Migrating data to a live on-premises CRM system?

While working on one of the project, we were required to move data from legacy system into a live production system with limited amount of black out window. One can calculate the time required based on number of records, server configuration, etc. and can even have a buffer. But you never know when Murphy will play his role.

Below are few points that, if considered can help Murphy stick to its seat and don’t show up. (Just some pre-checks that helps the processJ)

  • Confirm disk space on SQL, web and application server. Have some buffer space as logs will grow.
  • As your target system is a live production system, surly there will be some scheduled maintenance jobs running to maintain server health as part of disaster management. These jobs are life savers but problem with them is that these are scheduled in down time as they consume high resources. And unfortunately this is the only time when we can perform our import. Thus you may have to consider pausing the maintenance jobs. But do remember to turn them on once done with the import. Some of the resource consuming jobs are: consistency check, database backup, async-operation cleanup, POA, etc. Pausing this job helps the import utilize maximum available server resources.
  • Check the database log size and clear. The log will grow with the import and if it reaches the maximum available threshold the import processes throws timeout errors. Also check the shrink process, preferred if simple.
  • Check for CPU and Memory usage on SQL, web and application server.
  • Check for any blockage on SQL server, if any script is blocking or slow running queries.
  • Clear AsyncOperationBase table.
  • You may run into scenario where it is required to restart the SQL service. Make sure that this do not affect any other process. Also in case of NLB, upon restarting the SQL service switches the active node. Thus you will also have to consider that node for performance check.
  • You may have to disable the user logging and turn off the workflows and plugins.

And last but not the least is FULL database backup before start of the process. Also you make to do this in several passes due to amount of data and limited amount of down time. Identity the steps that can be performed outside black out that do not affect live system. This provides some extra time for completing the critical steps.

I would recommend to do the import in multiple small passes which helps in keeping the buffer and reduces the chances of breaking things or running on edges. After all “Rome was not built in a day”.

These are some of the steps that helped me. As always, these may not match exactly to your requirement but some of them will surly. And I don’t guaranty of anything from the steps as risk will be yours as it’s your production system.

If you have anything to add, please write in comment and I will update the content. Thanks!

DID YOU KNOW? What happens to Audit History on performing database level update?

062915_1445_DIDYOUKNOWW1.jpg

Note: This is all about UNSUPPORTED database level updates. These kind of updates are not recommended (unless you are left with no options).

Though any kind of database level updates are not recommended in CRM, there are scenarios where we are required to do so. I had one such requirement where I was required to perform a simple update (setting two options field value to Yes) for millions of records. Doing same using a supportive way (bulk update or bulk workflow execution) used to take days due to CRM service calls but same updates using SQL script was done in few hours.

I had auditing enabled for these records in CRM, but as this was unsupported db level update I was not expecting any help from audit history. What was expected form auditing was either of below:

  1. Change track as new update request in audit history. (If there is any kind of trigger return at db level to maintain auditing)
  2. No updates at all in audit history. (Assuming auditing are maintained by some kind of workflow/plugin that trigger at UI level)

Unfortunately, none of this happened and what happened was not at all audit friendly or expected. This is what happened:

The change made by SQL script was tracked at create event itself in audit history. But only the field value change was tracked and not the change date/time. The change date/time was still showing when the record was created.

Though these kind of changes are not recommended, but I still feel that in some cases these are required and this behavior of auditing will affect the consistency. I feel that things should be either completely baked or not baked at all than having them half baked. But again this is only what I feel. J

Example:

  1. I created a Contact record on 7/6/2015 at 11:52 AM with ‘First R~~~~~ Created’ set to ‘No’. Audit history shows something like below.

     

  2. Now on 7/6/2015 at 12:00 Noon, I update the value for ‘First R~~~~~ created’ to ‘Yes’ using a database level update script. Ideally there should have been a new entry made in audit log to track this change and there should be no change made to existing audit history records. But CRM updates the changed value in create event itself without even tracking the change time.

 

Hope this helps and thanks if you have come all the way down here reading the article.

DID YOU KNOW? Value for overriddencreatedon is swiped with createdon at database level

One of my college recently had a requirement to import few records into CRM with back dated values. Isn’t it a simple task? Just map/set the back dated value for ‘overriddencreatedon’ field while create/import of records. But has anyone checked what happens to the value for ‘createdon’ field?

All views and fields in CRM displays ‘createdon’ field and not ‘overriddencreatedon’. Thus, wherever we set value for ‘overriddencreatedon’, the value is set to ‘createdon’ field and the actual date/time when the record is created into CRM is set to ‘overriddencreatedon’.

E.g.: I’m creating a record on 2015-06-16 11:45:23.000 UTC into CRM with value for overriddencreatedon set to 2015-06-16 11:44:39.000 UTC. Upon completion, if you query the database you will see that the value what I set to overriddencreatedon is set to ‘createdon’ and the value for ‘overriddencreatedon’ is replaced with actual date/time when the record was created.

Update request at 2015-06-16 11:45:23.000 UTC:

Createdon = null

overriddencreatedon = 2015-06-16 11:44:39.000 UTC

Result:

Createdon = 2015-06-16 11:44:39.000 UTC

Overriddencreatedon = 2015-06-16 11:45:23.000 UTC

date

Note: Setting ‘overriddencreatedon’ on update of record is not supported.

Top 10 Data Migration Traps

While planing for the data migration item for one of my project, found this great article by Bruce Buxton which explains top 10 data migration traps. It was good to know as prevention is always better than cure 🙂

http://www.xrmconsultancy.com/2013/10/top-10-data-migration-traps/

Invoice and Invoice Line data import in Dynamics CRM

Recently I found that many people have issues importing old Invoices either from legacy system into Dynamics CRM. This is manly in scenarios where Products are not maintained in CRM. In this case import of Invoice Line throws error as the data import template for Invoice line do not have relationship with Invoice. Below are the steps that can be followed to fix the issue.

Invoice lines is a relationship between invoice and products. Thus if you need to add invoice lines, you need to add products using the “Product.xml” template file. But in our case as we don’t maintain Products in CRM, while importing Invoice lines, add invoice number of imported invoices in CRM in a separate column and select the Product as write-in in import file.

For importing Invoice lines, you need to download the “Invoice Product.xml” template file.

All the mentioned templates can be downloaded from “templates for Data Import” in Data Management section in Settings.

  1. Export Invoice Product.xml
  2. Add Invoice Line data in the xml with below values.
    1. Set “Select Product” column value to “Write In”.
    2. Add a new column “Invoice ID” and set the value to Invoice ID field of respective Invoice.
  3. Save and import the file.

4. Import the above xml file.

5. During Import wizard, on Review Mapping Summary Page, click on Edit.

6. In Map Fields, for Invoice ID row click on the look up reference and select “Invoice ID” field and click Ok.

7. Click Next and progress with the import process.

Probably issue with many users follow is at step 7. Invoice ID field is by default map to Invoice name and Invoice GUID field.

Hope this helps!

Importing Resolved cases in CRM 2011

Many a times we have a requirement to import historical data in CRM. Requirement looks good but when it comes to importing historical cases we are in problem. And in more problem if these are resolved cases coming from some non dynamics CRM product.

Import of Active cases is done quite well, but for resolved cases we get a error. Even if we set Status to Resolved and Status Reason to Problem Solved. Either we get an error that State Code is invalid or the record is imported but the Status could not be changed.

A workaround that worked for me was to resolve the case problematically by using CloseIncidentRequest message. Another workaround is to set up a dummy attribute on case record and have a workflow process on create of record which checks for the value of this dummy field and sets the status accordingly. But this will not work in case if resolution date are required to be kept same (old values).

https://social.microsoft.com/Forums/en-US/e0f19314-85a6-48cf-b3b4-393acaed5356/importchange-case-resolved-date?forum=crm

https://social.microsoft.com/Forums/en-US/3b76875f-0c7b-4c48-a38b-53b0a5271bca/importing-cases-that-are-resolved?forum=crm

 

**IMPORTANT**

It is recommended that you do an import process by a administrative user which has a timezone set to GMT/UTC.

The reason is in 2005 (2005 revision to dates of observance) the date when Daylight Savings Time starts and ends in the US was changed. As a result, if you import the data back in for previous years as a user that is not on GMT, some of the records will be off by an hour.

http://en.wikipedia.org/wiki/Daylight_saving_time_in_the_United_States

Migrating history information while data import in Dynamics CRM

When you import historical data in Dynamics CRM, you may want to override property information like Created By, Created On, Modified By, Modified On associated to the record. It seems possible because CRM has provided security privileges called “Override Created on or Created by for records during data import”. However, simply update the appropriate Security Role to enable this privilege does not completely meet all requirements to override such fields as the name sounds.

The truth is, only “Created On” field can be overridden and only if the ‘allow creating duplicate records’ option is checked during the import process. In other words, we cannot update this field on existing records. Only creating new records does work for overriding “Created On” field.
While the Created By, Modified On, Modified By fields cannot be updated once the data is imported even though the user has full privilege on “Override Created on or Created by for records during data import”.

A workaround for such request can be to have a plugin that fires on pre-create of the entity and set the values for remaining fields problematically.

 

**IMPORTANT**

It is recommended that you do an import process by a administrative user which has a timezone set to GMT/UTC.

The reason is in 2005 (2005 revision to dates of observance) the date when Daylight Savings Time starts and ends in the US was changed. As a result, if you import the data back in for previous years as a user that is not on GMT, some of the records will be off by an hour.

 

Note:

  • When one have a list of related entities for which data import is required, order in which data for these entities are imported is important. In some cases, you have many relationships, and you cannot practically determine the order in which they should be imported. In this case, do a two-pass operation.
  • It is recommended to map the record ID/GUID fields when the records are imported. If you map the GUID fields, all relationships will work when imported.
  • It is recommended to map createdon to overriddencreatedon. You can only populate this when the record is created, and this is what will set the created on date for the record to match the original version.
  • There are some fields that you do not want to map, such as version number, import sequence.
  • There are several fields that you cannot import like modified by, created by, modified on.
  •  Custom data cannot be reimported for closed opportunities.