Configuration Migration Tool Schema File Preparation using Adoxio.Dynamics.DevOps

The Adoxio.Dynamics.DevOps module includes several features that assist in the data management of Dynamics 365 projects. The features during export are the execution of schema file transformation rules, and unpacking exported data sets so that each record is stored in a single XML file. This blog post will cover the schema file transformation feature.

Introduction

The main efforts involved in performing data exports with Dynamics 365 are:

  1. Creating a schema file that defines the data to export – the entities and their fields – using the Configuration Migration Tool
  2. Running a data export against the source Dynamics 365 instance, using the selected schema file, and saving the exported data to the file system
  3. Preparing the data for storage in source control (e.g Git)

For background information on the purpose of the configuration migration tool and how to create schema files, refer to the following Microsoft documentation:

The rest of this post builds upon the basic knowledge of using the Configuration Migration Tool. Be sure to read the above links if you have not used it before. This is an advanced topic.

If you have not used the configuration migration tool and seen the schema file it creates, it would be helpful to do so before continuing with the rest of this article so that the concepts and information discussed make sense.

Schema file management

In an active development environment, creating a schema file can be a tedious process, because every time the environment’s schema changes in a way that affects the data that should be included during an export, the schema file needs to be updated. Some schema change scenarios that necessitate modifying the schema file include:

  • Creating and deleting entities
  • Creating and deleting fields
  • Creating and deleting relationships
  • Excluding existing entities and fields during export

In addition to schema changes, there are some data import behaviors that can be controlled by the schema file that necessitate modifying the schema file, including:

  • Disabling and enabling plugins on entities during import
  • Choosing the fields on an entity that are compared for record existence during import, with an update performed when a match is found

All of these changes may be manageable in a single person development effort if diligently keeping track of each individual change and modifying the schema file after every schema change, however in practice this is a very difficult and time-consuming effort to do correctly and consistently. This is even more difficult when working in a team with multiple people working against the same instance, which is very common in Dynamics 365.

The Adoxio.Dynamics.DevOps module assists in this regard by allowing for the definition of a set of schema file transformation rules. The transformation rules are a codified definition of modifications to a schema file and can be run repeatedly against updated schema files. The intended workflow to use is:

  1. Generate a schema file using the configuration migration tool that includes all of the entities from the source instance
  2. Define the schema file transformation rules that will adjust the entities and fields in the schema file
  3. Execute the schema file transformations against the generated schema file in step 1, to produce the updated schema file based on the rules in step 2
  4. Execute a data export with the configuration migration tool using the revised schema file generated in step 3

As additional schema changes are made to an instance, executing the previously defined schema transformation rules against a newly generated schema file will result in an updated schema file containing the latest components, while still respecting the codified transformation rules, without needing to spend a lot of time making changes within the configuration migration tool schema editing interface.

Example Settings

This is an example configuration of transformation rules as seen in the Full.ps1 export settings sample file:

CrmSchemaSettings = [PSCustomObject]@{
       Path = "$projectRoot\temp\export\schema.xml"
       Destination = "$projectRoot\crm\data\FabrikamFiber.schema.xml"
       EntityFilter = {$_.name -in 'account','contact'} # only export account and contact
       DisableAllEntityPlugins = $true # disable all plugins on all entities (or use DisableEntityPluginsFilter, don't use both)
       DisableEntityPluginsFilter = {$_.name -in 'account','contact'} # only disable plugins on account and contact during import
       UpdateComparePrimaryIdFilter = {$_.name -notin 'account','contact'} # all entities except for account and contact will be set to match on their primaryid field during import
       UpdateComparePrimaryNameFilter = {$_.name -in 'uom','uomschedule'} # only uom and uomschedule will be set to match on their primaryname field during import
       EntityUpdateCompareFields = @{
           abs_autonumberedentity = 'abs_entitylogicalname' # array of field names to match on
           incident = 'title','ticketnumber' # array of field names to match on
       }
       FieldFilter = {$_.name -notin 'createdby','createdon','createdonbehalfby','importsequencenumber','modifiedby','modifiedon','modifiedonbehalfby','organizationid','overriddencreatedon','ownerid','owningbusinessunit','owningteam','owninguser','timezoneruleversionnumber','utcconversiontimezonecode','versionnumber'} # include all but these fields on all entities
       EntityFieldFilters = @{
           contact = {$_.name -like 'adx_*' -or $_ -in 'contactid','createdon'} # export all fields that start with adx_ or the fields contactid and createdon
           team = {$_.name -notin 'businessunitid','teamid','name','isdefault'} # exclude all fields except businessunitid, teamid, name, and isdefault'
           businessunit = {$_.name -notin 'businessunitid','name'} # exclude all fields except businessunitid and name
           account = {$_ -in 'accountid','parentaccountid'} #  only export accountid and parentaccountid fields
       }
}

Each of the settings with a suffix of Filter is a PowerShell script block which is evaluated against each entity XML fragment in the configuration migration tool schema file. The evaluations that result in $true will cause the resulting entity XML to be transformed.

Setting Descriptions

Path specifies the file path of a configuration migration tool generated schema file. This file should be created by opening the configuration migration tool, adding all entities to the schema file, and saving the file to the specified file path. This file is considered temporary and is not intended to be saved to source control.

Destinationspecifies the new schema file that will be created after all of the transformation rules are executed. This newly produced file is what should be saved to source control. By convention, it’s helpful to store this file in the same folder where exported data is saved.

EntityFilter defines which entities will be kept in the schema file. When the condition result evaluates to $true, the entity xml will be kept; when $false the entity xml will be removed. The above example specifies that the account and contact entities will be retained, and all other entities will be removed.

DisableAllEntityPlugins defines whether plugins will be disabled or left as an enabled in all entities during import. the value $true will cause all entity plugins to be disabled during import, the value $false will leave all plugins as enabled during an import. This is a shorthand for the next setting.

DisableEntityPluginsFilter defines which entities will have their plugins disabled during import. This is a low level approach compared to the previous setting, to be used when selective control is desired over the entities that should have their plugins disabled during import. The example configuration shows how to disable plugins on only the account and contact entities.

In my experience, it’s preferable to have all plugins on all entities disabled during an import to avoid any runtime behavior from plugins adversely affecting a data import.

UpdateComparePrimaryIdFilter defines the entities which should match on their primary ID field during an import. Default import behavior is to match on the primary name field of entity during import, which is often not desirable because name fields can be changed by end-users in a system after the records are imported, whereas the ID field is hidden and can’t change which makes it more reliable to use when targeting previously imported records to update. The example shows how to define that all entities except for account and contact should match on their primary ID field during an import.

UpdateComparePrimaryNameFilter is similar to the previous setting, to define which entities should compare the primary name field for matching during import. This is the default behavior so this does not often need to be used, but could be if wanting to state the entities explicitly.

EntityUpdateCompareFields defines a dictionary of entities and the fields which they should compare against during import. This would be considered an unusual scenario, but is included for completeness with the comparison capabilities that can be defined by the schema file. The example shows two entities that needs to be compared by different sets of fields for existence during an import.

FieldFilter defines the fields that should be retained during export across all entities. This is most useful when used to define a common set of fields that should be excluded when an export is performed. For example in the sample configuration, it shows a set of fields that have the same name on all entities, and should be excluded because they are not important or always change between multiple systems.

EntityFieldFilters is similar to the previous setting, that allows for the definition of fields to be kept on a per entity basis. This is commonly used in tandem with the previous setting to define entity specific fields that should be kept or removed in the schema file.

Documentation for each of these settings can be seen by viewing help documentation for the Edit-CrmSchemaFile PowerShell function:

Get-Help -Name Edit-CrmSchemaFile -Full

Aside from the Path and Destination settings, all others are optional. Use only the ones that are needed for updating the schema file to suit your situation.

Executing the schema file transformation

Transformations can be executed using one of the following two invocations of the export script.

To execute as part of a complete export that includes all export steps, use the familiar export script execution:

./Export.ps1 -CrmConnectionName Online -ExportSettings Full

To execute as the only action taken during the export script, include the -Action parameter:

./Export.ps1 -CrmConnectionName Online -ExportSettings Full -Action Edit-CrmSchemaFile

To be continued

Upcoming blog posts will cover performing data exports given the prepared schema file and saving the exported data to the file system and Source control.