Overview: Data Exchange Framework and Sitecore Connect for Microsoft Dynamics 365 for Sales

Index

If you're here, congratulations: you're probably working on an ambitious and exciting project. This post is a high level overview involving the following technologies:

• Sitecore 9.3
• Data Exchange Framework 4.0 (docs, downloads)
• Sitecore Connect for Microsoft Dynamics 365 for Sales 4.0 (docs, downloads)
• xConnect / xDB

You use the Data Exchange Framework to model processes that synchronize data between multiple systems.

At the practical level, DEF is comprised of:

• Core database items
• Master database items
  • Settings
  • Templates
  • Branch templates
• DLLs
• Web.config adjustments regarding <dependentAssembly>
• Sitecore config files

Pipelines are how data is exchanged between systems. The pipeline batches are what actually run. Each pipeline batch has 1 or more pipelines, and each pipeline has one or more pipeline steps.

Pipeline batches can be triggered in many ways:

• Manually
• Programmatically
• Marketing Automation AKA "Data Exchange Framework Activity for Marketing Automation" (see downloads page)
• Scheduled tasks
• Web based push notifications
• Remote SDK / Tenant Service (docs)

The remote SDK is worth expanding upon here. The Sitecore docs say:

Data Exchange Framework Remote SDK allows pipeline batches, pipelines and other framework components to run outside of the Sitecore server. This can significantly reduce the load on your Sitecore server by moving much I/O and other processing to a separate machine.

When using the remote SDK, Sitecore's role in synchronization processes is limited to configuration. Sitecore items are used to configure framework components, but the framework components themselves run within the application domain that is using the remote SDK.

This connector extends DEF to facilitate synchronization of data between Dynamics and Sitecore. Both Dynamics 365 and on-premise versions of Dynamics are supported.

The Dynamics connector is comprised of:

• Master database items
  • Settings
  • Templates
  • Branch templates
• Sitecore config files
• Web.config adjustments regarding <dependentAssembly>
• DLLs
• JSON file (called the "xConnect collection model")
• Staging database (used for quick and efficient temporary storage during the synchronization process)

For comparison, another similar connector is "Sitecore Connect for Salesforce CRM".

Note: going forward, I will often use the term DEF to refer to both DEF and the Dynamics connector.

Once you install the Dynamics connector and create a new Data Exchange Tenant, you will see the following out-of-the-box pipeline batches:

In short:

• Pulling Dynamics contacts into Sitecore
• Pulling Dynamics marketing lists into Sitecore
• Pulling Dynamics "activities" into Sitecore (more on this in a following section)
• Pushing xConnect contacts to Dynamics
• Pushing xConnect email events to Dynamics

It might be a good idea to consider using DEF when:

• You want to manage and modify sync operations via items in the Content Editor
• You want to adjust connections on the fly with minimal or no code changes
• You have relatively simple architecture, such as XP Single (where CM and CD are the same instance) sitting on a virtual machine
• You want a lot of functionality in a short amount of time
• You are okay with using most of the functionality as-is, but doing custom overrides or additions when needed
• You want inspiration and ideas for your own custom implementation

In practice, you are almost certainly going to need a highly technical individual (developer) to get these running and configured properly. Although DEF is highly robust, it's not quite as simple as a content author installing a package and configuring a few items. It doesn't take much for your use cases to require some level of special configuration or customization.

You probably shouldn't use DEF when:

• You want full control
• You need a high level of test coverage
• You have a simple use case
• You don't need a comprehensive visual interface for managing your connections / flows

DEF is an impressive feat of engineering, but remember this quote by Tony Hoare:

Inside every large program is a small program struggling to get out.

Taking a fully custom code approach will always be more lightweight. There is also far more documentation, tutorials, and code samples for custom code as compared to content for configuring / customizing DEF. When I first started working with DEF and the Dynamics connector, I was surprised at how often Google searches returned no results. For that reason, my posts should allow me to lay claim to numerous long tail keywords. 😉

Make sure that your team has an excellent understanding of what data you want to sync between your systems and why. Challenge assumptions about your requirements, all the way to first principles.

For example, while Dynamics is powerful and can be customized to do virtually anything, that doesn't mean that you should. Microsoft and community support may be limited for your use case. Dynamics may be more cumbersome to work with and may cost more for storage and processing as compared to using other more specialized tools.

One real life example is that although Dynamics supports use cases involving "Contacts" or "Customers", Microsoft has specialized offerings which may be more suitable for your requirements. Particularly, they offer a "Customer Insights" or "Customer Data Platform".

Based on how Microsoft describes CDPs in general, it sounds like something worth researching if you haven't already:

A customer data platform (CDP) is marketing technology that pulls together customer data from multiple sources, then shares it with other applications.

Microsoft's CDP offering encompasses:

• Personalization
• Developing a wholistic view of customers
• Unifying fragmented data
• Deriving actionable insights
• Measuring customer activity (multi-channel activity from apps, websites, mailing lists, etc.)
• Ingesting data from multiple sources
• Contact merging (de-duping)

It's crucial that you understand the entire lifecycle of your sync process. It's not enough for a technical analyst on your team to come up with a flow diagram. The flow diagram also needs to be informed by what you actually end up building / configuring in your pipelines.

Questions to ask:

• What should happen if x step fails?
  • Retry?
  • Send to log?
  • Alert someone?
  • Wait?
  • Fire event?
  • Fail and exit?
• What should happen if x step times out or takes longer than expected to process?
• What should happen if Dynamics goes down?
• What should happen if xConnect goes down?
• What should happen if CM goes down?
• What should happen when a data entity does not pass a data validation as it's being processed?

DEF provides many avenues by which you can:

• Send information to logs
• Retry failed steps
• Override default functionality
• Detect failed validation conditions

For example, certain types of pipeline steps contain configurations such as:

If PII is a concern, make sure you understand Sitecore's default settings when it comes to storing / displaying / searching for contact information:

By default, it is not possible to search for personally identifiable information (PII) such as contact name. However, an administrator can enable indexing of PII sensitive data. By default, search is also disabled for anonymous contacts, because this operation consumes a lot of processing power due to the amount of data. However, it is possible for an administrator to enable indexing of anonymous data. You can only search for an exact name or email address. You cannot search for part of the name or email address.

Questions to ask:

• Should contact info be hidden in the Experience Profile UI?
• Should PII be stored in my index?
• Should Contacts be searchable?

From the docs:

xConnect is the service layer that sits in between the xDB Collection database and xDB index, and any trusted client or device that wants to read, write, or search contact and interaction data.

In other words, xConnect/xDB is what collects / manages / stores contact analytics information. I often refer to xConnect/xDB simply as xConnect or XC. As long as you've installed Sitecore properly and you're not having any certificate issues, with only a bit of setup, xConnect generally works beautifully.

When syncing XC with Dynamics, it's important to understand the "core collection model" that ships with XC:

Put into words, what we care about are these things:

• We have many Contacts
• Contacts can have many Identifiers (username, Dynamics ID, etc.)
• Each Contact can have many Interactions (in the context of this post, Interactions correlate to web sessions)
• Each Interaction has one or more Events
• There are various built-in Event types

All of this data is going to be stored and split within your xDB Collection database shards.

First make sure xConnect is working with no major errors in the logs and ensure that your xConnect certificate setup isn't going to cause problems down the road. See this post for more information.

Download the installation guides for DEF and the D365 connector (links are in the #Overview section). The installation steps are relatively straightforward.

Check out my Stack Exchange question about issues connecting to Dynamics. It contains comprehensive information about how to get connected successfully.

Some of the common connection errors are:

I found item serialization to be helpful while working with DEF in order to keep track of the changes I was making. I recommend it while you're still getting familiar with it.

DEF items have a verbose naming scheme, and serialized item paths will hit the path length limit in Windows. If you're using TDS, see this post about how to use aliases to shorten the paths.

Otherwise, be mindful of the types of items that should be serialized and updated and which ones shouldn't. For example, pipeline batch items should be deployed once and not automatically synced afterwards because the items contain information / references to the last pipeline run (timestamps, logs, etc).

Pipeline batches have options for the level of logging verbosity, as well as an option to include exception trace and telemetry information.

By default, DEF will log pipeline results to individual files in /App_Data/Logs/DataExchange/YOUR_DEF_TENANT_NAME/.

These log files can be large, and over time you could run into disk space issues, so consider setting up a cleanup task.

If your upstream environment is in Azure, consider integrating the DEF logs with Application Insights. Sitecore has a resource for this called AppInsights logger for Tenant Service.

DEF has a Write to Log Pipeline Step which is useful for quick debugging. Like most things in DEF, you can easily customize the code that powers this pipeline step. One reason you may want to do this is to prepend / append a string to your outputs to make them easier to spot in a large log file.

There are numerous built-in filters for Contacts, Marketing Lists, Activities, Events, and more. The built-in options depend on which Provider we're configuring for (Dynamics, xConnect, etc). Understanding how filters work and how to set up custom filters is crucial. Be sure to read through the docs.

You can use Filter Expressions to create filters with logic that's as complex as you need it to be using AND/OR logical operators. You can even nest them if you need to.

Below is a filter expression called Contact is My Test User. This uses AND logic to check the incoming Contact from Dynamics to ensure that it matches a specific Dynamics Contact ID and Dynamics Contact type. These kinds of filters will be crucial for you during development so you can run pipeline batches rapidly without syncing all your data, which could be millions of entities.

Here's how you would filter for the out of the box PageViewEvents. All three filters shown use the default Event Type Value Expression template {0095E139-E4CB-4F18-A07C-AE01C2062996}.

Modify the queries below as needed. Remember to run queries on all your shards.

Identifiers are stored as VARBINARY(700). The query below will convert the value to plain text for easier querying.

You can also view the values in plain text by copying the cell value, removing the 0x portion, and pasting into a hex-to-text decoder.

Once a contact has been loaded into Sitecore, the only way to remove them is programmatically or via SQL (not recommended except in local environments).

This Sitecore-endorsed repo has an example of how to implement programmatic deletion of contacts.

To remove all contacts locally run the following SQL commands (not recommended if you are storing XC contact IDs in Dynamics as the relationships will be broken):

Below is a typical example of what you'll see in an Interaction record's Events column.

This Interaction was generated by visiting a single page in a private browsing tab and then ending the session immediately after.

Note how the top PageViewEvent is the parent of the second custom event and also the parent of the last event. Expect to see many events containing Request took xxx ms to complete while developing. Sitecore automatically fires these.

There's still much to demonstrate. More posts on DEF / Dynamics / xConnect will follow. Hope this one helped you.

LAHWF,

Marcel