Lately I have been working with dual-write in different projects. In this post I will give an introduction to the subject from a technical perspective as well as functional and hopefully bring some clarity to what’s behind that magical door. Note that some parts of this post will be outdated when One Dynamics One Platform and unification of admin centers become reality. I will then update with a link to a new post.
What is dual-write?
Let’s take this from the beginning…
Ever since the Common Data Model (CDM) was announced, one of its purpose has been to enable easier integration and to make it simpler for us to talk to each other. In the xRM world there were accounts, on the ERP side, e.g. Dynamics 365 Finance and Operations (F&O), there were almost accounts (rather customers, vendors, prospects).
This is the announcement when Common Data Service (CDS) became GA. CDS was the service leveraging the possibility to store data in the cloud, CDM was the model, i.e. the definition of tables (such as account and contact) and relationships in between. Common Data Service was later rebranded to Dataverse. CDM still exist as a concept and name.
Dataverse, now we are getting closer to the subject. Dataverse is the part of Power Platform which lets you build data models, store data in the cloud and more. It is the back-end for model-driven Power Apps apps, such as the Dynamics 365 CE apps. The CE apps, that is also one step closer to the subject.
F&O does NOT have its data in Dataverse yet and dual-write is a way to sync data between F&O and Dataverse and the Dynamics 365 CE apps. Dual-write became generally available in May 2019.
Sync information between F&O and Dataverse
Dual-write, it’s the OOB infrastructure from Microsoft for setting up integration between F&O and Dataverse with Dynamics 365 apps enabled. Dynamics 365 Sales, Customer Service and Marketing are all a part of the traditional “CRM” application suite from Microsoft. The CE apps also include Field Service and Project Operations. To summarize, the following apps are supported for dual-write:
- Dynamics 365 Sales
- Dynamics 365 Customer Service
- Dynamics 365 Marketing
- Dynamics 365 Field Service
- Dynamics 365 Project Operations
There are dedicated pages in Microsoft Learn for Project Operations dual-write integration and Field Service dual-write integration.
The OOB data model in Dataverse and the additional OOB data models coming with different deployed solutions/products are extendable. You can create a custom app and extend the OOB data models and integrations, but the dual-write concept still requires Dataverse with Dynamics 365 apps in the bottom. If you extend on the Dataverse side, make sure you do it in a development environment and work in a solution.
Dual-write is one of the building blocks in the concept F&O 💙 Power Platform, the others being virtual entities, business events, data events and “add-ins”. Today it all starts by enabling Power Platform Integration for an F&O environment.
Technical perspective
Enabling Power Platform Integration for an F&O environment, that is something that is done though a wizard in Lifecycle Services (LCS) today. There are different ways by the way, enable during F&O deployment or after. Through a wizard, that sounds like an easy job, but there are certain things you need to be aware of. Such as establishing an environment strategy and awareness of the consequences when linking F&O to Dataverse and a thought-through plan about what OOB integration scenario you ant to start from. More about that in my next post.
Today in Power Platform Admin Center (PPAC) you will find the Power Platform environments automatically created for F&O environments (without Dataverse at first, until you choose to link environments). In PPAC you can create a new Power Platform environment, including a Dataverse database, to link to and use instead. For a certain Dataverse environment linked to F&O you will be able to see that F&O environment it is connected to. It’s also in PPAC that you manage different Power Platform and Dataverse Settings.
In the future, F&O environments and linking to Dataverse this will be administered in Power Platform Admin Center (PPAC). That’s the One Dynamics, One Admin experience.
Dual-write Building Blocks
Let’s have a look at the building blocks of dual-write. It’s basically a set of Dataverse solutions installed in the Dataverse environment. There is a dual-write core solution, containing the table maps foundation. Table maps are an essential part of dual-write (see this mapping reference).
The sample integration scenarios from Microsoft each contains a set of table maps. Each table map defines the mapping between a certain table in Dataverse and its columns and an entity and fields in F&O. Filters for what rows should be synced can be defined (e.g. only sync Accounts which are of Relationship Type Customer). These table maps are setup in F&O under Data Management, Dual-write, which can be seen as a “Dual-write UI”. It’s really just a UI for managing the table maps, they are actually stored in Dataverse. ALM for table maps is equal to Dataverse Solutions.
So, Dataverse is an ALM enabler, it’s on the Dataverse side (in Maker Portal) that you create your own Solution and add your in F&O modified table maps. A table map solution (i.e. a solution to which you add table maps) can contain your modified and custom table maps. A Solution can contain your extended Data Models and Business Logic on the Dataverse side too.
Depending on what sample scenario you want to start from you will be installing a set of different Dataverse solutions from Microsoft. Some of them will contain new tables, new layers to some of the OOB tables (e.g. new relationships, columns, views and additions to forms) and business logic (low-code such as business rules as well as code e.g. Dataverse plugins). Some of the solutions will contain table maps.
Logs exist on both Dataverse side (Plugin Trace Log) and F&O side (Data projects & in DualWriteErrorLog). It can be tricky for a Dataverse person to find the F&O logs and vise versa. Accessing logs are described in Microsoft Learn.
F&O dual-write UI and Dataverse as ALM enabler
Managing table maps configuration in F&O and the ALM story in Dataverse, how does that work? Let’s go back to after we have linked the F&O environment to a Power Platform / Dataverse environment.
When you have linked environments, depending on what choices you have made during the linking process you will have a different set of dual-write related solutions installed in your Dataverse environment. It’s now time to go and install a few more in order to get started with your specific integration scenario. ALWAYS FOLLOW Microsoft Learn setup instructions for your integration scenario.
Those setup instructions guide you both regarding what solutions to install as well as what table maps to start from. The instructions are also updated continuously and an important note section might have been added the next time you install the same set of solutions. OOB solutions development happens fast with this product.
When the solutions have been installed, you head over to F&O and the dual-write UI (Data Management, Dual-write). The first time you will need to do some initialization tasks such as choosing what F&O companies you would like to sync to Dataverse, apply solutions (choose the Dataverse solutions with table maps to use) and do a connection health check. Applying solutions is something you will be doing later on as well, each time you deploy table maps to an environment.
In the dual-write Admin UI you can view, create and modify table maps and set filters. You start from a table map when you want to do init sync or when you want to start, stop or pause the integration for a certain pair of F&O entity/Dataverse table. Follow the Microsoft Learn instructions and start the table maps specified for the chosen integration scenario. When you start a table map, you have the choice to do an initial sync (use the table map and sync for the first time to to an initial load of data).
The below picture shows what to do where. In F&O you do the work with table maps and on the Dataverse side, in Maker Portal, you start the table maps ALM journey. You can also extend the Dataverse side with tables and business logic. In PPAC you manage Dataverse settings, one setting in the center of attention for dual-write is the Plugin Trace Log. It’s also in PPAC that you can check for updates for the OOB dual-write related solutions!
Reset Link is used to re-establish the connection between F&O and Dataverse, you will need to use it e.g. when a copy from production has been done in a UAT environment. Before it was called unlink/link and you had to do some more manual steps (empty tables/entities), now Reset Link does that for us.
Data from F&O is exported in an F&O Data Project during init sync, logs can be found in F&O under Data Projects. Logs for the live sync can be found in a special F&O entity. There are plugin from the dual-write related solutions and you can turn on the Plugin Trace Log for the Dataverse environment and view the logs if necessary (i.e. in order to find more detailed error messages when trouble-shooting). Actually, the view part is best done with XrmToolBox and Plugin Trace Viewer.
There is more to both the dual-write UI and especially to Maker Portal than what is shown above. In the Admin UI we also have the possibility to do a “health check” for the connection and for the individual table maps, set alerts and view environment details. In Maker Portal, I will not even begin to mention it all. That’s the entrance to the Power Platform world, where you can create apps, chat bots, setup automation and more.
Functional perspective
Dual-write and data sync may sound very technical, but of course there is a functional perspective too. You will have business requirements to take into consideration, that one essential part of the functional perspective. Then we also have the OOB template integration scenarios. In order to use these, you need to have an understanding of how the OOB functionality works and in addition, taking the business requirements into consideration, how the applications are used (or should be) in your specific case. Understanding the CE side and the F&O side as is, that is a challenge of its own. Dual-write also has its own related business logic. There might be existing custom functionality to take into the picture.
Update your Dataverse apps so that it fits your business processes and make sure you understand both the intended use from Microsoft side as well as the needs and purpose of the integration in your specific case. Adjust the apps according to that. In Microsoft Learn you can read about GAB and e.g. to add Party table to the sitemap of you app, in the case where you are starting from the GAB templates. Apply this thinking to other scenarios as well.
Dual-write is a combined effort
Dual-write is meant to provide OOB integration possibilities between F&O and Dataverse (and the Dynamics CE apps). It is supposed to bridge the technical side of F&O and Dataverse. However, one thing must not be forgotten. Traditionally you are either an F&O person (which can be further split into functional or X++ / tech enthusiast as I understand it) or a “CRM/xRM” person (which also can be split into different categories of course) person. There are very few people out there having deep knowledge of it all.
The setup, configuration, extension and maintenance requires an understanding of both F&O and Dataverse / the CE app you intend to use. It’s simply a combined efforts job. I know that I, coming from the Dataverse side, need the expertise of someone from the F&O side. I need it to understand how everything is setup in F&O. I need an understanding for the business processes involved and how it all is meant to work. I sometimes need to for the technical parts as well, to understand where to find the logs and do more thorough trouble-shooting.
I can imagine an F&O person would need help understanding the table map ALM story, Dataverse solutions and best practice for it as well as all hidden functionality coming with the Dynamics 365 CE apps. Also for accessing and understanding the plugin logs. On top of that, together we need to figure out and learn how the business logic coming with dual-write plugins are supposed to work.
F&O meets Dataverse. But also, F&O person meets Dataverse person. We need to collaborate on this, on several levels. Dataverse and F&O sides – we need to combine our super powers and become stronger together!
Learn more
If you are interesting in providing feedback to Microsoft and to learn alongside other people, you can join the Insider Program & Dual-write yammer group. The dual-write start page in Microsoft Learn contains a great list of links. Here are a few of my favorites.
Microsoft Learn
What’s new or changed in dual-write Dual-write release info
Power Platform Integration Describes Power Platform Integration for F&O
Guidance for dual-write setup Sample scenarios for new/existing F&O/Dataverse. Is Bootstrap needed?
System requirements and prerequisites Step-by-step instructions for enabling dual-write
Init Sync Guidance Matrix To Init Sync or not to Init Sync
The Company concept About the F&O Company concept
Dual-write setup from Lifecycle Services About linking F&O to Dataverse
Dual-write mapping reference Explaining filters and dual-write templates
Dual-write ALM Explaining table maps ALM
Customization guidance for dual-write Guidance and dual-write best practices
Dual-write FAQ Common Questions and Answers
Other Microsoft resources
Power Platform Admin Center Integration with LCS – TechTalk
One Dynamics One Platform – TechTalk
Dynamics 365 Cross Apps Workshops (Register in order to see the recordings)
Microsoft Dynamics 365 FastTrack TechTalks: Dual Write Framework
Photo by Kyle Bushnell on Unsplash
Carina M. Claesson 