CDS, Data Integration, Model-Driven Apps, Power Platform, Virtual Entities

Virtual Entities – A forgotten feature which is still alive

Virtual Entities have been on my radar since it was released back in July 2017. I have not used it in any customer project yet though and you do not hear so much about it nowadays (as indicated in the title of this blog post). However, last year in October there was a session at eXtreme 365 where Matt Barbour said “Virtual Entities are not dead”. My friend Jonas was there to witness it live. Great to hear that Microsoft are doing investments within this area, i.e. it is not a completely forgotten chapter in the product portfolio.

I started to explore Virtual Entities last year and now it is time to document my findings. This blog post is the first one is a series of four blog posts. In this one I will give an introduction to what Virtual Entities are and describe the building blocks. I’ll also share the best links I’ve found for reading more about this subject.

In the next one I will give an example of how to create what is called a Custom Data Provider. That is what you need to develop if you cannot use one of the built-in Data Providers when you want to set up a Virtual Entity. It can be compared to pre-defined connectors for Power Apps – Canvas Apps and Custom Connectors. More about that comparison later in this blog post. In the third blog post we will look at how to register the Custom Data Provider and in the fourth and last blog post we will put the pieces together in order to get a functioning example of a Virtual Entity.

What are Virtual Entities?

Let us start with some words from the Microsoft documentation.

A virtual entity is a definition of an entity in the Common Data Service platform metadata without the associated physical tables for entity instances created in the Common Data Service database. Instead during runtime, when an entity instance is required, its state is dynamically retrieved from the associated external system.

Virtual entities are entities which are created and configured in the CDS as any other custom entity – with some extra steps, e.g. you mark a checkbox “Virtual Entity” and you specify a Data Source. What differs the most from the custom entities we are used to is that its data is not stored in the CDS, instead the data comes from another source.

The purpose of Virtual Entities is to provide a way to integrate data from other systems into your Model-Driven Apps without having the data stored both in the external system and in the Common Data Service. The end user experience should be like that there is no difference between the Virtual Entities and the other entities in the system. However, there are some limitations. For instance, the data for the Virtual Entities will be read-only and always organization owned.

Picture from the Microsoft docs. For On-Premises, but still good I think.

The Building Blocks

In the documentation you can read the following:

Each virtual entity type is associated with a virtual entity data provider and (optionally) some configuration information from an associated virtual entity data source.

Trying to register a Data Provider and not specifying a Data Source Entity, gives me the below. Not very optional. I assume what Microsoft refers to is if you want to add some extra configuration to the Data Source record – that is optional, but having a Data Source entity is mandatory.

Registering a new Data Provider without creating a Data Source entity

Taking a look at the On-Premises documentation gives me more information about the building blocks:

Virtual entities are made up of three main components, a data provider, a data source record, and a virtual entity. The data provider consists of plug-ins and a data source entity. The data source is an entity record in Dynamics 365 Customer Engagement (on-premises), which includes metadata that represents the schema of the connection parameters. Each virtual entity references a data source in the entity definition.

So a Virtual Entity is built up of three main components, a DATA PROVIDER, a DATA SOURCE record, and a VIRTUAL ENTITY. The Data Provider consists of plug-ins and a Data Source entity. There are already registered assemblies which comes with the platform and if you develop a custom Data Provider you will need to register both an assembly and a Data Provider.

I created a picture of my own, in order to make another visual representation of Virtual Entities.

Note! The Power Apps icon symbolizes a Model-Driven App in this case and the picture shows the building blocks of Virtual Entities as well as how the data flow using VE and not.

To the left we have Power Apps. In this case it is a Model-Driven App, since it is in our Model-Driven Apps we can use Virtual Entities. Then we have the Common Data Service (CDS) containing entities (standard/built-in entities and custom entities). There is a line that goes right through the CDS. To the left of that line – that is what is normally used for our Model-Driven Apps. We have the apps handling the UI and we have the data stored in entities in the Common Data Service. The end users can both view the data and update the data -> and it is updated in the CDS.

To the right of the line – that is what is used when you use one or several Virtual Entities. Then we have a Virtual Entity configured in the Common Data Service, we have a Data Provider, which e.g. is used when a user clicks on a virtual entity view or record in the Model-Driven App. The Data Provider contacts the external system and the data is retrieved from there and displayed in the Model-Driven App. The end user can NOT update data for Virtual Entities, it is a one direction flow.

Perhaps now you think, why the rocket? Well, you’ll see in my next blog post. Because then we will create a Custom Data Provider which connects to an API which gets us some deep space information. 🙂

The Data Provider is easier to grasp than the Data Source I believe. It consists of Retrieve and RetrieveMultiple plugins and it is simply the part which gets the data from the external system. When you register a new Data Provider in the Plugin Registration tool, then you get to create an entity which is the Data Source, which you saw in the picture earlier in this blog post – you cannot create a Data Provider without creating a new entity which is the Data Source. This is another entity than the Virtual Entity – the Virtual Entity you create yourself in the CDS as any other custom entity.

There are two built-in Data Providers, more about those in the next chapter. For now I will use the OData v4 Data Provider as an example and show you how it looks like in the Plugin Registration tool. If you open the tool you can see the following, there is an assembly registered and you can see there is e.g. a Retrieve and a RetrieveMultiple plugin in there. There is also a Data Provider registered.

The built-in OData v4 Data Provider in the Plugin Registration tool

About Data Providers

If you want to connect to an OData v4 web service, congratulations. As already mentioned, there is an OData Provider that you can use to connect with an OData v4 web service that accesses the external data. If the external data source is a Cosmos DB, congratulations, you get away with configuration only, i.e. a no-code solution. These two are the only Data Providers which are built-in.

When you want to configure a Virtual Entity using one of the built-in Data Providers, you need to create a new Data Source record, which is done from Settings -> Advanced Settings -> Administration -> Virtual Entity Data Sources. E.g. for the OData v4 Data Provider, it is in a record of a Data Source for that Data Provider that you specify the URL to the OData v4 endpoint. If you want to connect to some other data source for a Virtual Entity, then you have the possibility to do so by developing a Custom Data Provider.

In order to create a Custom Data Provider you need to be a pro developer though or at least have a pro developer friend. Luckily, I consider myself as both. Citizen and pro developer I mean. I am in between the two worlds, which is somehow convenient actually, since I then get to choose how to approach a requirement and I can go with no-code/low-code as far as that gets me and when I reach a point where no-code/low-code is no longer applicable I do not have to wait until a pro developer has time to help out. I just put on my pro developer hat and all is good.   

Virtual Entities vs. embedded Canvas Apps

Virtual Entities can be compared to embedded Canvas Apps, i.e. having a Canvas App embedded on a form in a Model-Driven App displaying data from an external data source. However, embedded Canvas Apps displays data for one record at a time – when the record is opened and you view the form. Virtual Entities provides a way to display several records at a time, not only related to a certain record. For instance, displaying them in a view from a menu item in the Sitemap of a Model-Driven App and to use the data almost as if the data was stored in the Common Data Service, though it is not. With other words, there is no need for unnecessary data replication.

With Model-Driven Apps you usually have the Common Data Service as data source (that is, if you do NOT use a Virtual Entity – then you could have any data source but sometimes it will require code as already pointed out). Canvas Apps on the other hand are more flexible, there are more (270+) pre-defined connectors and you can set up a custom connector without writing code, as long as you have an API with a good fit.

Ivan Ficko has written about these concepts and compared the two, I’ll provide you with the link at the end of this blog post.

Conclusions

Virtual Entities are used to display data from other systems in your Model-Driven Apps without having the data stored in the CDS. It might be an alternative for you if it is OK that the data is read-only and always organization owned. You should also take a look at the other limitations mentioned in the documentation from Microsoft.

If your source is a Cosmos DB or OData v4 endpoint you can use one of the two built-in Data Providers and get away with a Now-Code solution. If not, you can develop a Custom Data Provider.

Steps needed for setting up a Virtual Entity:

  1. If you are NOT using a built-in Data Provider, first develop a Custom Data Provider and register the assembly and the Data Provider.
  2. Create a Data Source record
  3. Create the Virtual Entity
  4. (Update the Virtual Entity with the Data Source – if you created the Virtual Entity before the Data Source Record and specified [None] as Data Source for that stage.)
  5. Add your Virtual Entity to the Sitemap of a Model-Driven App

I’ll provide you with visualization as well.

How to set up a Virtual Entity and use it in a Model-Driven App

To be continued

In the next blog post I will let you in on how to develop a Custom Data Provider. Until then – here is a great list of links. Those are links that I have gathered while exploring Virtual Entities.

First we have some documentation from Microsoft. Getting Started with Virtual Entities.

I assume you prefer working with the latest and the greatest which means you prefer NOT to work with Dynamics 365 CE (which nowadays refers to the on-premises version). Anyway, here is a link to the documentation about Virtual Entities but from the On-Premises point-of-view. Still, good to read to gain understanding for the overall concept. Here is also how to Add a Data Source using the OData v4 Data Provider

There is also a blog post from Microsoft from 2017, which describes Virtual Entities and gives an example of how to use the built-in OData v 4 Data Provider.

Ivan Ficko – the Dynamics Ninja – and his series of tree blog posts about Virtual Entities. These are three really good blog posts, do not miss to check it out if you want to know more about Virtual Entites. He also goes through each one of the built-in Data Providers and explains how to use them. You find the first one here.

As already mentioned, this Ninja has also written a great article about Virtual Entities vs. Embedded Canvas Apps. Go and check it out here.

I also found another great blog post within this topic, this one written by Jason Lattimer. He also has a great example on GitHub if you are looking for inspiration for creating your own data provider. Remember – I will also tell you more about how to do that in my next blog posts and provide you with another example.

This one is also really good to read in order to gain more understanding of Virtual Entities. For instance it explains different solutions to the fact that you need to use guids.  

A blog post from Alex Shlega about the same topic. We writes about it from an interesting angle, providing a proof of concept related to limiting access to Virtual Entities with the help of the RetrieveMultiple plugins.

Have you found any other interesting information, blog post, video or other community contribution related to this topic? Please let me know in a comment to this blog post.

3 thoughts on “Virtual Entities – A forgotten feature which is still alive”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s