Embeddings

Get started with Embeddings

Welcome to the tutorial for Embeddings in Dolittle, where you learn how to write a Microservice that event sources data coming from an external system. This external system is keeping track of the chefs in the kitchen, like a HR system. We’re going to be “freeing” this data by event sourcing with the help of Embeddings, so that other parts of the code can utilize it more freely.

Embeddings are like a combination of an Aggregate and a Projection. They have a collection of states (the read models) with unique keys (within the embedding). Whenever a new updated state is pushed/pulled from the external system, that updated state will be compared against its representative read model in the embedding. Whenever the states differ, the Runtime will call the embedding to resolve the difference into events. The embedding will then handle these events and modify its state to match the desired state.

The main point of embeddings is event source the changes coming from an external system. An embeddings read model exists so that we can commit the correct events and uphold its logic. Other event handlers, projections and microservices can then build upon these events.

Example of a embedding:

Diagram of embeddings

After this tutorial you will have:

  • a running Dolittle environment with a Runtime and a MongoDB, and
  • a Microservice that commits Events
  • an Embedding which creates events based on changes in the HR system.

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

Setup

Setup is the same as in the getting started tutorial.

Prerequisites:

Before getting started, your directory should look something like this:

└── Kitchen/
    ├── Program.cs
    └── Kitchen.csproj

Prerequisites:

Before getting started, your directory should look something like this:

└── kitchen/
    ├── .eslintrc
    ├── index.ts
    ├── package.json
    └── tsconfig.json

Start the Dolittle environment

Start the Dolittle environment with all the necessary dependencies (if you didn’t have it running already) with the following command:

$ docker run -p 50053:50053 -p 27017:27017 dolittle/runtime:latest-development

This will start a container with the Dolittle Development Runtime on port 50053 and a MongoDB server on port 27017. The Runtime handles committing the events and the embeddings, while the MongoDB is used for persistence.

Create the events

In this example, we want to event source the data coming from a mocked HR system. We want to keep track of hired employees, their workplace and whenever they retire. Let’s create 3 different EventTypes to signal whenever an employee is hired, transferred or retires:

EmployeeHired:

// EmployeeHired.cs
using Dolittle.SDK.Events;

namespace Kitchen
{
    [EventType("8fdf45bc-f484-4348-bcb0-4d6f134aaf6c")]
    public class EmployeeHired
    {
        public string Name { get; set; }

        public EmployeeHired(string name) => Name = name;
    }
}

EmployeeTransferred:

// EmployeeTransferred.cs
using Dolittle.SDK.Events;

namespace Kitchen
{
    [EventType("b27f2a39-a2d4-43a7-9952-62e39cbc7ebc")]
    public class EmployeeTransferred
    {
        public string Name { get; set; }
        public string From { get; set; }
        public string To { get; set; }

        public EmployeeTransferred(string name, string from, string to)
        {
            Name = name;
            From = from;
            To = to;
        }
    }
}

EmployeeRetired:

// EmployeeRetired.cs
using Dolittle.SDK.Events;

namespace Kitchen
{
    [EventType("1932beb4-c8cd-4fee-9a7e-a92af3693510")]
    public class EmployeeRetired
    {
        public string Name { get; set; }

        public EmployeeRetired(string name) => Name = name;
    }
}

EmployeeHired:

//EmployeeHired.ts
import { eventType } from '@dolittle/sdk.events';

@eventType('8fdf45bc-f484-4348-bcb0-4d6f134aaf6c')
export class EmployeeHired {
    constructor(readonly name: string) {}
}

EmployeeTransferred:

//EmployeeTransferred.ts
import { eventType } from '@dolittle/sdk.events';

@eventType('b27f2a39-a2d4-43a7-9952-62e39cbc7ebc')
export class EmployeeTransferred {
    constructor(readonly name: string, readonly from: string, readonly to: string) {}
}

EmployeeRetired:

//EmployeeRetired.ts
import { eventType } from '@dolittle/sdk.events';

@eventType('1932beb4-c8cd-4fee-9a7e-a92af3693510')
export class EmployeeRetired {
    constructor(readonly name: string) {}
}

Create an Employee Embedding

In this example, we want to events source the data coming from a mocked HR system by using Embeddings. Let’s create an Embedding that keeps track of the changes coming from the HR system by committing and handling of the events we made earlier:

// Employee.cs
using Dolittle.SDK.Projections;

namespace Kitchen
{
    [Embedding("e5577d2c-0de7-481c-b5be-6ef613c2fcd6")]
    public class Employee
    {
        public string Name { get; set; } = "";
        public string Workplace { get; set; } = "Unassigned";

        public object ResolveUpdateToEvents(Employee updatedEmployee, EmbeddingContext context)
        {
            if (Name != updatedEmployee.Name)
            {
                return new EmployeeHired(updatedEmployee.Name);
            }
            else if (Workplace != updatedEmployee.Workplace)
            {
                return new EmployeeTransferred(Name, updatedEmployee.Workplace);
            }

            throw new NotImplementedException();
        }

        public object ResolveDeletionToEvents(EmbeddingContext context)
        {
            return new EmployeeRetired(Name);
        }

        public void On(EmployeeHired @event, EmbeddingProjectContext context)
        {
            Name = @event.Name;
        }

        public void On(EmployeeTransferred @event, EmbeddingProjectContext context)
        {
            Workplace = @event.To;
        }

        public ProjectionResult<Employee> On(EmployeeRetired @event, EmbeddingProjectContext context)
        {
            return ProjectionResult<Employee>.Delete;
        }
    }
}

The [Embedding()] attribute identifies this embedding in the Runtime, and is used to keep track of the events that it creates and processes, it’s state and the retrying the handling of an event if the handler fails (throws an exception).

ResolveUpdateToEvents() method will be called whenever the current state of the embeddings read model is different from the updated state. This method needs to return one or many events that will update the read model so that it moves “closer” to matching the desired state. The Runtime will then apply the returned events onto the embeddings On() methods. If the states still differ, it will call the ResolveUpdateToEvents() method again until the read models current state matches the updated state. At that point, the events will be committed to the Event Log. If the On() methods fail, or the Runtime detects that the embeddings state is looping, the process will be stopped and no events will be committed. This means that events will only be committed if they successfully result in the states matching.

The ResolveDeletionToEvents() method is the same, except the resulting events have to result in the read model being deleted. This is done by returning a ProjectionResult<Employee>.Delete in the corresponding On() method.

The committed events are always public Aggregate events. The AggregateRootId is the same as the EmbeddingId, and the EventSourceId is computed from the read models key.

Unlike projections, you don’t need to specify a KeySelector for the On() methods. The Runtime will automatically calculate a unique EventSourceId for the committed events based on the embeddings Key.

// Employee.ts
import { CouldNotResolveUpdateToEvents, embedding, EmbeddingContext, EmbeddingProjectContext, on, resolveDeletionToEvents, resolveUpdateToEvents } from '@dolittle/sdk.embeddings';
import { ProjectionResult } from '@dolittle/sdk.projections';
import { EmployeeHired } from './EmployeeHired';
import { EmployeeRetired } from './EmployeeRetired';
import { EmployeeTransferred } from './EmployeeTransferred';

@embedding('e5577d2c-0de7-481c-b5be-6ef613c2fcd6')
export class Employee {

    constructor(
        public name: string = '',
        public workplace: string = 'Unassigned') {
    }

    @resolveUpdateToEvents()
    resolveUpdateToEvents(updatedEmployee: Employee, context: EmbeddingContext) {
        if (this.name !== updatedEmployee.name) {
            return new EmployeeHired(updatedEmployee.name);
        } else if (this.workplace !== updatedEmployee.workplace) {
            return new EmployeeTransferred(this.name, this.workplace, updatedEmployee.workplace);
        }

        throw new CouldNotResolveUpdateToEvents();
    }

    @resolveDeletionToEvents()
    resolveDeletionToEvents(context: EmbeddingContext) {
        return new EmployeeRetired(this.name);
    }

    @on(EmployeeHired)
    onEmployeeHired(event: EmployeeHired, context: EmbeddingProjectContext) {
        this.name = event.name;
    }

    @on(EmployeeTransferred)
    onEmployeeTransferred(event: EmployeeTransferred, context: EmbeddingProjectContext) {
        this.workplace = event.to;
    }

    @on(EmployeeRetired)
    onEmployeeRetired(event: EmployeeRetired, context: EmbeddingProjectContext) {
        return ProjectionResult.delete;
    }
}

Register the Employee embedding, and update and delete a read model

Let’s register the new event types and the embedding. Then we can update and delete a read model from it.

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;

namespace Kitchen
{
    class Program
    {
        public static async Task Main()
        {
            var client = Client
                .ForMicroservice("f39b1f61-d360-4675-b859-53c05c87c0e6")
                .WithEventTypes(eventTypes =>
                {
                    eventTypes.Register<EmployeeHired>();
                    eventTypes.Register<EmployeeTransferred>();
                    eventTypes.Register<EmployeeRetired>();
                })
                .WithEmbeddings(builder =>
                    builder.RegisterEmbedding<Employee>())
                .Build();
            _ = client.Start();

            // wait for the registration to complete
            await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);

            // mock of the state from the external HR system
            var updatedEmployee = new Employee
            {
                Name = "Mr. Taco",
                Workplace = "Street Food Taco Truck"
            };

            await client.Embeddings
                .ForTenant(TenantId.Development)
                .Update(updatedEmployee.Name, updatedEmployee);
            Console.WriteLine($"Updated {updatedEmployee.Name}.");

            await client.Embeddings
                .ForTenant(TenantId.Development)
                .Delete<Employee>(updatedEmployee.Name);
            Console.WriteLine($"Deleted {updatedEmployee.Name}.");

            // wait for the processing to finish before severing the connection
            await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
        }
    }
}

The Update() method tries to update the embeddings read model with the specified key to match the updated state by calling the embeddings ResolveUpdateToEvents() method. If no read model exists with the key, it will create one with the read model set to the embedding’s initial state.

The Delete() method will call the embeddings ResolveDeletionToEvents() for the specified key. This method then returns one or many events, which when handled will delete the read model.

// index.ts

import { Client } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';
import { Employee } from './Employee';
import { EmployeeHired } from './EmployeeHired';
import { EmployeeRetired } from './EmployeeRetired';
import { EmployeeTransferred } from './EmployeeTransferred';

const client = Client
    .forMicroservice('f39b1f61-d360-4675-b859-53c05c87c0e6')
    .withEventTypes(eventTypes => {
        eventTypes.register(EmployeeHired);
        eventTypes.register(EmployeeTransferred);
        eventTypes.register(EmployeeRetired);
    })
    .withEmbeddings(builder => {
        builder.register(Employee);
    })
    .build();

(async () => {

    // wait for the registration to complete
    setTimeout(async () => {
        // mock of the state from the external HR system
        const updatedEmployee = new Employee(
            'Mr. Taco',
            'Street Food Taco Truck');

        await client.embeddings
            .forTenant(TenantId.development)
            .update(Employee, updatedEmployee.name, updatedEmployee);
        console.log(`Updated ${updatedEmployee.name}`);

        await client.embeddings
            .forTenant(TenantId.development)
            .delete(Employee, updatedEmployee.name);
        console.log(`Deleted ${updatedEmployee.name}`);
    }, 1000);
})();

The update() method tries to update the embeddings read model with the specified key to match the updated state by calling the embeddings @resolveUpdateToEvents() decorated method. If no read model exists with the key, it will create one with the read model set to the embedding’s initial state.

The delete() method will call the embeddings @resolveDeletionToEvents() decorated method for the specified key. This method then returns one or many events, which when handled will delete the read model.

Run your microservice

Let’s run the code! It should commit events to the event log, one for hiring "Mr. Taco", one for transferring him to "Street Food Taco Truck", and one for Mr. Tacos retirement.

$ dotnet run
Updated Mr. Taco.
Deleted Mr. Taco.

$ npx ts-node index.ts
Updated Mr. Taco.
Deleted Mr. Taco.

Check the events

Let’s check the committed events from the event log:

You can look at the events in the database through a database viewer (like MongoDB Compass) or by querying the database running in the dolittle/runtime:latest-development image through the mongo shell.

MongoDB Compass:

MongoDB Compass with 2 events

mongo shell:

$ docker exec <mongo-container> mongo event_store --quiet --eval 'db.getCollection("event-log").find({}, {Content: 1})'
{ "_id" : NumberDecimal("0"), "Content" : { "Name" : "Mr. Taco" } }
{ "_id" : NumberDecimal("1"), "Content" : { "Name" : "Mr. Taco", "From" : "Unassigned", "To" : "Street Food Taco Truck" } }
{ "_id" : NumberDecimal("2"), "Content" : { "Name" : "Mr. Taco" } }

Check the persisted embeddings

We can also check the state of the embeddings read models from the database. Embeddings are saved into a database defined in the resources.json (defaults to embeddings), and each embedding gets it’s unique collection named embedding-<embedding-id>. Deleted embeddings are “soft-deleted” with the IsRemoved property marking their deletion. If the read model is later updated, IsRemoved will be set to false.

MongoDB Compass:

MongoDB Compass showing mr taco read model

mongo shell:

$ docker exec <mongo-container> mongo embeddings --quiet --eval 'db.getCollection("embedding-e5577d2c-0de7-481c-b5be-6ef613c2fcd6").find().pretty()'
{
	"_id" : "Mr. Taco",
	"Content" : "{\"Name\":\"Mr. Taco\",\"Workplace\":\"Street Food Taco Truck\"}",
	"IsRemoved" : true,
	"Version" : NumberLong(3)
}

Get the Embeddings

You can also get the read models and keys for an embedding. This can be useful when figuring out what states still exist in the external system compared to the embeddings read models or when debugging.

For example, the HR system might only return the currently hired employees. Any employees not returned by the HR system but still in the embedding could then be marked as retired.

/*
setup builder and update the read model first
*/
var mrTaco = client.Embeddings
    .ForTenant(TenantId.Development)
    .Get<Employee>("Mr. Taco");
var allEmployees = client.Embeddings
    .ForTenant(TenantId.Development)
    .GetAll<Employee>();
var employeeKeys = client.Embeddings
    .ForTenant(TenantId.Development)
    .GetKeys<Employee>();

/*
setup builder and update the read model first
*/
const mrTaco = client.embeddings
    .forTenant(TenantId.development)
    .get(Employee, 'Mr. Taco');
const allEmployees = client.embeddings
    .forTenant(TenantId.development)
    .getAll(Employee);
cosnt employeeKeys = client.embeddings
    .forTenant(TenantId.development)
    .getKeys(Employee);

If you try to get an embedding that doesn’t exist, the Runtime will return you the initial state of the embedding.

What’s next

Last modified July 19, 2021: Fix whitespace (f53fe43)