Documentation

• 1: Tutorials
• 1.1: Getting started
• 1.2: Aggregates
• 1.3: Projections
• 1.4: Event Horizon
• 2: Concepts
• 2.1: Overview
• 2.2: Events
• 2.3: Streams
• 2.4: Event Handlers & Filters
• 2.5: Projections
• 2.6: Tenants
• 2.7: Event Horizon
• 2.8: Event Store
• 2.9: Event Sourcing
• 2.10: Aggregates
• 2.11: Concurrency
• 2.12: Resource System
• 3: Platform
• 3.1: Requirements
• 3.2: Deploy an application
• 3.3: Update configurations
• 3.4: Update secrets
• 3.5: FAQ
• 3.6: Aigonix Studio
• 3.6.1: Overview
• 3.6.2: Application
• 3.6.3: Environment
• 3.6.4: Microservice
• 3.6.5: Integrations
• 3.6.6: Container Registry
• 4: References
• 4.1: Dolittle CLI
• 4.1.1: Runtime
• 4.1.1.1: Aggregates
• 4.1.1.1.1: List
• 4.1.1.1.2: Get
• 4.1.1.1.3: Events
• 4.1.1.2: Event Handlers
• 4.1.1.2.1: List
• 4.1.1.2.2: Get
• 4.1.1.2.3: Replay
• 4.1.1.3: Event Types
• 4.1.1.3.1: List
• 4.1.1.4: Projections
• 4.1.1.4.1: List
• 4.1.1.4.2: Get
• 4.1.1.4.3: Replay
• 4.2: Runtime
• 4.2.1: Compatibility
• 4.2.2: Configuration
• 4.2.3: Failures
• 5: Contributing
• 5.1: Tooling
• 5.1.1: Code Analysis
• 5.2: Documentation
• 5.2.1: Get started
• 5.2.2: Writing guide
• 5.2.3: Style guide
• 5.2.4: Structure overview
• 5.2.5: API documentation
• 5.3: Guidelines
• 5.3.1: The Vision
• 5.3.2: Code of conduct
• 5.3.3: Core values
• 5.3.4: Core principles
• 5.3.5: Logging
• 5.3.6: C# coding styles
• 5.3.7: C# Specifications
• 5.3.8: Copyright header

1 - Tutorials

1.1 - Getting started

Welcome to the tutorial for Dolittle, where you learn how to write a Microservice that keeps track of foods prepared by the chefs.

After this tutorial you will have:

• a running Dolittle environment with a Runtime and a MongoDB, and
• a Microservice that commits and handles Events

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

For a deeper dive into our Runtime, check our overview.

Setup

• C#
• TypeScript

$ dotnet new console
$ dotnet add package Dolittle.SDK

$ npm init
$ npm -D install typescript ts-node
$ npm install @dolittle/sdk
$ npx tsc --init --experimentalDecorators --target es6

Create an EventType

First we’ll create an EventType that represents that a dish has been prepared. Events represents changes in the system, a “fact that has happened”. As the event “has happened”, it’s immutable by definition, and we should name it in the past tense accordingly.

An EventType is a class that defines the properties of the event. It acts as a wrapper for the type of the event.

• C#
• TypeScript

// DishPrepared.cs
using Dolittle.SDK.Events;

[EventType("1844473f-d714-4327-8b7f-5b3c2bdfc26a")]
public class DishPrepared
{
    public DishPrepared (string dish, string chef)
    {
        Dish = dish;
        Chef = chef;
    }

    public string Dish { get; }
    public string Chef { get; }
}

// DishPrepared.ts
import { eventType } from '@dolittle/sdk.events';

@eventType('1844473f-d714-4327-8b7f-5b3c2bdfc26a')
export class DishPrepared {
    constructor(readonly Dish: string, readonly Chef: string) {}
}

Create an EventHandler

Now we need something that can react to dishes that have been prepared. Let’s create an EventHandler which prints the prepared dishes to the console.

• C#
• TypeScript

// DishHandler.cs
using Dolittle.SDK.Events;
using Dolittle.SDK.Events.Handling;
using Microsoft.Extensions.Logging;

[EventHandler("f2d366cf-c00a-4479-acc4-851e04b6fbba")]
public class DishHandler
{
    readonly ILogger _logger;

    public DishHandler(ILogger<DishHandler> logger)
    {
        _logger = logger;
    }

    public void Handle(DishPrepared @event, EventContext eventContext)
    {
        _logger.LogInformation("{Chef} has prepared {Dish}. Yummm!", @event.Chef, @event.Dish);
    }
}

// DishHandler.ts
import { inject } from '@dolittle/sdk.dependencyinversion';
import { EventContext } from '@dolittle/sdk.events';
import { eventHandler, handles } from '@dolittle/sdk.events.handling';
import { Logger } from 'winston';

import { DishPrepared } from './DishPrepared';

@eventHandler('f2d366cf-c00a-4479-acc4-851e04b6fbba')
export class DishHandler {
    constructor(
        @inject('Logger') private readonly _logger: Logger
    ) {}

    @handles(DishPrepared)
    dishPrepared(event: DishPrepared, eventContext: EventContext) {
        this._logger.info(`${event.Chef} has prepared ${event.Dish}. Yummm!`);
    }
}

Connect the client and commit an event

Let’s build a client that connects to the Runtime for a Microservice with the id "f39b1f61-d360-4675-b859-53c05c87c0e6". This sample Microservice is pre-configured in the development Docker image.

While configuring the client we register the EventTypes and EventHandlers so that the Runtime knows about them. Then we can prepare a delicious taco and commit it to the EventStore for the specified tenant.

• C#
• TypeScript

// Program.cs
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle()
    .Build();

await host.StartAsync();

var client = await host.GetDolittleClient();
await client.EventStore
    .ForTenant(TenantId.Development)
    .CommitEvent(
        content: new DishPrepared("Bean Blaster Taco", "Mr. Taco"),
        eventSourceId: "Dolittle Tacos");

await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';

import './DishHandler';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup()
        .connect();

    const preparedTaco = new DishPrepared('Bean Blaster Taco', 'Mr. Taco');

    await client.eventStore
        .forTenant(TenantId.development)
        .commit(preparedTaco, 'Dolittle Tacos');
})();

Start the Dolittle environment

Start the Dolittle environment with all the necessary dependencies with the following command:

$ docker run -p 50053:50053 -p 51052:51052 -p 27017:27017 -d dolittle/runtime:latest-development

This will start a container with the Dolittle Development Runtime on port 50053 and 51052 and a MongoDB server on port 27017. The Runtime handles committing the events and the event handlers while the MongoDB is used for persistence.

Run your microservice

Run your code, and get a delicious serving of taco:

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../GettingStarted
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
info: DishHandler[0]
      Mr. Taco has prepared Bean Blaster Taco. Yummm!

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.
info: Mr. Taco has prepared Bean Blaster Taco. Yummm!

Check the status of your microservice

With everything is up and running you can use the Dolittle CLI to check what’s going on.

Open a new terminal.

Now you can list the registered event types with the following command:

$ dolittle runtime eventtypes list
EventType
------------
DishPrepared

And check the status of the event handler with the following commands:

$ dolittle runtime eventhandlers list
EventHandler  Scope    Partitioned  Status
------------------------------------------
DishHandler   Default  ✅            ✅

$ dolittle runtime eventhandlers get DishHandler
Tenant                                Position  Status
------------------------------------------------------
445f8ea8-1a6f-40d7-b2fc-796dba92dc44  1         ✅

What’s next

• Learn how to use Aggregates implement rules.
• Learn how to use Projections to create read models.
• Learn how to deploy your application into our Platform.

1.2 - Aggregates

Welcome to the tutorial for Dolittle, where you learn how to write a Microservice that keeps track of foods prepared by the chefs.

After this tutorial you will have:

• a running Dolittle environment with a Runtime and a MongoDB,
• a Microservice that commits and handles Events and
• a stateful aggregate root that applies events and is controlled by an invariant

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

Pre requisites

This tutorial builds directly upon and that you have gone through our getting started guide; done the setup, created the EventType, EventHandler and connected the client

Create an AggregateRoot

An aggregate root is a class that upholds the rules (invariants) in your domain. An aggregate root is responsible for deciding which events should be committed. It exposes public methods that represents actions to be performed, and holds internal state to decide if the action is allowed.

Before one of the public methods is called, the internal state is rehydrated by calling the On-methods for all the events the aggregate root has already applied. These On-methods updates the internal state of the aggregate root, and must not have any other side-effects. When a public action method is executed, it can use this internal state decide either to apply events to be committed, or throw an error if the action is not allowed.

The following code implements an aggregate root for a Kitchen that only has enough ingredients to prepare two dishes:

• C#
• TypeScript

// Kitchen.cs
using System;
using Dolittle.SDK.Aggregates;
using Dolittle.SDK.Events;

[AggregateRoot("01ad9a9f-711f-47a8-8549-43320f782a1e")]
public class Kitchen : AggregateRoot
{
    int _ingredients = 2;

    public Kitchen(EventSourceId eventSource)
        : base(eventSource)
    {
    }

    public void PrepareDish(string dish, string chef)
    {
        if (_ingredients <= 0) throw new Exception("We have run out of ingredients, sorry!");
        Apply(new DishPrepared(dish, chef));
        Console.WriteLine($"Kitchen {EventSourceId} prepared a {dish}, there are {_ingredients} ingredients left.");
    }

    void On(DishPrepared @event)
        => _ingredients--;
}

// Kitchen.ts
import { aggregateRoot, AggregateRoot, on } from '@dolittle/sdk.aggregates';
import { EventSourceId } from '@dolittle/sdk.events';

import { DishPrepared } from './DishPrepared';

@aggregateRoot('01ad9a9f-711f-47a8-8549-43320f782a1e')
export class Kitchen extends AggregateRoot {
    private _ingredients: number = 2;

    constructor(eventSourceId: EventSourceId) {
        super(eventSourceId);
    }

    prepareDish(dish: string, chef: string) {
        if (this._ingredients <= 0) throw new Error('We have run out of ingredients, sorry!');
        this.apply(new DishPrepared(dish, chef));
        console.log(`Kitchen ${this.eventSourceId} prepared a ${dish}, there are ${this._ingredients} ingredients left.`);
    }

    @on(DishPrepared)
    onDishPrepared(event: DishPrepared) {
        this._ingredients--;
    }
}

Apply the event through an aggregate of the Kitchen aggregate root

Let’s expand upon the client built in the getting started guide. But instead of committing the event to the event store directly we perform an action on the aggregate that eventually applies and commits the event.

• C#
• TypeScript

// Program.cs
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle()
    .Build();

await host.StartAsync();

var client = await host.GetDolittleClient();

await client.Aggregates
    .ForTenant(TenantId.Development)
    .Get<Kitchen>("Dolittle Tacos")
    .Perform(kitchen => kitchen.PrepareDish("Bean Blaster Taco", "Mr. Taco"));

await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';

import  './DishHandler';
import { Kitchen } from './Kitchen';

(async () => {
    const client = await DolittleClient
        .setup()
        .connect();

    await client.aggregates
        .forTenant(TenantId.development)
        .get(Kitchen, 'Dolittle Tacos')
        .perform(kitchen => kitchen.prepareDish('Bean Blaster Taco', 'Mr. Taco'));
})();

Start the Dolittle environment

If you don’t have a Runtime already going from a previous tutorial, start the Dolittle environment with all the necessary dependencies with the following command:

$ docker run -p 50053:50053 -p 51052:51052 -p 27017:27017 -d dolittle/runtime:latest-development

This will start a container with the Dolittle Development Runtime on port 50053 and 51052 and a MongoDB server on port 27017. The Runtime handles committing the events and the event handlers while the MongoDB is used for persistence.

Run your microservice

Run your code twice, and get a two delicious servings of taco:

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../Aggregates
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
Kitchen Dolittle Tacos prepared a Bean Blaster Taco, there are 1 ingredients left.
info: DishHandler[0]
      Mr. Taco has prepared Bean Blaster Taco. Yummm!


$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../Aggregates
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
Kitchen Dolittle Tacos prepared a Bean Blaster Taco, there are 0 ingredients left.
info: DishHandler[0]
      Mr. Taco has prepared Bean Blaster Taco. Yummm!

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.
Kitchen Dolittle Tacos prepared a Bean Blaster Taco, there are 1 ingredients left.
info: Mr. Taco has prepared Bean Blaster Taco. Yummm!

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.
Kitchen Dolittle Tacos prepared a Bean Blaster Taco, there are 0 ingredients left.
info: Mr. Taco has prepared Bean Blaster Taco. Yummm!

Check the status of your Kitchen aggregate root

Open a new terminal for the Dolittle CLI and run the following commands:

$ dolittle runtime aggregates list
AggregateRoot  Instances
------------------------
Kitchen        1

$ dolittle runtime aggregates get Kitchen --wide
Tenant                                EventSource     AggregateRootVersion
--------------------------------------------------------------------------
445f8ea8-1a6f-40d7-b2fc-796dba92dc44  Dolittle Tacos  2

$ dolittle runtime aggregates events Kitchen "Dolittle Tacos" --wide
AggregateRootVersion  EventLogSequenceNumber  EventType     Public  Occurred                  
----------------------------------------------------------------------------------------------
0                     0                       DishPrepared  False   11/04/2021 14:04:19 +00:00
1                     1                       DishPrepared  False   11/04/2021 14:04:37 +00:00

Try to prepare a dish without any ingredients

Since we have already used up all the available ingredients, the Kitchen aggregate root should not allow us to prepare any more dishes. Run your code a third time, and you will see that the exception gets thrown from the aggregate root.

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../Aggregates
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
Unhandled exception. System.Exception: We have run out of ingredients, sorry!
... stack trace ...

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.

.../Kitchen.ts:20
        if (this._ingredients <= 0) throw new Error('We have run out of ingredients, sorry!');
                                          ^
Error: We have run out of ingredients, sorry!
... stack trace ...

You can verify that the Kitchen did not allow a third dish to be prepared, by checking the committed events:

$ dolittle runtime aggregates events Kitchen "Dolittle Tacos" --wide
AggregateRootVersion  EventLogSequenceNumber  EventType     Public  Occurred                  
----------------------------------------------------------------------------------------------
0                     0                       DishPrepared  False   11/04/2021 14:04:19 +00:00
1                     1                       DishPrepared  False   11/04/2021 14:04:37 +00:00

Events from aggregate roots are just normal events

The events applied (committed) from aggregate roots are handled the same way as events committed directly to the event store. You can verify this by checking the status of the DishHandler:

$ dolittle runtime eventhandlers get DishHandler
Tenant                                Position  Status
------------------------------------------------------
445f8ea8-1a6f-40d7-b2fc-796dba92dc44  2         ✅  

What’s next

• Learn how to use Projections to create read models.
• Learn how to deploy your application into our Platform.

1.3 - Projections

Welcome to the tutorial for Projections in Dolittle, where you learn how to write a Microservice that keeps track of food prepared by the chefs. After this tutorial you will have:

• a running Dolittle environment with a Runtime and a MongoDB, and
• a Microservice that commits Events
• 2 Projections that react to events and mutate the Read Model.

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

Setup

This tutorial builds directly upon the getting started guide and the files from the it.

• C#
• TypeScript

└── Projections/
    ├── DishHandler.cs
    ├── DishPrepared.cs
    ├── Program.cs
    └── Projections.csproj

└── projections/
    ├── .eslintrc
    ├── DishHandler.ts
    ├── DishPrepared.ts
    ├── index.ts
    ├── package.json
    └── tsconfig.json

Start the Dolittle environment

If you don’t have a Runtime already going from a previous tutorial, start the Dolittle environment with all the necessary dependencies with the following command:

$ docker run -p 50053:50053 -p 51052:51052 -p 27017:27017 -d dolittle/runtime:latest-development

This will start a container with the Dolittle Development Runtime on port 50053 and 51052 and a MongoDB server on port 27017. The Runtime handles committing the events and the event handlers while the MongoDB is used for persistence.

Create a DishCounter Projection

First, we’ll create a Projection that keeps track of the dishes and how many times the chefs have prepared them. Projections are a special type of event handler that mutate a read model based on incoming events.

• C#
• TypeScript

// DishCounter.cs
using Dolittle.SDK.Projections;

[Projection("98f9db66-b6ca-4e5f-9fc3-638626c9ecfa")]
public class DishCounter
{
    public string Name = "Unknown";
    public int NumberOfTimesPrepared = 0;

    [KeyFromProperty("Dish")]
    public void On(DishPrepared @event, ProjectionContext context)
    {
        Name = @event.Dish;
        NumberOfTimesPrepared ++;
    }
}

// DishCounter.ts
import { ProjectionContext, projection, on } from '@dolittle/sdk.projections';

import { DishPrepared } from './DishPrepared';

@projection('98f9db66-b6ca-4e5f-9fc3-638626c9ecfa')
export class DishCounter {
    name: string = 'Unknown';
    numberOfTimesPrepared: number = 0;

    @on(DishPrepared, _ => _.keyFromProperty('Dish'))
    on(event: DishPrepared, projectionContext: ProjectionContext) {
        this.name = event.Dish;
        this.numberOfTimesPrepared ++;
    }
}

Register and get the DishCounter Projection

Let’s register the projection, commit new DishPrepared events and get the projection to see how it reacted.

• C#
• TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle()
    .Build();
await host.StartAsync();

var client = await host.GetDolittleClient();
var eventStore = client.EventStore.ForTenant(TenantId.Development);
await eventStore.CommitEvent(new DishPrepared("Bean Blaster Taco", "Mr. Taco"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Bean Blaster Taco", "Mrs. Tex Mex"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Avocado Artillery Tortilla", "Mr. Taco"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Chili Canon Wrap", "Mrs. Tex Mex"), "Dolittle Tacos");

await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);

var dishes = await client.Projections
    .ForTenant(TenantId.Development)
    .GetAll<DishCounter>().ConfigureAwait(false);

foreach (var dish in dishes)
{
    Console.WriteLine($"The kitchen has prepared {dish.Name} {dish.NumberOfTimesPrepared} times");
}

await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';
import { setTimeout } from 'timers/promises';

import { DishCounter } from './DishCounter';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup()
        .connect();

    const eventStore = client.eventStore.forTenant(TenantId.development);

    await eventStore.commit(new DishPrepared('Bean Blaster Taco', 'Mr. Taco'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Bean Blaster Taco', 'Mrs. Tex Mex'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Avocado Artillery Tortilla', 'Mr. Taco'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Chili Canon Wrap', 'Mrs. Tex Mex'), 'Dolittle Tacos');

    await setTimeout(1000);

    for (const { name, numberOfTimesPrepared } of await client.projections.forTenant(TenantId.development).getAll(DishCounter)) {
        client.logger.info(`The kitchen has prepared ${name} ${numberOfTimesPrepared} times`);
    }
})();

Run your microservice

Run your code, and see the different dishes:

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../Projections
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      Projection 98f9db66-b6ca-4e5f-9fc3-638626c9ecfa registered with the Runtime, start handling requests
The kitchen has prepared Bean Blaster Taco 2 times
The kitchen has prepared Avocado Artillery Tortilla 1 times
The kitchen has prepared Chili Canon Wrap 1 times

$ npx ts-node index.ts
info: Projection 98f9db66-b6ca-4e5f-9fc3-638626c9ecfa registered with the Runtime, start handling requests.
info: The kitchen has prepared Bean Blaster Taco 2 times
info: The kitchen has prepared Avocado Artillery Tortilla 1 times
info: The kitchen has prepared Chili Canon Wrap 1 times

Add Chef read model

Let’s add another read model to keep track of all the chefs and . This time let’s only create the class for the read model:

• C#
• TypeScript

// Chef.cs
using System.Collections.Generic;

public class Chef
{
    public string Name = "";
    public List<string> Dishes = new();
}

// Chef.ts
export class Chef {
    constructor(
        public name: string = '',
        public dishes: string[] = []
    ) { }
}

Create and get the inline projection for Chef read model

You can also create a Projection inline in the client building steps instead of declaring a class for it.

Let’s create an inline Projection for the Chef read model:

• C#
• TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle(_ => _ 
        .WithProjections(_ => _
            .Create("0767bc04-bc03-40b8-a0be-5f6c6130f68b")
                .ForReadModel<Chef>()
                .On<DishPrepared>(_ => _.KeyFromProperty(_ => _.Chef), (chef, @event, projectionContext) =>
                {
                    chef.Name = @event.Chef;
                    if (!chef.Dishes.Contains(@event.Dish)) chef.Dishes.Add(@event.Dish);
                    return chef;
                })
        )
    )
    .Build();
await host.StartAsync();

var client = await host.GetDolittleClient();
var eventStore = client.EventStore.ForTenant(TenantId.Development);
await eventStore.CommitEvent(new DishPrepared("Bean Blaster Taco", "Mr. Taco"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Bean Blaster Taco", "Mrs. Tex Mex"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Avocado Artillery Tortilla", "Mr. Taco"), "Dolittle Tacos");
await eventStore.CommitEvent(new DishPrepared("Chili Canon Wrap", "Mrs. Tex Mex"), "Dolittle Tacos");

await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);

var dishes = await client.Projections
    .ForTenant(TenantId.Development)
    .GetAll<DishCounter>().ConfigureAwait(false);

foreach (var dish in dishes)
{
    Console.WriteLine($"The kitchen has prepared {dish.Name} {dish.NumberOfTimesPrepared} times");
}

var chef = await client.Projections
    .ForTenant(TenantId.Development)
    .Get<Chef>("Mrs. Tex Mex").ConfigureAwait(false);
Console.WriteLine($"{chef.Name} has prepared {string.Join(", ", chef.Dishes)}");

await host.WaitForShutdownAsync();

// index.ts
(async () => {
    const client = await DolittleClient
        .setup(builder => builder
            .withProjections(_ => _
                .create('0767bc04-bc03-40b8-a0be-5f6c6130f68b')
                    .forReadModel(Chef)
                    .on(DishPrepared, _ => _.keyFromProperty('Chef'), (chef, event, projectionContext) => {
                        chef.name = event.Chef;
                        if (!chef.dishes.includes(event.Dish)) chef.dishes.push(event.Dish);
                        return chef;
                    })
            )
        )
        .connect();

    const eventStore = client.eventStore.forTenant(TenantId.development);

    await eventStore.commit(new DishPrepared('Bean Blaster Taco', 'Mr. Taco'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Bean Blaster Taco', 'Mrs. Tex Mex'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Avocado Artillery Tortilla', 'Mr. Taco'), 'Dolittle Tacos');
    await eventStore.commit(new DishPrepared('Chili Canon Wrap', 'Mrs. Tex Mex'), 'Dolittle Tacos');

    await setTimeout(1000);

    for (const { name, numberOfTimesPrepared } of await client.projections.forTenant(TenantId.development).getAll(DishCounter)) {
        client.logger.info(`The kitchen has prepared ${name} ${numberOfTimesPrepared} times`);
    }

    const chef = await client.projections.forTenant(TenantId.development).get(Chef, 'Mrs. Tex Mex');
    client.logger.info(`${chef.name} has prepared ${chef.dishes.join(', ')}`);
})();

Run your microservice with the inline Chef projection

Run your code, and get a delicious serving of taco:

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: .../Projections
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      Projection 0767bc04-bc03-40b8-a0be-5f6c6130f68b registered with the Runtime, start handling requests
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      Projection 98f9db66-b6ca-4e5f-9fc3-638626c9ecfa registered with the Runtime, start handling requests
The kitchen has prepared Bean Blaster Taco 4 times
The kitchen has prepared Avocado Artillery Tortilla 2 times
The kitchen has prepared Chili Canon Wrap 2 times
Mrs. Tex Mex has prepared Bean Blaster Taco, Chili Canon Wrap

$ npx ts-node index.ts
info: Projection 0767bc04-bc03-40b8-a0be-5f6c6130f68b registered with the Runtime, start handling requests.
info: Projection 98f9db66-b6ca-4e5f-9fc3-638626c9ecfa registered with the Runtime, start handling requests.
info: The kitchen has prepared Bean Blaster Taco 4 times
info: The kitchen has prepared Avocado Artillery Tortilla 2 times
info: The kitchen has prepared Chili Canon Wrap 2 times
info: Mrs. Tex Mex has prepared Bean Blaster Taco,Chili Canon Wrap

What’s next

• Learn how to deploy your application into our Platform.

1.4 - Event Horizon

Welcome to the tutorial for Event Horizon, where you learn how to write a Microservice that produces public events of dishes prepared by chefs, and another microservice consumes those events.

After this tutorial you will have:

• a running Dolittle environment with two Runtimes and a MongoDB,
• a Producer Microservice that commits and handles a Public Event and filters it into a Public Stream and
• a Consumer Microservice that Subscribes to the consumers public stream over the Event Horizon and processes those public events

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

Prerequisites

This tutorial builds directly upon the getting started guide and the files from the it.

• C#
• TypeScript

Setup

This tutorial will have a setup with two microservices; one that produces public events, and a consumer that subscribes to those public events. Let’s make a folder structure that resembles that:

└── event-horizon-tutorial/
    ├── consumer/
    ├── producer/
    └── environment/
        └── docker-compose.yml

Go into both the consumer and the producer folders and initiate the project as we’ve gone through in our getting started guide. I.e copy over all the code from the getting started tutorial to the consumer and producer folders. You can choose different languages for the microservices if you want to.

We’ll come back to the docker-compose later in this tutorial.

Producer

Create a Public Filter

A public filter filters all public events that pass the filter into a public stream, which is special stream that another microservice can subscribe to.

A public filter is defined as a method that returns a partitioned filter result, which is an object with two properties:

• a boolean that says whether the event should be included in the public stream
• a partition id which is the partition that the event should belong to in the public stream.

Only public events get filtered through the public filters.

• C#
• TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Events.Filters;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithFilters(_ => _
            .CreatePublic("2c087657-b318-40b1-ae92-a400de44e507")
            .Handle((@event, eventContext) =>
            {
                Console.WriteLine($"Filtering event {@event} to public streams");
                return Task.FromResult(new PartitionedFilterResult(true, eventContext.EventSourceId.Value));
            })))
    .Build();

await host.StartAsync();
await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { EventContext } from '@dolittle/sdk.events';
import { PartitionedFilterResult } from '@dolittle/sdk.events.filtering';
import { TenantId } from '@dolittle/sdk.execution';

import './DishHandler';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withFilters(_ => _
                .createPublic('2c087657-b318-40b1-ae92-a400de44e507')
                    .handle((event: any, context: EventContext) => {
                        client.logger.info(`Filtering event ${JSON.stringify(event)} to public stream`);
                        return new PartitionedFilterResult(true, 'Dolittle Tacos');
                    })
            )
        )
        .connect();
})();

Notice that the returned PartitionedFilterResult has true and an unspecified PartitionId (which is the same as an empty GUID). This means that this filter creates a public stream that includes all public events, and that they are put into the unspecified partition of that stream.

Commit the public event

Now that we have a public stream we can commit public events to start filtering them. Let’s commit a DishPrepared event as a public event from the producer microservice:

• C#
• TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Events.Filters;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithFilters(_ => _
            .CreatePublic("2c087657-b318-40b1-ae92-a400de44e507")
            .Handle((@event, eventContext) =>
            {
                Console.WriteLine($"Filtering event {@event} to public streams");
                return Task.FromResult(new PartitionedFilterResult(true, eventContext.EventSourceId.Value));
            })))
    .Build();

await host.StartAsync();

var client = await host.GetDolittleClient();
var preparedTaco = new DishPrepared("Bean Blaster Taco", "Mr. Taco");
await client.EventStore
    .ForTenant(TenantId.Development)
    .CommitPublicEvent(preparedTaco, "Dolittle Tacos");

await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { EventContext } from '@dolittle/sdk.events';
import { PartitionedFilterResult } from '@dolittle/sdk.events.filtering';
import { TenantId } from '@dolittle/sdk.execution';

import './DishHandler';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withFilters(_ => _
                .createPublic('2c087657-b318-40b1-ae92-a400de44e507')
                    .handle((event: any, context: EventContext) => {
                        client.logger.info(`Filtering event ${JSON.stringify(event)} to public stream`);
                        return new PartitionedFilterResult(true, 'Dolittle Tacos');
                    })
            )
        )
        .connect();

    const preparedTaco = new DishPrepared('Bean Blaster Taco', 'Mr. Taco');

    await client.eventStore
        .forTenant(TenantId.development)
        .commitPublic(preparedTaco, 'Dolittle Tacos');
})();

Now we have a producer microservice with a public stream of DishPrepared events.

Consumer

Subscribe to the public stream of events

Let’s create another microservice that subscribes to the producer’s public stream.

• C#
• TypeScript

// Program.cs
using System;
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithEventHorizons(_ => _
            .ForTenant(TenantId.Development, subscriptions =>
                subscriptions
                    .FromProducerMicroservice("f39b1f61-d360-4675-b859-53c05c87c0e6")
                    .FromProducerTenant(TenantId.Development)
                    .FromProducerStream("2c087657-b318-40b1-ae92-a400de44e507")
                    .FromProducerPartition("Dolittle Tacos")
                    .ToScope("808ddde4-c937-4f5c-9dc2-140580f6919e"))
        )
        .WithEventHandlers(_ => _
            .Create("6c3d358f-3ecc-4c92-a91e-5fc34cacf27e")
            .InScope("808ddde4-c937-4f5c-9dc2-140580f6919e")
            .Partitioned()
            .Handle<DishPrepared>((@event, context) => Console.WriteLine($"Handled event {@event} from public stream"))
        ),
        configuration => configuration.WithRuntimeOn("localhost", 50055))
    .Build()
    .Run();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';

import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withEventHorizons(_ => {
                _.forTenant(TenantId.development, _ => _
                    .fromProducerMicroservice('f39b1f61-d360-4675-b859-53c05c87c0e6')
                        .fromProducerTenant(TenantId.development)
                        .fromProducerStream('2c087657-b318-40b1-ae92-a400de44e507')
                        .fromProducerPartition('Dolittle Tacos')
                        .toScope('808ddde4-c937-4f5c-9dc2-140580f6919e'));
            })
            .withEventHandlers(_ => _
                .create('6c3d358f-3ecc-4c92-a91e-5fc34cacf27e')
                    .inScope('808ddde4-c937-4f5c-9dc2-140580f6919e')
                    .partitioned()
                    .handle(DishPrepared, (event, context) => {
                        client.logger.info(`Handled event ${JSON.stringify(event)} from public stream`);
                     })
            )
        )
        .connect(_ => _
            .withRuntimeOn('localhost', 50055)
        );
})();

Now we have a consumer microservice that:

• Connects to another Runtime running on port 50055
• Subscribes to the producer’s public stream with the id of 2c087657-b318-40b1-ae92-a400de44e507 (same as the producer’s public filter)
• Puts those events into a Scope with id of 808ddde4-c937-4f5c-9dc2-140580f6919e
• Handles them incoming events in a scoped event handler with an id of 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e

There’s a lot of stuff going on the code so let’s break it down:

Connection to the Runtime

• C#
• TypeScript

configuration => configuration.WithRuntimeOn("localhost", 50055))

.withRuntimeOn('localhost', 50055)

This line configures the hostname and port of the Runtime for this client. By default, it connects to the Runtimes default port of 50053 on localhost.

Since we in this tutorial will end up with two running instances of the Runtime, they will have to run with different ports. The producer Runtime will be running on the default 50053 port, and the consumer Runtime will be running on port 50055. We’ll see this reflected in the docker-compose.yml file later in this tutorial.

Event Horizon

• C#
• TypeScript

// Program.cs
.WithEventHorizons(_ => _
    .ForTenant(TenantId.Development, subscriptions =>
        subscriptions
            .FromProducerMicroservice("f39b1f61-d360-4675-b859-53c05c87c0e6")
            .FromProducerTenant(TenantId.Development)
            .FromProducerStream("2c087657-b318-40b1-ae92-a400de44e507")
            .FromProducerPartition("Dolittle Tacos")
            .ToScope("808ddde4-c937-4f5c-9dc2-140580f6919e"))
)

.withEventHorizons(_ => {
    _.forTenant(TenantId.development, _ => _
        .fromProducerMicroservice('f39b1f61-d360-4675-b859-53c05c87c0e6')
            .fromProducerTenant(TenantId.development)
            .fromProducerStream('2c087657-b318-40b1-ae92-a400de44e507')
            .fromProducerPartition('Dolittle Tacos')
            .toScope('808ddde4-c937-4f5c-9dc2-140580f6919e'));
})

Here we define an event horizon subscription. Each subscription is submitted and managed by the Runtime. A subscription defines:

• The consumers Tenant
• The producer microservice, Public Stream and that streams Partition to get the events from
• The Scoped event-log of the consumer to put the subscribed events to

When the consumer’s Runtime receives a subscription, it will send a subscription request to the producer’s Runtime. If the producer accepts that request, the producer’s Runtime will start sending the public stream over to the consumer’s Runtime, one event at a time.

The acceptance depends on two things:

• The consumer needs to know where to access the other microservices, ie the URL address.
• The producer needs to give formal Consent for a tenant in another microservice to subscribe to public streams of a tenant.

We’ll setup the consent later.

The consumer will receive events from the producer and put those events in a specialized event-log that is identified by the scope’s id, so that events received over the event horizon don’t mix with private events. We’ll talk more about the scope when we talk about the scoped event handler.

Scoped Event Handler

• C#
• TypeScript

// Program.cs
.WithEventHandlers(_ => _
    .Create("6c3d358f-3ecc-4c92-a91e-5fc34cacf27e")
    .InScope("808ddde4-c937-4f5c-9dc2-140580f6919e")
    .Partitioned()
    .Handle<DishPrepared>((@event, context) => Console.WriteLine($"Handled event {@event} from public stream"))
)

.withEventHandlers(_ => _
    .create('6c3d358f-3ecc-4c92-a91e-5fc34cacf27e')
        .inScope('808ddde4-c937-4f5c-9dc2-140580f6919e')
        .partitioned()
        .handle(DishPrepared, (event, context) => {
            client.logger.info(`Handled event ${JSON.stringify(event)} from public stream`);
        })
)

Here we use the opportunity to create an event handler inline by using the client’s builder function. This way we don’t need to create a class and register it as an event handler.

This code will create a partitioned event handler with id 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e (same as from getting started) in a specific scope.

Remember, that the events from an event horizon subscription get put into a scoped event-log that is identified by the scope id. Having the scope id defined when creating an event handler signifies that it will only handle events in that scope and no other.

Setup your environment

Now we have the producer and consumer microservices Heads coded, we need to setup the environment for them to run in and configure their Runtimes to be connected. This configuration is provided by Dolittle when you’re running your microservices in our platform, but when running multiple services on your local machine you need to configure some of it yourself.

Let’s go to the environment folder we created in the beginning of this tutorial. Here we’ll need to configure:

• platform.json
• resources.json
• endpoints.json
• microservices.json
• event-horizon-consents.json.

Platform

platform.json configures the environment of a microservice. We have 2 microservices so they need to be configured with different identifiers and names.

Let’s create 2 files, consumer-platform.json and producer-producer.json:

//consumer-platform.json
{
    "applicationName": "EventHorizon Tutorial",
    "applicationID": "5bd8762f-6c39-4ba2-a141-d041c8668894",
    "microserviceName": "Consumer",
    "microserviceID": "a14bb24e-51f3-4d83-9eba-44c4cffe6bb9",
    "customerName": "Dolittle Tacos",
    "customerID": "c2d49e3e-9bd4-4e54-9e13-3ea4e04d8230",
    "environment": "Tutorial"
}

//producer-platform.json
{
    "applicationName": "EventHorizon Tutorial",
    "applicationID": "5bd8762f-6c39-4ba2-a141-d041c8668894",
    "microserviceName": "Producer",
    "microserviceID": "f39b1f61-d360-4675-b859-53c05c87c0e6",
    "customerName": "Dolittle Tacos",
    "customerID": "c2d49e3e-9bd4-4e54-9e13-3ea4e04d8230",
    "environment": "Tutorial"
}

Resources

resources.json configures where a microservices stores its event store. We have 2 microservices so they both need their own event store database. By default the database is called event_store.

Create 2 more files, consumer-resources.json and producer-resources.json:

//consumer-resources.json
{
    // the tenant to define this resource for
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": {
        "eventStore": {
            "servers": [
                // hostname of the mongodb
                "mongo"
            ],
            // the database name for the event store
            "database": "consumer_event_store"
        }
    }
}

//producer-resources.json
{
    // the tenant to define this resource for
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": {
        "eventStore": {
            "servers": [
                // hostname of the mongodb
                "mongo"
            ],
            // the database name for the event store
            "database": "producer_event_store"
        }
    }
}

Microservices

microservices.json configures where the producer microservice is so that the consumer can connect to it and subscribe to its events.

Let’s create a consumer-microservices.json file to define where the consumer can find the producer:

// consumer-microservices.json
{
    // the producer microservices id, hostname and port
    "f39b1f61-d360-4675-b859-53c05c87c0e6": {
        "host": "producer-runtime",
        "port": 50052
    }
}

Consent

event-horizon-consents.json configures the Consents that the producer gives to consumers.

Let’s create producer-event-horizon-consents.json where we give a consumer consent to subscribe to our public stream.

// producer-event-horizon-consents.json
{
    // the producer's tenant that gives the consent
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": [
        {
            // the consumer's microservice and tenant to give consent to
            "microservice": "a14bb24e-51f3-4d83-9eba-44c4cffe6bb9",
            "tenant": "445f8ea8-1a6f-40d7-b2fc-796dba92dc44",
            // the producer's public stream and partition to give consent to subscribe to
            "stream": "2c087657-b318-40b1-ae92-a400de44e507",
            "partition": "Dolittle Tacos",
            // an identifier for this consent. This is random
            "consent": "ad57aa2b-e641-4251-b800-dd171e175d1f"
        }
    ]
}

Configure docker-compose.yml

Now we can glue all the configuration files together in the docker-compose.yml. The configuration files are mounted inside /app.dolittle/ inside the dolittle/runtime image.

version: '3.8'
services:
  mongo:
    image: dolittle/mongodb
    hostname: mongo
    ports:
      - 27017:27017
    logging:
      driver: none

  consumer-runtime:
    image: dolittle/runtime:latest
    volumes:
      - ./consumer-platform.json:/app/.dolittle/platform.json
      - ./consumer-resources.json:/app/.dolittle/resources.json
      - ./consumer-microservices.json:/app/.dolittle/microservices.json
    ports:
      - 50054:50052
      - 50055:50053

  producer-runtime:
    image: dolittle/runtime:latest
    volumes:
      - ./producer-platform.json:/app/.dolittle/platform.json
      - ./producer-resources.json:/app/.dolittle/resources.json
      - ./producer-event-horizon-consents.json:/app/.dolittle/event-horizon-consents.json
    ports:
      - 50052:50052
      - 50053:50053

Start the environment

Start the docker-compose with this command

$ docker-compose up -d

This will spin up a MongoDB container and two Runtimes in the background.

Run your microservices

Run both the consumer and producer microservices in their respective folders, and see the consumer handle the events from the producer:

• C#
• TypeScript

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jakob/Git/dolittle/DotNET.SDK/Samples/Tutorials/EventHorizon/Producer
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      Public Filter 2c087657-b318-40b1-ae92-a400de44e507 registered with the Runtime, start handling requests
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
Filtering event DishPrepared to public streams
info: DishHandler[0]
      Mr. Taco has prepared Bean Blaster Taco. Yummm!

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jakob/Git/dolittle/DotNET.SDK/Samples/Tutorials/EventHorizon/Consumer
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e registered with the Runtime, start handling requests
Handled event DishPrepared from public stream

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.
info: Public Filter 2c087657-b318-40b1-ae92-a400de44e507 registered with the Runtime, start handling requests.
info: Filtering event {"Dish":"Bean Blaster Taco","Chef":"Mr. Taco"} to public stream
info: Mr. Taco has prepared Bean Blaster Taco. Yummm!

$ npx ts-node index.ts
info: EventHandler 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e registered with the Runtime, start handling requests.
info: Handled event {"Dish":"Bean Blaster Taco","Chef":"Mr. Taco"} from public stream

What’s next

• Learn how to deploy your application into our Platform
• Learn more about the Event Horizon

2 - Concepts

The Concepts section helps you learn about the abstractions and components of Dolittle.

To learn how to write a Dolittle application read our tutorial.

2.1 - Overview

Dolittle is an event-driven microservices platform built to harness the power of events. It’s a reliable ecosystem for microservices to thrive so that you can build complex applications with small, focused microservices that are loosely coupled, event driven and highly maintainable.

Components

• Events are “facts that have happened” in your system and they form the truth of the system.
• Event Handlers & Filter and Projections process events.
• The Runtime is the core of all Dolittle applications and manages connections from the SDKs and other Runtimes to its Event Store. The Runtime is packaged as a Docker image
• The SDK is a client-library that handles communication with the Runtime for your code.
• The Head is the user code that uses the Dolittle SDK. This is where your business-code lives, or is called from. You create this as a docker-image where your code uses the SDK. It will usually contain your domain-code and a frontend.
• The Event Store is the underlying database where the events are stored.
• A Microservice is one or more Heads talking to a Runtime.
• Microservices can produce public events and consume such events that flow over the Event Horizon.

flowchart LR
    subgraph MSP["Microservice (Consumer)"]
        subgraph H1[Head]
            F1[Frontend] --> Domain1
            subgraph B1[Backend]
                Domain1[Domain code] --> SDK1[SDK]
            end
        end
        SDK1 --> R1[Runtime]
        R1 --> ES1[(Event Store)]
    end
    subgraph MS1["Microservice (Producer)"]
        subgraph H2[Head]
            F2[Frontend] --> Domain2[Domain code]
            subgraph Backend
            Domain2 --> SDK2[SDK]
            end
        end
        SDK2 --> R2[Runtime]
        R2 --> ES2[(Event Store)]
    end
    R1 --Event Horizon gets<br/>public events--> R2

Event-Driven

Dolittle uses an Event-Driven Architecture and supports Event Sourcing, which means to “capture all changes to an applications state as a sequence of events”, these events then form the “truth” of the system. Events cannot be changed or deleted as they represent facts about things that have happened.

With event sourcing your applications state is no longer stored primarilly as a snapshot of your current state but rather as a whole history of all the state-changing events. These events can be replayed to recreate the state whenever needed.

For example: you can replay them in a test environment to see how a changed system would have behaved. By running through events up to a point in time the system can also reproduce the state it had at any point in time.

Event sourcing supports high scalability by being loose coupling. The events are the only thing that needs to be shared between different parts of the system, and separate parts can be made with different trade-offs for the scale they need to handle.

The history of events also forms a ready-made audit log to help with debugging and auditing.

Microservice

A microservice, in our parlance, consists of one or many heads talking to one Runtime. Each microservice is autonomous and has its own resources and event store.

The core idea is that a microservice is an independently scalable unit of deployment that can be reused in other parts of the software however you like. You could compose as one application running inside a single process, or you could spread it across a cluster. It really is a deployment choice once the software is giving you this freedom.

This diagram shows the anatomy of a microservice with one head.

flowchart LR
    Frontend --> Backend
    subgraph Head
        Backend --> SDK
    end
    SDK --> Runtime
    Runtime --> ES[(Event Store)]
    Runtime --> RC[(Read Cache)]

Multi-tenancy

Since compute is usually the most expensive resource, the Dolittle Runtime and SDK’s has been built from the ground up with multi-tenancy in mind. Multi-tenancy means that a single instance of the software and its supporting infrastructure serves multiple customers, making optimal use of resources. Dolittle supports multi-tenancy by separating the event stores and resources for each tenant so that each tenant only has access to its own data.

This diagram shows a microservice with 2 tenants, each of them with their own resources.

flowchart LR
    Frontend --> Backend
    subgraph Head
        Backend --> SDK
    end
    SDK --> Runtime
    Runtime --> ES1[("Tenant 1
    Event Store")]
    Runtime --> RC1[("Tenant 1
    Read Cache")]
    Runtime --> ES2[("Tenant 2
    Event Store")]
    Runtime --> RC2[("Tenant 2
    Read Cache")]

What Dolittle isn’t

Dolittle is not a traditional backend library nor an event driven message bus like Kafka. Dolittle uses Event Sourcing, which means that the state of the system is built from an append-only Event Store that has all the events ever produced by the application.

Dolittle isn’t a Command-Query Responsibility Segregation (CQRS) framework with formalized commands and queries, but it used to be. Dolittle allows you to write your own CQRS -abstractions on top of the SDK if you so desire.

Technology

• Runtime repository
• C# SDK repository
• JavaScript SDK repository
• The connection between the runtime and the SDKs is managed through gRPC calls, defined in our Contracts repository

The Event Store is implemented with MongoDB, and the resources -system give you access to a tenanted MongoDatabase for easy storage of your read-cache.

What’s next

• Read about Events
• Ready to Get Started?

2.2 - Events

An Event is a serializable representation of “a fact that has happened within your system”.

“A fact”

An event is a change (fact) within our system. The event itself contains all the relevant information concerning the change. At its simplest, an event can be represented by a name (type) if it’s enough to describe the change.

More usually, it is a simple Data Transfer Object (DTO) that contains state and properties that describe the change. It does not contain any calculations or behavior.

“that has happened”

As the event has happened, it cannot be changed, rejected, or deleted. This forms the basis of Event Sourcing If you wish to change the action or the state change that the event encapsulates, then it is necessary to initiate an action that results in another event that nullifies the impact of the first event.

This is common in accounting, for example: Sally adds 100$ into her bank, which would result in an event like “Add 100$ to Sally’s account”. But if the bank accidentally adds 1000$ instead of the 100$ then a correcting event should be played, like “Subtract 900$ from Sally’s account”. And with event sourcing, this information is preserved in the event store for eg. later auditing purposes.

Naming

To indicate that the event “has happened in the past”, it should be named as a verb in the past tense. Often it can contain the name of the entity that the change or action is affecting.

• ✅ DishPrepared
• ✅ ItemAddedToCart
• ❌ StartCooking
• ❌ AddItemToCart

“within your system”

An event represents something interesting that you wish to capture in your system. Instead of seeing state changes and actions as side effects, they are explicitly modeled within the system and captured within the name, state and shape of our Event.

State transitions are an important part of our problem space and should be modeled within our domain — Greg Young

Naming

An event should be expressed in language that makes sense in the domain, also known as Ubiquitous Language. You should avoid overly technical/CRUD-like events where such terms are not used in the domain.

For example, in the domain of opening up the kitchen for the day and adding a new item to the menu:

• ✅ KitchenOpened
• ✅ DishAddedToMenu
• ❌ TakeoutServerReady
• ❌ MenuListingElementUpdated

Main structure of an Event

This is a simplified structure of the main parts of an event. For the Runtime, the event is only a JSON-string which is saved into the Event Store.

Event {
    Content object
    EventLogSequenceNumber int
    EventSourceId string
    Public bool
    EventType {
        EventTypeId Guid
        Generation int
    }
}

For the whole structure of an event as defined in protobuf, please check Contracts.

Content

This is the content of the to be committed. It needs to be serializable to JSON.

EventLogSequenceNumber

This is the events position in the Event Log. It uniquely identifies the event.

EventSourceId

EventSourceId represents the source of the event like a “primary key” in a traditional database. The value of the event source id is simply a string, and we don’t enforce any particular rules or restrictions on the event source id. By default, partitioned event handlers use it for partitioning.

Public vs. Private

There is a basic distinction between private events and public events. In much the same way that you would not grant access to other applications to your internal database, you do not allow other applications to receive any of your private events.

Private events are only accessible within a single Tenant so that an event committed for one tenant cannot be handled outside of that tenant.

Public events are also accessible within a single tenant but they can also be added to a public Stream through a public filterfor other microservices to consume. Your public event streams essentially form a public API for the other microservices to subscribe to.

EventType

An EventType is the combination of an EventTypeId to uniquely identify the type of event it is and the event type’s Generation. This decouples the event from a programming language and enables the renaming of events as the domain language evolves.

For the Runtime, the event is just a JSON-string. It doesn’t know about the event’s content, properties, or type (in its respective programming language). The Runtime saves the event to the event log and from that point the event is ready to be processed by the EventHandlers & Filters. For this event to be serialized to JSON and then deserialized back to a type that the client’s filters and event handlers understand, an event type is required.

This diagram shows us a simplified view of committing a single event with the type of DishPrepared. The Runtime receives the event, and sends it back to us to be handled. Without the event type, the SDK wouldn’t know how to deserialize the JSON message coming from the Runtime.

sequenceDiagram
    participant SDK
    participant Runtime
    participant Event Store
    SDK->>Runtime: Commit(DishPrepared)
    Runtime->>Event Store: Serialize the event into<br/>JSON and save it
    Runtime->>SDK: Commit successful
    Runtime->>Runtime: Process the event in<br/>handlers and filters
    Runtime->>SDK: Send the JSON of the event<br/>to the event-handler
    SDK->>SDK: Deserialize according to EventTypeId<br/>found in JSON and call on the handler

Event types are also important when wanting to deserialize events coming from other microservices. As the other microservice could be written in a completely different programming language, event types provide a level of abstraction for deserializing the events.

Generations

As the code changes, the structures and contents of your events are also bound to change at some point. In most scenarios, you will see that you need to add more information to events. These iterations on the same event type are called generations. Whenever you add or change a property in an event, the generation should be incremented to reflect that it’s a new version of the event. This way the filters and handlers can handle different generations of an event.

2.3 - Streams

So, what is a stream? A stream is simply a list with two specific attributes:

• Streams are append-only. Meaning that items can only be put at the very end of the stream, and that the stream is not of a fixed length.
• Items in the stream immutable. The items or their order cannot change. An event stream is simply a stream of events. Each stream is uniquely identified within an Event Store by a GUID. An event can belong many streams, and in most cases it will at least belong to two streams (one being the event log).

As streams are append-only, an event can be uniquely identified by its position in a stream, including in the event log.

Event streams are perhaps the most important part of the Dolittle Runtime. To get a different and more detailed perspective on streams, please read our section on event sourcing and streams.

Rules

There are rules on streams to maintain idempotency and the predictability of Runtime. These rules are enforced by the Runtime:

• The ordering of the events cannot change
• Events can only be appended to the end of the stream
• Events cannot be removed from the stream
• A partitioned stream cannot be changed to be unpartitioned and vice versa

Partitions

If we dive deeper into event streams we’ll see that we have two types of streams in the Runtime; partitioned and unpartitioned streams.

A partitioned stream is a stream that is split into chunks. These chunks are uniquely identified by a PartitionId (string). Each item in a partitioned stream can only belong to a single partition.

An unpartitioned stream only has one chunk with a PartitionId of 00000000-0000-0000-0000-000000000000.

There are multiple reasons for partitioning streams. One of the benefits is that it gives a way for the developers to partition their events and the way they are processed in an Event Handler. Another reason for having partitions becomes apparent when needing to subscribe to other streams in other microservices. We’ll talk more about that in the Event Horizon section.

Public vs Private Streams

There are two different types of event streams; public and private. Private streams are exposed within their Tenant and public streams are additionally exposed to other microservices. Through the Event Horizon other microservices can subscribe to your public streams. Using a public filter you can filter out public events to public streams.

Stream Processor

A stream processor consists of an event stream and an event processor. It takes in a stream of events, calls the event processor to process the events in order, keeps track of which events have already been processed, which have failed and when to retry. Each stream processor can be seen as the lowest level unit-of-work in regards to streams and they all run at the same time, side by side, in parallel.

Since the streams are also uniquely identified by a stream id we can identify each stream processor by their SourceStream, EventProcessor pairing.

// structure of a StreamProcessor
StreamProcessor {
    SourceStream Guid
    EventProcessor Guid
    // the next event to be processed
    Position int
    // for keeping track of failures and retry attempts
    LastSuccesfullyProcessed DateTime
    RetryTime DateTime
    FailureReason string
    ProcessingAttempts int
    IsFailing bool
}

The stream processors play a central role in the Runtime. They enforce the most important rules of Event Sourcing; an event in a stream is not processed twice (unless the stream is being replayed) and that no event in a stream is skipped while processing.

Stream processors are constructs that are internal to the Runtime and there is no way for the SDK to directly interact with stream processors.

Dealing with failures

What should happen when a processor fails? We cannot skip faulty events, which means that the event processor has to halt until we can successfully process the event. This problem can be mitigated with a partitioned stream because the processing only stops for that single partition. This way we can keep processing the event stream even though one, or several, of the partitions fail. The stream processor will at some point retry processing the failing partitions and continue normally if it succeeds.

Event Processors

There are 2 different types of event processors:

• Filters that can create new streams
• Processors that process the event in the user’s code

These are defined by the user with Event Handlers & Filters.

When the processing of an event is completed it returns a processing result back to the stream processor. This result contains information on whether or not the processing succeeded or not. If it did not succeed it will say how many times it has attempted to process that event, whether or not it should retry and how long it will wait until retrying.

Multi-tenancy

When registering processors they are registered for every tenant in the Runtime, resulting in every tenant having their own copy of the stream processor.

Formula for calculating the total number of stream processors created:

(((2 x event handlers) + filters) x tenants)  + event horizon subscriptions = stream processors

Let’s provide an example:

For both the filter and the event processor “processors” only one stream processor is needed. But for event handlers we need two because it consists of both a filter and an event processor. If the Runtime has 10 tenants and the head has registered 20 event handlers we’d end up with a total of 20 x 2 x 10 = 400 stream processors.

2.4 - Event Handlers & Filters

In event-sourced systems it is usually not enough to just say that an Event occurred. You’d expect that something should happen as a result of that event occurring as well.

In the Runtime we can register 2 different processors that can process events; Event Handlers and Filters. They take in a Stream of events as an input and does something to each individual event.

Each of these processors is a combination of one or more Stream Processors and Event Processor. What it does to the event is dependent on what kind of processor it is. We’ll talk more about different processors later in this section.

Registration

In order to be able to deal with committed events, the heads needs to register their processors. The Runtime offers endpoints which initiates the registration of the different processors. Only registered processors will be ran. When the head disconnects from the Runtime all of the registered processors will be automatically unregistered and when it re-connects it will re-register them again. Processors that have been unregistered are idle in the Runtime until they are re-registered again.

Scope

Each processor processes events within a single scope. If not specified, they process events from the default scope. Events coming over the Event Horizon are saved to a scope defined by the event horizon Subscription.

Filters

The filter is a processor that creates a new stream of events from the event log. It is identified by a FilterId and it can create either a partitioned or unpartitioned stream. The processing in the filter itself is however not partitioned since it can only operate on the event log stream which is an unpartitioned stream.

flowchart LR
    EL[(Event Log)] --> StreamProcessor --> F[EventProcessor<br/>Filter code] --> S[(Stream)]

The filter is a powerful tool because it can create an entirely customized stream of events. It is up to the developer on how to filter the events, during filtering both the content and the metadata of the event is available for the filter to consider. If the filter creates a partitioned stream it also needs to include which partition the event belongs to.

However with great power comes great responsibility. The filters cannot be changed in a way so that it breaks the rules of streams. If it does, the Runtime would notice it and return a failed registration response to the head that tried to register the filter.

Public Filters

Since there are two types of streams there are two kinds of filters; public and private. They function in the same way, except that private filters creates private streams and a public filter creates public streams. Only public events can be filtered into a public stream.

Event Handlers

The event handler is a combination of a filter and an event processor. It is identified by an EventHandlerId which will be both the id of both the filter and the event processor.

flowchart LR
    subgraph implicit filter
        direction LR
        EL[(Event Log)] --> FSP[StreamProcessor] --> F[Filter based on<br/>EventType] --> S[(Stream)]
    end
    S --> SP[StreamProcessor]
    SP --> EP["EventProcessor<br/>Handle() function"]

The event handler’s filter is filtering events based on the EventType that the event handler handles.

Event handlers can be either partitioned or unpartitioned. Partitioned event handlers uses, by default, the EventSourceId of each event as the partition id. The filter follows the same rules for streams as other filters.

Changes to event handlers

As event handlers create a stream based on the types of events they handles, they have to uphold the rules of streams. Every time an event handler is registered the Runtime will check that these rules are upheld and that the event handlers definition wouldn’t invalidate the already existing stream. Most common ways of breaking the rules are:

Disallowed: Removing events from the stream

The event handler stops handling an event type that it has already handled. This would mean that events would have to be removed from the stream, breaking the append-only rule.

Given an event handler that handles DishPrepared and RecipeAdded events, and the following event log we would get the stream as follows:

flowchart LR
    subgraph Log[Event Log]
        DP1L[1: DishPrepared]
        RA1L[2: RecipeAdded]
        DA1L[3: DishAddedToMenu]:::Dashed
        DP2L[4: DishPrepared]
    end
    EH[Event Handler v1<br/>Handles:<br/>DishPrepared<br/>DishServed]
    DP1L --> EH --> DP1S
    RA1L --> EH --> RA1S
    DP2L --> EH --> DP2S
    subgraph S1[Stream before]
        DP1S[1: DishPrepared]
        RA1S[2: RecipeAdded]
        DP2S[3: DishPrepared]
    end

    classDef Dashed stroke-dasharray: 5, 5

The Event Handler creates an invalid stream by removing an already handled event type:

flowchart LR
    subgraph Log[Event Log]
        DP1L[1: DishPrepared]
        RA1L[2: RecipeAdded]:::Dashed
        DA1L[3: DishAddedToMenu]:::Dashed
        DP2L[4: DishPrepared]
    end
    EH2[Event Handler v2<br/>Handles:<br/>DishPrepared]
    DP1L --> EH2 --> DP1S2
    DP2L --> EH2 --> DP2S2

    subgraph S2[Stream after]
        DP1S2[1: DishPrepared]
        RA1S2{{?: RecipeAdded}}:::Error
        DP2S2[2: DishPrepared]
    end

    classDef Dashed stroke-dasharray: 5, 5
    classDef Error stroke:#ff0000,stroke-width:2px,stroke-dasharray: 5, 5

Since the RecipeAdded event-type has already been committed to the stream, the stream would have to be changed to remove the RecipeAdded event-type. This would break the append-only rule, as the stream would have to be changed. This change is invalid, and will be rejected by the Runtime.

Disallowed: Adding events in positions other than the end of the stream

The event handler starts handling a new event type that has already occurred in the event log. This would mean changing the ordering of events in the streams and break the append-only rule.

flowchart LR
    subgraph Log[Event Log]
        DP1L[1: DishPrepared]
        RA1L[2: RecipeAdded]:::Dashed
        DA1L[3: DishAddedToMenu]:::Dashed
        DP2L[4: DishPrepared]
    end
    EH[Event Handler v1<br/>Handles:<br/>DishPrepared]
    DP1L --> EH --> DP1S
    DP2L --> EH --> DP2S
    subgraph S1[Stream before]
        DP1S[1: DishPrepared]
        DP2S[3: DishPrepared]
    end

    classDef Dashed stroke-dasharray: 5, 5

The Event Handler creates an invalid stream by adding a new event at a position before the end of the existing stream:

flowchart LR
    subgraph Log[Event Log]
        DP1L[1: DishPrepared]
        RA1L[2: RecipeAdded]
        DA1L[3: DishAddedToMenu]:::Dashed
        DP2L[4: DishPrepared]
    end
    EH2[Event Handler v2<br/>Handles:<br/>DishPrepared<br/>RecipeAdded]
    DP1L --> EH2 --> DP1S2
    RA1L --> EH2 --> RA1S2
    DP2L --> EH2 --> DP2S2

    subgraph S2[Stream after]
        DP1S2[1: DishPrepared]
        RA1S2{{2: RecipeAdded}}:::Error
        DP2S2[3: DishPrepared]
    end

    classDef Dashed stroke-dasharray: 5, 5
    classDef Error stroke:#ff0000,stroke-width:2px

It is possible to add a new type of event into the handler if it doesn’t invalidate the stream. For example, you can add a new event type to the handler if it hasn’t ever been committed before any of the other types of events into the event log.

Replaying events

An event handler is meant to handle each events only once, however if you for some reason need to “replay” or “re-handle” all or some of the events for an event handler, you can use the Dolittle CLI to initiate this while the microservice is running.

The replay does not allow you to change what event types the event handler handles. To do this, you need to change the event handlers EventHandlerId. This registers a completely new event handler with the Runtime, and a completely new stream is created. This way no old streams are invalidated.

If you want to have an event handler for read models which replays all of its events whenever it changes, try using Projections instead, as they are designed to allow frequent changes.

Multi-tenancy

When registering processors they are registered for every tenant in the Runtime, resulting in every tenant having their own copy of the Stream Processor.

2.5 - Projections

A Projection is a special type of Event Handler, that only deals with updating or deleting Read Models based on Events that it handles. The read model instances are managed by the Runtime in a read model store, where they are fetched from whenever needed. This is useful, for when you want to create views from events, but don’t want to manually manage the read model database.

Read models defines the data views that you are interested in presenting, while a projection specifies how to compute this view from the event store. There is a one-to-one relationship between a projection and their corresponding read model. A projection can produce multiple instances of that read model and it will assign each of them a unique key. This key is based on the projections key selectors.

Example of a projection:

flowchart LR
    subgraph Business moments
        direction LR
        CR["Customer Registered<br/>Id: 123<br/>Name: John Doe"]
        DAO["Debit Account Opened<br/>Id: 456<br/>Balance: 0"]
        DP["Debit Performed<br/>Account: 456<br/>Amount: $20"]
        WP["Withdrawal Performed<br/>Account: 56<br/>Amount: $10"]
    end
    subgraph Operations
        CR --> O1["Customer = Id<br/>Name = Name"]
        DAO --> O2["Id = Id<br/>Type = Debit"]
        DP --> O3["Id = Id<br/>Amount += Amount"]
        WP --> O4["Id = Id<br/>Amount -= Amount"]
    end
    subgraph Read Model
        O1 --> RM
        O2 --> RM
        O3 --> RM
        O4 --> RM["Account Details
        Id: 456
        Type: Debit
        Customer: 123
        Name: John Doe
        Balance: $10"]
    end

Read model

A read model represents a view into the data in your system, and are used when you want to show data or build a view. It’s essentially a Data transfer object (DTO) specialized for reading. They are computed from the events, and are as such read-only object without any behaviour seen from the user interface. Some also refer to read models as materialized views.

As read models are computed objects, you can make as many as you want based on whatever events you would like. We encourage you to make every read model single purpose and specialized for a particular use. By splitting up or combining data so that a read model matches exactly what an end-user sees on a single page, you’ll be able to iterate on these views without having to worry how it will affect other pages.

On the other hand, if you end up having to fetch more than one read model to get the necessary data for a single page, you should consider combining those read models.

The read models are purely computed values, which you are free to throw them away or recreate lost ones at any point in time without loosing any data.

The Runtime stores the read models into a read model store, which is defined in the resources.json. Each read model gets its own unique key, which is defined by the projections key selector.

Projection

A projections purpose is to populate the data structure (read model) with information from the event store. Projections behave mostly like an event handler, but they don’t produce a Stream from the events that it handles. This means that changing a projection (like adding or removing handle methods from it) will always make it replay and recalculate the read models from the start of the Event Log. This makes it easier to iterate and develop these read models.

This is a simplified structure of a projection:

Projection {
    ProjectionId Guid
    Scope Guid
    ReadModel type
    EventTypes EventType[]
}

For the whole structure of a projections as defined in protobuf, please check Contracts.

Key selector

Each read model instance has a key, which uniquely identifies it within a projection. A projection handles multiple instances of its read models by fetching the read model with the correct key. It will then apply the changes of the on methods to that read model instance.

The projection fetches the correct read model instance by specifying the key selector for each on method. There are 3 different key selector:

• Event source based key selector, which defines the read model instances key as the events EventSourceId.
• Event property based key selector, which defines the key as the handled events property.
• Partition based key selector, which defines the key as the events streams PartitionId.

2.6 - Tenants

Dolittle supports having multiple tenants using the same software out of the box.

What is a Tenant?

A Tenant is a single client that’s using the hosted software and infrastructure. In a SaaS (Software-as-a-Service) domain, a tenant would usually be a single customer using the service. The tenant has its privileges and resources only it has access to.

What is Multi-tenancy?

In a multi-tenant application, the same instance of the software is used to serve multiple tenants. An example of this would be an e-commerce SaaS. The same basic codebase is used by multiple different customers, each who has their own customers and their own data.

Multi-tenancy allows for easier scaling, sharing of infrastructure resources, and easier maintenance and updates to the software.

flowchart TB
    T1((Tenant A)) --> A[Application]
    T2((Tenant B)) --> A
    T3((Tenant C)) --> A
    A --> DB1[(Tenant A)]
    A --> DB2[(Tenant B)]
    A --> DB3[(Tenant C)]

Multi-tenancy in Dolittle

In Dolittle, every tenant in a Microservice is identified by a GUID. Each tenant has their own Event Store, and Read Cache managed by the Runtime. These event stores are defined in the Runtime configuration files. The tenants all share the same Runtime, which is why you need to specify the tenant which to connect to when using the SDKs.

2.7 - Event Horizon

At the heart of the Dolittle runtime sits the concept of Event Horizon. Event horizon is the mechanism for a microservice to give Consent for another microservice to Subscribe to its Public Stream and receive Public Events.

flowchart BT
    subgraph Producer
        ProdEventLog[(Event Log)] -->|Public events| PublicFilter[Public Filter]
        PublicFilter -->|matches go into| PublicStream[(Public Stream)]
        Consent(((Consent))) -->|gives access to| PublicStream
    end
    subgraph Consumer
        direction LR
        Subscription(((Subscription)))
        Subscription -->|stores| ConEventLog[(Scoped Event Log)]
    end
    Subscription -->|asks for events| Consent

Producer

The producer is a Tenant in a Microservice that has one or more public streams that Consumer can subscribe to. Only public events are eligible for being filtered into a public stream.

Once an event moves past the event horizon, the producer will no longer see it. The producer doesn’t know or care, what happens with an event after it has gone past the event horizon.

Consent

The producer has to give consent for a consumer to subscribe to a Partition in the producers public stream. Consents are defined in event-horizon-consents.json.

Consumer

A consumer is a tenant that subscribes to a partition in one of the Producer’s public streams. The events coming from the producer will be stored into a Scoped Event Log in the consumer’s event store. This way even if the producer would get removed or deprecated, the produced events are still saved in the consumer. To process events from a scoped event log you need scoped event handlers & filters.

The consumer sets up the subscription and will keep asking the producer for events. The producers Runtime will check whether it has a consent for that specific subscription and will only allow events to flow if that consent exists. If the producer goes offline or doesn’t consent, the consumer will keep retrying.

Subscription

A subscription is setup by the consumer to receive events from a producer. Additionally the consumer has to add the producer to its microservices.json.

This is a simplified structure of a Subscription in the consumer.

Subscription {
    // the producers microservice, tenant, public stream and partition
    MicroserviceId Guid
    TenantId Guid
    PublicStreamId Guid
    PartitionId string
    // the consumers scoped event log
    ScopeId Guid
}

Event migration

2.8 - Event Store

An Event Store is a database optimized for storing Events in an Event Sourced system. The Runtime manages the connections and structure of the stored data. All Streams, Event Handlers & Filters, Aggregates and Event Horizon Subscriptions are being kept track inside the event store.

Events saved to the event store cannot be changed or deleted. It acts as the record of all events that have happened in the system from the beginning of time.

Each Tenant has their own event store database, which is configured in resources.json.

Scope

Events that came over the Event Horizon need to be put into a scoped collection so they won’t be mixed with the other events from the system.

Scoped collections work the same way as other collections, except you can’t have Public Streams or Aggregates.

Structure of the Event Store

• MongoDB

{
    // this it the events EventLogSequenceNumber,
    // which identifies the event uniquely within the event log
    "_id": "decimal",
    "Content": "object",
    // Aggregate metadata
    "Aggregate": {
        "wasAppliedByAggregate": "bool",
        // AggregateRootId
        "TypeId": "UUID",
        // AggregateRoot Version
        "TypeGeneration": "long",
        "Version": "decimal"
    },
    // EventHorizon metadata
    "EventHorizon": {
        "FromEventHorizon": "bool",
        "ExternalEventLogSequenceNumber": "decimal",
        "Received": "date",
        "Concent": "UUID"
    },
    // the committing microservices metadata
    "ExecutionContext": {
        // 
        "Correlation": "UUID",
        "Microservice": "UUID",
        "Tenant": "UUID",
        "Version": "object",
        "Environment": "string",
    },
    // the events metadata
    "Metadata": {
        "Occurred": "date",
        "EventSource": "string",
        // EventTypeId and Generation
        "TypeId": "UUID",
        "TypeGeneration": "long",
        "Public": "bool"
    }
}

{
    "EventSource": "string",
    // the AggregateRootId
    "AggregateType": "UUID",
    "Version": "decimal"
}

{
    // same as an Event in the "event-log" + Partition
    "Partition": "string",
}

{
    // id of the Stream the Filter creates
    "_id": "UUID",
    "Partitioned": "bool",
    "Public": "bool",
    "Filter": {
        "Type": "string",
        "Types": [
            // EventTypeIds to filter into the stream
        ]
    }
}

{
    "SourceStream": "UUID",
    "EventProcessor": "UUID",
    "Position": "decimal",
    "LastSuccesfullyProcessed": "date",
    // failure tracking information
    "RetryTime": "date",
    "FailureReason": "string",
    "ProcessingAttempts": "int",
    "IsFailing": "bool
}

{
    "Partitioned": true,
    "SourceStream": "UUID",
    "EventProcessor": "UUID",
    "Position": "decimal",
    "LastSuccessfullyProcessed": "date",
    "FailingPartitions": {
        // for each failing partition
        "<partition-id>": {
            // the position of the failing event in the stream
            "Position": "decimal",
            "RetryTime": "date",
            "Reason": "string",
            "ProcessingAttempts": "int",
            "LastFailed": "date"
        }
    }
}

{
    // producers microservice, tenant and stream info
    "Microservice": "UUID",
    "Tenant": "UUID",
    "Stream": "UUID",
    "Partition": "string",
    "Position": "decimal",
    "LastSuccesfullyProcessed": "date",
    "RetryTime": "date",
    "FailureReason": "string",
    "ProcessingAttempts": "int",
    "IsFailing": "bool
}

Commit vs Publish

We use the word Commit rather than Publish when talking about saving events to the event store. We want to emphasize that it’s the event store that is the source of truth in the system. The act of calling filters/event handlers comes after the event has been committed to the event store. We also don’t publish to any specific stream, event handler or microservice. After the event has been committed, it’s ready to be picked up by any processor that listens to that type of event.

2.9 - Event Sourcing

Event Sourcing is an approach that derives the current state of an application from the sequential Events that have happened within the application. These events are stored to an append-only Event Store that acts as a record for all state changes in the system.

Events are facts and Event Sourcing is based on the incremental accretion of knowledge about our application / domain. Events in the log cannot be changed or deleted. They represent things that have happened. Thus, in the absence of a time machine, they cannot be made to un-happen.

Here’s an overview of the data-flow in Event Sourcing:

flowchart TB
    Presentation --produces--> Events[/Events/]
    Events --stored in--> EventStore[(Event Store)]
    EventStore --- SendToConsumers["Events are<br/>sent to consumers"]:::transparent
    SendToConsumers --> External([External Systems])
    SendToConsumers --> Consumer --Generates the read cache--> ReadCache[(Read Cache)]
    ReadCache -->|Query for read data| Presentation

    classDef transparent stroke-width:0px,fill:#fff0;

Problem

A traditional model of dealing with data in applications is CRUD (create, read, update, delete). A typical example is to read data from the database, modify it, and update the current state of the data. Simple enough, but it has some limitations:

• Data operations are done directly against a central database, which can slow down performance and limit scalability
• Same piece of data is often accessed from multiple sources at the same time. To avoid conflicts, transactions and locks are needed
• Without additional auditing logs, the history of operations is lost. More importantly, the reason for changes is lost.

Advantages with Event Sourcing

• Horizontal scalability
  • With an event store, it’s easy to separate change handling and state querying, allowing for easier horizontal scaling. The events and their projections can be scaled independently of each other.
  • Event producers and consumers are decoupled and can be scaled independently.
• Flexibility
  • The Event Handlers react to events committed to the event store. The handlers know about the event and its data, but they don’t know or care what caused the event. This provides great flexibility and can be easily extended/integrated with other systems.
• Replayable state
  • The state of the application can be recreated by just re-applying the events. This enables rollbacks to any previous point in time.
  • Temporal queries make it possible to determine the state of the application/entity at any point in time.
• Events are natural
  • Events are easily modeled in domain terms, avoiding object-relational impedance mismatch. Events are simple objects describing actions.
• Audit log
  • The whole history of changes is recorded in an append-only store for later auditing.
  • Instead of being a simple record of reads/writes, the reason for change is saved within the events.

Problems with Event Sourcing

• Eventual consistency
  • As the events are separated from the projections made from them, there will be some delay between committing an event and handling it in handlers and consumers.
• Event store is append-only
  • As the event store is append-only, the only way to update an entity is to create a compensating event.
  • Changing the structure of events is hard as the old events still exist in the store and need to also be handled.

Projections

The Event Store defines how the events are written in the system, it does not define or prescribe how things are read or interpreted. Committed events will be made available to any potential subscribers, which can process the events in any way they require. One common scenario is to update a read model/cache of one or multiple views, also known as a projections or materialized views. As the Event Store is not ideal for querying data, a prepopulated view that reacts to changes is used instead. Dolittle has built-in support for a specific style of projection, and allows free-form handling of events through event handlers.

Compensating events

To negate the effect of an Event that has happened, another Event has to occur that reverses its effect. This can be seen in any mature Accounting domain where the Ledger is an immutable event store or journal. Entries in the ledger cannot be changed. The current balance can be derived at any point by accumulating all the changes (entries) that have been made and summing them up (credits and debts). In the case of mistakes, an explicit correcting action would be made to fix the ledger.

Commit vs Publish

Dolittle doesn’t publish events, rather they are committed. Events are committed to the event log, from which any potential subscribers will pick up the event from and process it. There is no way to “publish” to a particular subscriber as all the events are available on the event log, but you can create a Filter that creates a Stream.

Reason for change

By capturing all changes in the forms of events and modeling the why of the change (in the form of the event itself), an Event Sourced system keeps as much information as possible.

A common example is of a e-shopping that wants to test a theory:

A user who has an item in their shopping cart but does not proceed to buy it will be more likely to buy this item in the future

In a traditional CRUD system, where only the state of the shopping cart (or worse, completed orders) is captured, this hypothesis is hard to test. We do not have any knowledge that an item was added to the cart, then removed.

On the other hand, in an Event Sourced system where we have events like ItemAddedToCart and ItemRemovedFromCart, we can look back in time and check exactly how many people had an item in their cart at some point and did not buy it, subsequently did. This requires no change to the production system and no time to wait to gather sufficient data.

When creating an Event Sourced system we should not assume that we know the business value of all the data that the system generates, or that we always make well-informed decisions for what data to keep and what to discard.

Further reading

• Martin Fowler on Event Sourcing
• Microsoft on Event Sourcing pattern

2.10 - Aggregates

An Aggregate is Domain-driven design (DDD) term coined by Eric Evans. An aggregate is a collection of objects and it represents a concept in your domain, it’s not a container for items. It’s bound together by an Aggregate Root, which upholds the rules (invariants) to keep the aggregate consistent. It encapsulates the domain objects, enforces business rules, and ensures that the aggregate can’t be put into an invalid state.

Example

For example, in the domain of a restaurant, a Kitchen could be an aggregate, where it has domain objects like Chefs, Inventory and Menu and an operation PrepareDish.

The kitchen would make sure that:

• A Dish has to be on the Menu for it to be ordered
• The Inventory needs to have enough ingredients to make the Dish
• The Dish gets assigned to an available Chef

Here’s a simple C#ish example of what this aggregate root could look like:

public class Kitchen
{
    Chefs _chefs;
    Inventory _inventory;
    Menu _menu;

    public void PrepareDish(Dish dish)
    {
        if (!_menu.Contains(dish))
        {
            throw new DishNotOnMenu(dish);
        }
        foreach (var ingredient in dish.ingredients)
        {
            var foundIngredient = _inventory
                .GetIngredient(ingredient.Name);
            if (!foundIngredient)
            {
                throw new IngredientNotInInventory(ingredient);
            }

            if (foundIngredient.Amount < ingredient.Amount)
            {
                throw new InventoryOutOfIngredient(foundIngredient);
            }
        }
        var availableChef = _chefs.GetAvailableChef();
        if (!availableChef)
        {
            throw new NoAvailableChefs();
        }
        availableChef.IsAvailable = false;
    }
}

Aggregates in Dolittle

With Event Sourcing the aggregates are the key components to enforcing the business rules and the state of domain objects. Dolittle has a concept called AggregateRoot in the Event Store that acts as an aggregate root to the AggregateEvents applied to it. The root holds a reference to all the aggregate events applied to it and it can fetch all of them.

Structure of an AggregateRoot

This is a simplified structure of the main parts of an aggregate root.

AggregateRoot {
    AggregateRootId Guid
    EventSourceId string
    Version int
    AggregateEvents AggregateEvent[] {
        EventSourceId Guid
        AggregateRootId Guid
        // normal Event properties also included
        ...
    }
}

AggregateRootId

Identifies this specific type of aggregate root. In the kitchen example this would a unique id given to the Kitchen class to identify it from other aggregate roots.

EventSourceId

EventSourceId represents the source of the event like a “primary key” in a traditional database. In the kitchen example this would be the unique identifier for each instance of the Kitchen aggregate root.

Version

Version is the position of the next AggregateEvent to be processed. It’s incremented after each AggregateEvent has been applied by the AggregateRoot. This ensures that the root will always apply the events in the correct order.

AggregateEvents

The list holds the reference ids to the actual AggregateEvent instances that are stored in the Event Log. With this list the root can ask the Runtime to fetch all of the events with matching EventSourceId and AggregateRootId.

Designing aggregates

When building your aggregates, roots and rules, it is helpful to ask yourself these questions:

• “What is the impact of breaking this rule?"
• “What happens in the domain if this rule is broken?"
• “Am I modelling a domain concern or a technical concern?"
• “Can this rule be broken for a moment or does it need to be enforced immediately?"
• “Do these rules and domain objects break together or can they be split into another aggregate?"

Further reading

• Aggregates by Martin Fowler
• Aggregates in Domain Driven Design
• Uncovering Hidden Business Rules with DDD Aggregates by Nick Tune

2.11 - Concurrency

Introduction

When your Event handlers are processing long streams of events you might want to have them processed concurrently to speed up the processing. This can be achieved by setting the concurrency -attribute on the processor’s EventHandler -attribute.

// instead of
[EventHandler("...")]
public class SequentialHandler
{
    // ..
}

// use
[EventHandler(eventHandlerId: "...", concurrency: 100)]
public class ConcurrentHandler
{
    // ..
}

Sequential processing

An event handler with just an ID set in the EventHandler -attribute, the events will be processed sequentially. Having no concurrency set is the same as setting it to 1.

[EventHandler("...")]
public class HandleSequentially
{
    public Task Handle(CustomerCreated e, EventContext ctx)
    {
        //..
    }

    public Task Handle(CustomerDeleted e, EventContext ctx)
    {
        //..
    }
}

This is the default. All the events in the stream will be processed in-order as they arrive. When running a re-play or with an existing stream of events the processing will happen one-by-one.

Concurrent processing

An event handler with concurrency set in the EventHandler -attribute, the events will be processed concurrently.

[EventHandler(eventHandlerId: "...", concurrency: 100)]
public class HandleConcurrently
{
    public Task Handle(CustomerCreated e, EventContext ctx)
    {
        // ..
    }

    public Task Handle(CustomerDeleted e, EventContext ctx)
    {
        //..
    }
}

The stream of events will be split up by the EventSourceId of the events, and up to 100 (in this case) of these will be processed concurrently. This means that an event-stream of a million events with a thousand different event-sources will be processed 100 at a time. There will be one event-handler -instance per event-source.

If you are using Aggregates to commit events the EventSourceId will be the EventSourceId of the aggregate-root. If you are committing events directly to the Event Store as demonstrated in the Getting started tutorial the EventSourceId will be part of the call to the Event Store.

Enabling concurrency often yields a dramatic increase in speed of processing large streams of events, but there are consequences you need to be aware of when writing concurrent event-handlers.

Consequences

Your event-handler will no longer be a single processor moving along a stream of events, which will affect how they behave. You may no longer assume that something that happened “earlier” in the event-stream will have happened in your current handling-method, as it might still be waiting for processing on a different event-source-id. This introduces a dependency between the part of your system that inserts events (in particular the event-source and its Id), and the processing of these events.

Example

Let’s illustrate with an example.

We have created a system with customers and orders as different aggregates. We use a customer-number as the event-source-id for the customer, and the order-number as the event-source-id for the order.

We have an event-handler that handles CustomerCreated, CustomerDeleted, OrderCreated OrderLineAdded and OrderShipped -events.

[AggregateRoot("...")]
public class Customer : AggregateRoot
{
    readonly CustomerNumber _id;

    public Customer(EventSourceId id) : base(id) => _id = id;

    public Create() => Apply(new CustomerCreated(_id));
    public Delete() => Apply(new CustomerDeleted(_id));
}

[AggregateRoot("...")]
public class Order : AggregateRoot
{
    readonly OrderNumber _id;

    public Order(EventSourceId id) : base(id) => _id = id;

    public Create(CustomerNumber c) => Apply(new OrderCreated(_id, c));
    public Add(OrderLine line) => Apply(new OrderLineAdded(_id, line)
    public Ship() => Apply(new OrderShipped(_id));
}

In our system we have the following event-stream for a customer 5, and order 42 with 2 lines:

%%{init: {'gitGraph': { 'showBranches': false }}}%%
gitGraph
    commit id: "1: CustomerCreated(5)"
    commit id: "2: OrderCreated(42, 5)"
    commit id: "3: OrderLineAdded(42, 1)"
    commit id: "4: OrderLineAdded(42, 2)"
    commit id: "5: OrderShipped(42)"
    commit id: "6: CustomerDeleted(5)"

Our event-handler that handles these sequentially will get the events in that order, and process them.

[EventHandler("...")]
public class HandleSequentially
{
    public Task Handle(CustomerCreated e, EventContext ctx)
    {
        //..
    }

    public Task Handle(CustomerDeleted e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderCreated e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderLineAdded e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderShipped e, EventContext ctx)
    {
        //..
    }
}

A concurrent event-handler will process each event-source-id concurrently, so the CustomerCreated and CustomerDeleted -events will be processed in order, and the OrderCreated, OrderLineAdded and OrderShipped -events will be processed in-order, but the ordering between these two event-source-ids is no longer guaranteed.

[EventHandler("...", concurrency: 100)]
public class HandleConcurrentlyByEventSourceId
{
    public Task Handle(CustomerCreated e, EventContext ctx)
    {
        //..
    }

    public Task Handle(CustomerDeleted e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderCreated e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderLineAdded e, EventContext ctx)
    {
        //..
    }

    public Task Handle(OrderShipped e, EventContext ctx)
    {
        //..
    }
}

%%{init: {'gitGraph': { 'showBranches': false }}}%%
gitGraph
    commit id: "start"
    branch customer
    commit id: "CustomerCreated(5)"
    commit id: "CustomerDeleted(5)"
    checkout main
    branch order
    commit id: "OrderCreated(42, 5)"
    commit id: "OrderLineAdded(42, 1)"
    commit id: "OrderLineAdded(42, 2)"
    commit id: "OrderShipped(42)"

One possible actual ordering of these events as they run through your concurrently processing event-handler could be:

%%{init: {'gitGraph': { 'showBranches': false }}}%%
gitGraph
    commit id: "start"
    branch customer
    commit id: "CustomerCreated(5)"
    checkout main
    merge customer tag:"customer exists"
    branch order
    commit id: "OrderCreated(42, 5)"
    checkout main
    merge order tag:"order exists"
    checkout customer
    commit id: "CustomerDeleted(5)"
    checkout main
    merge customer tag:"customer deleted"
    checkout order
    commit id: "OrderLineAdded(42, 1)"
    checkout main
    merge order tag: "line 1 added"
    checkout order
    commit id: "OrderLineAdded(42, 2)"
    checkout main
    merge order tag: "line 2 added"
    checkout order
    commit id: "OrderShipped(42)"
    checkout main
     merge order tag: "order shipped"

As you can see, the CustomerDeleted -event is processed before the OrderLineAdded -events. If the handle-method for OrderLineAdded -event needs to access the customer, it will not be able to do so, as the customer has been deleted. The same is true for the OrderShipped -event, it might also need to access the customer, and it will not be able to do so.

Is this a good thing, is it even acceptable? That is up to you to decide. It is important to be aware of this, and to design your system accordingly.

If your processing exists to create an order-page for the customer, having the customer deletion happen before all the order-events might be a good thing, as you know this customer will end up deleting their account and you do not need to create a page that will never be shown. However, you need to be aware of it, and make sure that the handler does not crash when the customer has been deleted.

Mitigations

Concurrency adds complexity to your system, and you need to be aware of this complexity and design your system accordingly. There are ways to mitigate this complexity.

Option 1 - don’t use concurrency

The simplest way to mitigate this is to not use concurrent event-handlers. If you do not need the performance boost, or if you do not want to deal with the complexity of concurrent processing, just use sequential processing.

Option 2 - use the same EventSourceId

If you need to use concurrency the best way to deal with them is to have the same EventSourceId for the events that you want guarantee the ordering of. In the example above, if we want to guarantee that the OrderCreated, OrderLineAdded and OrderShipped -events are processed in-order, we can use the CustomerId as the EventSourceId for all of them.

In fact the events need not come from the same event-source, as long as they have the same EventSourceId they will be processed in-order.

[AggregateRoot("...")]
public class Customer : AggregateRoot
{
    readonly CustomerNumber _id;

    public Customer(EventSourceId id) : base(id) => _id = id;

    public Create() => Apply(new CustomerCreated(_id));
    public Delete() => Apply(new CustomerDeleted(_id));
}

[AggregateRoot("...")]
public class CustomerOrders : AggregateRoot
{
    readonly CustomerNumber _customerNumber;
    readonly OrderNumber _orderNumber;

    public Order(EventSourceId customerNumber) : base(customerNumber) => _customerId = id;

    public Create(OrderId o) => Apply(new OrderCreated(_orderNumber, _customerNumber));
    public Add(OrderLine line) => Apply(new OrderLineAdded(_orderNumber, line)
    public Ship() => Apply(new OrderShipped(_orderNumber));

    void On(OrderCreated e) => _orderNumber = e.OrderId;
}

Option 3 - consolidate to a single aggregate

You could put the orders into the customer -aggregate to guarantee that they are processed in-order.

[AggregateRoot("...")]
public class Customer : AggregateRoot
{
    readonly CustomerNumber _id;
    readonly List<Order> _orders = new List<Order>();

    public Customer(EventSourceId id) : base(id) => _id = id;

    public Create() => Apply(new CustomerCreated(_id));
    public Delete() => Apply(new CustomerDeleted(_id));

    public CreateOrder(OrderNumber o) => Apply(new OrderCreated(_id));

    public AddOrderLine(OrderNumber o, OrderLine line) =>
        _orders.Any(o => o.Number == o)
            ? Apply(new OrderLineAdded(_id, line)
            : throw new InvalidOperationException("Order does not exist");

    public ShipOrder(OrderNumber o) =>
        _orders.Single(o => o.Number == o).Shipped == false
            ? Apply(new OrderShipped(_id))
            : throw new InvalidOperationException("Order already shipped");

    void On(OrderCreated e) => _orders.Add(new Order(e.OrderId));
    void On(OrderLineAdded e) => _orders.Single(o => o.Id == e.OrderId).AddLine(e.Line);
    void On(OrderShipped e) => _orders.Single(o => o.Id == e.OrderId).Ship();

    class Order
    {
        public OrderNumber Number { get; init; }
        public bool Shipped { get; private set; } = false
        readonly List<OrderLine> _lines = new List<OrderLine>();

        public Order(OrderNumber number) => _number = number;

        public AddLine(OrderLine line) => _lines.Add(line);
        public Ship() => Shipped  = true;
    }
}

Option 4 - defensive handling

When you write your event-handlers you may be able to write them such that they can handle the events out of order, or at least only guaranteed in-order within an event-source. This is not always possible, but if it is, it is a good way to mitigate the complexity of concurrent processing.

This depends on what you do in your handler. If you are storing a read-model you may have to deal with partial or missing data, if you are calling on external services you might have to deal with them being unavailable, or not supporting the order in which you are calling them.

Summary

Activating concurrency can lead to great performance improvements, but it comes at a cost. To safely use concurrency you should be aware of the implications of concurrent processing, and design your system accordingly.

• handling order will depend on the EventSourceId of the events
• multiple event-handlers will be running concurrently
  • watch out for shared state
  • watch out for resource-usage
• good to have a single EventSourceId per unit of replay

Conclusion

Activating concurrency can lead to great performance improvements, but it comes at a cost. To safely use concurrency you should be aware of the implications of concurrent processing, and design your system accordingly.

2.12 - Resource System

When using the Dolittle SDK you get access to the Resource System which is a way to get access to storage. The Resources you get will be separated by Tenants and unique in your current context. This means that you can depend on the stored data not leaking between tenants. The event-store and the read-cache are the only permanent storage options available through the Resource System.

Read Cache

The Read Cache, or ReadModels, is available from the Resource System as an IMongoDatabase that you can reference in your code. The database will be connected, and you can use normal MongoDB queries to get access to the data you store there.

Example

A minimal example of an AspNet Core web application that uses the Dolittle SDK and the Read Cache could look like this:

using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using MongoDB.Driver;
using ResourceSystemDemo;

var builder = WebApplication.CreateBuilder(args);

// Add Dolittle to the web-application's host.
builder.Host.UseDolittle();

var app = builder.Build();

await app.StartAsync();

// get a client
var dolittleClient = await app.GetDolittleClient();

// get a reference the the development-tenant's read-cache
var tenantedReadCache = dolittleClient
    .Resources
    .ForTenant(TenantId.Development)
    .MongoDB
    .GetDatabase();

var id = Guid.NewGuid().ToString();

// the ReadModel has only Id and Value in this example
var readModel = new ReadModel(id, "this is my data");

// in this example the collection-name is "models"
const string collectionName = "models";
tenantedReadCache
    .GetCollection<ReadModel>(collectionName)
    .InsertOne(readModel);

// get the inserted read-model back from storage
var retrievedFromStorage = tenantedReadCache
    .GetCollection<ReadModel>(collectionName)
    .AsQueryable()
    .Where(rm => rm.Id == id)
    .Single();

Console.WriteLine(
    $"retrieved read-model was {retrievedFromStorage}"
);

// to keep running: await app.WaitForShutdownAsync();
await app.StopAsync();

In your actual code you would need to decide the collection-name and build your ReadModels to match your needs.

Note that the interactions with the tenantedReadCache are all done using a normal MongoDB driver. This means that you can use the full power of MongoDB to query and update your data.

Accessing the Event Store directly

The Event Store is available from the Dolittle SDK directly. The backing data-storage is MongoDB, but no MongoDB driver access is available from the SDK. To interact with the Event Store there are methods available to

• commit events
• commit public events
• get all the events for an aggregate-root
• get all the events in a stream for an aggregate-root

Committing events and public events

With a Dolittle Client from the Dolittle SDK you can commit events to the Event Store directly (skipping any aggregates).

// assume we have the app like above
var dolittleClient = await app.GetDolittleClient();

var tenantedEventStore = dolittleClient
    .EventStore
    .ForTenant(TenantId.Development);

// assume we have an Event -type called SomeEvent with Id and Value
var someEvent = new SomeEvent(
    Id: Guid.NewGuid().ToString(),
    Value: "this is my data");

tenantedEventStore
    .CommitEvent(
        content: someEvent,
        eventSourceId: "Demo"
    );

// assume we have another Event -type called SomePublicEvent
var somePublicEvent = new SomePublicEvent(
    Id: Guid.NewGuid().ToString(),
    Value: "this is data going on the public stream");

tenantedEventStore
    .CommitPublicEvent(
        content: somePublicEvent,
        eventSourceId: "Demo"
    );

Getting events for an aggregate-root

Committed events from the Event Store are available if they are associated with an aggregate-root. You need to know the aggregate-root’s ID and the event-source id to get the events. Remember that the aggregate-root ID identifies the type of aggregate-root and event-source ID identifies the instance of that aggregate-root-type.

You can either get all the events as a collection, or you can get it as a streaming IAsyncEnumerable.

We have a minimal aggregate-root type called Lookout with an aggregate-root id:

[AggregateRoot("B92DE697-2E09-4AE1-99A2-3BB72925B0AF")]
public class Lookout : AggregateRoot
{
    public void SeeSomething() =>
        Apply(new SomeEvent(Guid.NewGuid(), "something happened"));
}

We call on the lookout with the id “Alice” to see something through the Dolittle Client:

await dolittleClient
    .Aggregates
    .ForTenant(TenantId.Development)
    .Get<Lookout>("Alice")
    .Perform(
        alice => alice.SeeSomething()
    );

We can now access the events from an instance of that aggregate-root type like this:

var events = await dolittleClient
    .EventStore
    .ForTenant(TenantId.Development)
    .FetchForAggregate("B92DE697-2E09-4AE1-99A2-3BB72925B0AF", "Alice");

Console.WriteLine($"Alice has seen something {events.Count} times");

If we want to get the events as a streaming IAsyncEnumerable (there may be many events) we can do it like this:

var stream = dolittleClient
    .EventStore
    .FetchStreamForAggregate("B92DE697-2E09-4AE1-99A2-3BB72925B0AF", "Alice");

await foreach (var chunk in stream)
{
    Console.WriteLine("streaming to: " + chunk.AggregateRootVersion);
    foreach(var evt in chunk)
    {
        Console.WriteLine("evt: " + evt.Content);
    }
}

A much simpler way to handle events is usually to write an EventHandler or Projection that will handle the events for you.

Summary

The Resource System is a way to get access to storage in your Dolittle application. The Read Cache is a way to get access to a MongoDB database that is unique to your tenant. The Event Store is a way to get access to the events that have been committed to the Event Store by Aggregate-Roots for your tenant.

3 - Platform

Aigonix Platform is our PaaS(Platform-as-a-Service) solution for hosting your Dolittle microservices in the cloud.

3.1 - Requirements

To be compatible with the environment of the Aigonix platform, there are certain requirements we impose on your microservices. If they are not met, your application might behave unexpectedly - or in the worst case - not work at all. The following list of requirements is subject to change, but we will always notify you when you have an application running in our platform before making any changes.

1. Your application must use the resource system

To ensure data privacy, security and proper segregation of your tenant’s data, our platform has a resource management system. This system controls access and connection settings for resources on a per request basis and will provide your microservice with the necessary information for accessing these resources programmatically. The connection information will not be the same as when developing locally, so you must not embed connection settings in your code.

This requirement applies to read and write data to databases or files, or while making API-calls to services, both to internal resources provided by the Aigonix platform and external 3rd party services.

2. All your applications external endpoints must be configured and exposed through the platform

For the resource management system to work, and to protect your application and users from data leakage, we encrypt and authenticate all interactions with your application through the platform. This means that your microservices will be completely isolated by default, and all endpoints that should be accessible outside our platform needs to be exposed explicitly and configured with appropriate encryption and authentication schemes.

To enable same-origin authentication flows and adhere to internet best practices, the platform will take control of a set of URIs for the hostnames you have allocated to your application. The following paths and any sub-path of these (in any form of capitalisation) are reserved for the platform:

• /.well-known
• /robots.txt
• /sitemap
• /api/Dolittle
• /Dolittle

3. Your microservices must be stateless, scalable and probeable

To allow for efficient hosting of your application, we have to able to upgrade, re-start, move and scale your microservices to handle the load and perform necessary security upgrades. This means that you must not rely on any in-memory state for anything apart from the per-transaction state, and you must not rely on there being a single instance of your microservices at any point in time.

To ensure that your microservices are healthy and ready to perform work, your microservices must expose both liveness and readiness probes. The microservice should respond to the liveness probe whenever it has successfully started and is in a functional state, and should respond to the readiness probe whenever it is free to handle incoming requests from users.

4. Your application must adhere to semantic versioning of your microservices

We rely on semantic versioning to properly track changes of your microservices (from an operational aspect) and to decide on the correct course of action when new versions of your microservices are built. Minor or patch increments will result in automatic upgrades of your running microservices without any human interaction, while major increments require manual approval and potential updates of configuration or data structures. This means that you must increment the major number when making changes to your microservices that require changes in the platform for your application to work properly.

5. Your frontend must be a static single-page application

To ensure that any user-facing frontend is served quickly and with minimal data-usage, we serve your frontend using separate servers with appropriate caching, compression and CDN strategies. This means that your frontend must be built as a single-page application to static HTML, CSS and js files. These files must be built and versioned alongside your backend microservices to ensure that the frontend and backend versions are aligned and function properly.

3.2 - Deploy an application

Prerequisites

Familiar with the following:

• Docker containers
• Kubernetes
• Microsoft Azure

Recommendation

For users on Windows OS, we recommend that you use WSL/Ubuntu as your shell/terminal instead of CMD/Powershell.

• Install WSL on Windows 10

Installation

Install the following software:

• Docker Desktop
• Kubectl
• Azure CLI

Configuration

After an environment has been provisioned for you in the Dolittle PaaS, you will receive these details to use with the deployment commands in the following sections:

Subscription ID
Resource Group
Cluster Name
Application Namespace
ACR Registry
Image Repository
Deployment Name
Application URL

Setup

All commands are meant to be run in a terminal (Shell)

AZURE

Login to Azure:

az login

AKS - Azure Container Service

Get credentials from Dolittle’s AKS cluster

az aks get-credentials -g <Resource Group> -n <Cluster Name> --subscription <Subscription ID>

ACR - Azure Container Registry

Get credentials to Azure Container Registry

az acr login -n <ACR Registry> --subscription <Subscription ID>

Deployment

To deploy a new version of your application, follow these steps. For use semantic versioning, e.g. “1.0.0”.

Docker

Build your image

docker build -t <Image Repository>:<Tag> .

Push the image to ACR

docker push <Image Repository>:<Tag>

Kubernetes

Patch the Kubernetes deployment to run your new version

kubectl patch --namespace <Application Namespace> deployment <Deployment Name> -p '{"spec": { "template": { "spec": { "containers": [{ "name":"head", "image": "<Image Repository>:<Tag>"}] }}}}'

Debugging

kubectl commands:

Show the status of your application pods

kubectl -n <Application Namespace> get pods

Show deployed version of your application

kubectl -n <Application Namespace> get deployment -o wide

Show the logs of the last deployed version of the application

kubectl -n <Application Namespace> logs deployments/<Deployment Name>

Logs for the application, last 100 lines

kubectl -n <Application Namespace> logs deployments/<Deployment Name> --tail=100

3.3 - Update configurations

Prerequisites

Familiar with the following:

• Kubernetes
• yaml

Recommendation

For users on Windows OS, we recommend that you use WSL/Ubuntu as your shell/terminal instead of CMD/Poweshell.

• Install WSL on Windows 10

Installation

Install the following software:

• Kubectl

Configuration

After an environment has been provisioned for you in the Dolittle PaaS, you will receive a yaml file per environment. The files will be similar to this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: application-namespace
  name: app-dev-ms-env-variables
  labels:
    tenant: Customer
    application: App-Dev
    microservice: MS-A
data:
  OPENID_AUTHORITY: "yourapp.auth0.com"
  OPENID_CLIENT: "client-id"
  OPENID_CLIENTSECRET: "client-secret"

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: application-namespace
  name: app-dev-ms-config-files
  labels:
    tenant: Customer
    application: App-Dev
    microservice: MS-A
data:
  myapp.json: |
    {
       "somekey": "somevalue"
    }    

The files represent configmap resources in Kubernetes. We recommend that you store the files in a version control system(VCS) of your choice.

Purpose

Each yaml file consists of 2 configmaps per micro-service:

• app-dev-ms-env-variables: This configmap is for your environmental variables that will be passed on to the container at start up.
• app-dev-ms-config-files: This configmap is for add/override files. The default mount point is app/data

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: application-namespace
  name: app-dev-ms-env-variables
  labels:
    tenant: Customer
    application: App-Dev
    microservice: MS-A
data:

You may alter the content under data:

  OPENID_AUTHORITY: "yourapp.auth0.com"
  OPENID_CLIENT: "client-id"
  OPENID_CLIENTSECRET: "client-secret"
  connectionstring__myconnection: "strings"

Alter existing or add new key/value pairs.

myapp.json: |
    {
       "somekey": "somevalue"
    }    

customSetting.json: |
  {
    "settings": {
       "connection":"connectionstring"
     }
  }  

Alter existing or add new JSON data that will be linked to a specific file that will be available at runtime under app/data/

Setup

You need to setup your AKS credentials.

Update configurations

To update the configurations:

kubectl apply -f <filename>

You must be in the directory of the yaml file before running the command.

To update/add a single key in the config:

kubectl patch -n <Application Namespace> configmap <Configmap Name>  -p '{"data":{"my-key":"value that i want"}}'

To remove a single key from the configuration:

kubectl patch -n <Application Namespace> configmap <Configmap Name>  -p '{"data":{"my-key":null}}'

See configurations

JSON output

kubectl get -n <Application Namespace> configmap <Configmap Name> -o json

YAML output:

kubectl get -n <Application Namespace> configmap <Configmap Name> -o yaml

For an advanced print out, you need a tool called jq for parsing

kubectl get -n configmap -o json | jq -j ‘.data | to_entries | .[] | “(.key): (.value)\n”’

3.4 - Update secrets

Prerequisites

Familiar with the following:

• Kubernetes
• yaml

Recommendation

For users on Windows OS, we recommend that you use WSL/Ubuntu as your shell/terminal instead of CMD/Poweshell.

• Install WSL on Windows 10

Installation

Install the following software:

• Kubectl

Secrets

After an environment has been provisioned for you in the Dolittle PaaS, you will receive a yaml file per environment. The files will be similar to this:

---
apiVersion: v1
kind: Secret
metadata:
  namespace: application-namespace
  name: apps-dev-ms-secret-env-variables
  labels:
    tenant: Customer
    application: App-Dev
    microservice: MS-A
type: Opaque
data:
   OPENID_SECRET: b3BlbiBpZCBzZWNyZXQ=

The files represent the Secrets -resource in Kubernetes. We recommend that you store the files in a version control system(VCS) of your choice.

Purpose

Each yaml file consists of a secret per micro-service:

• app-dev-ms-secret-env-variables: This secret is for your environmental variables that will be passed on to the container at start up. One important thing to remember is that the values have to be encoded using base64.

---
apiVersion: v1
kind: Secret
metadata:
  namespace: application-namespace
  name: apps-dev-ms-secret-env-variables
  labels:
    tenant: Customer
    application: App-Dev
    microservice: MS-A
type: Opaque
data:

You may alter existing or add new key/value pairs.

  OPENID_SECRET: b3BlbiBpZCBzZWNyZXQ=
  DB_PASSWORD: c29tZSBwYXNzd29yZA==

Setup

You need to setup your AKS credentials.

Encode secrets

To encode values:

echo -n "my super secret pwd" | base64 -w0

The above command will give you:

bXkgc3VwZXIgc2VjcmV0IHB3ZA==

The value can then be added to the secrets:

MY_SECRET: bXkgc3VwZXIgc2VjcmV0IHB3ZA==

Update secrets

To update the secrets:

kubectl apply -f <filename>

You must be in the directory of the yaml file before running the command.

To update/add a single key in the secrets:

kubectl patch -n <Application Namespace> secret <Secrets Name>  -p '{"data":{"my-key":"value that i want encoded using base64"}}'

To remove a single key from the configuration:

kubectl patch -n <Application Namespace> secret <Secrets Name>  -p '{"data":{"my-key":null}}'

See secrets

JSON output:

kubectl get -n <Application Namespace> secret <Secrets Name> -o json

YAML output:

kubectl get -n <Application Namespace> secret <Secrets Name> -o yaml

For an advanced print out, you need a tool called jq for parsing the JSON in you shell:

kubectl get -n <Application Namespace> secret <Secrets Name> -o json | jq -j '.data | to_entries | .[] | "\(.key): \(.value)\n"'

3.5 - FAQ

Can I login without allowing cookies?

If you’re getting strange results with logon through Sentry or another OIDC service - check that you’re allowing cookies for the domains!

Without cookies you cannot logon - at all. Sorry!

3.6 - Aigonix Studio

Aigonix Studio is your management-tool to interact with services and products you run in the Aigonix Platform. It is a web-based application that is available at dolittle.studio.

3.6.1 - Overview

Aigonix Studio is the web-based interface for managing your Applications and Microservices in the Aigonix Platform. It is the main interface for interacting with the platform and is where you will spend most of your time when interacting with the platform.

In Studio you can create and manage Applications, Environments, Microservices and other products and services.

Getting started

To access Studio you need to be a customer of Aigonix. If you are not a customer, you can contact us to learn more and hopefully become one.

Once you have access to Studio, you can log in using your credentials at dolittle.studio. You can now create your first Application and start deploying your Microservices into that application.

Components

When you run your Microservices in the Aigonix Platform you will have access to a number of components that will help you manage your services.

You will define each Application with its Environments, and add Microservices to them. You define which Docker image to use for each Microservice, and whether or not you want to use the Dolittle Runtime. You can use a publicly available image, or store your container image in the provided container-registry.

If you use the Dolittle Runtime one will be made available to your Head and through it you will have access to Tenanted resources like the Event Store and Read Cache. If you do not use the Dolittle Runtime the service will run the assigned Docker-image without permanent storage (stateless).

You can make your services available to the internet if you so wish. If you do not they will only be available within the platform.

flowchart TB
    subgraph Legend
        S[Docker Container]
        R>Dolittle Runtime]
        D[(Database)]
        F[\Studio function/]
    end
    subgraph Customer
        subgraph app1[Application with 3 Environments]
            subgraph app1dev[Development Environment]
                subgraph App1DevMS1[Microservice with Runtime]
                    App1DevMS1head[Head] --SDK--> App1DevMS1runtime>Runtime]

                    App1DevMS1runtime --uses--> App1DevMS1EventStore[(Event Store)]
                    App1DevMS1runtime --makes available--> App1DevMS1ReadCache[(Read Cache)]
                end
                subgraph App1DevMS2[Microservice]
                    App1DevMS2Head[Head]

                end
                App1DevMS1EventStore --in--> app1devDB[(Dev Database)]
                App1DevMS1ReadCache --in--> app1devDB
                Logs1dev[\Log viewer/] -.gathers logs.-> App1DevMS1head
                Logs1dev -.gathers logs.-> App1DevMS2Head
            end
            subgraph app1test[Test Environment]
                subgraph App1TestMS1[Microservice with Runtime]
                    App1TestMS1Head[Head] --uses--> App1TestMS1Runtime>Runtime]

                    App1TestMS1Runtime --uses--> App1TestMS1EventStore[(Event Store)]
                    App1TestMS1Runtime --makes available--> App1TestMS1ReadCache[(Read Cache)]
                end
                subgraph App1TestMS2[Microservice]
                    App1TestMS2Head[Head]
                end
                App1TestMS1EventStore --in--> app1testDB[(Test Database)]
                App1TestMS1ReadCache --in--> app1testDB

                Logs1test[\Log viewer/] -.gathers logs.-> App1TestMS1Head
                Logs1test -.gathers logs.-> App1TestMS2Head
            end
            subgraph app1prod[Production Environment]
                subgraph App1ProdMS1[Microservice with Runtime]
                    App1ProdMS1Head[Head] --uses-->  App1ProdMS1Runtime>Runtime]

                    App1ProdMS1Runtime --uses--> App1ProdMS1EventStore[(Event Store)]
                    App1ProdMS1Runtime --makes available--> App1ProdMS1ReadCache[(Read Cache)]
                end
                subgraph App1ProdMS2[Microservice]
                    App1ProdMS2Head[Head]
                end

                App1ProdMS1EventStore --in--> app1prodDB[(Prod Database)]
                App1ProdMS1ReadCache --in--> app1prodDB

                Logs1prod[\Log viewer/] -.gathers logs.-> App1ProdMS1Head
                Logs1prod[\Log viewer/]  -.gathers logs.-> App1ProdMS2Head
            end
            Docs1[\Documentation/]


            app1devDB -.regular backups.-> Backups1[\Backups/]
            app1testDB -.regular backups.-> Backups1
            app1prodDB -.regular backups.-> Backups1
        end
        subgraph app2[Application with 1 Environment]
            subgraph app2prod[Production Environment]
                subgraph App2ProdMS1[Microservice with Runtime]
                    App2ProdMS1Head[Head] --uses--> App2ProdMS1Runtime>Runtime]

                    App2ProdMS1Runtime --uses--> App2ProdMS1EventStore[(Event Store)]
                    App2ProdMS1Runtime --makes available--> App2ProdMS1ReadCache[(Read Cache)]
                end
                subgraph App2ProdMS2[Microservice]
                    App2ProdMS2Head[Head]
                end
                App2ProdMS1EventStore --in--> app2prodDB[(Prod Database)]
                App2ProdMS1ReadCache --in--> app2prodDB

                Logs2prod[\Log viewer/] -.gathers logs.-> App2ProdMS1Head
                Logs2prod -.gathers logs.-> App2ProdMS2Head
            end

            Docs2[\Documentation/]

            app2prodDB -.regular backups.-> Backups2[\Backups/]
        end
        ACR[\Container Registry/]
    end

3.6.2 - Application

An Application is a logical grouping of Microservices. It is the top-level container for your Microservices and is used to group them together. An Application can have multiple Environments, which are used to separate your Microservices into different stages of your development process.

When you create a new application you will define a name for it and which environments the application should have.

Microservices

Each application acts as a separate area wherein your microservices run. When you navigate in an application in Studio you will see the services that are deployed in that application.

Each microservice will be listed with their name, container-image, which version of the Dolittle Runtime (if any) they are set up with, their publich URL (if any) and their status. If your application has multiple environments you can switch between them using the dropdown in the top right corner.

You can navigate into each Microservice to see more details about it, such as the logs for the service, the configuration for the service and metrics like the CPU and Memory consumption of the service. You can change the configuration of the service and restart it from this view.

Functions

There are several functions available in an Application -view. You can create new Microservices, and you can access the backups of the databases that are used by the services in that application.

Each application has its own docker-image container registry, which you can access from the application view.

There is a log-viewer, which consolidates logs from all the services in the application into one view. Here you can filter the logs by service, time and search for specific text. You can also have a live-view of the logs, which will update the view as new logs are generated.

There is a documentation-view in the application that contains sections on how to access the container-registry, how to access the underlying Kubernetes cluster if needed and how to setup Azure Pipelines to deploy your containers directly to the application.

3.6.3 - Environment

An environment is a grouping of Microservices within an Application. Environments are commonly used to separate instances of your Microservices into different stages of your development process.

When you create an Application you define which environment(s) you want in that Application. The available options are Development, Test and Production. You can choose to have all three, or just one or two of them.

Microservices

Each environment acts as a separate area wherein your microservices run. When you navigate to an Application in Studio it will list the Microservices running in the current environment. The current Environment is displayed and can be changed in a drop-down box on the top right corner of the page.

Databases

Each environment has its own database. The database is used by the Microservices in that environment to store their data. The database is a MongoDB database and is managed by the Aigonix Platform. You can access the database backups through the Studio interface. If necessary you can also access the database through port-forwarding to the Kubernetes cluster. You will need to contact Aigonix Support to get the elevated permissions needed to access the database directly.

3.6.4 - Microservice

A Microservice is a single unit of deployment in the Aigonix Platform. It is a containerized application that runs in a Kubernetes cluster. Each Microservice is deployed as a separate container in the cluster.

A Microservice lives inside an Application and can be deployed to multiple Environments in that application.

A Microservice in the Aigonix Platform is a Head and optionally a Runtime and data-storage. If you use the Dolittle Runtime you will have access to the Event Store and Read Cache for that Microservice.

If you do not use the Dolittle Runtime the Microservice will be stateless and will not have access to any storage. Any state you need to store will have to be stored in an external service. You will have access to the local file-system, but that is not persisted between Microservice restarts, and we do not recommend relying upon it.

Docker image

Each Microservice is deployed using a Docker image. You can use a publicly available image, or you can use a Docker image that you have built yourself and stored in the container-registry that is available for each application.

Head

Head is the name of the pod where your Microservice Docker image is deployed. It is the entry-point for your Microservice and is the only part of your Microservice that may be exposed to the internet. A Microservice must have a Head and can have a Runtime.

Dolittle Runtime

Each Microservice can be deployed with the Dolittle Runtime. The Dolittle Runtime is available as a docker image, and will be deployed alongside your Head. Your code communicates with the Runtime through the Dolittle SDK. The Runtime is used to connect to the Event Store and will make the resource-system available to you, where you can get the Tenanted Read Cache and to publish events to the Event Horizon.

Configuration

Each Microservice can be configured with a set of environment variables and configuration-files. These variables are available to your code and can be used to configure your Microservice. The Aigonix Studio gives you an interface to manage these variables and files.

Your code can read the environment variables and configuration-files. The files will be available under the /app/data -folder in your container. The Aigonix Platform will make sure that the files are available in the container when it is deployed. Therefore any files your image contains in /app/data will not be available when running in the Aigonix Platform.

You can upload files through Aigonix Studio, and you can also download the files and environment variables from Studio.

Secrets

You can also use the environment variables to store secrets, like connection strings and API-keys. Mark these environment-variables as secret to ensure that they be stored encrypted in the platform. You may want to look into a proper secret-management system, like Azure Key Vault or Amazon Key Management System if you have a lot of secrets, and just store the access-keys in the environment variables.

3.6.5 - Integrations

Integrations in Aigonix lets you work with your existing business-data in an ERP-system. The integrations support an Event Driven Architecture and is designed to make translation from ERP-specific terms to domain terms easy.

By setting up integrations your business data from an ERP -system (Infor M3) synchronize to an message stream. This data can then be used by your Microservices to provide functionality to your users and customers.

This article will introduce you to the concepts of the Aigonix integrations and show you how to set up a connection to an ERP -system and how to map data from the ERP -system into messages. It also describes how to consume these messages in your services.

Conceptual overview

graph TB
    People(People) --set up connection--> Connection[Aigonix integrations]
    People --use--> Services
    Connection <--changes--> ERP[ERP-system]
    Connection <--messages--> KT[/Messages/]
    Services <--reacts to--> KT

Central to this is the concept of a message-type. You define how to translate data from the ERP -system into messages. Message-types are defined in the Aigonix Studio -interface where you can select the tables and fields you want to translate. This mapping then translates the changes in business-data into messages whenever such a change is detected.

Connections

In order to translate data from the ERP -system into messages a connection must be specified. The connection defines how to connect to the ERP -system and the message-types how to translate the data into messages. Once you define the connection you can either run an instance of the ERP Integration service yourself (on-premises), or automatically host one in the Aigonix platform. This service is the conduit between the messages and the ERP -system. It discovers changes and translates those into messages. It also reacts to messages and calls programs in the ERP-system at your request.

Message-types

The message-types are defined in the Aigonix Studio -interface, and deployed to the ERP Integration service. They are defined in terms of the data in the ERP -system.

Setting up a connection

Step 0: Name and deploy the connection

Step 1: Connect to the Metadata Publisher

Step 2: Connect to ION