Event Horizon

Get started with the Event Horizon

Welcome to the tutorial for Event Horizon, where you learn how to write a Microservice that produces public events of dishes prepared by chefs, and another microservice consumes those events.

After this tutorial you will have:

a running Dolittle environment with two Runtimes and a MongoDB,
a Producer Microservice that commits and handles a Public Event and filters it into a Public Stream and
a Consumer Microservice that Subscribes to the consumers public stream over the Event Horizon and processes those public events

Use the tabs to switch between the C# and TypeScript code examples. Full tutorial code available on GitHub for C# and TypeScript.

Prerequisites

This tutorial builds directly upon the getting started guide and the files from the it.

C#
TypeScript

Prerequisites:

Setup

This tutorial will have a setup with two microservices; one that produces public events, and a consumer that subscribes to those public events. Let’s make a folder structure that resembles that:

└── event-horizon-tutorial/
    ├── consumer/
    ├── producer/
    └── environment/
        └── docker-compose.yml

Go into both the consumer and the producer folders and initiate the project as we’ve gone through in our getting started guide. I.e copy over all the code from the getting started tutorial to the consumer and producer folders. You can choose different languages for the microservices if you want to.

We’ll come back to the docker-compose later in this tutorial.

Producer

Create a `Public Filter`

A public filter filters all public events that pass the filter into a public stream, which is special stream that another microservice can subscribe to.

A public filter is defined as a method that returns a partitioned filter result, which is an object with two properties:

a boolean that says whether the event should be included in the public stream
a partition id which is the partition that the event should belong to in the public stream.

Only public events get filtered through the public filters.

C#
TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Events.Filters;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithFilters(_ => _
            .CreatePublic("2c087657-b318-40b1-ae92-a400de44e507")
            .Handle((@event, eventContext) =>
            {
                Console.WriteLine($"Filtering event {@event} to public streams");
                return Task.FromResult(new PartitionedFilterResult(true, eventContext.EventSourceId.Value));
            })))
    .Build();

await host.StartAsync();
await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { EventContext } from '@dolittle/sdk.events';
import { PartitionedFilterResult } from '@dolittle/sdk.events.filtering';
import { TenantId } from '@dolittle/sdk.execution';

import './DishHandler';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withFilters(_ => _
                .createPublic('2c087657-b318-40b1-ae92-a400de44e507')
                    .handle((event: any, context: EventContext) => {
                        client.logger.info(`Filtering event ${JSON.stringify(event)} to public stream`);
                        return new PartitionedFilterResult(true, 'Dolittle Tacos');
                    })
            )
        )
        .connect();
})();

Notice that the returned PartitionedFilterResult has true and an unspecified PartitionId (which is the same as an empty GUID). This means that this filter creates a public stream that includes all public events, and that they are put into the unspecified partition of that stream.

Commit the public event

Now that we have a public stream we can commit public events to start filtering them. Let’s commit a DishPrepared event as a public event from the producer microservice:

C#
TypeScript

// Program.cs
using System;
using System.Threading.Tasks;
using Dolittle.SDK;
using Dolittle.SDK.Events.Filters;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

var host = Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithFilters(_ => _
            .CreatePublic("2c087657-b318-40b1-ae92-a400de44e507")
            .Handle((@event, eventContext) =>
            {
                Console.WriteLine($"Filtering event {@event} to public streams");
                return Task.FromResult(new PartitionedFilterResult(true, eventContext.EventSourceId.Value));
            })))
    .Build();

await host.StartAsync();

var client = await host.GetDolittleClient();
var preparedTaco = new DishPrepared("Bean Blaster Taco", "Mr. Taco");
await client.EventStore
    .ForTenant(TenantId.Development)
    .CommitPublicEvent(preparedTaco, "Dolittle Tacos");

await host.WaitForShutdownAsync();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { EventContext } from '@dolittle/sdk.events';
import { PartitionedFilterResult } from '@dolittle/sdk.events.filtering';
import { TenantId } from '@dolittle/sdk.execution';

import './DishHandler';
import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withFilters(_ => _
                .createPublic('2c087657-b318-40b1-ae92-a400de44e507')
                    .handle((event: any, context: EventContext) => {
                        client.logger.info(`Filtering event ${JSON.stringify(event)} to public stream`);
                        return new PartitionedFilterResult(true, 'Dolittle Tacos');
                    })
            )
        )
        .connect();

    const preparedTaco = new DishPrepared('Bean Blaster Taco', 'Mr. Taco');

    await client.eventStore
        .forTenant(TenantId.development)
        .commitPublic(preparedTaco, 'Dolittle Tacos');
})();

Now we have a producer microservice with a public stream of DishPrepared events.

Consumer

Let’s create another microservice that subscribes to the producer’s public stream.

C#
TypeScript

// Program.cs
using System;
using Dolittle.SDK;
using Dolittle.SDK.Tenancy;
using Microsoft.Extensions.Hosting;

Host.CreateDefaultBuilder()
    .UseDolittle(_ => _
        .WithEventHorizons(_ => _
            .ForTenant(TenantId.Development, subscriptions => 
                subscriptions
                    .FromProducerMicroservice("f39b1f61-d360-4675-b859-53c05c87c0e6")
                    .FromProducerTenant(TenantId.Development)
                    .FromProducerStream("2c087657-b318-40b1-ae92-a400de44e507")
                    .FromProducerPartition("Dolittle Tacos")
                    .ToScope("808ddde4-c937-4f5c-9dc2-140580f6919e"))
        )
        .WithEventHandlers(_ => _
            .Create("6c3d358f-3ecc-4c92-a91e-5fc34cacf27e")
            .InScope("808ddde4-c937-4f5c-9dc2-140580f6919e")
            .Partitioned()
            .Handle<DishPrepared>((@event, context) => Console.WriteLine($"Handled event {@event} from public stream"))
        ),
        configuration => configuration.WithRuntimeOn("localhost", 50055))
    .Build()
    .Run();

// index.ts
import { DolittleClient } from '@dolittle/sdk';
import { TenantId } from '@dolittle/sdk.execution';

import { DishPrepared } from './DishPrepared';

(async () => {
    const client = await DolittleClient
        .setup(_ => _
            .withEventHorizons(_ => {
                _.forTenant(TenantId.development, _ => _
                    .fromProducerMicroservice('f39b1f61-d360-4675-b859-53c05c87c0e6')
                        .fromProducerTenant(TenantId.development)
                        .fromProducerStream('2c087657-b318-40b1-ae92-a400de44e507')
                        .fromProducerPartition('Dolittle Tacos')
                        .toScope('808ddde4-c937-4f5c-9dc2-140580f6919e'));
            })
            .withEventHandlers(_ => _
                .create('6c3d358f-3ecc-4c92-a91e-5fc34cacf27e')
                    .inScope('808ddde4-c937-4f5c-9dc2-140580f6919e')
                    .partitioned()
                    .handle(DishPrepared, (event, context) => {
                        client.logger.info(`Handled event ${JSON.stringify(event)} from public stream`);
                     })
            )
        )
        .connect(_ => _
            .withRuntimeOn('localhost', 50055)
        );
})();

Now we have a consumer microservice that:

Connects to another Runtime running on port 50055
Subscribes to the producer’s public stream with the id of 2c087657-b318-40b1-ae92-a400de44e507 (same as the producer’s public filter)
Puts those events into a Scope with id of 808ddde4-c937-4f5c-9dc2-140580f6919e
Handles them incoming events in a scoped event handler with an id of 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e

There’s a lot of stuff going on the code so let’s break it down:

configuration => configuration.WithRuntimeOn("localhost", 50055))

.withRuntimeOn('localhost', 50055)

This line configures the hostname and port of the Runtime for this client. By default, it connects to the Runtimes default port of 50053 on localhost.

Since we in this tutorial will end up with two running instances of the Runtime, they will have to run with different ports. The producer Runtime will be running on the default 50053 port, and the consumer Runtime will be running on port 50055. We’ll see this reflected in the docker-compose.yml file later in this tutorial.

Event Horizon

C#
TypeScript

// Program.cs
.WithEventHorizons(_ => _
    .ForTenant(TenantId.Development, subscriptions => 
        subscriptions
            .FromProducerMicroservice("f39b1f61-d360-4675-b859-53c05c87c0e6")
            .FromProducerTenant(TenantId.Development)
            .FromProducerStream("2c087657-b318-40b1-ae92-a400de44e507")
            .FromProducerPartition("Dolittle Tacos")
            .ToScope("808ddde4-c937-4f5c-9dc2-140580f6919e"))
)

.withEventHorizons(_ => {
    _.forTenant(TenantId.development, _ => _
        .fromProducerMicroservice('f39b1f61-d360-4675-b859-53c05c87c0e6')
            .fromProducerTenant(TenantId.development)
            .fromProducerStream('2c087657-b318-40b1-ae92-a400de44e507')
            .fromProducerPartition('Dolittle Tacos')
            .toScope('808ddde4-c937-4f5c-9dc2-140580f6919e'));
})

Here we define an event horizon subscription. Each subscription is submitted and managed by the Runtime. A subscription defines:

The consumers Tenant
The producer microservice, Public Stream and that streams Partition to get the events from
The Scoped event-log of the consumer to put the subscribed events to

When the consumer’s Runtime receives a subscription, it will send a subscription request to the producer’s Runtime. If the producer accepts that request, the producer’s Runtime will start sending the public stream over to the consumer’s Runtime, one event at a time.

The acceptance depends on two things:

The consumer needs to know where to access the other microservices, ie the URL address.
The producer needs to give formal Consent for a tenant in another microservice to subscribe to public streams of a tenant.

We’ll setup the consent later.

The consumer will receive events from the producer and put those events in a specialized event-log that is identified by the scope’s id, so that events received over the event horizon don’t mix with private events. We’ll talk more about the scope when we talk about the scoped event handler.

Scoped Event Handler

C#
TypeScript

// Program.cs
.WithEventHandlers(_ => _
    .Create("6c3d358f-3ecc-4c92-a91e-5fc34cacf27e")
    .InScope("808ddde4-c937-4f5c-9dc2-140580f6919e")
    .Partitioned()
    .Handle<DishPrepared>((@event, context) => Console.WriteLine($"Handled event {@event} from public stream"))
)

.withEventHandlers(_ => _
    .create('6c3d358f-3ecc-4c92-a91e-5fc34cacf27e')
        .inScope('808ddde4-c937-4f5c-9dc2-140580f6919e')
        .partitioned()
        .handle(DishPrepared, (event, context) => {
            client.logger.info(`Handled event ${JSON.stringify(event)} from public stream`);
        })
)

Here we use the opportunity to create an event handler inline by using the client’s builder function. This way we don’t need to create a class and register it as an event handler.

This code will create a partitioned event handler with id 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e (same as from getting started) in a specific scope.

Remember, that the events from an event horizon subscription get put into a scoped event-log that is identified by the scope id. Having the scope id defined when creating an event handler signifies that it will only handle events in that scope and no other.

Setup your environment

Now we have the producer and consumer microservices Heads coded, we need to setup the environment for them to run in and configure their Runtimes to be connected. This configuration is provided by Dolittle when you’re running your microservices in our platform, but when running multiple services on your local machine you need to configure some of it yourself.

Let’s go to the environment folder we created in the beginning of this tutorial. Here we’ll need to configure:

Platform

platform.json configures the environment of a microservice. We have 2 microservices so they need to be configured with different identifiers and names.

Let’s create 2 files, consumer-platform.json and producer-producer.json:

//consumer-platform.json
{
    "applicationName": "EventHorizon Tutorial",
    "applicationID": "5bd8762f-6c39-4ba2-a141-d041c8668894",
    "microserviceName": "Consumer",
    "microserviceID": "a14bb24e-51f3-4d83-9eba-44c4cffe6bb9",
    "customerName": "Dolittle Tacos",
    "customerID": "c2d49e3e-9bd4-4e54-9e13-3ea4e04d8230",
    "environment": "Tutorial"
}

//producer-platform.json
{
    "applicationName": "EventHorizon Tutorial",
    "applicationID": "5bd8762f-6c39-4ba2-a141-d041c8668894",
    "microserviceName": "Producer",
    "microserviceID": "f39b1f61-d360-4675-b859-53c05c87c0e6",
    "customerName": "Dolittle Tacos",
    "customerID": "c2d49e3e-9bd4-4e54-9e13-3ea4e04d8230",
    "environment": "Tutorial"
}

Resources

resources.json configures where a microservices stores its event store. We have 2 microservices so they both need their own event store database. By default the database is called event_store.

Create 2 more files, consumer-resources.json and producer-resources.json:

//consumer-resources.json
{
    // the tenant to define this resource for
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": {
        "eventStore": {
            "servers": [
                // hostname of the mongodb
                "mongo"
            ],
            // the database name for the event store
            "database": "consumer_event_store"
        }
    }
}

//producer-resources.json
{
    // the tenant to define this resource for
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": {
        "eventStore": {
            "servers": [
                // hostname of the mongodb
                "mongo"
            ],
            // the database name for the event store
            "database": "producer_event_store"
        }
    }
}

Development Tenant

The tenant id 445f8ea8-1a6f-40d7-b2fc-796dba92dc44 is the value of TenantId.Development.

Microservices

microservices.json configures where the producer microservice is so that the consumer can connect to it and subscribe to its events.

Let’s create a consumer-microservices.json file to define where the consumer can find the producer:

// consumer-microservices.json
{
    // the producer microservices id, hostname and port
    "f39b1f61-d360-4675-b859-53c05c87c0e6": {
        "host": "producer-runtime",
        "port": 50052
    }
}

event-horizon-consents.json configures the Consents that the producer gives to consumers.

Let’s create producer-event-horizon-consents.json where we give a consumer consent to subscribe to our public stream.

// producer-event-horizon-consents.json
{
    // the producer's tenant that gives the consent
    "445f8ea8-1a6f-40d7-b2fc-796dba92dc44": [
        {
            // the consumer's microservice and tenant to give consent to
            "microservice": "a14bb24e-51f3-4d83-9eba-44c4cffe6bb9",
            "tenant": "445f8ea8-1a6f-40d7-b2fc-796dba92dc44",
            // the producer's public stream and partition to give consent to subscribe to
            "stream": "2c087657-b318-40b1-ae92-a400de44e507",
            "partition": "Dolittle Tacos",
            // an identifier for this consent. This is random
            "consent": "ad57aa2b-e641-4251-b800-dd171e175d1f"
        }
    ]
}

Configure `docker-compose.yml`

Now we can glue all the configuration files together in the docker-compose.yml. The configuration files are mounted inside /app.dolittle/ inside the dolittle/runtime image.

version: '3.8'
services:
  mongo:
    image: dolittle/mongodb
    hostname: mongo
    ports:
      - 27017:27017
    logging:
      driver: none
 
  consumer-runtime:
    image: dolittle/runtime:latest
    volumes:
      - ./consumer-platform.json:/app/.dolittle/platform.json
      - ./consumer-resources.json:/app/.dolittle/resources.json
      - ./consumer-microservices.json:/app/.dolittle/microservices.json
    ports:
      - 50054:50052
      - 50055:50053

  producer-runtime:
    image: dolittle/runtime:latest
    volumes:
      - ./producer-platform.json:/app/.dolittle/platform.json
      - ./producer-resources.json:/app/.dolittle/resources.json
      - ./producer-event-horizon-consents.json:/app/.dolittle/event-horizon-consents.json
    ports:
      - 50052:50052
      - 50053:50053

Resource file naming

The configuration files mounted inside the image need to be named as they are defined in the configuration reference. Otherwise the Runtime can’t find them.

Start the environment

Start the docker-compose with this command

$ docker-compose up -d

This will spin up a MongoDB container and two Runtimes in the background.

Docker on Windows

Docker on Windows using the WSL2 backend can use massive amounts of RAM if not limited. Configuring a limit in the .wslconfig file can help greatly, as mentioned in this issue. The RAM usage is also lowered if you disable the WSL2 backend in Docker for Desktop settings.

Run your microservices

Run both the consumer and producer microservices in their respective folders, and see the consumer handle the events from the producer:

C#
TypeScript

Producer

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jakob/Git/dolittle/DotNET.SDK/Samples/Tutorials/EventHorizon/Producer
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      Public Filter 2c087657-b318-40b1-ae92-a400de44e507 registered with the Runtime, start handling requests
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests
Filtering event DishPrepared to public streams
info: DishHandler[0]
      Mr. Taco has prepared Bean Blaster Taco. Yummm!

Consumer

$ dotnet run
info: Dolittle.SDK.DolittleClientService[0]
      Connecting Dolittle Client
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/jakob/Git/dolittle/DotNET.SDK/Samples/Tutorials/EventHorizon/Consumer
info: Dolittle.SDK.Events.Processing.EventProcessors[0]
      EventHandler 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e registered with the Runtime, start handling requests
Handled event DishPrepared from public stream

Producer

$ npx ts-node index.ts
info: EventHandler f2d366cf-c00a-4479-acc4-851e04b6fbba registered with the Runtime, start handling requests.
info: Public Filter 2c087657-b318-40b1-ae92-a400de44e507 registered with the Runtime, start handling requests.
info: Filtering event {"Dish":"Bean Blaster Taco","Chef":"Mr. Taco"} to public stream
info: Mr. Taco has prepared Bean Blaster Taco. Yummm!

Consumer

$ npx ts-node index.ts
info: EventHandler 6c3d358f-3ecc-4c92-a91e-5fc34cacf27e registered with the Runtime, start handling requests.
info: Handled event {"Dish":"Bean Blaster Taco","Chef":"Mr. Taco"} from public stream

What’s next

Learn how to deploy your application into our Platform
Learn more about the Event Horizon

Last modified January 20, 2022: Fix tutorial samples (e40d3f3)

Event Horizon

Prerequisites

Setup

Producer

Create a Public Filter

Commit the public event

Consumer

Subscribe to the public stream of events

Connection to the Runtime

Event Horizon

Scoped Event Handler

Setup your environment

Platform

Resources

Development Tenant

Microservices

Consent

Configure docker-compose.yml

Resource file naming

Start the environment

Docker on Windows

Run your microservices

Producer

Consumer

Producer

Consumer

What’s next

Create a `Public Filter`

Configure `docker-compose.yml`