Event Store

Introduction to the Event Store

An Event Store is a database optimized for storing Events in an Event Sourced system. The Runtime manages the connections and structure of the stored data. All Streams, Event Handlers & Filters, Aggregates and Event Horizon Subscriptions are being kept track inside the event store.

Events saved to the event store cannot be changed or deleted. It acts as the record of all events that have happened in the system from the beginning of time.

Each Tenant has their own event store database, which is configured in resources.json.

Scope

Events that came over the Event Horizon need to be put into a scoped collection so they won’t be mixed with the other events from the system.

Scoped collections work the same way as other collections, except you can’t have Public Streams or Aggregates.

Structure of the Event Store

This is the structure of the event store implemented in MongoDB. It includes the following collections in the default Scope:

  • event-log
  • aggregates
  • stream-processor-states
  • stream-definitions
  • stream-<streamID>
  • public-stream-<streamID>

For scoped collections:

Following JSON structure examples have each property’s BSON type as the value.

event-log

The Event Log includes all the Events committed to the event store in chronological order. All streams are derived from the event log.

Aggregate events have "wasAppliedByAggregate": true set and events coming over the Event Horizon have "FromEventHorizon": true" set.

This is the structure of a committed event:

{
    // this it the events EventLogSequenceNumber,
    // which identifies the event uniquely within the event log
    "_id": "decimal",
    "Content": "object",
    // Aggregate metadata
    "Aggregate": {
        "wasAppliedByAggregate": "bool",
        // AggregateRootId
        "TypeId": "UUID",
        // AggregateRoot Version
        "TypeGeneration": "long",
        "Version": "decimal"
    },
    // EventHorizon metadata
    "EventHorizon": {
        "FromEventHorizon": "bool",
        "ExternalEventLogSequenceNumber": "decimal",
        "Received": "date",
        "Concent": "UUID"
    },
    // the committing microservices metadata
    "ExecutionContext": {
        // 
        "Correlation": "UUID",
        "Microservice": "UUID",
        "Tenant": "UUID",
        "Version": "object",
        "Environment": "string",
    },
    // the events metadata
    "Metadata": {
        "Occurred": "date",
        "EventSource": "UUID",
        // EventTypeId and Generation
        "TypeId": "UUID",
        "TypeGeneration": "long",
        "Public": "bool"
    }
}

aggregates

This collection keeps track of all instances of Aggregates registered with the Runtime.

{
    "EventSource": "UUID",
    // the AggregateRootId
    "AggregateType": "UUID",
    "Version": "decimal"
}

stream

A Stream contains all the events filtered into it. It’s structure is the same as the event-log, with the extra Partition property used for partitions

The streams StreamId is added to the collections name, eg. a stream with the id of 323bcdb2-5bbd-4f13-a7c3-b19bc2cc2452 would be in a collection called stream-323bcdb2-5bbd-4f13-a7c3-b19bc2cc2452.

{
    // same as an Event in the "event-log" + Partition
    "Partition": "UUID",
}

public-stream

The same as a stream, except only for Public Stream with the public prefix in collection name. Public streams can only exist on the default scope.

stream-definitions

This collection contains all Filters registered with the Runtime.

Filters defined by an Event Handler have a type of EventTypeId, while other filters have a type of Remote.

{
    // id of the Stream the Filter creates
    "_id": "UUID",
    "Partitioned": "bool",
    "Public": "bool",
    "Filter": {
        "Type": "string",
        "Types": [
            // EventTypeIds to filter into the stream
        ]
    }
}

stream-processor-states

This collection keeps track of all Stream Processors and their state. Partitioned streams will have a FailingPartitions property for tracking the fail information per partition.

{
    "SourceStream": "UUID",
    "EventProcessor": "UUID",
    "Position": "decimal",
    "LastSuccesfullyProcessed": "date",
    // failure tracking information
    "RetryTime": "date",
    "FailureReason": "string",
    "ProcessingAttempts": "int",
    "IsFailing": "bool
}

subscription-states

This collection keeps track of Event Horizon Subscriptions in a very similar way to stream-processor-states.

{
    // producers microservice, tenant and stream info
    "Microservice": "UUID",
    "Tenant": "UUID",
    "Stream": "UUID",
    "Partition": "UUID",
    "Position": "decimal",
    "LastSuccesfullyProcessed": "date",
    "RetryTime": "date",
    "FailureReason": "string",
    "ProcessingAttempts": "int",
    "IsFailing": "bool
}

Commit vs Publish

We use the word Commit rather than Publish when talking about saving events to the event store. We want to emphasize that it’s the event store that is the source of truth in the system. The act of calling filters/event handlers comes after the event has been committed to the event store. We also don’t publish to any specific stream, event handler or microservice. After the event has been committed, it’s ready to be picked up by any processor that listens to that type of event.

Last modified January 1, 0001