# How the engine is built

> The whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/architecture/

This section is the engine's design, written before its code. The code implements exactly what these pages say. Read it if you run the engine yourself, if you want to contribute, or if you want to know why a number on your screen can be trusted.

Everything here is the open-source engine **\[Engine]**. Sypnia Cloud runs this same engine and adds a private part around it: accounts, billing, the app, the GitHub App and a managed model for the agent. Where a page touches one of those, it carries a **\[Cloud]** badge.

> **In plain words**
>
> Your app sends small notes about what happens. The engine saves them as files, which is cheap. A fast reader looks at the new files for failures and opens a problem in under 2 minutes. A slower reader counts everything else every 5 minutes. Only the results go into the database. The agent reads a problem and writes one sentence about it.

The design is not new. It is the observability system that has run in production inside the first app that uses Sypnia since 2026-09-30, made multi-project, rewritten in Rust and made faster: minutes where the original waits for an hourly job. Each page says what was kept, what changed and why.

## The whole system

**Figure**: The engine on one page. The lit path is the fast path: a failure becomes a problem in under 2 minutes. Everything else is counted by the roll-up within 10.

1. Your app sends batches of events, every 10 seconds.
2. Ingest writes them to the bucket as segment objects (NDJSON, gzip), and one ledger row per segment to Postgres.
3. The fast path reads the new segments every 30 seconds.
4. It writes problems to Postgres.
5. The roll-up reads the open hour every 5 minutes. It writes numbers and the session index to Postgres, and per-session files and Parquet back to the bucket.
6. The agent reads a problem that is new, growing or came back, and writes the sentence.
7. Alerts are recorded, and sent to your webhook as a signed POST.
8. The API reads Postgres, and files from the bucket. The MCP server and the Console read through the API.
9. Your coding agent asks the MCP server. You read the Console or the API.

There are two stores and nothing else:

| Store                                | Holds                                                                                   | Why there                                                              |
| ------------------------------------ | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Bucket** (any S3-compatible store) | Raw events, per-session files, replay chunks, page snapshots, source maps, Parquet      | Written once, read rarely, large. A file costs almost nothing to keep. |
| **Postgres** (15 or later)           | Projects and keys, problems, hourly and daily numbers, the session index, alerts, usage | Read on every screen and every MCP call, small, updated in place.      |

And five moving parts, all inside one binary:

| Part          | Job                                                                 | Runs                                   |
| ------------- | ------------------------------------------------------------------- | -------------------------------------- |
| **Ingest**    | Checks the key and each event, writes segment objects to the Bucket | On every request from an SDK           |
| **Fast path** | Reads new segments, finds failures, creates and updates problems    | Every 30 seconds                       |
| **Roll-up**   | Counts everything else, assembles sessions, writes Parquet          | Every 5 minutes; Parquet once a day    |
| **The agent** | Reads a problem, writes the sentence, the cause and the prompt      | When a problem is new, growing or back |
| **Alerts**    | Records an alert, calls your webhook                                | When a rule fires                      |

The **API**, the **MCP server** and the **Console** only read what those parts wrote. They never compute.

## A request's life in 9 steps

The example is the one used across these docs: a shop whose release `4f2a91c` renamed `cart.total` to `cart.totalCents`. The checkout still reads the old name, so payments go out without an amount.

**Figure**: The nine steps on a clock. The promise is 2 minutes from the first failure to the problem and its alert.

1. **A customer clicks Pay.** At 09:12:04 the page calls `POST /api/checkout` and passes a trace id. The server calls the payment service, which answers `400`: the amount is missing. `@sypnia/node` records that failed call inside the trace. The page records the click, the failed request and the error it throws.
2. **The SDK batches.** The browser SDK keeps events in memory and sends them every 10 seconds, at 50 events, or when the page is hidden. One batch is one small JSON body with a `batch_id`.
3. **Ingest checks and accepts.** **\[Engine]** It finds the project from the ingest key in a cache, checks the origin, the size and each event, stamps `prj` and `rcv`, and appends the lines to an in-memory buffer. It does not query Postgres to do this.
4. **The Bucket gets a segment.** Within 2 seconds the buffer becomes one object, and only then does ingest answer the SDK. The events are now safe, whatever happens next.

   ```text
   prj_5k2m9x4q7w1z/events/production/dt=2026-03-12/hr=09/20260312T091216Z-7f3a9c.ndjson.gz
   ```
5. **The Fast path finds a new failure.** Every 30 seconds it takes the segments nobody has scanned yet, reads them and keeps only the failures. It finds a fingerprint it has never seen in this project and environment: `88a44b15205cb153`.
6. **A problem is born.** One transaction writes the problem, its first occurrence, the person and the session it hit, and marks the segment as scanned. In the example this is 09:13:02, 58 seconds after the click. The budget is 120.
7. **The agent writes the sentence.** If you configured a model, the agent builds a context (the failure, the stack with source lines, the trace, the release, the counts), calls the model and checks the answer. The technical title «createPayment answered 400 in POST /api/checkout» becomes «1 person tried to pay and could not. The Pay button does nothing since yesterday's change.»
8. **Alerts.** The transaction of step 6 also wrote an alert, held until the sentence is ready or until 100 seconds have passed since the failure arrived, whichever comes first. Then it is released: recorded, and sent to your webhook. **\[Cloud]** Sypnia Cloud also sends it by e-mail and shows it in the app.
9. **Your coding agent reads it.** The developer asks their agent what is failing. It calls the MCP tool `problem` with `pay-button-does-nothing` and gets the sentence, the cause with its evidence, the stack in the original source and the prompt. **\[Cloud]** On Sypnia Cloud the fix can also arrive as a pull request.

Meanwhile, every 5 minutes, the Roll-up counts the rest: the page view, the click, the rage click that follows, how long `POST /api/checkout` took. Those numbers are complete within 10 minutes.

## The design rules

Five rules explain almost every decision in this section.

**Figure**: Rule 1 and rule 5 in one picture: raw events stay files, only results become rows, and every one of them carries its project.

### 1. Raw events are files, not rows

An event is written once and almost never read again. A row in a database pays for indexes, vacuum and memory for ever; a gzip line in a bucket pays a fraction of a cent a year. The system this engine comes from learned it the hard way: its shared database ran out of memory once, and raw events were the one thing that was never in it.

Files also give **schema on read**. An event keeps every field it arrived with, including fields this version of the engine does not know. A newer SDK never breaks an older engine.

The type that reads a line carries that rule. Notice `extra`: it catches whatever the structs above it did not name, and it is written back unchanged.

> **Note: About the code on these pages**
>
> The Rust and SQL in this section are design excerpts: they show the shape the code will have and the names it will use. The repository opens at launch. If the code and a page disagree, the page is right and the code has a bug.

`crates/sypnia-core/src/event.rs`

```rust
/// One line of a segment. The tag is the wire field `type`.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Event {
    SessionStart(Browser<SessionStart>),
    Identify(Browser<Identify>),
    PageView(Browser<PageView>),
    PageLeave(Browser<PageLeave>),
    Click(Browser<Click>),
    RageClick(Browser<RepeatedClick>),
    DeadClick(Browser<RepeatedClick>),
    Vital(Browser<Vital>),
    Request(Browser<Request>),
    Error(Browser<ErrorEvent>),
    Custom(Browser<Custom>),
    FormInvalid(Browser<FormInvalid>),
    ServerRequest(Server<ServerRequest>),
    ServerError(Server<ServerError>),
    /// A `type` this version does not know. Stored and compacted like any
    /// other line, skipped by the parts that cannot read it. Never rejected.
    #[serde(untagged)]
    Unknown(serde_json::Map<String, serde_json::Value>),
}

/// The fields every browser event carries, around the fields of its type.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Browser<T> {
    #[serde(flatten)]
    pub base: Base,              // v, ts, env, ver, person, account, prj, rcv
    pub sid: SessionId,
    pub did: Option<DeviceId>,
    pub url: Option<String>,
    pub route: Option<Route>,
    pub dev: Option<Device>,
    #[serde(flatten)]
    pub body: T,
    /// Schema on read: every field not named above, exactly as it arrived.
    #[serde(flatten)]
    pub extra: serde_json::Map<String, serde_json::Value>,
}
```

### 2. Postgres holds only what is read often

Problems, numbers per hour, the list of sessions, alerts. A million events become a few thousand rows. That is why a small Postgres is enough, and why the engine needs no query engine beside it: nothing on your screen is computed from raw events at the moment you ask.

### 3. Every step can run twice safely

Functions get retried, processes get killed, networks drop answers after the work was done. So every write in the engine is one of three kinds:

- **an overwrite with the same key**: a segment, a replay chunk, a per-session file;
- **a replace**: the numbers of one hour are deleted and written again, whole;
- **an add guarded by a ledger**: a counter is increased in the same transaction that records which segment it came from, so the same segment can never be added twice.

This is the upsert behind a problem. The counters add, the dates widen, the title stays as first seen, and a resolved problem reopens by itself.

`crates/sypnia-db/sql/problem_upsert.sql`

```sql
INSERT INTO problems AS p
       (id, project_id, environment, fingerprint, kind, origin, title, route,
        first_seen, last_seen, first_release, last_release, occurrences, sample)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $11, $12, $13)
ON CONFLICT (project_id, environment, fingerprint) DO UPDATE SET
    occurrences  = p.occurrences + EXCLUDED.occurrences,
    first_seen   = LEAST(p.first_seen, EXCLUDED.first_seen),
    last_seen    = GREATEST(p.last_seen, EXCLUDED.last_seen),
    last_release = COALESCE(EXCLUDED.last_release, p.last_release),
    sample       = COALESCE(p.sample, EXCLUDED.sample),
    -- a resolved problem that happens again comes back; an ignored one stays ignored
    regressed    = p.regressed OR p.status = 'resolved',
    status       = CASE WHEN p.status = 'resolved' THEN 'open' ELSE p.status END
RETURNING p.id, p.status, p.regressed, (xmax = 0) AS is_new;
```

### 4. One binary

Ingest, the Fast path, the Roll-up, the agent, the API, the MCP server and the Console are one program, `sypnia`. On a server it runs as one long process. On AWS the same file runs as three functions. Both build the same `Engine` and differ only in what wakes it up.

Notice that the engine knows four things only through traits. Each is a piece you can swap: the store, the database, the model, the way an alert leaves.

`crates/sypnia/src/engine.rs`

```rust
/// Everything the engine needs from the outside world.
/// `serve` and the AWS functions build exactly one of these.
pub struct Engine {
    pub config: Arc<Config>,
    pub store: Arc<dyn ObjectStore>,        // S3, R2, MinIO, a folder in tests
    pub db: Arc<dyn Database>,              // Postgres
    pub ai: Option<Arc<dyn AiProvider>>,    // None until you set a model key
    pub sinks: Vec<Arc<dyn AlertSink>>,     // the record, then your webhooks
    pub clock: Arc<dyn Clock>,              // real time, or a recorded hour in tests
}

impl Engine {
    /// One pass of the worker: what `sypnia aggregate` runs once
    /// and `sypnia serve` runs every `processing.interval_seconds`.
    pub async fn pass(&self) -> Result<PassReport, EngineError> {
        let mut report = PassReport::default();
        report.fast_path = fast_path::run(self).await?;   // problems, under 2 minutes
        report.alerts = alerts::release_due(self).await?;
        report.uptime = uptime::run_due(self).await?;
        if rollup::is_due(self).await? {
            report.rollup = Some(rollup::run(self).await?); // numbers, within 10 minutes
        }
        Ok(report)
    }
}
```

### 5. Every row and every object carries the project

Every object key starts with the project id. Every table has `project_id` as the first column of its primary key. There is no shared table of events, no shared prefix and no query that can be written without a project. Deleting a project is deleting one prefix and the rows of one id.

In the code a store or database call cannot be made without a `Scope`, so forgetting the project is a compile error, not a leak.

`crates/sypnia-core/src/ids.rs`

```rust
/// `prj_` and 12 characters of [0-9a-z]. Cheap to clone, impossible to build from a bare string.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct ProjectId(Arc<str>);

impl TryFrom<String> for ProjectId {
    type Error = InvalidId;
    fn try_from(s: String) -> Result<Self, InvalidId> {
        let ok = s.len() == 16
            && s.starts_with("prj_")
            && s[4..].bytes().all(|b| b.is_ascii_digit() || b.is_ascii_lowercase());
        if ok { Ok(Self(s.into())) } else { Err(InvalidId::Project) }
    }
}

/// Where a piece of data belongs. Every key and every query starts from one.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Scope {
    pub project: ProjectId,
    pub env: Environment,   // "production" when the sender names none
}
```

## What changed from the system it comes from

|                       | The original                      | The Sypnia engine                               | Why                                                                      |
| --------------------- | --------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------ |
| Projects              | One app, two stages               | Any number; the project is in every key and row | Brief: multi-project from the first table                                |
| Freshness             | Hourly job, at minute 20          | Fast path every 30 s, roll-up every 5 min       | The 2-minute and 10-minute promises                                      |
| Reading raw events    | DuckDB inside the job             | Plain Rust, streaming                           | No external query engine; a 68 MB binding and a 1,200 MB ceiling removed |
| Writing raw events    | Amazon Data Firehose, 60 s buffer | The engine's own buffer (2 s); a queue on AWS   | No dependency outside AWS; a problem cannot wait 60 s in a buffer        |
| Hourly tables         | Replace the hour                  | The same                                        | Proven                                                                   |
| Counters across hours | Add, guarded by one run per hour  | Add, guarded by a ledger of segments            | Runs are now minutes apart and per project                               |
| Duplicated batches    | Stored twice                      | Stored once: `batch_id`                         | The incident of 2026-09-30, when a CORS mistake doubled batches          |
| Fingerprint           | FNV-1a over UTF-16 code units     | Identical, bit for bit                          | Existing groups must not split                                           |
| Identity              | Numeric user and club ids         | Opaque person and account ids                   | Brief §10.1                                                              |
| Spans                 | A flat list of database calls     | Generic spans with parent, kind and attributes  | Brief §10.5, OTLP                                                        |

## The pages
