# Data flow

> Every hop an event makes inside the engine, with its time budget, what waits for what, how the engine pushes back, what happens when a piece is down, and how the 2-minute and 10-minute promises are kept.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/architecture/data-flow/

This page follows an event through the engine hop by hop and puts a number on each one. It is for people who operate the engine and for anyone who wants to check that the freshness promises are arithmetic, not hope.

The promises, from [How it works](https://docs.sypnia.com/start/how-it-works):

- a new problem appears, and can alert, **within 2 minutes** of its first event;
- counts, performance numbers and the session list are complete **within 10 minutes**;
- a replay is watchable once its session has been idle for 2 minutes, or has ended.

> **Note: About the numbers and the code**
>
> Every time on this page is a **budget**: the most a hop is allowed to take, which the tests enforce. None is a benchmark of the Rust engine, which does not exist yet. Figures taken from the system this engine comes from say so. The Rust is a design excerpt; the repository opens at launch.

## The hops

**Figure**: The hops and their budgets. Above the line the SDK is waiting; below it nothing is. The Bucket is the point where an event is safe.

1. Your app sends a batch to Ingest, at most every 10 seconds.
2. Ingest writes one segment to the Bucket, then answers `200` with what it accepted and dropped.
3. Ingest writes the segment's ledger row to Postgres, after the answer.
4. The Fast path reads new segments every 30 seconds.
5. It writes the problem to Postgres and holds its alert.
6. The agent takes the job and writes the sentence.
7. Postgres releases the alert to Alerts.
8. The Roll-up reads the open hour every 5 minutes and writes numbers and sessions.
9. The API reads Postgres; the MCP server and the Console read through the API.

| # | Hop                                 | What happens                                                                                                         | Budget                                                | Synchronous?                         |
| - | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------ |
| 1 | Your app (SDK) → Ingest             | The browser SDK flushes every 10 s, at 50 events, or when the page is hidden. Server SDKs flush when a request ends. | 10 s                                                  | —                                    |
| 2 | Ingest → Bucket                     | Key, origin, sizes, each event; the lines wait in a buffer and are written as one segment.                           | 2 s in the buffer, 300 ms for the write               | Yes: the answer waits for the object |
| 3 | Ingest → Postgres                   | One ledger row that says «this segment exists».                                                                      | 50 ms                                                 | No: after the answer, retried        |
| 4 | Bucket → Fast path                  | The worker takes segments nobody scanned. In one process it is woken as soon as a segment lands.                     | 30 s (`processing.interval_seconds`)                  | No                                   |
| 5 | Fast path → Postgres                | Read the segment, keep the failures, upsert problems, record the alert. One transaction.                             | 5 s                                                   | No                                   |
| 6 | The agent                           | Symbolicate the stack, build the context, call the model, check the answer.                                          | 60 s (`ai.timeout_seconds`), less if the alert is due | No                                   |
| 7 | Alerts                              | Release the alert: mark it, send the webhook.                                                                        | 10 s per delivery (`webhooks.timeout_seconds`)        | No                                   |
| 8 | Bucket → Roll-up → Postgres         | Re-read the open hour, replace its numbers, add to the counters, write per-session files.                            | Starts every 5 min; 60 s per project and hour         | No                                   |
| 9 | Postgres → API, MCP server, Console | Indexed reads. Replay chunks and traces are fetched from the Bucket.                                                 | 300 ms for a list                                     | Yes, for the reader                  |

Only hop 2 makes anyone wait, and it makes the SDK wait on purpose: the answer `200` means the events are in the Bucket, not in the memory of a process that may die. Everything after that can fail and be run again without losing an event, because the segment is still there.

## What is synchronous

**Figure**: The only synchronous stretch. The answer leaves after the object exists and before anything is written to Postgres.

1. Your app sends one batch: `POST /v1/ingest/events` with a `batch_id` and 6 events.
2. Ingest finds the project in its key cache, then checks the origin, the sizes and each event. No database query.
3. It appends the lines to the buffer. The buffer waits at most 2 seconds and is shared with the other batches of the project.
4. Ingest writes the buffer to the Bucket as one segment.
5. The Bucket confirms that the object exists.
6. Only then does Ingest answer `200` with `accepted: 6, dropped: 0`.
7. After the answer, Ingest writes the segment's ledger row to Postgres, and retries until it succeeds.

A sequence diagram with four lifelines, time going down: Your app (SDK), Ingest, Bucket and Postgres. The SDK waits from message 1 to message 6; by message 7 it is gone. Two other endings stand beside the steps they replace: if the buffer is full, Ingest answers `503` with `Retry-After`; if the Bucket refuses the write, Ingest answers `503 unavailable` and the SDK sends the same `batch_id` again after 1 s, 4 s and 15 s.

Three things are worth noticing in the figure.

- **The answer is honest.** `accepted` counts events that are in an object. If the Bucket refuses the write, the SDK gets `503 unavailable` and sends the same batch again with the same `batch_id`.
- **The buffer is shared.** All batches of one project and environment that arrive inside the same 2 seconds leave in one object. A busy project writes at most 30 objects a minute, however many browsers send.
- **Postgres is not on the path.** The ledger row is written after the answer. If Postgres is down, ingest still accepts, and the row is written later.

On AWS a function cannot keep a buffer between requests, so the buffer is a queue: ingest answers once the batch is in the queue, and the worker writes the segment. The contract is the same, the durable point moves from the Bucket to the queue for a few seconds. See [Deployment shapes](https://docs.sypnia.com/architecture/deployment-shapes).

## Two clocks

The engine reads the same segments twice, at two speeds, for two different jobs.

**Figure**: Two clocks over the same files. The fast path answers «is something new broken?» every 30 seconds. The roll-up answers «how much of everything?» every 5 minutes.

A timeline from 09:10 to 09:25 with three rows. Top row, segments in the Bucket: files land at irregular moments; the one written at 09:12:16 holds the failed payment. Middle row, the Fast path: a tick every 30 seconds; the tick at 09:12:30 scans the 09:12:16 segment and finds a new problem, and from the first failure (09:12:04) to the problem (09:13:02) 58 seconds pass, against a budget of 120. Bottom row, the Roll-up: one run at 09:15, 09:20 and 09:25, each re-reading the whole hour 09; the failed payment's segment is counted in the 09:15 run, 2 minutes 56 seconds later, against a budget of 10 minutes. The Fast path writes problems, their occurrences and who they hit. The Roll-up writes page and API numbers, sessions, clicks, the traces index and usage.

|              | Fast path                                                   | Roll-up                                                                          |
| ------------ | ----------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Question     | Is something new broken, and for whom?                      | How much of everything happened?                                                 |
| Reads        | Each new segment, once                                      | The whole open hour, again, on every run                                         |
| Keeps        | Failures only: errors, failed calls, timeouts               | Every event                                                                      |
| Writes       | `problems`, `problem_occurrences`, `problem_hits`, `alerts` | Hourly and daily numbers, `sessions`, `traces`, `usage_daily`, per-session files |
| Every        | 30 s                                                        | 5 min                                                                            |
| Never writes | A number of the roll-up                                     | A problem                                                                        |

The split by table is what keeps both simple. Neither path ever updates a row the other one owns, so there is nothing to reconcile between them.

This is the fast path. Notice the lease: a segment is claimed for 2 minutes with one statement, read without holding a transaction open, and marked as scanned in the same transaction that writes its problems.

`crates/sypnia-pipeline/src/fast_path.rs`

```rust
pub async fn run(engine: &Engine) -> Result<FastPathReport, EngineError> {
    let mut report = FastPathReport::default();
    held::promote_due(engine, &mut report).await?; // browser calls that waited for their server half

    // UPDATE … WHERE key = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING …
    // Two workers never get the same segment; a crashed worker's lease runs out.
    while let Some(segment) = engine.db.lease_unscanned_segment(LEASE).await? {
        let scope = segment.scope();
        let mut failures = FailureFold::new(&scope);

        let mut lines = engine.store.read_lines(&segment.key).await?;
        while let Some(line) = lines.try_next().await? {
            // A line that does not parse is counted and skipped. It never stops the segment.
            let Ok(record) = serde_json::from_slice::<Record>(&line) else {
                report.unreadable += 1;
                continue;
            };
            if let Some(failure) = Failure::from_record(&record) {
                failures.push(failure);
            }
        }

        let mut tx = engine.db.begin().await?;
        let fresh = tx.record_batches(&scope, &segment).await?; // batch ids seen before are skipped
        let transitions = problems::apply(&mut tx, failures.only(&fresh)).await?;
        alerts::record(&mut tx, &transitions, engine.clock.now()).await?;
        tx.mark_scanned(&segment, fresh.skipped()).await?;
        tx.commit().await?; // problems, alerts and the ledger move together, or not at all

        agent::enqueue(engine, &transitions).await; // best effort; the next pass picks up what is missed
        report.absorb(&segment, &transitions);
    }
    Ok(report)
}
```

## How the 2 minutes are kept

The clock starts when the failure happens in your app. The worst case is the sum of the worst case of every hop.

| Hop                                                          | One process (`serve`)                           | AWS functions                                                         |
| ------------------------------------------------------------ | ----------------------------------------------- | --------------------------------------------------------------------- |
| SDK holds the event                                          | 10 s                                            | 10 s                                                                  |
| Buffer or queue                                              | 2 s                                             | 10 s (the queue's batching window)                                    |
| Segment write                                                | 0.3 s                                           | 0.3 s                                                                 |
| Wait for the fast path                                       | 30 s (0 when ingest and worker share a process) | 0 s (the queue wakes the worker)                                      |
| Scan and transaction                                         | 5 s                                             | 5 s                                                                   |
| **Problem exists**                                           | **47.3 s**                                      | **25.3 s**                                                            |
| Only for a browser call that waits for its server half: hold | 15 s                                            | 15 s                                                                  |
| …then the next pass                                          | 30 s                                            | 60 s (the one-minute schedule, if no traffic wakes the worker sooner) |
| **Problem exists, held case**                                | **92.3 s**                                      | **100.3 s**                                                           |

Both columns end under 120 seconds. The hold is explained in [Problem grouping](https://docs.sypnia.com/architecture/problem-grouping): a failed call seen by the browser waits up to 15 seconds for the failure behind it on the server, so that one request is one problem. It applies only to projects that also send server data.

**The alert** is written in the same transaction as the problem, held. It is released by whichever comes first:

1. the agent finishes the sentence;
2. 100 seconds have passed since the failure reached ingest (110 seconds since it happened, counting the SDK's 10).

So with a model configured, the agent has what is left of those 100 seconds, usually more than a minute. Without one, the alert is released at once, with the technical title. In the worst held case on AWS there is no time left for the model, and the alert leaves with the technical title; the sentence appears on the problem when it is written.

If you raise `processing.interval_seconds` above 30, the first column grows by the same amount, twice in the held case. At 60 the held case passes 2 minutes. That is why the setting stops at 60 and its description warns you.

## How the 10 minutes are kept

| Step                                                             | Worst case     |
| ---------------------------------------------------------------- | -------------- |
| SDK holds the event                                              | 10 s           |
| Buffer and write                                                 | 2.3 s          |
| Wait for the next roll-up                                        | 5 min          |
| The roll-up of this project and hour                             | 60 s           |
| **Numbers complete**                                             | **6 min 13 s** |
| Slack for one run that fails and is tried again on the next pass | 3 min 47 s     |

A roll-up that fails is not tried again five minutes later. `rollup::is_due` is true at every 5-minute mark **and** whenever the last run of a project did not finish, so the retry comes with the next 30-second pass.

Two details keep the numbers right across the top of the hour:

- A segment's hour is the hour it was **written**, not the time inside its events. Events sent at 09:59:58 can land in hour 10.
- An hour is re-read until 10 minutes after it ends, then closed. A segment written to a closed hour cannot exist, because segments are written with the current time.

### Replays

A replay chunk does not go through the buffer. Each upload is written to the Bucket as its own object before the answer. The API lists a session's chunks straight from the Bucket, so a recording can be played as soon as its chunks exist. The SDK uploads the last chunk when the page is hidden or the session goes idle, which is why the promise says «idle for 2 minutes, or ended».

A problem links to its sessions through `problem_hits`, written by the fast path. So the replay of the session behind a brand-new problem can be opened from the alert, minutes before that session shows in the session list.

## Back-pressure and the «slow down» signal

The engine never drops silently. Each kind of pressure has one answer, and the SDK knows what to do with each.

| Pressure                                                       | Where it shows                                       | The engine answers                                                                                        | The SDK does                                                                                            |
| -------------------------------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Too many requests from one address with one key                | Ingest, a counter in memory                          | `429 rate_limited` with `Retry-After`                                                                     | Waits, sends the same batch again                                                                       |
| The buffer of a project is full (32 MiB waiting to be written) | Ingest                                               | `503 unavailable` with `Retry-After: 2`                                                                   | Retries after 1 s, 4 s, 15 s with the same `batch_id`; keeps at most 500 events, the oldest leave first |
| The Bucket refuses writes                                      | Ingest                                               | `503 unavailable`                                                                                         | The same                                                                                                |
| The project reached its events limit                           | The answer of every batch and of `/v1/ingest/config` | `200` with `slow_down: { send: "failures" }`; other events are counted in `dropped_reasons.limit_reached` | Sends only errors and failed requests until `until`                                                     |
| The project reached its replays limit                          | The same                                             | `slow_down: { replay: false }`; chunks get `429 limit_reached`                                            | Stops recording until `until`                                                                           |
| The worker is behind                                           | The ledger: segments not scanned, hours not rolled   | Nothing changes for SDKs. After 30 minutes an `engine.failing` alert                                      | —                                                                                                       |
| Postgres has no free connection                                | API                                                  | `503` with `Retry-After: 2`                                                                               | Your client retries                                                                                     |

The slow-down state is decided away from the request. The roll-up counts usage; a pass compares it with the limits and writes the result on the project; ingest reads it from its cache. Notice that the function below is pure: the same counters always give the same answer, so it is trivial to test and safe to run on every pass.

`crates/sypnia-core/src/quota.rs`

```rust
/// What SDKs must hold back. `None` when the project is inside its limits,
/// which is always the case when a limit is 0 (no limit: the default of a self-hosted engine).
pub fn slow_down(usage: &Usage, limits: &Limits, period_end: DateTime<Utc>) -> Option<SlowDown> {
    let events_over = limits.events_per_month > 0 && usage.events >= limits.events_per_month;
    let replays_over = limits.replays_per_month > 0 && usage.replays >= limits.replays_per_month;

    match (events_over, replays_over) {
        (false, false) => None,
        (true, _) => Some(SlowDown {
            reason: SlowDownReason::EventsLimit,
            // Failures are always accepted, so problems are still found.
            send: Send::Failures,
            replay: false,
            until: period_end,
        }),
        (false, true) => Some(SlowDown {
            reason: SlowDownReason::ReplaysLimit,
            send: Send::All,
            replay: false,
            until: period_end,
        }),
    }
}
```

> **Warning: How far a project can go past its limit**
>
> Usage is counted by the roll-up and reaches ingest through a cache that is refreshed every 60 seconds. So a project can pass a limit by what it sends in about 7 minutes. The alternative, counting in the request, would put a database write on the hot path of every batch. The engine chooses to be a few minutes late and never slow.

## When a piece is down

| Down                      | What keeps working                                          | What stops                                            | What is lost                                                          | Recovery                                                                                        |
| ------------------------- | ----------------------------------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **Ingest**                | Reading: API, MCP server, Console                           | Receiving                                             | Events that the SDK cannot hold: it keeps 500 and retries three times | None needed                                                                                     |
| **Bucket**                | Lists and problems, from Postgres                           | Receiving (`503`), replays, session timelines, traces | Nothing accepted is lost; SDKs retry as above                         | None needed                                                                                     |
| **Postgres**              | Receiving: ingest uses its cache and keeps writing segments | The fast path, the roll-up, the API (`503`), new keys | Nothing. Segments wait in the Bucket                                  | The ledger rows that could not be written are written; then the worker catches up, oldest first |
| **The worker**            | Receiving and reading                                       | New problems, numbers, alerts, uptime checks          | Nothing                                                               | On start it scans everything not scanned and rolls every open hour of the last 48 hours         |
| **The AI provider**       | Everything else                                             | The sentence                                          | Nothing. `analysis.status` is `failed` after three tries              | Tried again when the problem grows, or on request                                               |
| **Your webhook receiver** | Everything else; alerts are still recorded                  | That delivery                                         | Nothing while retries last                                            | Deliveries are retried with growing waits; an `engine.failing` alert if they keep failing       |

Two rules make this table short:

- **The Bucket is the source of truth.** Everything in Postgres that comes from events can be built again from segments, with `sypnia aggregate --from … --to …`. The exceptions are what people wrote: statuses, notes, rules, keys.
- **A revoked key keeps being refused during a Postgres outage, and a valid key keeps working.** The cache holds its last good copy for up to one hour without a refresh. After that ingest answers `503` instead of guessing.

## What changed from the system this comes from

| There                                                   | Here                                             | Why                                                                                                               |
| ------------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- |
| One hourly job at minute 20 does everything             | A fast path every 30 s and a roll-up every 5 min | The 2-minute and 10-minute promises                                                                               |
| A managed buffer of 60 s or 5 MB in front of the bucket | The engine's own buffer of 2 s or 4 MiB          | 60 s is half of the 2-minute budget; and the self-hosted shape cannot depend on a service that exists only on AWS |
| The answer left before the write was confirmed          | The answer waits for the object                  | So `accepted` is true                                                                                             |
| A failed hour was retried by a 48-hour backfill         | The same 48-hour window, plus an immediate retry | Kept; it is what saved hours there                                                                                |

## Next
