Docs

Data flow

Every hop an event makes inside the engine, with its time budget, what waits for what, how the engine pushes back, what happens when a piece is down, and how the 2-minute and 10-minute promises are kept.

Applies to Engine At launch

View as Markdown
On this page
  1. The hops
  2. What is synchronous
  3. Two clocks
  4. How the 2 minutes are kept
  5. How the 10 minutes are kept
  6. Replays
  7. Back-pressure and the «slow down» signal
  8. When a piece is down
  9. What changed from the system this comes from
  10. Next

This page follows an event through the engine hop by hop and puts a number on each one. It is for people who operate the engine and for anyone who wants to check that the freshness promises are arithmetic, not hope.

The promises, from How it works:

  • a new problem appears, and can alert, within 2 minutes of its first event;
  • counts, performance numbers and the session list are complete within 10 minutes;
  • a replay is watchable once its session has been idle for 2 minutes, or has ended.

The hops

Figure. The hops and their budgets. Above the line the SDK is waiting; below it nothing is. The Bucket is the point where an event is safe.
  1. Your app sends a batch to Ingest, at most every 10 seconds.
  2. Ingest writes one segment to the Bucket, then answers 200 with what it accepted and dropped.
  3. Ingest writes the segment’s ledger row to Postgres, after the answer.
  4. The Fast path reads new segments every 30 seconds.
  5. It writes the problem to Postgres and holds its alert.
  6. The agent takes the job and writes the sentence.
  7. Postgres releases the alert to Alerts.
  8. The Roll-up reads the open hour every 5 minutes and writes numbers and sessions.
  9. The API reads Postgres; the MCP server and the Console read through the API.
#HopWhat happensBudgetSynchronous?
1Your app (SDK) → IngestThe browser SDK flushes every 10 s, at 50 events, or when the page is hidden. Server SDKs flush when a request ends.10 s—
2Ingest → BucketKey, origin, sizes, each event; the lines wait in a buffer and are written as one segment.2 s in the buffer, 300 ms for the writeYes: the answer waits for the object
3Ingest → PostgresOne ledger row that says «this segment exists».50 msNo: after the answer, retried
4Bucket → Fast pathThe worker takes segments nobody scanned. In one process it is woken as soon as a segment lands.30 s (processing.interval_seconds)No
5Fast path → PostgresRead the segment, keep the failures, upsert problems, record the alert. One transaction.5 sNo
6The agentSymbolicate the stack, build the context, call the model, check the answer.60 s (ai.timeout_seconds), less if the alert is dueNo
7AlertsRelease the alert: mark it, send the webhook.10 s per delivery (webhooks.timeout_seconds)No
8Bucket → Roll-up → PostgresRe-read the open hour, replace its numbers, add to the counters, write per-session files.Starts every 5 min; 60 s per project and hourNo
9Postgres → API, MCP server, ConsoleIndexed reads. Replay chunks and traces are fetched from the Bucket.300 ms for a listYes, for the reader

Only hop 2 makes anyone wait, and it makes the SDK wait on purpose: the answer 200 means the events are in the Bucket, not in the memory of a process that may die. Everything after that can fail and be run again without losing an event, because the segment is still there.

What is synchronous

Figure. The only synchronous stretch. The answer leaves after the object exists and before anything is written to Postgres.
  1. Your app sends one batch: POST /v1/ingest/events with a batch_id and 6 events.
  2. Ingest finds the project in its key cache, then checks the origin, the sizes and each event. No database query.
  3. It appends the lines to the buffer. The buffer waits at most 2 seconds and is shared with the other batches of the project.
  4. Ingest writes the buffer to the Bucket as one segment.
  5. The Bucket confirms that the object exists.
  6. Only then does Ingest answer 200 with accepted: 6, dropped: 0.
  7. After the answer, Ingest writes the segment’s ledger row to Postgres, and retries until it succeeds.
Read this diagram as text

A sequence diagram with four lifelines, time going down: Your app (SDK), Ingest, Bucket and Postgres. The SDK waits from message 1 to message 6; by message 7 it is gone. Two other endings stand beside the steps they replace: if the buffer is full, Ingest answers 503 with Retry-After; if the Bucket refuses the write, Ingest answers 503 unavailable and the SDK sends the same batch_id again after 1 s, 4 s and 15 s.

Three things are worth noticing in the figure.

  • The answer is honest. accepted counts events that are in an object. If the Bucket refuses the write, the SDK gets 503 unavailable and sends the same batch again with the same batch_id.
  • The buffer is shared. All batches of one project and environment that arrive inside the same 2 seconds leave in one object. A busy project writes at most 30 objects a minute, however many browsers send.
  • Postgres is not on the path. The ledger row is written after the answer. If Postgres is down, ingest still accepts, and the row is written later.

On AWS a function cannot keep a buffer between requests, so the buffer is a queue: ingest answers once the batch is in the queue, and the worker writes the segment. The contract is the same, the durable point moves from the Bucket to the queue for a few seconds. See Deployment shapes.

Two clocks

The engine reads the same segments twice, at two speeds, for two different jobs.

Figure. Two clocks over the same files. The fast path answers «is something new broken?» every 30 seconds. The roll-up answers «how much of everything?» every 5 minutes.
Read this diagram as text

A timeline from 09:10 to 09:25 with three rows. Top row, segments in the Bucket: files land at irregular moments; the one written at 09:12:16 holds the failed payment. Middle row, the Fast path: a tick every 30 seconds; the tick at 09:12:30 scans the 09:12:16 segment and finds a new problem, and from the first failure (09:12:04) to the problem (09:13:02) 58 seconds pass, against a budget of 120. Bottom row, the Roll-up: one run at 09:15, 09:20 and 09:25, each re-reading the whole hour 09; the failed payment’s segment is counted in the 09:15 run, 2 minutes 56 seconds later, against a budget of 10 minutes. The Fast path writes problems, their occurrences and who they hit. The Roll-up writes page and API numbers, sessions, clicks, the traces index and usage.

Fast pathRoll-up
QuestionIs something new broken, and for whom?How much of everything happened?
ReadsEach new segment, onceThe whole open hour, again, on every run
KeepsFailures only: errors, failed calls, timeoutsEvery event
Writesproblems, problem_occurrences, problem_hits, alertsHourly and daily numbers, sessions, traces, usage_daily, per-session files
Every30 s5 min
Never writesA number of the roll-upA problem

The split by table is what keeps both simple. Neither path ever updates a row the other one owns, so there is nothing to reconcile between them.

This is the fast path. Notice the lease: a segment is claimed for 2 minutes with one statement, read without holding a transaction open, and marked as scanned in the same transaction that writes its problems.

crates/sypnia-pipeline/src/fast_path.rs
pub async fn run(engine: &Engine) -> Result<FastPathReport, EngineError> {
    let mut report = FastPathReport::default();
    held::promote_due(engine, &mut report).await?; // browser calls that waited for their server half

    // UPDATE … WHERE key = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1) RETURNING …
    // Two workers never get the same segment; a crashed worker's lease runs out.
    while let Some(segment) = engine.db.lease_unscanned_segment(LEASE).await? {
        let scope = segment.scope();
        let mut failures = FailureFold::new(&scope);

        let mut lines = engine.store.read_lines(&segment.key).await?;
        while let Some(line) = lines.try_next().await? {
            // A line that does not parse is counted and skipped. It never stops the segment.
            let Ok(record) = serde_json::from_slice::<Record>(&line) else {
                report.unreadable += 1;
                continue;
            };
            if let Some(failure) = Failure::from_record(&record) {
                failures.push(failure);
            }
        }

        let mut tx = engine.db.begin().await?;
        let fresh = tx.record_batches(&scope, &segment).await?; // batch ids seen before are skipped
        let transitions = problems::apply(&mut tx, failures.only(&fresh)).await?;
        alerts::record(&mut tx, &transitions, engine.clock.now()).await?;
        tx.mark_scanned(&segment, fresh.skipped()).await?;
        tx.commit().await?; // problems, alerts and the ledger move together, or not at all

        agent::enqueue(engine, &transitions).await; // best effort; the next pass picks up what is missed
        report.absorb(&segment, &transitions);
    }
    Ok(report)
}

How the 2 minutes are kept

The clock starts when the failure happens in your app. The worst case is the sum of the worst case of every hop.

HopOne process (serve)AWS functions
SDK holds the event10 s10 s
Buffer or queue2 s10 s (the queue’s batching window)
Segment write0.3 s0.3 s
Wait for the fast path30 s (0 when ingest and worker share a process)0 s (the queue wakes the worker)
Scan and transaction5 s5 s
Problem exists47.3 s25.3 s
Only for a browser call that waits for its server half: hold15 s15 s
…then the next pass30 s60 s (the one-minute schedule, if no traffic wakes the worker sooner)
Problem exists, held case92.3 s100.3 s

Both columns end under 120 seconds. The hold is explained in Problem grouping: a failed call seen by the browser waits up to 15 seconds for the failure behind it on the server, so that one request is one problem. It applies only to projects that also send server data.

The alert is written in the same transaction as the problem, held. It is released by whichever comes first:

  1. the agent finishes the sentence;
  2. 100 seconds have passed since the failure reached ingest (110 seconds since it happened, counting the SDK’s 10).

So with a model configured, the agent has what is left of those 100 seconds, usually more than a minute. Without one, the alert is released at once, with the technical title. In the worst held case on AWS there is no time left for the model, and the alert leaves with the technical title; the sentence appears on the problem when it is written.

If you raise processing.interval_seconds above 30, the first column grows by the same amount, twice in the held case. At 60 the held case passes 2 minutes. That is why the setting stops at 60 and its description warns you.

How the 10 minutes are kept

StepWorst case
SDK holds the event10 s
Buffer and write2.3 s
Wait for the next roll-up5 min
The roll-up of this project and hour60 s
Numbers complete6 min 13 s
Slack for one run that fails and is tried again on the next pass3 min 47 s

A roll-up that fails is not tried again five minutes later. rollup::is_due is true at every 5-minute mark and whenever the last run of a project did not finish, so the retry comes with the next 30-second pass.

Two details keep the numbers right across the top of the hour:

  • A segment’s hour is the hour it was written, not the time inside its events. Events sent at 09:59:58 can land in hour 10.
  • An hour is re-read until 10 minutes after it ends, then closed. A segment written to a closed hour cannot exist, because segments are written with the current time.

Replays

A replay chunk does not go through the buffer. Each upload is written to the Bucket as its own object before the answer. The API lists a session’s chunks straight from the Bucket, so a recording can be played as soon as its chunks exist. The SDK uploads the last chunk when the page is hidden or the session goes idle, which is why the promise says «idle for 2 minutes, or ended».

A problem links to its sessions through problem_hits, written by the fast path. So the replay of the session behind a brand-new problem can be opened from the alert, minutes before that session shows in the session list.

Back-pressure and the «slow down» signal

The engine never drops silently. Each kind of pressure has one answer, and the SDK knows what to do with each.

PressureWhere it showsThe engine answersThe SDK does
Too many requests from one address with one keyIngest, a counter in memory429 rate_limited with Retry-AfterWaits, sends the same batch again
The buffer of a project is full (32 MiB waiting to be written)Ingest503 unavailable with Retry-After: 2Retries after 1 s, 4 s, 15 s with the same batch_id; keeps at most 500 events, the oldest leave first
The Bucket refuses writesIngest503 unavailableThe same
The project reached its events limitThe answer of every batch and of /v1/ingest/config200 with slow_down: { send: "failures" }; other events are counted in dropped_reasons.limit_reachedSends only errors and failed requests until until
The project reached its replays limitThe sameslow_down: { replay: false }; chunks get 429 limit_reachedStops recording until until
The worker is behindThe ledger: segments not scanned, hours not rolledNothing changes for SDKs. After 30 minutes an engine.failing alert—
Postgres has no free connectionAPI503 with Retry-After: 2Your client retries

The slow-down state is decided away from the request. The roll-up counts usage; a pass compares it with the limits and writes the result on the project; ingest reads it from its cache. Notice that the function below is pure: the same counters always give the same answer, so it is trivial to test and safe to run on every pass.

crates/sypnia-core/src/quota.rs
/// What SDKs must hold back. `None` when the project is inside its limits,
/// which is always the case when a limit is 0 (no limit: the default of a self-hosted engine).
pub fn slow_down(usage: &Usage, limits: &Limits, period_end: DateTime<Utc>) -> Option<SlowDown> {
    let events_over = limits.events_per_month > 0 && usage.events >= limits.events_per_month;
    let replays_over = limits.replays_per_month > 0 && usage.replays >= limits.replays_per_month;

    match (events_over, replays_over) {
        (false, false) => None,
        (true, _) => Some(SlowDown {
            reason: SlowDownReason::EventsLimit,
            // Failures are always accepted, so problems are still found.
            send: Send::Failures,
            replay: false,
            until: period_end,
        }),
        (false, true) => Some(SlowDown {
            reason: SlowDownReason::ReplaysLimit,
            send: Send::All,
            replay: false,
            until: period_end,
        }),
    }
}

When a piece is down

DownWhat keeps workingWhat stopsWhat is lostRecovery
IngestReading: API, MCP server, ConsoleReceivingEvents that the SDK cannot hold: it keeps 500 and retries three timesNone needed
BucketLists and problems, from PostgresReceiving (503), replays, session timelines, tracesNothing accepted is lost; SDKs retry as aboveNone needed
PostgresReceiving: ingest uses its cache and keeps writing segmentsThe fast path, the roll-up, the API (503), new keysNothing. Segments wait in the BucketThe ledger rows that could not be written are written; then the worker catches up, oldest first
The workerReceiving and readingNew problems, numbers, alerts, uptime checksNothingOn start it scans everything not scanned and rolls every open hour of the last 48 hours
The AI providerEverything elseThe sentenceNothing. analysis.status is failed after three triesTried again when the problem grows, or on request
Your webhook receiverEverything else; alerts are still recordedThat deliveryNothing while retries lastDeliveries are retried with growing waits; an engine.failing alert if they keep failing

Two rules make this table short:

  • The Bucket is the source of truth. Everything in Postgres that comes from events can be built again from segments, with sypnia aggregate --from … --to …. The exceptions are what people wrote: statuses, notes, rules, keys.
  • A revoked key keeps being refused during a Postgres outage, and a valid key keeps working. The cache holds its last good copy for up to one hour without a refresh. After that ingest answers 503 instead of guessing.

What changed from the system this comes from

ThereHereWhy
One hourly job at minute 20 does everythingA fast path every 30 s and a roll-up every 5 minThe 2-minute and 10-minute promises
A managed buffer of 60 s or 5 MB in front of the bucketThe engine’s own buffer of 2 s or 4 MiB60 s is half of the 2-minute budget; and the self-hosted shape cannot depend on a service that exists only on AWS
The answer left before the write was confirmedThe answer waits for the objectSo accepted is true
A failed hour was retried by a 48-hour backfillThe same 48-hour window, plus an immediate retryKept; it is what saved hours there

Next

  • Ingest

    The checks, the buffer and the object names in detail.
  • Aggregation

    What the roll-up does with an hour.
  • Problem grouping

    What the fast path does with a failure.

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close