# What happens when something fails

> Every hop of the engine with what can fail in it, where the data waits meanwhile, how many times it is tried, where it goes when it cannot be done, who is told and how it is replayed. Queues, dead-letter queues and health, in both deployment shapes.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/architecture/reliability/

For agents: Read this to know the engine's delivery guarantee (at-least-once between steps, idempotent steps), the failure table per hop, the intake queue and its dead-letter queue on AWS (retention, visibility timeout, max receives, redrive, alarms), the in-memory buffer of the one-process shape, quarantine of poison segments, work as rows in Postgres (jobs, alert releases, webhook deliveries), GET /healthz, GET /v1/engine/status and sypnia doctor.

This page answers one question: **is the engine fragile?** It goes through every hop an event makes, says what can fail there, where the data waits while it fails, how many times it is tried, where it goes when it cannot be done, who is told, and the exact command that replays it.

> **In plain words**
>
> Nothing your app sent, and the engine accepted, is thrown away because a step failed. Every step either finishes its work or leaves it where the next try finds it: in the browser for a few seconds, in a queue, in a file, or in a row with a time on it. Work that fails five times is set aside with its error written on it, so one bad file never blocks your project or anyone else's, and you are told. What can be lost is listed at the end of this page, and it is short: events your app could not hand over while the engine was unreachable.

> **Note: About the numbers and the code**
>
> Times and counts on this page are the design's rules: the tests enforce them. None is a benchmark of the Rust engine, which is not released yet. The Rust and SQL are design excerpts; the repository opens at launch. Incidents quoted are from the system this engine comes from.

## The guarantee

**Between steps, at least once. Inside a step, safe to repeat. Together: each event is counted once.**

No step of the engine trusts that it runs exactly one time. Functions are retried, queues deliver twice, processes are killed after doing the work and before saying so. Instead of pretending otherwise, the engine makes the second run harmless. Three mechanisms do it, and all three are on the pages that precede this one:

| Mechanism                                                              | What it guards                                                                                                                                                 | Where                                                                                                                              |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| **`batch_id`** on every stored line                                    | A batch that arrives twice, from an SDK that retried or a queue that delivered twice, is stored and never counted twice                                        | [Ingest](https://docs.sypnia.com/architecture/ingest#duplicates-and-idempotency)                                                   |
| **The ledger**: one row per segment, with `scanned_at` and `rolled_at` | A segment is scanned by one worker, and its counters are added in the same transaction that marks it. Both happen or neither does                              | [Aggregation](https://docs.sypnia.com/architecture/aggregation#exactly-once-the-ledger)                                            |
| **Upserts and unique keys**                                            | A problem has one row per fingerprint; an alert has one row per cause (`dedupe_key`); a job has one open row per problem; an hour's numbers are replaced whole | [Problem grouping](https://docs.sypnia.com/architecture/problem-grouping), [Storage](https://docs.sypnia.com/architecture/storage) |

The duplicate guard is one statement. Notice that it needs no lock and no «have I seen this?» read before the write: the primary key answers, and the ids that were **not** inserted are the duplicates.

`crates/sypnia-db/sql/batches_record.sql`

```sql
-- Run by the fast path inside the transaction that marks the segment as scanned.
-- $3 = the batch ids the segment carries. Returns the ones that are new.
INSERT INTO ingest_batches (project_id, batch_id, segment)
SELECT $1, batch_id, $2
  FROM unnest($3::text[]) AS t(batch_id)
ON CONFLICT (project_id, batch_id) DO NOTHING
RETURNING batch_id;
-- Every id of $3 that did not come back already belongs to an earlier segment.
-- Those go to segments.skipped, and from then on every reader skips their lines.
```

The second half of the guarantee is that **work is never held only in memory once it was accepted**. At every moment an accepted event is in one of four durable places:

| Place                                         | Holds                                                            | For how long                      |
| --------------------------------------------- | ---------------------------------------------------------------- | --------------------------------- |
| The intake queue **\[Engine]** AWS shape only | Validated batches that are not yet in a segment                  | Seconds normally; up to 4 days    |
| The Bucket                                    | Segments, replay chunks, per-session files                       | Until retention removes them      |
| The ledger in Postgres                        | Which segments exist, which are scanned, which are rolled        | With the segment                  |
| A row with a time on it                       | Agent jobs, held alerts, webhook deliveries, hours to roll again | Until done, or marked as given up |

The next figure is the data flow of the [previous page](https://docs.sypnia.com/architecture/data-flow) drawn for this question: every arrow that can fail has its retry loop, every waiting place is marked, and every exit for work that cannot be done is shown.

**Figure (diagram not drawn yet)**: The data flow with every retry loop, every place where work waits and every exit for work that cannot be done. Nothing leaves the picture: what is set aside stays visible and can be replayed.

It will show: FLOW DIAGRAM, left to right, same nodes and names as the «hops» diagram of architecture/data-flow, redrawn to show failure handling. NODES in order: «Your app (SDK)» · «Ingest» · a waiting place drawn as a small tray labelled «Intake queue (AWS) / buffer, 2 s (one process)» · «Bucket» · «Fast path» · «Roll-up» (below the Fast path, both reading from the Bucket) · «Postgres» · «The agent» · «Alerts» · «Your webhook» (dashed, outside the engine) · and on a lower lane «API · MCP server · Console» reading from Postgres and the Bucket. ON EVERY ARROW THAT CAN FAIL draw a small circular retry loop with a short label, exact texts: on SDK → Ingest «retry 1 s, 4 s, 15 s · same batch_id»; on Ingest → queue/Bucket «503: the SDK retries»; on queue → Bucket (worker writes the segment) «redelivered: 15 s, 1 min, 5 min, 15 min»; on Bucket → Fast path «lease 2 min · 5 tries»; on Bucket → Roll-up «next pass, then 1, 5, 15, 30 min»; on Fast path → The agent «3 tries: 5 s, 30 s»; on Alerts → Your webhook «7 tries over 21 hours»; on API → reader «503 + Retry-After». WAITING PLACES drawn as trays with a clock icon, exact labels: under Your app «queue of 500 events in memory»; the intake queue tray «up to 4 days»; under Bucket «segments wait here while Postgres is down»; under Postgres «rows with a time: jobs, held alerts, deliveries». EXITS for work that cannot be done, drawn as dashed arrows going DOWN to a bottom lane titled «Set aside, visible, replayable», with four boxes, exact labels: «Dead-letter queue (AWS) · 14 days» (from the intake queue), «Quarantined segment» (from the Fast path and the Roll-up), «analysis.status = failed» (from The agent), «delivery failed» (from Alerts). From each of the four boxes a thin return arrow back up to its source labelled with the replay: «redrive», «sypnia aggregate --from … --to …», «POST …/analyze», «POST …/retry». The lit element is the bottom lane. A small marker on the Bucket: «accepted events are safe from here on» (on AWS place a second, lighter marker on the intake queue: «on AWS: from here»). Numbered callouts (1) to (9) on the hops in the order of the table that follows.

## Hop by hop

One row per hop. «Tries» counts the first attempt. The sections after the table go deeper into the rows that need it.

| #  | Hop                                            | What can fail                                                                                                      | What happens, and where the data waits                                                                                                                                                                                                            | Gives up after, and goes to                                                                                          | Who is told                                                                                                 | How it is replayed                                                                                   | What you see meanwhile                                                                                         |
| -- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| 1  | Your app (SDK) → Ingest                        | Ingest unreachable, `429`, `503`, a timeout                                                                        | The browser SDK keeps the batch and sends it again after 1 s, 4 s and 15 s with the same `batch_id`, then with the next flush every 10 s. Events wait in its queue: 500 events in memory, the oldest leave first. Server SDKs retry and then drop | The queue of 500 overflows, or the tab closes: those events are lost                                                 | Nobody at once. A gap shows as `data_through` not advancing; a monitor of yours on `/healthz` sees it first | It cannot: the data never arrived                                                                    | Reading keeps working. No new data                                                                             |
| 2  | Ingest → the queue or the buffer               | The queue refuses the send (AWS). The buffer is full: 32 MiB waiting in all, or 8 MiB of one project (one process) | Ingest answers `503 unavailable` with `Retry-After: 2`. It has stored nothing, so the SDK's retry of hop 1 covers it                                                                                                                              | As hop 1                                                                                                             | Ingest's log line; `engine.failing` if it lasts 30 minutes                                                  | Not needed                                                                                           | As hop 1                                                                                                       |
| 3  | The queue or the buffer → Bucket               | The Bucket refuses the write                                                                                       | **One process:** the batches of that segment get `503` and their SDKs retry. **AWS:** the message stays in the queue and is delivered again after 15 s, 1 min, 5 min and 15 min                                                                   | **AWS:** 5 deliveries, then the dead-letter queue, kept 14 days                                                      | The dead-letter alarm; the queue-age alarm; `engine.failing`                                                | Redrive: one command, [below](#redrive)                                                              | Lists and problems still read from Postgres. Replays and timelines answer `503`                                |
| 4  | Ingest or worker → Postgres (the ledger row)   | Postgres down, no free connection                                                                                  | **One process:** the row is kept and retried; the segment is already in the Bucket. **AWS:** the message is not deleted until the row exists, so the queue redelivers it                                                                          | **One process:** never gives up; a restart finds the segment by listing. **AWS:** as hop 3                           | `engine.failing` after 30 minutes                                                                           | Nothing to do. After a restore: `sypnia aggregate --from <time>` lists the Bucket again              | Events are safe and not yet counted                                                                            |
| 5  | Bucket → Fast path → Postgres                  | The object cannot be read; a line that breaks the reader; the transaction fails; the worker dies mid-scan          | The segment's lease runs out after 2 minutes, or is given back with a wait of 10 s, 30 s, 2 min, 5 min. Another worker takes it. The segment waits in the Bucket, its row in the ledger                                                           | 5 tries, then **quarantined**: `quarantined_at` and the error on its ledger row                                      | An `engine.failing` alert that names the segment, at once                                                   | `sypnia aggregate --project <p> --from <t> --to <t>` gives it five new tries                         | Everything else of the project, and every other project, flows. The events of that one segment are not counted |
| 6  | Bucket → Roll-up → Postgres                    | Out of the memory budget, a read error, the commit fails, the worker dies                                          | Nothing was written: the run is one transaction. The hour is marked `failed` and tried again on the next pass, then after 1, 5 and 15 minutes, then every 30                                                                                      | Never, inside 48 hours. A run that fails 5 times on the same segment quarantines that segment and goes on without it | `engine.failing` after 30 minutes, naming the project and the hour                                          | By itself; or `sypnia aggregate --from … --to …` for hours older than 48 hours                       | Problems are on time. Numbers of that project and hour are late                                                |
| 7  | Fast path → the agent → the AI provider        | The provider answers an error, times out (60 s), or returns an answer that fails the check                         | The job is a row. It is tried again after 5 s and after 30 s. The alert does not wait: it leaves at its `release_by` with the technical title                                                                                                     | 3 tries, then the job is `dead` and the problem shows `analysis.status: failed` with the reason                      | The problem itself, in the API, the MCP server and the Console                                              | `POST /v1/problems/{problem}/analyze`; or by itself when the problem grows                           | The problem, its counts, its alert and its prompt, without the sentence                                        |
| 8  | Postgres → Alerts                              | The worker dies between writing the alert and releasing it                                                         | The alert is a row with `release_by`. Any pass releases what is past its time; releasing is one `UPDATE … WHERE released_at IS NULL`                                                                                                              | Never                                                                                                                | —                                                                                                           | Not needed                                                                                           | The alert, at most one pass late                                                                               |
| 9  | Alerts → your webhook                          | Your receiver is down, slow (10 s) or answers anything but `2xx`                                                   | The delivery is a row. 7 tries: at once, then after 1 min, 5 min, 30 min, 2 h, 6 h and 12 h, with the same event id                                                                                                                               | 7 tries, then the delivery is `failed`. After 3 days of nothing but failures the webhook is switched off             | `GET /v1/webhooks/{id}/deliveries`; an `engine.failing` alert when the webhook is switched off              | `POST /v1/webhooks/{webhook_id}/deliveries/{delivery_id}/retry`; `PATCH` the webhook to switch it on | The alert is recorded and shown in the Console and the API all along                                           |
| 10 | Postgres and Bucket → API, MCP server, Console | Postgres has no free connection; the Bucket does not answer                                                        | `503 unavailable` with `Retry-After: 2`, at once. The API does not queue                                                                                                                                                                          | —                                                                                                                    | The caller                                                                                                  | The caller retries                                                                                   | An error with a time to wait, never a wrong number                                                             |
| 11 | The schedule → the worker (AWS)                | A tick is missed, late, or delivered twice                                                                         | The next tick, a minute later, does everything the missed one would have: all due work is rows                                                                                                                                                    | Never                                                                                                                | The worker alarm if the function itself fails                                                               | Not needed                                                                                           | At most a minute of extra delay                                                                                |

Three things hold for every row:

- **The sender keeps the work until the receiver has it durably.** The SDK keeps the batch until `200`; the queue keeps the message until the worker deleted it; the ledger keeps the segment until its transaction committed.
- **Giving up is a state, not a disappearance.** A dead-letter queue, a quarantined segment, a `dead` job and a `failed` delivery are all things you can list, read the error of, and send again.
- **A failure delays; it does not corrupt.** A step that runs twice writes the same result. That is rule 3 of the [overview](https://docs.sypnia.com/architecture#3-every-step-can-run-twice-safely), and the replay tests of [the codebase](https://docs.sypnia.com/architecture/codebase#tests-that-replay-recorded-hours) run every step twice to prove it.

## The failures, one by one

### Ingest is down

SDKs hold what they can (hop 1) and reading is not affected: the API, the MCP server and the Console do not depend on ingest. On AWS «ingest is down» means the function fails or is throttled at its 40 copies; the address answers `503` or `429` and the SDKs retry. In the one-process shape it means the process is down, and the API with it, unless you run the roles apart.

What is lost is what the SDK could not hold: more than 500 events in a tab, a tab that closed, the last batch of a page that was sent with `sendBeacon` and got no answer. The engine cannot know about events it never received, so this loss shows only as a gap.

### The Bucket is down

Ingest answers `503` and nothing is accepted that is not stored, so there is nothing to recover: SDKs retry. On AWS, batches already in the queue wait there. Reading from Postgres keeps working: the problem list, a problem, numbers, alerts. Anything that needs an object answers `503`: replays, a session's timeline, a trace's detail.

### Postgres is down

Receiving continues. Ingest does not query Postgres: it uses its key cache, which stays valid for up to one hour without a refresh. Segments are written; the fast path, the roll-up and the API stop. When Postgres returns, the worker takes everything not scanned, oldest first and one project at a time in turns, and rolls every open hour.

| Shape         | Where events wait during the outage                                                                                                                                                       | If it lasts                                                                                                                                               |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| One process   | In the Bucket, as segments. Their ledger rows are kept in memory and retried; a worker that starts lists the last two hours of each project and adds any segment the ledger does not know | Longer than one hour: ingest answers `503`, because its key cache is too old to trust                                                                     |
| AWS functions | In the intake queue. A message is deleted only when its segment **and** its ledger row exist                                                                                              | Longer than about 20 minutes: batches reach their fifth delivery and move to the dead-letter queue, where they wait up to 14 days for one redrive command |

What Postgres holds that the Bucket cannot give back is what people wrote: statuses, notes, rules, keys, tokens. That is what its backups are for. Everything that comes from events is rebuilt with `sypnia aggregate --from … --to …`.

### A poison batch or segment

A «poison» input is one that fails every time it is tried: an object cut short, a line that triggers a bug in a reader, a value a table refuses. The danger is not the lost segment. It is a worker that takes it, dies, takes it again, and never reaches the work behind it.

Three layers stop that:

1. **A bad event never becomes a bad segment.** Ingest judges each event on its own: it cuts, cleans or drops it and says so in the answer. A line that still does not parse when read is counted (`unreadable`) and skipped. Neither stops anything.
2. **A bad message does not block the queue** (AWS). The worker answers the queue with the list of the messages it could not write, not with one failure for the whole delivery. The good ones are deleted; a bad one comes back alone and, after 5 deliveries, goes to the dead-letter queue.
3. **A bad segment is quarantined.** Every lease counts an attempt. A segment that was leased 5 times and never scanned is marked `quarantined_at`, with the last error, and no reader takes it again.

**Figure (diagram not drawn yet)**: A poison segment. It is tried five times with growing waits while everything around it is scanned, then set aside with its error. The object itself is never touched.

It will show: LANE DIAGRAM, time flowing left to right, three horizontal lanes. LANE 1 «Segments of prj_5k2m9x4q7w1z (shop)»: five file tiles in a row labelled «…091207Z», «…091216Z», «…091744Z» (this one drawn cracked, in the warning tone), «…091902Z», «…092015Z». LANE 2 «Fast path»: under each healthy tile a small check mark and an arrow down to a «Postgres» strip at the bottom labelled «problems, ledger: scanned_at». Under the cracked tile, five small attempt marks in a row with the waits between them, exact labels: «try 1 · fails», «+10 s · try 2», «+30 s · try 3», «+2 min · try 4», «+5 min · try 5», then a dashed arrow DOWN-RIGHT into a side box labelled «Quarantined · quarantined_at, error: unexpected end of gzip stream». From the side box two outgoing arrows: one to a bell icon labelled «engine.failing alert, names the segment», one curved arrow back to the cracked tile labelled «sypnia aggregate --project shop --from 09:00 --to 10:00 → five new tries». LANE 3 «Other projects»: a row of small tiles of a second project «prj_9c1t4h8n2b6d» all with check marks, and the note «turns are taken per project: nobody waits behind the bad segment». The tiles «…091902Z» and «…092015Z» of lane 1 must visibly be scanned WHILE the cracked tile is still being retried (place their check marks under tries 2 and 3). The lit element is the side box «Quarantined». Numbered callouts: (1) the cracked tile, (2) the five tries, (3) the quarantine box, (4) lane 3.

The whole decision is the function below. Notice that the attempt was already counted by the lease, so a worker that dies before reaching this code still uses up a try.

`crates/sypnia-pipeline/src/fast_path.rs`

```rust
const MAX_ATTEMPTS: u32 = 5;

/// How long a segment that failed waits before its next try.
fn backoff(attempts: u32) -> Duration {
    match attempts {
        0 | 1 => Duration::from_secs(10),
        2 => Duration::from_secs(30),
        3 => Duration::from_secs(120),
        _ => Duration::from_secs(300),
    }
}

/// Called by `run` when `scan_segment` returned an error. One bad segment never ends the pass.
async fn give_back(engine: &Engine, segment: &Segment, error: &EngineError) -> Result<Outcome, EngineError> {
    if segment.attempts < MAX_ATTEMPTS {
        // Free the lease early, with a wait. Another worker may be the one that succeeds.
        engine.db.release_segment(segment, backoff(segment.attempts), &error.to_string()).await?;
        return Ok(Outcome::Retrying);
    }
    // Out of the way of every reader. The object stays where it is; the hour is rolled without it.
    let mut tx = engine.db.begin().await?;
    tx.quarantine_segment(segment, &error.to_string()).await?;
    alerts::record_engine_failing(&mut tx, Failing::Quarantined(segment), engine.clock.now()).await?;
    tx.commit().await?;
    Ok(Outcome::Quarantined)
}
```

A quarantined segment is 2 seconds of one project's events, at most 4 MiB. Its events are not in any count until someone replays it. To do that, usually after upgrading to a version that fixes the reader:

```bash
sypnia aggregate --project shop --from 2026-03-12T09:00:00Z --to 2026-03-12T10:00:00Z
```

Processing a range again clears the quarantine of the segments in it, gives each five new tries, and adds to the ledger any segment of the range that is in the Bucket and not in the ledger. Nothing is counted twice: the segments that were already added keep their `rolled_at`.

### A slow project

One project must not make the others late, whatever it sends. Each part has its own guard:

| Part      | What a heavy project could take              | The guard                                                                                                                                                                                                                            |
| --------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Ingest    | The buffer                                   | One open segment per project. Of the 32 MiB that may wait in all, one project may hold 8 MiB; past that, **its** batches get `503`, the others do not. Rate limits are per key                                                       |
| Fast path | The worker's time                            | Turns. Each lease goes to the project that was served longest ago, and takes one segment. A project with a backlog of a thousand segments gets one turn in each round, like a project with one                                       |
| Roll-up   | Memory and the `processing.concurrency` runs | 256 MB per run, enforced; a run past its budget stops and marks the hour `failed`. Projects are taken least recently rolled first, and a failed hour waits 1, 5, 15, then 30 minutes between tries, so it cannot hold a run for ever |
| The agent | Model calls                                  | 100 analyses a day per project and 3 per problem; one open job per problem                                                                                                                                                           |
| API       | Connections                                  | A small bounded pool; `503` with `Retry-After` instead of a queue                                                                                                                                                                    |

The lease is one statement. Notice the order: first by when the project last had a turn, then by age. The index it reads holds only segments that are waiting, so it is small by construction.

`crates/sypnia-db/sql/segment_lease.sql`

```sql
-- $1 = the lease (2 minutes). Returns one segment, or none.
WITH pick AS (
    SELECT s.project_id, s.key
      FROM segments s
      LEFT JOIN project_turns t USING (project_id)
     WHERE s.scanned_at IS NULL
       AND s.quarantined_at IS NULL
       AND (s.leased_until IS NULL OR s.leased_until < now())  -- free, waited, or its worker died
     ORDER BY t.fast_path_at NULLS FIRST, s.created_at
     LIMIT 1
       FOR UPDATE OF s SKIP LOCKED                              -- two workers never get the same one
), turn AS (
    INSERT INTO project_turns (project_id, fast_path_at)
    SELECT project_id, now() FROM pick
    ON CONFLICT (project_id) DO UPDATE SET fast_path_at = EXCLUDED.fast_path_at
)
UPDATE segments s
   SET leased_until = now() + $1,
       attempts     = s.attempts + 1                            -- counted here, before any work
  FROM pick
 WHERE s.project_id = pick.project_id AND s.key = pick.key
RETURNING s.*;
```

### The AI provider fails or is slow

The agent is the only part that waits on a service the engine does not control, so nothing else waits on the agent. The alert is released at its `release_by` whether or not the sentence exists. A call has 60 seconds (`ai.timeout_seconds`); a job has 3 tries, after 5 and after 30 seconds; then the job is `dead` and the problem says `analysis.status: failed` with the provider's error. When 5 jobs in a row end that way the agent stops calling for 5 minutes, so an outage of the provider does not spend the worker's time on calls that cannot succeed; the jobs stay queued.

A failed analysis is tried again by itself when the problem grows or comes back, and at once when you ask: `POST /v1/problems/{problem}/analyze`.

### Your webhook receiver is down

The alert exists before any delivery is attempted: it is a row, shown in the Console and returned by `GET /v1/alerts`. The delivery is a second row with its own tries (hop 9). A receiver that was down for a day gets the alert on the seventh try, with the same event id, so it can ignore what it already handled. A receiver that was down for longer reads what it missed from `GET /v1/alerts`, or asks for single deliveries again.

### A deploy in the middle of a run

| Shape         | What a deploy does                                                                                                        | Why nothing breaks                                                                                                |
| ------------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| One process   | `SIGTERM`: requests get 20 seconds, open segments are flushed, the pass finishes its current step, then the process exits | A step that was cut is a lease that runs out or a transaction that never committed. The new process does it again |
| AWS functions | Calls in flight finish on the old version; new calls start on the new one                                                 | The same, and the queue keeps what a killed call did not delete                                                   |
| Both          | Old and new versions run side by side for a moment                                                                        | Migrations only add: an older binary keeps working against a newer schema                                         |

A worker killed in the middle of a roll-up leaves its hour marked `running`. Another worker takes that hour over after 20 minutes. This is the one case where numbers can be later than 10 minutes; problems are not affected, because a segment's lease is 2 minutes.

### The same run starts twice

It happens: a schedule that delivers a tick twice, two workers, a retry of a call that had in fact succeeded. Every piece of work has one claim that only one run can win:

| Work                        | The claim                                                                                           |
| --------------------------- | --------------------------------------------------------------------------------------------------- |
| Scanning a segment          | The lease: `FOR UPDATE SKIP LOCKED`                                                                 |
| Adding a segment's counters | `UPDATE segments SET rolled_at … WHERE rolled_at IS NULL`; a short count rolls the transaction back |
| An hour's numbers           | Replaced whole; two runs write the same rows                                                        |
| An agent job                | The claim of the [jobs table](#work-as-rows); one open job per problem                              |
| Releasing an alert          | `UPDATE … WHERE released_at IS NULL`                                                                |
| A webhook delivery          | The delivery row's lease; your receiver sees the same `evt_…` id if a try is repeated               |
| The daily job               | One job row per project and day                                                                     |

### Clocks that disagree

| Clock                         | Where it is wrong                                      | What the engine does                                                                                                                                                                                                                   |
| ----------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A device                      | `ts` in the past or the future                         | Each batch carries `sent_at`. When it differs from the arrival time by more than 2 minutes, the batch's times are shifted by the difference for ordering. An event older than 7 days is dropped as `too_old` and counted in the answer |
| An engine process             | Its time differs from the database's                   | Leases, `release_by` and every «is it due?» use the **database's** clock (`now()` in the statement), not the process's. `sypnia doctor` fails its `clock` check above 5 seconds of difference                                          |
| An ingest process, by minutes | A segment is named into an hour that is already closed | The ledger has a segment of a closed hour not yet rolled; the roll-up opens that hour again. Hours count by arrival, so nothing is lost or doubled                                                                                     |

### The queue backs up

On AWS, a worker that is slower than what arrives shows in the intake queue before it shows anywhere else. Nothing is lost while it lasts: the queue keeps a message for 4 days. Two things grow, and both are watched: the age of the oldest message (an alarm at 2 minutes) and the engine's own lag (an `engine.failing` alert at 30 minutes). SDKs notice nothing, because ingest answers once the batch is in the queue. The first thing to raise is the number of copies of the worker function.

## Queues and dead-letter queues

### On AWS: one queue and its dead-letter queue

The intake queue stands where the in-memory buffer is in the other shape. It is the only queue of the engine; why there is one and not four is on [Deployment shapes](https://docs.sypnia.com/architecture/deployment-shapes#why-a-queue-and-only-one). The template creates it with these values, and the reason for each:

| Setting                                 | Value                                      | Why                                                                                                           |
| --------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| Kind                                    | Standard                                   | Order does not matter and a duplicate is harmless: `batch_id`                                                 |
| A message                               | One validated batch, gzip, at most 256 KiB | A larger batch is written to the Bucket by ingest and the message only names it                               |
| Message retention                       | 4 days                                     | A worker can be broken for a long weekend without loss                                                        |
| Batching window                         | 10 seconds, or 500 messages, or 6 MB       | Segments stay few; a failure waits at most 10 seconds here                                                    |
| Visibility timeout                      | 15 minutes                                 | It may not be shorter than the worker's time limit. The worker shortens it itself for a message it gives back |
| Deliveries before the dead-letter queue | 5 (`maxReceiveCount`)                      | With the waits below, about 20 minutes of trying                                                              |
| Dead-letter queue retention             | 14 days, counted from the first send       | Time to look, fix and redrive                                                                                 |
| Encryption                              | At rest, with the service's keys           | Messages hold event data                                                                                      |

**Figure (diagram not drawn yet)**: The life of a message. Delivered, written and deleted is the normal path. A message that fails comes back after growing waits; after five deliveries it is parked in the dead-letter queue until someone redrives it.

It will show: STATE-AND-TIME DIAGRAM of ONE message through the intake queue, left to right. Start node «Ingest: SendMessage ok → 200 to the SDK». Then a horizontal track labelled «Intake queue» with the message drawn as a small envelope moving along it through these stations, each a rounded box with exact labels: (1) «Waiting · visible» → arrow «delivered with up to 10 s of other messages» → (2) «In flight · invisible, held by the worker». From (2) THREE outgoing branches. BRANCH A (accent colour, the happy path) → «Segment written + ledger row» → «Deleted» (end, check mark). BRANCH B (warning tone) labelled «the worker reports it as failed» → a box «Visible again after the wait» with a small table inside: «delivery 1 → 15 s», «2 → 1 min», «3 → 5 min», «4 → 15 min» → arrow back to station (1), labelled «delivered again: receive count +1». BRANCH C (dashed) labelled «the worker was killed: no answer» → «Visible again after 15 min (visibility timeout)» → back to station (1). From station (1) a downward dashed arrow labelled «receive count would pass 5» into a second track below labelled «Dead-letter queue · kept 14 days», where the envelope sits in a box «Parked · alarm: DeadLetterNotEmpty». From that box a curved arrow back UP to station (1) of the intake queue labelled «redrive: aws sqs start-message-move-task» and the note «arrives as a new message; duplicates are skipped by batch_id». The lit element is the dead-letter track and its redrive arrow. Under the whole diagram a thin time ruler for branch B: «0 · 15 s · 1 min 15 s · 6 min 15 s · 21 min 15 s → dead-letter queue». Numbered callouts (1) waiting, (2) in flight, (3) deleted, (4) the wait table, (5) dead-letter queue, (6) redrive.

**What the worker tells the queue.** A delivery brings many messages. The worker writes their segments and ledger rows, then answers with the ids of the messages it could not finish. Only those come back. Notice that an error never escapes the handler: a handler that fails as a whole would send the good messages back with the bad one.

`crates/sypnia/src/lambda.rs`

```rust
/// The queue case of `worker_event`. The answer names the messages to deliver again.
async fn intake_batch(engine: &Engine, batch: SqsEvent) -> SqsBatchResponse {
    let mut failed = Vec::new();
    // One group per project and environment: the messages that will share a segment.
    for group in intake::group(batch.records) {
        match intake::write_segment(engine, &group).await {
            // The object exists and so does its ledger row. Scanning may fail: the pass picks it up.
            Ok(segment) => {
                let _ = fast_path::scan(engine, std::slice::from_ref(&segment)).await;
            }
            Err(error) => {
                tracing::warn!(%error, scope = %group.scope, "segment not written; messages go back");
                for message in &group.messages {
                    // 15 s, 1 min, 5 min, 15 min, by how many times it was delivered.
                    queue::delay(engine, message, redelivery_wait(message.receive_count)).await;
                    failed.push(BatchItemFailure { item_identifier: message.id.clone() });
                }
            }
        }
    }
    SqsBatchResponse { batch_item_failures: failed }
}
```

A message that the worker gives back is seen again after its wait. A message whose worker was killed, out of memory or out of time, is seen again when the visibility timeout ends: up to 15 minutes later. Those events are late, not lost, and they are the one case on AWS where a problem can miss its 2 minutes without anything being broken for good.

#### Redrive

Messages in the dead-letter queue stay there for 14 days. Nothing reads them by itself: a batch that could not be written five times is either a bug to look at or an outage to wait out, and neither is helped by trying a sixth time at once. When the cause is gone, send them back:

```bash
# How many are parked
aws sqs get-queue-attributes --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --attribute-names ApproximateNumberOfMessages

# Look at one without removing it
aws sqs receive-message --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --max-number-of-messages 1 --visibility-timeout 0

# Send all of them back to the intake queue
aws sqs start-message-move-task --source-arn "$(aws sqs get-queue-attributes \
  --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)"
```

Redriven messages arrive as new ones and take the normal path. If some of them had in fact been written before, their batch ids are already known and their lines are skipped.

#### Alarms

The template creates three alarms. They are CloudWatch alarms because they must work when the engine itself does not; point them at a notification of yours.

| Alarm                | Fires when                                                                               | It means                                                        |
| -------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `DeadLetterNotEmpty` | The dead-letter queue has at least 1 message, for 1 minute                               | Batches were set aside. Look at one, fix, redrive               |
| `IntakeBacklog`      | The oldest message in the intake queue is older than 120 seconds, for 3 minutes in a row | The worker is not keeping up, or is not running                 |
| `WorkerFailing`      | The worker function ended in an error in 3 of the last 5 minutes                         | A bug, a missing permission, Postgres or the Bucket unreachable |

### On one process: the buffer

`sypnia serve` has no queue, because it does not need one: the process is the memory.

|                              |                                                                                                                                                                            |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Where a batch waits          | In the open segment of its project, in memory, for at most 2 seconds                                                                                                       |
| When the SDK is answered     | After the segment is in the Bucket. Not before                                                                                                                             |
| What a crash loses           | Nothing that was answered `200`. The batches in the open segments, at most 2 seconds of traffic, had not been answered: their SDKs get a network error and send them again |
| What a crash can still cost  | Those same batches, if the process stays down longer than the SDKs can hold them (hop 1)                                                                                   |
| A ledger row not yet written | Found again by listing: on start, a worker lists the last two hours of each project and adds what the ledger does not know. The daily job checks the day before as well    |
| A graceful stop              | Open segments are flushed before the process exits                                                                                                                         |

This is the honest difference between the two shapes. On AWS the durable point is the queue, a managed service that keeps working while functions fail. On one process the durable point is the Bucket, and between the SDK and the Bucket there is one process and 2 seconds. Run two `ingest` processes behind a load balancer if those seconds matter to you: an SDK's retry then reaches the other one.

### Work as rows

Everything after the Bucket that must happen later is a row in Postgres with a time on it, not a message in a queue. Three kinds of work use the pattern:

| Work                                                    | Table                                             | Tries and waits                               | Given up                        | Seen in                                                                      | Replayed with                                                   |
| ------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------- |
| The agent's analysis, the daily job, weekly suggestions | `jobs`                                            | 3: after 5 s and 30 s                         | `status = 'dead'`, `last_error` | The problem: `analysis.status: failed`. `GET /v1/engine/status`: `jobs.dead` | `POST /v1/problems/{problem}/analyze`                           |
| Releasing an alert                                      | `alerts` (`release_by`, `released_at`)            | Every pass, until done                        | Never                           | `GET /v1/alerts`                                                             | —                                                               |
| A webhook delivery                                      | `webhook_deliveries`                              | 7: after 1 min, 5 min, 30 min, 2 h, 6 h, 12 h | `status = 'failed'`, `error`    | `GET /v1/webhooks/{id}/deliveries?status=failed`                             | `POST /v1/webhooks/{webhook_id}/deliveries/{delivery_id}/retry` |
| An hour to roll again                                   | `hours` (`status`, `attempts`, `next_attempt_at`) | Next pass, then 1, 5, 15, 30 min              | Never inside 48 hours           | `GET /v1/engine/status`: `hours.failed`                                      | `sypnia aggregate --from … --to …`                              |

Why rows and not more queues:

- **The row is written in the transaction that causes it.** A problem is born and its job exists, or neither. A queue message sent after a commit can be lost between the two; one sent before can announce work that was rolled back.
- **A row can be read.** «What is waiting, since when, how many times was it tried, what did it say?» is a query, and the API answers it. A queue can only be counted.
- **Nothing more to deploy.** The one-process shape has no queue service at all, and the same code runs in both.

The table and the claim. Notice the second condition of the claim: a `running` job whose lease ran out belonged to a worker that died, and is taken over.

`migrations/0006_jobs.sql`

```sql
CREATE TABLE jobs (
    id              text PRIMARY KEY,                    -- job_…
    project_id      text NOT NULL REFERENCES projects ON DELETE CASCADE,
    kind            text NOT NULL CHECK (kind IN ('analysis', 'daily', 'suggestions')),
    subject         text NOT NULL,                       -- the problem id, the day, the week
    status          text NOT NULL DEFAULT 'queued'
                    CHECK (status IN ('queued', 'running', 'done', 'dead')),
    again           boolean NOT NULL DEFAULT false,      -- a trigger arrived while it ran: run once more
    attempts        integer NOT NULL DEFAULT 0,
    next_attempt_at timestamptz NOT NULL DEFAULT now(),
    leased_until    timestamptz,
    last_error      text,
    created_at      timestamptz NOT NULL DEFAULT now(),
    finished_at     timestamptz
);
-- One open job per subject: a second trigger sets `again` instead of inserting.
CREATE UNIQUE INDEX jobs_one_open ON jobs (project_id, kind, subject)
    WHERE status IN ('queued', 'running');
CREATE INDEX jobs_due ON jobs (next_attempt_at) WHERE status = 'queued';
```

`crates/sypnia-db/sql/job_claim.sql`

```sql
-- $1 = kind, $2 = the lease. Returns one job, or none.
UPDATE jobs j
   SET status = 'running', attempts = j.attempts + 1, leased_until = now() + $2
 WHERE j.id = (
        SELECT id FROM jobs
         WHERE kind = $1
           AND ((status = 'queued'  AND next_attempt_at <= now())
             OR (status = 'running' AND leased_until < now()))   -- its worker died
         ORDER BY next_attempt_at
         LIMIT 1
           FOR UPDATE SKIP LOCKED)
RETURNING j.*;
```

A job that fails goes back to `queued` with a later `next_attempt_at`, or to `dead` on its third failure. On AWS the worker calls itself to run a job at once; that call is only a shortcut. If it is lost, the minute tick finds the row. Lambda's own retries of that call are switched off, because the row is the retry.

## Health

Three things answer «is it working?», from the cheapest to the most complete.

### `GET /healthz`

Public, no credential. It answers `200` when the process is up and serving, and it does not test Postgres or the Bucket on purpose: a database outage must not take ingest out of a load balancer, because ingest works without the database.

```json
{ "status": "ok", "version": "1.0.0", "role": "all" }
```

Watch it from outside the machine the engine runs on. An engine that is down cannot raise its own alert.

### `GET /v1/engine/status`

With the root token **\[Engine]**. The state of the pipeline, across all projects. It is what the `pipeline` check of `sypnia doctor` prints and what the `engine.failing` alert is decided from.

```json
{
  "state": "ok",
  "engine_version": "1.0.0",
  "last_pass_at": "2026-03-12T10:34:10Z",
  "lag_seconds": 21,
  "rollup_lag_seconds": 190,
  "segments": { "unscanned": 2, "unrolled": 5, "quarantined": 0 },
  "hours": { "failed": 0 },
  "jobs": { "queued": 1, "running": 1, "dead": 0 },
  "alerts": { "held": 1 },
  "webhook_deliveries": { "pending": 0, "failed": 0 },
  "queue": { "waiting": 0, "oldest_seconds": 0, "dead_letter": 0 }
}
```

| Field                | Meaning                                                                                                                                                                                                                                       |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lag_seconds`        | **The freshness number.** The age of the oldest segment that the fast path has not scanned and that is not quarantined. On AWS, the age of the oldest message in the intake queue when that is older. `0` when nothing waits                  |
| `rollup_lag_seconds` | The age of the oldest segment not yet added by the roll-up                                                                                                                                                                                    |
| `state`              | `ok`: `lag_seconds` under 120 and `rollup_lag_seconds` under 600, the two promises. `behind`: over either. `failing`: `lag_seconds` over 1,800, or anything quarantined, an hour failed for 30 minutes, or a message in the dead-letter queue |
| `queue`              | The intake queue. `null` in the one-process shape                                                                                                                                                                                             |

Per project, the same idea is `data_through` in [`GET /v1/summary`](https://docs.sypnia.com/api/overview-and-authentication), which any `read` token can call and which Sypnia Cloud shows in the app **\[Cloud]**.

**The freshness objective.** `lag_seconds` under 120 and `rollup_lag_seconds` under 600. The engine measures both on every pass and writes them to its log; when either stays over 1,800 seconds it creates the `engine.failing` alert, which goes to your webhook like any other. The objective is a design rule, checked by the tests. A measured percentage will be published here once the engine has run at size.

### `sypnia doctor`

`doctor` tests what the engine depends on, one line each. Two of its checks are about this page:

```text
config    ok    14 settings read, 0 unknown
database  ok    postgres 16.4, schema 12 (up to date), 3 ms
storage   ok    s3://sypnia-shop: put, get, delete, presign
address   ok    https://sypnia.example.com/healthz answered 200
pipeline  ok    last pass 12 s ago, lag 21 s, 0 quarantined, 0 failed hours, 0 dead jobs
queue     ok    intake 0 waiting, dead-letter 0
ai        ok    openrouter, model <model>: answered in 0.9 s
webhook   skip  SYPNIA_ALERT_WEBHOOK_URL is not set (alerts are recorded only)
clock     ok    0.2 s from the database's clock
all checks passed
```

| Check      | Fails when                                                                                                       | It prints                                                                                |
| ---------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `pipeline` | No pass finished in the last 3 minutes; `lag_seconds` over 120; a quarantined segment; a failed hour; a dead job | Each quarantined segment with its error, and the `sypnia aggregate` line that replays it |
| `queue`    | The intake queue or its dead-letter queue cannot be read; the dead-letter queue is not empty                     | How many messages are parked, and the redrive command. Skipped in the one-process shape  |
| `clock`    | The machine's clock is more than 5 seconds from the database's                                                   | Both times                                                                               |

One more number is a health number and not only a log line: **problems symbolicated ÷ problems that had a release and a stack**, over 24 hours. The system this engine comes from once symbolicated nothing for six days and nobody noticed. `doctor` prints it, and [Problem grouping](https://docs.sypnia.com/architecture/problem-grouping#symbolication) explains it.

## What we do not promise

- **Events the engine never accepted.** While ingest is unreachable an SDK holds 500 events and retries; a tab that closes takes its queue with it; the last batch of a page, sent as the page hides, is sent once. A server SDK retries and then drops. There is no storage on the device.
- **Exactly-once delivery of a webhook.** It is at least once. Your receiver gets the same `evt_…` id again when a try is repeated; ignore an id you have handled.
- **The 2 minutes during an outage.** The freshness promises hold while the engine's pieces are up. During and right after a failure, data is late, by as long as the failure lasted plus the catch-up. It is not lost and not wrong.
- **The 2 minutes for a batch whose worker was killed** (AWS). It is delivered again when its visibility timeout ends, up to 15 minutes later.
- **The 10 minutes for an hour whose worker was killed mid-run.** Another worker takes the hour over after 20 minutes.
- **What people wrote, without a backup of Postgres.** Statuses, notes, rules, keys and tokens exist only there. Numbers and problems can be built again from the Bucket for as long as retention keeps the segments; older than that, they cannot.
- **A quarantined segment's events,** until someone replays it. They are safe in the Bucket and they are not in any count.
- **Order between events of different batches.** Timelines are ordered by the events' own times, corrected for the device's clock. Two events a few milliseconds apart on two devices can appear in either order.
- **An alarm when everything is down** in the one-process shape. If the only process stops, nothing inside it can tell you. Watch `/healthz` from somewhere else.
- **A limit on someone else's outage.** If your bucket provider or your Postgres loses data it had confirmed, the engine cannot bring it back.

## What changed from the system this comes from

| There                                                                                                 | Here                                                                                                          | Why                                              |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| A managed delivery stream; a record that failed was written under an errors prefix and nobody read it | A queue with a dead-letter queue, an alarm and a redrive command                                              | What is set aside must be visible and replayable |
| An hour that failed twice was never tried again                                                       | Tried on the next pass, then with growing waits, for 48 hours; an alert at 30 minutes                         | The incident of 2026-09-30                       |
| A corrupt line became a row of empty values                                                           | A line that does not parse is counted and skipped; a segment that cannot be read is quarantined after 5 tries | One bad input must cost only itself              |
| One job for one app: nothing to be fair to                                                            | Turns per project in the fast path, a memory budget per run, a share of the buffer                            | Many projects share one engine                   |
| Health was a log line                                                                                 | `/healthz`, `/v1/engine/status`, `sypnia doctor`, the `engine.failing` alert                                  | Six days without symbolication, unnoticed        |

## Next
