Docs

What happens when something fails

Every hop of the engine with what can fail in it, where the data waits meanwhile, how many times it is tried, where it goes when it cannot be done, who is told and how it is replayed. Queues, dead-letter queues and health, in both deployment shapes.

Applies to Engine At launch

View as Markdown
On this page
  1. The guarantee
  2. Hop by hop
  3. The failures, one by one
  4. Ingest is down
  5. The Bucket is down
  6. Postgres is down
  7. A poison batch or segment
  8. A slow project
  9. The AI provider fails or is slow
  10. Your webhook receiver is down
  11. A deploy in the middle of a run
  12. The same run starts twice
  13. Clocks that disagree
  14. The queue backs up
  15. Queues and dead-letter queues
  16. On AWS: one queue and its dead-letter queue
  17. On one process: the buffer
  18. Work as rows
  19. Health
  20. GET /healthz
  21. GET /v1/engine/status
  22. sypnia doctor
  23. What we do not promise
  24. What changed from the system this comes from
  25. Next

This page answers one question: is the engine fragile? It goes through every hop an event makes, says what can fail there, where the data waits while it fails, how many times it is tried, where it goes when it cannot be done, who is told, and the exact command that replays it.

The guarantee

Between steps, at least once. Inside a step, safe to repeat. Together: each event is counted once.

No step of the engine trusts that it runs exactly one time. Functions are retried, queues deliver twice, processes are killed after doing the work and before saying so. Instead of pretending otherwise, the engine makes the second run harmless. Three mechanisms do it, and all three are on the pages that precede this one:

MechanismWhat it guardsWhere
batch_id on every stored lineA batch that arrives twice, from an SDK that retried or a queue that delivered twice, is stored and never counted twiceIngest
The ledger: one row per segment, with scanned_at and rolled_atA segment is scanned by one worker, and its counters are added in the same transaction that marks it. Both happen or neither doesAggregation
Upserts and unique keysA problem has one row per fingerprint; an alert has one row per cause (dedupe_key); a job has one open row per problem; an hour’s numbers are replaced wholeProblem grouping, Storage

The duplicate guard is one statement. Notice that it needs no lock and no «have I seen this?» read before the write: the primary key answers, and the ids that were not inserted are the duplicates.

crates/sypnia-db/sql/batches_record.sql
-- Run by the fast path inside the transaction that marks the segment as scanned.
-- $3 = the batch ids the segment carries. Returns the ones that are new.
INSERT INTO ingest_batches (project_id, batch_id, segment)
SELECT $1, batch_id, $2
  FROM unnest($3::text[]) AS t(batch_id)
ON CONFLICT (project_id, batch_id) DO NOTHING
RETURNING batch_id;
-- Every id of $3 that did not come back already belongs to an earlier segment.
-- Those go to segments.skipped, and from then on every reader skips their lines.

The second half of the guarantee is that work is never held only in memory once it was accepted. At every moment an accepted event is in one of four durable places:

PlaceHoldsFor how long
The intake queue Engine AWS shape onlyValidated batches that are not yet in a segmentSeconds normally; up to 4 days
The BucketSegments, replay chunks, per-session filesUntil retention removes them
The ledger in PostgresWhich segments exist, which are scanned, which are rolledWith the segment
A row with a time on itAgent jobs, held alerts, webhook deliveries, hours to roll againUntil done, or marked as given up

The next figure is the data flow of the previous page drawn for this question: every arrow that can fail has its retry loop, every waiting place is marked, and every exit for work that cannot be done is shown.

Diagram to drawFLOW DIAGRAM, left to right, same nodes and names as the «hops» diagram of architecture/data-flow, redrawn to show failure handling. NODES in order: «Your app (SDK)» · «Ingest» · a waiting place drawn as a small tray labelled «Intake queue (AWS) / buffer, 2 s (one process)» · «Bucket» · «Fast path» · «Roll-up» (below the Fast path, both reading from the Bucket) · «Postgres» · «The agent» · «Alerts» · «Your webhook» (dashed, outside the engine) · and on a lower lane «API · MCP server · Console» reading from Postgres and the Bucket. ON EVERY ARROW THAT CAN FAIL draw a small circular retry loop with a short label, exact texts: on SDK → Ingest «retry 1 s, 4 s, 15 s · same batch_id»; on Ingest → queue/Bucket «503: the SDK retries»; on queue → Bucket (worker writes the segment) «redelivered: 15 s, 1 min, 5 min, 15 min»; on Bucket → Fast path «lease 2 min · 5 tries»; on Bucket → Roll-up «next pass, then 1, 5, 15, 30 min»; on Fast path → The agent «3 tries: 5 s, 30 s»; on Alerts → Your webhook «7 tries over 21 hours»; on API → reader «503 + Retry-After». WAITING PLACES drawn as trays with a clock icon, exact labels: under Your app «queue of 500 events in memory»; the intake queue tray «up to 4 days»; under Bucket «segments wait here while Postgres is down»; under Postgres «rows with a time: jobs, held alerts, deliveries». EXITS for work that cannot be done, drawn as dashed arrows going DOWN to a bottom lane titled «Set aside, visible, replayable», with four boxes, exact labels: «Dead-letter queue (AWS) · 14 days» (from the intake queue), «Quarantined segment» (from the Fast path and the Roll-up), «analysis.status = failed» (from The agent), «delivery failed» (from Alerts). From each of the four boxes a thin return arrow back up to its source labelled with the replay: «redrive», «sypnia aggregate --from … --to …», «POST …/analyze», «POST …/retry». The lit element is the bottom lane. A small marker on the Bucket: «accepted events are safe from here on» (on AWS place a second, lighter marker on the intake queue: «on AWS: from here»). Numbered callouts (1) to (9) on the hops in the order of the table that follows.
Figure. The data flow with every retry loop, every place where work waits and every exit for work that cannot be done. Nothing leaves the picture: what is set aside stays visible and can be replayed.

Hop by hop

One row per hop. «Tries» counts the first attempt. The sections after the table go deeper into the rows that need it.

#HopWhat can failWhat happens, and where the data waitsGives up after, and goes toWho is toldHow it is replayedWhat you see meanwhile
1Your app (SDK) → IngestIngest unreachable, 429, 503, a timeoutThe browser SDK keeps the batch and sends it again after 1 s, 4 s and 15 s with the same batch_id, then with the next flush every 10 s. Events wait in its queue: 500 events in memory, the oldest leave first. Server SDKs retry and then dropThe queue of 500 overflows, or the tab closes: those events are lostNobody at once. A gap shows as data_through not advancing; a monitor of yours on /healthz sees it firstIt cannot: the data never arrivedReading keeps working. No new data
2Ingest → the queue or the bufferThe queue refuses the send (AWS). The buffer is full: 32 MiB waiting in all, or 8 MiB of one project (one process)Ingest answers 503 unavailable with Retry-After: 2. It has stored nothing, so the SDK’s retry of hop 1 covers itAs hop 1Ingest’s log line; engine.failing if it lasts 30 minutesNot neededAs hop 1
3The queue or the buffer → BucketThe Bucket refuses the writeOne process: the batches of that segment get 503 and their SDKs retry. AWS: the message stays in the queue and is delivered again after 15 s, 1 min, 5 min and 15 minAWS: 5 deliveries, then the dead-letter queue, kept 14 daysThe dead-letter alarm; the queue-age alarm; engine.failingRedrive: one command, belowLists and problems still read from Postgres. Replays and timelines answer 503
4Ingest or worker → Postgres (the ledger row)Postgres down, no free connectionOne process: the row is kept and retried; the segment is already in the Bucket. AWS: the message is not deleted until the row exists, so the queue redelivers itOne process: never gives up; a restart finds the segment by listing. AWS: as hop 3engine.failing after 30 minutesNothing to do. After a restore: sypnia aggregate --from <time> lists the Bucket againEvents are safe and not yet counted
5Bucket → Fast path → PostgresThe object cannot be read; a line that breaks the reader; the transaction fails; the worker dies mid-scanThe segment’s lease runs out after 2 minutes, or is given back with a wait of 10 s, 30 s, 2 min, 5 min. Another worker takes it. The segment waits in the Bucket, its row in the ledger5 tries, then quarantined: quarantined_at and the error on its ledger rowAn engine.failing alert that names the segment, at oncesypnia aggregate --project <p> --from <t> --to <t> gives it five new triesEverything else of the project, and every other project, flows. The events of that one segment are not counted
6Bucket → Roll-up → PostgresOut of the memory budget, a read error, the commit fails, the worker diesNothing was written: the run is one transaction. The hour is marked failed and tried again on the next pass, then after 1, 5 and 15 minutes, then every 30Never, inside 48 hours. A run that fails 5 times on the same segment quarantines that segment and goes on without itengine.failing after 30 minutes, naming the project and the hourBy itself; or sypnia aggregate --from … --to … for hours older than 48 hoursProblems are on time. Numbers of that project and hour are late
7Fast path → the agent → the AI providerThe provider answers an error, times out (60 s), or returns an answer that fails the checkThe job is a row. It is tried again after 5 s and after 30 s. The alert does not wait: it leaves at its release_by with the technical title3 tries, then the job is dead and the problem shows analysis.status: failed with the reasonThe problem itself, in the API, the MCP server and the ConsolePOST /v1/problems/{problem}/analyze; or by itself when the problem growsThe problem, its counts, its alert and its prompt, without the sentence
8Postgres → AlertsThe worker dies between writing the alert and releasing itThe alert is a row with release_by. Any pass releases what is past its time; releasing is one UPDATE … WHERE released_at IS NULLNever—Not neededThe alert, at most one pass late
9Alerts → your webhookYour receiver is down, slow (10 s) or answers anything but 2xxThe delivery is a row. 7 tries: at once, then after 1 min, 5 min, 30 min, 2 h, 6 h and 12 h, with the same event id7 tries, then the delivery is failed. After 3 days of nothing but failures the webhook is switched offGET /v1/webhooks/{id}/deliveries; an engine.failing alert when the webhook is switched offPOST /v1/webhooks/{webhook_id}/deliveries/{delivery_id}/retry; PATCH the webhook to switch it onThe alert is recorded and shown in the Console and the API all along
10Postgres and Bucket → API, MCP server, ConsolePostgres has no free connection; the Bucket does not answer503 unavailable with Retry-After: 2, at once. The API does not queue—The callerThe caller retriesAn error with a time to wait, never a wrong number
11The schedule → the worker (AWS)A tick is missed, late, or delivered twiceThe next tick, a minute later, does everything the missed one would have: all due work is rowsNeverThe worker alarm if the function itself failsNot neededAt most a minute of extra delay

Three things hold for every row:

  • The sender keeps the work until the receiver has it durably. The SDK keeps the batch until 200; the queue keeps the message until the worker deleted it; the ledger keeps the segment until its transaction committed.
  • Giving up is a state, not a disappearance. A dead-letter queue, a quarantined segment, a dead job and a failed delivery are all things you can list, read the error of, and send again.
  • A failure delays; it does not corrupt. A step that runs twice writes the same result. That is rule 3 of the overview, and the replay tests of the codebase run every step twice to prove it.

The failures, one by one

Ingest is down

SDKs hold what they can (hop 1) and reading is not affected: the API, the MCP server and the Console do not depend on ingest. On AWS «ingest is down» means the function fails or is throttled at its 40 copies; the address answers 503 or 429 and the SDKs retry. In the one-process shape it means the process is down, and the API with it, unless you run the roles apart.

What is lost is what the SDK could not hold: more than 500 events in a tab, a tab that closed, the last batch of a page that was sent with sendBeacon and got no answer. The engine cannot know about events it never received, so this loss shows only as a gap.

The Bucket is down

Ingest answers 503 and nothing is accepted that is not stored, so there is nothing to recover: SDKs retry. On AWS, batches already in the queue wait there. Reading from Postgres keeps working: the problem list, a problem, numbers, alerts. Anything that needs an object answers 503: replays, a session’s timeline, a trace’s detail.

Postgres is down

Receiving continues. Ingest does not query Postgres: it uses its key cache, which stays valid for up to one hour without a refresh. Segments are written; the fast path, the roll-up and the API stop. When Postgres returns, the worker takes everything not scanned, oldest first and one project at a time in turns, and rolls every open hour.

ShapeWhere events wait during the outageIf it lasts
One processIn the Bucket, as segments. Their ledger rows are kept in memory and retried; a worker that starts lists the last two hours of each project and adds any segment the ledger does not knowLonger than one hour: ingest answers 503, because its key cache is too old to trust
AWS functionsIn the intake queue. A message is deleted only when its segment and its ledger row existLonger than about 20 minutes: batches reach their fifth delivery and move to the dead-letter queue, where they wait up to 14 days for one redrive command

What Postgres holds that the Bucket cannot give back is what people wrote: statuses, notes, rules, keys, tokens. That is what its backups are for. Everything that comes from events is rebuilt with sypnia aggregate --from … --to ….

A poison batch or segment

A «poison» input is one that fails every time it is tried: an object cut short, a line that triggers a bug in a reader, a value a table refuses. The danger is not the lost segment. It is a worker that takes it, dies, takes it again, and never reaches the work behind it.

Three layers stop that:

  1. A bad event never becomes a bad segment. Ingest judges each event on its own: it cuts, cleans or drops it and says so in the answer. A line that still does not parse when read is counted (unreadable) and skipped. Neither stops anything.
  2. A bad message does not block the queue (AWS). The worker answers the queue with the list of the messages it could not write, not with one failure for the whole delivery. The good ones are deleted; a bad one comes back alone and, after 5 deliveries, goes to the dead-letter queue.
  3. A bad segment is quarantined. Every lease counts an attempt. A segment that was leased 5 times and never scanned is marked quarantined_at, with the last error, and no reader takes it again.
Diagram to drawLANE DIAGRAM, time flowing left to right, three horizontal lanes. LANE 1 «Segments of prj_5k2m9x4q7w1z (shop)»: five file tiles in a row labelled «…091207Z», «…091216Z», «…091744Z» (this one drawn cracked, in the warning tone), «…091902Z», «…092015Z». LANE 2 «Fast path»: under each healthy tile a small check mark and an arrow down to a «Postgres» strip at the bottom labelled «problems, ledger: scanned_at». Under the cracked tile, five small attempt marks in a row with the waits between them, exact labels: «try 1 · fails», «+10 s · try 2», «+30 s · try 3», «+2 min · try 4», «+5 min · try 5», then a dashed arrow DOWN-RIGHT into a side box labelled «Quarantined · quarantined_at, error: unexpected end of gzip stream». From the side box two outgoing arrows: one to a bell icon labelled «engine.failing alert, names the segment», one curved arrow back to the cracked tile labelled «sypnia aggregate --project shop --from 09:00 --to 10:00 → five new tries». LANE 3 «Other projects»: a row of small tiles of a second project «prj_9c1t4h8n2b6d» all with check marks, and the note «turns are taken per project: nobody waits behind the bad segment». The tiles «…091902Z» and «…092015Z» of lane 1 must visibly be scanned WHILE the cracked tile is still being retried (place their check marks under tries 2 and 3). The lit element is the side box «Quarantined». Numbered callouts: (1) the cracked tile, (2) the five tries, (3) the quarantine box, (4) lane 3.
Figure. A poison segment. It is tried five times with growing waits while everything around it is scanned, then set aside with its error. The object itself is never touched.

The whole decision is the function below. Notice that the attempt was already counted by the lease, so a worker that dies before reaching this code still uses up a try.

crates/sypnia-pipeline/src/fast_path.rs
const MAX_ATTEMPTS: u32 = 5;

/// How long a segment that failed waits before its next try.
fn backoff(attempts: u32) -> Duration {
    match attempts {
        0 | 1 => Duration::from_secs(10),
        2 => Duration::from_secs(30),
        3 => Duration::from_secs(120),
        _ => Duration::from_secs(300),
    }
}

/// Called by `run` when `scan_segment` returned an error. One bad segment never ends the pass.
async fn give_back(engine: &Engine, segment: &Segment, error: &EngineError) -> Result<Outcome, EngineError> {
    if segment.attempts < MAX_ATTEMPTS {
        // Free the lease early, with a wait. Another worker may be the one that succeeds.
        engine.db.release_segment(segment, backoff(segment.attempts), &error.to_string()).await?;
        return Ok(Outcome::Retrying);
    }
    // Out of the way of every reader. The object stays where it is; the hour is rolled without it.
    let mut tx = engine.db.begin().await?;
    tx.quarantine_segment(segment, &error.to_string()).await?;
    alerts::record_engine_failing(&mut tx, Failing::Quarantined(segment), engine.clock.now()).await?;
    tx.commit().await?;
    Ok(Outcome::Quarantined)
}

A quarantined segment is 2 seconds of one project’s events, at most 4 MiB. Its events are not in any count until someone replays it. To do that, usually after upgrading to a version that fixes the reader:

Terminal
sypnia aggregate --project shop --from 2026-03-12T09:00:00Z --to 2026-03-12T10:00:00Z

Processing a range again clears the quarantine of the segments in it, gives each five new tries, and adds to the ledger any segment of the range that is in the Bucket and not in the ledger. Nothing is counted twice: the segments that were already added keep their rolled_at.

A slow project

One project must not make the others late, whatever it sends. Each part has its own guard:

PartWhat a heavy project could takeThe guard
IngestThe bufferOne open segment per project. Of the 32 MiB that may wait in all, one project may hold 8 MiB; past that, its batches get 503, the others do not. Rate limits are per key
Fast pathThe worker’s timeTurns. Each lease goes to the project that was served longest ago, and takes one segment. A project with a backlog of a thousand segments gets one turn in each round, like a project with one
Roll-upMemory and the processing.concurrency runs256 MB per run, enforced; a run past its budget stops and marks the hour failed. Projects are taken least recently rolled first, and a failed hour waits 1, 5, 15, then 30 minutes between tries, so it cannot hold a run for ever
The agentModel calls100 analyses a day per project and 3 per problem; one open job per problem
APIConnectionsA small bounded pool; 503 with Retry-After instead of a queue

The lease is one statement. Notice the order: first by when the project last had a turn, then by age. The index it reads holds only segments that are waiting, so it is small by construction.

crates/sypnia-db/sql/segment_lease.sql
-- $1 = the lease (2 minutes). Returns one segment, or none.
WITH pick AS (
    SELECT s.project_id, s.key
      FROM segments s
      LEFT JOIN project_turns t USING (project_id)
     WHERE s.scanned_at IS NULL
       AND s.quarantined_at IS NULL
       AND (s.leased_until IS NULL OR s.leased_until < now())  -- free, waited, or its worker died
     ORDER BY t.fast_path_at NULLS FIRST, s.created_at
     LIMIT 1
       FOR UPDATE OF s SKIP LOCKED                              -- two workers never get the same one
), turn AS (
    INSERT INTO project_turns (project_id, fast_path_at)
    SELECT project_id, now() FROM pick
    ON CONFLICT (project_id) DO UPDATE SET fast_path_at = EXCLUDED.fast_path_at
)
UPDATE segments s
   SET leased_until = now() + $1,
       attempts     = s.attempts + 1                            -- counted here, before any work
  FROM pick
 WHERE s.project_id = pick.project_id AND s.key = pick.key
RETURNING s.*;

The AI provider fails or is slow

The agent is the only part that waits on a service the engine does not control, so nothing else waits on the agent. The alert is released at its release_by whether or not the sentence exists. A call has 60 seconds (ai.timeout_seconds); a job has 3 tries, after 5 and after 30 seconds; then the job is dead and the problem says analysis.status: failed with the provider’s error. When 5 jobs in a row end that way the agent stops calling for 5 minutes, so an outage of the provider does not spend the worker’s time on calls that cannot succeed; the jobs stay queued.

A failed analysis is tried again by itself when the problem grows or comes back, and at once when you ask: POST /v1/problems/{problem}/analyze.

Your webhook receiver is down

The alert exists before any delivery is attempted: it is a row, shown in the Console and returned by GET /v1/alerts. The delivery is a second row with its own tries (hop 9). A receiver that was down for a day gets the alert on the seventh try, with the same event id, so it can ignore what it already handled. A receiver that was down for longer reads what it missed from GET /v1/alerts, or asks for single deliveries again.

A deploy in the middle of a run

ShapeWhat a deploy doesWhy nothing breaks
One processSIGTERM: requests get 20 seconds, open segments are flushed, the pass finishes its current step, then the process exitsA step that was cut is a lease that runs out or a transaction that never committed. The new process does it again
AWS functionsCalls in flight finish on the old version; new calls start on the new oneThe same, and the queue keeps what a killed call did not delete
BothOld and new versions run side by side for a momentMigrations only add: an older binary keeps working against a newer schema

A worker killed in the middle of a roll-up leaves its hour marked running. Another worker takes that hour over after 20 minutes. This is the one case where numbers can be later than 10 minutes; problems are not affected, because a segment’s lease is 2 minutes.

The same run starts twice

It happens: a schedule that delivers a tick twice, two workers, a retry of a call that had in fact succeeded. Every piece of work has one claim that only one run can win:

WorkThe claim
Scanning a segmentThe lease: FOR UPDATE SKIP LOCKED
Adding a segment’s countersUPDATE segments SET rolled_at … WHERE rolled_at IS NULL; a short count rolls the transaction back
An hour’s numbersReplaced whole; two runs write the same rows
An agent jobThe claim of the jobs table; one open job per problem
Releasing an alertUPDATE … WHERE released_at IS NULL
A webhook deliveryThe delivery row’s lease; your receiver sees the same evt_… id if a try is repeated
The daily jobOne job row per project and day

Clocks that disagree

ClockWhere it is wrongWhat the engine does
A devicets in the past or the futureEach batch carries sent_at. When it differs from the arrival time by more than 2 minutes, the batch’s times are shifted by the difference for ordering. An event older than 7 days is dropped as too_old and counted in the answer
An engine processIts time differs from the database’sLeases, release_by and every «is it due?» use the database’s clock (now() in the statement), not the process’s. sypnia doctor fails its clock check above 5 seconds of difference
An ingest process, by minutesA segment is named into an hour that is already closedThe ledger has a segment of a closed hour not yet rolled; the roll-up opens that hour again. Hours count by arrival, so nothing is lost or doubled

The queue backs up

On AWS, a worker that is slower than what arrives shows in the intake queue before it shows anywhere else. Nothing is lost while it lasts: the queue keeps a message for 4 days. Two things grow, and both are watched: the age of the oldest message (an alarm at 2 minutes) and the engine’s own lag (an engine.failing alert at 30 minutes). SDKs notice nothing, because ingest answers once the batch is in the queue. The first thing to raise is the number of copies of the worker function.

Queues and dead-letter queues

On AWS: one queue and its dead-letter queue

The intake queue stands where the in-memory buffer is in the other shape. It is the only queue of the engine; why there is one and not four is on Deployment shapes. The template creates it with these values, and the reason for each:

SettingValueWhy
KindStandardOrder does not matter and a duplicate is harmless: batch_id
A messageOne validated batch, gzip, at most 256 KiBA larger batch is written to the Bucket by ingest and the message only names it
Message retention4 daysA worker can be broken for a long weekend without loss
Batching window10 seconds, or 500 messages, or 6 MBSegments stay few; a failure waits at most 10 seconds here
Visibility timeout15 minutesIt may not be shorter than the worker’s time limit. The worker shortens it itself for a message it gives back
Deliveries before the dead-letter queue5 (maxReceiveCount)With the waits below, about 20 minutes of trying
Dead-letter queue retention14 days, counted from the first sendTime to look, fix and redrive
EncryptionAt rest, with the service’s keysMessages hold event data
Diagram to drawSTATE-AND-TIME DIAGRAM of ONE message through the intake queue, left to right. Start node «Ingest: SendMessage ok → 200 to the SDK». Then a horizontal track labelled «Intake queue» with the message drawn as a small envelope moving along it through these stations, each a rounded box with exact labels: (1) «Waiting · visible» → arrow «delivered with up to 10 s of other messages» → (2) «In flight · invisible, held by the worker». From (2) THREE outgoing branches. BRANCH A (accent colour, the happy path) → «Segment written + ledger row» → «Deleted» (end, check mark). BRANCH B (warning tone) labelled «the worker reports it as failed» → a box «Visible again after the wait» with a small table inside: «delivery 1 → 15 s», «2 → 1 min», «3 → 5 min», «4 → 15 min» → arrow back to station (1), labelled «delivered again: receive count +1». BRANCH C (dashed) labelled «the worker was killed: no answer» → «Visible again after 15 min (visibility timeout)» → back to station (1). From station (1) a downward dashed arrow labelled «receive count would pass 5» into a second track below labelled «Dead-letter queue · kept 14 days», where the envelope sits in a box «Parked · alarm: DeadLetterNotEmpty». From that box a curved arrow back UP to station (1) of the intake queue labelled «redrive: aws sqs start-message-move-task» and the note «arrives as a new message; duplicates are skipped by batch_id». The lit element is the dead-letter track and its redrive arrow. Under the whole diagram a thin time ruler for branch B: «0 · 15 s · 1 min 15 s · 6 min 15 s · 21 min 15 s → dead-letter queue». Numbered callouts (1) waiting, (2) in flight, (3) deleted, (4) the wait table, (5) dead-letter queue, (6) redrive.
Figure. The life of a message. Delivered, written and deleted is the normal path. A message that fails comes back after growing waits; after five deliveries it is parked in the dead-letter queue until someone redrives it.

What the worker tells the queue. A delivery brings many messages. The worker writes their segments and ledger rows, then answers with the ids of the messages it could not finish. Only those come back. Notice that an error never escapes the handler: a handler that fails as a whole would send the good messages back with the bad one.

crates/sypnia/src/lambda.rs
/// The queue case of `worker_event`. The answer names the messages to deliver again.
async fn intake_batch(engine: &Engine, batch: SqsEvent) -> SqsBatchResponse {
    let mut failed = Vec::new();
    // One group per project and environment: the messages that will share a segment.
    for group in intake::group(batch.records) {
        match intake::write_segment(engine, &group).await {
            // The object exists and so does its ledger row. Scanning may fail: the pass picks it up.
            Ok(segment) => {
                let _ = fast_path::scan(engine, std::slice::from_ref(&segment)).await;
            }
            Err(error) => {
                tracing::warn!(%error, scope = %group.scope, "segment not written; messages go back");
                for message in &group.messages {
                    // 15 s, 1 min, 5 min, 15 min, by how many times it was delivered.
                    queue::delay(engine, message, redelivery_wait(message.receive_count)).await;
                    failed.push(BatchItemFailure { item_identifier: message.id.clone() });
                }
            }
        }
    }
    SqsBatchResponse { batch_item_failures: failed }
}

A message that the worker gives back is seen again after its wait. A message whose worker was killed, out of memory or out of time, is seen again when the visibility timeout ends: up to 15 minutes later. Those events are late, not lost, and they are the one case on AWS where a problem can miss its 2 minutes without anything being broken for good.

Redrive

Messages in the dead-letter queue stay there for 14 days. Nothing reads them by itself: a batch that could not be written five times is either a bug to look at or an outage to wait out, and neither is helped by trying a sixth time at once. When the cause is gone, send them back:

Terminal
# How many are parked
aws sqs get-queue-attributes --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --attribute-names ApproximateNumberOfMessages

# Look at one without removing it
aws sqs receive-message --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --max-number-of-messages 1 --visibility-timeout 0

# Send all of them back to the intake queue
aws sqs start-message-move-task --source-arn "$(aws sqs get-queue-attributes \
  --queue-url "$SYPNIA_INTAKE_DEAD_LETTER_QUEUE_URL" \
  --attribute-names QueueArn --query Attributes.QueueArn --output text)"

Redriven messages arrive as new ones and take the normal path. If some of them had in fact been written before, their batch ids are already known and their lines are skipped.

Alarms

The template creates three alarms. They are CloudWatch alarms because they must work when the engine itself does not; point them at a notification of yours.

AlarmFires whenIt means
DeadLetterNotEmptyThe dead-letter queue has at least 1 message, for 1 minuteBatches were set aside. Look at one, fix, redrive
IntakeBacklogThe oldest message in the intake queue is older than 120 seconds, for 3 minutes in a rowThe worker is not keeping up, or is not running
WorkerFailingThe worker function ended in an error in 3 of the last 5 minutesA bug, a missing permission, Postgres or the Bucket unreachable

On one process: the buffer

sypnia serve has no queue, because it does not need one: the process is the memory.

Where a batch waitsIn the open segment of its project, in memory, for at most 2 seconds
When the SDK is answeredAfter the segment is in the Bucket. Not before
What a crash losesNothing that was answered 200. The batches in the open segments, at most 2 seconds of traffic, had not been answered: their SDKs get a network error and send them again
What a crash can still costThose same batches, if the process stays down longer than the SDKs can hold them (hop 1)
A ledger row not yet writtenFound again by listing: on start, a worker lists the last two hours of each project and adds what the ledger does not know. The daily job checks the day before as well
A graceful stopOpen segments are flushed before the process exits

This is the honest difference between the two shapes. On AWS the durable point is the queue, a managed service that keeps working while functions fail. On one process the durable point is the Bucket, and between the SDK and the Bucket there is one process and 2 seconds. Run two ingest processes behind a load balancer if those seconds matter to you: an SDK’s retry then reaches the other one.

Work as rows

Everything after the Bucket that must happen later is a row in Postgres with a time on it, not a message in a queue. Three kinds of work use the pattern:

WorkTableTries and waitsGiven upSeen inReplayed with
The agent’s analysis, the daily job, weekly suggestionsjobs3: after 5 s and 30 sstatus = 'dead', last_errorThe problem: analysis.status: failed. GET /v1/engine/status: jobs.deadPOST /v1/problems/{problem}/analyze
Releasing an alertalerts (release_by, released_at)Every pass, until doneNeverGET /v1/alerts—
A webhook deliverywebhook_deliveries7: after 1 min, 5 min, 30 min, 2 h, 6 h, 12 hstatus = 'failed', errorGET /v1/webhooks/{id}/deliveries?status=failedPOST /v1/webhooks/{webhook_id}/deliveries/{delivery_id}/retry
An hour to roll againhours (status, attempts, next_attempt_at)Next pass, then 1, 5, 15, 30 minNever inside 48 hoursGET /v1/engine/status: hours.failedsypnia aggregate --from … --to …

Why rows and not more queues:

  • The row is written in the transaction that causes it. A problem is born and its job exists, or neither. A queue message sent after a commit can be lost between the two; one sent before can announce work that was rolled back.
  • A row can be read. «What is waiting, since when, how many times was it tried, what did it say?» is a query, and the API answers it. A queue can only be counted.
  • Nothing more to deploy. The one-process shape has no queue service at all, and the same code runs in both.

The table and the claim. Notice the second condition of the claim: a running job whose lease ran out belonged to a worker that died, and is taken over.

migrations/0006_jobs.sql
CREATE TABLE jobs (
    id              text PRIMARY KEY,                    -- job_…
    project_id      text NOT NULL REFERENCES projects ON DELETE CASCADE,
    kind            text NOT NULL CHECK (kind IN ('analysis', 'daily', 'suggestions')),
    subject         text NOT NULL,                       -- the problem id, the day, the week
    status          text NOT NULL DEFAULT 'queued'
                    CHECK (status IN ('queued', 'running', 'done', 'dead')),
    again           boolean NOT NULL DEFAULT false,      -- a trigger arrived while it ran: run once more
    attempts        integer NOT NULL DEFAULT 0,
    next_attempt_at timestamptz NOT NULL DEFAULT now(),
    leased_until    timestamptz,
    last_error      text,
    created_at      timestamptz NOT NULL DEFAULT now(),
    finished_at     timestamptz
);
-- One open job per subject: a second trigger sets `again` instead of inserting.
CREATE UNIQUE INDEX jobs_one_open ON jobs (project_id, kind, subject)
    WHERE status IN ('queued', 'running');
CREATE INDEX jobs_due ON jobs (next_attempt_at) WHERE status = 'queued';
crates/sypnia-db/sql/job_claim.sql
-- $1 = kind, $2 = the lease. Returns one job, or none.
UPDATE jobs j
   SET status = 'running', attempts = j.attempts + 1, leased_until = now() + $2
 WHERE j.id = (
        SELECT id FROM jobs
         WHERE kind = $1
           AND ((status = 'queued'  AND next_attempt_at <= now())
             OR (status = 'running' AND leased_until < now()))   -- its worker died
         ORDER BY next_attempt_at
         LIMIT 1
           FOR UPDATE SKIP LOCKED)
RETURNING j.*;

A job that fails goes back to queued with a later next_attempt_at, or to dead on its third failure. On AWS the worker calls itself to run a job at once; that call is only a shortcut. If it is lost, the minute tick finds the row. Lambda’s own retries of that call are switched off, because the row is the retry.

Health

Three things answer «is it working?», from the cheapest to the most complete.

GET /healthz

Public, no credential. It answers 200 when the process is up and serving, and it does not test Postgres or the Bucket on purpose: a database outage must not take ingest out of a load balancer, because ingest works without the database.

JSON
{ "status": "ok", "version": "1.0.0", "role": "all" }

Watch it from outside the machine the engine runs on. An engine that is down cannot raise its own alert.

GET /v1/engine/status

With the root token Engine. The state of the pipeline, across all projects. It is what the pipeline check of sypnia doctor prints and what the engine.failing alert is decided from.

JSON
{
  "state": "ok",
  "engine_version": "1.0.0",
  "last_pass_at": "2026-03-12T10:34:10Z",
  "lag_seconds": 21,
  "rollup_lag_seconds": 190,
  "segments": { "unscanned": 2, "unrolled": 5, "quarantined": 0 },
  "hours": { "failed": 0 },
  "jobs": { "queued": 1, "running": 1, "dead": 0 },
  "alerts": { "held": 1 },
  "webhook_deliveries": { "pending": 0, "failed": 0 },
  "queue": { "waiting": 0, "oldest_seconds": 0, "dead_letter": 0 }
}
FieldMeaning
lag_secondsThe freshness number. The age of the oldest segment that the fast path has not scanned and that is not quarantined. On AWS, the age of the oldest message in the intake queue when that is older. 0 when nothing waits
rollup_lag_secondsThe age of the oldest segment not yet added by the roll-up
stateok: lag_seconds under 120 and rollup_lag_seconds under 600, the two promises. behind: over either. failing: lag_seconds over 1,800, or anything quarantined, an hour failed for 30 minutes, or a message in the dead-letter queue
queueThe intake queue. null in the one-process shape

Per project, the same idea is data_through in GET /v1/summary, which any read token can call and which Sypnia Cloud shows in the app Cloud.

The freshness objective. lag_seconds under 120 and rollup_lag_seconds under 600. The engine measures both on every pass and writes them to its log; when either stays over 1,800 seconds it creates the engine.failing alert, which goes to your webhook like any other. The objective is a design rule, checked by the tests. A measured percentage will be published here once the engine has run at size.

sypnia doctor

doctor tests what the engine depends on, one line each. Two of its checks are about this page:

Text
config    ok    14 settings read, 0 unknown
database  ok    postgres 16.4, schema 12 (up to date), 3 ms
storage   ok    s3://sypnia-shop: put, get, delete, presign
address   ok    https://sypnia.example.com/healthz answered 200
pipeline  ok    last pass 12 s ago, lag 21 s, 0 quarantined, 0 failed hours, 0 dead jobs
queue     ok    intake 0 waiting, dead-letter 0
ai        ok    openrouter, model <model>: answered in 0.9 s
webhook   skip  SYPNIA_ALERT_WEBHOOK_URL is not set (alerts are recorded only)
clock     ok    0.2 s from the database's clock
all checks passed
CheckFails whenIt prints
pipelineNo pass finished in the last 3 minutes; lag_seconds over 120; a quarantined segment; a failed hour; a dead jobEach quarantined segment with its error, and the sypnia aggregate line that replays it
queueThe intake queue or its dead-letter queue cannot be read; the dead-letter queue is not emptyHow many messages are parked, and the redrive command. Skipped in the one-process shape
clockThe machine’s clock is more than 5 seconds from the database’sBoth times

One more number is a health number and not only a log line: problems symbolicated ÷ problems that had a release and a stack, over 24 hours. The system this engine comes from once symbolicated nothing for six days and nobody noticed. doctor prints it, and Problem grouping explains it.

What we do not promise

  • Events the engine never accepted. While ingest is unreachable an SDK holds 500 events and retries; a tab that closes takes its queue with it; the last batch of a page, sent as the page hides, is sent once. A server SDK retries and then drops. There is no storage on the device.
  • Exactly-once delivery of a webhook. It is at least once. Your receiver gets the same evt_… id again when a try is repeated; ignore an id you have handled.
  • The 2 minutes during an outage. The freshness promises hold while the engine’s pieces are up. During and right after a failure, data is late, by as long as the failure lasted plus the catch-up. It is not lost and not wrong.
  • The 2 minutes for a batch whose worker was killed (AWS). It is delivered again when its visibility timeout ends, up to 15 minutes later.
  • The 10 minutes for an hour whose worker was killed mid-run. Another worker takes the hour over after 20 minutes.
  • What people wrote, without a backup of Postgres. Statuses, notes, rules, keys and tokens exist only there. Numbers and problems can be built again from the Bucket for as long as retention keeps the segments; older than that, they cannot.
  • A quarantined segment’s events, until someone replays it. They are safe in the Bucket and they are not in any count.
  • Order between events of different batches. Timelines are ordered by the events’ own times, corrected for the device’s clock. Two events a few milliseconds apart on two devices can appear in either order.
  • An alarm when everything is down in the one-process shape. If the only process stops, nothing inside it can tell you. Watch /healthz from somewhere else.
  • A limit on someone else’s outage. If your bucket provider or your Postgres loses data it had confirmed, the engine cannot bring it back.

What changed from the system this comes from

ThereHereWhy
A managed delivery stream; a record that failed was written under an errors prefix and nobody read itA queue with a dead-letter queue, an alarm and a redrive commandWhat is set aside must be visible and replayable
An hour that failed twice was never tried againTried on the next pass, then with growing waits, for 48 hours; an alert at 30 minutesThe incident of 2026-09-30
A corrupt line became a row of empty valuesA line that does not parse is counted and skipped; a segment that cannot be read is quarantined after 5 triesOne bad input must cost only itself
One job for one app: nothing to be fair toTurns per project in the fast path, a memory budget per run, a share of the bufferMany projects share one engine
Health was a log line/healthz, /v1/engine/status, sypnia doctor, the engine.failing alertSix days without symbolication, unnoticed

Next

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close