Docs

How the engine is built

The whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.

Applies to Engine At launch

View as Markdown
On this page
  1. The whole system
  2. A request’s life in 9 steps
  3. The design rules
  4. 1. Raw events are files, not rows
  5. 2. Postgres holds only what is read often
  6. 3. Every step can run twice safely
  7. 4. One binary
  8. 5. Every row and every object carries the project
  9. What changed from the system it comes from
  10. The pages

This section is the engine’s design, written before its code. The code implements exactly what these pages say. Read it if you run the engine yourself, if you want to contribute, or if you want to know why a number on your screen can be trusted.

Everything here is the open-source engine Engine. Sypnia Cloud runs this same engine and adds a private part around it: accounts, billing, the app, the GitHub App and a managed model for the agent. Where a page touches one of those, it carries a Cloud badge.

The design is not new. It is the observability system that has run in production inside the first app that uses Sypnia since 2026-09-30, made multi-project, rewritten in Rust and made faster: minutes where the original waits for an hourly job. Each page says what was kept, what changed and why.

The whole system

Figure. The engine on one page. The lit path is the fast path: a failure becomes a problem in under 2 minutes. Everything else is counted by the roll-up within 10.
  1. Your app sends batches of events, every 10 seconds.
  2. Ingest writes them to the bucket as segment objects (NDJSON, gzip), and one ledger row per segment to Postgres.
  3. The fast path reads the new segments every 30 seconds.
  4. It writes problems to Postgres.
  5. The roll-up reads the open hour every 5 minutes. It writes numbers and the session index to Postgres, and per-session files and Parquet back to the bucket.
  6. The agent reads a problem that is new, growing or came back, and writes the sentence.
  7. Alerts are recorded, and sent to your webhook as a signed POST.
  8. The API reads Postgres, and files from the bucket. The MCP server and the Console read through the API.
  9. Your coding agent asks the MCP server. You read the Console or the API.

There are two stores and nothing else:

StoreHoldsWhy there
Bucket (any S3-compatible store)Raw events, per-session files, replay chunks, page snapshots, source maps, ParquetWritten once, read rarely, large. A file costs almost nothing to keep.
Postgres (15 or later)Projects and keys, problems, hourly and daily numbers, the session index, alerts, usageRead on every screen and every MCP call, small, updated in place.

And five moving parts, all inside one binary:

PartJobRuns
IngestChecks the key and each event, writes segment objects to the BucketOn every request from an SDK
Fast pathReads new segments, finds failures, creates and updates problemsEvery 30 seconds
Roll-upCounts everything else, assembles sessions, writes ParquetEvery 5 minutes; Parquet once a day
The agentReads a problem, writes the sentence, the cause and the promptWhen a problem is new, growing or back
AlertsRecords an alert, calls your webhookWhen a rule fires

The API, the MCP server and the Console only read what those parts wrote. They never compute.

A request’s life in 9 steps

The example is the one used across these docs: a shop whose release 4f2a91c renamed cart.total to cart.totalCents. The checkout still reads the old name, so payments go out without an amount.

Figure. The nine steps on a clock. The promise is 2 minutes from the first failure to the problem and its alert.
  1. A customer clicks Pay. At 09:12:04 the page calls POST /api/checkout and passes a trace id. The server calls the payment service, which answers 400: the amount is missing. @sypnia/node records that failed call inside the trace. The page records the click, the failed request and the error it throws.

  2. The SDK batches. The browser SDK keeps events in memory and sends them every 10 seconds, at 50 events, or when the page is hidden. One batch is one small JSON body with a batch_id.

  3. Ingest checks and accepts. Engine It finds the project from the ingest key in a cache, checks the origin, the size and each event, stamps prj and rcv, and appends the lines to an in-memory buffer. It does not query Postgres to do this.

  4. The Bucket gets a segment. Within 2 seconds the buffer becomes one object, and only then does ingest answer the SDK. The events are now safe, whatever happens next.

    Text
    prj_5k2m9x4q7w1z/events/production/dt=2026-03-12/hr=09/20260312T091216Z-7f3a9c.ndjson.gz
  5. The Fast path finds a new failure. Every 30 seconds it takes the segments nobody has scanned yet, reads them and keeps only the failures. It finds a fingerprint it has never seen in this project and environment: 88a44b15205cb153.

  6. A problem is born. One transaction writes the problem, its first occurrence, the person and the session it hit, and marks the segment as scanned. In the example this is 09:13:02, 58 seconds after the click. The budget is 120.

  7. The agent writes the sentence. If you configured a model, the agent builds a context (the failure, the stack with source lines, the trace, the release, the counts), calls the model and checks the answer. The technical title «createPayment answered 400 in POST /api/checkout» becomes «1 person tried to pay and could not. The Pay button does nothing since yesterday’s change.»

  8. Alerts. The transaction of step 6 also wrote an alert, held until the sentence is ready or until 100 seconds have passed since the failure arrived, whichever comes first. Then it is released: recorded, and sent to your webhook. Cloud Sypnia Cloud also sends it by e-mail and shows it in the app.

  9. Your coding agent reads it. The developer asks their agent what is failing. It calls the MCP tool problem with pay-button-does-nothing and gets the sentence, the cause with its evidence, the stack in the original source and the prompt. Cloud On Sypnia Cloud the fix can also arrive as a pull request.

Meanwhile, every 5 minutes, the Roll-up counts the rest: the page view, the click, the rage click that follows, how long POST /api/checkout took. Those numbers are complete within 10 minutes.

The design rules

Five rules explain almost every decision in this section.

Figure. Rule 1 and rule 5 in one picture: raw events stay files, only results become rows, and every one of them carries its project.

1. Raw events are files, not rows

An event is written once and almost never read again. A row in a database pays for indexes, vacuum and memory for ever; a gzip line in a bucket pays a fraction of a cent a year. The system this engine comes from learned it the hard way: its shared database ran out of memory once, and raw events were the one thing that was never in it.

Files also give schema on read. An event keeps every field it arrived with, including fields this version of the engine does not know. A newer SDK never breaks an older engine.

The type that reads a line carries that rule. Notice extra: it catches whatever the structs above it did not name, and it is written back unchanged.

crates/sypnia-core/src/event.rs
/// One line of a segment. The tag is the wire field `type`.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Event {
    SessionStart(Browser<SessionStart>),
    Identify(Browser<Identify>),
    PageView(Browser<PageView>),
    PageLeave(Browser<PageLeave>),
    Click(Browser<Click>),
    RageClick(Browser<RepeatedClick>),
    DeadClick(Browser<RepeatedClick>),
    Vital(Browser<Vital>),
    Request(Browser<Request>),
    Error(Browser<ErrorEvent>),
    Custom(Browser<Custom>),
    FormInvalid(Browser<FormInvalid>),
    ServerRequest(Server<ServerRequest>),
    ServerError(Server<ServerError>),
    /// A `type` this version does not know. Stored and compacted like any
    /// other line, skipped by the parts that cannot read it. Never rejected.
    #[serde(untagged)]
    Unknown(serde_json::Map<String, serde_json::Value>),
}

/// The fields every browser event carries, around the fields of its type.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Browser<T> {
    #[serde(flatten)]
    pub base: Base,              // v, ts, env, ver, person, account, prj, rcv
    pub sid: SessionId,
    pub did: Option<DeviceId>,
    pub url: Option<String>,
    pub route: Option<Route>,
    pub dev: Option<Device>,
    #[serde(flatten)]
    pub body: T,
    /// Schema on read: every field not named above, exactly as it arrived.
    #[serde(flatten)]
    pub extra: serde_json::Map<String, serde_json::Value>,
}

2. Postgres holds only what is read often

Problems, numbers per hour, the list of sessions, alerts. A million events become a few thousand rows. That is why a small Postgres is enough, and why the engine needs no query engine beside it: nothing on your screen is computed from raw events at the moment you ask.

3. Every step can run twice safely

Functions get retried, processes get killed, networks drop answers after the work was done. So every write in the engine is one of three kinds:

  • an overwrite with the same key: a segment, a replay chunk, a per-session file;
  • a replace: the numbers of one hour are deleted and written again, whole;
  • an add guarded by a ledger: a counter is increased in the same transaction that records which segment it came from, so the same segment can never be added twice.

This is the upsert behind a problem. The counters add, the dates widen, the title stays as first seen, and a resolved problem reopens by itself.

crates/sypnia-db/sql/problem_upsert.sql
INSERT INTO problems AS p
       (id, project_id, environment, fingerprint, kind, origin, title, route,
        first_seen, last_seen, first_release, last_release, occurrences, sample)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $11, $12, $13)
ON CONFLICT (project_id, environment, fingerprint) DO UPDATE SET
    occurrences  = p.occurrences + EXCLUDED.occurrences,
    first_seen   = LEAST(p.first_seen, EXCLUDED.first_seen),
    last_seen    = GREATEST(p.last_seen, EXCLUDED.last_seen),
    last_release = COALESCE(EXCLUDED.last_release, p.last_release),
    sample       = COALESCE(p.sample, EXCLUDED.sample),
    -- a resolved problem that happens again comes back; an ignored one stays ignored
    regressed    = p.regressed OR p.status = 'resolved',
    status       = CASE WHEN p.status = 'resolved' THEN 'open' ELSE p.status END
RETURNING p.id, p.status, p.regressed, (xmax = 0) AS is_new;

4. One binary

Ingest, the Fast path, the Roll-up, the agent, the API, the MCP server and the Console are one program, sypnia. On a server it runs as one long process. On AWS the same file runs as three functions. Both build the same Engine and differ only in what wakes it up.

Notice that the engine knows four things only through traits. Each is a piece you can swap: the store, the database, the model, the way an alert leaves.

crates/sypnia/src/engine.rs
/// Everything the engine needs from the outside world.
/// `serve` and the AWS functions build exactly one of these.
pub struct Engine {
    pub config: Arc<Config>,
    pub store: Arc<dyn ObjectStore>,        // S3, R2, MinIO, a folder in tests
    pub db: Arc<dyn Database>,              // Postgres
    pub ai: Option<Arc<dyn AiProvider>>,    // None until you set a model key
    pub sinks: Vec<Arc<dyn AlertSink>>,     // the record, then your webhooks
    pub clock: Arc<dyn Clock>,              // real time, or a recorded hour in tests
}

impl Engine {
    /// One pass of the worker: what `sypnia aggregate` runs once
    /// and `sypnia serve` runs every `processing.interval_seconds`.
    pub async fn pass(&self) -> Result<PassReport, EngineError> {
        let mut report = PassReport::default();
        report.fast_path = fast_path::run(self).await?;   // problems, under 2 minutes
        report.alerts = alerts::release_due(self).await?;
        report.uptime = uptime::run_due(self).await?;
        if rollup::is_due(self).await? {
            report.rollup = Some(rollup::run(self).await?); // numbers, within 10 minutes
        }
        Ok(report)
    }
}

5. Every row and every object carries the project

Every object key starts with the project id. Every table has project_id as the first column of its primary key. There is no shared table of events, no shared prefix and no query that can be written without a project. Deleting a project is deleting one prefix and the rows of one id.

In the code a store or database call cannot be made without a Scope, so forgetting the project is a compile error, not a leak.

crates/sypnia-core/src/ids.rs
/// `prj_` and 12 characters of [0-9a-z]. Cheap to clone, impossible to build from a bare string.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct ProjectId(Arc<str>);

impl TryFrom<String> for ProjectId {
    type Error = InvalidId;
    fn try_from(s: String) -> Result<Self, InvalidId> {
        let ok = s.len() == 16
            && s.starts_with("prj_")
            && s[4..].bytes().all(|b| b.is_ascii_digit() || b.is_ascii_lowercase());
        if ok { Ok(Self(s.into())) } else { Err(InvalidId::Project) }
    }
}

/// Where a piece of data belongs. Every key and every query starts from one.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Scope {
    pub project: ProjectId,
    pub env: Environment,   // "production" when the sender names none
}

What changed from the system it comes from

The originalThe Sypnia engineWhy
ProjectsOne app, two stagesAny number; the project is in every key and rowBrief: multi-project from the first table
FreshnessHourly job, at minute 20Fast path every 30 s, roll-up every 5 minThe 2-minute and 10-minute promises
Reading raw eventsDuckDB inside the jobPlain Rust, streamingNo external query engine; a 68 MB binding and a 1,200 MB ceiling removed
Writing raw eventsAmazon Data Firehose, 60 s bufferThe engine’s own buffer (2 s); a queue on AWSNo dependency outside AWS; a problem cannot wait 60 s in a buffer
Hourly tablesReplace the hourThe sameProven
Counters across hoursAdd, guarded by one run per hourAdd, guarded by a ledger of segmentsRuns are now minutes apart and per project
Duplicated batchesStored twiceStored once: batch_idThe incident of 2026-09-30, when a CORS mistake doubled batches
FingerprintFNV-1a over UTF-16 code unitsIdentical, bit for bitExisting groups must not split
IdentityNumeric user and club idsOpaque person and account idsBrief §10.1
SpansA flat list of database callsGeneric spans with parent, kind and attributesBrief §10.5, OTLP

The pages

  • Data flow

    Every hop with its time budget, what waits and what does not, and how the 2 and 10 minutes are kept.
  • Storage

    The bucket as a tree, the Postgres schema, sizes per million events, and what is never stored.
  • Ingest

    Keys, origins, limits, the buffer, duplicates, and why ingest does not query the database.
  • Aggregation

    The roll-up without a SQL engine: group-bys in memory, histograms, sessions, the ledger, Parquet.
  • Problem grouping

    The fingerprint, worked by hand; source maps; one request, one problem; the states of a problem.
  • The agent's pipeline

    Triggers, the context and its budget, providers, the checked answer, the prompt.
  • Deployment shapes

    The same binary as three AWS functions or one process, what each costs, and the pieces you can swap.
  • The codebase

    The crates, the main traits, the rules of the code, and how a contribution flows.

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close