How the engine is built
The whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.
This section is the engine’s design, written before its code. The code implements exactly what these pages say. Read it if you run the engine yourself, if you want to contribute, or if you want to know why a number on your screen can be trusted.
Everything here is the open-source engine Engine. Sypnia Cloud runs this same engine and adds a private part around it: accounts, billing, the app, the GitHub App and a managed model for the agent. Where a page touches one of those, it carries a Cloud badge.
The design is not new. It is the observability system that has run in production inside the first app that uses Sypnia since 2026-09-30, made multi-project, rewritten in Rust and made faster: minutes where the original waits for an hourly job. Each page says what was kept, what changed and why.
The whole system
- Your app sends batches of events, every 10 seconds.
- Ingest writes them to the bucket as segment objects (NDJSON, gzip), and one ledger row per segment to Postgres.
- The fast path reads the new segments every 30 seconds.
- It writes problems to Postgres.
- The roll-up reads the open hour every 5 minutes. It writes numbers and the session index to Postgres, and per-session files and Parquet back to the bucket.
- The agent reads a problem that is new, growing or came back, and writes the sentence.
- Alerts are recorded, and sent to your webhook as a signed POST.
- The API reads Postgres, and files from the bucket. The MCP server and the Console read through the API.
- Your coding agent asks the MCP server. You read the Console or the API.
There are two stores and nothing else:
| Store | Holds | Why there |
|---|---|---|
| Bucket (any S3-compatible store) | Raw events, per-session files, replay chunks, page snapshots, source maps, Parquet | Written once, read rarely, large. A file costs almost nothing to keep. |
| Postgres (15 or later) | Projects and keys, problems, hourly and daily numbers, the session index, alerts, usage | Read on every screen and every MCP call, small, updated in place. |
And five moving parts, all inside one binary:
| Part | Job | Runs |
|---|---|---|
| Ingest | Checks the key and each event, writes segment objects to the Bucket | On every request from an SDK |
| Fast path | Reads new segments, finds failures, creates and updates problems | Every 30 seconds |
| Roll-up | Counts everything else, assembles sessions, writes Parquet | Every 5 minutes; Parquet once a day |
| The agent | Reads a problem, writes the sentence, the cause and the prompt | When a problem is new, growing or back |
| Alerts | Records an alert, calls your webhook | When a rule fires |
The API, the MCP server and the Console only read what those parts wrote. They never compute.
A request’s life in 9 steps
The example is the one used across these docs: a shop whose release 4f2a91c renamed cart.total to cart.totalCents. The checkout still reads the old name, so payments go out without an amount.
-
A customer clicks Pay. At 09:12:04 the page calls
POST /api/checkoutand passes a trace id. The server calls the payment service, which answers400: the amount is missing.@sypnia/noderecords that failed call inside the trace. The page records the click, the failed request and the error it throws. -
The SDK batches. The browser SDK keeps events in memory and sends them every 10 seconds, at 50 events, or when the page is hidden. One batch is one small JSON body with a
batch_id. -
Ingest checks and accepts. Engine It finds the project from the ingest key in a cache, checks the origin, the size and each event, stamps
prjandrcv, and appends the lines to an in-memory buffer. It does not query Postgres to do this. -
The Bucket gets a segment. Within 2 seconds the buffer becomes one object, and only then does ingest answer the SDK. The events are now safe, whatever happens next.
Text prj_5k2m9x4q7w1z/events/production/dt=2026-03-12/hr=09/20260312T091216Z-7f3a9c.ndjson.gz -
The Fast path finds a new failure. Every 30 seconds it takes the segments nobody has scanned yet, reads them and keeps only the failures. It finds a fingerprint it has never seen in this project and environment:
88a44b15205cb153. -
A problem is born. One transaction writes the problem, its first occurrence, the person and the session it hit, and marks the segment as scanned. In the example this is 09:13:02, 58 seconds after the click. The budget is 120.
-
The agent writes the sentence. If you configured a model, the agent builds a context (the failure, the stack with source lines, the trace, the release, the counts), calls the model and checks the answer. The technical title «createPayment answered 400 in POST /api/checkout» becomes «1 person tried to pay and could not. The Pay button does nothing since yesterday’s change.»
-
Alerts. The transaction of step 6 also wrote an alert, held until the sentence is ready or until 100 seconds have passed since the failure arrived, whichever comes first. Then it is released: recorded, and sent to your webhook. Cloud Sypnia Cloud also sends it by e-mail and shows it in the app.
-
Your coding agent reads it. The developer asks their agent what is failing. It calls the MCP tool
problemwithpay-button-does-nothingand gets the sentence, the cause with its evidence, the stack in the original source and the prompt. Cloud On Sypnia Cloud the fix can also arrive as a pull request.
Meanwhile, every 5 minutes, the Roll-up counts the rest: the page view, the click, the rage click that follows, how long POST /api/checkout took. Those numbers are complete within 10 minutes.
The design rules
Five rules explain almost every decision in this section.
1. Raw events are files, not rows
An event is written once and almost never read again. A row in a database pays for indexes, vacuum and memory for ever; a gzip line in a bucket pays a fraction of a cent a year. The system this engine comes from learned it the hard way: its shared database ran out of memory once, and raw events were the one thing that was never in it.
Files also give schema on read. An event keeps every field it arrived with, including fields this version of the engine does not know. A newer SDK never breaks an older engine.
The type that reads a line carries that rule. Notice extra: it catches whatever the structs above it did not name, and it is written back unchanged.
/// One line of a segment. The tag is the wire field `type`.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Event {
SessionStart(Browser<SessionStart>),
Identify(Browser<Identify>),
PageView(Browser<PageView>),
PageLeave(Browser<PageLeave>),
Click(Browser<Click>),
RageClick(Browser<RepeatedClick>),
DeadClick(Browser<RepeatedClick>),
Vital(Browser<Vital>),
Request(Browser<Request>),
Error(Browser<ErrorEvent>),
Custom(Browser<Custom>),
FormInvalid(Browser<FormInvalid>),
ServerRequest(Server<ServerRequest>),
ServerError(Server<ServerError>),
/// A `type` this version does not know. Stored and compacted like any
/// other line, skipped by the parts that cannot read it. Never rejected.
#[serde(untagged)]
Unknown(serde_json::Map<String, serde_json::Value>),
}
/// The fields every browser event carries, around the fields of its type.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Browser<T> {
#[serde(flatten)]
pub base: Base, // v, ts, env, ver, person, account, prj, rcv
pub sid: SessionId,
pub did: Option<DeviceId>,
pub url: Option<String>,
pub route: Option<Route>,
pub dev: Option<Device>,
#[serde(flatten)]
pub body: T,
/// Schema on read: every field not named above, exactly as it arrived.
#[serde(flatten)]
pub extra: serde_json::Map<String, serde_json::Value>,
}2. Postgres holds only what is read often
Problems, numbers per hour, the list of sessions, alerts. A million events become a few thousand rows. That is why a small Postgres is enough, and why the engine needs no query engine beside it: nothing on your screen is computed from raw events at the moment you ask.
3. Every step can run twice safely
Functions get retried, processes get killed, networks drop answers after the work was done. So every write in the engine is one of three kinds:
- an overwrite with the same key: a segment, a replay chunk, a per-session file;
- a replace: the numbers of one hour are deleted and written again, whole;
- an add guarded by a ledger: a counter is increased in the same transaction that records which segment it came from, so the same segment can never be added twice.
This is the upsert behind a problem. The counters add, the dates widen, the title stays as first seen, and a resolved problem reopens by itself.
INSERT INTO problems AS p
(id, project_id, environment, fingerprint, kind, origin, title, route,
first_seen, last_seen, first_release, last_release, occurrences, sample)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $11, $12, $13)
ON CONFLICT (project_id, environment, fingerprint) DO UPDATE SET
occurrences = p.occurrences + EXCLUDED.occurrences,
first_seen = LEAST(p.first_seen, EXCLUDED.first_seen),
last_seen = GREATEST(p.last_seen, EXCLUDED.last_seen),
last_release = COALESCE(EXCLUDED.last_release, p.last_release),
sample = COALESCE(p.sample, EXCLUDED.sample),
-- a resolved problem that happens again comes back; an ignored one stays ignored
regressed = p.regressed OR p.status = 'resolved',
status = CASE WHEN p.status = 'resolved' THEN 'open' ELSE p.status END
RETURNING p.id, p.status, p.regressed, (xmax = 0) AS is_new;4. One binary
Ingest, the Fast path, the Roll-up, the agent, the API, the MCP server and the Console are one program, sypnia. On a server it runs as one long process. On AWS the same file runs as three functions. Both build the same Engine and differ only in what wakes it up.
Notice that the engine knows four things only through traits. Each is a piece you can swap: the store, the database, the model, the way an alert leaves.
/// Everything the engine needs from the outside world.
/// `serve` and the AWS functions build exactly one of these.
pub struct Engine {
pub config: Arc<Config>,
pub store: Arc<dyn ObjectStore>, // S3, R2, MinIO, a folder in tests
pub db: Arc<dyn Database>, // Postgres
pub ai: Option<Arc<dyn AiProvider>>, // None until you set a model key
pub sinks: Vec<Arc<dyn AlertSink>>, // the record, then your webhooks
pub clock: Arc<dyn Clock>, // real time, or a recorded hour in tests
}
impl Engine {
/// One pass of the worker: what `sypnia aggregate` runs once
/// and `sypnia serve` runs every `processing.interval_seconds`.
pub async fn pass(&self) -> Result<PassReport, EngineError> {
let mut report = PassReport::default();
report.fast_path = fast_path::run(self).await?; // problems, under 2 minutes
report.alerts = alerts::release_due(self).await?;
report.uptime = uptime::run_due(self).await?;
if rollup::is_due(self).await? {
report.rollup = Some(rollup::run(self).await?); // numbers, within 10 minutes
}
Ok(report)
}
}5. Every row and every object carries the project
Every object key starts with the project id. Every table has project_id as the first column of its primary key. There is no shared table of events, no shared prefix and no query that can be written without a project. Deleting a project is deleting one prefix and the rows of one id.
In the code a store or database call cannot be made without a Scope, so forgetting the project is a compile error, not a leak.
/// `prj_` and 12 characters of [0-9a-z]. Cheap to clone, impossible to build from a bare string.
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(try_from = "String", into = "String")]
pub struct ProjectId(Arc<str>);
impl TryFrom<String> for ProjectId {
type Error = InvalidId;
fn try_from(s: String) -> Result<Self, InvalidId> {
let ok = s.len() == 16
&& s.starts_with("prj_")
&& s[4..].bytes().all(|b| b.is_ascii_digit() || b.is_ascii_lowercase());
if ok { Ok(Self(s.into())) } else { Err(InvalidId::Project) }
}
}
/// Where a piece of data belongs. Every key and every query starts from one.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Scope {
pub project: ProjectId,
pub env: Environment, // "production" when the sender names none
}What changed from the system it comes from
| The original | The Sypnia engine | Why | |
|---|---|---|---|
| Projects | One app, two stages | Any number; the project is in every key and row | Brief: multi-project from the first table |
| Freshness | Hourly job, at minute 20 | Fast path every 30 s, roll-up every 5 min | The 2-minute and 10-minute promises |
| Reading raw events | DuckDB inside the job | Plain Rust, streaming | No external query engine; a 68 MB binding and a 1,200 MB ceiling removed |
| Writing raw events | Amazon Data Firehose, 60 s buffer | The engine’s own buffer (2 s); a queue on AWS | No dependency outside AWS; a problem cannot wait 60 s in a buffer |
| Hourly tables | Replace the hour | The same | Proven |
| Counters across hours | Add, guarded by one run per hour | Add, guarded by a ledger of segments | Runs are now minutes apart and per project |
| Duplicated batches | Stored twice | Stored once: batch_id | The incident of 2026-09-30, when a CORS mistake doubled batches |
| Fingerprint | FNV-1a over UTF-16 code units | Identical, bit for bit | Existing groups must not split |
| Identity | Numeric user and club ids | Opaque person and account ids | Brief §10.1 |
| Spans | A flat list of database calls | Generic spans with parent, kind and attributes | Brief §10.5, OTLP |
The pages
- Every hop with its time budget, what waits and what does not, and how the 2 and 10 minutes are kept.
- The bucket as a tree, the Postgres schema, sizes per million events, and what is never stored.
- Keys, origins, limits, the buffer, duplicates, and why ingest does not query the database.
- The roll-up without a SQL engine: group-bys in memory, histograms, sessions, the ledger, Parquet.
- The fingerprint, worked by hand; source maps; one request, one problem; the states of a problem.
- Triggers, the context and its budget, providers, the checked answer, the prompt.
- The same binary as three AWS functions or one process, what each costs, and the pieces you can swap.
- The crates, the main traits, the rules of the code, and how a contribution flows.