Docs

The codebase

The engine's repository, crate by crate, with the main traits and types, the rules every line of code follows, the tests that replay recorded hours, how to run it on your machine and how a contribution flows.

Applies to Engine At launch

View as Markdown
On this page
  1. The workspace
  2. The main traits and types
  3. The rules of the code
  4. Tests that replay recorded hours
  5. Run it on your machine
  6. Start Postgres and a bucket
  7. Point the engine at them
  8. Create the tables
  9. Run the engine
  10. Create a project and send the example
  11. See the problem
  12. How a contribution flows
  13. Where to start reading
  14. Next

This page is a map of the engine’s source code, for people who want to read it, run it or change it. It is the last page of the section on purpose: everything it names was explained on an earlier one.

The workspace

One Cargo workspace, eleven crates, one binary.

Diagram to drawDEPENDENCY GRAPH, drawn in four horizontal layers, top to bottom, arrows pointing DOWN from a crate to what it depends on. LAYER 1 (top), one node, the lit element: «sypnia (the binary)» with the sub-label «CLI · serve · lambda · Engine». LAYER 2, five nodes side by side: «sypnia-ingest» (sub-label «gates, buffer, OTLP»), «sypnia-pipeline» (sub-label «fast path, roll-up, sessions, Parquet»), «sypnia-agent» (sub-label «context, providers, check, prompt»), «sypnia-alerts» (sub-label «rules, release, webhook»), «sypnia-api» (sub-label «HTTP API, MCP server, Console»). LAYER 3, three nodes: «sypnia-store» (sub-label «ObjectStore, keys»), «sypnia-db» (sub-label «Database, migrations, upserts»), «sypnia-fingerprint» (sub-label «hash64, frames»). LAYER 4 (bottom), one wide node: «sypnia-core» (sub-label «Event, ids, Scope, Histogram, ports, errors · no I/O»). ARROWS: the binary → each of the five layer-2 crates; sypnia-ingest → sypnia-store, sypnia-db, sypnia-fingerprint; sypnia-pipeline → sypnia-store, sypnia-db, sypnia-fingerprint; sypnia-agent → sypnia-store, sypnia-db; sypnia-alerts → sypnia-db; sypnia-api → sypnia-store, sypnia-db, sypnia-agent (label on this one arrow: «the prompt»); every layer-3 crate → sypnia-core. To the right, outside the layers, a dashed node «sypnia-testkit» (sub-label «recorded hours, in-memory adapters, fixed clock») with dashed arrows to sypnia-core, sypnia-store and sypnia-db, and a note «dev-dependency only». A rule written under the diagram: «arrows only point down: no layer-2 crate depends on another, except api → agent for the prompt». Numbered callouts: (1) the binary, (2) layer 2, (3) layer 3, (4) sypnia-core.
Figure. The crates and what each depends on. The parts of the engine do not know each other; they meet in the binary.
Text
sypnia/
├── Cargo.toml                 the workspace, the shared lints, the pinned dependencies
├── crates/
│   ├── sypnia-core/           (4) types and traits; no network, no disk, no clock of its own
│   ├── sypnia-fingerprint/    (3) hash64, message and frame rules, per-platform lists
│   ├── sypnia-store/          (3) ObjectStore over S3-compatible stores and over a folder; Keys
│   ├── sypnia-db/             (3) Database over Postgres; every SQL statement of the engine
│   │   └── migrations/        numbered, additive, forward-only
│   ├── sypnia-ingest/         (2) the seven gates, the buffer, replay and snapshot uploads, OTLP mapping
│   ├── sypnia-pipeline/       (2) fast path, problems, roll-up folds, sessions, symbolication, Parquet, retention
│   ├── sypnia-agent/          (2) context, redaction, providers, the check, the prompt template
│   ├── sypnia-alerts/         (2) alert rules, release, sinks, webhook deliveries, uptime checks
│   ├── sypnia-api/            (2) the HTTP API, the MCP server, the Console's files
│   ├── sypnia/                (1) main, CLI, serve, lambda, Engine
│   └── sypnia-testkit/        recorded hours and in-memory adapters, for tests only
├── spec/                      openapi.yaml, events.schema.json, mcp.json, config.yaml, cli.yaml
├── vectors/                   fingerprint and histogram test vectors, shared with the SDKs
├── deploy/                    sypnia-aws.yaml, docker-compose.yaml, Dockerfile
└── console/                   the Console: plain files, embedded in the binary at build time
CrateWhat it doesExplained on
sypnia-coreEvent, ProjectId, Scope, Histogram, the quota function, the four port traits, the error types. It does no I/O, so it compiles anywhere and its tests need nothingOverview
sypnia-fingerprinthash64 over UTF-16 code units, message normalization, frame parsing, the lists of noisy frames and build-hash rules per platformProblem grouping
sypnia-storeObjectStore implemented over the object_store crate, and over a folder. Keys, the only place that knows the bucket layoutStorage
sypnia-dbDatabase implemented with sqlx. Every statement lives here as a .sql file checked at build time. The ledger. MigrationsStorage, Aggregation
sypnia-ingestThe axum handlers for events, replay chunks, snapshots, config and OTLP. ProjectCache. Intake and its two implementations: the buffer and the queueIngest
sypnia-pipelinefast_path, problems::transition, the folds, session assembly, symbolication, compaction, retentionData flow, Aggregation
sypnia-agentbuild_context, redact, AiProvider with ChatCompletions and AnthropicMessages, check, the prompt template, weekly suggestionsThe agent’s pipeline
sypnia-alertsThe nine alert rules, release, AlertSink with the record and the webhook, uptime checksProblem grouping
sypnia-apiThe read side: every endpoint of spec/openapi.yaml that is not ingest, the MCP server, the ConsoleAPI reference
sypniamain, Engine, serve, lambda, the admin commandsDeployment shapes
sypnia-testkitTestKit: a folder as the Bucket, a throwaway Postgres, a clock you set, a provider that answers from a file, and the recorded hoursThis page

Two rules shape the graph (2):

  • The parts do not depend on each other. Ingest does not know the pipeline exists; the pipeline does not know the API exists. They share types from sypnia-core and data in the Bucket and Postgres, which is exactly how they are deployed apart.
  • sypnia-core does no I/O. A function there takes values and returns values. That is where the rules that must never drift live: the fingerprint’s inputs, the histogram, the quota decision, the state transition.

The main traits and types

NameKindCrateOne line
EventenumcoreOne line of a segment, tagged by type, with extra for unknown fields and Unknown for unknown types
ProjectId, ScopestructscoreA project, and a project with an environment. Nothing reads or writes without a Scope
Fingerprintstructcore16 hex characters
Histogram<N>structcoreFixed cells; record, merge, percentile
ObjectStoretraitcore, implemented in storeput, read_lines, get, list, delete_prefix, sign
Databasetraitcore, implemented in dbOne method per statement, grouped by table
AiProvidertraitagentcomplete: text in, text out, token counts
AlertSinktraitcore, implemented in alertsdeliver one released alert
Intaketraitingestappend validated lines; returns when they are durable
Clocktraitcorenow. Real in production, set by hand in tests
KeysstructstoreBuilds every object key from a Scope
EnginestructsypniaThe traits above, wired once; pass runs one round of the worker
Failure, TransitionenumspipelineWhat the fast path extracts from a line, and what happened to its problem
HourFold, DeltaFoldstructspipelineThe two sets of group-bys of the roll-up
Redacted<T>structagentA value that passed redaction. The only thing a provider accepts
EngineError and one error enum per crateenumseachEvery failure has a variant; the API maps variants to the error codes of the spec

The rules of the code

Six rules. Each is enforced by something that fails a build, not by a reviewer’s memory.

RuleWhat it meansEnforced by
No unsafeNot one block, in any crateunsafe_code = "forbid" in the workspace lints
No panic on inputNothing an SDK, a browser, a bucket or a provider sends can stop a process. No unwrap, expect, panic!, unchecked index or unchecked arithmetic outside testsClippy lints set to deny; a fuzz target on validate and on the fingerprint
Every write is idempotentAn overwrite with the same key, a replace, or an add guarded by the ledger. A test runs every step twice and comparesThe replay tests below
Nothing without a projectNo store or database method takes a bare string where a project belongsThe types: Scope, Keys, ObjectKey
Time comes from the ClockNo Utc::now() outside the one real clockA lint on the disallowed method; tests set the time
The spec is the contractA response type that disagrees with spec/openapi.yaml, an event type that disagrees with events.schema.json, or a setting that is not in config.yaml fails CIContract tests generated from spec/

The first two are a few lines at the top of the workspace. Notice that the lints are deny, not warn: a build with an unwrap in engine code does not produce a binary.

Cargo.toml
[workspace]
members = ["crates/*"]
resolver = "2"

[workspace.package]
edition = "2021"
license = "AGPL-3.0-only"
rust-version = "1.82"

[workspace.lints.rust]
unsafe_code = "forbid"
missing_debug_implementations = "warn"

[workspace.lints.clippy]
unwrap_used = "deny"            # tests opt out with #[cfg(test)] allow
expect_used = "deny"
panic = "deny"
indexing_slicing = "deny"       # use .get(), or prove the bound in the type
arithmetic_side_effects = "deny" # counters use saturating_add
disallowed_methods = "deny"     # clippy.toml lists chrono::Utc::now and std::time::SystemTime::now

[workspace.dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "signal"] }
axum = "0.7"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["raw_value"] }
sqlx = { version = "0.8", features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "json"] }
object_store = { version = "0.11", features = ["aws"] }

What «no panic on input» looks like in practice: an error is a value with a variant, and the edge of the engine turns it into the answer the spec promises. Notice that the mapping is one exhaustive match: adding a variant without deciding its status code does not compile.

crates/sypnia-ingest/src/error.rs
#[derive(Debug, thiserror::Error)]
pub enum IngestError {
    #[error("the ingest key is missing, unknown or revoked")]
    Unauthorized,
    #[error("an API token was sent to an ingest endpoint")]
    WrongCredential,
    #[error("origin not allowed for this key")]
    OriginNotAllowed,
    #[error("too many requests from this address")]
    RateLimited { retry_after: Duration },
    #[error("the body is not valid JSON or gzip")]
    InvalidBody,
    #[error("more than 200 events in one batch")]
    TooManyEvents,
    #[error("the body is larger than the limit after decompression")]
    PayloadTooLarge,
    #[error("storage is not reachable")]
    Unavailable,
}

impl IntoResponse for IngestError {
    fn into_response(self) -> Response {
        let (status, code, retry_after) = match &self {
            Self::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized", None),
            Self::WrongCredential => (StatusCode::FORBIDDEN, "wrong_credential", None),
            Self::OriginNotAllowed => (StatusCode::FORBIDDEN, "origin_not_allowed", None),
            Self::RateLimited { retry_after } => (StatusCode::TOO_MANY_REQUESTS, "rate_limited", Some(*retry_after)),
            Self::InvalidBody => (StatusCode::BAD_REQUEST, "invalid_body", None),
            Self::TooManyEvents => (StatusCode::BAD_REQUEST, "too_many_events", None),
            Self::PayloadTooLarge => (StatusCode::PAYLOAD_TOO_LARGE, "payload_too_large", None),
            Self::Unavailable => (StatusCode::SERVICE_UNAVAILABLE, "unavailable", Some(Duration::from_secs(2))),
        };
        api_error(status, code, self.to_string(), retry_after) // { "error": { code, message, doc_url, request_id } }
    }
}

Tests that replay recorded hours

A unit test can prove a function. It cannot prove that an hour of real traffic, arriving in the order and the pieces it really arrived in, ends as the right rows. So the engine’s main tests are recordings.

A recorded hour is a folder: every batch that reached ingest in an hour, each with the time it arrived, and next to it the rows and objects that hour must produce. The first one is the synthetic hour of the system this engine comes from: three sessions with vitals, clicks, errors, requests and their server records. Its expected rows are the ones the original pipeline produced, which makes the test a proof of parity, not only of consistency.

Diagram to drawFLOW DIAGRAM of one replay test, left to right, with a loop. Node 1 (file stack icon): «Recorded hour» with the sub-label «hours/shop-2026-03-12T09/ · 41 batches with their arrival times». Arrow labelled «replayed in order, clock set to each arrival time» to node 2: «TestKit» drawn as a frame containing a small copy of the engine: «Ingest» → «Bucket (a folder)» → «Fast path» and «Roll-up» → «Postgres (throwaway)». Arrow from the frame to node 3: «Snapshot» with the sub-label «every table, every object key, as text». Arrow labelled «compare» to node 4: «Expected snapshot (in the repository)», with a check mark; this comparison is the lit element. From node 3 a loop arrow going back into TestKit, with four small numbered tags along it, exact labels: «1 run the pass again», «2 send every batch a second time», «3 stop the roll-up before its commit, then run it again», «4 run two workers at once». Under the loop the line: «after each one the snapshot must be identical». Numbered callouts: (1) the recorded hour, (2) TestKit, (3) the comparison, (4) the loop.
Figure. A replay test. An hour is played through the whole engine, and then four things that happen in production are done to it. The rows must not change.

Notice that the test below never asserts on a single number. It asserts that the whole result equals the recorded one, and then that nothing a retry can do changes it.

crates/sypnia-pipeline/tests/replay_shop_hour.rs
#[tokio::test]
async fn the_shop_hour_survives_everything_a_retry_can_do() {
    // A folder as the Bucket, a throwaway Postgres with the migrations applied, a clock we set.
    let kit = TestKit::start().await;
    let hour = RecordedHour::load("hours/shop-2026-03-12T09");

    // Each batch goes through the real ingest handler at the time it really arrived.
    kit.replay(&hour).await;
    kit.run_until_idle().await; // passes of the worker, until the ledger has nothing new
    let first = kit.snapshot().await; // every table and every object key, as sorted text
    insta::assert_snapshot!("shop_hour", first);

    // 1. Running again changes nothing.
    kit.engine.pass().await.unwrap();
    assert_eq!(first, kit.snapshot().await, "a second pass changed the result");

    // 2. Every batch arrives a second time: stored, never counted.
    kit.replay(&hour).await;
    kit.run_until_idle().await;
    assert_eq!(first.rows(), kit.snapshot().await.rows(), "duplicated batches were counted");

    // 3. The roll-up dies just before its commit, then runs again.
    let fresh = TestKit::start().await;
    fresh.replay(&hour).await;
    fresh.fail_once(FailPoint::BeforeRollupCommit);
    assert!(fresh.engine.pass().await.is_err());
    fresh.run_until_idle().await;
    assert_eq!(first, fresh.snapshot().await, "a failed run left something behind");

    // 4. Two workers at once: one claim wins, the result is the same.
    let racing = TestKit::start().await;
    racing.replay(&hour).await;
    let (a, b) = tokio::join!(racing.engine.pass(), racing.engine.pass());
    assert!(a.is_ok() && b.is_ok());
    racing.run_until_idle().await;
    assert_eq!(first, racing.snapshot().await, "two workers disagreed");
}

The other kinds of test, and what each one guards:

TestsGuard
vectors/fingerprints.jsonThe same inputs and expected fingerprints are run by this crate and by the SDKs. A difference of one bit fails both builds. The worked example of Problem grouping is vector 1
vectors/histograms.jsonCells and percentiles for known values, shared with anything that reads the arrays
Contract testsEvery example in spec/openapi.yaml and spec/events.schema.json is sent to the real handlers; every response is checked against its schema
Ported suitesThe ingest, aggregation, symbolication and upsert tests of the system this comes from, translated: its exception lists for false dead clicks and foreign errors are product knowledge
Fuzzingvalidate, the stack parser and the OTLP mapper must return, never panic, on any bytes
Budget testsA recorded hour at ten times the size must roll inside 256 MB and inside its time budget

Run it on your machine

At the end you will have: the engine running locally in one process, a project, and the example problem in the list.

You need Rust (the version in rust-toolchain.toml) and Docker, for Postgres and an S3-compatible store.

  1. Start Postgres and a bucket

    Terminal
    docker compose -f deploy/docker-compose.dev.yaml up -d
    You should see
    Two containers running: Postgres on port 5432 and MinIO on port 9000, with a bucket named sypnia-dev.
  2. Point the engine at them

    Terminal
    cp .env.example .env    # base URL http://localhost:8080, the local database and bucket, a root token
    set -a; source .env; set +a
    You should see
    echo $SYPNIA_BASE_URL prints http://localhost:8080.
  3. Create the tables

    Terminal
    cargo run -- migrate
    You should see
    The list of applied migrations, ending with database is up to date.
  4. Run the engine

    Terminal
    cargo run -- serve
    You should see
    Text
    sypnia 1.0.0  role=all  listen=0.0.0.0:8080
    database  ok  postgres 16.4, schema 12
    storage   ok  s3://sypnia-dev (us-east-1)
    ai        off (SYPNIA_AI_PROVIDER=none)
    ready     http://localhost:8080
  5. Create a project and send the example

    In a second terminal, with the same .env loaded:

    Terminal
    cargo run -- project create --name Shop
    curl -X POST "$SYPNIA_BASE_URL/v1/ingest/events" \
      -H "Authorization: Bearer <the spk_ key that was printed>" \
      -H "Content-Type: application/json" \
      --data @crates/sypnia-testkit/hours/shop-2026-03-12T09/batch-0007.json
    You should see
    {"accepted":2,"dropped":0}. In the first terminal, within a few seconds, a line of the worker that says 1 problem (1 new).
  6. See the problem

    Terminal
    cargo run -- problems --project shop
    You should see
    One open problem, createPayment answered 400 in POST /api/checkout, with 1 person. The Console shows the same at http://localhost:8080/console.

To run the tests: cargo test --workspace. The replay tests start their own Postgres through Docker and need nothing else. cargo run -- doctor checks a configuration piece by piece when something does not start.

How a contribution flows

The engine is built documentation-first, and a contribution follows the same order.

Diagram to drawFLOW DIAGRAM, left to right, six steps as nodes joined by arrows, with a lower lane of automatic checks. Upper lane, exact labels: 1 «Issue: what should change, and why»; 2 «The docs change first» with the sub-label «the page, and the file in spec/ if a contract moves», drawn as the lit element; 3 «The code, with its tests»; 4 «Pull request»; 5 «Review by a maintainer»; 6 «Merge, then a release». Lower lane under step 4, a row of seven small check tiles joined to the pull request by a bracket labelled «CI: all must pass», exact labels: «format», «lints: no unsafe, no unwrap», «unit tests», «replay tests», «shared vectors», «contract tests against spec/», «migrations only add». An arrow from step 5 back to step 3 labelled «changes asked». A side note attached to step 2: «a behaviour change without a docs change is not reviewed». A side note attached to step 6: «the changelog names it». Numbered callouts are the step numbers.
Figure. How a change gets in. The page changes before the code, and seven automatic checks stand between a pull request and a review.
  1. Open an issue that says what should change and why. For a bug, a recorded batch that shows it is the best report.
  2. Change the docs first. If the change alters behaviour, the page that describes it changes in the same pull request, and so does the file in spec/ when an endpoint, an event, a setting or a command moves. A reviewer reads the page before the code.
  3. Write the code and its tests. A new path through the pipeline comes with a recorded hour, or an addition to one. A new SQL statement comes with the test that runs it twice.
  4. Open the pull request. CI runs the seven checks of the figure. The migration check refuses a migration that drops or renames: old and new versions of the engine must be able to run against the same tables during an upgrade.
  5. A maintainer reviews. They look for the rules of this page before anything else: is every new write idempotent, does anything take a project as a bare string, can any input reach a panic.
  6. Merge and release. Versions follow the API’s rule: inside version 1, things are only added.

Some changes are not accepted as pull requests, because they are decisions and not code: a new required service next to the Bucket and Postgres, a change to the fingerprint that would split existing problems, a change to the histogram bounds, an integration that sends alerts to a chat product. Open an issue to discuss those.

The engine and the Console are under AGPL-3.0; the SDKs and @sypnia/mcp are under MIT. What that means for someone who self-hosts, modifies or contributes, and the terms a contribution is accepted under, are on Contributing.

The Sypnia engine and Sypnia Cloud compared
Sypnia engineOpen source, self-hostedSypnia CloudPaid service, with a free tier
This repositoryEverything on this page: the eleven crates, the specs, the deploy files, the Console.The same code, deployed by us from the same releases.
Not in this repositoryNothing is held back from the engine.Accounts and organizations, billing, the app, e-mail delivery, the GitHub App and the managed model live in a private repository. They use the engine through its public API.

Where to start reading

If you want to understandOpen
What an event iscrates/sypnia-core/src/event.rs
What happens to a requestcrates/sypnia-ingest/src/events.rs, then buffer.rs
How a problem is borncrates/sypnia-pipeline/src/fast_path.rs, then problems/transition.rs
How a number is countedcrates/sypnia-pipeline/src/fold/pages.rs, then crates/sypnia-db/src/rollup_commit.rs
Why two failures are the samecrates/sypnia-fingerprint/src/lib.rs and vectors/fingerprints.json
What the model is sentcrates/sypnia-agent/src/context.rs, then check.rs
How the two shapes differcrates/sypnia/src/serve.rs and lambda.rs
Whether a rule really holdscrates/sypnia-pipeline/tests/replay_shop_hour.rs

Next

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close