The codebase
The engine's repository, crate by crate, with the main traits and types, the rules every line of code follows, the tests that replay recorded hours, how to run it on your machine and how a contribution flows.
On this page
This page is a map of the engine’s source code, for people who want to read it, run it or change it. It is the last page of the section on purpose: everything it names was explained on an earlier one.
The workspace
One Cargo workspace, eleven crates, one binary.
sypnia/
├── Cargo.toml the workspace, the shared lints, the pinned dependencies
├── crates/
│ ├── sypnia-core/ (4) types and traits; no network, no disk, no clock of its own
│ ├── sypnia-fingerprint/ (3) hash64, message and frame rules, per-platform lists
│ ├── sypnia-store/ (3) ObjectStore over S3-compatible stores and over a folder; Keys
│ ├── sypnia-db/ (3) Database over Postgres; every SQL statement of the engine
│ │ └── migrations/ numbered, additive, forward-only
│ ├── sypnia-ingest/ (2) the seven gates, the buffer, replay and snapshot uploads, OTLP mapping
│ ├── sypnia-pipeline/ (2) fast path, problems, roll-up folds, sessions, symbolication, Parquet, retention
│ ├── sypnia-agent/ (2) context, redaction, providers, the check, the prompt template
│ ├── sypnia-alerts/ (2) alert rules, release, sinks, webhook deliveries, uptime checks
│ ├── sypnia-api/ (2) the HTTP API, the MCP server, the Console's files
│ ├── sypnia/ (1) main, CLI, serve, lambda, Engine
│ └── sypnia-testkit/ recorded hours and in-memory adapters, for tests only
├── spec/ openapi.yaml, events.schema.json, mcp.json, config.yaml, cli.yaml
├── vectors/ fingerprint and histogram test vectors, shared with the SDKs
├── deploy/ sypnia-aws.yaml, docker-compose.yaml, Dockerfile
└── console/ the Console: plain files, embedded in the binary at build time| Crate | What it does | Explained on |
|---|---|---|
sypnia-core | Event, ProjectId, Scope, Histogram, the quota function, the four port traits, the error types. It does no I/O, so it compiles anywhere and its tests need nothing | Overview |
sypnia-fingerprint | hash64 over UTF-16 code units, message normalization, frame parsing, the lists of noisy frames and build-hash rules per platform | Problem grouping |
sypnia-store | ObjectStore implemented over the object_store crate, and over a folder. Keys, the only place that knows the bucket layout | Storage |
sypnia-db | Database implemented with sqlx. Every statement lives here as a .sql file checked at build time. The ledger. Migrations | Storage, Aggregation |
sypnia-ingest | The axum handlers for events, replay chunks, snapshots, config and OTLP. ProjectCache. Intake and its two implementations: the buffer and the queue | Ingest |
sypnia-pipeline | fast_path, problems::transition, the folds, session assembly, symbolication, compaction, retention | Data flow, Aggregation |
sypnia-agent | build_context, redact, AiProvider with ChatCompletions and AnthropicMessages, check, the prompt template, weekly suggestions | The agent’s pipeline |
sypnia-alerts | The nine alert rules, release, AlertSink with the record and the webhook, uptime checks | Problem grouping |
sypnia-api | The read side: every endpoint of spec/openapi.yaml that is not ingest, the MCP server, the Console | API reference |
sypnia | main, Engine, serve, lambda, the admin commands | Deployment shapes |
sypnia-testkit | TestKit: a folder as the Bucket, a throwaway Postgres, a clock you set, a provider that answers from a file, and the recorded hours | This page |
Two rules shape the graph (2):
- The parts do not depend on each other. Ingest does not know the pipeline exists; the pipeline does not know the API exists. They share types from
sypnia-coreand data in the Bucket and Postgres, which is exactly how they are deployed apart. sypnia-coredoes no I/O. A function there takes values and returns values. That is where the rules that must never drift live: the fingerprint’s inputs, the histogram, the quota decision, the state transition.
The main traits and types
| Name | Kind | Crate | One line |
|---|---|---|---|
Event | enum | core | One line of a segment, tagged by type, with extra for unknown fields and Unknown for unknown types |
ProjectId, Scope | structs | core | A project, and a project with an environment. Nothing reads or writes without a Scope |
Fingerprint | struct | core | 16 hex characters |
Histogram<N> | struct | core | Fixed cells; record, merge, percentile |
ObjectStore | trait | core, implemented in store | put, read_lines, get, list, delete_prefix, sign |
Database | trait | core, implemented in db | One method per statement, grouped by table |
AiProvider | trait | agent | complete: text in, text out, token counts |
AlertSink | trait | core, implemented in alerts | deliver one released alert |
Intake | trait | ingest | append validated lines; returns when they are durable |
Clock | trait | core | now. Real in production, set by hand in tests |
Keys | struct | store | Builds every object key from a Scope |
Engine | struct | sypnia | The traits above, wired once; pass runs one round of the worker |
Failure, Transition | enums | pipeline | What the fast path extracts from a line, and what happened to its problem |
HourFold, DeltaFold | structs | pipeline | The two sets of group-bys of the roll-up |
Redacted<T> | struct | agent | A value that passed redaction. The only thing a provider accepts |
EngineError and one error enum per crate | enums | each | Every failure has a variant; the API maps variants to the error codes of the spec |
The rules of the code
Six rules. Each is enforced by something that fails a build, not by a reviewer’s memory.
| Rule | What it means | Enforced by |
|---|---|---|
No unsafe | Not one block, in any crate | unsafe_code = "forbid" in the workspace lints |
| No panic on input | Nothing an SDK, a browser, a bucket or a provider sends can stop a process. No unwrap, expect, panic!, unchecked index or unchecked arithmetic outside tests | Clippy lints set to deny; a fuzz target on validate and on the fingerprint |
| Every write is idempotent | An overwrite with the same key, a replace, or an add guarded by the ledger. A test runs every step twice and compares | The replay tests below |
| Nothing without a project | No store or database method takes a bare string where a project belongs | The types: Scope, Keys, ObjectKey |
Time comes from the Clock | No Utc::now() outside the one real clock | A lint on the disallowed method; tests set the time |
| The spec is the contract | A response type that disagrees with spec/openapi.yaml, an event type that disagrees with events.schema.json, or a setting that is not in config.yaml fails CI | Contract tests generated from spec/ |
The first two are a few lines at the top of the workspace. Notice that the lints are deny, not warn: a build with an unwrap in engine code does not produce a binary.
[workspace]
members = ["crates/*"]
resolver = "2"
[workspace.package]
edition = "2021"
license = "AGPL-3.0-only"
rust-version = "1.82"
[workspace.lints.rust]
unsafe_code = "forbid"
missing_debug_implementations = "warn"
[workspace.lints.clippy]
unwrap_used = "deny" # tests opt out with #[cfg(test)] allow
expect_used = "deny"
panic = "deny"
indexing_slicing = "deny" # use .get(), or prove the bound in the type
arithmetic_side_effects = "deny" # counters use saturating_add
disallowed_methods = "deny" # clippy.toml lists chrono::Utc::now and std::time::SystemTime::now
[workspace.dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "time", "signal"] }
axum = "0.7"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["raw_value"] }
sqlx = { version = "0.8", features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "json"] }
object_store = { version = "0.11", features = ["aws"] }What «no panic on input» looks like in practice: an error is a value with a variant, and the edge of the engine turns it into the answer the spec promises. Notice that the mapping is one exhaustive match: adding a variant without deciding its status code does not compile.
#[derive(Debug, thiserror::Error)]
pub enum IngestError {
#[error("the ingest key is missing, unknown or revoked")]
Unauthorized,
#[error("an API token was sent to an ingest endpoint")]
WrongCredential,
#[error("origin not allowed for this key")]
OriginNotAllowed,
#[error("too many requests from this address")]
RateLimited { retry_after: Duration },
#[error("the body is not valid JSON or gzip")]
InvalidBody,
#[error("more than 200 events in one batch")]
TooManyEvents,
#[error("the body is larger than the limit after decompression")]
PayloadTooLarge,
#[error("storage is not reachable")]
Unavailable,
}
impl IntoResponse for IngestError {
fn into_response(self) -> Response {
let (status, code, retry_after) = match &self {
Self::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized", None),
Self::WrongCredential => (StatusCode::FORBIDDEN, "wrong_credential", None),
Self::OriginNotAllowed => (StatusCode::FORBIDDEN, "origin_not_allowed", None),
Self::RateLimited { retry_after } => (StatusCode::TOO_MANY_REQUESTS, "rate_limited", Some(*retry_after)),
Self::InvalidBody => (StatusCode::BAD_REQUEST, "invalid_body", None),
Self::TooManyEvents => (StatusCode::BAD_REQUEST, "too_many_events", None),
Self::PayloadTooLarge => (StatusCode::PAYLOAD_TOO_LARGE, "payload_too_large", None),
Self::Unavailable => (StatusCode::SERVICE_UNAVAILABLE, "unavailable", Some(Duration::from_secs(2))),
};
api_error(status, code, self.to_string(), retry_after) // { "error": { code, message, doc_url, request_id } }
}
}Tests that replay recorded hours
A unit test can prove a function. It cannot prove that an hour of real traffic, arriving in the order and the pieces it really arrived in, ends as the right rows. So the engine’s main tests are recordings.
A recorded hour is a folder: every batch that reached ingest in an hour, each with the time it arrived, and next to it the rows and objects that hour must produce. The first one is the synthetic hour of the system this engine comes from: three sessions with vitals, clicks, errors, requests and their server records. Its expected rows are the ones the original pipeline produced, which makes the test a proof of parity, not only of consistency.
Notice that the test below never asserts on a single number. It asserts that the whole result equals the recorded one, and then that nothing a retry can do changes it.
#[tokio::test]
async fn the_shop_hour_survives_everything_a_retry_can_do() {
// A folder as the Bucket, a throwaway Postgres with the migrations applied, a clock we set.
let kit = TestKit::start().await;
let hour = RecordedHour::load("hours/shop-2026-03-12T09");
// Each batch goes through the real ingest handler at the time it really arrived.
kit.replay(&hour).await;
kit.run_until_idle().await; // passes of the worker, until the ledger has nothing new
let first = kit.snapshot().await; // every table and every object key, as sorted text
insta::assert_snapshot!("shop_hour", first);
// 1. Running again changes nothing.
kit.engine.pass().await.unwrap();
assert_eq!(first, kit.snapshot().await, "a second pass changed the result");
// 2. Every batch arrives a second time: stored, never counted.
kit.replay(&hour).await;
kit.run_until_idle().await;
assert_eq!(first.rows(), kit.snapshot().await.rows(), "duplicated batches were counted");
// 3. The roll-up dies just before its commit, then runs again.
let fresh = TestKit::start().await;
fresh.replay(&hour).await;
fresh.fail_once(FailPoint::BeforeRollupCommit);
assert!(fresh.engine.pass().await.is_err());
fresh.run_until_idle().await;
assert_eq!(first, fresh.snapshot().await, "a failed run left something behind");
// 4. Two workers at once: one claim wins, the result is the same.
let racing = TestKit::start().await;
racing.replay(&hour).await;
let (a, b) = tokio::join!(racing.engine.pass(), racing.engine.pass());
assert!(a.is_ok() && b.is_ok());
racing.run_until_idle().await;
assert_eq!(first, racing.snapshot().await, "two workers disagreed");
}The other kinds of test, and what each one guards:
| Tests | Guard |
|---|---|
vectors/fingerprints.json | The same inputs and expected fingerprints are run by this crate and by the SDKs. A difference of one bit fails both builds. The worked example of Problem grouping is vector 1 |
vectors/histograms.json | Cells and percentiles for known values, shared with anything that reads the arrays |
| Contract tests | Every example in spec/openapi.yaml and spec/events.schema.json is sent to the real handlers; every response is checked against its schema |
| Ported suites | The ingest, aggregation, symbolication and upsert tests of the system this comes from, translated: its exception lists for false dead clicks and foreign errors are product knowledge |
| Fuzzing | validate, the stack parser and the OTLP mapper must return, never panic, on any bytes |
| Budget tests | A recorded hour at ten times the size must roll inside 256 MB and inside its time budget |
Run it on your machine
At the end you will have: the engine running locally in one process, a project, and the example problem in the list.
You need Rust (the version in rust-toolchain.toml) and Docker, for Postgres and an S3-compatible store.
Create a project and send the example
In a second terminal, with the same
.envloaded:Terminal cargo run -- project create --name Shop curl -X POST "$SYPNIA_BASE_URL/v1/ingest/events" \ -H "Authorization: Bearer <the spk_ key that was printed>" \ -H "Content-Type: application/json" \ --data @crates/sypnia-testkit/hours/shop-2026-03-12T09/batch-0007.jsonYou should see{"accepted":2,"dropped":0}. In the first terminal, within a few seconds, a line of the worker that says1 problem (1 new).
To run the tests: cargo test --workspace. The replay tests start their own Postgres through Docker and need nothing else. cargo run -- doctor checks a configuration piece by piece when something does not start.
How a contribution flows
The engine is built documentation-first, and a contribution follows the same order.
- Open an issue that says what should change and why. For a bug, a recorded batch that shows it is the best report.
- Change the docs first. If the change alters behaviour, the page that describes it changes in the same pull request, and so does the file in
spec/when an endpoint, an event, a setting or a command moves. A reviewer reads the page before the code. - Write the code and its tests. A new path through the pipeline comes with a recorded hour, or an addition to one. A new SQL statement comes with the test that runs it twice.
- Open the pull request. CI runs the seven checks of the figure. The migration check refuses a migration that drops or renames: old and new versions of the engine must be able to run against the same tables during an upgrade.
- A maintainer reviews. They look for the rules of this page before anything else: is every new write idempotent, does anything take a project as a bare string, can any input reach a panic.
- Merge and release. Versions follow the API’s rule: inside version 1, things are only added.
Some changes are not accepted as pull requests, because they are decisions and not code: a new required service next to the Bucket and Postgres, a change to the fingerprint that would split existing problems, a change to the histogram bounds, an integration that sends alerts to a chat product. Open an issue to discuss those.
The engine and the Console are under AGPL-3.0; the SDKs and @sypnia/mcp are under MIT. What that means for someone who self-hosts, modifies or contributes, and the terms a contribution is accepted under, are on Contributing.
| Sypnia engineOpen source, self-hosted | Sypnia CloudPaid service, with a free tier | |
|---|---|---|
| This repository | Everything on this page: the eleven crates, the specs, the deploy files, the Console. | The same code, deployed by us from the same releases. |
| Not in this repository | Nothing is held back from the engine. | Accounts and organizations, billing, the app, e-mail delivery, the GitHub App and the managed model live in a private repository. They use the engine through its public API. |
Where to start reading
| If you want to understand | Open |
|---|---|
| What an event is | crates/sypnia-core/src/event.rs |
| What happens to a request | crates/sypnia-ingest/src/events.rs, then buffer.rs |
| How a problem is born | crates/sypnia-pipeline/src/fast_path.rs, then problems/transition.rs |
| How a number is counted | crates/sypnia-pipeline/src/fold/pages.rs, then crates/sypnia-db/src/rollup_commit.rs |
| Why two failures are the same | crates/sypnia-fingerprint/src/lib.rs and vectors/fingerprints.json |
| What the model is sent | crates/sypnia-agent/src/context.rs, then check.rs |
| How the two shapes differ | crates/sypnia/src/serve.rs and lambda.rs |
| Whether a rule really holds | crates/sypnia-pipeline/tests/replay_shop_hour.rs |
Next
- Back to the overview, with the map in hand.
- The licences in plain words, and the terms of a contribution.
- From
cargo runto a real deployment.