# Storage

> The bucket layout as a tree, the Postgres schema with its main tables, how data is partitioned and expired, how much a million events weigh, and what is never stored.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/architecture/storage/

The engine keeps data in two places and nowhere else: one bucket in any S3-compatible store, and one Postgres database. This page shows exactly what is in each, down to the object names and the columns.

> **Note: About the code and the numbers**
>
> The SQL and Rust here are design excerpts; the repository opens at launch. Sizes are estimates with their arithmetic shown. Where a figure was measured on the system this engine comes from, the text says so.

## The bucket

Every key starts with the project id. Under it, one folder per kind of data.

**Figure**: The bucket of one project. Ingest writes three branches, the roll-up two, your CI one.

1. `events/`: the segments, written by Ingest. Raw events, at most 2 seconds or 4 MiB each.
2. `sessions/`: one session's events, in order, one file per hour. Written by the Roll-up.
3. `replays/`: replay chunks, one every 10 seconds. Written by Ingest.
4. `snapshots/`: page snapshots for click maps. Written by Ingest.
5. `sourcemaps/`: one folder per release. Written by your CI.
6. `parquet/`: yesterday's events, compacted. Written by the Roll-up.

A tree drawn like a file explorer. The root is your bucket (plus `storage.prefix`, if you set one). Under it there is one folder per project, `prj_5k2m9x4q7w1z/`, and nothing is shared between projects. The project folder has six branches, each with example files; the same tree is written as text right below this figure.

The same tree as text, to copy:

```text
prj_5k2m9x4q7w1z/
├── events/production/dt=2026-03-12/hr=09/
│   ├── 20260312T091207Z-2b81e4.ndjson.gz        (1) a segment
│   └── 20260312T091216Z-7f3a9c.ndjson.gz
├── sessions/8f3a6c1e9b2d4f07a5c3e1b8d6f4a2c0/
│   └── hr=2026-03-12-09.ndjson.gz               (2) a session, one file per hour
├── replays/8f3a6c1e9b2d4f07a5c3e1b8d6f4a2c0/
│   ├── 000000.json.gz                           (3) replay chunks
│   └── 000001.json.gz
├── snapshots/production/route=checkout/dev=desktop/
│   └── 1773306724000.json.gz                    (4) a page snapshot
├── sourcemaps/4f2a91c/
│   ├── index.json                               (5) built file → its map
│   └── assets/checkout-3f9c.js.map
└── parquet/production/dt=2026-03-11/
    └── events.parquet                           (6) one day, compacted
```

| # | Prefix        | Written by                            | Key                                                                                                                                                                                                                             | Kept (default)                          |
| - | ------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| 1 | `events/`     | Ingest                                | `events/<environment>/dt=<day>/hr=<hour>/<written at>-<writer>.ndjson.gz`. The day and hour are when the segment was **written**, in UTC. `<writer>` is 6 hex characters chosen by the process, so two processes never collide. | 90 days (`retention.events_days`)       |
| 2 | `sessions/`   | Roll-up                               | `sessions/<sid>/hr=<day>-<hour>.ndjson.gz`. Written again, whole, each time the roll-up sees new events of that session in that hour.                                                                                           | 90 days                                 |
| 3 | `replays/`    | Ingest                                | `replays/<sid>/<seq, 6 digits>.json.gz`. Sending the same `seq` again replaces the object. At most 720 per session.                                                                                                             | 30 days (`retention.replays_days`)      |
| 4 | `snapshots/`  | Ingest                                | `snapshots/<environment>/route=<route key>/dev=<device>/<ts>.json.gz`                                                                                                                                                           | 90 days (`retention.snapshots_days`)    |
| 5 | `sourcemaps/` | Your CI, through a link the API signs | `sourcemaps/<release>/index.json` and `sourcemaps/<release>/<path>.map`                                                                                                                                                         | 90 days after the release was last seen |
| 6 | `parquet/`    | Roll-up, once a day                   | `parquet/<environment>/dt=<day>/events.parquet`                                                                                                                                                                                 | 90 days                                 |

Names are built in one place. Notice that nothing in this module takes a bare string for the project: a key cannot be built without a `Scope`, and the route goes through one function that makes any path safe as a key.

`crates/sypnia-store/src/keys.rs`

```rust
/// Builds every object key of the engine. The only place where key layout lives.
pub struct Keys<'a> {
    prefix: &'a str, // `storage.prefix`, usually empty
    scope: &'a Scope,
}

impl<'a> Keys<'a> {
    /// `<prefix>prj_…/events/production/dt=2026-03-12/hr=09/20260312T091216Z-7f3a9c.ndjson.gz`
    pub fn segment(&self, written: DateTime<Utc>, writer: WriterId) -> ObjectKey {
        ObjectKey::new(format!(
            "{}{}/events/{}/dt={}/hr={}/{}-{}.ndjson.gz",
            self.prefix,
            self.scope.project,
            self.scope.env,
            written.format("%Y-%m-%d"),
            written.format("%H"),
            written.format("%Y%m%dT%H%M%SZ"),
            writer, // 6 hex characters, fixed for the life of the process
        ))
    }

    /// Chunks sort by name, so a listing is already in playback order.
    pub fn replay_chunk(&self, sid: &SessionId, seq: u16) -> ObjectKey {
        ObjectKey::new(format!("{}{}/replays/{}/{:06}.json.gz", self.prefix, self.scope.project, sid, seq))
    }

    pub fn snapshot(&self, route: &Route, dev: Device, ts: i64) -> ObjectKey {
        ObjectKey::new(format!(
            "{}{}/snapshots/{}/route={}/dev={}/{}.json.gz",
            self.prefix, self.scope.project, self.scope.env, route_key(route), dev, ts,
        ))
    }
}

/// `/orders/:id` → `orders~_id`. Same rule as the system this comes from.
fn route_key(route: &Route) -> String {
    let trimmed = route.as_str().trim_start_matches('/');
    if trimmed.is_empty() {
        return "root".to_owned();
    }
    trimmed
        .chars()
        .map(|c| match c {
            ':' => '_',
            '/' => '~',
            c if c.is_ascii_alphanumeric() || matches!(c, '_' | '~' | '.' | '-') => c,
            _ => '-',
        })
        .collect()
}
```

### What is inside an object

**A segment** is gzip NDJSON: one event per line, exactly as the SDK sent it after validation, plus three fields the engine writes. This is the failed call of the example, as stored:

`one line of a segment`

```json
{"v":1,"type":"server_error","ts":1773306724519,"service":"shop-api","env":"production","ver":"4f2a91c","tid":"7c2e9a4b1d8f4e6a9b3c5d7e1f2a4b6c","sid":"8f3a6c1e9b2d4f07a5c3e1b8d6f4a2c0","person":{"id":"u_1042"},"account":{"id":"acc_77"},"method":"POST","path":"/api/checkout","route":"/api/checkout","name":"HTTP 400","msg":"amount is required (POST /v1/payments) in POST /api/checkout","stack":null,"digest":null,"src":"span","span":"3c4d5e6f708192a3","fp":"88a44b15205cb153","prj":"prj_5k2m9x4q7w1z","rcv":1773306725102,"bid":"5d0c7a1e-93b4-4f6a-8e2d-1c7b9a3f5e60"}
```

| Stamped field | Meaning                                                                                    |
| ------------- | ------------------------------------------------------------------------------------------ |
| `prj`         | The project, from the ingest key. Whatever the sender put there is replaced.               |
| `rcv`         | When ingest received the event, in milliseconds, by the engine's clock.                    |
| `bid`         | The `batch_id` the event arrived in. It is how a batch that arrived twice is counted once. |

**A per-session file** starts with one header line, then the session's events of that hour ordered by `ts`, each exactly as in its segment. The header lets a reader know what it has without reading the file:

`first line of sessions/8f3a…a2c0/hr=2026-03-12-09.ndjson.gz`

```json
{"type":"_session","v":1,"prj":"prj_5k2m9x4q7w1z","env":"production","sid":"8f3a6c1e9b2d4f07a5c3e1b8d6f4a2c0","hour":"2026-03-12T09:00:00Z","events":14,"first_ts":1773306690000,"last_ts":1773306731400,"server_records":2,"written_at":"2026-03-12T09:15:03Z"}
```

**A replay chunk** and **a page snapshot** are the body the SDK sent, gzipped, stored with `Content-Type: application/json` and `Content-Encoding: gzip` so a browser can fetch them through a signed link and get JSON. The engine does not look inside: masking happened in the browser.

**The Parquet file** has one typed column per field of the [event schema](https://docs.sypnia.com/reference/event-schema) and one more, `extra`, a JSON string with every field the engine did not know. Its schema is fixed by the engine's version, not guessed from the day's data. The engine never reads these files. They are for you: any tool that reads Parquet can query a day of events.

## Postgres

Postgres holds what a screen or an MCP call reads. Every table starts with `project_id`.

**Figure**: The main tables in four groups. The fast path owns the problems group, the roll-up owns the numbers group, and the ledger keeps both honest.

1. `projects`: every other table carries its `project_id`.
2. `segments`: the ledger. One row per segment in the Bucket, with when the Fast path scanned it and when the Roll-up counted it.
3. `problems`: one row per problem. Its occurrences, hits, trace links, analyses and alerts hang from it.
4. `perf_pages_hourly`: one row per hour, route and device, with a histogram instead of an average.

Nineteen tables in four groups; a line with a crow's foot is a one-to-many relation and lands on the column it uses. Who gets in: `projects` has many `ingest_keys` and many `api_tokens`. The ledger: `segments` has many `ingest_batches`; `hours` stands alone. Problems, written by the Fast path: `problems` has many `problem_occurrences`, `problem_hits`, `trace_failures`, `analyses` and `alerts`. Numbers, written by the Roll-up: `sessions`, `traces`, `perf_pages_hourly`, `perf_api_hourly`, `clicks_daily`, `usage_daily` and `uptime_results`. Two looser links cross between the last two groups: `problem_hits.key` points to `sessions.sid`, and `traces.trace_id` to `trace_failures.trace_id`.

### Who gets in

`migrations/0001_projects.sql`

```sql
CREATE TABLE projects (
    id          text PRIMARY KEY CHECK (id ~ '^prj_[0-9a-z]{12}$'),
    slug        text NOT NULL UNIQUE,
    name        text NOT NULL,
    settings    jsonb NOT NULL DEFAULT '{}',   -- replay rules, blocked routes, alert numbers, the agent's language
    limits      jsonb NOT NULL DEFAULT '{}',   -- events and replays per month; absent or 0 = no limit
    slow_down   jsonb,                         -- written by a pass, read by ingest through its cache
    has_server  boolean NOT NULL DEFAULT false,-- has this project ever sent server events?
    created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE ingest_keys (
    id                    text PRIMARY KEY,            -- key_…
    project_id            text NOT NULL REFERENCES projects ON DELETE CASCADE,
    key                   text NOT NULL UNIQUE,        -- spk_…: public by design, so stored as it is
    allowed_origins       text[] NOT NULL DEFAULT '{}',-- empty = any origin
    rate_limit_per_minute integer NOT NULL DEFAULT 600,
    revoked_at            timestamptz
);

CREATE TABLE api_tokens (
    id          text PRIMARY KEY,                      -- tok_…
    project_id  text NOT NULL REFERENCES projects ON DELETE CASCADE,
    name        text NOT NULL,
    token_hash  bytea NOT NULL UNIQUE,                 -- sha256(sps_…). The token is shown once and never stored.
    scopes      text[] NOT NULL CHECK (scopes <@ ARRAY['read', 'replay', 'admin']),
    revoked_at  timestamptz
);
```

There is no `users` table and no `organizations` table. Those belong to Sypnia Cloud's private part **\[Cloud]**, in a database of its own. The engine knows projects, keys and tokens, and one root token that lives in its configuration, never in a table.

### The ledger

Three small tables make every step safe to repeat. [Aggregation](https://docs.sypnia.com/architecture/aggregation) explains how they are used.

`migrations/0002_ledger.sql`

```sql
-- One row per segment object. Written by ingest after the object exists.
CREATE TABLE segments (
    project_id   text        NOT NULL,
    key          text        NOT NULL,
    environment  text        NOT NULL,
    hour         timestamptz NOT NULL,            -- the hr= of the key
    events       integer     NOT NULL,
    bytes        integer     NOT NULL,
    batch_ids    text[]      NOT NULL,            -- the batches inside
    skipped      text[]      NOT NULL DEFAULT '{}', -- of those, the ones an earlier segment already carried
    leased_until timestamptz,                     -- a worker is scanning it
    scanned_at   timestamptz,                     -- the fast path is done with it
    rolled_at    timestamptz,                     -- the roll-up added it to the counters
    PRIMARY KEY (project_id, key)
);
CREATE INDEX segments_unscanned ON segments (key) WHERE scanned_at IS NULL;
CREATE INDEX segments_by_hour   ON segments (project_id, environment, hour);

-- One row per batch for 48 hours: how a batch that arrives twice is counted once.
CREATE TABLE ingest_batches (
    project_id text        NOT NULL,
    batch_id   text        NOT NULL,
    segment    text        NOT NULL,
    seen_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_id, batch_id)
);

-- One row per project, environment and hour: the roll-up's run guard.
CREATE TABLE hours (
    project_id  text        NOT NULL,
    environment text        NOT NULL,
    hour        timestamptz NOT NULL,
    status      text        NOT NULL CHECK (status IN ('open', 'running', 'failed', 'closed')),
    events      integer     NOT NULL DEFAULT 0,
    last_run_at timestamptz,
    error       text,
    PRIMARY KEY (project_id, environment, hour)
);
```

### Problems

`migrations/0003_problems.sql`

```sql
CREATE TABLE problems (
    id            text PRIMARY KEY,                    -- prb_…
    project_id    text NOT NULL REFERENCES projects ON DELETE CASCADE,
    environment   text NOT NULL,
    fingerprint   text NOT NULL CHECK (fingerprint ~ '^[0-9a-f]{16}$'),
    slug          text NOT NULL,
    old_slugs     text[] NOT NULL DEFAULT '{}',        -- every slug it ever had keeps working
    kind          text NOT NULL CHECK (kind IN ('error', 'failed_call', 'timeout')),
    origin        text NOT NULL CHECK (origin IN ('browser', 'server')),
    status        text NOT NULL DEFAULT 'open' CHECK (status IN ('open', 'resolved', 'ignored')),
    regressed     boolean NOT NULL DEFAULT false,
    title         text NOT NULL,                       -- the technical title, as first seen
    route         text,
    first_seen    timestamptz NOT NULL,
    last_seen     timestamptz NOT NULL,
    first_release text,
    last_release  text,
    occurrences   bigint  NOT NULL DEFAULT 0,
    people        integer NOT NULL DEFAULT 0,          -- rows of problem_hits with kind 'person'
    sessions      integer NOT NULL DEFAULT 0,          -- rows of problem_hits with kind 'session'
    people_at_last_alert integer NOT NULL DEFAULT 0,   -- what «growing» is measured against
    growing_until timestamptz,
    resolved_at   timestamptz,
    resolved_by   text,
    resolved_in_release text,
    note          text,
    sample        jsonb,                               -- one full occurrence: message, stack, original stack, trace id
    analysis      jsonb,                               -- the agent's last answer, as the API returns it
    created_at    timestamptz NOT NULL DEFAULT now(),
    UNIQUE (project_id, environment, fingerprint),
    UNIQUE (project_id, slug)
);

-- Occurrences per hour, for the timeline. Arrays are capped at 200 like in the system this comes from.
CREATE TABLE problem_occurrences (
    project_id  text        NOT NULL,
    problem_id  text        NOT NULL REFERENCES problems ON DELETE CASCADE,
    hour        timestamptz NOT NULL,
    occurrences integer     NOT NULL,
    person_keys text[]      NOT NULL DEFAULT '{}',
    session_ids text[]      NOT NULL DEFAULT '{}',
    people      integer     NOT NULL DEFAULT 0,        -- GREATEST(stored, cardinality(person_keys))
    sessions    integer     NOT NULL DEFAULT 0,
    PRIMARY KEY (project_id, problem_id, hour)
);

-- Who and which sessions a problem hit, once each. This is what makes «3 people» exact.
CREATE TABLE problem_hits (
    project_id text        NOT NULL,
    problem_id text        NOT NULL REFERENCES problems ON DELETE CASCADE,
    kind       text        NOT NULL CHECK (kind IN ('person', 'session')),
    key        text        NOT NULL,   -- person id, or `device:<did>` for someone never identified; or the session id
    first_seen timestamptz NOT NULL,
    last_seen  timestamptz NOT NULL,
    PRIMARY KEY (project_id, problem_id, kind, key)
);
```

`people` is not an estimate. Each person a problem hits is one row in `problem_hits`; the insert uses `ON CONFLICT DO NOTHING`, and the number of rows it really inserted is added to `problems.people`. The same person failing twice adds nothing. Session rows expire with the session; person rows stay as long as the problem, so the count never goes down.

### Numbers

The hourly tables keep one histogram per metric instead of the values. A histogram is an array of counters with fixed bounds, so two of them are merged by adding position by position, and a percentile is read from the array when you ask. [Aggregation](https://docs.sypnia.com/architecture/aggregation) has the bounds and the function.

`migrations/0004_numbers.sql`

```sql
CREATE TABLE perf_pages_hourly (
    project_id  text        NOT NULL,
    environment text        NOT NULL,
    hour        timestamptz NOT NULL,
    route       text        NOT NULL,
    device      text        NOT NULL,          -- mobile | tablet | desktop
    views       integer NOT NULL DEFAULT 0,
    sessions    integer NOT NULL DEFAULT 0,    -- distinct sessions in the hour: exact, because the hour is re-read whole
    errors      integer NOT NULL DEFAULT 0,
    rage_clicks integer NOT NULL DEFAULT 0,
    dead_clicks integer NOT NULL DEFAULT 0,
    lcp_hist    integer[] NOT NULL,            -- 15 cells, milliseconds
    inp_hist    integer[] NOT NULL,            -- 15 cells
    ttfb_hist   integer[] NOT NULL,            -- 15 cells
    cls_hist    integer[] NOT NULL,            -- 10 cells
    PRIMARY KEY (project_id, environment, hour, route, device)
) PARTITION BY RANGE (hour);

CREATE TABLE perf_api_hourly (
    project_id  text        NOT NULL,
    environment text        NOT NULL,
    hour        timestamptz NOT NULL,
    method      text        NOT NULL,
    route       text        NOT NULL,
    requests        integer NOT NULL DEFAULT 0,   -- as the browser saw them
    ok              integer NOT NULL DEFAULT 0,   -- 200 to 399
    rejected        integer NOT NULL DEFAULT 0,   -- 401, 403, 404, 409, 422 and the other 4xx that are not failures
    failed          integer NOT NULL DEFAULT 0,   -- no answer, 400, 500 and above
    dur_hist        integer[] NOT NULL,
    server_requests integer NOT NULL DEFAULT 0,   -- as the server saw them
    server_failed   integer NOT NULL DEFAULT 0,
    timeouts        integer NOT NULL DEFAULT 0,
    db_calls        integer NOT NULL DEFAULT 0,
    db_ms           bigint  NOT NULL DEFAULT 0,
    server_dur_hist integer[] NOT NULL,
    PRIMARY KEY (project_id, environment, hour, method, route)
) PARTITION BY RANGE (hour);

-- The index of traces. The spans stay in the bucket: `segment` says which object holds them.
CREATE TABLE traces (
    project_id  text        NOT NULL,
    trace_id    text        NOT NULL,          -- 32 hex, a W3C trace id
    started_at  timestamptz NOT NULL,
    environment text        NOT NULL,
    service     text,
    method      text,
    route       text,
    status      integer,
    duration_ms integer     NOT NULL,
    spans       integer     NOT NULL DEFAULT 0,
    failed      boolean     NOT NULL DEFAULT false,
    partial     boolean     NOT NULL DEFAULT false,  -- a full record replaces a partial one
    sid         text,
    problem_id  text,
    segment     text        NOT NULL,
    PRIMARY KEY (project_id, started_at, trace_id)
) PARTITION BY RANGE (started_at);
```

The session index is one row per session, widened as more of the session arrives:

`migrations/0004_numbers.sql`

```sql
CREATE TABLE sessions (
    project_id  text        NOT NULL,
    sid         text        NOT NULL,          -- 32 hex, chosen by the SDK
    environment text        NOT NULL,
    started_at  timestamptz NOT NULL,
    ended_at    timestamptz NOT NULL,
    person_id   text,
    account_id  text,
    device_id   text,
    release     text,
    device      text,
    browser     text,
    os          text,
    locale      text,
    entry_route text,
    routes      text[]  NOT NULL DEFAULT '{}', -- in order of first visit, at most 50
    page_views  integer NOT NULL DEFAULT 0,
    clicks      integer NOT NULL DEFAULT 0,
    rage_clicks integer NOT NULL DEFAULT 0,
    dead_clicks integer NOT NULL DEFAULT 0,
    errors      integer NOT NULL DEFAULT 0,
    requests    integer NOT NULL DEFAULT 0,
    failed_requests integer NOT NULL DEFAULT 0,
    has_replay  boolean NOT NULL DEFAULT false,
    PRIMARY KEY (project_id, sid)
);
CREATE INDEX sessions_recent  ON sessions (project_id, environment, ended_at DESC);
CREATE INDEX sessions_person  ON sessions (project_id, person_id, ended_at DESC) WHERE person_id IS NOT NULL;
CREATE INDEX sessions_trouble ON sessions (project_id, environment, ended_at DESC)
    WHERE errors > 0 OR rage_clicks > 0 OR dead_clicks > 0 OR failed_requests > 0;
```

### Uptime, alerts, usage

`migrations/0005_alerts_usage.sql`

```sql
CREATE TABLE uptime_results (
    project_id text        NOT NULL,
    check_id   text        NOT NULL,           -- chk_…
    at         timestamptz NOT NULL,
    ok         boolean     NOT NULL,
    status     integer,                        -- the HTTP status, null when there was no answer
    ms         integer,
    reason     text,
    PRIMARY KEY (project_id, check_id, at)
) PARTITION BY RANGE (at);

CREATE TABLE alerts (
    id           text PRIMARY KEY,             -- alt_…
    project_id   text NOT NULL REFERENCES projects ON DELETE CASCADE,
    environment  text NOT NULL,
    kind         text NOT NULL,                -- problem.new, problem.regressed, problem.growing, uptime.down, …
    subject_type text NOT NULL,
    subject_id   text NOT NULL,
    dedupe_key   text NOT NULL,                -- one alert per cause: `problem.new:prb_8f3k2m9x4q7w`
    title        text NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    release_by   timestamptz NOT NULL,         -- the latest moment it may leave
    released_at  timestamptz,                  -- null while it waits for the agent's sentence
    acknowledged_at timestamptz,
    acknowledged_by text,
    UNIQUE (project_id, dedupe_key)
);

-- What a project used, per day. Written by the roll-up, read by quotas and by GET /v1/usage.
CREATE TABLE usage_daily (
    project_id text    NOT NULL,
    day        date    NOT NULL,
    events     bigint  NOT NULL DEFAULT 0,     -- counted events: errors, traces, performance samples
    events_all bigint  NOT NULL DEFAULT 0,     -- every event stored, counted or not
    replays    integer NOT NULL DEFAULT 0,     -- sessions with a recording
    analyses   integer NOT NULL DEFAULT 0,
    dropped_over_limit bigint NOT NULL DEFAULT 0,
    PRIMARY KEY (project_id, day)
);
```

The other tables follow the same patterns and are listed in the migrations: `people` and `accounts` (the names and e-mails your app sent with `identify()`), `releases`, `clicks_daily`, `click_elements_daily`, `vital_elements_daily`, `custom_events_daily`, `form_failures_daily`, `uptime_checks`, `alert_rules`, `webhooks`, `webhook_deliveries`, `analyses`, `trace_failures`, `held_failures`, `suggestions`.

## Partitions and retention

**Figure**: How long each thing lives by default. Problems outlive everything else, so the engine can still say «came back» months later.

A bar per kind of data on an axis of days that is not to scale. In Postgres: `ingest_batches` (the duplicate guard), 2 days; hourly and daily numbers, 400 days (`retention.numbers_days`); problems, with their sentence, status and daily counts, for as long as the project exists. In the Bucket: replay chunks, 30 days (`retention.replays_days`); raw events in segments, 90 days (`retention.events_days`); per-session files and the session index, 90 days; page snapshots, 90 days (`retention.snapshots_days`); Parquet, 90 days; source maps, 90 days after the release was last seen (`retention.sourcemaps_days`).

**Partitions.** The tables that grow with time and are read by time are partitioned by month: `perf_pages_hourly`, `perf_api_hourly`, `traces`, `uptime_results` and the daily click tables. The daily job creates the partitions of the next two months and drops the ones older than `retention.numbers_days`. Dropping a partition is instant and leaves nothing to vacuum.

The engine creates partitions itself, in plain SQL, and needs no extension. The alternative was `pg_partman` or `pg_cron`, which many managed Postgres services do not offer. `sessions` and `problems` are not partitioned: they are updated by key, and a partition key in their primary key would make every upsert name a date.

**Retention.** One job a day, per project:

| Data                                                      | How it is removed                                                                                                                                                                                      |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Segments, Parquet, snapshots                              | Whole `dt=` folders older than the limit are listed and deleted.                                                                                                                                       |
| Per-session files, replay chunks                          | The job reads expired rows of `sessions`, deletes the two folders of each `sid`, then deletes the row. For replays past their own, shorter limit it deletes the chunks and sets `has_replay` to false. |
| Source maps                                               | Releases not seen for 90 days lose their folder.                                                                                                                                                       |
| Hourly and daily numbers                                  | Old partitions are dropped.                                                                                                                                                                            |
| `ingest_batches`, old ledger rows, delivered webhook rows | Deleted by age.                                                                                                                                                                                        |
| Problems                                                  | Never, while the project exists. Their example occurrence loses its session link when the session expires.                                                                                             |

The job deletes by listing because retention is the engine's rule, not the store's. It works the same on Amazon S3, Cloudflare R2 and MinIO, and it lets Sypnia Cloud give each plan its own number of days **\[Cloud]**. On AWS you may add a lifecycle rule as a second guard; the template adds one for unfinished uploads only.

## How much a million events weigh

These are estimates. The first row uses a figure measured on the system this engine comes from: about 16 MB of gzip NDJSON a day for 110 to 150 thousand records a day.

| Data                  | Arithmetic                                                                                                                                                                                                                                                | Per million events                       |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| Segments              | 16 MB ÷ 130,000 records ≈ 123 bytes an event, compressed                                                                                                                                                                                                  | ≈ 125 MB                                 |
| Per-session files     | The same events again, grouped by session; server records without a session are left out                                                                                                                                                                  | ≤ 125 MB                                 |
| Parquet               | Not measured. Columnar files with ZSTD are usually smaller than gzip JSON; we budget the same size until we measure it                                                                                                                                    | ≤ 125 MB                                 |
| Replay chunks         | Measured there: a short recorded session is 22 chunks, 751 KB. About 7,700 sessions per million events (1,000 sessions a day at 130,000 records). With 10 % recorded from the start and every session with trouble, budget 15 %: 1,150 sessions × 0.75 MB | ≈ 870 MB                                 |
| Page snapshots        | Measured there: 70 to 107 KB each. They grow with routes, not with events                                                                                                                                                                                 | a few MB                                 |
| **Bucket, total**     |                                                                                                                                                                                                                                                           | **≈ 1.2 GB**, most of it replays         |
| `sessions`            | 7,700 rows × about 600 bytes, twice that with indexes                                                                                                                                                                                                     | ≈ 10 MB                                  |
| Hourly numbers        | They grow with routes, not events: 100 routes × 3 devices × 24 hours × about 400 bytes                                                                                                                                                                    | ≈ 3 MB a day, whatever the traffic       |
| `ingest_batches`      | About 230,000 batches (30 batches per session) × about 120 bytes, kept 48 hours                                                                                                                                                                           | ≈ 28 MB, temporary                       |
| Problems and the rest | Thousands of rows at most                                                                                                                                                                                                                                 | under 5 MB                               |
| **Postgres, total**   |                                                                                                                                                                                                                                                           | **≈ 20 MB** kept, plus ≈ 30 MB temporary |

Two things can break these numbers, and both are capped. A long session can record up to 720 chunks of 200 KB, 144 MB; measured there, one recording reached 158 MB before the cap existed. And source maps are per release, not per event: measured there, one deploy of a large app is 965 maps and 109 MB.

At 1.2 GB per million events, 90 days of a project that sends a million events a month is about 3.6 GB in the bucket. Storage is not what a deployment pays for. Writes are: [Deployment shapes](https://docs.sypnia.com/architecture/deployment-shapes) has that arithmetic.

## What is never stored

| Never stored                                                             | Because                                                                                                                                       |
| ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| What a person types                                                      | The SDK masks it before anything leaves the browser. The engine never receives it.                                                            |
| Request and response bodies, cookies, headers, query strings, URL hashes | No event has a field for them. `url` is a path.                                                                                               |
| IP addresses with events                                                 | Ingest uses the address for the rate limit, in memory, and does not write it to an event, an object or a row. There is no geolocation.        |
| Raw events in Postgres                                                   | Rule 1 of the [overview](https://docs.sypnia.com/architecture).                                                                               |
| Replay content in Postgres                                               | A session row has `has_replay` and nothing else about the recording.                                                                          |
| An API token, the root token, the AI key, a webhook secret in plain text | Tokens are stored as a SHA-256 hash; the others live in the configuration and are never printed, logged or returned.                          |
| Span attributes that look like secrets                                   | Attributes whose name contains `authorization`, `cookie`, `password`, `secret` or `token` are removed at ingest.                              |
| What was sent to the model                                               | The engine stores the analysis and its token counts, not the context. It can be rebuilt on request with `GET /v1/problems/{problem}/context`. |

Names and e-mails are stored only when your app sends them with `identify()`. They live in `people` and `accounts`, and are deleted with `DELETE /v1/people/{person_id}`. See [What is collected](https://docs.sypnia.com/privacy/what-is-collected).

## Next
