# Ingest

> How the engine receives events, replay chunks, page snapshots, source maps and OTLP traces, what it checks, how it turns batches into objects, how it handles duplicates, and why it does not query the database to do it.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/architecture/ingest/

Ingest is the door of the engine. It receives what the SDKs send, checks it, and writes it to the Bucket. It analyses nothing. That is what keeps it fast, cheap and hard to break.

This page is the design behind the [Ingest API](https://docs.sypnia.com/api/ingest). The API page says what to send; this one says what happens to it.

> **Note: About the code and the numbers**
>
> The Rust here is a design excerpt; the repository opens at launch. Limits are the ones in the API and event contracts. Times are budgets, not benchmarks.

## The HTTP surface

| Endpoint                   | Body                                                  | Limit                                                     | Goes to                                                  |
| -------------------------- | ----------------------------------------------------- | --------------------------------------------------------- | -------------------------------------------------------- |
| `POST /v1/ingest/events`   | A batch: up to 200 events and a `batch_id`            | 2 MiB after decompression                                 | The buffer, then a segment under `events/`               |
| `POST /v1/ingest/replay`   | One replay chunk                                      | 4,000,000 bytes after decompression; 720 chunks a session | One object under `replays/`, written before the answer   |
| `POST /v1/ingest/snapshot` | One page snapshot                                     | 4,000,000 bytes after decompression                       | One object under `snapshots/`                            |
| `GET /v1/ingest/config`    | —                                                     | —                                                         | Answered from the cache: recording rules and `slow_down` |
| `POST /v1/otlp/v1/traces`  | An OTLP `ExportTraceServiceRequest`, protobuf or JSON | 4,000,000 bytes; 2,000 spans                              | Mapped to events, then the same buffer                   |

All five take an ingest key, `spk_…`, in `Authorization: Bearer` or, for `navigator.sendBeacon`, in the `key` query parameter. An API token sent here is refused with `403 wrong_credential`, so a secret that ends up in a browser by mistake does not work quietly.

Source maps do not pass through ingest. Your CI asks the API for upload links with an API token, and uploads each file straight to the Bucket. See [the source-map section below](#source-maps-and-replay-chunks).

## The gates

A batch passes seven gates, in this order. Each one that fails answers at once with one error code, and the cheapest checks come first.

**Figure (diagram not drawn yet)**: The seven gates of a batch. Gates 1 to 5 can refuse the whole request; gates 6 and 7 drop single events and say so in the answer.

It will show: FLOW DIAGRAM, left to right, a single line of seven gate-shaped nodes (a gate = a small rectangle with a notch or a thin vertical bar on its left), then two end nodes. On the far left the node «Your app (SDK)» with an arrow labelled «POST /v1/ingest/events» into gate 1. Gates and exact labels: 1 «Key» (sub-label «cache lookup»); 2 «Origin»; 3 «Rate limit» (sub-label «per key and address, in memory»); 4 «Decode» (sub-label «gzip by its first bytes, size cap»); 5 «Shape» (sub-label «events is a list of at most 200»); 6 «Each event» (sub-label «cut, clean, drop, stamp»); 7 «Quota» (sub-label «slow_down from the cache»). After gate 7 the node «Buffer», then an arrow labelled «one segment, at most 2 s later» to the node «Bucket», then a return arrow from Bucket back along the top to «Your app (SDK)» labelled «200 { accepted, dropped, slow_down }». UNDER each gate a short downward arrow ending in a small error tag with exact text: gate 1 «401 unauthorized · 403 wrong_credential»; gate 2 «403 origin_not_allowed»; gate 3 «429 rate_limited»; gate 4 «400 invalid_body · 413 payload_too_large»; gate 5 «400 too_many_events»; gate 6 «no error: counted in dropped»; gate 7 «no error: counted in dropped_reasons.limit_reached»; under Buffer «503 unavailable». A bracket over gates 1 to 7 labelled «no database query». The lit element is that bracket. Numbered callouts (1) to (7) on the gates.

### (1) The key

The key is looked up in a cache held in memory, not in Postgres. A key that is unknown or revoked gets `401 unauthorized`.

Notice the three ages in the lookup below. A hit is used as it is for 60 seconds. A miss asks Postgres once and remembers «no such key» for 60 seconds, so a wrong key cannot be used to hammer the database. And when Postgres does not answer, an entry up to one hour old is still good.

`crates/sypnia-ingest/src/project_cache.rs`

```rust
impl ProjectCache {
    /// The project behind an ingest key. One hash-map read on the hot path.
    pub async fn lookup(&self, key: &IngestKey) -> Result<Arc<ProjectEntry>, IngestError> {
        let now = self.clock.now();
        if let Some(slot) = self.entries.get(key) {
            match slot.value() {
                Slot::Known(entry) if entry.loaded_at + FRESH > now => return Ok(entry.clone()),
                Slot::Unknown { since } if *since + NEGATIVE > now => return Err(IngestError::Unauthorized),
                _ => {} // stale: refresh below
            }
        }
        match self.db.project_for_key(key).await {
            Ok(Some(entry)) => {
                let entry = Arc::new(entry);
                self.entries.insert(key.clone(), Slot::Known(entry.clone()));
                Ok(entry)
            }
            Ok(None) => {
                self.entries.insert(key.clone(), Slot::Unknown { since: now });
                Err(IngestError::Unauthorized)
            }
            // Postgres is down: keep serving what we knew, for at most an hour.
            Err(_) => match self.entries.get(key).as_deref() {
                Some(Slot::Known(entry)) if entry.loaded_at + STALE_OK > now => Ok(entry.clone()),
                _ => Err(IngestError::Unavailable),
            },
        }
    }
}
```

A long-running process also refreshes every entry in the background each 60 seconds, so in practice the request never waits for Postgres. That is how a revoked key is refused within one minute, as [Projects, environments, keys and tokens](https://docs.sypnia.com/concepts/projects-keys-and-tokens) promises.

### (2) The origin

If the request has an `Origin` header and the key has a list of allowed origins, the origin must be in the list, or the answer is `403 origin_not_allowed`. A request with no `Origin`, such as one from a server SDK, passes: a key used from a server is protected by its rate limit, not by a header the sender controls.

One layer owns CORS, and it is the engine. The system this comes from learned why: its function URL and its handler both added CORS headers, the browser rejected the answer **after** the data was stored, the SDK retried, and batches arrived twice. In the AWS template the address in front of the functions adds no CORS headers of its own.

### (3) The rate limit

Each key has a number of requests a minute from one address (default 600, `rate_limit.ingest_per_minute`). The counter is a token bucket in the memory of the process. The address is read from the connection, or from the forwarding header when `trusted_proxy_hops` says a proxy is in front. It is used for this count and nowhere else: it is not written to an event, a log line or a row.

Because the counter is per process, several processes or functions each count on their own, and the real limit is higher than the setting. The alternative is a shared counter in Postgres or in another service, on the path of every request. A rate limit is a guard against a runaway loop, not a bill; an approximate one is enough.

### (4) Decoding

The body may be gzip. Ingest detects it from the first two bytes, `1f 8b`, whatever the `Content-Type` says, because `sendBeacon` cannot set `Content-Encoding`. It decompresses as a stream and stops at the limit, so a small body that expands to gigabytes costs 2 MiB of memory and a `413`, not the process.

### (5) and (6) The shape, then each event

`events` must be a list of at most 200. After that, each event is judged on its own. One bad event never costs the rest of the batch.

| What ingest finds                                                                                 | What it does                                                    |
| ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| A field the contract does not list                                                                | Keeps it, as sent                                               |
| A `type` the contract does not list                                                               | Keeps the event. Only the parts that know the type will read it |
| A string longer than its limit                                                                    | Cuts it to the limit                                            |
| A listed field with the wrong type or a value outside its list                                    | Removes the field, keeps the event                              |
| A required field missing or unusable: no `type`, `ts` not positive, a browser event without `sid` | Drops the event: `invalid`                                      |
| A vital outside its plausible range (an LCP over 120 s, an INP over 60 s, a CLS over 10)          | Drops the event: `invalid`                                      |
| `ts` more than 7 days in the past                                                                 | Drops the event: `too_old`                                      |
| An event from a route in the project's blocked routes                                             | Drops the event: `blocked_route`                                |
| A span attribute whose name contains `authorization`, `cookie`, `password`, `secret` or `token`   | Removes the attribute                                           |
| `prj`, `rcv` or `bid` sent by the app                                                             | Replaces them with the engine's own                             |

The plausibility caps exist twice on purpose, in the SDK and here. Measured on the system this comes from: two browsers once reported INP values of 5 to 7 days and dragged a p75 to the top of the scale.

### (7) The quota

The cache entry carries the project's `slow_down`, decided elsewhere (see [Data flow](https://docs.sypnia.com/architecture/data-flow)). When it says `send: failures`, ingest keeps errors and failed requests and counts the rest as `limit_reached`. The answer carries the same `slow_down`, so the SDK stops sending what would be dropped.

## The handler

This is the whole request, in order. Notice two things: nothing in it awaits Postgres, and the last `await` returns only when the segment that carries these events is in the Bucket.

`crates/sypnia-ingest/src/events.rs`

```rust
pub async fn ingest_events(
    State(ingest): State<Arc<Ingest>>,
    key: IngestKeyAuth,                       // Bearer header or ?key=; an sps_ token is 403 wrong_credential
    origin: Option<TypedHeader<Origin>>,
    ClientAddr(addr): ClientAddr,
    body: Bytes,
) -> Result<Json<IngestResult>, IngestError> {
    let project = ingest.projects.lookup(&key.0).await?;            // (1) 401
    project.check_origin(origin.as_deref())?;                       // (2) 403
    ingest.limiter.check(&project.key_id, addr)?;                   // (3) 429
    let batch: RawBatch = decode(&body, MAX_BATCH_BYTES)?;          // (4) 400, 413
    if batch.events.len() > MAX_EVENTS_PER_BATCH {
        return Err(IngestError::TooManyEvents);                     // (5) 400
    }

    let now = ingest.clock.now();
    // No batch_id? The body's own hash stands in, so an identical retry is still caught.
    let batch_id = batch.batch_id.unwrap_or_else(|| BatchId::from_content(&body));
    let mut lines = Lines::new(&project.id, &batch_id, now);
    let mut dropped = Dropped::default();

    for raw in batch.events {
        match validate(raw, &project, now) {                        // (6) cut, clean, stamp
            Ok(event) if project.slow_down.allows(&event) => lines.push(&event),
            Ok(_) => dropped.add(Reason::LimitReached),             // (7)
            Err(reason) => dropped.add(reason),
        }
    }

    let accepted = lines.len();
    if accepted > 0 {
        // Returns when the segment holding these lines exists. 503 if it could not be written.
        ingest.intake.append(lines).await?;
    }
    Ok(Json(IngestResult {
        accepted,
        dropped: dropped.total(),
        dropped_reasons: dropped.into_map(),
        slow_down: project.slow_down.clone(),
    }))
}
```

## From batches to objects

One object per batch would be the simplest design and the most expensive one. Measured on the system this comes from: object writes cost more than object storage. So batches are grouped.

**Figure (diagram not drawn yet)**: Group commit. Batches that arrive within the same 2 seconds leave in one object, and all of them are answered when it exists.

It will show: DIAGRAM OF A GROUP COMMIT, time flowing left to right over a 2-second window marked «0 s» at the left edge and «2 s» at the right edge. On the left, three source nodes stacked: «browser A», «browser B», «shop-api (server SDK)». From each, one or two small batch tiles travel right into a wide open container in the centre labelled «Buffer: prj_5k2m9x4q7w1z · production». Exact tile labels: from browser A «batch 0b9f… · 6 events» arriving at 0.2 s; from shop-api «batch 5d0c… · 2 events» at 0.6 s; from browser B «batch 71aa… · 11 events» at 1.4 s. Each tile, once inside the container, has a thin dotted line going back to its source, ending in a small clock icon labelled «waiting». At the 2 s edge the container closes and becomes one file tile labelled «20260312T091216Z-7f3a9c.ndjson.gz · 19 events · 3 batches» with an arrow labelled «one PUT» into the node «Bucket». From the Bucket, three return arrows fan back to the three sources, each labelled «200», drawn in the accent colour: this fan is the lit element. Below the container two small trigger tags: «closes at 2 s» and «or at 4 MiB, whichever comes first». Below the Bucket a dashed arrow to a small node «Postgres» labelled «ledger row, after the answers». Numbered callouts: (1) on the container, (2) on the PUT arrow, (3) on the fan of answers, (4) on the dashed ledger arrow.

The buffer (1) is one open segment per project and environment. It closes when the first of two things happens:

| Trigger                   | Value     | Why this value                                                                    |
| ------------------------- | --------- | --------------------------------------------------------------------------------- |
| Time since its first line | 2 seconds | Small against the 120-second budget, and at most 30 objects a minute per project  |
| Size before compression   | 4 MiB     | A worker reads a segment whole into memory to scan it; this keeps that read small |

A third limit protects the process: if more than 32 MiB of lines are waiting to be written, across all projects, new batches get `503 unavailable` with `Retry-After: 2` until the Bucket catches up.

When a segment closes it is written with one `PUT` (2), and every request that put lines in it is answered (3). Then, off the request path, one row goes into the `segments` ledger (4). If that insert fails it is kept and tried again; a worker that starts also lists the last two hours of each project in the Bucket and adds any segment the ledger does not know.

Notice in the excerpt that the lock is never held across an `await`, and that every waiter gets the result of the same write: either all the batches of a segment are stored, or none is and each SDK retries its own.

`crates/sypnia-ingest/src/buffer.rs`

```rust
#[async_trait]
impl Intake for Buffer {
    async fn append(&self, lines: Lines) -> Result<(), IntakeError> {
        let (done, wait) = oneshot::channel();
        let scope = lines.scope();
        let full = {
            let mut open = self.open.lock(); // parking_lot: short, never held across an await
            if self.waiting_bytes.load(Ordering::Relaxed) + lines.bytes() > MAX_WAITING_BYTES {
                return Err(IntakeError::Full); // 503 with Retry-After
            }
            let segment = open
                .entry(scope.clone())
                .or_insert_with(|| OpenSegment::new(self.clock.now()));
            segment.push(lines, done);
            segment.bytes() >= MAX_SEGMENT_BYTES
        };
        if full {
            self.flush(&scope).await; // the size trigger; a timer task covers the 2 seconds
        }
        wait.await.map_err(|_| IntakeError::Closed)?
    }
}

impl Buffer {
    async fn flush(&self, scope: &Scope) {
        let Some(segment) = self.open.lock().remove(scope) else { return };
        let key = Keys::new(&self.prefix, scope).segment(self.clock.now(), self.writer);
        let (body, summary, waiters) = segment.finish(); // gzip, counts, batch ids

        let result = self.store.put(&key, body, PutOptions::ndjson_gzip()).await;
        if result.is_ok() {
            self.ledger.register(SegmentRow::new(scope, &key, summary)); // queued, retried, never awaited here
        }
        for waiter in waiters {
            let _ = waiter.send(result.clone().map_err(IntakeError::from));
        }
    }
}
```

### The same thing on AWS

A function cannot keep an open segment between two requests, and many copies of it run at once. So `Intake` has a second implementation there: a queue.

|                          | One process (`serve`)                             | AWS functions                                                                  |
| ------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------ |
| `Intake::append`         | Adds to the in-memory buffer, waits for the `PUT` | Sends the validated lines as one queue message, waits for the queue to confirm |
| Who writes the segment   | The same process, 2 seconds later                 | The worker function, woken by the queue with up to 10 seconds of messages      |
| The answer means         | The events are in the Bucket                      | The events are in the queue, which keeps them for 4 days                       |
| Segment name and content | Identical                                         | Identical                                                                      |

A batch larger than 192 KiB after compression does not go into a message: ingest writes it as its own segment and sends a message that only names it. The alternative for AWS was a managed delivery stream, as in the system this comes from. It was left out because it cannot name objects by project without an extra feature that buffers for at least 60 seconds, half of the 2-minute budget.

## Duplicates and idempotency

Networks retry. A browser can send a batch, lose the answer and send it again. Each kind of upload has one rule that makes the second arrival harmless.

| Upload                     | What makes it idempotent                                                                                                                                                                                                                                     |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| A batch of events          | `batch_id`. Every stored line carries it as `bid`. The first worker that scans a segment records its batch ids; a batch id seen before, in any segment of the last 48 hours, is skipped by every reader. The lines stay in the object; they are not counted. |
| A batch without `batch_id` | Ingest uses the SHA-256 of the body as the id. A retry of the same bytes is caught; the same events re-sent in a different body are not.                                                                                                                     |
| A replay chunk             | The key is `replays/<sid>/<seq>`. The same `seq` again replaces the object.                                                                                                                                                                                  |
| A page snapshot            | The key ends in the snapshot's own `ts`. The same snapshot again replaces the object.                                                                                                                                                                        |
| An OTLP request            | The SHA-256 of the body. OTLP exporters retry with the same bytes.                                                                                                                                                                                           |

Dedupe happens when segments are read, not when batches are received, for one reason: receiving must not depend on Postgres. The price is that a duplicated batch takes space in the Bucket. That is cheap; a doubled count is not.

## Source maps and replay chunks

**Replay chunks** go through ingest, one request per chunk, because ingest must check three things before storing one: that the route is not in the project's blocked routes (`403 not_recordable`), that the session has fewer than 720 chunks (`422 session_full`), and that the project is inside its replays limit (`429 limit_reached`). The chunk is written before the answer, with `Content-Type: application/json` and `Content-Encoding: gzip`. Ingest does not look inside it.

The 4,000,000-byte limit is not arbitrary. On AWS a function receives at most 6 MB in one request; the SDK's chunks of 200 KB and this cap keep a good margin under it.

**Source maps** never pass through ingest. A deploy of a large app is hundreds of files and, measured on the system this comes from, about 109 MB. So the upload is in three steps, all with an API token of scope `admin`:

1. Your CI calls `POST /v1/releases/{version}/sourcemaps` with the list of files.
2. The API answers with one signed upload link per file, valid for 15 minutes (`storage.presign_ttl_seconds`).
3. Your CI uploads each file straight to the Bucket, under `sourcemaps/<release>/`, and the `index.json` that maps each built file to its map.

`sypnia sourcemaps upload` does the three steps for you.

## OTLP intake

`POST /v1/otlp/v1/traces` accepts what any OpenTelemetry exporter sends and turns it into the same events a Sypnia server SDK would have sent. After this mapping nothing downstream knows the difference.

| In the OTLP request                                                            | Becomes                                                                                                                                                             |
| ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Resource `service.name`                                                        | `service`                                                                                                                                                           |
| Resource `service.version`                                                     | `ver`, the release                                                                                                                                                  |
| Resource `deployment.environment.name` (or the older `deployment.environment`) | `env`; `production` when missing                                                                                                                                    |
| A span of kind `SERVER`                                                        | One `server_request`: `tid`, `span`, `parent`, `method`, `route` from `http.route`, `status`, `dur`                                                                 |
| Spans of the same trace and service under it                                   | Its `spans[]`: `db` when the span has `db.system.name`, `http` when it has `http.request.method`, else `internal`; at most 200 kept, the rest counted in `span_n`   |
| Span status `ERROR` on a child span                                            | That span's `status: error`, with `code` from `http.response.status_code` or `db.response.status_code`                                                              |
| An `exception` span event                                                      | One `server_error`: `name` from `exception.type`, `msg` from `exception.message`, `stack` from `exception.stacktrace`. The engine computes `fp`, because no SDK did |
| Span attribute `session.id` (32 hex characters)                                | `sid`: the request joins that session's timeline                                                                                                                    |
| Other span events, links, metrics, logs                                        | Dropped. Refused spans are counted in `partialSuccess.rejectedSpans`                                                                                                |

Spans of one trace can arrive in different requests. Ingest does not wait for them: each request is mapped on its own, and a `SERVER` span that arrives without its children is stored as it is. The roll-up joins what shares a `tid` when it builds the `traces` index.

## Why ingest does not query Postgres

Three reasons, in order of weight.

1. **Availability.** If receiving needed the database, every database problem would be data loss. As designed, a Postgres outage costs freshness and nothing else.
2. **Connections.** The system this comes from once exhausted its connection pooler: each copy of a request-serving function opened its own pool. Ingest is the most copied part of the engine; keeping it off the database removes the largest source of connections.
3. **Latency.** A hash-map read takes less than a microsecond. A query takes a network round trip, on every batch from every browser.

What ingest reads instead is small, and all of it comes from the cache entry of the key:

| Cached per ingest key                        | Used for                                         | How fresh |
| -------------------------------------------- | ------------------------------------------------ | --------- |
| Project id, key id, `revoked_at`             | Gate 1                                           | 60 s      |
| `allowed_origins`                            | Gate 2                                           | 60 s      |
| `rate_limit_per_minute`                      | Gate 3                                           | 60 s      |
| Blocked routes, replay rules, sampling rates | Gate 6, `/v1/ingest/config`                      | 60 s      |
| `slow_down`                                  | Gate 7 and the answer                            | 60 s      |
| `has_server`                                 | Nothing in ingest; it rides along for the worker | 60 s      |

The only write ingest makes to Postgres is the ledger row: **one row per flushed segment**, not one per request and not one per event. It happens after the answer, on one connection per process, and it is allowed to fail. A request never waits on Postgres.

If Postgres is down when a segment is flushed, the segment is still written to the Bucket and the SDKs are still answered. The ledger row is kept in memory and retried; if the process dies before it succeeds, a worker recovers it by listing the Bucket and adding every segment the ledger does not know. Until then the events are safe and not yet counted.

### Logs

Ingest writes one JSON line per request: status, events accepted and dropped, bytes, milliseconds. Never the key, the address or the body. Requests that succeed with nothing dropped are sampled at 1 %. Measured on the system this comes from: logging every success cost about 18 MB of logs a day for one app.

## What changed from the system this comes from

| There                                                             | Here                                                  | Why                                                                      |
| ----------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------ |
| A signed-in user's token, verified against the app's auth service | A public ingest key, allowed origins and a rate limit | Brief §10.2: the ingest key is the only credential for sending at launch |
| No dedupe: a retried batch was stored and counted twice           | `batch_id` on every line                              | The CORS incident of 2026-09-30                                          |
| No rate limit, no quota                                           | Both                                                  | A public key needs them                                                  |
| No limit on how old `ts` may be                                   | 7 days                                                | An old clock or a stuck queue must not rewrite last month                |
| Server records skipped ingest and wrote to the stream directly    | Everything goes through ingest                        | One place stamps the project; a server needs only a key and a URL        |
| Unknown fields removed by an allow-list                           | Unknown fields kept                                   | Brief §10.4: schema on read                                              |

## Next
