Ingest
How the engine receives events, replay chunks, page snapshots, source maps and OTLP traces, what it checks, how it turns batches into objects, how it handles duplicates, and why it does not query the database to do it.
On this page
- The HTTP surface
- The gates
- (1) The key
- (2) The origin
- (3) The rate limit
- (4) Decoding
- (5) and (6) The shape, then each event
- (7) The quota
- The handler
- From batches to objects
- The same thing on AWS
- Duplicates and idempotency
- Source maps and replay chunks
- OTLP intake
- Why ingest does not query Postgres
- Logs
- What changed from the system this comes from
- Next
Ingest is the door of the engine. It receives what the SDKs send, checks it, and writes it to the Bucket. It analyses nothing. That is what keeps it fast, cheap and hard to break.
This page is the design behind the Ingest API. The API page says what to send; this one says what happens to it.
The HTTP surface
| Endpoint | Body | Limit | Goes to |
|---|---|---|---|
POST /v1/ingest/events | A batch: up to 200 events and a batch_id | 2 MiB after decompression | The buffer, then a segment under events/ |
POST /v1/ingest/replay | One replay chunk | 4,000,000 bytes after decompression; 720 chunks a session | One object under replays/, written before the answer |
POST /v1/ingest/snapshot | One page snapshot | 4,000,000 bytes after decompression | One object under snapshots/ |
GET /v1/ingest/config | — | — | Answered from the cache: recording rules and slow_down |
POST /v1/otlp/v1/traces | An OTLP ExportTraceServiceRequest, protobuf or JSON | 4,000,000 bytes; 2,000 spans | Mapped to events, then the same buffer |
All five take an ingest key, spk_…, in Authorization: Bearer or, for navigator.sendBeacon, in the key query parameter. An API token sent here is refused with 403 wrong_credential, so a secret that ends up in a browser by mistake does not work quietly.
Source maps do not pass through ingest. Your CI asks the API for upload links with an API token, and uploads each file straight to the Bucket. See the source-map section below.
The gates
A batch passes seven gates, in this order. Each one that fails answers at once with one error code, and the cheapest checks come first.
(1) The key
The key is looked up in a cache held in memory, not in Postgres. A key that is unknown or revoked gets 401 unauthorized.
Notice the three ages in the lookup below. A hit is used as it is for 60 seconds. A miss asks Postgres once and remembers «no such key» for 60 seconds, so a wrong key cannot be used to hammer the database. And when Postgres does not answer, an entry up to one hour old is still good.
impl ProjectCache {
/// The project behind an ingest key. One hash-map read on the hot path.
pub async fn lookup(&self, key: &IngestKey) -> Result<Arc<ProjectEntry>, IngestError> {
let now = self.clock.now();
if let Some(slot) = self.entries.get(key) {
match slot.value() {
Slot::Known(entry) if entry.loaded_at + FRESH > now => return Ok(entry.clone()),
Slot::Unknown { since } if *since + NEGATIVE > now => return Err(IngestError::Unauthorized),
_ => {} // stale: refresh below
}
}
match self.db.project_for_key(key).await {
Ok(Some(entry)) => {
let entry = Arc::new(entry);
self.entries.insert(key.clone(), Slot::Known(entry.clone()));
Ok(entry)
}
Ok(None) => {
self.entries.insert(key.clone(), Slot::Unknown { since: now });
Err(IngestError::Unauthorized)
}
// Postgres is down: keep serving what we knew, for at most an hour.
Err(_) => match self.entries.get(key).as_deref() {
Some(Slot::Known(entry)) if entry.loaded_at + STALE_OK > now => Ok(entry.clone()),
_ => Err(IngestError::Unavailable),
},
}
}
}A long-running process also refreshes every entry in the background each 60 seconds, so in practice the request never waits for Postgres. That is how a revoked key is refused within one minute, as Projects, environments, keys and tokens promises.
(2) The origin
If the request has an Origin header and the key has a list of allowed origins, the origin must be in the list, or the answer is 403 origin_not_allowed. A request with no Origin, such as one from a server SDK, passes: a key used from a server is protected by its rate limit, not by a header the sender controls.
One layer owns CORS, and it is the engine. The system this comes from learned why: its function URL and its handler both added CORS headers, the browser rejected the answer after the data was stored, the SDK retried, and batches arrived twice. In the AWS template the address in front of the functions adds no CORS headers of its own.
(3) The rate limit
Each key has a number of requests a minute from one address (default 600, rate_limit.ingest_per_minute). The counter is a token bucket in the memory of the process. The address is read from the connection, or from the forwarding header when trusted_proxy_hops says a proxy is in front. It is used for this count and nowhere else: it is not written to an event, a log line or a row.
Because the counter is per process, several processes or functions each count on their own, and the real limit is higher than the setting. The alternative is a shared counter in Postgres or in another service, on the path of every request. A rate limit is a guard against a runaway loop, not a bill; an approximate one is enough.
(4) Decoding
The body may be gzip. Ingest detects it from the first two bytes, 1f 8b, whatever the Content-Type says, because sendBeacon cannot set Content-Encoding. It decompresses as a stream and stops at the limit, so a small body that expands to gigabytes costs 2 MiB of memory and a 413, not the process.
(5) and (6) The shape, then each event
events must be a list of at most 200. After that, each event is judged on its own. One bad event never costs the rest of the batch.
| What ingest finds | What it does |
|---|---|
| A field the contract does not list | Keeps it, as sent |
A type the contract does not list | Keeps the event. Only the parts that know the type will read it |
| A string longer than its limit | Cuts it to the limit |
| A listed field with the wrong type or a value outside its list | Removes the field, keeps the event |
A required field missing or unusable: no type, ts not positive, a browser event without sid | Drops the event: invalid |
| A vital outside its plausible range (an LCP over 120 s, an INP over 60 s, a CLS over 10) | Drops the event: invalid |
ts more than 7 days in the past | Drops the event: too_old |
| An event from a route in the project’s blocked routes | Drops the event: blocked_route |
A span attribute whose name contains authorization, cookie, password, secret or token | Removes the attribute |
prj, rcv or bid sent by the app | Replaces them with the engine’s own |
The plausibility caps exist twice on purpose, in the SDK and here. Measured on the system this comes from: two browsers once reported INP values of 5 to 7 days and dragged a p75 to the top of the scale.
(7) The quota
The cache entry carries the project’s slow_down, decided elsewhere (see Data flow). When it says send: failures, ingest keeps errors and failed requests and counts the rest as limit_reached. The answer carries the same slow_down, so the SDK stops sending what would be dropped.
The handler
This is the whole request, in order. Notice two things: nothing in it awaits Postgres, and the last await returns only when the segment that carries these events is in the Bucket.
pub async fn ingest_events(
State(ingest): State<Arc<Ingest>>,
key: IngestKeyAuth, // Bearer header or ?key=; an sps_ token is 403 wrong_credential
origin: Option<TypedHeader<Origin>>,
ClientAddr(addr): ClientAddr,
body: Bytes,
) -> Result<Json<IngestResult>, IngestError> {
let project = ingest.projects.lookup(&key.0).await?; // (1) 401
project.check_origin(origin.as_deref())?; // (2) 403
ingest.limiter.check(&project.key_id, addr)?; // (3) 429
let batch: RawBatch = decode(&body, MAX_BATCH_BYTES)?; // (4) 400, 413
if batch.events.len() > MAX_EVENTS_PER_BATCH {
return Err(IngestError::TooManyEvents); // (5) 400
}
let now = ingest.clock.now();
// No batch_id? The body's own hash stands in, so an identical retry is still caught.
let batch_id = batch.batch_id.unwrap_or_else(|| BatchId::from_content(&body));
let mut lines = Lines::new(&project.id, &batch_id, now);
let mut dropped = Dropped::default();
for raw in batch.events {
match validate(raw, &project, now) { // (6) cut, clean, stamp
Ok(event) if project.slow_down.allows(&event) => lines.push(&event),
Ok(_) => dropped.add(Reason::LimitReached), // (7)
Err(reason) => dropped.add(reason),
}
}
let accepted = lines.len();
if accepted > 0 {
// Returns when the segment holding these lines exists. 503 if it could not be written.
ingest.intake.append(lines).await?;
}
Ok(Json(IngestResult {
accepted,
dropped: dropped.total(),
dropped_reasons: dropped.into_map(),
slow_down: project.slow_down.clone(),
}))
}From batches to objects
One object per batch would be the simplest design and the most expensive one. Measured on the system this comes from: object writes cost more than object storage. So batches are grouped.
The buffer (1) is one open segment per project and environment. It closes when the first of two things happens:
| Trigger | Value | Why this value |
|---|---|---|
| Time since its first line | 2 seconds | Small against the 120-second budget, and at most 30 objects a minute per project |
| Size before compression | 4 MiB | A worker reads a segment whole into memory to scan it; this keeps that read small |
A third limit protects the process: if more than 32 MiB of lines are waiting to be written, across all projects, new batches get 503 unavailable with Retry-After: 2 until the Bucket catches up.
When a segment closes it is written with one PUT (2), and every request that put lines in it is answered (3). Then, off the request path, one row goes into the segments ledger (4). If that insert fails it is kept and tried again; a worker that starts also lists the last two hours of each project in the Bucket and adds any segment the ledger does not know.
Notice in the excerpt that the lock is never held across an await, and that every waiter gets the result of the same write: either all the batches of a segment are stored, or none is and each SDK retries its own.
#[async_trait]
impl Intake for Buffer {
async fn append(&self, lines: Lines) -> Result<(), IntakeError> {
let (done, wait) = oneshot::channel();
let scope = lines.scope();
let full = {
let mut open = self.open.lock(); // parking_lot: short, never held across an await
if self.waiting_bytes.load(Ordering::Relaxed) + lines.bytes() > MAX_WAITING_BYTES {
return Err(IntakeError::Full); // 503 with Retry-After
}
let segment = open
.entry(scope.clone())
.or_insert_with(|| OpenSegment::new(self.clock.now()));
segment.push(lines, done);
segment.bytes() >= MAX_SEGMENT_BYTES
};
if full {
self.flush(&scope).await; // the size trigger; a timer task covers the 2 seconds
}
wait.await.map_err(|_| IntakeError::Closed)?
}
}
impl Buffer {
async fn flush(&self, scope: &Scope) {
let Some(segment) = self.open.lock().remove(scope) else { return };
let key = Keys::new(&self.prefix, scope).segment(self.clock.now(), self.writer);
let (body, summary, waiters) = segment.finish(); // gzip, counts, batch ids
let result = self.store.put(&key, body, PutOptions::ndjson_gzip()).await;
if result.is_ok() {
self.ledger.register(SegmentRow::new(scope, &key, summary)); // queued, retried, never awaited here
}
for waiter in waiters {
let _ = waiter.send(result.clone().map_err(IntakeError::from));
}
}
}The same thing on AWS
A function cannot keep an open segment between two requests, and many copies of it run at once. So Intake has a second implementation there: a queue.
One process (serve) | AWS functions | |
|---|---|---|
Intake::append | Adds to the in-memory buffer, waits for the PUT | Sends the validated lines as one queue message, waits for the queue to confirm |
| Who writes the segment | The same process, 2 seconds later | The worker function, woken by the queue with up to 10 seconds of messages |
| The answer means | The events are in the Bucket | The events are in the queue, which keeps them for 4 days |
| Segment name and content | Identical | Identical |
A batch larger than 192 KiB after compression does not go into a message: ingest writes it as its own segment and sends a message that only names it. The alternative for AWS was a managed delivery stream, as in the system this comes from. It was left out because it cannot name objects by project without an extra feature that buffers for at least 60 seconds, half of the 2-minute budget.
Duplicates and idempotency
Networks retry. A browser can send a batch, lose the answer and send it again. Each kind of upload has one rule that makes the second arrival harmless.
| Upload | What makes it idempotent |
|---|---|
| A batch of events | batch_id. Every stored line carries it as bid. The first worker that scans a segment records its batch ids; a batch id seen before, in any segment of the last 48 hours, is skipped by every reader. The lines stay in the object; they are not counted. |
A batch without batch_id | Ingest uses the SHA-256 of the body as the id. A retry of the same bytes is caught; the same events re-sent in a different body are not. |
| A replay chunk | The key is replays/<sid>/<seq>. The same seq again replaces the object. |
| A page snapshot | The key ends in the snapshot’s own ts. The same snapshot again replaces the object. |
| An OTLP request | The SHA-256 of the body. OTLP exporters retry with the same bytes. |
Dedupe happens when segments are read, not when batches are received, for one reason: receiving must not depend on Postgres. The price is that a duplicated batch takes space in the Bucket. That is cheap; a doubled count is not.
Source maps and replay chunks
Replay chunks go through ingest, one request per chunk, because ingest must check three things before storing one: that the route is not in the project’s blocked routes (403 not_recordable), that the session has fewer than 720 chunks (422 session_full), and that the project is inside its replays limit (429 limit_reached). The chunk is written before the answer, with Content-Type: application/json and Content-Encoding: gzip. Ingest does not look inside it.
The 4,000,000-byte limit is not arbitrary. On AWS a function receives at most 6 MB in one request; the SDK’s chunks of 200 KB and this cap keep a good margin under it.
Source maps never pass through ingest. A deploy of a large app is hundreds of files and, measured on the system this comes from, about 109 MB. So the upload is in three steps, all with an API token of scope admin:
- Your CI calls
POST /v1/releases/{version}/sourcemapswith the list of files. - The API answers with one signed upload link per file, valid for 15 minutes (
storage.presign_ttl_seconds). - Your CI uploads each file straight to the Bucket, under
sourcemaps/<release>/, and theindex.jsonthat maps each built file to its map.
sypnia sourcemaps upload does the three steps for you.
OTLP intake
POST /v1/otlp/v1/traces accepts what any OpenTelemetry exporter sends and turns it into the same events a Sypnia server SDK would have sent. After this mapping nothing downstream knows the difference.
| In the OTLP request | Becomes |
|---|---|
Resource service.name | service |
Resource service.version | ver, the release |
Resource deployment.environment.name (or the older deployment.environment) | env; production when missing |
A span of kind SERVER | One server_request: tid, span, parent, method, route from http.route, status, dur |
| Spans of the same trace and service under it | Its spans[]: db when the span has db.system.name, http when it has http.request.method, else internal; at most 200 kept, the rest counted in span_n |
Span status ERROR on a child span | That span’s status: error, with code from http.response.status_code or db.response.status_code |
An exception span event | One server_error: name from exception.type, msg from exception.message, stack from exception.stacktrace. The engine computes fp, because no SDK did |
Span attribute session.id (32 hex characters) | sid: the request joins that session’s timeline |
| Other span events, links, metrics, logs | Dropped. Refused spans are counted in partialSuccess.rejectedSpans |
Spans of one trace can arrive in different requests. Ingest does not wait for them: each request is mapped on its own, and a SERVER span that arrives without its children is stored as it is. The roll-up joins what shares a tid when it builds the traces index.
Why ingest does not query Postgres
Three reasons, in order of weight.
- Availability. If receiving needed the database, every database problem would be data loss. As designed, a Postgres outage costs freshness and nothing else.
- Connections. The system this comes from once exhausted its connection pooler: each copy of a request-serving function opened its own pool. Ingest is the most copied part of the engine; keeping it off the database removes the largest source of connections.
- Latency. A hash-map read takes less than a microsecond. A query takes a network round trip, on every batch from every browser.
What ingest reads instead is small, and all of it comes from the cache entry of the key:
| Cached per ingest key | Used for | How fresh |
|---|---|---|
Project id, key id, revoked_at | Gate 1 | 60 s |
allowed_origins | Gate 2 | 60 s |
rate_limit_per_minute | Gate 3 | 60 s |
| Blocked routes, replay rules, sampling rates | Gate 6, /v1/ingest/config | 60 s |
slow_down | Gate 7 and the answer | 60 s |
has_server | Nothing in ingest; it rides along for the worker | 60 s |
The only write ingest makes to Postgres is the ledger row: one row per flushed segment, not one per request and not one per event. It happens after the answer, on one connection per process, and it is allowed to fail. A request never waits on Postgres.
If Postgres is down when a segment is flushed, the segment is still written to the Bucket and the SDKs are still answered. The ledger row is kept in memory and retried; if the process dies before it succeeds, a worker recovers it by listing the Bucket and adding every segment the ledger does not know. Until then the events are safe and not yet counted.
Logs
Ingest writes one JSON line per request: status, events accepted and dropped, bytes, milliseconds. Never the key, the address or the body. Requests that succeed with nothing dropped are sampled at 1 %. Measured on the system this comes from: logging every success cost about 18 MB of logs a day for one app.
What changed from the system this comes from
| There | Here | Why |
|---|---|---|
| A signed-in user’s token, verified against the app’s auth service | A public ingest key, allowed origins and a rate limit | Brief §10.2: the ingest key is the only credential for sending at launch |
| No dedupe: a retried batch was stored and counted twice | batch_id on every line | The CORS incident of 2026-09-30 |
| No rate limit, no quota | Both | A public key needs them |
No limit on how old ts may be | 7 days | An old clock or a stuck queue must not rewrite last month |
| Server records skipped ingest and wrote to the stream directly | Everything goes through ingest | One place stamps the project; a server needs only a key and a URL |
| Unknown fields removed by an allow-list | Unknown fields kept | Brief §10.4: schema on read |
Next
- Where the objects land and what they are called.
- What reads the segments next.
- Every field, limit and error, for anyone writing a client.