Docs

Ingest

How the engine receives events, replay chunks, page snapshots, source maps and OTLP traces, what it checks, how it turns batches into objects, how it handles duplicates, and why it does not query the database to do it.

Applies to Engine At launch

View as Markdown
On this page
  1. The HTTP surface
  2. The gates
  3. (1) The key
  4. (2) The origin
  5. (3) The rate limit
  6. (4) Decoding
  7. (5) and (6) The shape, then each event
  8. (7) The quota
  9. The handler
  10. From batches to objects
  11. The same thing on AWS
  12. Duplicates and idempotency
  13. Source maps and replay chunks
  14. OTLP intake
  15. Why ingest does not query Postgres
  16. Logs
  17. What changed from the system this comes from
  18. Next

Ingest is the door of the engine. It receives what the SDKs send, checks it, and writes it to the Bucket. It analyses nothing. That is what keeps it fast, cheap and hard to break.

This page is the design behind the Ingest API. The API page says what to send; this one says what happens to it.

The HTTP surface

EndpointBodyLimitGoes to
POST /v1/ingest/eventsA batch: up to 200 events and a batch_id2 MiB after decompressionThe buffer, then a segment under events/
POST /v1/ingest/replayOne replay chunk4,000,000 bytes after decompression; 720 chunks a sessionOne object under replays/, written before the answer
POST /v1/ingest/snapshotOne page snapshot4,000,000 bytes after decompressionOne object under snapshots/
GET /v1/ingest/config——Answered from the cache: recording rules and slow_down
POST /v1/otlp/v1/tracesAn OTLP ExportTraceServiceRequest, protobuf or JSON4,000,000 bytes; 2,000 spansMapped to events, then the same buffer

All five take an ingest key, spk_…, in Authorization: Bearer or, for navigator.sendBeacon, in the key query parameter. An API token sent here is refused with 403 wrong_credential, so a secret that ends up in a browser by mistake does not work quietly.

Source maps do not pass through ingest. Your CI asks the API for upload links with an API token, and uploads each file straight to the Bucket. See the source-map section below.

The gates

A batch passes seven gates, in this order. Each one that fails answers at once with one error code, and the cheapest checks come first.

Diagram to drawFLOW DIAGRAM, left to right, a single line of seven gate-shaped nodes (a gate = a small rectangle with a notch or a thin vertical bar on its left), then two end nodes. On the far left the node «Your app (SDK)» with an arrow labelled «POST /v1/ingest/events» into gate 1. Gates and exact labels: 1 «Key» (sub-label «cache lookup»); 2 «Origin»; 3 «Rate limit» (sub-label «per key and address, in memory»); 4 «Decode» (sub-label «gzip by its first bytes, size cap»); 5 «Shape» (sub-label «events is a list of at most 200»); 6 «Each event» (sub-label «cut, clean, drop, stamp»); 7 «Quota» (sub-label «slow_down from the cache»). After gate 7 the node «Buffer», then an arrow labelled «one segment, at most 2 s later» to the node «Bucket», then a return arrow from Bucket back along the top to «Your app (SDK)» labelled «200 { accepted, dropped, slow_down }». UNDER each gate a short downward arrow ending in a small error tag with exact text: gate 1 «401 unauthorized · 403 wrong_credential»; gate 2 «403 origin_not_allowed»; gate 3 «429 rate_limited»; gate 4 «400 invalid_body · 413 payload_too_large»; gate 5 «400 too_many_events»; gate 6 «no error: counted in dropped»; gate 7 «no error: counted in dropped_reasons.limit_reached»; under Buffer «503 unavailable». A bracket over gates 1 to 7 labelled «no database query». The lit element is that bracket. Numbered callouts (1) to (7) on the gates.
Figure. The seven gates of a batch. Gates 1 to 5 can refuse the whole request; gates 6 and 7 drop single events and say so in the answer.

(1) The key

The key is looked up in a cache held in memory, not in Postgres. A key that is unknown or revoked gets 401 unauthorized.

Notice the three ages in the lookup below. A hit is used as it is for 60 seconds. A miss asks Postgres once and remembers «no such key» for 60 seconds, so a wrong key cannot be used to hammer the database. And when Postgres does not answer, an entry up to one hour old is still good.

crates/sypnia-ingest/src/project_cache.rs
impl ProjectCache {
    /// The project behind an ingest key. One hash-map read on the hot path.
    pub async fn lookup(&self, key: &IngestKey) -> Result<Arc<ProjectEntry>, IngestError> {
        let now = self.clock.now();
        if let Some(slot) = self.entries.get(key) {
            match slot.value() {
                Slot::Known(entry) if entry.loaded_at + FRESH > now => return Ok(entry.clone()),
                Slot::Unknown { since } if *since + NEGATIVE > now => return Err(IngestError::Unauthorized),
                _ => {} // stale: refresh below
            }
        }
        match self.db.project_for_key(key).await {
            Ok(Some(entry)) => {
                let entry = Arc::new(entry);
                self.entries.insert(key.clone(), Slot::Known(entry.clone()));
                Ok(entry)
            }
            Ok(None) => {
                self.entries.insert(key.clone(), Slot::Unknown { since: now });
                Err(IngestError::Unauthorized)
            }
            // Postgres is down: keep serving what we knew, for at most an hour.
            Err(_) => match self.entries.get(key).as_deref() {
                Some(Slot::Known(entry)) if entry.loaded_at + STALE_OK > now => Ok(entry.clone()),
                _ => Err(IngestError::Unavailable),
            },
        }
    }
}

A long-running process also refreshes every entry in the background each 60 seconds, so in practice the request never waits for Postgres. That is how a revoked key is refused within one minute, as Projects, environments, keys and tokens promises.

(2) The origin

If the request has an Origin header and the key has a list of allowed origins, the origin must be in the list, or the answer is 403 origin_not_allowed. A request with no Origin, such as one from a server SDK, passes: a key used from a server is protected by its rate limit, not by a header the sender controls.

One layer owns CORS, and it is the engine. The system this comes from learned why: its function URL and its handler both added CORS headers, the browser rejected the answer after the data was stored, the SDK retried, and batches arrived twice. In the AWS template the address in front of the functions adds no CORS headers of its own.

(3) The rate limit

Each key has a number of requests a minute from one address (default 600, rate_limit.ingest_per_minute). The counter is a token bucket in the memory of the process. The address is read from the connection, or from the forwarding header when trusted_proxy_hops says a proxy is in front. It is used for this count and nowhere else: it is not written to an event, a log line or a row.

Because the counter is per process, several processes or functions each count on their own, and the real limit is higher than the setting. The alternative is a shared counter in Postgres or in another service, on the path of every request. A rate limit is a guard against a runaway loop, not a bill; an approximate one is enough.

(4) Decoding

The body may be gzip. Ingest detects it from the first two bytes, 1f 8b, whatever the Content-Type says, because sendBeacon cannot set Content-Encoding. It decompresses as a stream and stops at the limit, so a small body that expands to gigabytes costs 2 MiB of memory and a 413, not the process.

(5) and (6) The shape, then each event

events must be a list of at most 200. After that, each event is judged on its own. One bad event never costs the rest of the batch.

What ingest findsWhat it does
A field the contract does not listKeeps it, as sent
A type the contract does not listKeeps the event. Only the parts that know the type will read it
A string longer than its limitCuts it to the limit
A listed field with the wrong type or a value outside its listRemoves the field, keeps the event
A required field missing or unusable: no type, ts not positive, a browser event without sidDrops the event: invalid
A vital outside its plausible range (an LCP over 120 s, an INP over 60 s, a CLS over 10)Drops the event: invalid
ts more than 7 days in the pastDrops the event: too_old
An event from a route in the project’s blocked routesDrops the event: blocked_route
A span attribute whose name contains authorization, cookie, password, secret or tokenRemoves the attribute
prj, rcv or bid sent by the appReplaces them with the engine’s own

The plausibility caps exist twice on purpose, in the SDK and here. Measured on the system this comes from: two browsers once reported INP values of 5 to 7 days and dragged a p75 to the top of the scale.

(7) The quota

The cache entry carries the project’s slow_down, decided elsewhere (see Data flow). When it says send: failures, ingest keeps errors and failed requests and counts the rest as limit_reached. The answer carries the same slow_down, so the SDK stops sending what would be dropped.

The handler

This is the whole request, in order. Notice two things: nothing in it awaits Postgres, and the last await returns only when the segment that carries these events is in the Bucket.

crates/sypnia-ingest/src/events.rs
pub async fn ingest_events(
    State(ingest): State<Arc<Ingest>>,
    key: IngestKeyAuth,                       // Bearer header or ?key=; an sps_ token is 403 wrong_credential
    origin: Option<TypedHeader<Origin>>,
    ClientAddr(addr): ClientAddr,
    body: Bytes,
) -> Result<Json<IngestResult>, IngestError> {
    let project = ingest.projects.lookup(&key.0).await?;            // (1) 401
    project.check_origin(origin.as_deref())?;                       // (2) 403
    ingest.limiter.check(&project.key_id, addr)?;                   // (3) 429
    let batch: RawBatch = decode(&body, MAX_BATCH_BYTES)?;          // (4) 400, 413
    if batch.events.len() > MAX_EVENTS_PER_BATCH {
        return Err(IngestError::TooManyEvents);                     // (5) 400
    }

    let now = ingest.clock.now();
    // No batch_id? The body's own hash stands in, so an identical retry is still caught.
    let batch_id = batch.batch_id.unwrap_or_else(|| BatchId::from_content(&body));
    let mut lines = Lines::new(&project.id, &batch_id, now);
    let mut dropped = Dropped::default();

    for raw in batch.events {
        match validate(raw, &project, now) {                        // (6) cut, clean, stamp
            Ok(event) if project.slow_down.allows(&event) => lines.push(&event),
            Ok(_) => dropped.add(Reason::LimitReached),             // (7)
            Err(reason) => dropped.add(reason),
        }
    }

    let accepted = lines.len();
    if accepted > 0 {
        // Returns when the segment holding these lines exists. 503 if it could not be written.
        ingest.intake.append(lines).await?;
    }
    Ok(Json(IngestResult {
        accepted,
        dropped: dropped.total(),
        dropped_reasons: dropped.into_map(),
        slow_down: project.slow_down.clone(),
    }))
}

From batches to objects

One object per batch would be the simplest design and the most expensive one. Measured on the system this comes from: object writes cost more than object storage. So batches are grouped.

Diagram to drawDIAGRAM OF A GROUP COMMIT, time flowing left to right over a 2-second window marked «0 s» at the left edge and «2 s» at the right edge. On the left, three source nodes stacked: «browser A», «browser B», «shop-api (server SDK)». From each, one or two small batch tiles travel right into a wide open container in the centre labelled «Buffer: prj_5k2m9x4q7w1z · production». Exact tile labels: from browser A «batch 0b9f… · 6 events» arriving at 0.2 s; from shop-api «batch 5d0c… · 2 events» at 0.6 s; from browser B «batch 71aa… · 11 events» at 1.4 s. Each tile, once inside the container, has a thin dotted line going back to its source, ending in a small clock icon labelled «waiting». At the 2 s edge the container closes and becomes one file tile labelled «20260312T091216Z-7f3a9c.ndjson.gz · 19 events · 3 batches» with an arrow labelled «one PUT» into the node «Bucket». From the Bucket, three return arrows fan back to the three sources, each labelled «200», drawn in the accent colour: this fan is the lit element. Below the container two small trigger tags: «closes at 2 s» and «or at 4 MiB, whichever comes first». Below the Bucket a dashed arrow to a small node «Postgres» labelled «ledger row, after the answers». Numbered callouts: (1) on the container, (2) on the PUT arrow, (3) on the fan of answers, (4) on the dashed ledger arrow.
Figure. Group commit. Batches that arrive within the same 2 seconds leave in one object, and all of them are answered when it exists.

The buffer (1) is one open segment per project and environment. It closes when the first of two things happens:

TriggerValueWhy this value
Time since its first line2 secondsSmall against the 120-second budget, and at most 30 objects a minute per project
Size before compression4 MiBA worker reads a segment whole into memory to scan it; this keeps that read small

A third limit protects the process: if more than 32 MiB of lines are waiting to be written, across all projects, new batches get 503 unavailable with Retry-After: 2 until the Bucket catches up.

When a segment closes it is written with one PUT (2), and every request that put lines in it is answered (3). Then, off the request path, one row goes into the segments ledger (4). If that insert fails it is kept and tried again; a worker that starts also lists the last two hours of each project in the Bucket and adds any segment the ledger does not know.

Notice in the excerpt that the lock is never held across an await, and that every waiter gets the result of the same write: either all the batches of a segment are stored, or none is and each SDK retries its own.

crates/sypnia-ingest/src/buffer.rs
#[async_trait]
impl Intake for Buffer {
    async fn append(&self, lines: Lines) -> Result<(), IntakeError> {
        let (done, wait) = oneshot::channel();
        let scope = lines.scope();
        let full = {
            let mut open = self.open.lock(); // parking_lot: short, never held across an await
            if self.waiting_bytes.load(Ordering::Relaxed) + lines.bytes() > MAX_WAITING_BYTES {
                return Err(IntakeError::Full); // 503 with Retry-After
            }
            let segment = open
                .entry(scope.clone())
                .or_insert_with(|| OpenSegment::new(self.clock.now()));
            segment.push(lines, done);
            segment.bytes() >= MAX_SEGMENT_BYTES
        };
        if full {
            self.flush(&scope).await; // the size trigger; a timer task covers the 2 seconds
        }
        wait.await.map_err(|_| IntakeError::Closed)?
    }
}

impl Buffer {
    async fn flush(&self, scope: &Scope) {
        let Some(segment) = self.open.lock().remove(scope) else { return };
        let key = Keys::new(&self.prefix, scope).segment(self.clock.now(), self.writer);
        let (body, summary, waiters) = segment.finish(); // gzip, counts, batch ids

        let result = self.store.put(&key, body, PutOptions::ndjson_gzip()).await;
        if result.is_ok() {
            self.ledger.register(SegmentRow::new(scope, &key, summary)); // queued, retried, never awaited here
        }
        for waiter in waiters {
            let _ = waiter.send(result.clone().map_err(IntakeError::from));
        }
    }
}

The same thing on AWS

A function cannot keep an open segment between two requests, and many copies of it run at once. So Intake has a second implementation there: a queue.

One process (serve)AWS functions
Intake::appendAdds to the in-memory buffer, waits for the PUTSends the validated lines as one queue message, waits for the queue to confirm
Who writes the segmentThe same process, 2 seconds laterThe worker function, woken by the queue with up to 10 seconds of messages
The answer meansThe events are in the BucketThe events are in the queue, which keeps them for 4 days
Segment name and contentIdenticalIdentical

A batch larger than 192 KiB after compression does not go into a message: ingest writes it as its own segment and sends a message that only names it. The alternative for AWS was a managed delivery stream, as in the system this comes from. It was left out because it cannot name objects by project without an extra feature that buffers for at least 60 seconds, half of the 2-minute budget.

Duplicates and idempotency

Networks retry. A browser can send a batch, lose the answer and send it again. Each kind of upload has one rule that makes the second arrival harmless.

UploadWhat makes it idempotent
A batch of eventsbatch_id. Every stored line carries it as bid. The first worker that scans a segment records its batch ids; a batch id seen before, in any segment of the last 48 hours, is skipped by every reader. The lines stay in the object; they are not counted.
A batch without batch_idIngest uses the SHA-256 of the body as the id. A retry of the same bytes is caught; the same events re-sent in a different body are not.
A replay chunkThe key is replays/<sid>/<seq>. The same seq again replaces the object.
A page snapshotThe key ends in the snapshot’s own ts. The same snapshot again replaces the object.
An OTLP requestThe SHA-256 of the body. OTLP exporters retry with the same bytes.

Dedupe happens when segments are read, not when batches are received, for one reason: receiving must not depend on Postgres. The price is that a duplicated batch takes space in the Bucket. That is cheap; a doubled count is not.

Source maps and replay chunks

Replay chunks go through ingest, one request per chunk, because ingest must check three things before storing one: that the route is not in the project’s blocked routes (403 not_recordable), that the session has fewer than 720 chunks (422 session_full), and that the project is inside its replays limit (429 limit_reached). The chunk is written before the answer, with Content-Type: application/json and Content-Encoding: gzip. Ingest does not look inside it.

The 4,000,000-byte limit is not arbitrary. On AWS a function receives at most 6 MB in one request; the SDK’s chunks of 200 KB and this cap keep a good margin under it.

Source maps never pass through ingest. A deploy of a large app is hundreds of files and, measured on the system this comes from, about 109 MB. So the upload is in three steps, all with an API token of scope admin:

  1. Your CI calls POST /v1/releases/{version}/sourcemaps with the list of files.
  2. The API answers with one signed upload link per file, valid for 15 minutes (storage.presign_ttl_seconds).
  3. Your CI uploads each file straight to the Bucket, under sourcemaps/<release>/, and the index.json that maps each built file to its map.

sypnia sourcemaps upload does the three steps for you.

OTLP intake

POST /v1/otlp/v1/traces accepts what any OpenTelemetry exporter sends and turns it into the same events a Sypnia server SDK would have sent. After this mapping nothing downstream knows the difference.

In the OTLP requestBecomes
Resource service.nameservice
Resource service.versionver, the release
Resource deployment.environment.name (or the older deployment.environment)env; production when missing
A span of kind SERVEROne server_request: tid, span, parent, method, route from http.route, status, dur
Spans of the same trace and service under itIts spans[]: db when the span has db.system.name, http when it has http.request.method, else internal; at most 200 kept, the rest counted in span_n
Span status ERROR on a child spanThat span’s status: error, with code from http.response.status_code or db.response.status_code
An exception span eventOne server_error: name from exception.type, msg from exception.message, stack from exception.stacktrace. The engine computes fp, because no SDK did
Span attribute session.id (32 hex characters)sid: the request joins that session’s timeline
Other span events, links, metrics, logsDropped. Refused spans are counted in partialSuccess.rejectedSpans

Spans of one trace can arrive in different requests. Ingest does not wait for them: each request is mapped on its own, and a SERVER span that arrives without its children is stored as it is. The roll-up joins what shares a tid when it builds the traces index.

Why ingest does not query Postgres

Three reasons, in order of weight.

  1. Availability. If receiving needed the database, every database problem would be data loss. As designed, a Postgres outage costs freshness and nothing else.
  2. Connections. The system this comes from once exhausted its connection pooler: each copy of a request-serving function opened its own pool. Ingest is the most copied part of the engine; keeping it off the database removes the largest source of connections.
  3. Latency. A hash-map read takes less than a microsecond. A query takes a network round trip, on every batch from every browser.

What ingest reads instead is small, and all of it comes from the cache entry of the key:

Cached per ingest keyUsed forHow fresh
Project id, key id, revoked_atGate 160 s
allowed_originsGate 260 s
rate_limit_per_minuteGate 360 s
Blocked routes, replay rules, sampling ratesGate 6, /v1/ingest/config60 s
slow_downGate 7 and the answer60 s
has_serverNothing in ingest; it rides along for the worker60 s

The only write ingest makes to Postgres is the ledger row: one row per flushed segment, not one per request and not one per event. It happens after the answer, on one connection per process, and it is allowed to fail. A request never waits on Postgres.

If Postgres is down when a segment is flushed, the segment is still written to the Bucket and the SDKs are still answered. The ledger row is kept in memory and retried; if the process dies before it succeeds, a worker recovers it by listing the Bucket and adding every segment the ledger does not know. Until then the events are safe and not yet counted.

Logs

Ingest writes one JSON line per request: status, events accepted and dropped, bytes, milliseconds. Never the key, the address or the body. Requests that succeed with nothing dropped are sampled at 1 %. Measured on the system this comes from: logging every success cost about 18 MB of logs a day for one app.

What changed from the system this comes from

ThereHereWhy
A signed-in user’s token, verified against the app’s auth serviceA public ingest key, allowed origins and a rate limitBrief §10.2: the ingest key is the only credential for sending at launch
No dedupe: a retried batch was stored and counted twicebatch_id on every lineThe CORS incident of 2026-09-30
No rate limit, no quotaBothA public key needs them
No limit on how old ts may be7 daysAn old clock or a stuck queue must not rewrite last month
Server records skipped ingest and wrote to the stream directlyEverything goes through ingestOne place stamps the project; a server needs only a key and a URL
Unknown fields removed by an allow-listUnknown fields keptBrief §10.4: schema on read

Next

  • Storage

    Where the objects land and what they are called.
  • Aggregation

    What reads the segments next.
  • Ingest API

    Every field, limit and error, for anyone writing a client.

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close