Docs

Problem grouping

How failures become problems, with the exact fingerprint algorithm worked by hand, symbolication with source maps, the rule that makes one request one problem across browser and server, the states of a problem and what triggers an alert.

Applies to Engine At launch

View as Markdown
On this page
  1. From a failure to a problem
  2. The fingerprint
  3. The algorithm
  4. Worked by hand
  5. One request, one problem
  6. Symbolication
  7. The states of a problem
  8. What a transition writes
  9. What triggers an alert
  10. A problem, as the API returns it
  11. What changed from the system this comes from
  12. Next

A problem is a group of failures with one cause. This page is how the engine decides that two failures are the same, how it turns a minified stack into your own files and lines, and how a problem moves between its states. It is the design behind Problems.

Grouping is done by the fast path: the reader that scans each new segment for failures every 30 seconds. Data flow shows where it sits.

From a failure to a problem

Diagram to drawFLOW DIAGRAM, left to right, five stages joined by arrows. Stage 1, node «Segment» (file icon) with the label «one line: server_error, fp 88a44b15205cb153». Stage 2, node «Fast path» with three stacked sub-steps: «is it a failure?», «is it part of a request already known?», «fingerprint»; arrow in labelled «every 30 s». Stage 3, a decision diamond «fingerprint known in this project and environment?» with two exits: exit «no» to a node «New problem» (drawn as the lit element, accent colour, with a small star), exit «yes» to a node «Same problem: add 1 occurrence». Both nodes then point to stage 4, node «Postgres» with four small row tiles under it: «problems», «problem_occurrences», «problem_hits», «alerts». Stage 5, from Postgres two outgoing arrows: one to node «The agent» labelled «symbolicate, then the sentence», one to node «Alerts» labelled «held until the sentence or 100 s». A side input into stage 2 from below: node «Bucket: sourcemaps/4f2a91c/» with a dashed arrow going not to the Fast path but to «The agent», labelled «original files and lines». Numbered callouts: (1) on «is it a failure?», (2) on «is it part of a request already known?», (3) on «fingerprint», (4) on the diamond, (5) on the dashed source-map arrow.
Figure. A failure's path. The fingerprint decides between a new problem and one more occurrence; source maps are applied after, and never change the grouping.

(1) What counts as a failure. The fast path looks at four kinds of line and ignores every other.

LineA failure whenKindFingerprint comes from
error (browser)AlwayserrorThe SDK: fp
server_errorAlwayserror, or failed_call when src is spanThe SDK: fp. The engine computes it for OTLP
server_requestpartial is true and no full record of the same trace existstimeoutThe SDK: fp
request (browser)No answer, 400, or 500 and abovefailed_callThe engine: no SDK field carries it

A 401, 403, 404, 409 or 422 is not a failure. Neither is a rage click or a slow page. Those are counted by the roll-up and never reach this code.

(2), (3), (4) are the next three sections. (5) is symbolication.

The fingerprint

A fingerprint is 16 hex characters. Failures with the same fingerprint, in the same project and environment, are the same problem.

It is computed where the failure happens, in the SDK, so that the SDK can also drop a repeat of the same error within 5 seconds. The engine carries the same algorithm for the cases where no SDK did the work: OTLP exceptions and failed browser calls. Both must give the same bytes for the same input, so the algorithm is written down here exactly, and the same test vectors run in TypeScript and in Rust.

The algorithm

For an error, three parts joined with |:

Text
name | normalized message | function@file of the first frame of your own code
  1. name: the error’s name. Error when there is none.

  2. The message, normalized, in this order:

    StepReplaceWith
    1URLs<url>
    2Paths of built chunks, such as /_next/static/chunks/…<chunk>
    3UUIDs<uuid>
    4Quoted strings of up to 200 characters"…"
    5Runs of 12 or more hex characters<hex>
    6Numbers#
    7Runs of whitespaceone space
    8Everything after the first 300 charactersnothing
  3. The top frame: read the first 30 lines of the stack, in the V8 form (at fn (file:line:col)) or the Gecko form (fn@file:line:col). Take the first frame whose file is not noise. Noise is node_modules, the framework’s own runtime chunks, react-dom, scheduler, node:internal and <anonymous>. If every frame is noise, take the first. Then clean the file: remove the origin, the query and the hash; remove a build hash of the form - followed by 8 or more hex characters; cut everything before the build folder. The part is function@file, or empty when there is no stack.

If the framework gave the error a digest of 6 or more digits, the fingerprint is the hash of digest: and the digest instead, because that id is already stable.

For a failed call, the call, how it failed, and the request it happened in:

Text
call | span name | status or error code | METHOD | route

For a timeout: timeout | METHOD | route.

The hash, hash64, is FNV-1a with 32 bits, run twice over the same string with two different seeds, the two results written as 8 hex characters each and joined.

Value
Seed of the first half0x811c9dc5
Seed of the second half0x050c5d1f
Prime0x01000193
One steph = (h XOR unit) × prime, keeping the low 32 bits
A unitOne UTF-16 code unit of the string

Notice that the function takes no project and no time: a fingerprint depends only on the failure.

crates/sypnia-fingerprint/src/lib.rs
const PRIME: u32 = 0x0100_0193;
const SEED_A: u32 = 0x811c_9dc5;
const SEED_B: u32 = 0x050c_5d1f;

/// FNV-1a, 32 bits, over UTF-16 code units. Must match the SDKs bit for bit.
fn fnv1a_utf16(text: &str, seed: u32) -> u32 {
    text.encode_utf16()
        .fold(seed, |hash, unit| (hash ^ u32::from(unit)).wrapping_mul(PRIME))
}

/// 16 hex characters: two 32-bit hashes of the same text with different seeds.
pub fn hash64(text: &str) -> Fingerprint {
    Fingerprint::from_halves(fnv1a_utf16(text, SEED_A), fnv1a_utf16(text, SEED_B))
}

pub fn error_fingerprint(name: Option<&str>, message: &str, stack: Option<&str>, digest: Option<&str>) -> Fingerprint {
    if let Some(digest) = digest.filter(|d| d.len() >= 6 && d.bytes().all(|b| b.is_ascii_digit())) {
        return hash64(&format!("digest:{digest}"));
    }
    let frame = stack
        .and_then(top_frame) // first frame of your own code, V8 or Gecko
        .map(|f| format!("{}@{}", f.function, normalize_frame_file(&f.file)))
        .unwrap_or_default();
    hash64(&format!(
        "{}|{}|{}",
        name.unwrap_or("Error"),
        normalize_message(message),
        frame,
    ))
}

pub fn call_fingerprint(span_name: &str, code: &str, method: &str, route: &str) -> Fingerprint {
    hash64(&format!("call|{span_name}|{code}|{}|{route}", method.to_ascii_uppercase()))
}

pub fn timeout_fingerprint(method: &str, route: &str) -> Fingerprint {
    hash64(&format!("timeout|{}|{route}", method.to_ascii_uppercase()))
}

Worked by hand

The shop’s checkout page throws this when the payment is refused:

Text
ApiError: POST /api/checkout failed with 400 (amount_required)
    at o (https://shop.example/assets/checkout-3f9c.js:1:48211)
    at async HTMLButtonElement.<anonymous> (https://shop.example/assets/checkout-3f9c.js:1:49002)
Diagram to drawSTEP DIAGRAM, top to bottom, like a recipe card with five numbered rows; each row has an input box on the left, an arrow with the rule's name in the middle, and an output box on the right. Use a monospace face inside the boxes and keep every string exactly as written here. Row 1, rule «name»: input «ApiError: POST /api/checkout failed with 400 (amount_required)» → output «ApiError». Row 2, rule «normalize the message: numbers become #»: input «POST /api/checkout failed with 400 (amount_required)» → output «POST /api/checkout failed with # (amount_required)», with the «#» highlighted. Row 3, rule «top frame: first frame of your own code»: input «at o (https://shop.example/assets/checkout-3f9c.js:1:48211)» → output «o@/assets/checkout-3f9c.js», with a small note «origin, line and column removed». Row 4, rule «join with |»: three small input chips (the outputs of rows 1 to 3) → one long output box «ApiError|POST /api/checkout failed with # (amount_required)|o@/assets/checkout-3f9c.js». Row 5, rule «hash64: FNV-1a over UTF-16 units, two seeds»: the long box → two half boxes side by side «15228057» (label «seed 0x811c9dc5») and «93228d19» (label «seed 0x050c5d1f»), which join into the final box «1522805793228d19», drawn as the lit element in the accent colour with the label «the fingerprint». Numbered callouts (1) to (5) on the rows.
Figure. The fingerprint of the checkout error, step by step. Change the order number or the customer and nothing here changes; change the line of code that throws and it does.
StepResult
(1) NameApiError
(2) Message, normalized. Rule 6 turns 400 into #; nothing else matchesPOST /api/checkout failed with # (amount_required)
(3) Top frame. The first frame, o, is your own code. Its file loses the origin, the line and the column. -3f9c is four characters, shorter than the eight a build hash needs, so it stayso@/assets/checkout-3f9c.js
(4) JoinedApiError|POST /api/checkout failed with # (amount_required)|o@/assets/checkout-3f9c.js
(5) hash64, first half, seed 0x811c9dc515228057
(5) hash64, second half, seed 0x050c5d1f93228d19
Fingerprint1522805793228d19

The first two steps of the first half, to check an implementation against: the string starts with A (unit 0x41) and p (0x70).

Text
h0 = 0x811c9dc5
h1 = (h0 XOR 0x41) × 0x01000193 mod 2^32 = 0x811c9d84 × 0x01000193 = 0xc40bf6cc
h2 = (h1 XOR 0x70) × 0x01000193 mod 2^32 = 0xc40bf6bc × 0x01000193 = 0x5ad569f4
…
h86 = 0x15228057

And the failure behind it on the server, the call that the payment service refused:

FailureHashed stringFingerprint
The server’s failed callcall|POST /v1/payments|400|POST|/api/checkout88a44b15205cb153
If that request had hung insteadtimeout|POST|/api/checkoutc405205e99c5aa6c

88a44b15205cb153 is the problem of the example story. Why it is that one and not the browser’s is the next section.

One request, one problem

(2) in the first figure. When the customer clicks Pay, three failures are reported for the same request:

  1. the server’s call to the payment service answered 400 (server_error, from @sypnia/node);
  2. the page’s call to POST /api/checkout answered 400 (request, from @sypnia/browser);
  3. the page threw ApiError (error, from @sypnia/browser).

Three problems for one click would be noise. The rule: the deepest failure in a trace is the problem; what the person saw is evidence inside it.

Diagram to drawTRACE DIAGRAM drawn as a waterfall, time left to right, with three horizontal bars stacked and indented like a call tree, plus a result panel on the right. Bar 1 (top, widest) labelled «browser · POST /api/checkout · 412 ms · answered 400», with a small tag «request» and a second small tag attached at its right end «error: ApiError, 4 ms later, same batch». Bar 2 (indented, inside bar 1) labelled «shop-api · POST /api/checkout · 398 ms · 400» with the tag «server_request». Bar 3 (indented further, inside bar 2, near its right end) labelled «POST /v1/payments · 340 ms · 400» with the tag «span: failed call»; bar 3 is the lit element, in the accent colour, with the note «deepest failure = the problem». A thin vertical line on the left links the three bars, labelled «one trace id: 7c2e9a4b…». RESULT PANEL on the right, titled «What the engine keeps»: one card «Problem 88a44b15205cb153 · createPayment answered 400 in POST /api/checkout» with three lines inside: «occurrences +1», «person u_1042, session 8f3a6c1e…», «evidence: the page's failed call and its ApiError». Under the card, two crossed-out ghost cards labelled «not opened: problem for the browser call» and «not opened: problem for ApiError». Numbered callouts: (1) on bar 3, (2) on bar 1, (3) on the ApiError tag.
Figure. One click, three failures, one problem. The failed call at the bottom of the trace is the problem; the two things the browser saw are attached to it.

How the fast path applies it:

FailureWhat the fast path does
A failure from the server, with a trace id (1)It is the problem. One row goes into trace_failures: «trace 7c2e9a4b… belongs to problem prb_8f3k2m9x4q7w», kept 48 hours. If the same trace has two server failures, the one in the deepest span wins.
A failed request from the browser, with a trace id (2)Look up the trace in trace_failures. Found: it is evidence of that problem. It adds no occurrence; it adds the person and the session if the server record did not carry them. Not found, and the project has sent server data: hold it for 15 seconds in held_failures, then look again. Still not found, or the project has only the browser SDK: it is its own problem.
A browser error in the same batch and session as a failed request, at most 2 seconds after it (3)It follows that request: evidence if the request is evidence, part of the request’s problem otherwise.
A browser error with no failed request before itIts own problem. This is most browser errors: a TypeError in your code has no request behind it.

The hold exists because the two halves travel separately. The server’s record usually arrives first, since the server sends when the request ends and the browser waits up to 10 seconds to send its batch. The 15 seconds cover the opposite order. They are part of the 2-minute budget worked out in Data flow, and they are spent only by projects that send server data.

A project with only the browser SDK gets the page’s failed call as the problem, exactly as Problems says.

Symbolication

A browser reports the stack of the code it ran: minified, in one line, with names like o. Source maps turn it back into your files.

(5) in the first figure. Symbolication happens after the problem exists and before the agent reads it. It never changes the fingerprint: grouping uses the minified frame, so a problem does not split when maps arrive late or are missing.

Rule
WhenWhen a problem is born or comes back; again when the maps of its release are uploaded after the failure; and for the first occurrence in each new release
Which releaseThe ver of the occurrence. No ver, no symbolication
Finding the mapsourcemaps/<release>/index.json maps each built file to its map. Bundlers name maps independently of chunks, so guessing <file>.map is only the fallback
FramesThe first 30. V8 and Gecko forms. A frame with no map stays as it is
Outputsample.stack_original, the stack with original file:line:column and function names, and where: the file, line and function of the first frame of your own code
LimitsA cache of 50 maps or 200 MB; a single map over 64 MB is skipped; 60 seconds for the step
On failureThe problem keeps its minified stack. The step never blocks a problem or an alert

For the checkout error the result is:

Text
ApiError: POST /api/checkout failed with 400 (amount_required)
    at confirmPayment (src/checkout/checkout.tsx:142:11)
    at async onPayClick (src/checkout/PayButton.tsx:31:5)

One number is exposed for this step: problems symbolicated ÷ problems that had a release and a stack, over the last 24 hours, shown by sypnia doctor. The reason is an incident in the system this comes from: for six days no error was symbolicated, because a build option broke the map reader, and nobody noticed. A log line is not a health number.

The states of a problem

A problem has one status, open, resolved or ignored, and an open problem can carry marks: new, growing, came back. Together they are the states below.

Diagram to drawSTATE MACHINE DIAGRAM. Five rounded state nodes. Top left: «Open · new» (drawn as the lit element, accent colour). Top right: «Open · growing». Centre: «Open» (plain, no mark). Bottom left: «Resolved». Bottom right: «Ignored». Bottom centre, between Resolved and Open: «Open · came back». A small filled start dot at the far top left with an arrow into «Open · new» labelled «first failure with this fingerprint → alert problem.new». TRANSITIONS with exact labels: «Open · new» → «Open» labelled «24 h after the first failure»; «Open · new» → «Open · growing» and «Open» → «Open · growing», both labelled «people ≥ 2 × people at last alert, and ≥ 5 → alert problem.growing»; «Open · growing» → «Open» labelled «24 h later»; from each of the three open states → «Resolved» (draw as one merged arrow from a bracket around the open states) labelled «a person or a token resolves it»; «Resolved» → «Open · came back» labelled «it happens again (in the fix release or later, if one was named) → alert problem.regressed»; «Open · came back» → «Resolved» labelled «resolved again»; from the bracket of open states → «Ignored» labelled «a person ignores it (admin)»; «Ignored» self-loop labelled «it happens again: counted, no alert, stays ignored»; «Ignored» → «Open» and «Resolved» → «Open», both dashed, labelled «reopened by hand». Inside every open state a tiny note «new occurrences are counted». A legend at the bottom: solid arrow «the engine does it», dashed arrow «a person does it»; draw the arrows «resolves», «ignores» and «reopened by hand» dashed accordingly. Numbered callouts: (1) on the start arrow, (2) on the growing arrows, (3) on the Resolved → came back arrow, (4) on the Ignored self-loop.
Figure. The states of a problem. The engine makes only three moves by itself: a problem is born, it grows, it comes back. Everything else is a person.

The engine decides a transition with one pure function. It receives the problem as it was before this segment and as it is after, and answers what happened. Notice that it cannot write anything: the caller writes the transition, the alert and the ledger row in one transaction, so a transition can never be recorded twice.

crates/sypnia-pipeline/src/problems/transition.rs
#[derive(Debug, PartialEq)]
pub enum Transition {
    Born,
    CameBack,
    Growing { from: u32, to: u32 },
    Quiet, // counted, nothing to tell anyone
}

pub fn transition(
    before: Option<&ProblemState>,
    after: &ProblemState,
    seen: &Seen,          // what this segment added: releases, people
    rules: &GrowingRule,  // factor 2, min_people 5, unless the project changed them
) -> Transition {
    let Some(before) = before else {
        return Transition::Born;
    };
    match before.status {
        Status::Ignored => Transition::Quiet,
        Status::Resolved => {
            // «Old tabs»: with a fix release named, only a failure in that release or a later one reopens.
            let reopens = match &before.resolved_in_release {
                Some(fix) => seen.releases.iter().any(|r| r.is_same_or_later_than(fix)),
                None => true,
            };
            if reopens { Transition::CameBack } else { Transition::Quiet }
        }
        Status::Open => {
            let threshold = before
                .people_at_last_alert
                .saturating_mul(rules.factor)
                .max(rules.min_people);
            if after.people >= threshold && after.people > before.people_at_last_alert {
                Transition::Growing { from: before.people_at_last_alert, to: after.people }
            } else {
                Transition::Quiet
            }
        }
    }
}

Follow the example of Problems through it. The problem is born with 1 person, and people_at_last_alert becomes 1. At 3 people the threshold is max(1 × 2, 5) = 5: quiet. At 5 people: growing, and people_at_last_alert becomes 5. The next threshold is 10, then 20.

«Later» for a release means first seen by the engine at the same time as the fix release or after it. The engine does not compare version strings: a short git commit has no order.

What a transition writes

All in the one transaction that also marks the segment as scanned:

TransitionproblemsAlertThe agent
BornA new row, status = open, people_at_last_alert = peopleproblem.newSymbolicate, then analyse
Came backstatus = open, regressed = true; resolved_at and resolved_by stay as historyproblem.regressedSymbolicate, then analyse again
Growinggrowing_until = now + 24 h, people_at_last_alert = peopleproblem.growingAnalyse again, inside the daily limit per problem
QuietCounters and last_seen onlyNoneNothing

What triggers an alert

For problems, exactly the three transitions above, and only when the project’s rule for that kind is on for the problem’s environment. By default the three are on for production only.

Each alert row has a dedupe_key with a unique index, which is what makes «one problem is one alert» a property of the database instead of a hope:

Alertdedupe_keySo that
problem.newproblem.new:<problem id>A problem is new once in its life
problem.regressedproblem.regressed:<problem id>:<resolved_at>Once for each time it was resolved
problem.growingproblem.growing:<problem id>:<people at last alert>Once for each step: 5, 10, 20

The alert is inserted with released_at empty and release_by set to 100 seconds after the failure reached ingest. It is released when the agent has written the sentence, or at release_by, whichever is first. Releasing means: set released_at, write the title (the sentence, or the technical title), and hand it to each alert sink. The agent’s pipeline shows that hand-off, and Deployment shapes shows the sinks.

The other alert kinds (uptime, a page that got slower, an API route failing, usage, the engine itself) are produced by the pass and the roll-up from numbers, not by this code. Their rules are in Alerts and the daily summary.

A problem, as the API returns it

After three people and five failures, GET /v1/problems/pay-button-does-nothing answers with the row of problems, its analysis, and links:

GET /v1/problems/pay-button-does-nothing
{
  "id": "prb_8f3k2m9x4q7w",
  "slug": "pay-button-does-nothing",
  "fingerprint": "88a44b15205cb153",
  "kind": "failed_call",
  "origin": "server",
  "status": "open",
  "regressed": false,
  "title": "createPayment answered 400 in POST /api/checkout",
  "environment": "production",
  "route": "/checkout",
  "where": { "file": "src/checkout/checkout.tsx", "line": 142, "function": "confirmPayment" },
  "first_seen": "2026-03-12T09:12:04Z",
  "last_seen": "2026-03-12T10:31:47Z",
  "first_release": "4f2a91c",
  "last_release": "4f2a91c",
  "occurrences": 5,
  "people": 3,
  "sessions": 3,
  "resolved_at": null,
  "resolved_by": null,
  "resolved_in_release": null,
  "note": null,
  "analysis": {
    "status": "ready",
    "sentence": "3 people tried to pay today and could not. The Pay button does nothing since yesterday's change.",
    "confidence": "high",
    "runs": 3
  },
  "pull_request": null,
  "url": "https://sypnia.example.com/console/problems/pay-button-does-nothing",
  "created_at": "2026-03-12T09:13:02Z"
}

The analysis is shortened here; its full shape is in the API reference. occurrences is 5 and not 15: each click produced three failures and one occurrence.

What changed from the system this comes from

ThereHereWhy
Error groups keyed by fingerprint and stageProblems keyed by project, environment and fingerprintMulti-project
Grouped once an hourEvery 30 seconds, by the fast pathThe 2-minute promise
Browser and server failures of one request were separate groupsOne problem, by traceBrief: one cause, one sentence
People and sessions per day, from arrays capped at 200The same per hour for the timeline, plus problem_hits for an exact lifetime count«3 people» is the product’s sentence; it must be exact
A 4xx was never a server issue; the browser kept its codeA failed call is no answer, 400, or 500 and above, in the browser tooSo an app with only the browser SDK gets problems
Symbolication inside the hourly job, 200 groups an hourAfter each new or returning problemA new problem needs its real file within the 2 minutes, for the agent
Chunk and digest rules of one framework inside the fingerprintThe same rules, as one list per platformAny framework, not one
hash64 over UTF-16 units, in JavaScriptThe same, bit for bit, in RustExisting groups keep their fingerprints

Next

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close