Problem grouping
How failures become problems, with the exact fingerprint algorithm worked by hand, symbolication with source maps, the rule that makes one request one problem across browser and server, the states of a problem and what triggers an alert.
A problem is a group of failures with one cause. This page is how the engine decides that two failures are the same, how it turns a minified stack into your own files and lines, and how a problem moves between its states. It is the design behind Problems.
Grouping is done by the fast path: the reader that scans each new segment for failures every 30 seconds. Data flow shows where it sits.
From a failure to a problem
(1) What counts as a failure. The fast path looks at four kinds of line and ignores every other.
| Line | A failure when | Kind | Fingerprint comes from |
|---|---|---|---|
error (browser) | Always | error | The SDK: fp |
server_error | Always | error, or failed_call when src is span | The SDK: fp. The engine computes it for OTLP |
server_request | partial is true and no full record of the same trace exists | timeout | The SDK: fp |
request (browser) | No answer, 400, or 500 and above | failed_call | The engine: no SDK field carries it |
A 401, 403, 404, 409 or 422 is not a failure. Neither is a rage click or a slow page. Those are counted by the roll-up and never reach this code.
(2), (3), (4) are the next three sections. (5) is symbolication.
The fingerprint
A fingerprint is 16 hex characters. Failures with the same fingerprint, in the same project and environment, are the same problem.
It is computed where the failure happens, in the SDK, so that the SDK can also drop a repeat of the same error within 5 seconds. The engine carries the same algorithm for the cases where no SDK did the work: OTLP exceptions and failed browser calls. Both must give the same bytes for the same input, so the algorithm is written down here exactly, and the same test vectors run in TypeScript and in Rust.
The algorithm
For an error, three parts joined with |:
name | normalized message | function@file of the first frame of your own code-
name: the error’s name.Errorwhen there is none. -
The message, normalized, in this order:
Step Replace With 1 URLs <url>2 Paths of built chunks, such as /_next/static/chunks/…<chunk>3 UUIDs <uuid>4 Quoted strings of up to 200 characters "…"5 Runs of 12 or more hex characters <hex>6 Numbers #7 Runs of whitespace one space 8 Everything after the first 300 characters nothing -
The top frame: read the first 30 lines of the stack, in the V8 form (
at fn (file:line:col)) or the Gecko form (fn@file:line:col). Take the first frame whose file is not noise. Noise isnode_modules, the framework’s own runtime chunks,react-dom,scheduler,node:internaland<anonymous>. If every frame is noise, take the first. Then clean the file: remove the origin, the query and the hash; remove a build hash of the form-followed by 8 or more hex characters; cut everything before the build folder. The part isfunction@file, or empty when there is no stack.
If the framework gave the error a digest of 6 or more digits, the fingerprint is the hash of digest: and the digest instead, because that id is already stable.
For a failed call, the call, how it failed, and the request it happened in:
call | span name | status or error code | METHOD | routeFor a timeout: timeout | METHOD | route.
The hash, hash64, is FNV-1a with 32 bits, run twice over the same string with two different seeds, the two results written as 8 hex characters each and joined.
| Value | |
|---|---|
| Seed of the first half | 0x811c9dc5 |
| Seed of the second half | 0x050c5d1f |
| Prime | 0x01000193 |
| One step | h = (h XOR unit) × prime, keeping the low 32 bits |
| A unit | One UTF-16 code unit of the string |
Notice that the function takes no project and no time: a fingerprint depends only on the failure.
const PRIME: u32 = 0x0100_0193;
const SEED_A: u32 = 0x811c_9dc5;
const SEED_B: u32 = 0x050c_5d1f;
/// FNV-1a, 32 bits, over UTF-16 code units. Must match the SDKs bit for bit.
fn fnv1a_utf16(text: &str, seed: u32) -> u32 {
text.encode_utf16()
.fold(seed, |hash, unit| (hash ^ u32::from(unit)).wrapping_mul(PRIME))
}
/// 16 hex characters: two 32-bit hashes of the same text with different seeds.
pub fn hash64(text: &str) -> Fingerprint {
Fingerprint::from_halves(fnv1a_utf16(text, SEED_A), fnv1a_utf16(text, SEED_B))
}
pub fn error_fingerprint(name: Option<&str>, message: &str, stack: Option<&str>, digest: Option<&str>) -> Fingerprint {
if let Some(digest) = digest.filter(|d| d.len() >= 6 && d.bytes().all(|b| b.is_ascii_digit())) {
return hash64(&format!("digest:{digest}"));
}
let frame = stack
.and_then(top_frame) // first frame of your own code, V8 or Gecko
.map(|f| format!("{}@{}", f.function, normalize_frame_file(&f.file)))
.unwrap_or_default();
hash64(&format!(
"{}|{}|{}",
name.unwrap_or("Error"),
normalize_message(message),
frame,
))
}
pub fn call_fingerprint(span_name: &str, code: &str, method: &str, route: &str) -> Fingerprint {
hash64(&format!("call|{span_name}|{code}|{}|{route}", method.to_ascii_uppercase()))
}
pub fn timeout_fingerprint(method: &str, route: &str) -> Fingerprint {
hash64(&format!("timeout|{}|{route}", method.to_ascii_uppercase()))
}Worked by hand
The shop’s checkout page throws this when the payment is refused:
ApiError: POST /api/checkout failed with 400 (amount_required)
at o (https://shop.example/assets/checkout-3f9c.js:1:48211)
at async HTMLButtonElement.<anonymous> (https://shop.example/assets/checkout-3f9c.js:1:49002)| Step | Result |
|---|---|
| (1) Name | ApiError |
(2) Message, normalized. Rule 6 turns 400 into #; nothing else matches | POST /api/checkout failed with # (amount_required) |
(3) Top frame. The first frame, o, is your own code. Its file loses the origin, the line and the column. -3f9c is four characters, shorter than the eight a build hash needs, so it stays | o@/assets/checkout-3f9c.js |
| (4) Joined | ApiError|POST /api/checkout failed with # (amount_required)|o@/assets/checkout-3f9c.js |
(5) hash64, first half, seed 0x811c9dc5 | 15228057 |
(5) hash64, second half, seed 0x050c5d1f | 93228d19 |
| Fingerprint | 1522805793228d19 |
The first two steps of the first half, to check an implementation against: the string starts with A (unit 0x41) and p (0x70).
h0 = 0x811c9dc5
h1 = (h0 XOR 0x41) × 0x01000193 mod 2^32 = 0x811c9d84 × 0x01000193 = 0xc40bf6cc
h2 = (h1 XOR 0x70) × 0x01000193 mod 2^32 = 0xc40bf6bc × 0x01000193 = 0x5ad569f4
…
h86 = 0x15228057And the failure behind it on the server, the call that the payment service refused:
| Failure | Hashed string | Fingerprint |
|---|---|---|
| The server’s failed call | call|POST /v1/payments|400|POST|/api/checkout | 88a44b15205cb153 |
| If that request had hung instead | timeout|POST|/api/checkout | c405205e99c5aa6c |
88a44b15205cb153 is the problem of the example story. Why it is that one and not the browser’s is the next section.
One request, one problem
(2) in the first figure. When the customer clicks Pay, three failures are reported for the same request:
- the server’s call to the payment service answered
400(server_error, from@sypnia/node); - the page’s call to
POST /api/checkoutanswered400(request, from@sypnia/browser); - the page threw
ApiError(error, from@sypnia/browser).
Three problems for one click would be noise. The rule: the deepest failure in a trace is the problem; what the person saw is evidence inside it.
How the fast path applies it:
| Failure | What the fast path does |
|---|---|
| A failure from the server, with a trace id (1) | It is the problem. One row goes into trace_failures: «trace 7c2e9a4b… belongs to problem prb_8f3k2m9x4q7w», kept 48 hours. If the same trace has two server failures, the one in the deepest span wins. |
A failed request from the browser, with a trace id (2) | Look up the trace in trace_failures. Found: it is evidence of that problem. It adds no occurrence; it adds the person and the session if the server record did not carry them. Not found, and the project has sent server data: hold it for 15 seconds in held_failures, then look again. Still not found, or the project has only the browser SDK: it is its own problem. |
A browser error in the same batch and session as a failed request, at most 2 seconds after it (3) | It follows that request: evidence if the request is evidence, part of the request’s problem otherwise. |
A browser error with no failed request before it | Its own problem. This is most browser errors: a TypeError in your code has no request behind it. |
The hold exists because the two halves travel separately. The server’s record usually arrives first, since the server sends when the request ends and the browser waits up to 10 seconds to send its batch. The 15 seconds cover the opposite order. They are part of the 2-minute budget worked out in Data flow, and they are spent only by projects that send server data.
A project with only the browser SDK gets the page’s failed call as the problem, exactly as Problems says.
Symbolication
A browser reports the stack of the code it ran: minified, in one line, with names like o. Source maps turn it back into your files.
(5) in the first figure. Symbolication happens after the problem exists and before the agent reads it. It never changes the fingerprint: grouping uses the minified frame, so a problem does not split when maps arrive late or are missing.
| Rule | |
|---|---|
| When | When a problem is born or comes back; again when the maps of its release are uploaded after the failure; and for the first occurrence in each new release |
| Which release | The ver of the occurrence. No ver, no symbolication |
| Finding the map | sourcemaps/<release>/index.json maps each built file to its map. Bundlers name maps independently of chunks, so guessing <file>.map is only the fallback |
| Frames | The first 30. V8 and Gecko forms. A frame with no map stays as it is |
| Output | sample.stack_original, the stack with original file:line:column and function names, and where: the file, line and function of the first frame of your own code |
| Limits | A cache of 50 maps or 200 MB; a single map over 64 MB is skipped; 60 seconds for the step |
| On failure | The problem keeps its minified stack. The step never blocks a problem or an alert |
For the checkout error the result is:
ApiError: POST /api/checkout failed with 400 (amount_required)
at confirmPayment (src/checkout/checkout.tsx:142:11)
at async onPayClick (src/checkout/PayButton.tsx:31:5)One number is exposed for this step: problems symbolicated ÷ problems that had a release and a stack, over the last 24 hours, shown by sypnia doctor. The reason is an incident in the system this comes from: for six days no error was symbolicated, because a build option broke the map reader, and nobody noticed. A log line is not a health number.
The states of a problem
A problem has one status, open, resolved or ignored, and an open problem can carry marks: new, growing, came back. Together they are the states below.
The engine decides a transition with one pure function. It receives the problem as it was before this segment and as it is after, and answers what happened. Notice that it cannot write anything: the caller writes the transition, the alert and the ledger row in one transaction, so a transition can never be recorded twice.
#[derive(Debug, PartialEq)]
pub enum Transition {
Born,
CameBack,
Growing { from: u32, to: u32 },
Quiet, // counted, nothing to tell anyone
}
pub fn transition(
before: Option<&ProblemState>,
after: &ProblemState,
seen: &Seen, // what this segment added: releases, people
rules: &GrowingRule, // factor 2, min_people 5, unless the project changed them
) -> Transition {
let Some(before) = before else {
return Transition::Born;
};
match before.status {
Status::Ignored => Transition::Quiet,
Status::Resolved => {
// «Old tabs»: with a fix release named, only a failure in that release or a later one reopens.
let reopens = match &before.resolved_in_release {
Some(fix) => seen.releases.iter().any(|r| r.is_same_or_later_than(fix)),
None => true,
};
if reopens { Transition::CameBack } else { Transition::Quiet }
}
Status::Open => {
let threshold = before
.people_at_last_alert
.saturating_mul(rules.factor)
.max(rules.min_people);
if after.people >= threshold && after.people > before.people_at_last_alert {
Transition::Growing { from: before.people_at_last_alert, to: after.people }
} else {
Transition::Quiet
}
}
}
}Follow the example of Problems through it. The problem is born with 1 person, and people_at_last_alert becomes 1. At 3 people the threshold is max(1 × 2, 5) = 5: quiet. At 5 people: growing, and people_at_last_alert becomes 5. The next threshold is 10, then 20.
«Later» for a release means first seen by the engine at the same time as the fix release or after it. The engine does not compare version strings: a short git commit has no order.
What a transition writes
All in the one transaction that also marks the segment as scanned:
| Transition | problems | Alert | The agent |
|---|---|---|---|
| Born | A new row, status = open, people_at_last_alert = people | problem.new | Symbolicate, then analyse |
| Came back | status = open, regressed = true; resolved_at and resolved_by stay as history | problem.regressed | Symbolicate, then analyse again |
| Growing | growing_until = now + 24 h, people_at_last_alert = people | problem.growing | Analyse again, inside the daily limit per problem |
| Quiet | Counters and last_seen only | None | Nothing |
What triggers an alert
For problems, exactly the three transitions above, and only when the project’s rule for that kind is on for the problem’s environment. By default the three are on for production only.
Each alert row has a dedupe_key with a unique index, which is what makes «one problem is one alert» a property of the database instead of a hope:
| Alert | dedupe_key | So that |
|---|---|---|
problem.new | problem.new:<problem id> | A problem is new once in its life |
problem.regressed | problem.regressed:<problem id>:<resolved_at> | Once for each time it was resolved |
problem.growing | problem.growing:<problem id>:<people at last alert> | Once for each step: 5, 10, 20 |
The alert is inserted with released_at empty and release_by set to 100 seconds after the failure reached ingest. It is released when the agent has written the sentence, or at release_by, whichever is first. Releasing means: set released_at, write the title (the sentence, or the technical title), and hand it to each alert sink. The agent’s pipeline shows that hand-off, and Deployment shapes shows the sinks.
The other alert kinds (uptime, a page that got slower, an API route failing, usage, the engine itself) are produced by the pass and the roll-up from numbers, not by this code. Their rules are in Alerts and the daily summary.
A problem, as the API returns it
After three people and five failures, GET /v1/problems/pay-button-does-nothing answers with the row of problems, its analysis, and links:
{
"id": "prb_8f3k2m9x4q7w",
"slug": "pay-button-does-nothing",
"fingerprint": "88a44b15205cb153",
"kind": "failed_call",
"origin": "server",
"status": "open",
"regressed": false,
"title": "createPayment answered 400 in POST /api/checkout",
"environment": "production",
"route": "/checkout",
"where": { "file": "src/checkout/checkout.tsx", "line": 142, "function": "confirmPayment" },
"first_seen": "2026-03-12T09:12:04Z",
"last_seen": "2026-03-12T10:31:47Z",
"first_release": "4f2a91c",
"last_release": "4f2a91c",
"occurrences": 5,
"people": 3,
"sessions": 3,
"resolved_at": null,
"resolved_by": null,
"resolved_in_release": null,
"note": null,
"analysis": {
"status": "ready",
"sentence": "3 people tried to pay today and could not. The Pay button does nothing since yesterday's change.",
"confidence": "high",
"runs": 3
},
"pull_request": null,
"url": "https://sypnia.example.com/console/problems/pay-button-does-nothing",
"created_at": "2026-03-12T09:13:02Z"
}The analysis is shortened here; its full shape is in the API reference. occurrences is 5 and not 15: each click produced three failures and one occurrence.
What changed from the system this comes from
| There | Here | Why |
|---|---|---|
| Error groups keyed by fingerprint and stage | Problems keyed by project, environment and fingerprint | Multi-project |
| Grouped once an hour | Every 30 seconds, by the fast path | The 2-minute promise |
| Browser and server failures of one request were separate groups | One problem, by trace | Brief: one cause, one sentence |
| People and sessions per day, from arrays capped at 200 | The same per hour for the timeline, plus problem_hits for an exact lifetime count | «3 people» is the product’s sentence; it must be exact |
A 4xx was never a server issue; the browser kept its code | A failed call is no answer, 400, or 500 and above, in the browser too | So an app with only the browser SDK gets problems |
| Symbolication inside the hourly job, 200 groups an hour | After each new or returning problem | A new problem needs its real file within the 2 minutes, for the agent |
| Chunk and digest rules of one framework inside the fingerprint | The same rules, as one list per platform | Any framework, not one |
hash64 over UTF-16 units, in JavaScript | The same, bit for bit, in Rust | Existing groups keep their fingerprints |
Next
- What happens to a problem after it is born.
- The same rules, from the reader’s side.
- How to upload the maps this page uses.