Docs

Storage layout

Where a self-hosted engine keeps each kind of data, how much space it takes, how to measure it, and which bucket rules to add and which to avoid.

Applies to Engine At launch

View as Markdown
On this page
  1. The bucket
  2. Where the bucket is
  3. Postgres
  4. How big it gets
  5. Measure your own
  6. Rules on the bucket
  7. What not to do by hand
  8. Move to another bucket
  9. Related

Engine This page is for whoever operates a self-hosted engine and needs to answer three questions: where is each thing, how big will it get, and which rules should the bucket have. The design behind it, down to the object names and the table columns, is in Storage; this page does not repeat it. On Sypnia Cloud none of this is yours to look after.

The engine keeps data in two places and nowhere else: one bucket and one Postgres database. The engine’s own disk holds nothing you need to keep.

The bucket

Every key starts with the project’s id. Under it there are six folders, one per kind of data.

Text
<storage.prefix>prj_5k2m9x4q7w1z/
├── events/      raw events, as they arrived          written by ingest
├── sessions/    the same events, one file per session and hour   written by the worker
├── replays/     replay chunks                        written by ingest
├── snapshots/   page snapshots, for click maps       written by ingest
├── sourcemaps/  one folder per release               written by your CI
└── parquet/     one file per day, for your own tools written by the worker
FolderWhat makes it growKept by defaultSetting
events/Events90 daysSYPNIA_RETENTION_EVENTS_DAYS
sessions/Events that belong to a session90 daysThe same
parquet/Events90 daysThe same
replays/Recorded sessions, and how long they last30 daysSYPNIA_RETENTION_REPLAYS_DAYS
snapshots/Routes of your app, not traffic90 daysSYPNIA_RETENTION_SNAPSHOTS_DAYS
sourcemaps/Deploys90 days after the release was last seenSYPNIA_RETENTION_SOURCEMAPS_DAYS

Three things to know as an operator:

  • events/ is the source of truth. Everything in Postgres that comes from events can be built again from it with sypnia aggregate --from … --to …. Never edit, move or rename objects in it.
  • replays/ is where the space goes. Recordings are about three quarters of a typical bucket.
  • parquet/ is for you. The engine writes it and never reads it. Point any tool that reads Parquet at a day of events.

If you set SYPNIA_STORAGE_PREFIX, everything above sits under that prefix, so the bucket can be shared with something else.

Where the bucket is

DeploymentWhere
AWS templateAn S3 bucket. Its name is the stack output BucketName.
Docker Compose, as writtenThe bucket sypnia inside MinIO, on the Docker volume bucket.
Your own choiceWhatever SYPNIA_STORAGE_BUCKET and SYPNIA_STORAGE_ENDPOINT say.

Postgres

Postgres holds what a screen or an MCP call reads: problems, the session index, the numbers per hour and per day, and who gets in. Raw events and replay content never go into it.

WhatGrows withKept by default
Projects, keys, tokens, settings, alert rules, webhooks, uptime checksWhat you createWhile the project exists
Problems, with their sentence, status and daily countsDistinct failures, not trafficWhile the project exists
The session index: one row per sessionSessions90 days (SYPNIA_RETENTION_EVENTS_DAYS)
Hourly and daily numbers: performance, click maps, usage, uptimeRoutes of your app × hours, not traffic400 days (SYPNIA_RETENTION_NUMBERS_DAYS)
The trace indexRequests90 days
The duplicate guardBatches of the last 48 hours48 hours

The tables that grow with time are partitioned by month. The engine creates the next partitions and drops the old ones itself, in plain SQL. It needs no extension, so any managed Postgres works. You do not need to schedule a vacuum or a cleanup job.

DeploymentWhere
AWS template, database created by itAn RDS instance. Its name is the stack output DatabaseIdentifier.
Docker Compose, as writtenThe database sypnia in the service db, on the Docker volume db.
Your own choiceWhatever SYPNIA_DATABASE_URL says.

How big it gets

These are estimates. The first figure, about 123 bytes per stored event, was measured on the system this engine comes from; the replay figures were measured there too. The arithmetic for every row is in Storage.

Per million eventsSize
events/, sessions/ and parquet/, about 125 MB each≈ 375 MB
replays/, with the default sampling≈ 870 MB
snapshots/A few MB, whatever the traffic
Bucket≈ 1.2 GB
Postgres, rows that grow with traffic≈ 20 MB, plus ≈ 30 MB that lives 48 hours

Source maps are apart, because they grow with deploys: measured on the system this comes from, one deploy of a large app is 965 maps and 109 MB.

What that means once retention has filled up, with the defaults:

Small: 130,000 events a dayMedium: 1.3 million events a day
Events a month4 million40 million
events/, sessions/, parquet/, 90 days375 MB × 4 × 3 ≈ 4.5 GB≈ 45 GB
replays/, 30 days870 MB × 4 ≈ 3.5 GB≈ 35 GB
Bucket, without source maps≈ 8 GB≈ 80 GB
Session index, 90 days10 MB × 4 × 3 ≈ 120 MB≈ 1.2 GB
Hourly numbers, 400 days, for 100 routes3 MB a day × 400 ≈ 1.2 GBThe same: they do not grow with traffic
Postgres≈ 1.5 GB≈ 2.5 GB

Two things can break these numbers, and both are capped: one session can record at most 720 chunks, about 144 MB, and one session sends at most 5 page snapshots.

Measure your own

Estimates stop mattering once you have a week of real data.

The bucket, per project and per folder:

AWS

Terminal
aws s3 ls s3://<BucketName>/prj_5k2m9x4q7w1z/replays/ --recursive --summarize | tail -2
Text
Total Objects: 31204
   Total Size: 1065353216

Postgres, whole and by table:

SQL
SELECT pg_size_pretty(pg_database_size(current_database()));

SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;

What a project stored this month, without touching the stores: GET /v1/usage returns its events, replays and analyses. See the API reference.

Rules on the bucket

The engine deletes expired data itself, once a day, by listing and deleting. It works the same on Amazon S3, Cloudflare R2 and MinIO, and it is the only retention you need. The bucket does not need lifecycle rules to stay small.

RuleAdd it?Why
Remove unfinished multipart uploads after 7 daysYes. The AWS template adds it.A source-map upload that was cut off leaves parts that no listing shows and that are billed.
Block all public accessYes. The AWS template does.Nothing in the bucket is ever read without a signed link.
Expire everything in the bucket after N daysNo.Source maps of a release that is still live would disappear, and stacks would turn unreadable. Retention per kind of data is the engine’s job.
Expire one folder of one project, as a second guardOnly if you want a guard against a worker that stopped.A lifecycle rule filters by the start of the key, and every key starts with the project id, so it is one rule per project and folder: prj_…/replays/, for example. Give it more days than the engine’s setting, so the engine always deletes first.
Move objects to a colder storage classNo.Most objects are a few kilobytes, below what colder classes bill as a minimum, and the worker reads recent hours more than once. A replay must also open at once.
VersioningOff, unless you need it.Per-session files are written again on every pass, so versions pile up. Worse, a deleted person would survive in old versions. If you turn it on, add a rule that removes old versions after a few days.
Replication to another regionYour choice.It doubles storage and write costs. A database backup protects what cannot be rebuilt; see Upgrades, backups, retention.

To keep data for less time, change the retention settings, not the bucket: Upgrades, backups, retention.

What not to do by hand

  • Do not delete objects to free space. Lower a retention setting instead: the engine then deletes the objects and the rows that point at them together. An object deleted by hand leaves a session in the list that answers 410 expired when opened.
  • Do not delete rows to remove a person or a session. Use the API, which removes the rows and the objects: Deleting data.
  • Do not point two engines with different databases at the same bucket and prefix. Each would delete what the other does not know. Use SYPNIA_STORAGE_PREFIX to share a bucket.

Move to another bucket

Stop the engine, copy every object with its key unchanged, change the storage settings, start the engine and run sypnia doctor. Postgres needs no change: it stores keys, not addresses. SDKs retry while the engine is stopped, so a short stop loses nothing; see what happens when a piece is down.

Sections

Start hereWhat Sypnia is, how it works, what is open source and what is the paid cloud, and the two ways to get your first problem in plain words.ConceptsThe ten ideas Sypnia is made of, each defined precisely, from a project and its keys to problems, traces and alerts.SDKsHow your app sends data to Sypnia. One SDK for the browser, one for Node, one for Next.js, and the OpenTelemetry intake for every other backend.The agentHow Sypnia analyses a problem, what it writes, what it sends to a model and what it never does.MCP and skillsConnect your coding agent to Sypnia so it can ask what is failing in production, read what it needs to fix it, and set Sypnia up for you.API referenceThe Sypnia HTTP API: one API for the open-source engine and for Sypnia Cloud, with every endpoint, parameter, response and error.CLI referenceThe sypnia command: how to install it, the variables it reads, what its exit codes mean, and where each command is documented.Self-hostingWhat you run when you host the Sypnia engine yourself, what you need, what it costs, and what you do not get compared with Sypnia Cloud.How the engine is builtThe whole Sypnia engine on one page, from a click in your app to the sentence your coding agent reads, with the design rules and the code that holds them.Sypnia CloudHow the paid service works, from organizations and roles to plans, limits, billing, where your data lives, the GitHub App and support.Privacy and securityWhat Sypnia collects from your app and its users, what it never collects, how masking works, how to delete data, and how the service is protected.LicensingWhich licence covers each part of Sypnia, what you may do with it, how the name may be used, and the terms of a contribution.GuidesStep-by-step recipes for the common ways to use Sypnia, from apps made with AI builders and Next.js on Vercel to Supabase, source maps in CI, alert e-mails from your own engine, and moving from Sentry.ReferenceThe pages you look things up in, from the event schema and the glossary to what is planned, the changelog and the common questions.
↑↓ to move↵ to openEsc to close