# Storage layout

> Where a self-hosted engine keeps each kind of data, how much space it takes, how to measure it, and which bucket rules to add and which to avoid.

Applies to: the Sypnia engine (open source, self-hosted). Status: at launch.  
Source: https://docs.sypnia.com/self-hosting/storage-layout/

For agents: Operator's view of where the self-hosted Sypnia engine stores data: the six bucket prefixes under each project id with their retention settings, the Postgres tables that grow, estimated sizes per million events and for a small and a medium project, commands to measure real usage, and which lifecycle, versioning and storage-class rules are safe on the bucket.

**\[Engine]** This page is for whoever operates a self-hosted engine and needs to answer three questions: where is each thing, how big will it get, and which rules should the bucket have. The design behind it, down to the object names and the table columns, is in [Storage](https://docs.sypnia.com/architecture/storage); this page does not repeat it. On Sypnia Cloud none of this is yours to look after.

The engine keeps data in two places and nowhere else: one bucket and one Postgres database. The engine's own disk holds nothing you need to keep.

## The bucket

Every key starts with the project's id. Under it there are six folders, one per kind of data.

```text
<storage.prefix>prj_5k2m9x4q7w1z/
├── events/      raw events, as they arrived          written by ingest
├── sessions/    the same events, one file per session and hour   written by the worker
├── replays/     replay chunks                        written by ingest
├── snapshots/   page snapshots, for click maps       written by ingest
├── sourcemaps/  one folder per release               written by your CI
└── parquet/     one file per day, for your own tools written by the worker
```

| Folder        | What makes it grow                        | Kept by default                         | Setting                            |
| ------------- | ----------------------------------------- | --------------------------------------- | ---------------------------------- |
| `events/`     | Events                                    | 90 days                                 | `SYPNIA_RETENTION_EVENTS_DAYS`     |
| `sessions/`   | Events that belong to a session           | 90 days                                 | The same                           |
| `parquet/`    | Events                                    | 90 days                                 | The same                           |
| `replays/`    | Recorded sessions, and how long they last | 30 days                                 | `SYPNIA_RETENTION_REPLAYS_DAYS`    |
| `snapshots/`  | Routes of your app, not traffic           | 90 days                                 | `SYPNIA_RETENTION_SNAPSHOTS_DAYS`  |
| `sourcemaps/` | Deploys                                   | 90 days after the release was last seen | `SYPNIA_RETENTION_SOURCEMAPS_DAYS` |

Three things to know as an operator:

- **`events/` is the source of truth.** Everything in Postgres that comes from events can be built again from it with `sypnia aggregate --from … --to …`. Never edit, move or rename objects in it.
- **`replays/` is where the space goes.** Recordings are about three quarters of a typical bucket.
- **`parquet/` is for you.** The engine writes it and never reads it. Point any tool that reads Parquet at a day of events.

If you set `SYPNIA_STORAGE_PREFIX`, everything above sits under that prefix, so the bucket can be shared with something else.

### Where the bucket is

| Deployment                                                                       | Where                                                               |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [AWS template](https://docs.sypnia.com/self-hosting/deploy-aws)                  | An S3 bucket. Its name is the stack output `BucketName`.            |
| [Docker Compose](https://docs.sypnia.com/self-hosting/deploy-docker), as written | The bucket `sypnia` inside MinIO, on the Docker volume `bucket`.    |
| Your own choice                                                                  | Whatever `SYPNIA_STORAGE_BUCKET` and `SYPNIA_STORAGE_ENDPOINT` say. |

## Postgres

Postgres holds what a screen or an MCP call reads: problems, the session index, the numbers per hour and per day, and who gets in. Raw events and replay content never go into it.

| What                                                                   | Grows with                              | Kept by default                            |
| ---------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------ |
| Projects, keys, tokens, settings, alert rules, webhooks, uptime checks | What you create                         | While the project exists                   |
| Problems, with their sentence, status and daily counts                 | Distinct failures, not traffic          | While the project exists                   |
| The session index: one row per session                                 | Sessions                                | 90 days (`SYPNIA_RETENTION_EVENTS_DAYS`)   |
| Hourly and daily numbers: performance, click maps, usage, uptime       | Routes of your app × hours, not traffic | 400 days (`SYPNIA_RETENTION_NUMBERS_DAYS`) |
| The trace index                                                        | Requests                                | 90 days                                    |
| The duplicate guard                                                    | Batches of the last 48 hours            | 48 hours                                   |

The tables that grow with time are partitioned by month. The engine creates the next partitions and drops the old ones itself, in plain SQL. It needs no extension, so any managed Postgres works. You do not need to schedule a vacuum or a cleanup job.

| Deployment                           | Where                                                                 |
| ------------------------------------ | --------------------------------------------------------------------- |
| AWS template, database created by it | An RDS instance. Its name is the stack output `DatabaseIdentifier`.   |
| Docker Compose, as written           | The database `sypnia` in the service `db`, on the Docker volume `db`. |
| Your own choice                      | Whatever `SYPNIA_DATABASE_URL` says.                                  |

## How big it gets

These are estimates. The first figure, about 123 bytes per stored event, was measured on the system this engine comes from; the replay figures were measured there too. The arithmetic for every row is in [Storage](https://docs.sypnia.com/architecture/storage#how-much-a-million-events-weigh).

| Per million events                                       | Size                                          |
| -------------------------------------------------------- | --------------------------------------------- |
| `events/`, `sessions/` and `parquet/`, about 125 MB each | ≈ 375 MB                                      |
| `replays/`, with the default sampling                    | ≈ 870 MB                                      |
| `snapshots/`                                             | A few MB, whatever the traffic                |
| **Bucket**                                               | **≈ 1.2 GB**                                  |
| **Postgres**, rows that grow with traffic                | **≈ 20 MB**, plus ≈ 30 MB that lives 48 hours |

Source maps are apart, because they grow with deploys: measured on the system this comes from, one deploy of a large app is 965 maps and 109 MB.

What that means once retention has filled up, with the defaults:

|                                             | Small: 130,000 events a day | Medium: 1.3 million events a day        |
| ------------------------------------------- | --------------------------- | --------------------------------------- |
| Events a month                              | 4 million                   | 40 million                              |
| `events/`, `sessions/`, `parquet/`, 90 days | 375 MB × 4 × 3 ≈ 4.5 GB     | ≈ 45 GB                                 |
| `replays/`, 30 days                         | 870 MB × 4 ≈ 3.5 GB         | ≈ 35 GB                                 |
| **Bucket, without source maps**             | **≈ 8 GB**                  | **≈ 80 GB**                             |
| Session index, 90 days                      | 10 MB × 4 × 3 ≈ 120 MB      | ≈ 1.2 GB                                |
| Hourly numbers, 400 days, for 100 routes    | 3 MB a day × 400 ≈ 1.2 GB   | The same: they do not grow with traffic |
| **Postgres**                                | **≈ 1.5 GB**                | **≈ 2.5 GB**                            |

Two things can break these numbers, and both are capped: one session can record at most 720 chunks, about 144 MB, and one session sends at most 5 page snapshots.

## Measure your own

Estimates stop mattering once you have a week of real data.

**The bucket, per project and per folder:**

**AWS**

```bash
aws s3 ls s3://<BucketName>/prj_5k2m9x4q7w1z/replays/ --recursive --summarize | tail -2
```

```text
Total Objects: 31204
   Total Size: 1065353216
```

**Docker**

```bash
docker compose run --rm --entrypoint sh bucket -c \
  'mc alias set local http://minio:9000 sypnia "$MINIO_PASSWORD" >/dev/null && mc du --depth 2 local/sypnia'
```

```text
1.0GiB	31204 objects	sypnia/prj_5k2m9x4q7w1z/replays
412MiB	9870 objects	sypnia/prj_5k2m9x4q7w1z/events
```

**Postgres, whole and by table:**

```sql
SELECT pg_size_pretty(pg_database_size(current_database()));

SELECT relname AS table, pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;
```

**What a project stored this month**, without touching the stores: `GET /v1/usage` returns its events, replays and analyses. See the [API reference](https://docs.sypnia.com/api/projects-keys-and-tokens).

## Rules on the bucket

The engine deletes expired data itself, once a day, by listing and deleting. It works the same on Amazon S3, Cloudflare R2 and MinIO, and it is the only retention you need. The bucket does not need lifecycle rules to stay small.

| Rule                                                | Add it?                                                 | Why                                                                                                                                                                                                                                                       |
| --------------------------------------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Remove unfinished multipart uploads after 7 days    | **Yes.** The AWS template adds it.                      | A source-map upload that was cut off leaves parts that no listing shows and that are billed.                                                                                                                                                              |
| Block all public access                             | **Yes.** The AWS template does.                         | Nothing in the bucket is ever read without a signed link.                                                                                                                                                                                                 |
| Expire everything in the bucket after N days        | **No.**                                                 | Source maps of a release that is still live would disappear, and stacks would turn unreadable. Retention per kind of data is the engine's job.                                                                                                            |
| Expire one folder of one project, as a second guard | Only if you want a guard against a worker that stopped. | A lifecycle rule filters by the start of the key, and every key starts with the project id, so it is one rule per project and folder: `prj_…/replays/`, for example. Give it more days than the engine's setting, so the engine always deletes first.     |
| Move objects to a colder storage class              | **No.**                                                 | Most objects are a few kilobytes, below what colder classes bill as a minimum, and the worker reads recent hours more than once. A replay must also open at once.                                                                                         |
| Versioning                                          | **Off**, unless you need it.                            | Per-session files are written again on every pass, so versions pile up. Worse, a [deleted person](https://docs.sypnia.com/privacy/deleting-data) would survive in old versions. If you turn it on, add a rule that removes old versions after a few days. |
| Replication to another region                       | Your choice.                                            | It doubles storage and write costs. A database backup protects what cannot be rebuilt; see [Upgrades, backups, retention](https://docs.sypnia.com/self-hosting/upgrades-backups-retention).                                                               |

To keep data for less time, change the retention settings, not the bucket: [Upgrades, backups, retention](https://docs.sypnia.com/self-hosting/upgrades-backups-retention#retention).

## What not to do by hand

- **Do not delete objects to free space.** Lower a retention setting instead: the engine then deletes the objects and the rows that point at them together. An object deleted by hand leaves a session in the list that answers `410 expired` when opened.
- **Do not delete rows to remove a person or a session.** Use the API, which removes the rows and the objects: [Deleting data](https://docs.sypnia.com/privacy/deleting-data).
- **Do not point two engines with different databases at the same bucket and prefix.** Each would delete what the other does not know. Use `SYPNIA_STORAGE_PREFIX` to share a bucket.

## Move to another bucket

Stop the engine, copy every object with its key unchanged, change the storage settings, start the engine and run `sypnia doctor`. Postgres needs no change: it stores keys, not addresses. SDKs retry while the engine is stopped, so a short stop loses nothing; see [what happens when a piece is down](https://docs.sypnia.com/architecture/data-flow#when-a-piece-is-down).

## Related
