Operations

Running a Basalt instance after the first docker compose up: putting it on the internet, keeping the data, upgrading, and the things that will surprise you. If you have not installed yet, start at Self-hosting.

Putting it on the internet

The app speaks plain HTTP on host port 8080 and terminates no TLS of its own. Put a reverse proxy in front of it — Caddy, nginx, Traefik — and give it three jobs: TLS, WebSockets, and (see below) blocking sign-up.

Caddy is the shortest complete example:

notes.example.com {
    reverse_proxy localhost:8080
}

That is enough: Caddy obtains a certificate, and its reverse_proxy passes WebSocket upgrades through unchanged.

With nginx you have to say so explicitly, because Basalt uses two WebSocket planes — /api/v1/collab for document editing and /api/v1/events for cache invalidation and presence — and neither works without upgrade headers:

server {
    server_name notes.example.com;
    # ... TLS directives ...

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;   # map $http_upgrade
        proxy_set_header Host       $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 3600s;   # collab sockets are long-lived
    }
}

If the editor sits on "Connecting…" forever while the rest of the app works, the WebSocket upgrade is being eaten by the proxy — that is the first thing to check.

Then two settings, both mandatory once a proxy exists:

1. BASE_URL=https://notes.example.com in infra/docker/.env. It drives auth cookies, callbacks and the cross-origin write check. Get it wrong and you can load the app but not stay signed in.
2. TRUST_PROXY — set it to your proxy's address or CIDR. Without it the sign-in rate limiter counts your proxy as the one client, so ten failed sign-ins by anyone lock out everyone. TRUST_PROXY=true is worse than leaving it off: it believes any X-Forwarded-For a client invents. See Configuration → Behind a reverse proxy.

Two domains: Basalt and Lithic side by side

Lithic is a second app on the same server, the same database and the same release (ADR-0018, ADR-0023) — an outliner over the same workspaces, served from its own domain. infra/docker/compose.services.yml is the shipped stack for that shape:

# infra/docker/.env
AUTH_SECRET=…
POSTGRES_PASSWORD=…
BASE_URL=https://basalt.example.com
TRUST_PROXY=172.16.0.0/12          # your edge proxy's network
BASALT_IMAGE_TAG=v1.4.0            # pin a release, not `main`
LITHIC_IMAGE_TAG=v1.4.0

docker compose -f infra/docker/compose.services.yml up -d --wait

Both containers speak plain HTTP (host 8080 and 8081 by default) and terminate no TLS. Point your reverse proxy at them by Host header, with the same WebSocket and timeout care as the single-domain snippets above — the collab socket reaches the app through both proxies, so a missing Upgrade header in either one produces the same permanent "Connecting…".

Four things about this topology are worth understanding before you change it.

Lithic's /api is proxied, not called cross-origin. The nginx container forwards /api to app so that the browser only ever sees one origin, and that is mandatory rather than tidy: the session cookie belongs to BASE_URL's origin, and the server refuses any mutating request or WebSocket upgrade whose Origin matches neither its own Host nor BASE_URL. A browser on the Lithic domain talking straight to the Basalt domain therefore gets a 403 at sign-in. Do not "simplify" it by pointing Lithic's fetches at the other host.

BASE_URL stays the Basalt domain. It is the instance's canonical origin — invitation links, password resets and auth callbacks all use it — and Lithic works from its own window.location rather than from this value.

Exactly one app replica, always. The collab server holds Yjs documents in process memory and the event bus is in-process (ADR-0005). A second replica would hold its own copy of every open document, and two people editing the same page through different replicas would diverge permanently — not a merge conflict, two different documents. So no autoscaling, no scale-to-zero, no replicas: 2. The lithic container has no such constraint (it is a file server), but there is nothing to gain by scaling it either.

Deploys replace containers. That is what the attachment volume below exists for; a PaaS that redeploys by replacing the app container is exactly the case that used to destroy uploads.

Closing registration

Anyone who can reach your instance can create an account. There is no invite-only mode, no allowlist, and no setting that turns sign-up off. On a public domain that means strangers get accounts.

They do not get your data — a new account has no workspace and no membership, so it sees nothing — but it is still an unbounded row in your user table and an attack surface you did not choose.

Until the product has a switch, block the endpoint at the proxy. Sign-up is a single route:

POST /api/auth/sign-up/email

Caddy:

notes.example.com {
    @signup path /api/auth/sign-up/email
    respond @signup 403

    reverse_proxy localhost:8080
}

nginx:

location = /api/auth/sign-up/email {
    return 403;
}

Open it briefly to create your own account, then close it and add everyone else through invitations — with the caveat in the next section.

Adding people

Workspace → Settings → Invitations creates an invitation and shows you a link to <BASE_URL>/invite/<token>.

Whether it is also emailed depends on you. With SMTP_HOST and MAIL_FROM configured (Configuration → Email) the invitation is sent to the address as well, in the language of your own browser, and the panel says so. With no mail configured nothing is sent and you deliver the link yourself — which is the default, and a perfectly good way to run this.

The link is shown once. The token is stored only as a hash, so it cannot be displayed again; if it is lost, revoke the invitation and create a new one.

One thing to know before you rely on it: the invitee needs an account, which means sign-up has to be open at the moment they register. If you closed it above, open it for the minute it takes them. The invitation is bound to the address it was sent to, so they have to register with that exact address.

Passwords

With mail configured, there is a password reset. The sign-in screen offers "Forgot your password?", the link that arrives works once and expires within an hour, and using it signs every existing session of that account out — somebody resetting a password has usually lost control of the account, and leaving the other session alive would make the reset theatre.

Without mail there is no reset, and the screen says so rather than leaving somebody waiting for a message that will never come: a reset works by proving you can read a mailbox, and an instance that cannot send to one has nothing to prove it against. There is deliberately no instance-level override — an operator who could reset any password would be an operator who could enter any workspace, which is exactly what the back office is built not to be.

So on an instance without mail, the only fix for a forgotten password is to delete the account row and have them register again — losing nothing but the login, since content belongs to the workspace:

docker compose -f infra/docker/compose.selfhost.yml exec db \
  psql -U basalt -d basalt -c "delete from \"user\" where email = 'them@example.com'"

Their workspace membership goes with the account, so re-invite them afterwards. Take a backup first.

When mail is configured but nothing arrives

Queued messages are rows in mail_message. The body is encrypted; everything you need to diagnose delivery is not:

docker compose -f infra/docker/compose.selfhost.yml exec db psql -U basalt -d basalt \
  -c "select created_at, kind, recipient, status, attempt, error
        from mail_message order by created_at desc limit 20"

status is pending (waiting or retrying), sent (the relay accepted it — not the same as "it arrived") or dead (given up on). error carries the relay's own refusal, which is usually the whole answer: "relay access denied", "sender address rejected". The metrics basalt_mail_enqueued and basalt_mail_deliveries{outcome="dead"} say the same thing without a shell.

Backup

Postgres — the source of truth

Documents, page tree, databases, users, permissions, comments and history all live in Postgres. Dump it while it runs:

docker compose -f infra/docker/compose.selfhost.yml \
  exec -T db pg_dump -U basalt -Fc basalt > basalt-$(date +%F).dump

Restore into a fresh database container:

docker compose -f infra/docker/compose.selfhost.yml \
  exec -T db pg_restore -U basalt -d basalt --clean --if-exists < basalt-2026-07-26.dump

The secret

Keep a copy of infra/docker/.env with your dumps. AUTH_SECRET is not in the database, and losing it signs every user out permanently — a restored dump with a new secret is a working instance nobody can get back into.

Volume snapshots

Snapshotting the basalt-db volume works as an alternative, but only with the stack stopped: a file-level copy of a running Postgres data directory is not guaranteed to be consistent. pg_dump is the online option.

Attachments

Uploaded images are not in the database, so pg_dump does not contain them. Where they are depends on one setting, and both answers are safe:

• STORAGE_DRIVER=local (the default) — on disk at /data/attachments inside the app container, which both shipped compose files mount as the named volume basalt-files. Back that volume up alongside your database dumps.
• STORAGE_DRIVER=s3 — in your bucket, backed up by whatever backs up your bucket. The recommended choice for a real deployment; see Configuration → Storage for the five lines it takes.

This used to be a data-loss bug, and if you installed before the volume was shipped you still have it. The local driver wrote into the container's own writable layer with nothing mounted, so anything that replaced the app container — up -d --build, the documented upgrade — deleted every uploaded image while its database row survived, leaving the page with an image block that serves 404. Nothing warned you.

Check whether your stack has the volume before your next upgrade:

docker compose -f infra/docker/compose.selfhost.yml \
  exec app sh -c 'mount | grep /data/attachments'

One line of output means the directory is a real mount and the bytes survive a container replacement. NO output means they do not.

Ask mount, not ls. The image itself creates /data/attachments (it has to, so Docker can initialise the volume with the right ownership), so the directory is always there and always listable — an ls succeeds identically whether the volume is mounted or the bytes are sitting in the container's writable layer waiting to be deleted. A check that cannot fail is worse than no check, because it reassures.

If there is no mount, pull the newer compose file and treat everything uploaded so far as lost on the next rebuild — but copy it out first, while the container is still running:

# Path depends on which version you are on. Newer images use /data/attachments;
# older ones kept the store inside the code tree.
docker compose -f infra/docker/compose.selfhost.yml \
  cp app:/data/attachments ./attachments \
  || docker compose -f infra/docker/compose.selfhost.yml \
       cp app:/app/apps/server/.storage ./attachments

A schedule, and the restore you have actually done

Everything above describes how to take a backup. Two things turn that into a backup you have:

Automate it, and put the dumps somewhere the server cannot reach. A dump on the same disk as the database survives a dropped table and nothing else. If the attachments already live in a bucket (STORAGE_DRIVER=s3), that bucket is the natural destination for the dumps too — one place off the machine, one set of credentials, one thing to check. Coolify and most PaaS layers can schedule a Postgres dump to an S3 target without any of the commands above; use that rather than a cron job you will forget you wrote.

A workable rhythm for a small instance: nightly dumps, kept 30 days, plus a monthly one kept a year. Storage is cents; the decision you are buying is "how far back can we go", and a month is short if a corruption is noticed late.

Restore one before you need to. A backup that has never been restored is a hypothesis. Do it once, into a throwaway stack, and time it:

# A scratch database, not the live one.
docker run --rm -d --name restore-test -e POSTGRES_PASSWORD=x -e POSTGRES_DB=basalt postgres:16-alpine
docker exec -i restore-test pg_restore -U postgres -d basalt --clean --if-exists < basalt-2026-07-29.dump
docker exec restore-test psql -U postgres -d basalt -c 'select count(*) from block;'
docker rm -f restore-test

If the row count looks like your instance, the dump is good. What you learn from the timing matters as much: knowing a restore takes four minutes changes what you do at 3 a.m., and knowing it takes two hours changes what you promise people.

Three things a dump does not contain, all of which have bitten somebody:

• AUTH_SECRET — keep a copy of .env with the dumps, or a restored instance is one nobody can sign in to.
• Attachments, unless they are in a bucket. With STORAGE_DRIVER=local the volume needs its own backup, and it needs to be taken at the same time as the dump — a database from Tuesday with images from Sunday is a half restore.
• Anything written after the dump. That is what the frequency above is for.

Export is not a backup

Settings → General offers a workspace export: a zip with one Markdown file per page, the images those pages reference, and a JSON manifest of the structure Markdown cannot express (collections, databases, fields, row memberships, views).

It exists so your content is portable — readable without Basalt, importable elsewhere — not so you can rebuild an instance from it. It carries no users, no permissions, no comment threads and no document history, and it contains only what the exporting account is allowed to read (pages it cannot read are skipped and counted in the manifest).

For restoring an instance, use pg_dump.

Upgrading

Back up first, whichever route you take. Migrations are forward-only — there is no downgrade, and rolling back means restoring the dump into the old version.

docker compose -f infra/docker/compose.selfhost.yml \
  exec -T db pg_dump -U basalt -Fc basalt > basalt-pre-upgrade.dump

From published images (compose.services.yml) an upgrade is a pull:

docker compose -f infra/docker/compose.services.yml pull
docker compose -f infra/docker/compose.services.yml up -d --wait

Which version you get depends on the tag in .env. BASALT_IMAGE_TAG=main follows the default branch and moves under you; a version tag (v1.4.0) stays put until you change it, and is what a deployment you care about should pin. The running instance tells you what it is: basalt_build_info{version=…} on /metrics.

From source (compose.selfhost.yml) it is a rebuild:

git pull
docker compose -f infra/docker/compose.selfhost.yml up -d --build --wait

Migrations run automatically when the new server boots, under an advisory lock, so a restart storm cannot run them twice. Both routes replace the app container, which is why the attachment volume above is not optional.

Monitoring

Three endpoints, aimed at three different machines.

Liveness and readiness are different questions

/api/v1/health answers from memory and touches nothing. That is deliberate: its answer may be allowed to restart the container, and a health check that talked to Postgres would turn a thirty-second database blip into a restart loop across every replica — killed for the failure of something they were waiting for anyway.

/api/v1/ready is the one that checks dependencies, and its answer only ever decides routing:

curl -s http://localhost:8080/api/v1/ready
{"status":"ready","checks":{"database":"ok","migrations":"applied"},"pendingMigrations":0}

It returns 503 with the same shape when the database does not answer within two seconds, or when migration files are present that this database has not applied. That second check is the one people skip and then regret: during an upgrade the new server listens before its migrations finish, and traffic routed into that window gets 500s from code expecting a column that is not there yet. Wire readiness — not health — into the proxy's upstream check.

Both are unauthenticated, because a load balancer holds no credential. Neither discloses anything: ok/unreachable, applied/pending, and a count.

Scraping

/metrics serves the Prometheus text format. By default it answers only a loopback caller — from inside the container — so a fresh instance behind a reverse proxy that forwards / does not publish its telemetry to the internet by accident:

docker compose -f infra/docker/compose.selfhost.yml exec app \
  wget -qO- http://127.0.0.1:3000/metrics | head

To scrape from your monitoring host, set METRICS_TOKEN (see Configuration → Monitoring; it needs an override file, like everything else that is not one of the three core variables) and give the scraper the token:

# prometheus.yml
scrape_configs:
  - job_name: basalt
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ['notes.example.com']
    scheme: https
    authorization:
      credentials: '<the value of METRICS_TOKEN>'

Once the token is set it is required from everywhere, loopback included. METRICS=off removes the endpoint entirely.

What is measured

Names are stable; this is the contract, not an implementation detail.

A stalled worker does not look like an idle one. The webhook dispatcher and the collaboration re-authorization sweep are polling loops: they tick on a timer whether or not there is work, so their delivery counters are flat both when nothing is happening and when the loop is dead. The heartbeat (basalt_worker_last_run_timestamp_seconds) is what separates the two. The materializer is different — it runs when a document changes, so its timestamp means "the search projection last caught up", and it should be read against edit traffic rather than against the clock.

What is deliberately not measured

No label anywhere carries a workspace name, a page title, a user, an email address, an IP address or any id. Two reasons, and either alone would be enough: a scrape endpoint is copied into monitoring systems, retained for a year and forwarded to third parties, which is not a purpose anyone consented to; and per-workspace or per-page labels grow without bound and are the standard way to kill a Prometheus server — taking your monitoring down with it.

The cost is honest: these metrics cannot tell you which workspace is slow. That question belongs to the admin back office and the audit log, which are authenticated and tenant-aware. /metrics answers "is the instance healthy, and which route is not".

Scraping is also free of database work: rendering the page reads counters that are already in memory. A scrape cannot add load to an instance that is already struggling — which is exactly when it gets scraped hardest.

The three alerts worth having

More alerts is not better monitoring. These three are the ones that page a person, chosen because each one fires only when users are already affected or about to be, and because between them they cover the three ways this system actually breaks: it stops answering, it runs out of connections, or a background loop quietly dies.

1. The instance is failing requests.

sum(rate(basalt_http_requests_total{status=~"5.."}[5m]))
  / sum(rate(basalt_http_requests_total[5m])) > 0.02

for 5m. Server errors are never normal — 4xx are, so they are excluded on purpose (a bad client or a wrong password must not page anyone). Rate over five minutes rather than a raw count, so a quiet instance with one error does not wake anybody. When it fires, break it down by (route): the point of the route label is that the answer is one line away.

2. The database pool is exhausted.

histogram_quantile(0.95, sum(rate(basalt_db_pool_acquire_seconds_bucket[5m])) by (le)) > 0.1

for 10m. This is the failure that presents as "everything is slow" with no slow query anywhere: every handler waits for a connection instead. The histogram is used rather than basalt_db_pool_connections{state="waiting"} because the gauge is a sample — a queue that forms and drains between two scrapes is invisible in it, and that is precisely the shape of the incident. Ten minutes, because bursts are survivable and only a sustained queue means the pool is genuinely too small for the load.

3. A background worker has stopped.

basalt_worker_up{worker=~"webhook_dispatcher|collab_reauth"} == 0
  or time() - basalt_worker_last_run_timestamp_seconds{worker=~"webhook_dispatcher|collab_reauth"} > 300

for 5m. Both halves are needed and the second alone is the trap: if the loop never started, the heartbeat series does not exist and an age comparison is never true — the alert stays silent for exactly the failure it was written for. basalt_worker_up == 0 covers that case. This is the only alert here that fires with no user-visible symptom yet, and that is the reason it exists: a dead dispatcher means webhooks silently stop arriving, and a dead re-authorization sweep means someone whose access was revoked keeps their editing socket. Both are discovered days later, by the wrong person.

Why not a fourth. The obvious candidate is latency — a p99 alert on basalt_http_request_duration_seconds. It is a dashboard panel, not a page: per-route latency is dominated by which routes were called, so it fires on a change of usage as readily as on a regression, and the ADR-0011 budgets already gate that in CI where the hardware is known. Quota refusals, failed sign-ins and webhook dead-letters belong on a dashboard for the same reason — they are business signals to look at during working hours, not reasons to wake somebody. Every alert that can fire without anyone acting on it makes the three above less likely to be believed.

What is traced

Nothing, unless you asked for it. Tracing is off, and an instance with no OTEL_* configuration does not load the code — see Configuration → Tracing for the four variables and how to point them at a collector.

It is worth being clear about when this is and is not the thing you want. Everything above answers what is failing, how often, how slowly, and it does so with no second piece of software. A trace answers one question those cannot: this single request took 900 ms — where did they go? If you are not asking that, tracing costs you a collector to run and buys you nothing.

When it is on, six spans exist:

Read that table for what is missing as much as for what is there. There is no span per database statement, which is what the stock OpenTelemetry instrumentation would give you: dozens of spans per request repeating what basalt_db_pool_acquire_seconds already says, at a cost paid by every request and a bill paid per span. Where a query genuinely is the story it has a span of its own. The background workers are not traced either — their problem is throughput, which is what the heartbeats and counters above are for.

The same rule as the metric labels applies, and is asserted against a real collector in the test suite: no span carries a workspace, page, user, title, address or URL. Route patterns, counts, booleans and closed enums, and nothing else. So a trace tells you where the time went, never whose request it was — and, like /metrics, it cannot answer "which workspace is slow". That is the price of being safe to forward to a third-party backend, which is what a tracing pipeline is.

If the collector is unreachable, spans are dropped, a warning is logged at most once a minute, and requests are unaffected.

Troubleshooting

The app container will not start, logs say invalid server configuration. The environment failed validation and the message names the variable. Most often AUTH_SECRET shorter than 32 characters, or a BASE_URL that is not a full URL (it needs the scheme).

docker compose up --build fails during the web build with initial JS is … over the … budget. This is a build-time quality gate on the app bundle, not a configuration problem — the tree you checked out cannot currently be built. Check out a tag or commit where it passes.

Everything loads but sign-in never sticks. BASE_URL does not match the URL in the browser's address bar. Cookies are issued for the BASE_URL origin and the cross-origin check rejects writes from anywhere else.

The editor says "Connecting…" forever. The /api/v1/collab WebSocket is not getting through the proxy. See the nginx snippet above; with HTTP/2 in front of a HTTP/1.1 backend, some proxies complete the upgrade and then fail to bridge the data frames.

Sign-in returns 429 for everybody at once. The rate limiter is counting your reverse proxy as the client. That is TRUST_PROXY — see Configuration.

Images render broken after an upgrade. The attachment store was in the container rather than in a volume, so replacing the container took the bytes with it. This is fixed in the shipped compose files, so it means you upgraded from a stack that predates the volume — see Attachments for the check and what is recoverable.

Is it alive?

curl -fsS http://localhost:8080/api/v1/health     # -> {"status":"ok"}
curl -fsS http://localhost:8080/api/v1/ready      # -> {"status":"ready", …}
docker compose -f infra/docker/compose.selfhost.yml logs -f app

The container also carries a Docker HEALTHCHECK against the health endpoint, so docker compose ps reports health without any extra tooling. ready is the one to ask when the app is up but behaving strangely right after an upgrade — {"checks":{"migrations":"pending"}} says the schema is still catching up. See Monitoring for what else the instance will tell you.