Operating it

OnceFirst production setup, in this order

Done alone, against production. It happens once and never again, which is exactly why it is written down.

%%{init: {"flowchart": {"padding": 14, "nodeSpacing": 34, "rankSpacing": 40, "useMaxWidth": true}}}%%
flowchart TB
  B1["1 migrate 0001 → 0005"]
  B2["2 fix the legal entities
real registered name and number"]
  B3["3 import people"]
  B4["4 check the tree
exactly one person with no supervisor"]
  B5["5 import money,
then delete both files"]
  B6["6 break the super_admin deadlock
one grant, directly in SQL"]
  B7["7 sign in with your own account"]
  B8["8 verify the floor
from a second, ordinary account"]

  B1 --> B2 --> B3 --> B4 --> B5 --> B6 --> B7 --> B8
  B4 -.->|"wrong? stop —
compensation goes nowhere"| B3
  B8 -.->|"only then tell
anyone it exists"| DONE(["live"])

1. Migrate

   0001 through 0005. Creates the schema, the permission catalogue, RLS, and the departments and legal entities that employees point at.

2. Fix the entities — before importing anyone

   Set the real registered name and registration number on both. 0005 deliberately leaves the numbers null rather than plausible-looking, because these appear on statutory filings. The Indonesian entity’s details are not yet known, and this step cannot be completed today.

3. Import people

   Run the importer against the master workbook. It emits SQL rather than applying it — read the file, then run it.

4. Check the tree — before compensation goes anywhere

   Exactly one employee may have no supervisor, and it must be the right one. Spot-check five reporting lines against the tracker.

5. Import money

   Read the compensation file, run it, then delete both files. They hold real salaries, bank accounts and national ids, and they are gitignored for that reason.

6. Break the super_admin deadlock

   The step nothing else can do for you. A fresh production database has nobody holding core.role.manage, and the sign-in trigger grants only employee — so until one grant is made directly in SQL, the platform is permanently unadministerable, including by you.

   Then grant the rest through the app, so the audit trail records them.

7. Sign in

   Your own Workspace account. The trigger creates your Account against the Employee row imported in step 3.

8. Verify the floor — from a second, ordinary account

   The base compensation table must return zero rows; the self-access view must return exactly one, theirs. This is the only test that the RLS policies work in production rather than in CI. Do it before anyone else is told the platform exists.

Weekly-ishThe release loop

Roughly fifteen minutes. Never on a Friday. Never during payroll week.

%%{init: {"flowchart": {"padding": 14, "nodeSpacing": 40, "rankSpacing": 42, "useMaxWidth": true}}}%%
flowchart TB
  R1["1 · Restore
last night's dump, onto your machine"] --> R2["2 · Rehearse
apply the migrations
run the payroll suite"]
  R2 -->|"fails"| STOP["STOP
production is untouched
fix the migration, start again"]
  R2 -->|"passes"| R3["3 · Migrate
the same migrations,
against production"]
  R3 -->|"fails halfway"| DIAG["DO NOT PROMOTE
schema is partly migrated,
old code still running — which
usually still works.
Diagnose before anything else."]
  R3 -->|"ok"| R4["4 · Promote
ff-only merge, push release
this is what deploys"]
  R4 --> R5["5 · Smoke
sign in · one payslip ·
one leave balance ·
the alert channel is quiet"]

1. Restore

   Last night’s dump, onto your machine. Encrypted disk or anonymised dump — real salaries, real ids. Not optional and not automated: tying the restore to every release is how you find out the backups work, weekly, instead of during an incident.

2. Rehearse

   Apply the new migrations to the restored copy, run the payroll suite. If this fails, stop. Nothing has touched production. Fix the migration and start again.

3. Migrate

   The same migrations, against production. If this fails halfway, do not promote. The schema is partly migrated and old code is still running — which usually still works, because old code ignores new columns. Diagnose before doing anything else.

4. Promote

   Fast-forward release and push. This is what deploys.

5. Smoke

   Sign in · open one payslip · check one leave balance · check the alert channel is quiet.

Migrations before promotion, always. New code against an old schema breaks; old code against a new schema usually survives.

NightlyBackups, restore, upgrades

Backups

Nightly pg_dump from the jobs service to R2, encrypted at rest, 35-day retention, access logged. An alarm fires when a dump does not land — take it seriously the first time, because a silently broken backup is the normal way self-hosted setups discover they have no backups. Check monthly that last night’s object exists and is a plausible size; a 4 KB dump is an error message.

Uploaded files live in R2 via the Storage backend, not on a Zeabur volume, so they are not part of the Postgres dump and do not need separate backing up.

Restoring production

You are here because data is wrong or gone. Slow down — the wrong restore loses more than the incident did.

1. Stop writes

   Take the Node service down so nothing lands mid-restore.

2. Snapshot what you have now

   However broken. If the restore is wrong you cannot get back without this.

3. Pick the dump

   Nightly means up to 24 hours of loss. If the damage is narrower, consider extracting the affected rows rather than restoring everything.

4. Restore into a scratch database first

   And look at it. Never straight over production.

5. Restore, bring the service back, smoke test

6. Write down what happened

   While you remember. Especially how far back the loss went, and who needs telling. If a closed payroll month is involved, check the Statutory Rate versions survived — a closed month must reproduce its original figures.

Upgrades

Pinned versions across about ten containers; nothing upgrades itself. Bump dev, leave it a week and use it, then bump production in a release window one container at a time. Postgres major versions are a dump and restore, not in place, rehearsed fully first. Not during payroll week.

When it breaksFour failures and where to look

%%{init: {"flowchart": {"padding": 14, "nodeSpacing": 36, "rankSpacing": 40, "useMaxWidth": true}}}%%
flowchart TB
  D["The app is down
check cheapest first"] --> C1{"Cloudflare Pages —
latest deploy live?
domain attached, cert valid?
CNAME still resolving at GoDaddy?"}
  C1 -->|"no"| F1["fix Pages, or the GoDaddy record"]
  C1 -->|"yes"| C2{"Kong container up?
config unchanged
since the last upgrade?"}
  C2 -->|"no"| F2["the usual answer
everything Supabase goes through Kong"]
  C2 -->|"yes"| C3{"Postgres up?
disk full?"}
  C3 -->|"no"| F3["disk is the common one
bloat, WAL, or an unvacuumed table.
Dev shares this disk."]
  C3 -->|"yes"| C4["Node service —
read the logs, check it
can reach Postgres"]

CadenceMonthly and annually

Monthly

• Backup object exists and is a plausible size
• Restore rehearsed — happens automatically if you released this month
• ops.error_log: anything recurring that has been ignored? Group by fingerprint over the last 30 days
• Disk headroom on the Postgres volume
• Any container more than one version behind?

Annually

• Google OAuth client secret still valid
• Resend domain verification still valid
• The 19 December close and 1 January accrual actually ran
• The retention job has destroyed what it should have