Skip to main content

​
After reading this

  • You will know what each health endpoint checks
  • You will know how the scheduler’s advisory locks prevent duplicate work
  • You will know how to back up, restore, and verify a restore actually works
  • You will know how to upgrade safely

​
Health and logging

All public probe responses expose only status and version. With PostgreSQL down, liveness stays 200 and readiness becomes 503. Every service rotates Docker JSON logs (10 MB x 3 files by default). The Next.js runtime cache is pruned at startup and every six hours, retaining at most 512 MB and seven days by default. Tune these without editing Compose:

​
Scheduler and queue runbook

The scheduler is a single, restartable worker. It runs the following jobs on fixed intervals using PostgreSQL advisory locks — a second scheduler container or restart receives 409 Already running and skips rather than doing duplicate work: Queue rows left processing by a crash become eligible again after five minutes. Mail deliveries left sending beyond the reconciliation lease become unknown; they are never resent automatically because SMTP/IMAP may have accepted the message before the process lost its response. Investigate the provider outcome before resolving them.

​
Candidate email retries and attempts

Every claim of an email_outbox row counts as one attempt, including claims that end in a crash or a stall:
  • A failed attempt is retried with a growing delay (1, 2, 3, then 4 minutes).
  • After five attempts the row ends failed with its last error. It is not retried again automatically. It stays in Settings → Email → Delivery log, where an operator can Retry it (one new attempt, now) or Cancel it.
  • A row whose worker stalls is reclaimed while attempts remain. If the fifth attempt also stalls, the row ends failed with an explanatory error instead of looping.
  • Once the provider has accepted an email, the row is recorded as accepted and is never sent again, even if the local bookkeeping that follows fails. If the success record keeps failing, the row is held as failed with its provider message ID, so no automatic path or Delivery log action sends it again.
  • SMTP can accept a message and then stop responding. When that happens the outcome is unknown. The row ends failed with a Delivery outcome unknown note and is not retried automatically. Check the sent folder before retrying it from the Delivery log.
  • SMTP waits are bounded: 10 seconds to connect, 10 seconds for the greeting, and 20 seconds of socket inactivity.
  • Retry records the previous attempt count and error in the row’s payload (operatorRetries) before it starts a fresh attempt.
The email-outbox job runs for at most 120 seconds: it starts no new row after that and hands unstarted rows back to the queue. The scheduler’s timeout for the job is 150 seconds, so a normal run always finishes first. Failed runs record a short error message with no addresses or query parameters.

​
Cron run records

cron_runs uses one vocabulary: succeeded, failed, or skipped. The app records the runs it executes, including the error text of failed runs. The scheduler records a fallback row only when the app recorded nothing for that run (for example a 409 skip or an unreachable app). A request that timed out is not recorded by the scheduler, because the app may still finish and record its own result. Rows older than 30 days are pruned by the scheduler every six hours. Metrics count each run once, and older success rows are reported as succeeded. Only succeeded runs count as a fresh success: a skipped tick means another run still holds the lock, so it does not refresh a job’s last-success time in the doctor.

​
Email warnings in doctor output

The doctor’s email check reports counts of failed, overdue pending, and stuck processing outbox rows. It shows WARNING when failed rows exceed 10, overdue pending rows exceed 25, or any processing row is stuck. A warning does not fail the healthcheck: a bad candidate email should not restart or take the instance out of rotation. Review the Delivery log when it appears. The scheduler container’s healthcheck runs its built-in doctor. It requires a recent successful-or-skipped record for each scheduled job, not merely a running container. The configured value is the minimum stale window; slower jobs also get a window of two scheduled intervals so daily reconciliation jobs are not falsely reported as stale:
Change this only when the instance is intentionally suspended for longer than five minutes. After a deployment, a restart, or an alert, run these from the installation directory:
The last command reports freshness per job plus email and webhook queue counts.
Do not manually invoke cron URLs or put CRON_SECRET in a URL. If a job is stale, first confirm app readiness, then restart only the scheduler with docker compose restart scheduler.
If queue counts grow while scheduler runs are fresh, fix the corresponding SMTP, webhook destination, or mailbox credentials; retries remain durable in PostgreSQL. Alert on an unhealthy scheduler, a stale job, and disk usage or backup failures — not merely on a non-zero queue count. Uploads and PostgreSQL data are never pruned automatically. Monitor them with docker system df -v and keep encrypted backups outside the VPS. The image exposes serve, migrate, scheduler, and doctor. Runtime migrations use committed SQL through drizzle-orm; Drizzle Kit and the TypeScript toolchain are build-time only.

​
Backup, restore, and upgrade

Backup stops app and scheduler, keeps PostgreSQL running, creates a pg_dump -Fc, includes local uploads and config, and checksums every included file before it can succeed. By default it creates a local rollback archive readable only by the deployment owner — this is convenient for a failed update, not a replacement for off-host disaster recovery. Restore verifies every archived file checksum, creates a safety backup before changing data, stops the app, scheduler, and migration services to avoid a database race, restores PostgreSQL and local uploads, then requires readiness to succeed. A failed restore keeps the safety archive under backups/.
S3 objects are not embedded in the archive. Use bucket versioning and an off-provider backup policy alongside npx @harly/cli backup.
Upgrades take a backup first, run the single migrator, wait for readiness, and finish with doctor. Migrations are forward-only — restore the full backup when a release does not declare schema-compatible image rollback. For managed platforms, use their managed PostgreSQL backups and snapshots together with S3 bucket encryption and versioning; they do not require installing age.

​
Verify your backups actually restore

A backup you have never restored is a guess. Prove recovery end-to-end against a throwaway installation with the destructive harness:
It seeds known rows and an uploaded object, backs them up, drops the entire database schema and deletes the upload, restores from the archive, and fails loudly unless row counts, a content checksum, the owner and user count, and the upload’s SHA-256 all come back identical.
This harness refuses to run without HARLY_DESTRUCTIVE_OK=1 because it wipes the database it points at. Never aim it at production.

​
Development database

Repository contributors use a separate development Compose file, not the production stack:
It lives at tooling/docker/compose.dev.yml. Development needs Node 22+, pnpm, Docker, about 4 GB of available RAM, and 10 GB of free disk for a comfortable full-monorepo workflow. Turbopack’s persistent filesystem cache is disabled to prevent multi-gigabyte .next growth. pnpm clean removes generated workspace caches without touching PostgreSQL volumes.

​
Seeding the demo workspace

pnpm db:seed:demo replaces every workspace-scoped row of the target workspace with synthetic demo data. The CLI refuses to run until you name that workspace in the confirmation, so an accidental run cannot wipe real hiring data:
--confirm-wipe=<workspace-slug> works too. The slug must match exactly. The guard counts every row in a workspace-scoped table except configuration (settings, custom roles, document access roles and pipeline stage definitions) and the checklist and sample-data bookkeeping rows. A workspace with more than 25 such records must also be pinned with DEMO_WORKSPACE_ID. Never run this against a production database that holds real candidates. With DEMO_MODE=true, the scheduled demo reset also requires DEMO_WORKSPACE_ID. Without it the reset refuses to run and deletes nothing. The reset never wipes a per-visitor sandbox; those expire on their own.

​
Database extensions

​
Workspace search and the pg_trgm extension

Workspace search uses trigram indexes when PostgreSQL has the pg_trgm extension. The migration that creates them tries to install the extension and never fails the upgrade if it cannot: managed PostgreSQL plans sometimes refuse CREATE EXTENSION. Without the extension search still works, but name, email and title lookups scan the tables. Install pg_trgm before you upgrade, with a role that may create extensions, or ask your provider to enable it. The migration only runs once, so if the extension is added later the trigram indexes are not created retroactively.

​
Troubleshooting

​
Scheduler doctor reports a stale job

Symptom: docker compose exec -T scheduler node /app/runtime.mjs doctor shows a job past its staleness window. Cause: the scheduler container restarted, hung, or the instance was suspended longer than HARLY_SCHEDULER_STALE_AFTER_SECONDS. Solution: confirm app readiness first, then docker compose restart scheduler. Do not invoke cron routes manually.

​
Restore fails checksum verification

Symptom: npx @harly/cli restore exits before touching data. Cause: the archive is corrupted or was modified after the backup ran. Solution: use a different backup archive. The safety backup created before the attempted restore is preserved under backups/.

Self-hosting overview

VPS requirements and DNS setup.

Proxy modes

Choose Caddy, an external reverse proxy, or local-only.