Reliability and observability

CR0 baseline and recovery — 2026-10-22

The baseline recorded image failures, unready backend pods and failed backups. The GKE recovery restored readiness, expanded MinIO from 30 to 60Gi and verified a standard CronJob backup in 38 seconds. The independent GCS copy reconciles 3709 current object names/sizes, verifies six small/large sample hashes and restores SQL using a key outside GKE. On 2026-10-23 the user accepted desktop/report behavior and the live backend fallback/return preserved two ready replicas, SQL aggregates and volume identities: CR0 is complete. The nightly backup still targets MinIO; recurring offsite backups, point-in-time consistency, all-object content hashes, broader RPO/retention governance and future Cloud Run rollback qualification remain separate from this completed baseline.¹

Terminal retirement — publication boundary, 2026-10-23

Backend/Admin terminal retirement is deployed on GKE: new interactive terminals and relay operation are disabled, while command/script Remote Actions and protected read-only history remain. The repository also removes the native PTY runner; no new installed-desktop release was published as part of CR0. Explicit Redis resource cleanup remains separate. See the retirement plan and publication evidence; earlier local implementation descriptions below are historical.²

Reliability model

The API can be replicated, but any state that must survive a request or pod belongs in PostgreSQL or durable object storage. Process memory is suitable only for caches whose loss is harmless.

Implemented durability model

• OAuth challenges, sessions, refresh families and one-time exchange codes live in PostgreSQL and use atomic consumption.
• Import and synchronization jobs use persisted status, lease owner/expiry, heartbeat, retry count, checkpoint, cancellation and expired-lease recovery.
• The desktop monitoring queue is encrypted by monstaff-core with a per-user AES-256-GCM key kept in the OS credential store, replays with persisted exponential back-off up to a terminal failure, and sends lease and idempotency headers without persisting bearer tokens.
• Server idempotency records deduplicate timer, activity and screenshot writes; PostgreSQL also enforces one running timer per user.
• Desktop-update upload uses a checksum-bound presigned path or a bounded stream instead of buffering an unbounded artifact.
• Screenshot deletion is recoverable: metadata tracks deletion state and orphan reporting instead of silently losing the object relationship.
• GitHub, AI and S3 clients have bounded timeouts; retry policy distinguishes transient from permanent failures.
• Agent telemetry uses a durable PostgreSQL ingestion outbox, leased ClickHouse cursor and retryable governance jobs; local load/backup/delete/restore preserves all 20,000 synthetic rows.
• Report run, delivery, schedule and retention workers use persisted leases/checkpoints. Unknown provider outcomes fail closed, object deletion is due-time tracked and portable bundles are digest verified before import.

Health semantics

The backend exposes:

• /health/live: process viability only;
• /health/ready: a bounded PostgreSQL readiness check;
• optional dependency status: S3, GitHub and AI providers, surfaced without necessarily removing the pod from service.

Implemented telemetry

• structured, redacted request logs and request IDs;
• route request count and latency histograms;
• import/sync queue depth and oldest-pending-age gauges;
• OAuth outcome counters without codes or tokens;
• denial events for security-sensitive authorization failures;
• an optional PrometheusRule bundle under k8s/observability/ for availability, latency and queue-age symptoms.

Operational objectives

These are initial operational objectives, not measured production claims:

Signal	Initial objective
Login, timer and screenshot API availability	99.9% monthly
Core write latency	p95 below 1 second
Bounded admin-report latency	p95 below 2 seconds
Oldest pending durable job	below 5 minutes during normal capacity

Production alert thresholds and error budgets require owner approval after representative traffic measurement.

Performance direction

Report aggregation is executed by bounded database queries; GitHub synchronization is isolated in durable jobs. Bounded local regressions cover 1,000 customers/5,000 engagements, 5,000 dashboard work items/7,500 transitions, 500 compliance installations and 20,000 Hubstaff source records; real renderer and ClickHouse recovery canaries pass. Production p95, memory, authorized-provider behavior and traffic-derived objectives remain rollout evidence.³⁴

Recovery boundary

Kubernetes includes an encrypted database/object backup CronJob, versioned MinIO buckets, disruption budgets and a restore-drill script. The integration suite successfully dumps PostgreSQL, restores into a second empty database and verifies migration plus representative domain counts. A successful production-equivalent isolated cluster/object restore with measured RPO/RTO is still required before calling disaster recovery verified.⁴

Confirmed input-activity incident (2026-09-08)

Production native sessions inspected for September 5–8 contain regular activity snapshots with zero aggregate input counts. The installed 0.3.27 Core reports Capturing and an empty delivery queue, which does not prove successful event collection. The macOS collector incorrectly passes the common-modes pseudo-mode to CFRunLoopRunInMode; Apple requires a concrete mode. Separately, the backend overwrites the session percentage with each batch percentage instead of aggregating the full session. These are confirmed source defects; correction and real-device delivery verification remain open. Historical percentages cannot be reconstructed from the inspected zero-input snapshots. This incident qualifies the earlier capture/readiness claims and is distinct from agent telemetry.⁵

Input activity repair — implemented locally

The native listener now runs in the macOS default mode and reports capability loss on an unexpected loop exit. The backend derives activity from persisted observed seconds under a per-entry transaction lock, preserving contributions across quiet, duplicate, reordered and concurrent batches. The real OS dispatch regression detects the original defect; database integration passes. This supersedes the implementation-open part of the September 8 diagnosis; published and installed live verification are still pending. Previously absent input events cannot be reconstructed.⁶

Version 0.3.29 preparation passed full local checks, builds, integration and native regressions. Publication and installed live evidence remain pending.⁷

The 0.3.29 native CI gate exposed pre-existing Tauri formatting drift. Formatting is corrected and the candidate advances to 0.3.30; local native formatting and Clippy pass. The activity logic is unchanged. Delivery remains pending.⁸

Idle decisions after activity resumes

A locally reproduced Core defect rejected Discard and Stop with IDLE_DURATION_UNAVAILABLE when resumed input reset the live OS idle counter to zero. Keep bypassed this duration check. The local correction preserves the pending episode duration separately until resolution, allowing both commands to use the original inactivity duration. The negative-control regression fails with the old accessor and passes with the correction; installed-client qualification and release remain pending.⁹

Input activity 0.3.30 — published and verified live

Version 0.3.30 is published from 2c00d47 with all tag-bound Quality, release and promotion jobs successful. The cumulative backend 34598db is deployed with two healthy updated replicas. A signed/notarized Apple Silicon DMG was installed locally after the old updater downloaded but deferred installation; GUI and Core report 0.3.30 and existing input permissions remain granted. Real input counters now reach production. Live aggregation matched 54% from 298/550 observed seconds despite a latest 100% batch; a later latest-zero batch retained the correct 42% from 332/780. Timer context and agent capture were restored, with no pending delivery queue. This closes the activity incident’s release/live gates, while historical missing events remain unrecoverable and the separate updater deferral is not fixed here.¹⁰

Database audit operational findings — 2026-09-08

Current live observation qualifies the configured-backup statements above: the three retained backup Jobs are failed, the latest reports DeadlineExceeded, and the CronJob has no lastSuccessfulTime. No external backup or restore was verified by this audit. PostgreSQL retains eight chat rows with nonempty content beyond the current organization retention setting; projectless activity is excluded from the general retention filter, although none of the sampled projectless snapshots was already expired. The database engineering review records evidence, limits and recommended remediation. These findings remain open; the audit changed no runtime behavior.¹¹

Footnotes

1. CR0 live baseline and local recovery qualification. ↩

2. Local interactive-terminal and Redis retirement. ↩

3. Active workstream quality, load and recovery evidence. ↩

4. Residual active-plan repository-local closure. ↩ ↩²

5. Production and repository diagnostic evidence. ↩

6. Implementation and regression evidence. ↩

7. 0.3.29 local verification. ↩

8. 0.3.30 CI correction. ↩

9. Idle resolution duration repair. ↩

10. Published release and installed live evidence. ↩

11. Database audit evidence. ↩