feat(analytics): enforce finite retention in the worker service #10

Merged

eleith merged 2 commits from feat/analytics-retention into main 2026-11-05 19:52:07 +00:00

Conversation 0 Commits 2 Files Changed 41

+1224 -199

eleith-gitea-agent Bot commented 2026-11-05 03:31:41 +00:00

Copy link

Copy source

Reworked after #11

This replaces the previous in-web retention implementation. The worker-service foundation in #11 has landed; scheduled retention now belongs to the separate OpenWorkflow service. Web's Piscina thread remains exclusively for bounded, nonblocking visit classification and insertion.

Policy and storage

• Replace analytics.enabled / retention_days with analytics.retention_months: 0 | 3 | 9 | 12, default 3. Update YAML and editor-schema consumers accordingly; old keys are rejected.

• Compute retention cutoffs in UTC calendar months, clamping month-end dates while preserving time. Delete expired visits in indexed passes of up to 1,000 rows.

• Zero disables web collection and pauses the native recurring analytics schedule. Existing visits receive a one-off purge; an empty database queues no cleanup run. The service and its operational polling remain available.

• Both paths remain configuration-relative and distinct:

  database:
    path: ./data/goto.sqlite
    worker_path: ./data/worker.sqlite
  

• goto.sqlite gains the timestamp index and one operational visit_collection row. collect_after = NULL disables insertion; a timestamp fences out visits captured before or at the last re-enable. The ingestion helper checks that row inside its existing single INSERT, so queued visits cannot recreate purged data or return after a quick re-enable. No HTTP SQLite wait, per-visit workflow, new queue, link/configuration mirror, lifetime counter, or identity tracking is added.

• A positive workflow transaction completes any interrupted zero-policy purge before reopening collection with a new fence, and rechecks current policy under the owned connection. Retention zero still initializes an empty visits schema, but creates no ingestion pool or recurring analytics work.

• Keep When's two migration callers, with a public SQLite adapter migration-lock hook that reserves the writer before Kysely reads migration history. Transactional DDL remains enabled, and the existing five-second busy timeout is configured before WAL setup. Lock waits remain bounded; this is not an unlimited-startup-availability claim or a new hard dependency of redirects on worker health.

Service and lifecycle

• Local workflow spec and implementation under apps/worker/src/workflows/; purpose-specific retention and lifecycle services follow When's layout.
• Native hourly UTC schedule with 1–60 seconds of scheduled-only jitter, chosen in a persisted step and applied using native durable sleep. Startup and live policy maintenance—including zero-policy purge—remain immediate. No custom scheduler, Node delay timer, shared jobs package, or dummy workflow.
• Pause the schedule before starting the engine at zero; explicitly resume it for positive policies. Already-delayed scheduled work does no cleanup if collection is disabled when it resumes.
• Await policy workflow completion before accepting a subsequent configuration candidate. Drain configuration application before stopping the engine, its backend, and visit storage. Restart-only database changes reject the entire candidate; failed policy reconciliation requests controlled shutdown while keeping zero intent effective.
• Preserve Directory's tslog 5.2.0 and sanitized engine messages; no Pino dependency.
• Split ingestion pool creation/draining from admission and acknowledgement handling. Preserve the 1,000-outstanding / 32 KiB event bounds, strict FIFO, and best-effort failure policy.
• Preserve ordinary web HMR. Only application/recorder/pool lifetime files and compiled ingestion artifacts require targeted restarts.
• Use .js for the ESM service smoke script. Add DB source-resolution/mount support for service development, its actual workspace dependency, and DB paths/checks in both publishing pipelines. Production remains compiled JS without development loaders.

There are still no statistics/report queries, dashboard changes, invalid/base visit expansion, or derivative summaries in this PR. Zero and shorter policies delete history; test destructive changes only with disposable data or an explicit owner-approved copy.

Commits

1. c9e60c2 — finite configuration, async monitor contract, storage safeguards, and web admission.
2. 1c75121 — service workflows, scheduling/jitter, lifecycle, packaging/CI, and focused tests.

The old draft commit was replaced on explicit owner approval, using a branch-specific force-with-lease. Existing PR discussion is preserved; no merge, release, tag, or deployment was performed.

Verification

• Final readability, correctness/security, test-quality, and When/Directory reference-parity reviews are clean.
• The test-review finding is resolved by one focused real-SQLite unit: hold the connection, start positive retention, change policy to zero, release, and assert collection remains disabled. No migration test harness or matrix was added.
• Full pnpm verify passed: 457 unit tests, 7 web integration tests, and 5 service-process smoke cases; schema/builds/types/formatting/ESLint pass, with Svelte zero errors/warnings.
• Process cases cover compiled and source startup, storage-change shutdown, real live 3 → 0 → 9 retention, and occupied-port partial-startup cleanup. Timeouts fail the tests; shutdown must be natural and output assertions wait for stream closure.
• Fresh offline pruned production artifact tested outside the repository: dependencies resolve inside the artifact; tslog present/Pino absent; real compiled service startup pruning; private health; zero purge; native schedule pause/resume; stale insert rejection both while disabled and after re-enable; fresh insertion; natural SIGTERM code 0 and secret-safe drained output. The compiled DB helper is exercised here; the existing web HTTP story covers the actual compiled Piscina transport.
• Worker/config/DB scoped lint, Changesets status, and Git whitespace checks passed. The committed tree was compared against every reviewed/passing file hash, including the smoke-script rename.

These are local checks. Remote CI must pass on the updated head. Agent-run Docker/Compose verification was not performed; container/manual checks remain owner-only. Owner application data/configuration was not accessed or modified, and the development services were kept stopped.

## Reworked after #11 This replaces the previous in-web retention implementation. The worker-service foundation in #11 has landed; **scheduled retention now belongs to the separate OpenWorkflow service**. Web's Piscina thread remains exclusively for bounded, nonblocking visit classification and insertion. ## Policy and storage - Replace `analytics.enabled` / `retention_days` with **`analytics.retention_months: 0 | 3 | 9 | 12`**, default **3**. Update YAML and editor-schema consumers accordingly; old keys are rejected. - Compute retention cutoffs in UTC calendar months, clamping month-end dates while preserving time. Delete expired visits in indexed passes of up to 1,000 rows. - Zero disables web collection and pauses the native recurring analytics schedule. Existing visits receive a one-off purge; an empty database queues no cleanup run. The service and its operational polling remain available. - Both paths remain configuration-relative and distinct: ```yaml database: path: ./data/goto.sqlite worker_path: ./data/worker.sqlite ``` - `goto.sqlite` gains the timestamp index and one operational `visit_collection` row. `collect_after = NULL` disables insertion; a timestamp fences out visits captured before or at the last re-enable. The ingestion helper checks that row **inside its existing single INSERT**, so queued visits cannot recreate purged data or return after a quick re-enable. No HTTP SQLite wait, per-visit workflow, new queue, link/configuration mirror, lifetime counter, or identity tracking is added. - A positive workflow transaction completes any interrupted zero-policy purge before reopening collection with a new fence, and rechecks current policy under the owned connection. Retention zero still initializes an **empty visits schema**, but creates no ingestion pool or recurring analytics work. - Keep When's two migration callers, with a public SQLite adapter migration-lock hook that reserves the writer before Kysely reads migration history. Transactional DDL remains enabled, and the existing five-second busy timeout is configured before WAL setup. Lock waits remain bounded; this is not an unlimited-startup-availability claim or a new hard dependency of redirects on worker health. ## Service and lifecycle - Local workflow spec and implementation under `apps/worker/src/workflows/`; purpose-specific retention and lifecycle services follow When's layout. - Native hourly UTC schedule with **1–60 seconds of scheduled-only jitter**, chosen in a persisted step and applied using native durable sleep. Startup and live policy maintenance—including zero-policy purge—remain immediate. No custom scheduler, Node delay timer, shared jobs package, or dummy workflow. - Pause the schedule before starting the engine at zero; explicitly resume it for positive policies. Already-delayed scheduled work does no cleanup if collection is disabled when it resumes. - Await policy workflow completion before accepting a subsequent configuration candidate. Drain configuration application before stopping the engine, its backend, and visit storage. Restart-only database changes reject the entire candidate; failed policy reconciliation requests controlled shutdown while keeping zero intent effective. - Preserve **Directory's tslog 5.2.0** and sanitized engine messages; no Pino dependency. - Split ingestion pool creation/draining from admission and acknowledgement handling. Preserve the 1,000-outstanding / 32 KiB event bounds, strict FIFO, and best-effort failure policy. - Preserve ordinary web HMR. Only application/recorder/pool lifetime files and compiled ingestion artifacts require targeted restarts. - Use `.js` for the ESM service smoke script. Add DB source-resolution/mount support for service development, its actual workspace dependency, and DB paths/checks in both publishing pipelines. Production remains compiled JS without development loaders. There are still no statistics/report queries, dashboard changes, invalid/base visit expansion, or derivative summaries in this PR. Zero and shorter policies delete history; test destructive changes only with disposable data or an explicit owner-approved copy. ## Commits 1. `c9e60c2` — finite configuration, async monitor contract, storage safeguards, and web admission. 2. `1c75121` — service workflows, scheduling/jitter, lifecycle, packaging/CI, and focused tests. The old draft commit was replaced on explicit owner approval, using a branch-specific force-with-lease. Existing PR discussion is preserved; no merge, release, tag, or deployment was performed. ## Verification - Final readability, correctness/security, test-quality, and When/Directory reference-parity reviews are clean. - The test-review finding is resolved by one focused real-SQLite unit: hold the connection, start positive retention, change policy to zero, release, and assert collection remains disabled. No migration test harness or matrix was added. - **Full `pnpm verify` passed:** **457 unit tests**, **7 web integration tests**, and **5 service-process smoke cases**; schema/builds/types/formatting/ESLint pass, with Svelte zero errors/warnings. - Process cases cover compiled and source startup, storage-change shutdown, real live `3 → 0 → 9` retention, and occupied-port partial-startup cleanup. Timeouts fail the tests; shutdown must be natural and output assertions wait for stream closure. - Fresh offline pruned production artifact tested **outside the repository**: dependencies resolve inside the artifact; tslog present/Pino absent; real compiled service startup pruning; private health; zero purge; native schedule pause/resume; stale insert rejection both while disabled and after re-enable; fresh insertion; natural SIGTERM code 0 and secret-safe drained output. The compiled DB helper is exercised here; the existing web HTTP story covers the actual compiled Piscina transport. - Worker/config/DB scoped lint, Changesets status, and Git whitespace checks passed. The committed tree was compared against every reviewed/passing file hash, including the smoke-script rename. **These are local checks. Remote CI must pass on the updated head.** Agent-run Docker/Compose verification was not performed; container/manual checks remain owner-only. Owner application data/configuration was not accessed or modified, and the development services were kept stopped.

eleith-gitea-agent Bot referenced this pull request 2026-11-05 14:56:51 +00:00

feat(worker): introduce the OpenWorkflow service foundation #11

eleith added 2 commits 2026-11-05 19:26:24 +00:00

feat(analytics): add finite retention policy and storage safeguards c9e60c25b2

feat(worker): run retention through native workflow schedules 1c75121e5d

eleith force-pushed feat/analytics-retention from cde8fc97b0 to 1c75121e5d 2026-11-05 19:26:24 +00:00 Compare

eleith-gitea-agent Bot changed title from feat(analytics): enforce finite visit retention to feat(analytics): enforce finite retention in the worker service 2026-11-05 19:27:49 +00:00

eleith merged commit 1c75121e5d into main 2026-11-05 19:52:07 +00:00

eleith deleted branch feat/analytics-retention 2026-11-05 19:52:07 +00:00

eleith-gitea-agent Bot referenced this pull request 2026-11-05 21:31:59 +00:00

refactor(analytics): make collection changes restart-only #12

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

eleith-gitea-agent Bot eleith (eleith)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: eleith/goto#10