eleith/hledger-web-reports
1
1 Fork 0
Code Issues Pull Requests Packages Releases Activity

Clarify report ownership and simplify docs and tests #43

Merged
eleith merged 1 commits from refactor/report-ownership-cleanup into main 2026-11-07 18:38:26 +00:00
Conversation 0 Commits 1 Files Changed 40
+993 -4209
40 changed files with 993 additions and 4209 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.changeset/README.md
+8 -26
View File
@@ -1,30 +1,12 @@
# Changesets
Like When, deployable apps use a fixed release group: `@hledger-reports/web` and
`@hledger-reports/worker` stay on the same version. Shared packages remain private
implementation dependencies, not independently released packages.
Add one short note per coherent change. Describe its effect plainly, not its
implementation history. Record shared-package changes under the affected app;
web and worker are versioned together.
- Include a descriptive changeset with each coherent review proposal. Explain the
behavior, scope, and important limitations; one note covers the change, not each file.
- For internal/tooling-only changes with no app-visible effect, use an empty
changeset (`pnpm changeset --empty`) and add a useful review summary. This records
the change without scheduling an app release.
- Run `pnpm changeset` to record an app change as a patch, minor, or major release.
Shared-package changes should describe their user-visible effect under the
affected app, not an ignored package. The fixed group versions both apps together.
- Run `pnpm changeset status` to inspect the pending release plan.
- On a clean, reviewed `main`, run `pnpm version:apps` to consume pending notes,
update app versions, and generate app changelogs. Review and commit those changes
and any lockfile update before tagging.
- Run `pnpm tag:apps`, then push the release commit and its two new app tags.
This creates tags for private apps; it does **not** publish to npm or push Git refs.
Use `pnpm changeset` to add a note and `pnpm changeset status` to inspect pending
releases. Adding a note does not release anything.
App tags are `@hledger-reports/web@X.Y.Z` and `@hledger-reports/worker@X.Y.Z`.
Their service-owned pipelines verify the tag matches the package version before
publishing container versions. Main keeps its existing `latest` and SHA images;
release tags publish `X.Y.Z`, `X.Y`, and `X` aliases without moving `latest`.
The initial workflow supports ordinary stable releases, not prerelease channels.
No versions are bumped and no release tags are created merely by adding a
changeset. Release commands remain explicit, as in When. There is no npm publish
step, release bot, or automatic deployment.
Releases require explicit approval: run `pnpm version:apps`, review and commit the
version/changelog changes, then use `pnpm tag:apps` and push the approved app tags.
Tags publish container images, not npm packages or deployments.
.changeset/clarify-report-ownership.md
+6
View File
@@ -0,0 +1,6 @@
---
'@hledger-reports/web': patch
'@hledger-reports/worker': patch
---
Clarify report preparation and sidebar data ownership. Shorten documentation and focus tests on our data handling and resource lifetimes rather than Chart.js or repeated hledger calculations. Financial results and app behavior remain unchanged.
.changeset/present-responsive-homepage.md
+1 -1
View File
@@ -25,7 +25,7 @@ count, recorded strict check, preparation time, saved hledger version and curren
Exact precision and incomplete valuations remain explicit throughout.
This release requires financial format 8. Follow the
[manual paired activation steps](../docs/development.md#financial-format-8-activation)
[upgrade guidance](../docs/development.md#private-data-and-upgrades)
for compatible web, worker and rebuilt shared packages, then explicitly prepare
reports again. Unsupported formats 7, 6 and older remain untouched; account IDs may
reset. There is no automatic preparation, migration or deletion. Authentication and
README.md
+19 -61
View File
@@ -1,75 +1,33 @@
# hledger web reports
A local, read-only financial app for an hledger journal. Hledger owns accounting;
the web reads saved reports and never opens journals or executes hledger.
A read-only financial app for an hledger journal. Browse saved reports and account
activity, or explicitly request preparation and live queries. Hledger owns accounting.
## Product
## Run locally
- **Overview** shows selected-period Income, Spending and Transactions, plus latest
whole-journal Net worth, with income/spending and category charts and exact tables.
- **Accounts** links to configured Cash/Investment registers and the saved
**Net worth** trend at `/reports/networth`.
- **Settings** owns explicit report preparation, status and a six-fact Journal
summary. Browsing, startup and journal edits never prepare reports automatically.
- **Live queries**, linked from Settings, run a separate recorded-cost report on
explicit request.
Use Docker with host networking enabled. Create private
`apps/web/config/hledger-reports.yaml` from the
[example](apps/web/config/hledger-reports.example.yaml), without overwriting existing
configuration. Select your journal and supply the [sign-in secrets](docs/authentication.md).
Exact quantities, separate commodities and incomplete valuations stay visible.
Saved timestamps describe preparation, not freshness. Saved reports remain readable
when the worker is unavailable. See [web usage and routes](apps/web/README.md).
## Start with Docker Compose
Create private `apps/web/config/hledger-reports.yaml` from the
[safe example](apps/web/config/hledger-reports.example.yaml), without overwriting
an existing configuration. Select your read-only journal tree under
`apps/web/journals/` (`main.journal`), or set `JOURNAL_DIR` to another directory.
Configure exactly one sign-in strategy and supply its secrets and `AUTH_SECRET`
through your secret tooling; see [authentication setup](docs/authentication.md).
No host Node, pnpm or hledger installation is needed for Compose.
The worker mounts `apps/web/journals/`, or `JOURNAL_DIR`, read-only at `/journal`.
Use that container path in the configuration.
```sh
cd apps/web
cd apps/web &&
mkdir -m 700 -p data &&
docker compose -f docker-compose.dev.yml up --build --wait
```
Open **http://127.0.0.1:5173/**, sign in, then use **Prepare reports** in **Settings**.
Reload Settings for status and navigate to saved results. Stop with
`docker compose -f docker-compose.dev.yml down`; keep the private data folder.
Open **http://127.0.0.1:5173/**, sign in and choose **Prepare reports** in Settings.
Reload for status. Browsing and journal edits never prepare reports automatically.
Both services share ignored **`apps/web/data/`**, containing sensitive SQLite jobs,
financial results and saved reports. Protect and back up the whole folder. Existing
named-volume users should follow the [safe copy guide](docs/development.md#inspecting-data-and-preserving-the-previous-volume).
Keep configuration, journals and the shared data folder private and backed up.
Upgrading from PR3's format 6 requires [manual paired format-8 activation](docs/development.md#financial-format-8-activation).
Unsupported files are preserved, not migrated, deleted or automatically regenerated.
## Guides
Sign-in remains mandatory. Development is loopback-bound; compiled preview needs
an existing [trusted HTTPS proxy](docs/authentication.md#trusted-proxy-contract).
This is not a public deployment recipe.
## Documentation owners
| Document | Owns |
| ---------------------------------------------- | --------------------------------------------------------- |
| [Web](apps/web/README.md) | User behavior, routes and presentation file map |
| [Architecture](docs/architecture.md) | Request, public-data and phase boundaries |
| [Native library](packages/hledger/README.md) | Hledger commands, exact output, dates, counts and version |
| [Snapshot store](packages/snapshots/README.md) | Format validation, commit and preservation |
| [Development](docs/development.md) | Activation, local/compiled runs, fixtures, checks and CI |
| [Worker](apps/worker/README.md) | Job execution, service resources and query retention |
| [Configuration](packages/config/README.md) | YAML, paths, labels and register membership |
| [Philosophy](docs/philosophy.md) | Narrow ownership and simplicity principles |
```text
apps/web SvelteKit saved views, Settings and live queries
apps/worker OpenWorkflow implementations and owned native fixtures
packages/config Configuration schema, validation and path resolution
packages/hledger Native execution and ready report data
packages/jobs Shared job definitions and queue connection
packages/snapshots Complete report publications and validation evidence
```
All workspace packages are internal. Source and tests are co-located; development
uses source exports and production uses compiled JavaScript.
- [Development and operations](docs/development.md)
- [Configuration](packages/config/README.md)
- [Authentication](docs/authentication.md)
- [Architecture](docs/architecture.md)
- [Development principles](docs/philosophy.md)
apps/web/README.md
+4 -226
View File
@@ -1,229 +1,7 @@
# Web app
Svelte 5/SvelteKit with adapter-node. The web displays saved reports and accepts
explicit background-work requests. It never opens journals or executes hledger.
See [architecture](../../docs/architecture.md) for request and public-data boundaries.
Displays saved financial reports and accepts explicit preparation and query requests.
Server loaders expose public page data; journals and private provenance stay off the client.
## Pages
| Page | Behavior |
| ----------------------------------------------------------- | ----------------------------------------------------------------------- |
| `/`, `/?months=1\|3\|6\|12` | Saved Overview; default 3 months |
| `/reports/networth`, `/reports/networth?months=1\|3\|6\|12` | Saved native net-worth history; default 3 months |
| `/registers/cash`, `/registers/investments` | Configured members from the saved account directory |
| `/registers/cash/<id>`, `/registers/investments/<id>` | Saved account activity with ordinary pagination |
| `/holdings` | Explicit investment holdings report for configured members |
| `/settings` | Preparation/status, Journal summary and link to Live queries |
| `/queries` | Explicit recorded-cost balance tool; reload clears its displayed result |
`/reports` is a namespace only, with no index or redirect; there is no
`/reports/net-worth` alias. `/prepare` has no redirect or POST alias.
Browsing does not check inputs, infer freshness, query job status or submit work.
Without a complete supported financial set, saved views say reports are unavailable;
use Settings to prepare them. Saved reports remain readable while preparing or
with the worker stopped. Open pages do not refresh automatically.
## Overview periods
Bare `/` and the header's Overview link select **3 months**. Choose **1/3/6/12
months**, or use a URL such as `/?months=12`; there is no remembered preference.
The route accepts exactly one literal `months` value or no query. Repeated, empty,
unsupported values and unknown keys return 400.
With JavaScript, selection and Back/Forward use already-loaded data without data,
API, job or native requests, preserving the snapshot, focus and scroll. Before or
without JavaScript, the native select and Apply button submit the same GET route.
Direct links and reloads can observe a newer saved set.
Four headlines show selected-period **Income**, **Spending** and **Transactions**,
plus **latest whole-journal Net worth**. Transactions counts actual primary-date
transactions, including transfers and opening balances, not postings or chart rows.
The window includes the latest recorded primary transaction's month and preceding
calendar months, not the computer's month. Latest net worth and Accounts remain at
the saved cutoff regardless of selection. There is no income-less-expenses headline.
Two equal **40rem** cards show monthly income/spending and depth-2 expense categories,
with exact tables visible by default. On wide screens each spans two headline tiles
plus their gap. Pinned headings and keyboard-accessible scrolling keep all table rows
available. The category plot stays in a bounded **19rem** slot without a chart scroll
wrapper; full accessible captions remain available when visual captions are clamped.
### Spending by category chart
One horizontal **Top 5 + Other** chart ranks signed net spending within a single
commodity, with native-order ties. Other appears last only when more than five
eligible categories exist and sums the remainder exactly; a real category named
Other remains distinct. Native zeros and refunds retain their signs. Missing
commodities are omitted, not turned into zero bars. Ambiguous duplicate amounts
for the same commodity in any row make that commodity's chart unavailable rather
than silently combining them.
After hydration, a native **Spending chart currency** selector appears when several
commodities are available. If a period removes the chosen commodity, it falls back
to an available choice. Period/currency changes reuse saved data without requests.
The chart selector is absent without JavaScript; the complete raw category table
still shows every commodity and category in native order, never filtered or truncated.
After that complete table, a native **Chart values** disclosure provides an ordered
text alternative to the chart, including exact grouped Other values and commodity.
The canvas is described by that list. The disclosure is available when the client
can produce the chart; it does not replace or modify the raw table.
This chart-only exact grouping does not convert currencies, change native totals
or authorize browser accounting elsewhere. See the narrow
[arithmetic principle](../../docs/philosophy.md#ownership-before-abstraction).
## Net worth report
The entire Net worth footer row in Accounts links to **`/reports/networth`** and
marks the active page. It stays available without configured groups or saved
balances, with honest unavailable values.
The report shows the saved twelve-point native trend and a visible exact table.
Its own 1/3/6/12-month selector defaults to 3 and has the same URL, history and GET
behavior as Overview. It selects loaded points without recalculation. The chosen
native currency and incomplete valuations remain explicit; selection does not
change latest NET or Accounts balances.
## Settings journal summary
Settings' ordinary **Prepare reports** form requests Overview and all account
registers together. Concurrent requests join active work; a later request prepares
again even with unchanged inputs. The empty form works without JavaScript and
redirects with 303 to `/settings`. Reload for this deployment's queue status, then
navigate to saved reports. There is no polling or automatic follow-up after edits.
A failed job does not guarantee unchanged reports; reload to inspect the available
result. See [publication/recovery](../../packages/snapshots/README.md#publication-and-reads).
Below execution status and controls, **Journal summary** shows six facts:
- **Journal**: full recorded date span (including posting/secondary dates), total
whole-journal transactions and last recorded strict-check outcome/time.
- **Saved report**: preparation time, saved hledger version and report currency.
The summary owns the single preparation/check timestamp display; status above is
execution-only. Validation is independent of saved finances: unavailable validation
does not hide readable reports, and a recorded pass is not a freshness guarantee.
No journal/request identity or native diagnostic reaches the browser.
Follow **Live queries** to `/queries` for a separate current-input calculation.
Each Run returns exact recorded-cost results; reload clears the display. Closing
HTTP does not cancel durable work, and there is no user-facing run history.
## Account labels and register membership
Configure exact Cash/Investment members and optional app-wide labels in shared YAML.
A register without members is hidden and returns 404, including direct account URLs.
Label-only entries do not enable registers. Missing saved members appear unavailable
without fabricated links or counts. Holdings uses investment membership; without
investment members it is unavailable too.
Labels appear throughout presentation while canonical identities remain available.
They do not alter native job inputs, report scope or saved IDs. Preparation still
covers all native accounts. See [account configuration](../../packages/config/README.md#account-presentation).
## Accounts navigation
Signed-in pages have a permanent right rail on desktop, a right overlay drawer on
medium screens and a full-screen drawer on narrow screens. The compact branded
header keeps Overview and Settings left, with accessible Accounts/sign-out icon
actions right. Sign-out is a native POST, including without JavaScript.
Cash/Investments headings open configured register directories; accounts link to
saved activity. Holdings is reached from an investment account register, not the
header or Accounts. Close, Escape, backdrop and navigation dismiss the drawer.
Without JavaScript, Accounts is a native disclosure with real links. Public/sign-in
pages have no financial navigation.
Accounts shows Cash raw balances, Investment valuations including cash, and native
whole-journal net worth—not sums of configured accounts. The balance cutoff is
visible, without a preparation timestamp. Unavailable and incomplete values stay
explicit. Shell and page observe the same saved set on navigation/reload.
## Financial presentation
Exact quantity strings and separate commodities remain visible. Display formatting
never rounds meaningful precision or converts quantities to floating point. Charts
use approximate geometry; exact tables and individual native tooltips retain the
producer values. Category Other is the sole chart-derived tooltip amount. Values
outside numeric range or ambiguous duplicate-commodity bags retain the exact table
instead of a misleading chart. Mixed/unconverted valuations are visibly incomplete,
not consolidated scalars or invented zeros. Empty and undated results are explicit.
See the [native contract](../../packages/hledger/README.md#overview-data) for financial
meaning, dates and valuation evidence.
Cash pages show 50 rows; Investment pages show 50 whole transactions. Full recorded
account activity, including declared-parent subtree history, is not filtered by
Overview's period. Native running balances do not reset at page boundaries.
First/Previous/Next/Last are ordinary links. Account names remain exact data, never
paths or slugs; IDs may reset across an unsupported-format upgrade.
### Transaction dialogs
Cash breakdowns and Investment details use a shared Bits UI shell. Escape/Close
dismiss; backdrop clicks do not. Focus returns to the opener or main content.
Navigation/replacement data clears the selection. The portal inherits theme and
supports desktop dialogs/narrow bottom sheets; Bits UI owns focus trapping and
scroll lock. Detail buttons are disabled before hydration; financial tables work
without JavaScript.
## Files and state
Compose shares private ignored **`apps/web/data/`** with the worker. Vite denies
raw journal/data/SQLite serving. Nothing here is a public static asset.
```text
data/snapshots/current/
overview.json
transactions/<account-id>.json
```
The worker publishes complete immutable sets. Server loaders project safe page data,
never entire stored envelopes or source/assertion positions. See
[storage and preservation](../../packages/snapshots/README.md#current-format-and-preservation)
and [manual paired activation](../../docs/development.md#financial-format-8-activation).
Do not delete volumes to simulate an empty app or copy into populated data folders;
use the [safe copy guide](../../docs/development.md#inspecting-data-and-preserving-the-previous-volume).
Sign-in and no-store financial responses remain mandatory. Configuration is selected
explicitly at startup; restart after YAML edits. Authentication, origins, POST guards
and compiled proxy requirements are owned by the
[authentication guide](../../docs/authentication.md); YAML by the
[config package](../../packages/config/README.md). `/healthz` is process liveness,
not financial readiness. Public `/schema.json` serves only the bundled editor schema.
## Run and check
Use the [development guide](../../docs/development.md#live-development) for startup,
[focused checks](../../docs/development.md#focused-local-checks) for test/build/browser
commands and disposable fixture setup. Tests are co-located with implementations;
route tests omit the reserved `+` prefix. Test each contract at its owning layer.
```text
src/lib/
charts/ ChartCanvas.svelte, theme.ts, theme.fixture.ts,
rendering.integration.test.ts
overview/ data.ts, select-period.ts, KeyFigures, FinancialValue
income-spending/ IncomeSpending.svelte and chart.ts
spending-categories/ SpendingCategories.svelte, presentation.ts and chart.ts
reports/
networth/ NetWorth.svelte and chart.ts
report-card/ ReportCard.svelte and private ScrollArea.svelte
report-period/ PeriodControls.svelte and period-query.ts (URL choices/parsing)
navigation/ AppHeader, AccountSidebar, AccountsDrawer and public data.ts
registers/ Cash and Investment presentation, transaction dialogs
holdings/ Investment holdings presentation
settings/ data.ts, JournalSummary and single-presentation ValidationStatus
queries/ Live-query presentation
server/ Request coordination (including preparation.ts), pinned saved reads
and public projections
amounts.ts Exact display-only amount formatting
dates.ts Calendar dates and explicit-UTC timestamps
styles/ Reset/theme, layout and native controls
```
Shared CSS is imported once. Feature styles retain financial table structure and
alignment. Chart colors/fonts resolve from CSS after mounting. Icons compile from
`virtual:icons/ph/*`; artwork lives in `static/branding/` and `static/favicon.ico`.
There is no runtime icon service or appearance configuration.
See the [quickstart](../../README.md#run-locally) and
[development guide](../../docs/development.md).
apps/web/src/lib/charts/rendering.integration.test.ts
-89
View File
@@ -1,89 +0,0 @@
import { execFile } from 'node:child_process';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { expect, it } from 'vitest';
it('renders extreme bars and lines with gaps and constant values without hanging tick generation', async () => {
// A Vitest timeout cannot interrupt Chart.js's synchronous scale loop; bound a child process.
const { stdout } = await promisify(execFile)(
process.execPath,
[
'--import',
fileURLToPath(new URL('../../../../../tools/register.js', import.meta.url)),
'--input-type=module',
'-e',
`
import assert from 'node:assert/strict';
import { theme } from ${JSON.stringify(new URL('./theme.fixture.ts', import.meta.url).href)};
import { Chart, BarController, BarElement, LineController, LineElement, PointElement, CategoryScale, LinearScale, Tooltip, Legend, BasicPlatform } from 'chart.js';
import { incomeSpendingChart } from ${JSON.stringify(new URL('../overview/income-spending/chart.ts', import.meta.url).href)};
import { spendingCategoriesChart } from ${JSON.stringify(new URL('../overview/spending-categories/chart.ts', import.meta.url).href)};
import { spendingCategoriesPresentation } from ${JSON.stringify(new URL('../overview/spending-categories/presentation.ts', import.meta.url).href)};
import { netWorthChart } from ${JSON.stringify(new URL('../reports/networth/chart.ts', import.meta.url).href)};
Chart.register(BarController, BarElement, LineController, LineElement, PointElement, CategoryScale, LinearScale, Tooltip, Legend);
const extreme = '17' + '0'.repeat(307);
const mixed = '85' + '0'.repeat(306);
for (const [positive, negative] of [[extreme, null], [null, '-' + extreme], [mixed, '-' + mixed]]) {
const amounts = quantity => quantity === null ? [] : [{ commodity: 'USD', quantity }];
const vertical = incomeSpendingChart([{
begin: '2026-04-01', endExclusive: '2026-05-01',
income: amounts(positive), expenses: amounts(negative)
}], 'USD', theme);
const presentation = spendingCategoriesPresentation([
{ account: 'Outgoings:Travel', amounts: amounts(positive) },
{ account: 'Outgoings:Refunds', amounts: amounts(negative) }
], 'USD');
assert.ok(presentation);
const horizontal = spendingCategoriesChart(presentation, theme);
const line = netWorthChart({ currency: 'USD', points: [positive, negative, positive].map((quantity, index) => ({
date: ['2026-02-28', '2026-03-31', '2026-04-16'][index], amounts: amounts(quantity), unconverted: []
})) }, theme);
const singleLine = netWorthChart({ currency: 'USD', points: [{
date: '2026-04-16', amounts: amounts(positive ?? negative), unconverted: []
}] }, theme);
for (const [config, axis] of [[vertical, 'y'], [horizontal, 'x'], [line, 'y'], [singleLine, 'y']]) {
assert.ok(config);
// BasicPlatform exercises real Chart.js scale/layout code without browser rasterization.
const canvas = { width: 700, height: 280, getContext: () => context };
const context = new Proxy({
canvas, measureText: text => ({ width: String(text).length * 7 }), getLineDash: () => []
}, { get: (target, key) => key in target ? target[key] : () => {} });
config.platform = BasicPlatform;
const chart = new Chart(canvas, config);
assert.ok(Number.isFinite(chart.scales[axis].min));
assert.ok(Number.isFinite(chart.scales[axis].max));
for (let index = 0; index < config.data.datasets.length; index++) {
const elements = chart.getDatasetMeta(index).data;
for (let point = 0; point < elements.length; point++) {
const element = elements[point];
if (config.type === 'line' && config.data.datasets[index].data[point] === null) {
assert.equal(element.skip, true);
} else {
assert.ok(Number.isFinite(element[axis]));
if (config.type === 'line') assert.ok(Number.isFinite(element.x));
}
}
}
if (config.type === 'line' && config === line) {
const segments = chart.getDatasetMeta(0).dataset.segments;
assert.ok(segments.every(segment => {
for (let point = segment.start; point <= segment.end; point++) {
if (config.data.datasets[0].data[point] === null) return false;
}
return true;
}));
}
chart.destroy();
}
}
console.log('rendered both bar orientations and lines at extreme coordinates');
`
],
{
cwd: fileURLToPath(new URL('../../../', import.meta.url)),
timeout: 3000,
killSignal: 'SIGKILL'
}
);
expect(stdout.trim()).toBe('rendered both bar orientations and lines at extreme coordinates');
}, 10_000);
apps/web/src/lib/server/accounts.test.ts → apps/web/src/lib/navigation/sidebar.server.test.ts
+2 -34
View File
@@ -1,7 +1,7 @@
import type { AccountConfig } from '@hledger-reports/config';
import { expect, it } from 'vitest';
import type { ReportPublication } from '@hledger-reports/snapshots';
import { projectAccountSidebar, registerDirectory, registerMembers } from './accounts.js';
import { expect, it } from 'vitest';
import { projectAccountSidebar } from './sidebar.server.js';
const publication: ReportPublication = {
journal: '/PRIVATE/journal',
@@ -166,35 +166,3 @@ it('keeps native undated values distinct from an unavailable publication without
]
});
});
it('selects only exact, explicit register members in configuration order', () => {
const accounts: AccountConfig[] = [
{ account: 'assets:bank', register: 'cash' },
{ account: 'assets:brokerage', register: 'investment' },
{ account: 'assets:bank:savings', label: 'Savings' },
{ account: ' Assets:bank ', register: 'cash' }
];
expect(registerMembers(accounts, 'cash')).toEqual(['assets:bank', ' Assets:bank ']);
expect(registerMembers(accounts, 'investment')).toEqual(['assets:brokerage']);
});
it('joins canonical names without inventing IDs or counts and ignores label changes', () => {
const accounts: AccountConfig[] = [
{ account: 'missing', register: 'cash' },
{ account: 'assets:bank', label: 'Checking', register: 'cash' },
{ account: 'ASSETS', register: 'cash' }
];
const catalog: Parameters<typeof registerDirectory>[1] = [
{ id: 'abcd1234', name: 'assets:bank', access: 'ready', rowCount: 0 },
{ id: 'ambig000', name: 'ASSETS', access: 'ambiguous', rowCount: null },
{ id: 'other000', name: 'missing:child', access: 'ready', rowCount: 3 }
];
const directory = registerDirectory(registerMembers(accounts, 'cash'), catalog);
expect(directory).toEqual([
{ id: null, name: 'missing', access: 'missing', rowCount: null },
{ id: 'abcd1234', name: 'assets:bank', access: 'ready', rowCount: 0 },
{ id: 'ambig000', name: 'ASSETS', access: 'ambiguous', rowCount: null }
]);
const relabeled = accounts.map((entry) => ({ ...entry, label: 'New label' }));
expect(registerDirectory(registerMembers(relabeled, 'cash'), catalog)).toEqual(directory);
});
apps/web/src/lib/server/accounts.ts → apps/web/src/lib/navigation/sidebar.server.ts
+4 -24
View File
@@ -1,14 +1,13 @@
import type { AccountConfig } from '@hledger-reports/config';
import type { StoredAccount } from '@hledger-reports/snapshots';
import type { RegisterAccount } from '#lib/accounts.js';
import type { AccountSidebarData, SidebarAccount } from '#lib/navigation/data.js';
import type { SavedReportsState } from './reports.js';
import type { SavedReportsState } from '#lib/server/reports.js';
import type { AccountSidebarData, SidebarAccount } from './data.js';
export function projectAccountSidebar(
configuredAccounts: readonly AccountConfig[],
reports: SavedReportsState
savedReports: SavedReportsState
): AccountSidebarData {
const { publication, readFailed } = reports;
const { publication, readFailed } = savedReports;
const groups: AccountSidebarData['groups'] = [];
for (const kind of ['cash', 'investment'] as const) {
const members = configuredAccounts.filter((entry) => entry.register === kind);
@@ -25,7 +24,6 @@ export function projectAccountSidebar(
});
groups.push({ kind, accounts });
}
// Use hledger's whole-journal NET, never a sum of potentially overlapping configured rows.
const latestNetWorth = publication?.overview.netWorth.points.at(-1);
const netWorth = latestNetWorth
? {
@@ -58,21 +56,3 @@ function projectSidebarValue(
unconverted
};
}
export function registerMembers(
accounts: readonly AccountConfig[],
kind: 'cash' | 'investment'
): string[] {
return accounts.filter((entry) => entry.register === kind).map((entry) => entry.account);
}
export function registerDirectory(
members: readonly string[],
catalog: readonly Pick<StoredAccount, 'id' | 'name' | 'access' | 'rowCount'>[]
): RegisterAccount[] {
return members.map((name) => {
const saved = catalog.find((entry) => entry.name === name);
if (!saved) return { id: null, name, access: 'missing', rowCount: null };
return { id: saved.id, name, access: saved.access, rowCount: saved.rowCount };
});
}
apps/web/src/lib/server/registers/accounts.test.ts
+35
View File
@@ -0,0 +1,35 @@
import type { AccountConfig } from '@hledger-reports/config';
import { expect, it } from 'vitest';
import { registerDirectory, registerMembers } from './accounts.js';
it('selects only exact, explicit register members in configuration order', () => {
const accounts: AccountConfig[] = [
{ account: 'assets:bank', register: 'cash' },
{ account: 'assets:brokerage', register: 'investment' },
{ account: 'assets:bank:savings', label: 'Savings' },
{ account: ' Assets:bank ', register: 'cash' }
];
expect(registerMembers(accounts, 'cash')).toEqual(['assets:bank', ' Assets:bank ']);
expect(registerMembers(accounts, 'investment')).toEqual(['assets:brokerage']);
});
it('joins canonical names without inventing IDs or counts and ignores label changes', () => {
const accounts: AccountConfig[] = [
{ account: 'missing', register: 'cash' },
{ account: 'assets:bank', label: 'Checking', register: 'cash' },
{ account: 'ASSETS', register: 'cash' }
];
const catalog: Parameters<typeof registerDirectory>[1] = [
{ id: 'abcd1234', name: 'assets:bank', access: 'ready', rowCount: 0 },
{ id: 'ambig000', name: 'ASSETS', access: 'ambiguous', rowCount: null },
{ id: 'other000', name: 'missing:child', access: 'ready', rowCount: 3 }
];
const directory = registerDirectory(registerMembers(accounts, 'cash'), catalog);
expect(directory).toEqual([
{ id: null, name: 'missing', access: 'missing', rowCount: null },
{ id: 'abcd1234', name: 'assets:bank', access: 'ready', rowCount: 0 },
{ id: 'ambig000', name: 'ASSETS', access: 'ambiguous', rowCount: null }
]);
const relabeled = accounts.map((entry) => ({ ...entry, label: 'New label' }));
expect(registerDirectory(registerMembers(relabeled, 'cash'), catalog)).toEqual(directory);
});
apps/web/src/lib/server/registers/accounts.ts
+21
View File
@@ -0,0 +1,21 @@
import type { AccountConfig } from '@hledger-reports/config';
import type { StoredAccount } from '@hledger-reports/snapshots';
import type { RegisterAccount } from '#lib/accounts.js';
export function registerMembers(
accounts: readonly AccountConfig[],
kind: 'cash' | 'investment'
): string[] {
return accounts.filter((entry) => entry.register === kind).map((entry) => entry.account);
}
export function registerDirectory(
members: readonly string[],
catalog: readonly Pick<StoredAccount, 'id' | 'name' | 'access' | 'rowCount'>[]
): RegisterAccount[] {
return members.map((name) => {
const saved = catalog.find((entry) => entry.name === name);
if (!saved) return { id: null, name, access: 'missing', rowCount: null };
return { id: saved.id, name, access: saved.access, rowCount: saved.rowCount };
});
}
apps/web/src/lib/server/reports.ts
-1
View File
@@ -5,7 +5,6 @@ import {
type ReportPublication
} from '@hledger-reports/snapshots';
// Validation is independent of financial publication and queue status.
export async function readSavedValidation(
snapshotDirectory: string,
journal: string
apps/web/src/routes/(protected)/+layout.server.ts
+3 -3
View File
@@ -1,5 +1,5 @@
import type { AccountLabel } from '#lib/accounts.js';
import { projectAccountSidebar } from '#lib/server/accounts.js';
import { projectAccountSidebar } from '#lib/navigation/sidebar.server.js';
import { readSavedReports } from '#lib/server/reports.js';
import type { LayoutServerLoad } from './$types';
@@ -11,10 +11,10 @@ export const load = (async (event) => {
const accountLabels: AccountLabel[] = configuredAccounts.flatMap(({ account, label }) =>
label === undefined ? [] : [{ account, label }]
);
const reports = await readSavedReports(locals);
const savedReports = await readSavedReports(locals);
return {
accountLabels,
pathname,
accountSidebar: projectAccountSidebar(configuredAccounts, reports)
accountSidebar: projectAccountSidebar(configuredAccounts, savedReports)
};
}) satisfies LayoutServerLoad;
apps/web/src/routes/(protected)/holdings/+page.server.ts
+1 -1
View File
@@ -1,7 +1,7 @@
import type { HoldingsReport } from '@hledger-reports/hledger';
import { SafeHoldingsJobError, validateHoldingsRequest } from '@hledger-reports/jobs';
import { error, fail, type ActionFailure } from '@sveltejs/kit';
import { registerMembers } from '#lib/server/accounts.js';
import { registerMembers } from '#lib/server/registers/accounts.js';
import { runHoldingsReport } from '#lib/server/holdings.js';
import { InvalidBody, readRequestText } from '#lib/server/http.js';
import { readSavedReports } from '#lib/server/reports.js';
apps/web/src/routes/(protected)/registers/cash/+page.server.ts
+1 -1
View File
@@ -1,4 +1,4 @@
import { registerMembers, registerDirectory } from '#lib/server/accounts.js';
import { registerMembers, registerDirectory } from '#lib/server/registers/accounts.js';
import type { CashAccountsData } from '#lib/registers/cash/data.js';
import { error } from '@sveltejs/kit';
import { readSavedReports } from '#lib/server/reports.js';
apps/web/src/routes/(protected)/registers/cash/[accountId]/+page.server.ts
+1 -1
View File
@@ -1,4 +1,4 @@
import { registerMembers } from '#lib/server/accounts.js';
import { registerMembers } from '#lib/server/registers/accounts.js';
import { readAccountTransactions } from '@hledger-reports/snapshots';
import { error } from '@sveltejs/kit';
import type { CashRegisterData } from '#lib/registers/cash/data.js';
apps/web/src/routes/(protected)/registers/investments/+page.server.ts
+1 -1
View File
@@ -1,4 +1,4 @@
import { registerMembers, registerDirectory } from '#lib/server/accounts.js';
import { registerMembers, registerDirectory } from '#lib/server/registers/accounts.js';
import type { InvestmentAccountsData } from '#lib/registers/investment/data.js';
import { error } from '@sveltejs/kit';
import { readSavedReports } from '#lib/server/reports.js';
apps/web/src/routes/(protected)/registers/investments/[accountId]/+page.server.ts
+1 -1
View File
@@ -1,4 +1,4 @@
import { registerMembers } from '#lib/server/accounts.js';
import { registerMembers } from '#lib/server/registers/accounts.js';
import { readAccountTransactions } from '@hledger-reports/snapshots';
import { error } from '@sveltejs/kit';
import type {
apps/worker/README.md
+6 -152
View File
@@ -1,154 +1,8 @@
# @hledger-reports/worker
# Worker
A separate Node process for native hledger work. It uses the thin [shared job definitions and queue connection](../../packages/jobs/README.md) from `@hledger-reports/jobs`; web never imports worker implementations or executes hledger.
Executes requested hledger jobs and background query-history cleanup. It shares the
web app's configuration and private queue, but runs as a separate process. Startup
does not request financial preparation.
The service entry is `src/index.ts` (`dist/src/index.js` in production), exporting `startWorker`. The independent read-only inspection command is `src/inspect.ts`.
## Run the worker
From the repository root:
```bash
pnpm worker:dev --config /absolute/deployment/config/hledger-reports.yaml
```
Development requires an explicit deployment configuration; there is no fixture default or configuration discovery. Alternatively set `CONFIG_PATH`. From this package directory:
```bash
CONFIG_PATH=/absolute/deployment/config/hledger-reports.yaml pnpm dev
# Compiled production entry (after a workspace build)
CONFIG_PATH=/absolute/deployment/config/hledger-reports.yaml pnpm start
```
The shared YAML includes `journal`, `url.app` and one auth strategy, with `version: 1` defaulted. Worker validates the full raw structure, including account-list shapes and duplicate identities, but resolves only journal/version, never web-only secrets. Account labels and register membership do not affect native preparation or report scope. Journal environment references are resolved before startup; see the [configuration contract](../../packages/config/README.md). The fixed deployment `data/jobs/` directory alongside `config/` holds the private SQLite queue, shared with web through the selected config path. Prepared reports are inspectable at `data/snapshots/current/overview.json` and `data/snapshots/current/transactions/<8-character-id>.json`. Both directories contain sensitive financial data. Compose bind-mounts the shared host folder `apps/web/data/` at `/app/data/` in web and worker, allowing direct inspection without putting data in an image or exposing it through HTTP. The previous named volume is not automatically moved or deleted; use the [copy guide](../../docs/development.md#inspecting-data-and-preserving-the-previous-volume) before switching existing data.
Startup loads configuration and verifies hledger **1.99.5**, opens the queue, registers workflows, starts openworkflow with **concurrency 1** and serves health. Startup requests no financial preparation. Financial results do not gate health: a missing or invalid journal fails safely without preventing operational health. The loopback health endpoint defaults to `http://127.0.0.1:9000/healthz`; set `WORKER_PORT` to change it. This is a process-health endpoint, not a journal-validity or query-success guarantee. No query HTTP endpoint runs in this process.
## Workflow layout
```text
src/workflows/
index.ts
index.test.ts
prepare-reports/
run.ts
run.test.ts
run.integration.test.ts
drain.integration.test.ts
edits.integration.test.ts
test-fixtures.ts
cost-balance/
run.ts
run.test.ts
query-retention/
run.ts
run.test.ts
run.integration.test.ts
```
The folder names the job; `run.ts` implements it. Registration stays explicit in
`workflows/index.ts`. Tests are next to the implementation, without repeating the
job name in every filename. Preparation's test fixtures are excluded from production
builds. The web requests these jobs; this process performs them.
## Explicit report preparation
The worker executes requested jobs only. Startup, idle time, edits and restart do
not request financial work. There are no source hashes, filesystem watches,
staleness checks or automatic follow-ups. Scheduled query retention is independent
operational maintenance. Web owns requesting/joining work through Settings; see
[web behavior](../web/README.md#settings-journal-summary).
`src/workflows/index.ts` explicitly registers all five implementations.
`prepare-reports/run.ts` implements `prepareReportsSpec`: persisted SDK name
`hledger.prepare-dashboard`, version **1**, **void input**, **OverviewData output**,
**maximumAttempts 1**, no schedule. Concurrent explicit requests join active work;
a later request starts a new run even with unchanged inputs. Failure remains failed
until another explicit request. The workflow identity/retry policy is unchanged.
The handler awaits standalone `checkJournal(journal)`, records its completed outcome
and continues only after a recorded pass. Incomplete checking preserves earlier
evidence; check/evidence-write failure stops before financial staging. A recorded
pass is not proof of unchanged inputs or completed finances.
Afterward it reads the prior publication, tolerating missing, corrupt or unsupported
data so an explicit job can replace it. Only supported same-journal data supplies
previous account IDs. It creates a private stage and awaits
`prepareReports(journal, { writeRows: stage.writeRows })`, then publishes Overview,
account summaries and staged rows together. The handler returns **its own calculated
OverviewData**, not a readback of whichever publication is selected, and awaits
stage discard in `finally`.
There are no replay-cached SDK financial checkpoints. Older queued runs with legacy
steps can finish under this handler without changing those steps. SQLite persists
the handler's Overview result, not duplicate account-history arrays or private source
positions. Native diagnostics and source paths never enter job output.
- [Architecture](../../docs/architecture.md#prepared-reports) owns cross-layer phases.
- [Native library](../../packages/hledger/README.md#overview-planning-and-interpretation)
owns commands, version checking, dates/counts and execution cost.
- [Snapshots](../../packages/snapshots/README.md#publication-and-reads) owns commit,
validation records, read selection and preservation. Post-commit alias failure can
fail the job while leaving a new valid set selectable; there is no rollback.
- [Format-8 activation](../../docs/development.md#financial-format-8-activation) owns
the manual paired web/worker/shared-package upgrade.
## Register preparation resources
Native rows are written sequentially with backpressure into whole-account files;
browser pagination does not reduce native preparation cost or whole-file read cost.
The producer has no application-imposed native deadline, output/row/tuple cap or
saved-file size cap. Native/Node memory can still be exhausted; there is no universal
journal-capacity guarantee. See the [native resource contract](../../packages/hledger/README.md#one-hledger-process-boundary).
Cancellation kills/reaps native work and waits for active writes before cleanup.
The handler discards only its own unfinished stage; completed sets may already be
selected and are retained. Prepared-data retention is separate from SQLite query
retention below. Health remains independent of report calculation.
## Current-journal queries
`liveBalanceQuerySpec` is `hledger.balance-query`, version **2**. `src/workflows/cost-balance/run.ts` owns its handler. Each explicit Run validates `{ query, asOf }` and calls the [native cost-balance adapter](../../packages/hledger/README.md) against the configured journal. That library owns native command details, strict validation, exact quantities and execution safeguards. The workflow returns `{ query, asOf, rows, total }` directly; SQLite persists that financial output, with no memoized step or separate result file.
Journal edits appear on the next fresh Run **without restarting the worker**. A bad journal fails safely; correcting it allows the next Run to succeed. These explicit queries always read current input and do not request saved-report preparation. Restart both processes after changing the YAML configuration itself.
The SDK handles queue execution and persisted state. The workflow uses one attempt; a failure does not retry indefinitely. The web-owned requester awaits the result with a **35-second** timeout. Closing the page does not cancel queued or running work. No cancellation UI, cross-query reuse, or general report catalog is provided. Scheduled query retention is described below.
Graceful SIGINT/SIGTERM stops SDK polling, waits for active financial work, and closes health and SQLite connections. Shutdown calls are idempotent.
## Query job retention
OpenWorkflow runs `hledger.query-retention` **daily at 00:MM UTC**. The worker samples MM from **1–59 once per startup**, rather than synchronizing every instance at midnight; that minute stays fixed until restart. The schedule identity stays stable and an administrative SDK pause survives restart. There is no separate startup purge or application timer.
Changing the startup minute reschedules the next future occurrence. Unlike an unchanged SDK cron, this cannot guarantee overdue catch-up after restart; repeated restarts can postpone cleanup.
A failed maintenance run does not retry indefinitely or disable query processing. `src/workflows/query-retention/run.ts` owns collection. Each execution deletes at most **100** standalone `hledger.balance-query` rows (versions **1 and 2**) in the **default namespace**, ordered by `finished_at` then ID. Only `completed`, `succeeded`, `failed`, or `canceled` rows with a non-null `finished_at` **strictly earlier than execution time minus seven days** qualify. Creation time and the query's input date do not determine retention. Seven days is an eligibility threshold, not a maximum storage age: backlogs wait for later daily runs. The row cap does not guarantee scan/runtime bounds.
Active/unknown statuses, other workflows/versions/namespaces, idempotency-keyed or scheduled runs, parent/child relationships, step attempts and signals are excluded. Neither graph records nor legacy files are collected. The implementation opens only an existing queue file, enables foreign keys and a one-second busy timeout, closes its connection, and exposes a fixed safe error on failure. One DELETE selects and removes the batch atomically, trusting SDK ISO timestamps. Maintenance history itself is not purged. There is no startup purge, migration, VACUUM, checkpointing, physical shrinkage, or secure-erasure guarantee.
This is temporary private SDK-table coupling ([upstream issue](https://github.com/openworkflowdev/openworkflow/issues/303)), not a general SDK retention API. Stop web and worker and take a consistent private jobs-directory backup before an SDK upgrade; review this SQL against schema changes and restore the backup for rollback rather than downgrading a migrated database. Take a consistent private-data backup before deploying retention if old job data must be kept. SQLite/WAL and backups may retain deleted financial data; deletion does not remove backups. Journals, prices, snapshots and legacy input/result files remain untouched. [Prepared-data lifetime](../../packages/snapshots/README.md) is separate.
## Legacy captured-input queries
The version-1 `retiredBalanceQuerySpec` (`hledger.balance-query`) is an opaque, schema-less contract: its input is `unknown` and its output is `never`. It remains registered only to reject pending captured-input runs once with: “Captured-input queries are no longer supported. Run a new query.” It does not validate or interpret their legacy payloads, read captured-input/result files, or run them against the current journal. This explicit handler is necessary because the SDK claims runs without filtering registered versions and would otherwise reschedule missing definitions indefinitely.
Existing input generations, pointers, and result files are left untouched; scheduled SQLite retention does not migrate or purge those files. New version-2 runs create no capture or result-file directories. SQLite itself contains query inputs and financial output: treat the entire `data/jobs/` directory, including legacy files, as sensitive. Do not log native errors, query bodies, or financial values. Exposed report errors are generic; use hledger directly for detailed diagnostics.
## Retained preflight command
```bash
# Repository root: inspect the small code fixture in an owned temporary deployment
pnpm worker:inspect
# This package directory: inspect your own explicit configuration
pnpm inspect --config /absolute/deployment/config/hledger-reports.yaml
# Configuration only: no native checks, journal reads or storage creation
pnpm config:validate --config /absolute/deployment/config/hledger-reports.yaml
```
Root `worker:inspect` runs the single-purpose `fixtures/inspect.js` launcher with Node 26. It materializes the `lotsJournal` code literal and a minimal JSON config under an owned OS temporary directory, waits for `src/inspect.ts`, and cleans up in `finally`. No journal files or fixture YAML are tracked. The launcher does not start a worker or web service or forward inspection options; use the package commands above for your deployment.
`--config-only` validates the full resolved web configuration, including authentication secrets, using the current environment and prints only success metadata or a safe failure code. Explicit `--config` takes precedence over `CONFIG_PATH`; neither mode discovers personal configuration.
Default preflight uses worker-only configuration resolution (no web secret requirement), verifies hledger **1.99.5** and runs read-only `check --strict` on the configured journal. It does not create query jobs, prepare snapshots or write validation evidence. It remains read-only, separate from worker startup, preparation preflight recording and query execution.
Tests exercise real hledger/SQLite workflows with temporary files materialized from small fictional code fixtures: whole-set preparation and recovery, live edits, current includes, read-only source handling, safe failures and correction, durable query output without query capture/result files, legacy rejection without file removal, and startup/shutdown ordering. They require the pinned native engine on PATH. The shared code literal and inspection launcher are under this app's [fixtures](fixtures/README.md).
See [development and operations](../../docs/development.md) for startup, inspection
and data-preservation guidance.
apps/worker/fixtures/README.md
+3 -24
View File
@@ -1,25 +1,4 @@
# Code fixtures
# Fictional fixtures
These small, fictional, hand-checkable inputs are owned by the worker application. No journal files are tracked, private or fictional. Tests and the inspection command materialize the code literal only in their own temporary directories and remove those directories afterward.
## `journal.ts`
The named `lotsJournal` export is the exact **61-line nine-transaction native FIFO lot fixture** for pinned hledger **1.99.5**. Native and worker integration tests share it. It covers two investment accounts, VTI/BND securities, brokerage cash, purchases at different prices, a partial sale, a lot-preserving transfer, salary, expenses, and dated prices. hledger infers the realized-gain posting without writing to the source.
At April 30, 2026, expected securities value is **3,290 USD**, basis **2,430 USD**, unrealized gain **860 USD**, and realized gain **250 USD**. Checking holds **8,520 USD**, brokerage cash **200 USD**, and net worth is **12,010 USD**. Native holdings returns three account/security summaries or five lot rows. This is correctness coverage, not a capacity benchmark.
## `inspect.js`
With the workspace already installed, Node **26** and hledger **1.99.5** on `PATH`, run from the repository root:
```bash
pnpm worker:inspect
# Equivalent direct invocation
node apps/worker/fixtures/inspect.js
```
This single-operation launcher imports `lotsJournal` using native TypeScript support. It creates an owned deployment directory under `os.tmpdir()`, writes the journal and a minimal JSON configuration in `config/`, and runs the existing `src/inspect.ts` command in a child Node process with the development import hook. It waits for that command, preserves its exit code, and removes its temporary deployment in `finally`.
The configuration supplies version 1, a relative journal path, an app URL and a fictional credentials strategy with a password environment reference. Account settings use their empty-list default. Worker inspection validates the shared raw configuration but does not resolve web-only secrets, so `AUTH_PASSWORD` and `AUTH_SECRET` are not required. This is not a web configuration launcher or a persistent worker deployment.
Inspection verifies the native version and performs a read-only strict journal check. It does not create jobs, prepare snapshots, or write validation evidence. There is no static fixture YAML or repository-local fixture data directory. For normal worker development, select your own explicit deployment configuration as described in the [worker documentation](../README.md).
Small code fixtures for tests and `pnpm worker:inspect`. Use only owned disposable
test directories. Never commit journal files or use operator data.
docs/architecture.md
+16 -209
View File
@@ -1,218 +1,25 @@
# Architecture
The product has separate read, request and produce paths:
The web reads saved reports. Explicit requests go through a private job queue to
the worker, which runs hledger and saves the results.
```text
Protected shell / saved pages → share saved financial set → render
Settings → read saved validation outcome/time → render
Settings form → request durable job → check + record → prepare complete set
Live query Run → request durable job → hledger → return exact result
Prepare request → journal check → hledger reports → complete saved set
Browse request → one selected saved set → public page data
Live query → worker → hledger → requested result
```
Web and worker share configuration and private storage. OpenWorkflow/SQLite
coordinates jobs; storage is a library, not another HTTP service.
Hledger owns accounting. The adapter preserves exact values; storage publishes
complete sets; the web exposes only the fields each page needs. Journal paths,
source locations and other private metadata stay on the server.
## Ownership
A request keeps one saved set so navigation and page data agree. Recorded journal
checks are separate from financial publication. Neither saved results nor a passed
check proves that the journal has not changed.
| Owner | Responsibility |
| -------------------- | ------------------------------------------------------------------ |
| `packages/hledger` | Native execution, exact decoding and report-ready financial data |
| `apps/worker` | Requested job execution, phase orchestration and service resources |
| `packages/snapshots` | Complete publication, pinned reads, IDs and validation records |
| `packages/jobs` | Job definitions, input contracts and queue connection lifetime |
| `apps/web` | Request validation, public projection and presentation |
| `packages/config` | Explicit configuration and fixed deployment paths |
Web and worker share configuration and private storage. SQLite coordinates jobs;
financial browsing does not depend on a running worker. Startup does not request
preparation, though previously requested jobs can resume.
Hledger owns accounting. Web never opens journals, launches hledger, interprets its
wire JSON or reconstructs balances. Worker coordinates, storage persists, and web
presents. The native package owns the sole process boundary, including cancellation
and backpressure; see its [execution contract](../packages/hledger/README.md#one-hledger-process-boundary).
The narrowly permitted browser arithmetic is category-chart grouping only, not a
shared money utility or permission to calculate financial totals.
## Prepared reports
The worker's preparation handler has visible ordered phases:
1. Await a standalone strict check and record its completed outcome. Continue only
after a recorded pass; incomplete checks preserve prior evidence.
2. Read the prior publication for supported same-journal account-ID reuse and create
a private stage.
3. Await native production and staged row writes. The producer verifies the engine
version and uses a cached financial journal view for Overview and full account
history; native planning/interpretation belongs to the
[hledger library](../packages/hledger/README.md#complete-prepared-reports).
4. Publish the complete result and return this execution's calculated Overview,
not a readback of another selected publication. Await stage discard in `finally`.
The worker does not checkpoint financial phases as replay-cached SDK steps. Queue
identity/retry behavior remains owned by the [worker](../apps/worker/README.md#explicit-report-preparation).
Storage commits the complete immutable directory, containing Overview, account
catalog and one whole transaction file per account. `current` is an inspection
alias, not the application read-selection mechanism. Before commit, failure keeps
the previous complete set; an alias failure after commit can fail the job while
leaving the new valid set selectable. Do not interpret job failure as proof that
reports are unchanged. Format validation, ordering, account-ID reuse and preservation
have one owner: the [snapshot store](../packages/snapshots/README.md#publication-and-reads).
Deployment sequencing belongs to [format-8 activation](development.md#financial-format-8-activation).
Financial timestamps describe saved preparation, not current-input freshness.
Exact private journal matching withholds other-journal data; it is not an input
fingerprint or atomic multi-file snapshot guarantee. Browsing never triggers
migration, preparation or cleanup.
## Homepage periods and presentation
Routes own HTTP semantics and explicit public whitelists. Protected saved-data
leaves await their parent and share the shell's pinned publication. Availability
and read-failure flags describe state without exposing diagnostics. Financial
responses are no-store.
| Consumer | Public financial data |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Overview | `asOf`, four ranges with counts/income/expenses/categories, monthly income history and only the latest native NET point with currency/amounts/unconverted evidence |
| `/reports/networth` | `asOf`, native currency and twelve NET points with dates, amounts and conversion evidence |
| Settings `JournalSummaryData` | Preparation time, coverage, saved hledger version and report currency from `netWorth.currency` |
| Accounts navigation | Configured identity/label/access, raw or valued bags, latest whole-journal NET and cutoff/read context |
| Register directories | Configured identity/access/count fields |
| Register details | Selected page's public transaction/detail fields |
Overview does not send unused range NET totals, assets/liabilities sides or full
net-worth history. Native/stored NET totals and sides remain intact. Neither
Overview nor the net-worth leaf reads validation or sends preparation/version/coverage
metadata. Settings reads validation independently and passes only outcome/time and
read state separately from `JournalSummaryData`. No consumer receives the private
publication directory, journal/request identity or source/assertion positions.
Avoid object spreads of stored records into PageData.
`lib/report-period/period-query.ts` owns the literal 1/3/6/12-month URL policy and
default 3. Both period-aware loaders translate invalid input to HTTP 400
**before parent/read access**. Co-located `PeriodControls.svelte` owns the shared
control: SvelteKit shallow selection
uses loaded data, while the native GET fallback uses server validation. Ordinary
navigation/reload may select a new publication; shallow selection cannot. History
may restore an earlier coherent view, with no latest-on-resume guarantee.
`overview/select-period.ts` selects a saved range and trailing income history. Net worth
slices its own twelve points. These are distinct operations, not a generic report
controller. `/reports` is a namespace for the concrete `/reports/networth` leaf,
not a catalog or registry. See [web usage](../apps/web/README.md#overview-periods) for
interactions, chart/table layout and no-JavaScript behavior.
### Presentation ownership
- `overview/KeyFigures.svelte` composes four headlines. Overview-local
`FinancialValue.svelte` renders exact bags and valuation states without forcing
Accounts' differently worded states through the same component.
- `overview/income-spending/IncomeSpending.svelte`,
`overview/spending-categories/SpendingCategories.svelte` and
`reports/networth/NetWorth.svelte` own their presentation and exact tables, each
with `chart.ts` and tests. Their builders are `incomeSpendingChart`,
`spendingCategoriesChart` and `netWorthChart`.
- `report-card/ReportCard.svelte` owns heading/chart/details geometry, using
`report-card` and `--report-card-height`. Its private `ScrollArea.svelte` keeps
Bits UI enhancement and native overflow fallback together.
- `settings/JournalSummary.svelte` composes Settings' six facts;
`ValidationStatus.svelte` presents the independent check, with types in `data.ts`.
Preparation execution status does not duplicate its timestamps.
- `overview/spending-categories/presentation.ts` exports
`spendingCategoriesPresentation` and `SpendingCategoriesPresentation`: exact
ranking, Other and guards in one model for canvas and ordered text. Its `chart.ts`
owns only Chart.js configuration. Native bags and the complete raw table stay
untouched; the feature puts Chart values **after** that table, and `ChartCanvas`
associates the canvas with the list via `aria-describedby`.
- `charts/` owns only shared `ChartCanvas.svelte`, theme code/fixture and rendering
integration tests. Other chart builders retain native-owned values and approximate
geometry only. No server rollup, stored chart field or generic arithmetic layer.
## Shared saved reads and account navigation
`lib/server/reports.ts` assigns a request-local `Promise<SavedReportsState>` before
filesystem work. Layout, saved-data leaves and account-selecting actions share that
selection, including missing/error results. Register details read one account file
from the pinned directory and paginate in memory, not every account history.
There is no global cache or public set pin.
The protected layout tracks the full destination URL, including query changes;
leaves await `parent()`. Ordinary changed-path/query navigation refreshes shell and
leaf together. Request-local sharing alone would not refresh a retained browser
layout. No custom router, automatic prefetch, polling or input inspection is added.
Settings alone reads queue status and independent validation.
The protected layout owns header/main composition and rail placement.
`navigation/AppHeader.svelte` owns brand/navigation/actions, not report or auth
controllers. `AccountSidebar.svelte` renders the same content in rail, drawer and
native disclosure. `navigation/data.ts` owns its narrow public types;
`lib/server/accounts.ts` explicitly projects them from configured accounts and the
private publication. Native whole-journal NET is not a configured-account sum:
parents can overlap and configured membership is only a subset.
`AccountsDrawer.svelte` owns hydration fallback, media-query subscription/cleanup,
responsive transitions, navigation closure and focus-restoration policy. Bits UI
owns portal, focus trap, scroll lock and dismissal mechanics. Dismissal restores the
trigger; navigation permits destination focus. A same-URL drawer link still refreshes
saved reports; its matching Kit `afterNavigate` callback focuses main after the
router's focus reset. Switching to the rail closes the modal. The Net worth footer uses the existing navigation callback, including with
no configured accounts or readable balances. User-visible responsive behavior is
owned by the [web README](../apps/web/README.md#accounts-navigation).
## Last recorded validation
The native checker owns completed versus incomplete execution. The worker records
Passed only after complete exit 0; normal nonzero records Could not validate and
stops financial work. Nonzero may mean missing/unreadable input, not invalid
accounting. Incomplete execution preserves prior evidence; evidence-write failure
stops preparation before financial staging. See [native checks](../packages/hledger/README.md)
for execution semantics and [recorded validation](../packages/snapshots/README.md#recorded-validation)
for immutable storage, selection and retention.
Only Settings reads evidence, independently of financial availability and queue
state. Missing/unsupported/other-journal evidence is unknown; unreadable evidence
is unavailable and does not hide readable finances. Only outcome/time is public.
The check and financial batch can see different edits; financial production remains
independently strict. A saved pass survives later financial failure but proves
neither completed publication nor freshness. Read-only inspection never records
validation evidence.
## Explicit execution
Web's `lib/server/preparation.ts` owns submitting/joining preparation and reading
status. Status reads never submit. `lib/server/queries.ts` validates and submits a
fresh live query, then awaits its result with a bounded HTTP wait. Neither helper
owns the shared queue connection. Closing HTTP does not cancel durable work.
`packages/jobs` defines job/input contracts and opens/closes the SDK connection;
it does not own submission policy or execution. Worker registration remains explicit
in `workflows/index.ts`, with each concrete job's `run.ts` and tests together.
Scheduled query retention is operational maintenance, not report preparation.
Already requested jobs may resume after restart; startup creates no new request.
Origin/body guards and authentication remain unchanged and are owned by the
[authentication guide](authentication.md). Configuration is explicit at startup;
web resolves auth secrets, worker does not. No public report API, source watcher,
input hash, freshness loop or automatic browser/native work is added.
## Code organization and boundaries
The [web file map](../apps/web/README.md#run-and-check) and
[native source layout](../packages/hledger/README.md#source-layout) locate features.
Shared server helpers stay in `lib/server`, enforcing SvelteKit's server-only import
boundary. Native wire interpretation stays in the producer, not route loaders.
`lib/amounts.ts` owns display-only formatting without rounding meaningful precision,
combining bags or converting quantities to `Number`. `lib/dates.ts` owns deterministic
calendar dates and explicit-UTC timestamps; markup retains ISO `datetime` values.
CSS owns chart colors/fonts, resolved by `ChartCanvas` after mount/theme changes;
server-rendered exact tables do not depend on browser styling APIs.
Shared CSS imports once; feature styles own financial alignment and table structure.
`DetailDialog.svelte` owns only the common interactive shell, while registers own
payload-bound selection and financial contents. Keep resource cleanup visible and
avoid generic table schemas or controllers without consumers.
Tests stay beside their owner: native interpretation, storage publication, web
projection/period parsing and a small real browser journey suite. See
[philosophy](philosophy.md) for scope and [development](development.md#focused-local-checks)
for commands. Persisted format/workflow changes require explicit compatibility
decisions; internal names do not require aliases.
See [development](development.md) for commands and [principles](philosophy.md) for
how we change the code.
docs/authentication.md
+23 -129
View File
@@ -1,139 +1,33 @@
# Shared access
# Authentication
One deployment serves one journal. Every signed-in user can browse saved reports,
prepare reports and run live queries or Holdings. There are no roles, managed users,
signup, invitations, personal journals or application email allowlists.
One deployment serves one journal. Sign-in is required, and every admitted user
has the same access to reports and explicit financial jobs.
Local development remains loopback-bound. A compiled web instance can also sit
behind a trusted HTTPS proxy; authentication and canonical Host/Origin checks still
apply. This is a request-compatibility contract, not a production deployment recipe.
## Setup
## Trusted-proxy contract
Use the [configuration example](../apps/web/config/hledger-reports.example.yaml).
Choose exactly one strategy: credentials or OIDC.
The application listener must be reachable only through the trusted proxy. Compiled
Compose sets adapter-node's runtime `PROTOCOL_HEADER=x-forwarded-proto` and
`ADDRESS_HEADER=x-real-ip`. The proxy must overwrite both headers with the actual
public scheme and client address, preserve the public `Host`, and be the only route
to the upstream. Do not trust browser-supplied values or append-only header chains.
- Set `url.app` to the exact browser origin. `localhost` and `127.0.0.1` are different origins.
- Supply a strong password and a stable random `AUTH_SECRET` of at least 32 characters through private environment or secret tooling.
- Keep secrets out of commits, command arguments and logs. Restart services after configuration changes.
The framework constructs request URLs from scheme and Host. Both `event.url.origin`
and the origin of `request.url` must equal normalized YAML `url.app`, the authoritative
public origin. There is no custom raw Host parser. Forwarded host is ignored by
default; only an explicit native `HOST_HEADER` setting changes that policy, and its
chosen header must also be overwritten by the trusted proxy.
For OIDC, set the issuer, client ID and client secret. Register
`<url.app>/auth/callback/oidc` with the provider and restrict admission there; the
app has no separate user allowlist. Sign-out clears the app session, not the
provider session.
There is no per-build public URL (`paths.origin` or Docker `ARG ORIGIN`) and no runtime
`ORIGIN` override. Adapter-node 6's `ORIGIN` is not a replacement for protocol headers;
without a trusted protocol it assumes HTTPS, so its compiled entry point is not a
direct HTTP browser preview.
## Trusted proxy
Authentication copies `event.getClientAddress()` into its own private client-address
header, replacing any incoming value. The framework owns proxy/IP policy; the app
does not parse forwarding chains. Remote clients still need a session, and unsafe
methods require an exact `Origin` matching `url.app`. Financial responses remain
no-store.
Compiled web requires a trusted HTTPS proxy. Only that proxy may reach the
upstream. Preserve the public `Host` and overwrite `X-Forwarded-Proto` and
`X-Real-IP` with trusted values, never browser-supplied headers.
Local Vite/mise development and listener bindings are unchanged; Vite derives its
own request origin. The owned compiled browser fixtures instead use a test-only
loopback HTTP boundary, not a deployment option or authentication bypass. Broader
production work remains deferred.
Set `APP_URL` to the public HTTPS origin when using the Compose example. Use Vite
development for direct local HTTP access. Do not expose the upstream publicly.
## Configuration
## Sessions
Preserve your journal selection and data directory. Add `url` and exactly one auth
strategy to the existing private `config/hledger-reports.yaml`; do not replace it
with an example. Journal-only configuration is no longer sufficient for web startup.
```yaml
version: 1
journal: /journal/main.journal
url:
app: http://127.0.0.1:5173
auth:
secret: '${AUTH_SECRET}'
credentials:
username: household
password: '${AUTH_PASSWORD}'
```
Supply a strong password and a stable random secret of at least 32 characters
through your environment/secret tooling. Generate a secret explicitly, for example
with `openssl rand -base64 32`; the application never generates a fallback. Keep
secret output out of logs, commits and shell command arguments. A literal YAML
secret is supported, but the YAML file must remain private. Omitting `auth.secret`
means `${AUTH_SECRET}`, not unauthenticated access.
`url.app` is an absolute origin, without credentials, path, query or fragment. It
must match the browser address and framework request URLs. The example above is
for local Vite development; compiled proxy setups use their public HTTPS origin.
The tracked Compose example uses `${APP_URL:-http://127.0.0.1:3000}`; development
Compose defaults `APP_URL` to `http://127.0.0.1:5173` but honors an explicit override
such as `http://localhost:5173`. Recreate containers after changing that environment
value and browse using the exact configured origin. The compiled proxy setup must
set `APP_URL` to its public origin; its loopback HTTP fallback is for fixtures, not
a direct compiled browser preview. Compose passes `AUTH_SECRET`,
`AUTH_PASSWORD` and `OIDC_CLIENT_SECRET` to web, not worker. Other environment
references require explicit deployment environment wiring. Restart both services
when changing the shared file; no financial reports are prepared on restart or login.
Web resolves and validates all fields before opening its jobs client. Worker
validates the same complete raw structure but resolves only the journal and version;
it does not require web-only environment secrets. `pnpm config:validate --config
<path>` deliberately validates the complete web configuration using the current
environment. Native inspection still uses worker-only resolution.
## OIDC instead of credentials
Replace `credentials` with:
```yaml
oidc:
issuer: https://identity.example.com/tenant
client_id: hledger-reports
client_secret: '${OIDC_CLIENT_SECRET}'
```
Register the redirect URI **`<url.app>/auth/callback/oidc`** with the provider and
configure its application assignment/admission policy. The application trusts that
policy, not an additional local allowlist. It requests `openid profile email` and
uses discovery, authorization code exchange with PKCE, nonce/state verification,
verified ID tokens and user information. Issuers require HTTPS except for explicit
loopback development providers. Provider compatibility is exercised with an owned
test provider, not a real identity service or its admission settings.
Sign-out is local: it clears this application's cookies, not the provider's session.
A provider session may therefore allow immediate sign-in again.
## Session policy
Better Auth owns encrypted, HttpOnly, SameSite=Lax session cookies. HTTPS uses secure
cookies. Sessions expire exactly seven days after sign-in, without sliding renewal,
and survive process restart with the same secret. No authentication database is
created; the financial queue remains a separate operational store.
Changing a password or removing a provider assignment does not immediately revoke
an existing cookie. Sign-out removes the browser's cookies but cannot revoke copies
of a stateless cookie. Expiry or rotating the configured auth secret ends those
sessions; rotation signs everyone out after web restarts.
SvelteKit route groups declare the access boundary: `(public)` holds public pages
and resources; `(protected)` holds financial pages and APIs. A hook enforces that
policy for page-data loads, actions and endpoints as well as HTML pages. Unclassified
application routes also require a session. SvelteKit owns framework assets and 404s.
Sign-in uses a normal SvelteKit action with `use:enhance`, so it works both with and
without JavaScript. Sign-out is a native POST. Only the OIDC callback is exposed as
a Better Auth HTTP route; sign-in/out dispatch through its SDK from server actions
and use the SDK cookie utilities. This retains its HTTP CSRF and rate-limit checks
without exposing unused SDK endpoints. Auth bodies are bounded and errors/logs omit
credentials, provider diagnostics, codes and tokens. Financial responses and session-bearing
pages are no-store; the public schema retains its separate cache policy.
## Browser tests
Owned browser deployments create fictional credentials and sign in through the
normal endpoint. Their private temporary cookie state is removed afterward; there
is no production auth bypass. External fixture tests require explicit
`PLAYWRIGHT_USERNAME` and `PLAYWRIGHT_PASSWORD`, matching that fixture's configured
credentials. Never point mutation-capable financial browser tests at personal data.
Sessions survive restart with the same secret. Changing a password or provider
assignment does not immediately revoke existing sessions. Rotate the auth secret
and restart web to sign everyone out. Local sign-out cannot revoke copied cookies.
docs/development.md
+38 -434
View File
@@ -1,466 +1,70 @@
# Local containers and CI
# Development and operations
Run commands from the repository root unless noted. Docker Compose is sufficient;
Node, pnpm and hledger do not need to be installed on the host.
## Containers
## Live development
Create the ignored live config once, from `apps/web` (this refuses to overwrite an existing file):
Follow the [quickstart](../README.md#run-locally). The containers target Linux/amd64
and require Docker host networking. From the repository root:
```sh
(umask 077; set -C; cat config/hledger-reports.example.yaml > config/hledger-reports.yaml)
```
Keep `main.journal` and its includes under `apps/web/journals/`, or set `JOURNAL_DIR` to another host directory. The example selects `/journal/main.journal`, the worker's read-only container path. For host-native runs, change `journal` to a host-visible path; relative paths resolve from the YAML file. Do not use the example to replace an existing operator configuration.
Before starting, supply `AUTH_SECRET` and the configured credentials/OIDC secret
through your secret tooling. Existing live YAML must add `url.app` and exactly
one auth strategy without changing its journal selection. See
[authentication setup](authentication.md). The example uses `APP_URL`; development
Compose defaults it to `http://127.0.0.1:5173` and honors an explicit `APP_URL`
override, including one supplied by mise. Recreate containers after changing it;
`localhost` and `127.0.0.1` are distinct origins. The compiled upstream listens on port `3000`.
Do not use either authenticated configuration to open a public listener.
Start development:
```sh
cd apps/web
mkdir -m 700 -p data &&
docker compose -f docker-compose.dev.yml up --build --wait
```
This uses the checked-in Compose files and your ignored live configuration.
A mise/fnox Docker task is an owner-local convenience, not part of the checked-in
project. The remaining commands below run from the repository root.
Open **http://127.0.0.1:5173/** and sign in. Use **Prepare reports** in
**Settings** (`/settings`), reload for status, then navigate to saved results.
Startup and browsing never prepare reports. For interactions and routes, see
[web usage](../apps/web/README.md#pages). Unsupported saved data requires the
[activation steps](#financial-format-8-activation), not deletion or manual file edits.
The automated tests use owned fictional fixtures, never operator journals. Native
fixture semantics belong to the [library](../packages/hledger/README.md#overview-data);
external browser deployments must use the combined fixture described under
[focused checks](#focused-local-checks).
App/package source directories are mounted read-only. Vite updates the UI and
Node's watcher restarts the worker on imported source changes. Dependency,
Dockerfile, or non-mounted configuration changes require another `up --build`.
After journal/include/price edits, request **Prepare reports** in Settings again when you want a
new saved set. There is no file watcher, input hashing, freshness comparison or automatic
follow-up job. Native includes are resolved when the requested calculation runs.
Explicit query Runs still read current input directly. Configuration changes require
restarting the processes. The default journal **directory** mount exposes atomic
editor replacements and glob additions without container recreation.
This runs the current working tree, including uncommitted source changes, rather
than an older staged revision.
```sh
# Follow both processes.
docker compose -f apps/web/docker-compose.dev.yml logs --follow
# Stop; keep prepared data and queued/completed jobs.
docker compose -f apps/web/docker-compose.dev.yml down
```
Both services use `/app/config/hledger-reports.yaml` and bind-mount the same host folder,
**`apps/web/data/`**, at `/app/data/`. Compose reads ignored `apps/web/config/hledger-reports.yaml`; a missing file fails instead of creating a host directory. The worker alone mounts `${JOURNAL_DIR:-./journals}` read-only at `/journal`.
YAML includes `version` (default `1`), `journal`, `url.app` and one auth strategy. Both processes need the same journal environment values. Only web resolves authentication secrets; worker validates their raw shape without reading them. `pnpm worker:inspect` creates its fictional journal and config in an owned temporary directory and removes them afterward; worker development requires your explicit configuration.
Web/worker processes run as the image's unprivileged `node` user (UID 1000).
Use `down` without `--volumes` to preserve data. Restart after configuration changes;
rebuild after dependency or Dockerfile changes.
The journal mount is explicit, not a personal-journal discovery mechanism.
Keep live YAML, journals and local Compose overrides untracked and outside images.
The journal path in that YAML must resolve to the journal's **container path**.
For native includes and editor rename-over saves, mount the required journal tree as a
read-only **directory**, not individual files. A single-file bind mount can keep the
old inode after a host editor replacement; a calculation cannot read a replacement
that the container cannot see.
Treat the entire host data folder as sensitive: SQLite contains query parameters,
full financial results and preparation history, while the inspectable report set lives in
`data/snapshots/current/`. See [prepared reports](../packages/snapshots/README.md#current-format-and-preservation)
for publication, recovery, growth and untouched unsupported files.
Compiled images use `apps/web/docker-compose.yml` instead. Stop development before
switching, configure the public URL, and use a [trusted HTTPS proxy](authentication.md#trusted-proxy).
For direct local HTTP browsing, use development mode.
Storage layout, format validation, account-ID reuse and preservation are owned by
[snapshots](../packages/snapshots/README.md#publication-and-reads). Do not remove
active account files individually. There is no application-imposed native deadline
or file-size cap; larger journals can still exhaust resources. See the
[native execution contract](../packages/hledger/README.md#one-hledger-process-boundary).
## Host development
**SQLite query retention is enabled**. See [retention scope, scheduling and backup precautions](../apps/worker/README.md#query-job-retention);
it is not prepared-file cleanup.
## Financial format-8 activation
The current financial format is **8**, with required checked hledger version and
all four native period transaction counts alongside existing coverage/financial
data. PR3 used format **6**; format **7** was an unlanded intermediate. Both and all
older formats are unsupported. See the
[storage contract](../packages/snapshots/README.md#current-format-and-preservation)
for validation and preservation, and the
[native contract](../packages/hledger/README.md#overview-data) for producer semantics.
1. Stop old preparation execution before upgrading.
2. Deploy compatible web **and** worker revisions, including rebuilt shared packages,
together. Independent image publication is not coordinated activation. For a
local compiled pair, after stopping the old stack:
```sh
docker compose -f apps/web/docker-compose.yml up --build --wait web worker
```
Compiled browsing still requires the trusted proxy described below. For local
Vite development, use the development Compose file instead.
3. Explicitly **Prepare reports again** in Settings, including with an existing
format-6 or format-7 set, then reload/navigate to saved reports.
Unsupported files remain untouched but unread. Account IDs may reset, so old account
URLs may no longer resolve. There is no automatic preparation, migration or deletion.
Do not clear data or edit envelopes to complete activation. Authentication,
configuration, the v1 void-input workflow and independent validation evidence do
not change.
## Register configuration
Add exact account entries to the existing private YAML to enable the corresponding
registers. The safe example and owned browser fixtures configure checking for Cash
and brokerage/retirement for Investments. Adapt these names to your journal; do not
copy fictional members over an existing operator configuration. An omitted list or
register with no members hides that view and makes its URLs unavailable.
Optional labels are literal app-wide display text, not native aliases. Label-only
entries can name categories or query results without enabling a register. Holdings
uses investment membership for account selection and displayed results; with no
investment members, it is hidden and unavailable too. Missing saved members remain
visibly unavailable; restarting after a label/membership change does not prepare
reports or rewrite data. See
[account presentation](../packages/config/README.md#account-presentation).
## Configuration filename transition
Before updating an existing deployment, stop both services and preserve the current `reports.docker.yaml` (or your selected YAML) as `config/hledger-reports.yaml`, refusing any overwrite. Keep its journal selection exactly as configured, including the container path. Do not replace it with the example. `version: 1` may be added, but omission defaults to `1`.
The new Compose file selects that ignored live file. Explicit native `CONFIG_PATH` or `--config` overrides still accept other filenames; update your service environment to the canonical path when ready. There is no startup migration or filesystem discovery. Restart both processes after config changes; no reports are prepared automatically.
The container journal path `/journal/main.journal`, host journal directory and sibling `data/jobs`/`data/snapshots` locations are unchanged. Renaming only the YAML file within its existing `config/` directory does not move data or change journal identity. Never delete or recreate data to complete this transition.
From the repository root, `pnpm config:validate --config apps/web/config/hledger-reports.yaml` checks the selected config and environment expansion without opening journals, running hledger or creating data. `pnpm worker:inspect` remains the separate native fixture preflight. `/schema.json` serves the generated schema, not configuration; use the browser-facing URL in the YAML language-server comment (`http://127.0.0.1:5173` for default Vite development).
## Inspecting data and preserving the previous volume
The bind mount keeps runtime data visible on the host:
```text
apps/web/data/
jobs/queue.sqlite
snapshots/current/overview.json
snapshots/current/transactions/7m2q9k4x.json
snapshots/current/transactions/2b8n5r6p.json
```
For file contents and private metadata, see the
[storage layout](../packages/snapshots/README.md#publication-and-reads). Inspect
published files read-only. Do not edit queue files or completed datasets while the
services are running. Runtime contents are ignored
by Git, formatting/lint tools and Docker builds, and Vite denies raw data serving.
The entire folder is Git-ignored, with no tracked placeholder. Create it on the host
before first startup so Docker does not create a root-owned bind directory. The containers
still run as UID 1000; this folder must be writable by that user. On the owner's Linux checkout,
the host and container UIDs match. Keep financial directories private.
**Existing named-volume data is not moved or deleted automatically.** Before the
first start with the new bind mount, stop both writers and copy the old volume if
you want to preserve queued jobs, SQLite results and existing publications. For the
default Compose project, run from `apps/web`. If the old service containers still
exist and `data/` is absent or empty, the short path is:
Use the Node and pnpm requirements in [package.json](../package.json). Install the
pinned hledger executable, dependencies and shared packages:
```sh
docker compose -f docker-compose.dev.yml stop &&
mkdir -m 700 -p data &&
docker compose -f docker-compose.dev.yml cp worker:/app/data/. ./data &&
docker compose -f docker-compose.dev.yml up --build --wait
sh tools/install-hledger.sh .cache/bin
export PATH="$PWD/.cache/bin:$PATH"
pnpm install --frozen-lockfile
pnpm build:packages
```
Do not use that merging copy on an already-populated target. If you already used
`down`, the containers are gone but the old volume remains; this alternative
refuses a populated destination:
Run web and worker in separate terminals with the same explicit `CONFIG_PATH`:
```sh
docker compose -f docker-compose.dev.yml down &&
mkdir -m 700 -p data &&
docker volume inspect hledger-web-reports_reports-data >/dev/null &&
docker compose -f docker-compose.dev.yml run --rm --no-deps --entrypoint node \
-v hledger-web-reports_reports-data:/old-data:ro worker -e '
const fs = require("node:fs");
const target = "/app/data";
if (fs.readdirSync(target).length)
throw Error("Host data folder is not empty; nothing was overwritten");
for (const name of fs.readdirSync("/old-data")) {
fs.cpSync("/old-data/" + name, target + "/" + name,
{ recursive: true, force: false, errorOnExist: true, preserveTimestamps: true });
}
console.log("Copied data; the original volume remains unchanged.");
' &&
docker compose -f docker-compose.dev.yml up --build --wait
CONFIG_PATH=/absolute/deployment/config/hledger-reports.yaml pnpm dev
CONFIG_PATH=/absolute/deployment/config/hledger-reports.yaml pnpm worker:dev
```
Both command chains stop on any failure: inspection/copy must succeed before startup. The alternative refuses to merge into an already-populated host folder. Use your actual old
volume name if you changed the Compose project name. If you deliberately start
with an empty folder, the services create a fresh queue; use Prepare reports in Settings to create
saved reports. The old volume still remains available. Never use `down -v` to migrate.
Use host-visible journal paths outside Docker. `pnpm worker:inspect` checks a
small fictional journal in a temporary directory; it does not prepare reports.
## Manual startup checks
## Checks
Use the normal development Compose file and **http://127.0.0.1:5173/**; the owner's local mise/fnox Docker task can run the same commands. Do not use `down -v`.
After [review approval](philosophy.md#review), run the affected package's tests,
type check and lint. Build shared packages when their runtime code changes.
- **Warm boot:** keep the data folder, stop/start services and open Overview. Saved reports render immediately; Settings retains their preparation timestamp. Editing inputs while stopped does not change that: a new set requires an explicit request.
- **Worker unavailable:** leave web running and stop worker. Browsing saved reports still works. A submitted preparation waits in the queue; starting the worker executes that already-requested job. Restart alone must not create a new request.
- **Journal changes:** preserve a copy of the fictional fixture, then edit a salary or applicable price. Reloading Overview still shows the saved values. Visit Settings, submit Prepare reports, reload its status, then reload Overview to see the new values. Restore the fixture and explicitly prepare again afterward.
- **Failure/retry:** request preparation against an invalid fictional journal. The job fails safely, worker health remains available and saved reports remain readable. Fix the source and explicitly request preparation again. Neither source restoration nor page reload retries the failed job. No raw native diagnostics should reach the browser.
- **Cold boot:** automated tests use disposable deployments. For an optional manual check, stop services and move only the prepared directory aside using the same worker image and existing data bind mount:
```sh
cd apps/web
docker compose -f docker-compose.dev.yml down &&
docker compose -f docker-compose.dev.yml run --rm --no-deps --entrypoint node worker -e 'const fs=require("node:fs"); const p="/app/data/snapshots"; if(fs.existsSync(p)){const b=p+".manual-backup-"+Date.now(); fs.renameSync(p,b); console.log(b);}' &&
docker compose -f docker-compose.dev.yml up --build --wait
```
Overview now reports that no saved reports exist. Visit Settings and submit Prepare reports; reload Overview after preparation finishes. The printed backup is preserved, as are the SQLite queue, configuration and journals. To restore it, stop services, move the newly prepared directory to a separate backup, then rename the saved backup to `snapshots` using that same one-off Node command pattern. Never delete the volume to simulate a cold start.
Settings exposes independent recorded validation; a failed preparation need not
mean unchanged finances after a post-commit alias failure. Follow the
[validation phase boundary](architecture.md#last-recorded-validation) and
[publication/recovery contract](../packages/snapshots/README.md#publication-and-reads)
when inspecting failures. Read-only `worker:inspect` never writes evidence.
Preparation timestamps and validation records do not prove current-input freshness.
## Compiled preview
This compiled setup requires an existing trusted HTTPS proxy under the
[trusted-proxy contract](authentication.md#trusted-proxy-contract), with YAML
`url.app` (the example's `APP_URL`) set to its public origin. Compose supplies
`PROTOCOL_HEADER=x-forwarded-proto` and `ADDRESS_HEADER=x-real-ip`; the proxy must
overwrite both and preserve public Host. It must be the only route to the upstream.
Stop live development before switching; both modes use the same project, data
folder, and worker health port.
```sh
docker compose -f apps/web/docker-compose.dev.yml down
docker compose -f apps/web/docker-compose.yml up --build --wait
```
Browse through that proxy, not directly at `http://127.0.0.1:3000/`: adapter-node 6
assumes HTTPS without a trusted protocol. For direct local HTTP browsing, keep using
Vite development. No proxy service or new listener binding is added here.
These are pruned production images running
compiled JavaScript, not Vite or the TypeScript development loader. Source edits
need another build. Web has no hledger executable; only the worker image does.
Worker health is at `http://127.0.0.1:9000/healthz`; web also has `/healthz`.
Web startup is independent of worker health so cached reports remain available. Worker health does not wait for report calculation. These endpoints establish
process readiness/liveness, not the freshness of every financial report.
```sh
# Stop while preserving the existing financial data volume.
docker compose -f apps/web/docker-compose.yml down
```
## Focused local checks
During development, start with the tests for the changed behavior:
```sh
pnpm --filter @hledger-reports/web test src/lib/server/accounts.test.ts
pnpm --filter @hledger-reports/config test src/accounts.test.ts
pnpm --filter @hledger-reports/web check
```
Build shared packages first when their runtime code changed (`pnpm build:packages`).
Use the affected package's lint command for a source change, and `check:schema` when
changing the configuration contract. Do not rerun every workspace test, native
preflight, browser journey and artifact probe after a markup or copy edit. The
[testing philosophy](philosophy.md#tests-while-the-foundations-are-changing) explains
what we retain and when to add coverage.
Use the [web file map](../apps/web/README.md#run-and-check) for co-located feature
tests: `report-period/period-query`, `overview/select-period`, feature `chart` and
`spending-categories/presentation`; shared rendering is `charts/rendering.integration`.
Native interpretation and storage validation stay tested at their package owners.
The full workspace gate remains available for cross-cutting changes and CI. In a
disposable container:
```sh
docker compose -f apps/web/docker-compose.yml run --build --rm checks
```
This runs type checks, lint, unit/integration tests, builds, and native preflight
with Node **26.5.0**, pnpm **11.9.0**, and checksum-verified hledger **1.99.5**.
Tests have their own loopback network and do not mount your source or data volume.
For the compiled **fictional-fixture** deployment, Chromium uses the test-only override:
```sh
docker compose -f apps/web/docker-compose.yml -f apps/web/docker-compose.e2e.yml run --build --rm e2e
```
This is the GitHub compiled-fixture path, not a command for personal data. The
override replaces config, journal and data mounts with the owned
`apps/web/.cache/browser-fixture/` layout. It does not use operator paths.
Prefer the host-native browser command below for ordinary local use.
Fixture preparation belongs to test code, not shell assembly in a pipeline:
```sh
# From the repository root; supply fictional AUTH_SECRET/AUTH_PASSWORD for Compose.
PLAYWRIGHT_USERNAME=browser-reader node apps/web/e2e/prepare.js
```
The command calls `e2e/fixture.js` and refuses an existing deployment directory.
It writes config, a complete fictional journal tree, empty data and an ownership
marker with private modes. Config retains environment references, not secret values.
Before starting containers, GitHub invokes `e2e/permissions.js` in its root fixture
container to prepare the marked mount for UID 1000; its cleanup restores checkout
ownership. These commands target only the owned cache fixture, never live data.
Woodpecker runs Playwright, which owns `e2e/server.js` startup and teardown. That
server uses the same `fixture.js` helper in a fresh OS temporary directory. Neither
pipeline assembles journal text. Both exercise the same small code fixtures,
including the earlier EUR-only entry for period-dependent currency availability.
An external `PLAYWRIGHT_BASE_URL` must target that prepared fictional layout and
matching credentials; the URL override alone does not configure or isolate it.
No journal files are tracked or admitted to the Docker context. Small fictional
literals in `*.fixture.ts`, the worker's `fixtures/journal.ts` and the browser's
category fixture are materialized only into temporary or ignored directories.
Do not add private inputs or rename journal files to evade this boundary.
The override runs `e2e/web.js`: an owned loopback HTTP boundary that overwrites protocol
and address headers with the actual HTTP scheme and socket IP before invoking the
exported native server handler. Browsers send normal headers and sign in normally;
there is no authentication bypass.
For Vite Compose fixture testing, use `docker-compose.dev.yml` alone against the
same disposable inputs. The small smoke suite covers sign-in, explicit preparation,
saved reports/registers, Holdings, live queries, navigation, period/history/GET
behavior and category chart/text alternatives. It exercises real services, not
mock publications. Financial and storage edge cases belong to package tests. Run
browser checks for changed journeys/integration, not every copy edit. Output is in
the terminal; disposable browser containers are removed afterward.
For a host-native browser run, install dependencies and Chromium, then:
Browser tests need built apps, Chromium and its OS dependencies:
```sh
pnpm --filter @hledger-reports/web exec playwright install chromium
pnpm build
pnpm --filter @hledger-reports/web test:e2e
pnpm --filter @hledger-reports/web test:e2e --trace off
```
The GitHub compiled-image test uses only fictional fixture data and explicitly
prepares bind-directory ownership for the unprivileged image user; this does not
change normal runtime users or grant access to private host data.
Leave `PLAYWRIGHT_BASE_URL` unset for isolated runs. The test server creates and
cleans up fictional data; never point browser tests at a personal deployment.
This mode requires host hledger 1.99.5 and browser OS dependencies. Playwright
uses the same owned HTTP boundary for compiled web and starts the worker on ports
3000/4193 respectively, against a fresh temporary
deployment with its own generated fictional journal tree, and removes it on shutdown. Port 3000
must be free: Playwright refuses to reuse an existing server. It never reuses the
development data volume or mutates an operator's journal. Browser helpers observe
reports through the UI rather than publishing fixture snapshots. Use `PLAYWRIGHT_BASE_URL` only with the externally staged fictional deployment
described above.
## Private data and upgrades
## Networking and platforms
- Keep live configuration, journals and runtime data out of Git and images.
- Compose shares `apps/web/data/` between web and worker. It contains sensitive jobs and financial results and must be writable by container UID 1000.
- Stop both services before backing up the whole data directory. Upgrade web and worker together; restore a consistent backup rather than downgrade a migrated database.
- Do not delete data or reset volumes to fix unavailable reports. Request preparation explicitly and reload to inspect the result.
- Background maintenance removes eligible older balance-query jobs, not saved reports. Deletion is not secure erasure; SQLite and backups may retain data.
The checked-in development/preview Compose files still use **host networking** and
bind listeners to **127.0.0.1**. They do not expose the app or configure a proxy.
The compiled app supports an existing trusted proxy that preserves the public Host
and exclusively controls upstream access. See the [trusted-proxy contract](authentication.md#trusted-proxy-contract)
for client-address forwarding and origin requirements. Authentication remains mandatory.
## Automation
Use Linux Docker Engine, or Docker Desktop 4.34+ with host networking enabled.
Stop existing native servers using ports 3000, 5173, or 9000 before starting the
corresponding stack. The initial worker/browser setup targets **linux/amd64**;
there is no claimed native ARM hledger image. Desktop may need emulation.
This is a local test/preview setup, **not a public deployment recipe**.
## CI and image publication
The app-owned `dev → builder → pruner → prod` Docker stages, source-mounted dev
services, separate worker, and gate-before-publication lifecycle follow When.
Like When, production uses Alpine. The upstream Linux hledger executable is
statically linked; its version and SHA-256 are fixed in `tools/install-hledger.sh`.
Failures never fall back to a distro or latest hledger version. The browser image and `@playwright/test` are
kept on **1.61.1**, as in When.
Like When, each provider has separate app/package pipelines, with no aggregate
`ci.yml`:
| Owner | GitHub | Woodpecker |
| -------------- | --------------------------------- | --------------------------- |
| Web | `.github/workflows/web.yml` | `.woodpecker/web.yml` |
| Worker | `.github/workflows/worker.yml` | `.woodpecker/worker.yml` |
| Config | `.github/workflows/config.yml` | `.woodpecker/config.yml` |
| Native adapter | `.github/workflows/hledger.yml` | `.woodpecker/hledger.yml` |
| Jobs | `.github/workflows/jobs.yml` | `.woodpecker/jobs.yml` |
| Snapshots | `.github/workflows/snapshots.yml` | `.woodpecker/snapshots.yml` |
Each pipeline checks/lints/tests its owner, building dependencies first. Package
pipelines do not publish images. Each app publishes only its own image after its
own gates pass; web additionally owns browser integration and workspace-tooling
lint. Path filters include shared build inputs and relevant package dependencies.
Web also watches worker changes because its browser test uses the real worker.
These are independent pipelines, not a cross-pipeline publication barrier.
- **GitHub:** web's browser job tests the actual compiled Compose images. Main
pushes publish web to `ghcr.io/<owner>/<repository>` and worker to the `-worker`
sibling using `GITHUB_TOKEN`.
- **Woodpecker:** web follows When's shared-workspace Node/browser steps and tests
an isolated compiled deployment with the verified native CLI. The app pipelines
publish to `git.eleith.com/eleith.private/hledger-web-reports` and the `-worker` sibling.
Enable directory-based pipelines and provide the existing When-style
`gitea_docker_registry` secret with package-write access. The agent must permit
the Docker buildx plugin, as it does for When. Newer Woodpecker versions require
`woodpeckerci/plugin-docker-buildx:6.0.3` to be explicitly included in the agent's
`WOODPECKER_PLUGINS_PRIVILEGED` allowlist; do not enable arbitrary privileged
pipeline steps.
Main pushes publish `latest` and `sha-<full-commit>` images. App release tags
publish `X.Y.Z`, `X.Y`, and `X` aliases after checking the tag matches that app's
package version. Release tags never move `latest`; GitHub also retains its SHA
image on tagged builds. Pull requests/manual checks do not publish. Neither
pipeline deploys a public service, reads personal journals, or needs production
config. Deployment/notification policy remains deferred. GitHub execution requires
a GitHub mirror; Woodpecker requires repository activation on that server.
## Releases
Follow When's manual [Changesets](../.changeset/README.md) workflow. Web and worker
form a fixed version group; shared packages are ignored for release purposes.
All packages remain private. Changesets versions/tags the apps and generates their
changelogs; it does not publish npm packages, push refs, or deploy services.
1. Record an app change with `pnpm changeset`. For changes in a shared package,
describe the effect under the affected app, not the ignored package.
2. Inspect the pending plan with `pnpm changeset status`.
3. Run `pnpm version:apps` to consume pending notes and update both app versions and
changelogs. Review and commit those changes (and any lockfile update) through
the normal review process.
4. Once that release commit is on a clean, reviewed `main`, create the app tags:
```sh
pnpm tag:apps
git push origin main '@hledger-reports/web@X.Y.Z' '@hledger-reports/worker@X.Y.Z'
```
Replace `X.Y.Z` with the actual app version. Pushing these namespaced tags runs
each service's own pipeline and publishes its versioned container image. A
mismatched tag fails before registry login/publication. The initial image
release path accepts ordinary stable `X.Y.Z` versions, not prerelease channels.
Adding release tooling or writing a changeset does not itself bump a version or
create a tag. There is no release bot, automatic version commit, or npm publish
step. Private package tagging is enabled deliberately because these app tags,
rather than npm releases, drive image publication.
GitHub and Woodpecker definitions live in `.github/workflows/` and `.woodpecker/`.
Both use owned fictional fixtures. Releases are [manual](../.changeset/README.md);
publishing images does not deploy the app.
docs/philosophy.md
+25 -105
View File
@@ -1,115 +1,35 @@
# Philosophy
# Development principles
A small, read-only financial app for an hledger journal. The product should be easy
to use and the code should be easy to explain. Necessary complexity belongs behind
clear inputs and outputs, not scattered across its consumers.
## Readable code first
## Read, request, produce
- Make folder, file and function names explain ownership. Keep feature code and its tests together.
- Let operations show meaningful phases. Keep resource lifetimes and cleanup together.
- Improve awkward producer output before adding consumer workarounds. Prefer deletion over speculative abstractions.
- Use comments only for necessary external context, not to narrate code. Keep docs and PR descriptions plain and short.
- **Browsing reads saved files.** Missing reports and unavailable evidence remain
explicit. Neither saved finances nor a recorded check proves current-input
freshness. Rendering must not inspect journals, query jobs or request work.
- **Users request work.** Preparation and live queries are explicit, separate
operations. Startup and journal edits do not prepare reports; open pages do not
update themselves.
- **Jobs produce data.** Record a completed check before financial work, publish only
complete results, and let navigation/reload observe saved data. Concrete page and
phase ownership belongs to [architecture](architecture.md).
## Financial boundaries
## Ownership before abstraction
Hledger owns accounting. Preserve exact values, separate commodities and explicit
missing or ambiguous states. Category-chart grouping is the only browser arithmetic:
exact, within one saved period and commodity, without changing native data.
Hledger owns accounting. The library interprets exact output and owns the process
boundary; web requests work, workers execute, storage persists, and web presents
saved data. Operational version/inspection checks are explicit exceptions, not
browser financial jobs. Shared job code owns contracts and queue access, not a
second job framework.
Browsing reads saved data; financial work requires an explicit request. Preserve
private data and completed publications. Unsupported reports require explicit
preparation, not automatic migration or deletion.
Validation evidence is independent of financial publication, not a gate on readable
saved finances or proof of freshness. Preserve evidence and completed user data
across failures; concrete phase/failure rules belong to
[architecture](architecture.md#last-recorded-validation) and
[storage](../packages/snapshots/README.md#publication-and-reads).
## Tests
If a consumer compensates for awkward output, improve the producer contract before
extracting more helpers.
Keep tests that protect valuable rules we own: financial fidelity, privacy, access,
complete publication and resource ownership. Test at the owning layer with a few
representative cases, not repeated matrices. Do not test hledger, Chart.js, framework
behavior, UI copy or appearance. Exact financial strings are data, not copy.
The sole owner-approved browser-arithmetic exception is **Spending by category
chart grouping**: local exact scaled-decimal `BigInt` ranking and Top 5 + Other
within one selected saved period and exact commodity. It must not combine ambiguous
same-commodity entries within a native row, invent zeros or convert currencies.
Native bags, full exact tables and accounting totals remain untouched; approximate
numbers are for canvas geometry only. This is not a shared money utility, server
rollup or stored financial field. Other charts retain native-owned rules. See
[presentation ownership](architecture.md#presentation-ownership) and
[chart behavior](../apps/web/README.md#spending-by-category-chart).
Use disposable fictional inputs, never operator journals. Keep a small browser
smoke suite for important workflows. Test counts and coverage are not goals.
All workspace packages are internal. Retain APIs for actual consumers, not because
an old experiment or test once needed them. Preserve useful financial guarantees
and existing user data, not accidental implementation structure.
## Review
Prepared reports are regenerable outputs. Support one complete current format rather
than carrying older file shapes and missing-card branches through the app. If saved
data is unsupported, ask for explicit preparation again; do not silently migrate,
regenerate or delete it. This does not remove safeguards for current writes or genuine
empty reports.
## Readable complexity
- Name files and functions for the responsibility they have now. Use `process` for
executable lifecycle, `checks` for validation, and the actual report name for its
implementation. Use `hledger` to identify its output, rather than ambiguous `native`
labels that could mean platform code or UI controls.
- Distinguish verbs: `prepare` calculates report data, `read` retrieves saved data,
and `request` submits a job. Folder context should make those responsibilities
clear without redundant filename prefixes.
- Make the folder tree explain the app. Group code by feature or a concrete shared
responsibility, such as charts; name files for their purpose within that folder.
Product boundaries matter more than file count: a small feature can have its own
folder, while genuinely shared mechanisms can remain at the package root.
- Keep tests beside the code they cover, using matching names such as `net-worth.ts`
and `net-worth.test.ts`. Label cross-module integration tests honestly. SvelteKit
route tests omit the reserved `+` prefix: `+page.server.ts` and `page.server.test.ts`.
- Treat file placement as part of assigning ownership, not a cosmetic step at the
end. Do not create empty folders or generic helpers for hypothetical features.
- Let the main operation describe its phases. Extract a function when its name
explains a meaningful operation, not to meet an arbitrary line limit.
- Keep resource ownership and cleanup visible. Avoid splitting one stateful
lifecycle among independently mutable services.
- Share code when the meaning is shared, not merely because two loops look alike.
Share concrete mechanisms between actual consumers; keep feature data and
rendering with the feature rather than building a catalog or controller.
- Prefer deletion over speculative registries, configuration knobs, generic data
layers or reusable controllers with no current consumer.
## Tests while the foundations are changing
Tests should make this small app cheaper to change, not turn each iteration into a
release qualification exercise. Keep a small set around core behavior we own:
exact native values, complete publication, explicit work, access control and resource
ownership. Add a meaningful failure case when it protects one of those contracts.
- Test a rule at its owning layer. Do not repeat parser, financial, authentication or
storage cases in every route and browser test.
- Prefer a few representative inputs over exhaustive permutations of types, strings,
framework responses or hypothetical callers. Internal APIs do not need compatibility
tests for consumers we do not have.
- UI tests should protect financial presentation meaning, not CSS classes, HTML shape,
every label or framework behavior. Type checking and a small browser smoke suite
cover routine wiring while the design is still moving.
- Browser tests exercise real user journeys: sign in, request preparation, read saved
reports and run an on-demand report. Do not build a mock publication system or
duplicate each chart's edge cases in a browser.
- During iteration, run the affected tests and relevant static checks. Run browser
smoke tests when changing a journey or its integration. Full workspace, artifact and
deployment checks are for changes that actually cross those boundaries, not every
markup or copy edit.
- Expand coverage when a contract settles or a real regression demonstrates a gap.
Test counts and coverage percentages are not product-progress targets. Do not keep
obsolete tests hidden behind another testing mode.
Tests remain co-located and use disposable fictional data. Never substitute an
operator's journal or live service for an owned test fixture.
Correct accounting, coherent publication and safe local access remain essential.
They are not reasons to retain unnecessary automatic behavior. See
[architecture](architecture.md) for the concrete boundaries.
Prepare one commit's changes unstaged, then stop for owner review. After approval,
run code, readability and test reviews, then relevant checks. Stage and commit only
with approval and passing checks. Push and open a PR only with approval; the owner
merges. PR descriptions briefly explain the change and checks, without a process diary.
packages/config/README.md
+14 -129
View File
@@ -1,138 +1,23 @@
# @hledger-reports/config
# Configuration
The canonical deployment file is `config/hledger-reports.yaml`:
Web and worker use the same private `config/hledger-reports.yaml`. Start from the
[example](../../apps/web/config/hledger-reports.example.yaml) only for a new deployment;
never overwrite an existing journal selection or secrets.
```yaml
# yaml-language-server: $schema=http://127.0.0.1:5173/schema.json
version: 1
journal: ../journals/main.journal
url:
app: http://127.0.0.1:5173
auth:
credentials:
username: household
password: '${AUTH_PASSWORD}'
```
- `journal` selects the journal. Relative paths resolve from the configuration file; Docker needs the worker's container path.
- `url.app` is the exact browser origin. Configure one [authentication strategy](../../docs/authentication.md).
- `accounts` selects exact account names, optional display labels and `cash` or `investment` register membership. Labels and names are literal; membership is presentation, not authorization or journal scope.
Use the browser-facing schema URL.
The endpoint serves the bundled editor artifact, never live configuration. The
tracked `schema.json` package export is also available for offline editors.
Environment references support `${NAME}` and `${NAME:-fallback}`. Account names
and labels are not expanded. Restart services after changes.
## Input and resolution
Data lives in the sibling `data/` directory beside `config/`; web and worker must
select the same deployment. Do not move or replace existing data to change labels.
`version` defaults to `1` when omitted; other values are rejected. `journal` is
required and nonblank, with no control characters. Unknown fields are rejected:
storage directories and reporting currency are not settings.
`url.app` is the required, authoritative canonical HTTP(S) origin. Both framework
request URL origins must match its normalized value; compiled proxy setups derive
their scheme and Host through adapter-node, not a build-time public URL. Local Vite
development is unchanged. `auth` requires exactly one
`credentials` or `oidc` branch; its optional `secret` defaults to `${AUTH_SECRET}`
and must resolve to at least 32 characters. There is no generated fallback. See
[authentication](../../docs/authentication.md) for provider and session semantics.
Deployment/auth string settings support `${NAME}`, `${NAME:-fallback}` and `$$` for a literal
`$`. Names use uppercase letters, digits and underscores, with no leading digit.
Fallback applies only to an unset variable, not an empty value. Expansion is
single-pass, does not interpret YAML and does not expand variables inside a
replacement. Missing variables and invalid resolved values fail safely.
`validateConfig(input, { env })` returns a new private `Config` with defaults and
interpolation applied; it does not mutate the raw `ConfigInput`. Omit `env` to use
`process.env`. Defaults never repair nulls, arrays or malformed containers.
`loadConfig(path, { env })` reads only the explicit selection and resolves all web
settings, including secrets. `loadWorkerConfig` and `validateWorkerConfig` check the
full raw contract but resolve and return only `version` and `journal`; worker does
not read web-only environment variables. Both loading functions resolve relative
journal paths from the YAML file, not the working directory. Neither opens journals,
creates storage, watches files or discovers personal configuration. Restart both
processes after configuration changes; loading never prepares reports.
Only one YAML 1.2 core document is accepted. Duplicate keys, unsupported tags,
ambiguous shapes and unsafe inputs are rejected. Errors expose fixed codes and
known field paths, never values, environment variable names, private paths,
unknown property names or parser excerpts.
## Account presentation
Append an optional `accounts` list to the shared file:
```yaml
accounts:
- account: assets:bank:checking
label: Everyday checking
register: cash
- account: assets:investments:brokerage
label: Brokerage
register: investment
- account: expenses:living
label: Living expenses
```
Names and labels are **literal**, including dollar signs and `${...}`. They are not
interpolated, trimmed, case-folded, prefix-matched or normalized. `account` is the
exact native identity; labels are nonblank app-wide display text. Duplicate account
entries are rejected, even if one is label-only. Different accounts may share a label.
Omitted `label` displays the native name. Omitted `register` is label-only; the one
optional register value is `cash` or `investment`. No user IDs, slugs or patterns are
accepted. App-owned account IDs remain stable across label edits.
Omitted/empty `accounts` disables both register views. A register without members
is hidden and its routes return 404. Direct account routes require exact membership
in that register. A configured member absent from saved data is shown as unavailable,
not given an invented ID, count or balance. Loading does not prepare or repair data.
Holdings shares the investment membership: its selector and results include only
exact configured investment accounts, and nonmember selections are rejected server-side.
With no investment members, Holdings is hidden and returns 404. “All investment accounts”
filters native account groups without recomputing their values or including implicit
children. An enabled All request can still use a manually selected date before reports
have been prepared; individual selections require a current saved account ID.
Membership is presentation, not journal scope or user authorization. Preparation
still covers all native accounts, and live queries keep their native inputs. The
Holdings worker still produces native reports; the web filters their account groups
before returning results. Private queue output is not a filtered account-access store.
Worker checks configuration shape/duplicates but does not use presentation settings
in financial execution. Restart after changes; preserve the journal and
existing data directories.
## Editor artifact and operator validation
TypeBox owns the input fields and derived types. The editor schema describes raw
input, including version and secret defaults and environment references; loaded configuration is resolved
private state, not a public page payload or a resource container.
Validate configuration without reading the journal:
```sh
# From this package
pnpm generate:schema
pnpm check:schema
# From the repository root; no journal/native/queue work
pnpm config:validate --config apps/web/config/hledger-reports.yaml
pnpm config:validate --config /absolute/deployment/config/hledger-reports.yaml
```
The schema check compares generated bytes and fails on drift without rewriting.
CI runs it explicitly. Configuration validation uses the current environment;
`CONFIG_PATH` is accepted when `--config` is absent. Native journal validation
remains the separate worker inspection command.
## Fixed deployment layout
```text
<deployment>/
config/hledger-reports.yaml
data/jobs/
data/snapshots/
```
`resolveDataPaths(configPath)` derives these runtime locations, outside the YAML
contract. Web and worker must select the same deployment. Renaming a file within
its existing `config/` directory preserves the data locations; files in the same
config directory are not independent deployments. Keep the selected journal path
unchanged to retain its saved-report identity. No data migration occurs.
The repository-local live YAML is ignored; only a safe example is tracked.
Explicit `CONFIG_PATH`/`--config` selections may use another filename. Authentication and
canonical-origin protections remain separate from configuration loading.
The running web app serves the editor schema at `/schema.json`.
packages/hledger/README.md
+5 -365
View File
@@ -1,367 +1,7 @@
# @hledger-reports/hledger
# Hledger
Hledger library, pinned to **1.99.5**. It reads journal data and organizes neutral, chart/table-ready datasets rather than exposing CLI-shaped results or Svelte chart props. Hledger owns accounting; this library is not a second accounting engine. Process execution and exact decoding stay inside the library; web imports hledger types only. Only operations used by the apps are exported; hledger JSON types, codecs and process helpers stay private.
Runs hledger for explicit worker jobs and converts its output into prepared reports.
Hledger owns accounting; quantities are preserved as exact decimal strings.
- `prepareReports(path, { writeRows, signal? })` calculates Overview and all account activity by running hledger for the manual preparation job. Unlike storage's `readReportPublication`, it prepares new data rather than reading a saved report.
- `getHledgerVersion()` verifies the inspected engine version; its return is the
native banner. Preparation calls it once in a version-only process and saves the
validated numeric identifier `1.99.5`, not that banner.
- `checkJournal(path): Promise<void>` runs read-only strict checking with ambient hledger config disabled. It returns only after complete consumption, normal exit 0 and cleanup; sanitized `JournalCheckError.kind` distinguishes `nonzero-exit` from `not-completed`. A normal nonzero supports **Could not validate**, not an invalid-accounting diagnosis: input may be missing, unreadable or malformed. Launch/abort/signal/stream/unexpected errors mean incomplete execution.
- `runCostBalance(path, expression, asOf)` runs one native `balance --strict --cost --flat` expression against the supplied journal path and its current includes. Native strict validation happens on every execution, not only at worker startup. It freezes relative-date context, uses an inclusive as-of cutoff, validates the returned structure, and preserves reported quantities as strings. It does not perform market valuation or implicit FX conversion. There are no application-imposed native runtime/output/row caps; this is not a journal-capacity or native-memory guarantee.
- Exact JSON ingestion reads integer mantissas as BigInt from original JSON tokens,
never rounded JavaScript numbers. The existing codec fails closed on
scientific-notation mantissa tokens; exactness is not a universal claim for
extreme quantities.
## Source layout
Feature modules own both hledger wire interpretation and ready report data. Matching
`*.test.ts` files live beside their modules; the combined producer also has
`prepare-reports.integration.test.ts`.
```text
src/
├── index.ts # Narrow public operations and ready-data types
├── process.ts # Only hledger process boundary
├── checks.ts # Engine version and strict journal checks
├── prepare-reports.ts # Combined calculation and staged row delivery
├── overview/
│ ├── batch.ts # Grouped Overview commands and output
│ ├── report.ts # OverviewData and report organization
│ ├── ranges.ts # Native 1/3/6/12-month totals and shared history
│ ├── income-statement.ts # Hledger income-statement JSON decoding
│ ├── balance-sheet.ts # Hledger balance-sheet JSON decoding
│ └── net-worth.ts # Interprets hledger net-worth results
├── accounts/
│ └── balances.ts # Bulk raw/valued inclusive account summaries
├── transactions/
│ ├── account-rows.ts # Register context and selected-posting indexes
│ └── transaction.ts # Full exported transaction/posting/amount metadata
├── queries/
│ └── cost-balance.ts # Recorded-cost query
├── amounts.ts # Shared exact quantities and amount projection
├── report-stream.ts # Incremental report framing
├── journal-stats.ts # Coverage, primary anchor, private currency and counts
└── report-date.ts # Shared ISO report-date formatting
```
The feature folders reflect the product views, even where a feature currently
needs only one module and its tests. Shared execution and preparation stay at the
root because they serve more than one feature. There is no client resource or
separate Overview executor. Storage and publication remain caller-owned.
## Overview data
`prepareReports` returns this Overview alongside the account directory. All fields are required; undated results use explicit nulls and empty histories. Report interpretation produces financial fields without timestamps; the preparation operation adds `preparedAt` once, after all account writes and successful hledger completion:
```ts
type OverviewMonths = 1 | 3 | 6 | 12;
type NativeValue = { amounts: ReportedAmount[]; unconverted: string[] };
interface OverviewRange {
transactionCount: number;
begin: string;
endExclusive: string;
income: ReportedAmount[];
expenses: ReportedAmount[];
net: ReportedAmount[];
expenseCategories: {
depth: 2;
rows: { account: string; amounts: ReportedAmount[] }[];
};
}
interface OverviewData {
asOf: string | null;
preparedAt: string;
hledgerVersion: string; // validated '1.99.5', not the native banner
ranges: Record<OverviewMonths, OverviewRange> | null;
balanceSheet: { assets: NativeValue; liabilities: NativeValue } | null;
netWorth: {
currency: string | null;
points: ({ date: string } & NativeValue)[];
};
journalCoverage: {
transactionCount: number;
firstDate: string | null;
lastDate: string | null;
};
incomeExpenses: {
months: {
begin: string;
endExclusive: string;
income: ReportedAmount[];
expenses: ReportedAmount[];
}[];
};
}
// ReportedAmount = { commodity: string; quantity: string }
```
`journalCoverage` is the whole-journal transaction count and inclusive native date span. It comes from the existing **cached batch's unfiltered stats**, alongside the financial report—not the earlier discovery process. Whole-journal coverage itself needs no additional native command; filtered period counts below use four. The native span includes secondary and posting dates, so it can extend beyond the latest primary transaction (`asOf`) or any prepared reporting range. The library converts the native exclusive span end to an inclusive `lastDate`; web does not interpret native statistics or perform that conversion.
Empty journals have coverage count `0` and null dates, with required
`hledgerVersion`. Price directives and unexpanded periodic rules do not create
recorded transactions. Runtime statistics and source paths are not exposed.
Coverage does not prove completeness or freshness. The parsed `baseCurrency`
remains private planning data for financial valuation, not a stored statistics
object.
Each 1/3/6/12-month range requires `transactionCount: number`, a safe nonnegative
integer from native `stats -b BEGIN -e EXACT_END_EXCLUSIVE` in the existing cached
second financial batch. Bounds are the range's `begin` (inclusive) and
`endExclusive` (`asOf` plus one day), without monthly interval expansion. It counts
actual transactions by **primary transaction date**, including transfers and opening
balances, independently of income/expense classification. Posting-date and secondary
(`date2`) overrides do not move a transaction into or out of the count. It is not a
posting count, sum of register/chart rows or TypeScript reconstruction. A dated
period with no matching transactions has count `0`; undated reports keep
`ranges: null`, not four invented zero-count ranges. Native preparation computes all
four counts once; browser selection only reads the selected prepared count.
Each range's `expenseCategories` contains native expense-account totals over that selected range, at maximum absolute account depth 2. For example, `expenses:food:groceries` and `expenses:food:restaurants` roll into `expenses:food`; depth is counted from the beginning of the full account name. Native flat reporting keeps shallower direct postings separate rather than displaying inclusive parent totals alongside their children. Full native account names and order are retained, including expense roots not literally named `expenses`.
Range income and expenses come from native subreport `prTotals.prrTotal`, NET from `cbrTotals.prrTotal`, and categories from native row `prrTotal` bags—not TypeScript sums of monthly columns, subtraction or child aggregation. Native signs and refunds are retained.
`netWorth` contains twelve native historical balance-sheet NET amount bags: the previous eleven month ends and a final point at `asOf`. Hledger performs assets-minus-liabilities and dated market conversion; the library does not sum balances or calculate FX. Every point uses only prices available on or before its valuation date, without falling back to transaction cost or fetching quotes. For the original fixture the April 16 point is **11825.00 USD**, not the **12010.00 USD** obtained using later April 30 quotes.
The target is hledger's **inferred base currency** from the same loaded journal, not a hardcoded USD or a new application setting. This is an inference, not a reporting-currency directive: `commodity` declares a commodity and `D` supplies input defaults. The inferred identifier is preserved exactly, including case, spaces, commas and an unnamed commodity. One target applies to the entire prepared trend. A different second-pass inference fails preparation instead of mixing currencies. The chosen target remains visible in the UI.
Each point's `amounts` is the native NET bag. `unconverted` lists non-target commodities remaining in either the native asset or liability totals, even if those units cancel in NET. Such a point is incomplete and is not plotted as a consolidated total; the exact table retains its native mixed/partial amounts. Missing native amounts are not invented target-currency zeros; a native explicit zero is retained. Empty journals return `{ currency: null, points: [] }`.
`balanceSheet` copies the latest native assets and liabilities side totals, with their native signs and separate `unconverted` lists; it does not recompute them from NET or account summaries. `NativeValue` always contains an amount bag and explicit conversion evidence, not a guaranteed target-currency scalar.
Native amount bags are not guaranteed to have one entry per commodity: distinct cost annotations can leave multiple entries. Consumers must not silently select the first or combine them as a presentation shortcut. Empty commodity IDs, exact quantity strings, duplicate entries and explicit zeros are preserved; empty native bags stay `[]`.
Each range begins on the first day of its earliest month and ends exclusively at `asOf` plus one day, including partial trailing months. The shared `incomeExpenses.months` history comes only from the twelve-month statement and retains twelve native calendar-month columns, including empty months. Those columns' `endExclusive` boundaries remain calendar boundaries, even for the clipped trailing month. `preparedAt` is the library's current preparation timestamp, not a snapshot-writer or SDK timestamp. Consumers receive ready JSON-serializable data; native tuples, raw income JSON and BigInt decoding stay inside the library. Consumer projections are owned by
[web architecture](../../docs/architecture.md#homepage-periods-and-presentation).
`asOf` is the latest recorded **primary transaction date**, not today or SDK request time. Assets, transfers, gains and recorded future-dated transactions can anchor the window. Price directives and unexpanded periodic rules do not count. Gain accounts (`type:G`) are excluded from income/expense values, not date discovery. Net worth uses native asset/liability classification; it does not inherit the income report's gain filter. Each 1/3/6/12-month range covers the as-of month and its preceding calendar months. Its trailing month runs through the latest primary transaction date without claiming completeness. For example, `2026-04-16` anchors the shared history at May 2025–April 2026 and a six-month selection at November 2025–April 2026 (the default three-month selection is February–April 2026), regardless of today's date.
No dated transactions returns the standard dataset with `asOf: null`, a current `preparedAt`, required `hledgerVersion`, `journalCoverage: { transactionCount: 0, firstDate: null, lastDate: null }`, `ranges: null`, `balanceSheet: null`, `incomeExpenses: { months: [] }` and `netWorth: { currency: null, points: [] }`, not fabricated zero months. The income/category report requests no cost, market valuation or FX conversion. Its amounts are not available cash, cash flow or net worth: transfers, loans and asset-sale proceeds differ. This neutral dataset supports the exact table and grouped bars without chart-library configuration in this package.
### Overview planning and interpretation
`overview/batch.ts` plans commands and decodes the framed Overview batch; it does not execute hledger. `journal-stats.ts` decodes metadata; `overview/income-statement.ts` and `overview/balance-sheet.ts` describe/decode their hledger wire formats. `overview/report.ts` owns `OverviewData` and organizes financial fields, delegating range/history organization to `overview/ranges.ts` and net-worth/side interpretation to `overview/net-worth.ts`. The combined preparation operation below owns when these steps happen; there is no separate Overview executor.
Unfiltered strict native statistics discover the primary anchor and inferred target. The actual prepared batch reads unfiltered statistics plus four bounded period `stats` commands, four income statements, twelve date-specific historical balance sheets, two bulk account balance reports and all account registers from the same loaded journal. For dated journals the existing **18 financial report commands** are unchanged; **four filtered stats commands** run inside the cached production batch, not four processes. There are still two financial processes (discovery plus cached production). The worker's separate strict preflight plus preparation's one version-only
`getHledgerVersion()` call make **four processes for the whole job**, still **three
journal parses**; version checking supplies no journal flags or reads. Known static markers delimit the reports; decoding from the end avoids mistaking metadata-path text or escaped JSON strings for a report boundary.
Each income command retains `--flat --depth=2 --change --monthly --empty`, its 1/3/6/12-month bounds, `not:type:G` and `not:date:AS_OF_PLUS_1..`. Each balance sheet uses `--historical --flat --empty --depth=1`, its month start, the day after its valuation date, `--value=EXACT_DATE,INFERRED_TARGET`, JSON output and the matching negative date cutoff. It does **not** use monthly interval expansion or the income gain filter. Native historical accumulation retains opening balances from before the displayed window.
Both passes disable ambient config and read the source journal and native includes read-only. The second `stats` primary date and inferred currency must match discovery; a changed anchor or target fails preparation and leaves the last good published dataset intact. Counts and span may change at the same anchor; the second parsed view supplies both coverage and financial data. For an empty journal, the actual batch still confirms statistics and account identity, without inventing financial periods. This does not guarantee atomic multi-file reads during initial parsing.
Each income report starts on the first day of its range and uses a command end at the first day of the following month. Native `--empty` retains every monthly column, including holes; only the twelve-month statement supplies published monthly history. The absolute **negative date query** at as-of plus one day keeps the native cutoff consistent within the monthly period; positive date filtering or a mid-month `-e` alone is unsafe under native interval expansion. This does not discard recorded future transactions based on the clock. `not:type:G` filters only income-statement values.
All reports must finish and validate before the combined producer succeeds. A failed later balance sheet or account register cannot publish only earlier results. Exact JSON decoding and interpretation stay in the library, checking the consumed structure and expected periods without a second comprehensive financial schema. Unsupported scientific-notation mantissa tokens fail closed before publication.
## Complete prepared reports
`prepareReports(journal, { writeRows, signal? })` calculates an Overview and account
transaction history together:
```ts
const { overview, accounts } = await prepareReports(journal, {
async writeRows(accountName, rows) {
// Append these rows to this account's private staged output.
}
});
// overview: OverviewData
// accounts: PreparedAccountSummary[]
```
```ts
interface PreparedAccountRow {
accountDate: string;
otherAccounts: string[];
isSplit: boolean;
change: DetailedAmount[];
balance: DetailedAmount[];
transaction: PreparedTransaction;
accountPostingIndexes: number[];
}
interface PreparedAccountSummary {
name: string;
access: 'ready' | 'ambiguous';
rowCount: number | null;
summary: PreparedBalance;
}
type PreparedBalance =
| { status: 'ambiguous' }
| { status: 'undated' }
| { status: 'ready'; balances: ReportedAmount[]; valuation: NativeValue };
```
Native strict discovery runs `stats` and `accounts --flat --directives` in one
cached `run`. A second, single cached `run` supplies the unfiltered stats, four
bounded period stats, the four-income/twelve-balance-sheet/two-bulk-balance financial
plan, the actual account list,
all native `accounts --find --directives` identity qualifications, and then all
full-history `aregister --historical --empty -Ojson` reports. Every published
financial value comes from this second loaded journal. Its anchor, inferred
currency and actual account list must match discovery. Both processes disable
ambient config. The financial batch remains independently `--strict`. This is not
an atomic filesystem snapshot. The standalone strict check precedes this financial path; command/process
accounting is specified under [planning and interpretation](#overview-planning-and-interpretation).
The check can observe different edits from financial production; a pass proves
neither identical inputs, freshness nor completed publication. Worker/storage record
outcomes; read-only inspection does not. The caller owns publication and cleanup;
see [storage](../snapshots/README.md#current-format-and-preservation) and
[activation](../../docs/development.md#financial-format-8-activation).
The account summaries contain only native used/declared accounts, sorted by literal name;
the directory does not expand implicit parents. An explicitly declared parent has native
**subtree** history. UI-only ancestor headings must not request extra registers.
Unused declared leaves remain ready with count zero and an empty `writeRows(name, [])` call.
Names retain native whitespace, punctuation and Unicode. They are identities,
never file paths; storage owns stable IDs and file layout, not this library. Control/unrepresentable identities
fail preparation. Native directive-prefixed names and validated static framing
prevent account names from being mistaken for protocol markers.
Qualification uses native matching, not JavaScript case folding. If a requested
name differs from its native match, or multiple requested names share one
canonical match, **all involved accounts** are ambiguous, with null counts and
no row writes. Their entire native register results are ignored rather than
financially filtered in JavaScript. Their summary is `{ status: 'ambiguous' }`, even
without a date; other undated accounts have `{ status: 'undated' }`.
For dated ready accounts, two bulk `balance --tree --no-elide --declared --historical
--empty -e AS_OF_PLUS_1 -Ojson not:date:AS_OF_PLUS_1..` commands provide raw inclusive
`balances` and end-valued `valuation`. Only the valued command adds
`--value=AS_OF,INFERRED_TARGET` as one literal argument. No per-configured-account
reports, role filters, cost conversion or JS financial sums are added. Investment
totals include uninvested cash and other non-lot assets.
Rows match exact full native names, never shortened labels or fuzzy/case-folded
names; existing native `--find` qualification remains authoritative. A missing
expected row fails preparation. Known declared-unused accounts retain native `[]`,
not fake zeros or missing-data states. Valuation's `unconverted` list also collects
residual commodities from exact `name + ':'` descendants, even when opposing child
exposures cancel in the inclusive parent. This collects evidence, not amounts;
incomplete valuation does not change register access. Summaries cover only the
existing declared/used directory, not extra implicit parents.
Rows retain native order and cumulative historical balances across callback
boundaries. `PreparedAccountRow` combines the native **account-relative** report
facts (`accountDate`, `isSplit`, `otherAccounts`, `change`, `balance`) with the
original exported transaction and selected posting indexes. Description and transaction status
have one authoritative location in `transaction`, not duplicate legacy table fields.
`transaction` retains the **original transaction exported in that same tuple**:
- `index` (journal index, meaningful only within this prepared set), original
`date`, nullable `secondaryDate`, native `status`, `code`, full `description`,
derived `payee`/`note`, `comment`, `precedingComment`, ordered `tags`, all
`postings` in native order, and private `sourcePositions`.
- Each posting retains `account`, `amounts`, nullable explicit `date` and
`secondaryDate`, own `status`, `effectiveStatus`, `comment`, ordered `tags`,
native `kind`, nullable `balanceAssertion` (amount, total/inclusive flags and
private source position), and nullable native `transactionIndex` string.
- `DetailedAmount` extends `ReportedAmount` with nullable recursive `cost`
(`kind: 'UnitCost' | 'TotalCost'`, `amount`), nullable `costBasis` (nullable
`amount`, `date`, `label`), and nullable `style` (`commoditySide`,
`commoditySpaced`, `decimalMark`, `digitGroups`, `precision`, `rounding`).
Change/balance, posting, assertion, cost and basis quantities all use the same
exact string codec. No monetary sums or conversions are performed here.
`PreparedTransaction`, `PreparedPosting`, `DetailedAmount`, `AmountStyle`, and
`SourcePosition` are public types alongside `PreparedAccountRow`. All fields
in the prepared contract are required; nullable fields contain explicit nulls.
The financial report format and required-container validation are owned by
[snapshots](../snapshots/README.md#current-format-and-preservation).
### Mapping and privacy
This library owns the native-to-prepared mapping. For the current unfiltered
commands, `accountDate` is the account-relative date, distinct from the original
`transaction.date`; original secondary and posting dates are also retained.
Preparation fails if native context changes anything beyond the account date,
rather than silently discarding metadata.
After native account identity qualification, `accountPostingIndexes` selects the
exact canonical account and its `:` subtree **case-sensitively**, including virtual
postings. These indexes refer to `transaction.postings`, not the complement of
`otherAccounts`. Transaction indexes are meaningful only within this prepared set,
not stable financial IDs across preparations or journal edits.
Payee/note use native description semantics: without `|`, both retain the full
untrimmed description; otherwise the first `|` separates two trimmed parts.
Description notes remain distinct from comments. Each posting's effective status
inherits the transaction status only when its own status is `Unmarked`.
The transaction header status is not an aggregate of selected posting statuses,
which may differ. Tags retain exported order, duplicates and inheritance.
Source positions, including assertion positions, are **private storage provenance**.
Never pass the complete prepared record into browser PageData: consumers must
explicitly project safe fields and omit provenance.
Exactness preserves exported decimal values, not approximate floating-point
values. Native aregister strips investment costs/basis even from its original
transaction; the mapper retains annotations when exported but cannot recover
precision or annotations lost upstream. This report is **not a verbatim source
archive or complete investment record**. No extra print command is issued.
History is not restricted to Overview's ranges or primary `asOf`; no
register date bounds, clock context, cost or valuation are requested. Exact native
JSON numeric tokens feed the existing BigInt codec; quantities such as `0.0049`
are not rounded through CSV or JavaScript numbers. Every ordered mixed,
explicit-zero and unnamed-commodity entry is preserved; empty bags stay empty.
Scientific-notation mantissas still fail closed.
Output is streamed with fatal UTF-8 decoding, strict incremental top-level row
array framing and callback backpressure. Only the metadata/financial prefix,
one raw tuple and an internal chunk of at most 50 projected rows are retained,
not the complete native JSON or an entire account history. Chunk boundaries are
not persisted pagination. `writeRows(accountName, rows)` is awaited before consuming more
rows, including during cancellation. Abort kills the native child
immediately, but the active callback must settle before this facade returns or
throws; staging can then be discarded without a late write recreating files.
Callback I/O is caller-owned, so a stalled callback can delay completion and cleanup.
**Writes must remain private/staged**: callbacks can run before a later native error. Only successful EOF, zero exit and completed validation allow the
returned Overview/account summaries and staged rows to be published together. Caller code
owns cleanup and atomic publication; this library does not store snapshots.
Callback failure, cancellation, malformed/truncated/extra output and resource
failures kill and reap the child and return a generic error without diagnostics.
There are no application-imposed native execution deadlines, stdout/prefix/tuple/row
caps or saved-file size caps. Sequential callback chunks of at most 50 rows are
backpressure, not capacity limits. Hledger rendering and Node parsing can still use
substantial memory; no journal-capacity guarantee is implied. There are no separately
exported Overview or account-register execution APIs.
## One hledger process boundary
`process.ts` is the only production module that launches hledger. It owns
argument-array execution, SIGKILL cancellation, backpressure, EOF/exit validation
and cleanup. There is no shell command builder,
provider registry or configurable executable. Native stderr and nested process
errors never become public diagnostics.
Its private `readHledger` buffers responses with fatal UTF-8 decoding. The prepared
batch consumes byte chunks through `consumeHledger`; both use the same process
lifecycle, without application time/output caps. There is no separate buffered
execution implementation. Consumers cannot return a successful partial read before
EOF or successful process exit. Cancellation kills native work immediately but waits
for active caller-owned writes before returning, so cleanup cannot race a late write.
Financial operations are called by worker jobs, never web handlers or page loaders.
Version checking at worker startup, the version-only call during preparation and
the explicit `worker:inspect` command are operational checks, not extra financial
batches. Overview and account history currently remain coordinated by
one preparation job. The existing live cost-balance query remains separate; deciding
which curated or user-authored exploratory reports to offer is future product work,
not a reason to expose a generic CLI API now.
`amounts.ts` owns the exact numeric codec and common reported-amount shape/projection.
Projection preserves order, duplicate commodities and exact quantity strings. Query
input validation and cost-annotation restrictions stay in the cost-balance operation.
`transactions/account-rows.ts` owns the private hledger register tuple;
`organizeAccountRow` produces `PreparedAccountRow` with account-relative facts
and canonical posting selection; `transactions/transaction.ts` owns full
transaction/posting metadata and recursive exact amount projection.
Tests cover the pure financial plan/decoding/organization and the actual combined
producer. Independent native fixture oracles remain test-only; exports serve actual app
consumers, not test-only execution APIs. Process
safety tests live next to `process.ts`. The finite range/side/balance outputs extend
the framed financial prefix; register exports and callback backpressure remain unchanged.
Development uses source exports and the root Node loader; production uses compiled `dist/src`. `pnpm build`, `check`, `lint`, and `test` operate on this package.
The caller owns staging, publication and cleanup. Browsing reads saved reports;
it does not run hledger.
packages/hledger/fixtures/checking.fixture.ts
-60
View File
@@ -1,60 +0,0 @@
export const checkingJournal = `; Fictional checking register, including investment cash movements.
; As of 2026-04-30: eight checking rows, ending 8520.00 USD.
; Tests preserve native register output, not portfolio valuations or gains.
commodity USD 1.00
commodity VTI 1.0000 ; lots: FIFO
commodity BND 1.0000 ; lots: FIFO
account assets:bank:checking ; type: C
account assets:investments:brokerage ; type: A
account assets:investments:retirement ; type: A
account equity:opening ; type: E
account income:salary ; type: R
account income:capital gains ; type: G
account expenses:living ; type: X
P 2026-04-01 VTI 100.00 USD
P 2026-04-01 BND 80.00 USD
P 2026-04-15 VTI 150.00 USD
P 2026-04-15 BND 81.00 USD
P 2026-04-30 VTI 160.00 USD
P 2026-04-30 BND 82.00 USD
2026-04-01 Opening cash
assets:bank:checking 10000.00 USD
equity:opening -10000.00 USD
2026-04-02 Employer | Salary
assets:bank:checking 1000.00 USD
income:salary -1000.00 USD
2026-04-03 Local Market | Living expenses
expenses:living 100.00 USD
assets:bank:checking -100.00 USD
2026-04-04 Brokerage | Buy VTI and BND
assets:investments:brokerage 10.00 VTI @ 100.00 USD
assets:investments:brokerage 5.00 BND @ 80.00 USD
assets:bank:checking -1400.00 USD
2026-04-05 Brokerage | Second VTI purchase
assets:investments:brokerage 10.00 VTI @ 120.00 USD
assets:bank:checking -1200.00 USD
2026-04-06 Transfer | Fund brokerage cash
assets:investments:brokerage 200.00 USD
assets:bank:checking -200.00 USD
2026-04-07 Retirement | Buy VTI
assets:investments:retirement 3.00 VTI @ 110.00 USD
assets:bank:checking -330.00 USD
2026-04-15 Brokerage | Sell five VTI
assets:investments:brokerage -5.00 VTI @ 150.00 USD
assets:bank:checking 750.00 USD
2026-04-16 Transfer | Move two VTI to retirement
assets:investments:brokerage -2.00 VTI
assets:investments:retirement 2.00 VTI
`;
packages/hledger/fixtures/register.fixture.ts
-57
View File
@@ -1,57 +0,0 @@
export const registerJournal = `; Native aregister semantics, not an application-owned financial model.
commodity USD 1.00
commodity EUR 1.000
commodity BTC 1.0000000000
commodity STOCK 1.
2024-12-31 Opening cash
assets:cash 10.00 USD
equity:opening
2025-01-05 Two selected posting dates
assets:cash:a 2.00 USD ; date:2025-01-05
assets:cash:b 3.00 USD ; date:2025-01-06
income:other -5.00 USD
2025-01-08 Internal subtree transfer
assets:cash:a -1.00 USD
assets:cash:b 1.00 USD
2025-02-10 Future original date
assets:cash 7.00 USD ; date:2025-01-12
income:other -7.00 USD
2025-01-20 * Split counterparts
assets:cash -3.00 USD
expenses:one 1.00 USD
expenses:two 2.00 USD
2025-01-25 Future selected posting with in-range counterpart
assets:cash 4.00 USD ; date:2025-02-01
income:other -4.00 USD
2025-01-10 Literal account and neighbours
assets:a.b+(c)[d]{e}^$|q?\\z* 1.00 USD
assets:a.b+(c)[d]{e}^$|q?\\z*:child 2.00 USD
assets:a.b+(c)[d]{e}^$|q?\\z*sibling 4.00 USD
equity:opening -7.00 USD
2025-01-15 Mixed exact quantities
assets:mix 9007199254740993.01 USD
assets:mix 2.500 EUR
assets:mix -0.1234567890 BTC
equity:opening
2025-01-16 Costed security
assets:mix 2 STOCK @ 3.00 USD
assets:funding -6.00 USD
; Lowercase appears first in the journal, but native account selection picks Case.
2025-01-11 Lowercase case-colliding account
assets:case 2.00 USD
equity:opening -2.00 USD
2025-01-12 Uppercase case-colliding account
assets:Case 1.00 USD
equity:opening -1.00 USD
`;
packages/hledger/src/accounts/balances.integration.test.ts
+1 -1
View File
@@ -6,7 +6,7 @@ import { promisify } from 'node:util';
import { expect, it } from 'vitest';
import { lotsJournal } from '../../../../apps/worker/fixtures/journal.js';
import { formatQuantity, parseExactJson, type Amount } from '../amounts.js';
import { prepareReports } from '../prepare-reports.js';
import { prepareReports } from '../prepare-reports/run.js';
const exec = promisify(execFile);
packages/hledger/src/index.ts
+2 -2
View File
@@ -15,8 +15,8 @@ export type { CostBalanceReport } from './queries/cost-balance.js';
export type { ReportedAmount } from './amounts.js';
export { runHoldings } from './holdings/report.js';
export type { HoldingPosition, HoldingsAccount, HoldingsReport } from './holdings/report.js';
export { prepareReports } from './prepare-reports.js';
export type { PreparedAccountSummary } from './prepare-reports.js';
export { prepareReports } from './prepare-reports/run.js';
export type { PreparedAccountSummary } from './prepare-reports/batch.js';
export type { PreparedBalance } from './accounts/balances.js';
export type { PreparedAccountRow } from './transactions/account-rows.js';
export type {
packages/hledger/src/overview/transaction-counts.integration.test.ts
+1 -1
View File
@@ -3,7 +3,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterAll, beforeAll, expect, it } from 'vitest';
import { transactionCountsJournal } from '../../fixtures/transaction-counts.fixture.js';
import { prepareReports } from '../prepare-reports.js';
import { prepareReports } from '../prepare-reports/run.js';
let directory: string;
let journal: string;
packages/hledger/src/prepare-reports.integration.test.ts
-594
View File
@@ -1,594 +0,0 @@
import { execFile } from 'node:child_process';
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { promisify } from 'node:util';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { lotsJournal } from '../../../apps/worker/fixtures/journal.js';
import { cashJournal } from '../fixtures/cash.fixture.js';
import { registerJournal } from '../fixtures/register.fixture.js';
import { formatQuantity, parseExactJson, type Amount } from './amounts.js';
import type { HledgerAccountRegisterRow, PreparedAccountRow } from './transactions/account-rows.js';
import type { DetailedAmount, NativeTransaction } from './transactions/transaction.js';
import { prepareReports } from './prepare-reports.js';
const exec = promisify(execFile);
const amounts = (bag: Amount[]) =>
bag.map((amount) => ({
commodity: amount.acommodity,
quantity: formatQuantity(amount.aquantity)
}));
const quantities = (bag: PreparedAccountRow['change']) =>
bag.map(({ commodity, quantity }) => ({ commodity, quantity }));
describe('prepared account reports against independent native registers', () => {
let directory: string;
let journal: string;
let fixture: string;
const original = registerJournal;
let input: string;
let reports: Awaited<ReturnType<typeof prepareReports>>;
const accounts = new Map<string, PreparedAccountRow[]>();
beforeAll(async () => {
directory = await mkdtemp(join(tmpdir(), 'prepared-account-semantics-'));
journal = join(directory, 'main.journal');
fixture = join(directory, 'register.journal');
await writeFile(fixture, original);
// The old raw-register fixture did not require declarations. Qualify an owned
// copy for the application's strict producer, without changing any postings.
const { stdout: declarations } = await exec('hledger', [
'--no-conf',
'--lang=en',
'-f',
fixture,
'accounts',
'--flat',
'--directives'
]);
input = `${declarations}account assets:unused\n\n${original}`;
await writeFile(journal, input);
reports = await prepareReports(journal, {
async writeRows(name, rows) {
const previous = accounts.get(name) ?? [];
accounts.set(name, [...previous, ...rows]);
}
});
});
afterAll(async () => {
if (directory) await rm(directory, { recursive: true, force: true });
});
it('matches full-history native rows for every addressable account without mutating sources', async () => {
for (const account of reports.accounts) {
if (account.access !== 'ready') continue;
const pattern = `^${account.name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`;
const { stdout } = await exec('hledger', [
'--no-conf',
'--lang=en',
'--strict',
'-f',
journal,
'aregister',
'--historical',
'--empty',
'-Ojson',
pattern
]);
const native = parseExactJson(stdout) as HledgerAccountRegisterRow[];
expect(
accounts
.get(account.name)!
.map(
({
accountDate,
transaction: { description, status },
isSplit,
otherAccounts,
change,
balance
}) => ({
accountDate,
description,
status,
isSplit,
otherAccounts,
change: quantities(change),
balance: quantities(balance)
})
)
).toEqual(
native.map(([, contextual, isSplit, otherAccounts, change, balance]) => ({
accountDate: contextual.tdate,
description: contextual.tdescription,
status: contextual.tstatus,
isSplit,
otherAccounts,
change: amounts(change),
balance: amounts(balance)
}))
);
expect(account.rowCount).toBe(native.length);
}
expect(await readFile(fixture, 'utf8')).toBe(original);
expect(await readFile(journal, 'utf8')).toBe(input);
});
it('retains contextual dates, native subtree zero-change activity, status and attached balances', () => {
const rows = accounts.get('assets:cash')!;
expect(
rows.map(({ accountDate, transaction: { description }, change, balance }) => [
accountDate,
description,
quantities(change),
quantities(balance)
])
).toEqual([
[
'2024-12-31',
'Opening cash',
[{ commodity: 'USD', quantity: '10.00' }],
[{ commodity: 'USD', quantity: '10.00' }]
],
[
'2025-01-05',
'Two selected posting dates',
[{ commodity: 'USD', quantity: '5.00' }],
[{ commodity: 'USD', quantity: '15.00' }]
],
[
'2025-01-08',
'Internal subtree transfer',
[{ commodity: 'USD', quantity: '0.00' }],
[{ commodity: 'USD', quantity: '15.00' }]
],
[
'2025-01-12',
'Future original date',
[{ commodity: 'USD', quantity: '7.00' }],
[{ commodity: 'USD', quantity: '22.00' }]
],
[
'2025-01-20',
'Split counterparts',
[{ commodity: 'USD', quantity: '-3.00' }],
[{ commodity: 'USD', quantity: '19.00' }]
],
[
'2025-02-01',
'Future selected posting with in-range counterpart',
[{ commodity: 'USD', quantity: '4.00' }],
[{ commodity: 'USD', quantity: '23.00' }]
]
]);
expect(rows[2].isSplit).toBe(false);
expect(rows[2].otherAccounts).toHaveLength(2);
expect(rows[4]).toMatchObject({ isSplit: true, transaction: { status: 'Cleared' } });
});
it('treats regex punctuation literally and includes children but not prefix siblings', () => {
const rows = accounts.get('assets:a.b+(c)[d]{e}^$|q?\\z*')!;
expect(rows).toHaveLength(1);
expect(quantities(rows[0].change)).toEqual([{ commodity: 'USD', quantity: '3.00' }]);
});
it('retains mixed units, unsafe-integer precision, native fractions and unconverted securities', () => {
const rows = accounts.get('assets:mix')!;
expect(rows).toHaveLength(2);
expect(quantities(rows[0].change)).toEqual([
{ commodity: 'BTC', quantity: '-0.1234567890' },
{ commodity: 'EUR', quantity: '2.500' },
{ commodity: 'USD', quantity: '9007199254740993.01' }
]);
expect(quantities(rows[1].change)).toEqual([{ commodity: 'STOCK', quantity: '2' }]);
});
it('distinguishes genuinely empty accounts from native-selector ambiguity', () => {
expect(accounts.get('assets:unused')).toEqual([]);
expect(reports.accounts.find(({ name }) => name === 'assets:unused')).toEqual({
name: 'assets:unused',
access: 'ready',
rowCount: 0,
summary: {
status: 'ready',
balances: [],
valuation: { amounts: [], unconverted: [] }
}
});
for (const name of ['assets:Case', 'assets:case']) {
expect(reports.accounts.find((account) => account.name === name)).toEqual({
name,
access: 'ambiguous',
rowCount: null,
summary: { status: 'ambiguous' }
});
expect(accounts.has(name)).toBe(false);
}
});
});
// cashJournal is fictional and declared/strict on its own: salary,
// payments, transfers, 75/25 and repeated-category splits, metadata and foreign units.
// This independent native oracle does not invoke transaction/account-row organizers;
// mapper unit tests separately cover recursive annotations and unsafe-integer quantities.
describe('fictional cash metadata against independent register/original/transacted native commands', () => {
let directory: string;
let cash: string;
const rowsByAccount = new Map<string, PreparedAccountRow[]>();
let prepared: Awaited<ReturnType<typeof prepareReports>>;
const source = cashJournal;
beforeAll(async () => {
directory = await mkdtemp(join(tmpdir(), 'prepared-cash-semantics-'));
cash = join(directory, 'cash.journal');
await writeFile(cash, source);
prepared = await prepareReports(cash, {
async writeRows(name, rows) {
rowsByAccount.set(name, [...(rowsByAccount.get(name) ?? []), ...rows]);
}
});
});
afterAll(async () => {
if (directory) await rm(directory, { recursive: true, force: true });
});
const bankRows = () => rowsByAccount.get('assets:Cash')!;
const find = (description: string) =>
bankRows().find((row) => row.transaction.description === description)!;
// hledger 1.99.5 AccountTransactionsReport.hs: accountTransactionsReportItem
// retains every posting and changes only contextual tdate for this unfiltered command.
// Its encoder exports poriginal as null; this is not an original-posting archive.
it('retains every exported transaction/posting field and amount style, with only tdate differing in context', async () => {
for (const account of prepared.accounts) {
if (account.access !== 'ready') continue;
const { stdout } = await exec('hledger', [
'--no-conf',
'--lang=en',
'--strict',
'-f',
cash,
'aregister',
'--historical',
'--empty',
'-Ojson',
`^${account.name}$`
]);
const native = parseExactJson(stdout) as HledgerAccountRegisterRow[];
const query = `acct:^${account.name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}(:|$)`;
const print = async (valued: boolean) => {
const { stdout } = await exec('hledger', [
'--no-conf',
'--lang=en',
'--strict',
'-f',
cash,
'print',
'--round=none',
...(valued ? ['--value=transacted'] : []),
'-Ojson',
query
]);
return new Map(
(parseExactJson(stdout) as NativeTransaction[]).map((transaction) => [
transaction.tindex,
transaction
])
);
};
const originals = await print(false);
const transacted = await print(true);
const rows = rowsByAccount.get(account.name)!;
expect(rows).toHaveLength(native.length);
for (const [
index,
[registerOriginal, context, isSplit, otherAccounts, change, balance]
] of native.entries()) {
expect({ ...context, tdate: registerOriginal.tdate }).toEqual(registerOriginal);
const original = originals.get(registerOriginal.tindex)!;
const valued = transacted.get(registerOriginal.tindex)!;
const row = rows[index];
expect(row.accountDate).toBe(context.tdate);
expect(row.transaction.description).toBe(context.tdescription);
expect(row.transaction.status).toBe(context.tstatus);
expect(row.isSplit).toBe(isSplit);
expect(row.otherAccounts).toEqual(otherAccounts);
expect(row.change).toEqual(change.map(cashAmount));
expect(row.balance).toEqual(balance.map(cashAmount));
// hledger 1.99.5 Transaction.hs: no separator leaves both parts untrimmed;
// otherwise split at the first | and strip each part, independently of comments.
const separator = original.tdescription.indexOf('|');
let payee = original.tdescription;
let note = original.tdescription;
if (separator >= 0) {
payee = original.tdescription.slice(0, separator).trim();
note = original.tdescription.slice(separator + 1).trim();
}
const postings = original.tpostings.map((p, postingIndex) => {
// hledger 1.99.5 Posting.hs: only Unmarked inherits the header status.
const effectiveStatus = p.pstatus === 'Unmarked' ? original.tstatus : p.pstatus;
const assertion = p.pbalanceassertion;
const balanceAssertion = assertion && {
amount: cashAmount(assertion.baamount),
total: assertion.batotal,
inclusive: assertion.bainclusive,
sourcePosition: {
file: assertion.baposition.sourceName,
line: assertion.baposition.sourceLine,
column: assertion.baposition.sourceColumn
}
};
return {
account: p.paccount,
amounts: p.pamount.map(cashAmount),
transactedAmounts: amounts(valued.tpostings[postingIndex].pamount),
date: p.pdate,
secondaryDate: p.pdate2,
status: p.pstatus,
effectiveStatus,
comment: p.pcomment,
tags: p.ptags,
kind: p.preal,
transactionIndex: p.ptransaction_,
balanceAssertion
};
});
expect(row.transaction).toEqual({
index: original.tindex,
date: original.tdate,
secondaryDate: original.tdate2,
status: original.tstatus,
code: original.tcode,
description: original.tdescription,
payee,
note,
comment: original.tcomment,
precedingComment: original.tprecedingcomment,
tags: original.ttags,
sourcePositions: original.tsourcepos.map((p) => ({
file: p.sourceName,
line: p.sourceLine,
column: p.sourceColumn
})),
postings
});
for (const posting of original.tpostings) expect(posting.poriginal).toBeNull();
expect(JSON.parse(JSON.stringify(row))).toEqual(row);
}
}
expect(await readFile(cash, 'utf8')).toBe(source);
});
it('preserves hand-checked salary, payment, transfer, 75/25 split and repeated counterpart detail', () => {
expect(bankRows().map((row) => [row.change[0].quantity, row.balance[0].quantity])).toEqual([
['1000.00', '1000.00'],
['-100.00', '900.00'],
['-50.00', '850.00'],
['-100.00', '750.00'],
['-30.00', '720.00'],
['-20.00', '700.00'],
['0.00', '700.00'],
['5.00', '705.00'],
['6.00', '711.00'],
['-10.00', '701.00']
]);
const salary = find('Example Employer | January salary');
expect(salary.transaction).toMatchObject({
index: 1,
date: '2026-01-02',
secondaryDate: '2026-01-03',
code: 'PAY-1',
status: 'Cleared',
payee: 'Example Employer',
note: 'January salary',
comment: '\nPayroll memo\nproject: alpha, project: beta\n',
precedingComment: '; Fictional payroll documentation\n',
tags: [
['project', 'alpha'],
['project', 'beta']
]
});
expect(salary.transaction.postings[0]).toMatchObject({
comment: 'Bank deposit memo\n',
status: 'Unmarked',
effectiveStatus: 'Cleared',
tags: [
['type', 'A'],
['bank', 'fictional']
]
});
const payment = find('Example Shop | Supplies | office');
expect(payment.transaction.postings[0]).toMatchObject({
status: 'Cleared',
effectiveStatus: 'Cleared',
comment: 'Card posting memo\n',
balanceAssertion: {
total: true,
inclusive: true,
amount: { commodity: 'USD', quantity: '900.00' }
}
});
expect(payment.transaction.status).toBe('Pending');
expect(find('Transfer to savings').otherAccounts).toEqual(['assets:savings']);
const split = find('Example Market | Weekly split');
expect(split.accountPostingIndexes).toEqual([0]);
expect(split.isSplit).toBe(true);
expect(
split.transaction.postings.map((p) => [
p.account,
p.amounts[0].quantity,
p.comment,
p.effectiveStatus
])
).toEqual([
['assets:Cash', '-100.00', 'Bank split memo\n', 'Pending'],
['expenses:food', '75.00', 'Groceries memo\n', 'Cleared'],
['expenses:office', '25.00', 'Stationery memo\n', 'Cleared']
]);
const repeated = find('Repeated category');
expect(repeated.isSplit).toBe(false);
expect(repeated.otherAccounts).toEqual(['expenses:food', 'expenses:food']);
expect(repeated.transaction.postings.slice(1).map((p) => p.amounts[0].quantity)).toEqual([
'10.00',
'20.00'
]);
});
// hledger 1.99.5 AccountName.hs: accountNameToAccountRegex uses ^escapedName(:|$)
// case-sensitively after first-match identity qualification, including virtual postings.
// Native summarisePostingAccounts suppresses virtual counterparts when real ones exist;
// internal transfers list selected accounts themselves, so OtherAccounts is no complement.
// These fixtures also demonstrate mixed Cleared/Pending postings under an Unmarked header.
it('uses canonical case-sensitive parent matching, not complement of OtherAccounts or guessed notes/status', () => {
expect(prepared.accounts.find((a) => a.name === 'assets:Cash')?.access).toBe('ready');
const row = find('Parent case and prefix neighbours');
expect(row.accountPostingIndexes).toEqual([0, 1]);
expect(row.transaction.postings.map((p) => p.account)).toEqual([
'assets:Cash',
'assets:Cash:Child',
'assets:cash:child',
'assets:Cashbox',
'equity:opening',
'memo:budget'
]);
expect(row.otherAccounts).toEqual(['assets:cash:child', 'assets:Cashbox', 'equity:opening']);
expect(row.transaction.postings[5]).toMatchObject({
kind: 'VirtualPosting',
comment: 'Virtual counterpart memo\n'
});
const internal = find('Internal subtree transfer');
expect(internal.otherAccounts).toEqual(['assets:Cash', 'assets:Cash:wallet']);
expect(internal.accountPostingIndexes).toEqual([0, 1]);
expect(internal.transaction.status).toBe('Unmarked');
expect(
internal.accountPostingIndexes.map(
(index) => internal.transaction.postings[index].effectiveStatus
)
).toEqual(['Cleared', 'Pending']);
const virtual = find('Virtual selected child');
expect(virtual.accountPostingIndexes).toEqual([0]);
expect(virtual.transaction.postings[0].kind).toBe('VirtualPosting');
});
it('retains original/secondary/posting dates and exported counterpart units, not stripped costs', () => {
const dated = find('Dated payment');
expect(dated.accountDate).toBe('2026-01-08');
expect(dated.transaction).toMatchObject({ date: '2026-02-01', secondaryDate: '2026-02-03' });
expect(dated.transaction.postings[0]).toMatchObject({
date: '2026-01-08',
secondaryDate: '2026-01-09'
});
const foreign = find('Foreign currency payment');
expect(quantities(foreign.change)).toEqual([{ commodity: 'USD', quantity: '-10.00' }]);
expect(foreign.transaction.postings[1].amounts[0]).toMatchObject({
commodity: 'EUR',
quantity: '8.00',
cost: { kind: 'UnitCost', amount: { commodity: 'USD', quantity: '1.25' } },
costBasis: null
});
expect(foreign.transaction.postings[1].transactedAmounts).toEqual([
{ commodity: 'USD', quantity: '10' }
]);
});
});
type CashNativeAmount = HledgerAccountRegisterRow[0]['tpostings'][number]['pamount'][number];
function cashAmount(amount: CashNativeAmount): DetailedAmount {
// Independent recursive native oracle: original print annotations are not
// register's stripped costs and transacted values are not cost basis.
const style = amount.astyle;
const basis = amount.acostbasis;
return {
commodity: amount.acommodity,
quantity: formatQuantity(amount.aquantity),
cost: amount.acost
? { kind: amount.acost.tag, amount: cashAmount(amount.acost.contents) }
: null,
costBasis: basis
? {
amount: basis.cbCost === null ? null : cashAmount(basis.cbCost),
date: basis.cbDate,
label: basis.cbLabel
}
: null,
style: style
? {
commoditySide: style.ascommodityside,
commoditySpaced: style.ascommodityspaced,
decimalMark: style.asdecimalmark,
digitGroups: style.asdigitgroups,
precision: style.asprecision,
rounding: style.asrounding
}
: null
};
}
it('preserves native transaction prices, basis and transacted value independently on the owned lots fixture', async () => {
const directory = await mkdtemp(join(tmpdir(), 'prepared-lots-semantics-'));
const lots = join(directory, 'lots.journal');
try {
await writeFile(lots, lotsJournal);
const rowsByAccount = new Map<string, PreparedAccountRow[]>();
await prepareReports(lots, {
async writeRows(name, rows) {
rowsByAccount.set(name, [...(rowsByAccount.get(name) ?? []), ...rows]);
}
});
const print = async (valued: boolean) => {
const { stdout } = await exec('hledger', [
'--no-conf',
'--lang=en',
'--strict',
'-f',
lots,
'print',
'--round=none',
...(valued ? ['--value=transacted'] : []),
'-Ojson'
]);
return new Map(
(parseExactJson(stdout) as NativeTransaction[]).map((transaction) => [
transaction.tindex,
transaction
])
);
};
const originals = await print(false);
const transacted = await print(true);
for (const rows of rowsByAccount.values()) {
for (const { transaction } of rows) {
const original = originals.get(transaction.index)!;
const valued = transacted.get(transaction.index)!;
expect(new Set(original.tpostings.map((p) => p.preal))).toEqual(new Set(['RealPosting']));
expect(transaction.postings.map((p) => p.kind)).toEqual(
original.tpostings.map((p) => p.preal)
);
expect(transaction.postings.map((p) => p.amounts)).toEqual(
original.tpostings.map((p) => p.pamount.map(cashAmount))
);
expect(transaction.postings.map((p) => p.transactedAmounts)).toEqual(
valued.tpostings.map((p) => amounts(p.pamount))
);
}
}
const sale = rowsByAccount
.get('assets:investments:brokerage')!
.find((row) => row.transaction.description === 'Brokerage | Sell five VTI')!;
expect(sale.investment.securities[0].action).toBe('sell');
expect(sale.investment.issues).not.toContain('virtual-postings');
const disposal = sale.transaction.postings.find((p) =>
p.amounts.some((a) => a.commodity === 'VTI')
)!;
expect(disposal.amounts[0].cost?.amount.quantity).toMatch(/^150(?:\.0+)?$/);
expect(disposal.amounts[0].costBasis?.amount?.quantity).toMatch(/^100(?:\.0+)?$/);
expect(disposal.transactedAmounts).toEqual([
{ commodity: 'USD', quantity: expect.stringMatching(/^-750(?:\.0+)?$/) }
]);
const gain = sale.transaction.postings.find((p) => p.account === 'income:capital gains')!;
expect(gain.amounts[0].quantity).toMatch(/^-250(?:\.0+)?$/);
expect(await readFile(lots, 'utf8')).toBe(lotsJournal);
} finally {
await rm(directory, { recursive: true, force: true });
}
});
packages/hledger/src/prepare-reports.test.ts
-890
View File
@@ -1,890 +0,0 @@
import { execFile, type ChildProcessWithoutNullStreams } from 'node:child_process';
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { promisify } from 'node:util';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { lotsJournal } from '../../../apps/worker/fixtures/journal.js';
import { cashJournal } from '../fixtures/cash.fixture.js';
import { checkingJournal } from '../fixtures/checking.fixture.js';
import { formatQuantity, parseExactJson, type Amount } from './amounts.js';
import { prepareReports } from './prepare-reports.js';
import type { HledgerAccountRegisterRow, PreparedAccountRow } from './transactions/account-rows.js';
import { transactionFixture } from './transactions/transaction.fixture.js';
const physical = vi.hoisted(() => ({ spawn: vi.fn() }));
vi.mock('node:child_process', async (importOriginal) => {
const actual = await importOriginal<typeof import('node:child_process')>();
return { ...actual, spawn: physical.spawn };
});
const exec = promisify(execFile);
const actual = await vi.importActual<typeof import('node:child_process')>('node:child_process');
const failure = 'Prepared transactions could not be read. Check the journal.';
const directories: string[] = [];
beforeEach(() => {
physical.spawn.mockReset().mockImplementation(actual.spawn);
});
afterEach(async () => {
vi.restoreAllMocks();
await Promise.all(
directories.splice(0).map((path) => rm(path, { recursive: true, force: true }))
);
});
async function journal(text: string): Promise<string> {
const dir = await mkdtemp(join(tmpdir(), 'prepared-reports-'));
directories.push(dir);
const path = join(dir, 'fictional.journal');
await writeFile(path, text);
return path;
}
function collector() {
const rowsByName = new Map<string, PreparedAccountRow[]>();
return {
rowsByName,
writeRows: vi.fn(async (name: string, rows: readonly PreparedAccountRow[]) => {
const account = rowsByName.get(name) ?? [];
account.push(...rows);
rowsByName.set(name, account);
})
};
}
const money = (bag: Amount[]) =>
bag.map((a) => ({ commodity: a.acommodity, quantity: formatQuantity(a.aquantity) }));
const quantities = (bag: PreparedAccountRow['change']) =>
bag.map(({ commodity, quantity }) => ({ commodity, quantity }));
const financialRow = ({
accountDate,
transaction: { description, status },
otherAccounts,
isSplit,
change,
balance
}: PreparedAccountRow) => ({
accountDate,
description,
status,
otherAccounts,
isSplit,
change: quantities(change),
balance: quantities(balance)
});
function oracleRows(json: string) {
return (parseExactJson(json) as HledgerAccountRegisterRow[]).map(
([, t, isSplit, otherAccounts, change, balance]) => ({
accountDate: t.tdate,
description: t.tdescription,
status: t.tstatus,
isSplit,
otherAccounts,
change: money(change),
balance: money(balance)
})
);
}
async function oracle(path: string, name: string) {
return oracleRows(
(
await exec('hledger', [
'--no-conf',
'--lang=en',
'--strict',
'-f',
path,
'aregister',
'--historical',
'--empty',
'-Ojson',
`^${name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`
])
).stdout
);
}
it('uses one version-only process and two journal reads for the combined financial view and native JSON rows for exactly seven actual accounts', async () => {
const checking = await journal(checkingJournal);
const collected = collector();
const prepared = await prepareReports(checking, collected);
expect(physical.spawn).toHaveBeenCalledTimes(3);
expect(physical.spawn.mock.calls[0][1]).toEqual(['--no-conf', '--lang=en', '--version']);
const discoveryArgs = physical.spawn.mock.calls[1][1] as string[];
expect(discoveryArgs.slice(0, 9)).toEqual([
'--no-conf',
'--lang=en',
'--strict',
'-f',
checking,
'run',
'--',
'stats',
'--'
]);
expect(discoveryArgs).toContain('--directives');
const args = physical.spawn.mock.calls[2][1] as string[];
expect(args.slice(0, 8)).toEqual([
'--no-conf',
'--lang=en',
'--strict',
'-f',
checking,
'run',
'--',
'stats'
]);
expect(args.filter((a) => a === 'stats')).toHaveLength(5);
expect(args.filter((a) => a === 'balancesheet')).toHaveLength(12);
expect(args.filter((a) => a === 'incomestatement')).toHaveLength(4);
expect(args.filter((a) => a === 'balance')).toHaveLength(2);
expect(args.filter((a) => a === 'aregister')).toHaveLength(7);
expect(args.filter((a) => a === 'print')).toHaveLength(14);
for (const type of ['A', 'C', 'L', 'R', 'X', 'G']) expect(args).toContain(`type:${type}`);
expect(args).toContain('tag:lots');
for (let i = 0; i < args.length; i++) {
if (args[i] !== 'print') continue;
expect(args[i + 1]).toBe('--round=none');
const valued = args[i + 2] === '--value=transacted';
expect(args[i + (valued ? 3 : 2)]).toBe('-Ojson');
expect(args[i + (valued ? 4 : 3)]).toMatch(/^acct:\^.*\(:\|\$\)$/);
}
expect(args.lastIndexOf('--find')).toBeLessThan(args.indexOf('aregister'));
for (let i = 0; i < args.length; i++) {
if (args[i] !== 'aregister') continue;
expect(args.slice(i, i + 5)).toEqual([
'aregister',
'--historical',
'--empty',
'-Ojson',
expect.stringMatching(/^\^.*\$$/)
]);
}
expect(args).not.toContain('--today');
expect(prepared.overview.asOf).toBe('2026-04-16');
expect(prepared.overview.hledgerVersion).toBe('1.99.5');
expect(prepared.overview.journalCoverage).toEqual({
transactionCount: 9,
firstDate: '2026-04-01',
lastDate: '2026-04-16'
});
expect(prepared.overview.incomeExpenses.months).toHaveLength(12);
expect(prepared.overview.incomeExpenses.months.at(-1)).toEqual({
begin: '2026-04-01',
endExclusive: '2026-05-01',
income: [{ commodity: 'USD', quantity: '1000.00' }],
expenses: [{ commodity: 'USD', quantity: '100.00' }]
});
expect(prepared.overview.netWorth?.points.at(-1)).toEqual({
date: '2026-04-16',
amounts: [{ commodity: 'USD', quantity: '11825.00' }],
unconverted: []
});
expect(Object.keys(prepared).sort()).toEqual(['accounts', 'overview']);
expect(prepared.accounts).toHaveLength(7);
expect(prepared.accounts.some((a) => a.name === 'assets')).toBe(false);
for (const account of prepared.accounts) {
expect(account.access).toBe('ready');
const rows = collected.rowsByName.get(account.name)!;
expect(rows.map(financialRow)).toEqual(await oracle(checking, account.name));
expect(account).toMatchObject({
name: account.name,
access: 'ready',
rowCount: rows.length,
summary: { status: 'ready' }
});
}
expect(JSON.parse(JSON.stringify(prepared))).toEqual(prepared);
});
it('delivers richer cash rows and both print exports through the same cached financial call', async () => {
const cash = await journal(cashJournal);
const collected = collector();
const prepared = await prepareReports(cash, collected);
expect(physical.spawn).toHaveBeenCalledTimes(3);
const args = physical.spawn.mock.calls[2][1] as string[];
expect(args.filter((arg) => arg === 'print')).toHaveLength(2 * prepared.accounts.length);
expect(args).not.toContain('--lots');
expect(args.some((arg) => arg.startsWith('--infer'))).toBe(false);
const rows = collected.rowsByName.get('assets:Cash')!;
expect(rows).toHaveLength(10);
expect(
rows.find((row) => row.transaction.description === 'Parent case and prefix neighbours')!
.accountPostingIndexes
).toEqual([0, 1]);
expect(rows[3].transaction.postings.map((posting) => posting.amounts[0].quantity)).toEqual([
'-100.00',
'75.00',
'25.00'
]);
});
it('preserves full native history, contextual dates, all exact mixed bags, statuses and cumulative balances across row slices', async () => {
const selected = 'assets:cash, 東京';
const counterpart = 'equity:opening, é';
const path = await journal(`commodity USD 1.00
commodity EUR 1.000
commodity 1.0000
account assets ; type: A
account ${selected}
account ${counterpart} ; type: E
account expenses:refund ; type: X
account expenses:split ; type: X
account income:unused gain ; type: G
account assets:unused
account assets:a.b+(c)[d]{e}^$|q?\\z*
2020-01-01 * Tiny
${selected} 0.0049 USD
${counterpart}
2020-01-02 ! Mixed
${selected} 9007199254740993.01 EUR
${selected} -0.00123 USD
${counterpart}
2020-01-03 Refund
expenses:refund -0.0049 USD
${selected}
2020-01-04 Split
${selected} -3 USD
expenses:refund 1 USD
expenses:split 2 USD
2020-01-05 Zero
${selected} 0 USD
${counterpart} 0 USD
2020-01-06 Unnamed
${selected} 0.0037
${counterpart}
2020-01-07 Literal regex
assets:a.b+(c)[d]{e}^$|q?\\z* 1 USD
${counterpart}
2036-01-01=2035-02-02 Primary future contextual past
${selected} 7 USD ; date:2031-03-04, date2:2032-04-05
${counterpart} -7 USD
${Array.from(
{ length: 105 },
(_, i) => `2021-01-01 Entry ${i}, λ
${selected} 0.0049 USD
${counterpart}
`
).join('\n')}`);
const collected = collector();
const prepared = await prepareReports(path, collected);
const account = prepared.accounts.find((a) => a.name === selected)!;
const rows = collected.rowsByName.get(account.name)!;
const expected = await oracle(path, selected);
expect(rows).toHaveLength(112);
expect(rows.map(financialRow)).toEqual(expected);
expect(quantities(rows.find((r) => r.transaction.description === 'Tiny')!.change)).toEqual([
{ commodity: 'USD', quantity: '0.0049' }
]);
expect(rows.find((r) => r.transaction.description === 'Mixed')!.change).toHaveLength(2);
expect(
quantities(rows.find((r) => r.transaction.description === 'Unnamed')!.change)
).toContainEqual({
commodity: '',
quantity: '0.0037'
});
expect(rows.find((r) => r.transaction.description === 'Split')!.isSplit).toBe(true);
expect(new Set(rows.map((r) => r.transaction.status))).toEqual(
new Set(['Cleared', 'Pending', 'Unmarked'])
);
expect(prepared.overview.asOf).toBe('2036-01-01');
expect(
rows.find((r) => r.transaction.description === 'Primary future contextual past')!.accountDate
).toBe('2031-03-04');
for (const name of ['assets:unused', 'income:unused gain']) {
const unused = prepared.accounts.find((a) => a.name === name)!;
expect(unused).toMatchObject({ access: 'ready', rowCount: 0 });
expect(collected.rowsByName.get(unused.name)).toEqual([]);
}
const parent = prepared.accounts.find((a) => a.name === 'assets')!;
expect(collected.rowsByName.get(parent.name)!.map(financialRow)).toEqual(
await oracle(path, 'assets')
);
const literal = prepared.accounts.find((a) => a.name.includes('a.b+'))!;
expect(collected.rowsByName.get(literal.name)!.map(financialRow)).toEqual(
await oracle(path, literal.name)
);
});
it('publishes the cached financial include view despite source edits during row writes', async () => {
const path = await journal(
'commodity USD 1.00\naccount assets:cash ; type: A\naccount equity:opening ; type: E\ninclude detail.journal\n'
);
const detail = join(path, '..', 'detail.journal');
const content = (quantity: string) =>
Array.from(
{ length: 105 },
(_, i) => `2026-04-16 Native ${i}\n assets:cash ${quantity} USD\n equity:opening\n`
).join('\n');
await writeFile(detail, content('1.00'));
const { overview: beforeOverview } = await prepareReports(path, {
writeRows: async () => {}
});
const beforeCash = await oracle(path, 'assets:cash');
const beforeEquity = await oracle(path, 'equity:opening');
const collected = collector();
let changed = false;
const result = await prepareReports(path, {
writeRows: async (name, rows) => {
await collected.writeRows(name, rows);
if (!changed) {
changed = true;
await writeFile(detail, content('2.00'));
}
}
});
expect(changed).toBe(true);
expect({ ...result.overview, preparedAt: '' }).toEqual({
...beforeOverview,
preparedAt: ''
});
for (const account of result.accounts) {
expect(collected.rowsByName.get(account.name)!.map(financialRow)).toEqual(
account.name === 'assets:cash' ? beforeCash : beforeEquity
);
}
const { overview: afterOverview } = await prepareReports(path, { writeRows: async () => {} });
expect(afterOverview.netWorth).not.toEqual(result.overview.netWorth);
});
it('preserves marker-named and Unicode account identities without treating names as paths', async () => {
const names = [
'__HLEDGER_PREPARED_REGISTERS__',
'__HLEDGER_PREPARED_FIND__',
'assets:../東京, é'
];
const path = await journal(`${names.map((name) => `account ${name}`).join('\n')}\n`);
const collected = collector();
const result = await prepareReports(path, collected);
expect(result.accounts.map((account) => account.name)).toEqual([...names].sort());
for (const account of result.accounts) {
expect(account).toMatchObject({ access: 'ready', rowCount: 0 });
expect(collected.rowsByName.get(account.name)).toEqual([]);
}
});
it('uses native identity qualification and disables BOTH colliding accounts without writing their reports', async () => {
const path = await journal(`commodity USD
account assets:case
account assets:Case
account equity
2026-01-01 Collision
assets:case 1 USD
assets:Case 2 USD
equity
`);
const collected = collector();
const prepared = await prepareReports(path, collected);
const collisions = prepared.accounts.filter((a) => a.name.startsWith('assets:'));
expect(collisions).toHaveLength(2);
for (const account of collisions) {
expect(account).toMatchObject({ access: 'ambiguous', rowCount: null });
expect(collected.rowsByName.has(account.name)).toBe(false);
}
});
it.each(['', 'account assets:unused\naccount equity:unused\n'])(
'prepares an empty journal with only actual declared accounts (%#)',
async (text) => {
const path = await journal(text);
const collected = collector();
const prepared = await prepareReports(path, collected);
expect(prepared.overview).toMatchObject({
asOf: null,
hledgerVersion: '1.99.5',
incomeExpenses: { months: [] },
netWorth: { currency: null, points: [] }
});
expect(prepared.accounts).toHaveLength(text ? 2 : 0);
for (const account of prepared.accounts) {
expect(account.rowCount).toBe(0);
expect(collected.rowsByName.get(account.name)).toEqual([]);
}
const args = physical.spawn.mock.calls[2][1] as string[];
expect(args).not.toContain('incomestatement');
expect(args).not.toContain('balancesheet');
}
);
// Fault injection uses real OS children, not a new public/test-only execution API.
const stats = 'Base currency : none\nLast txn : none\nTxns : 0\nTxns span : to\n';
const accounts = '\n__HLEDGER_PREPARED_ACCOUNTS__\naccount assets:cash\n';
const discovery = `${stats}${accounts}\n__HLEDGER_PREPARED_DISCOVERY_END__\n`;
const rolePrefix =
['ASSETS', 'CASH', 'LIABILITIES', 'INCOME', 'EXPENSES', 'GAINS']
.map(
(role) =>
`\n__HLEDGER_PREPARED_${role}__\n${role === 'ASSETS' || role === 'CASH' ? 'account assets:cash\n' : ''}`
)
.join('') + '\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n';
const financialPrefix = `${stats}${accounts}\n__HLEDGER_PREPARED_FIND__\naccount assets:cash\n${rolePrefix}\n__HLEDGER_PREPARED_REGISTERS__\n`;
const originalEnd = '\n\n__HLEDGER_PREPARED_ORIGINAL_END__\n';
const transactedEnd = '\n\n__HLEDGER_PREPARED_TRANSACTED_END__\n';
const end = '\n\n__HLEDGER_PREPARED_REGISTER_END__\n';
const nativeRowTransaction = {
...transactionFixture(),
tdate: '2031-01-01',
tdescription: 'λ, test'
};
const prefix = `${financialPrefix}[${JSON.stringify(nativeRowTransaction)}]${originalEnd}[${JSON.stringify(nativeRowTransaction)}]${transactedEnd}`;
const row = JSON.stringify([
nativeRowTransaction,
nativeRowTransaction,
false,
['other, 東京'],
[],
[]
]);
function rowsProtocol(count: number): string {
const transactions = Array.from({ length: count }, (_, index) => ({
...nativeRowTransaction,
tindex: index + 1,
tpostings: nativeRowTransaction.tpostings.map((posting) => ({
...posting,
ptransaction_: String(index + 1)
}))
}));
const prints = JSON.stringify(transactions);
const rows = JSON.stringify(
transactions.map((transaction) => [transaction, transaction, false, ['other, 東京'], [], []])
);
return `${financialPrefix}${prints}${originalEnd}${prints}${transactedEnd}${rows}${end}`;
}
function injectVersion(output = 'hledger 1.99.5, full SECRET banner\n', exit = 0) {
injectDiscovery(output, exit, false);
}
function injectDiscovery(output = discovery, exit = 0, version = true) {
if (version) injectVersion();
physical.spawn.mockImplementationOnce(() => {
const child = actual.spawn(process.execPath, [
'-e',
`process.stdin.pipe(process.stdout,{end:false});process.stdin.on('end',()=>process.stdout.write('',()=>process.exit(${exit})));`
]);
child.stdin.on('error', () => {});
child.stdin.end(output);
return child;
});
}
function inject(output: string | Buffer, exit = 0, hang = false): ChildProcessWithoutNullStreams {
injectDiscovery();
const child = actual.spawn(process.execPath, [
'-e',
`process.stdin.pipe(process.stdout,{end:false});process.stdin.on('end',()=>process.stdout.write('',()=>{${hang ? 'setInterval(()=>{},1000)' : `process.exit(${exit})`}}));`
]);
child.stdin.on('error', () => {}); // Early termination intentionally closes this test pipe.
physical.spawn.mockImplementationOnce(() => {
child.stdin.end(output);
return child;
});
return child;
}
async function rejected(output: string | Buffer, exit = 0) {
const child = inject(output, exit);
const writeRows = vi.fn(async () => {});
const error = await prepareReports('/private/SECRET', { writeRows }).catch(
(error: unknown) => error
);
expect(error).toBeInstanceOf(Error);
expect((error as Error).message).toBe(failure);
expect((error as Error).cause).toBeUndefined();
expect(child.exitCode !== null || child.signalCode !== null).toBe(true);
return writeRows;
}
describe('native protocol and lifecycle', () => {
it('timestamps the completed set after the final row callback, not the financial prefix', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-01-01T00:00:00Z'));
inject(`${prefix}[${row}]${end}`);
const result = await prepareReports('/private', {
writeRows: async () => {
vi.setSystemTime(new Date('2026-01-01T00:02:30Z'));
}
});
expect(result.overview.preparedAt).toBe('2026-01-01T00:02:30.000Z');
expect(result.overview.asOf).toBeNull();
expect(result.overview.hledgerVersion).toBe('1.99.5');
expect(JSON.stringify(result)).not.toContain('SECRET');
expect(result.accounts[0].rowCount).toBe(1);
} finally {
vi.useRealTimers();
}
});
it('preserves blank native amount bags, native row order and UTF-8 split across physical chunks', async () => {
injectDiscovery();
const bytes = Buffer.from(`${prefix}[${row}]${end}`);
physical.spawn.mockImplementationOnce(() =>
actual.spawn(process.execPath, [
'-e',
`const b=Buffer.from(${JSON.stringify(bytes.toString('base64'))},'base64');let i=0;function next(){if(i<b.length)process.stdout.write(b.subarray(i,i+=1),()=>setImmediate(next));}next();`
])
);
const collected = collector();
const result = await prepareReports('/private', collected);
expect(result.accounts[0].rowCount).toBe(1);
expect(financialRow(collected.rowsByName.get('assets:cash')![0])).toEqual({
accountDate: '2031-01-01',
description: 'λ, test',
status: 'Pending',
otherAccounts: ['other, 東京'],
isSplit: false,
change: [],
balance: []
});
});
it.each([
`${prefix}[${row}`, // truncated tuple array
`${prefix}[${row},]${end}`, // trailing comma
`${prefix}[${row} ${row}]${end}`, // missing comma
`${prefix}[{}]${end}`, // non-tuple
`${prefix}[${row.slice(0, -1)}]${end}`, // incomplete tuple
`${prefix}[${row}]\n__UNEXPECTED_MARKER__\n`,
`${prefix}[${row}]${end}extra`,
`${prefix}[${row}]${end}${end}`,
`${prefix}[${row.replace('[],[]', '[{"acommodity":"USD","aquantity":{"decimalMantissa":1e-20,"decimalPlaces":2}}],[]')}]${end}`,
Buffer.concat([Buffer.from(prefix), Buffer.from([0xff]), Buffer.from(`[]${end}`)])
])(
'rejects incomplete, malformed, extra, unsupported-number or invalid UTF-8 output (%#)',
async (output) => {
await rejected(output);
}
);
it('does not return a partial dataset after a late native nonzero exit, even after row writes', async () => {
const writes = await rejected(rowsProtocol(51), 9);
expect(writes).toHaveBeenCalledTimes(2);
});
it('applies writeRows backpressure and kills/reaps on callback failure', async () => {
const child = inject(rowsProtocol(105), 0, true);
let release: (() => void) | undefined;
const held = new Promise<void>((resolve) => {
release = resolve;
});
let entered: (() => void) | undefined;
const firstWrite = new Promise<void>((resolve) => {
entered = resolve;
});
const writeRows = vi.fn(async () => {
entered!();
await held;
throw new Error('SECRET /path 123.45 USD');
});
const result = prepareReports('/private', { writeRows });
await firstWrite;
await new Promise((resolve) => setTimeout(resolve, 30));
expect(writeRows).toHaveBeenCalledTimes(1);
release!();
await expect(result).rejects.toThrow(failure);
expect(child.signalCode).toBe('SIGKILL');
});
it('kills/reaps on abort but waits for the active row write before returning', async () => {
const child = inject(rowsProtocol(50), 0, true);
const childClosed = new Promise<void>((resolve) => {
child.once('close', () => resolve());
});
const controller = new AbortController();
let entered: (() => void) | undefined;
const firstWrite = new Promise<void>((resolve) => {
entered = resolve;
});
let release: (() => void) | undefined;
const held = new Promise<void>((resolve) => {
release = resolve;
});
let writeFinished = false;
let settled = false;
const result = prepareReports('/private', {
signal: controller.signal,
writeRows: async () => {
entered!();
await held;
writeFinished = true;
}
});
void result.then(
() => {
settled = true;
},
() => {
settled = true;
}
);
try {
await firstWrite;
controller.abort();
await childClosed;
expect(child.signalCode).toBe('SIGKILL');
expect(settled).toBe(false);
expect(writeFinished).toBe(false);
} finally {
release!();
}
await expect(result).rejects.toThrow(failure);
expect(writeFinished).toBe(true);
});
it('kills/reaps a blocked native read when aborted', async () => {
const child = inject(prefix.slice(0, -5), 0, true);
const controller = new AbortController();
const result = prepareReports('/private', {
signal: controller.signal,
writeRows: async () => {}
});
await vi.waitFor(() => expect(physical.spawn).toHaveBeenCalledTimes(3));
controller.abort();
await expect(result).rejects.toThrow(failure);
expect(child.signalCode).toBe('SIGKILL');
});
it('handles spawn failure without exposing a native executable path', async () => {
injectDiscovery();
let child: ReturnType<typeof actual.spawn>;
physical.spawn.mockImplementationOnce(() => {
child = actual.spawn('/nonexistent/prepared-reports-SECRET');
return child;
});
await expect(prepareReports('/private', { writeRows: async () => {} })).rejects.toThrow(
failure
);
expect(child!.exitCode).not.toBeNull();
});
it('owns the version-only process through EOF and cancellation before reading any journal', async () => {
let child: ChildProcessWithoutNullStreams;
physical.spawn.mockImplementationOnce(() => {
child = actual.spawn(process.execPath, [
'-e',
'process.stdout.write("hledger 1.99.5, SECRET banner\\n");setInterval(()=>{},1000);'
]);
return child;
});
const controller = new AbortController();
const result = prepareReports('/private', {
signal: controller.signal,
writeRows: async () => {}
});
await new Promise<void>((resolve) => child!.stdout.once('data', () => resolve()));
controller.abort();
await expect(result).rejects.toThrow(failure);
expect(physical.spawn).toHaveBeenCalledTimes(1);
expect(physical.spawn.mock.calls[0][1]).toEqual(['--no-conf', '--lang=en', '--version']);
expect(child!.signalCode).toBe('SIGKILL');
});
it('does not start a pre-aborted call', async () => {
await expect(
prepareReports('/private', { signal: AbortSignal.abort(), writeRows: async () => {} })
).rejects.toThrow(failure);
expect(physical.spawn).not.toHaveBeenCalled();
});
it('accepts long account and commodity identities without changing their spelling', async () => {
const name = `assets:${'x'.repeat(4097)}`;
const commodity = 'V'.repeat(4097);
injectDiscovery(discovery.replaceAll('assets:cash', name));
injectDiscovery(
`${financialPrefix}[]${originalEnd}[]${transactedEnd}[]${end}`
.replaceAll('assets:cash', name)
.replace(
'\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n',
`\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n${commodity}\n`
),
0,
false
);
const collected = collector();
const result = await prepareReports('/private', collected);
expect(result.accounts).toEqual([
{ name, access: 'ready', rowCount: 0, summary: { status: 'undated' } }
]);
expect(collected.rowsByName.get(name)).toEqual([]);
});
it('rejects a changed native account list instead of mixing views', async () => {
await rejected(`${prefix.replace('account assets:cash\n', 'account assets:other\n')}[]${end}`);
});
it('does not confuse stats-path markers or marker-named accounts with protocol boundaries', async () => {
injectDiscovery(`Main file : /private${accounts}part\n${discovery}`);
physical.spawn.mockImplementationOnce(() =>
actual.spawn(process.execPath, [
'-e',
`process.stdout.write(${JSON.stringify(`Main file : /private\n__HLEDGER_PREPARED_REGISTERS__\npart\n${prefix}[]${end}`)})`
])
);
const result = await prepareReports('/private', { writeRows: async () => {} });
expect(result.accounts[0].rowCount).toBe(0);
});
it('rejects a forged complete protocol in a stats path unless physical EOF also validates', async () => {
await rejected(`${prefix}[]${end}rest-of-path\n${prefix}[]${end}`);
});
it('rejects unsafe control identities and native discovery diagnostics generically', async () => {
injectDiscovery(discovery.replace('assets:cash', 'assets:\tSECRET'));
await expect(prepareReports('/private', { writeRows: async () => {} })).rejects.toThrow(
failure
);
expect(physical.spawn).toHaveBeenCalledTimes(2);
injectDiscovery('SECRET /private 1.23 USD', 9);
await expect(prepareReports('/private', { writeRows: async () => {} })).rejects.toThrow(
failure
);
});
});
describe('original/transacted native export protocol', () => {
const print = JSON.stringify(nativeRowTransaction);
const wire = (originals: string, transacted: string, register = `[${row}]`) =>
`${financialPrefix}${originals}${originalEnd}${transacted}${transactedEnd}${register}${end}`;
it.each([
wire(`[${print},${print}]`, `[${print}]`),
wire(`[${print}]`, `[${print},${print}]`),
wire(`[${print}]`, '[]'),
wire('[]', `[${print}]`),
wire('[]', '[]'),
wire(`[${print}]`, `[${print.replace('λ, test', 'changed')}]`),
wire(`[${print}]`, `[${print}]`, `[${row.replaceAll('λ, test', 'changed')}]`),
wire(`[${print}]`, `[${print}]`, `[${row},${row}]`),
wire(`[${row}]`, `[${print}]`),
wire(`[${print}]`, `[${row}]`),
wire(`[${print}]`, `[${print.slice(0, -1)}]`),
wire(`[${print}]`, `[${print}]`).replace('__HLEDGER_PREPARED_ORIGINAL_END__', '__WRONG_END__')
])(
'fails closed on duplicate/missing/mismatched records and crossed frames (%#)',
async (output) => {
await rejected(output);
}
);
it('rejects duplicate lot commodity identities', async () => {
const output = wire(`[${print}]`, `[${print}]`).replace(
'\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n',
'\n__HLEDGER_PREPARED_LOT_COMMODITIES__\nVTI\nVTI\n'
);
const writes = await rejected(output);
expect(writes).not.toHaveBeenCalled();
});
it('discards unknown CI-only print records without contaminating account-local cash evidence', async () => {
const extra = structuredClone(nativeRowTransaction);
extra.tindex = 8;
extra.tpostings[0].paccount = 'assets:Cash';
extra.tpostings[1].paccount = 'assets:cashbox';
extra.tpostings[0].pamount[0].acommodity = 'SECRET-UNIT';
const extraJson = JSON.stringify(extra);
inject(wire(`[${extraJson},${print}]`, `[${print},${extraJson}]`));
const collected = collector();
const result = await prepareReports('/private', collected);
expect(result.accounts[0].rowCount).toBe(1);
expect(JSON.stringify(collected.rowsByName.get('assets:cash'))).not.toContain('SECRET-UNIT');
});
});
it('uses actual native role query sets, not account-name or posting-tag approximations', async () => {
const path = await journal(`commodity USD 1.00
commodity XYZ 1.000 ; lots: FIFO
account arbitrary pool ; type: C
account arbitrary holdings ; type: A
account arbitrary equity ; type: E
account arbitrary income ; type: R
2026-01-01 Open
arbitrary pool 100 USD
arbitrary equity
2026-01-02 Buy
arbitrary holdings 1 XYZ @ 20 USD
arbitrary pool
2026-01-03 Receipt
arbitrary pool 5 USD
arbitrary income
`);
const collected = collector();
await prepareReports(path, collected);
const buy = collected.rowsByName.get('arbitrary holdings')![0];
expect(buy.transaction.postings.map((posting) => posting.kind)).toEqual([
'RealPosting',
'RealPosting'
]);
expect(buy.investment.issues).toEqual([]);
expect(buy.investment.cashPostingIndexes).toEqual([1]);
expect(buy.investment.securities[0].commodity).toBe('XYZ');
expect(buy.investment.securities[0].action).toBe('buy');
const receipt = collected.rowsByName
.get('arbitrary pool')!
.find((row) => row.transaction.description === 'Receipt')!;
expect(receipt.transaction.postings.map((posting) => posting.kind)).toEqual([
'RealPosting',
'RealPosting'
]);
expect(receipt.investment.cash.direction).toBe('in');
expect(receipt.investment.cash.unclassifiedChange).toEqual([]);
expect(receipt.investment.incomePostingIndexes).toEqual([1]);
expect(receipt.investment.securities).toEqual([]);
});
it('retains coalesced multi-lot price/value and explicitly unknown basis without --lots semantics', async () => {
const path = await journal(`${lotsJournal}
2026-04-17 Multi lot sale
assets:investments:brokerage -8 VTI @ 150.00 USD
assets:bank:checking 1200.00 USD
`);
const collected = collector();
await prepareReports(path, collected);
const row = collected.rowsByName
.get('assets:investments:brokerage')!
.find((row) => row.transaction.description === 'Multi lot sale')!;
expect(row.transaction.postings).toHaveLength(3);
const security = row.transaction.postings[0];
expect(security.amounts).toHaveLength(1);
expect(security.amounts[0]).toMatchObject({
commodity: 'VTI',
quantity: expect.stringMatching(/^-8(?:\.0+)?$/),
cost: {
kind: 'UnitCost',
amount: { commodity: 'USD', quantity: expect.stringMatching(/^150(?:\.0+)?$/) }
},
costBasis: { amount: null, date: null, label: null }
});
expect(security.transactedAmounts).toEqual([
{ commodity: 'USD', quantity: expect.stringMatching(/^-1200(?:\.0+)?$/) }
]);
expect(
row.transaction.postings.find((p) => p.account === 'income:capital gains')!.amounts[0].quantity
).toMatch(/^-300(?:\.0+)?$/);
expect(row.accountPostingIndexes).toEqual([0]);
});
it('matches a literal Unicode account query and retains a native-omitted conversion posting only in original data', async () => {
const selected = 'assets:literal [東京]';
const path = await journal(`commodity USD 1.00
account ${selected} ; type: C
account equity:opening ; type: E
account equity:conversion ; type: E
2026-01-01 Conversion marker
${selected} 10 USD
equity:opening -10 USD
equity:conversion 0 USD ; _conversion-posting:
`);
const collected = collector();
await prepareReports(path, collected);
const row = collected.rowsByName.get(selected)![0];
expect(row.transaction.postings).toHaveLength(3);
expect(row.transaction.postings[2].transactedAmounts).toBeNull();
expect(row.transaction.postings[0].transactedAmounts).toEqual([
{ commodity: 'USD', quantity: expect.stringMatching(/^10(?:\.0+)?$/) }
]);
expect(row.accountPostingIndexes).toEqual([0]);
});
packages/hledger/src/prepare-reports.ts
-305
View File
@@ -1,305 +0,0 @@
import { decodeOverviewReports, planOverviewReports } from './overview/batch.js';
import { organizeOverviewData, type OverviewData } from './overview/report.js';
import {
planAccountBalances,
decodeAccountBalances,
organizeAccountBalance,
type PreparedBalance
} from './accounts/balances.js';
import { journalMetadata } from './journal-stats.js';
import { getHledgerVersion, supportedHledgerVersion } from './checks.js';
import { ReportStreamReader } from './report-stream.js';
import {
parseAccountRegisterRow,
organizeAccountRow,
type PreparedAccountRow
} from './transactions/account-rows.js';
import { consumeHledger, readHledger } from './process.js';
import { AccountTransactionExports } from './transactions/export.js';
import { accountInvestmentContext, type InvestmentMetadata } from './transactions/investment.js';
export interface PreparedAccountSummary {
name: string;
access: 'ready' | 'ambiguous';
rowCount: number | null;
summary: PreparedBalance;
}
type AccountRowWriter = (accountName: string, rows: readonly PreparedAccountRow[]) => Promise<void>;
const accountMarker = '\n__HLEDGER_PREPARED_ACCOUNTS__\n';
const discoveryEnd = '\n__HLEDGER_PREPARED_DISCOVERY_END__\n';
const findMarker = '\n__HLEDGER_PREPARED_FIND__\n';
const registersMarker = '\n__HLEDGER_PREPARED_REGISTERS__\n';
const registerEnd = '\n__HLEDGER_PREPARED_REGISTER_END__\n';
const originalEnd = '\n__HLEDGER_PREPARED_ORIGINAL_END__\n';
const transactedEnd = '\n__HLEDGER_PREPARED_TRANSACTED_END__\n';
const roleQueries = [
['assets', 'A', '\n__HLEDGER_PREPARED_ASSETS__\n'],
['cash', 'C', '\n__HLEDGER_PREPARED_CASH__\n'],
['liabilities', 'L', '\n__HLEDGER_PREPARED_LIABILITIES__\n'],
['income', 'R', '\n__HLEDGER_PREPARED_INCOME__\n'],
['expenses', 'X', '\n__HLEDGER_PREPARED_EXPENSES__\n'],
['gains', 'G', '\n__HLEDGER_PREPARED_GAINS__\n']
] as const;
const lotsMarker = '\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n';
const failureMessage = 'Prepared transactions could not be read. Check the journal.';
function echo(marker: string): string[] {
return ['--', 'echo', marker.trimEnd()];
}
function accountNames(text: string): string[] {
if (text === '') return [];
if (!text.endsWith('\n')) throw new Error(failureMessage);
const names = text
.slice(0, -1)
.split('\n')
.map((line) => {
if (!line.startsWith('account ')) throw new Error(failureMessage);
const name = line.slice('account '.length);
// Names are identities, never paths. Reject controls and unrepresentable
// identities rather than normalizing native whitespace or Unicode.
if (!name.trim() || /\p{Cc}/u.test(name)) {
throw new Error(failureMessage);
}
return name;
});
if (new Set(names).size !== names.length) throw new Error(failureMessage);
return names;
}
function splitLast(text: string, marker: string): [string, string] {
const index = text.lastIndexOf(marker);
if (index < 0) throw new Error(failureMessage);
return [text.slice(0, index), text.slice(index + marker.length)];
}
function pattern(name: string): string {
return `^${name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`;
}
function decodePreparedPrefix(
prefix: string,
names: string[],
discovery: ReturnType<typeof journalMetadata>,
plan: ReturnType<typeof planOverviewReports>
) {
// These native query sets are part of the same financial prefix and
// cached journal, never inferred from account names or inherited posting tags.
const [beforeLots, lots] = splitLast(prefix, lotsMarker);
prefix = beforeLots;
const metadata: InvestmentMetadata = {
assets: new Set(),
cash: new Set(),
liabilities: new Set(),
income: new Set(),
expenses: new Set(),
gains: new Set(),
lotCommodities: commodityNames(lots)
};
for (const [role, , marker] of [...roleQueries].reverse()) {
const [before, text] = splitLast(prefix, marker);
metadata[role] = new Set(accountNames(text));
prefix = before;
}
const canonical: string[] = [];
for (let index = names.length - 1; index >= 0; index--) {
const [before, found] = splitLast(prefix, findMarker);
const matches = accountNames(found);
if (matches.length !== 1) throw new Error(failureMessage);
canonical.unshift(matches[0]);
prefix = before;
}
const [financial, actualNames] = splitLast(prefix, accountMarker);
const actual = accountNames(actualNames).sort();
if (actual.length !== names.length || actual.some((name, index) => name !== names[index])) {
throw new Error(failureMessage);
}
const accountReports = decodeAccountBalances(financial, discovery.asOf);
const overview = organizeOverviewData(
decodeOverviewReports(accountReports.overviewBatch, discovery, plan)
);
// Native --find defines equivalence; JS case folding does not.
const counts = new Map<string, number>();
for (const name of canonical) counts.set(name, (counts.get(name) ?? 0) + 1);
const accounts: PreparedAccountSummary[] = names.map((name, index) => {
const nativeName = canonical[index];
const access = nativeName !== name || counts.get(nativeName) !== 1 ? 'ambiguous' : 'ready';
const rowCount = access === 'ambiguous' ? null : 0;
const summary = organizeAccountBalance(
name,
access,
discovery.asOf,
discovery.baseCurrency,
accountReports
);
return { name, access, rowCount, summary };
});
return {
overview,
accounts,
metadata
};
}
function commodityNames(text: string): ReadonlySet<string> {
if (text === '') return new Set();
if (!text.endsWith('\n')) throw new Error(failureMessage);
const names = text.slice(0, -1).split('\n');
// Commodities emits raw symbols, not quoted commodity directives.
if (names.some((name) => /\p{Cc}/u.test(name)) || new Set(names).size !== names.length)
throw new Error(failureMessage);
return new Set(names);
}
async function writePreparedAccountRows(
reader: ReportStreamReader,
account: PreparedAccountSummary,
metadata: InvestmentMetadata,
writeRows: AccountRowWriter,
check: () => void
): Promise<void> {
const exports = new AccountTransactionExports(account.name, account.access === 'ambiguous');
await reader.array('object', async (json) => exports.original(json));
await reader.expect(`\n${originalEnd}`);
await reader.array('object', async (json) => exports.transacted(json));
await reader.expect(`\n${transactedEnd}`);
const context = accountInvestmentContext(exports.transactions(), metadata, account.name);
let chunk: PreparedAccountRow[] = [];
const write = async () => {
check();
// Cancellation kills native immediately, but caller-owned writes must settle
// before preparation returns and its staging can safely be discarded.
await writeRows(account.name, chunk);
check();
chunk = [];
};
await reader.array('array', async (json) => {
const tuple = parseAccountRegisterRow(json);
if (account.access === 'ambiguous') return;
const transaction = exports.take(tuple[0].tindex);
chunk.push(organizeAccountRow(tuple, account.name, transaction, context));
account.rowCount = (account.rowCount ?? 0) + 1;
if (chunk.length === 50) await write();
});
await reader.expect(`\n${registerEnd}`);
if (account.access === 'ready' && (chunk.length > 0 || account.rowCount === 0)) await write();
}
/**
* One cached native view supplies every financial report and full-history register.
* Row writes MUST be private/staged: only successful return permits publication.
*/
export async function prepareReports(
journal: string,
options: {
writeRows: AccountRowWriter;
signal?: AbortSignal;
}
): Promise<{ overview: OverviewData; accounts: PreparedAccountSummary[] }> {
try {
options.signal?.throwIfAborted();
// Operational version check only: no journal flags or extra journal read.
await getHledgerVersion((args) => readHledger(args, { signal: options.signal }));
options.signal?.throwIfAborted();
const inputFlags = ['--no-conf', '--lang=en', '--strict', '-f', journal];
// A discovery process only plans commands. No values from it are published.
const discoveryText = await readHledger(
[
...inputFlags,
'run',
'--',
'stats',
...echo(accountMarker),
'--',
'accounts',
'--flat',
'--directives',
...echo(discoveryEnd)
],
{ signal: options.signal }
);
options.signal?.throwIfAborted();
if (!discoveryText.endsWith(discoveryEnd)) throw new Error(failureMessage);
const [stats, namesText] = splitLast(
discoveryText.slice(0, -discoveryEnd.length),
accountMarker
);
const names = accountNames(namesText).sort();
const discovery = journalMetadata(stats);
const plan = planOverviewReports(discovery);
const args = [
...inputFlags,
'run',
'--',
...plan.commands,
...planAccountBalances(discovery),
...echo(accountMarker),
'--',
'accounts',
'--flat',
'--directives'
];
for (const name of names) {
args.push(...echo(findMarker), '--', 'accounts', '--find', '--directives', pattern(name));
}
for (const [, type, marker] of roleQueries) {
args.push(...echo(marker), '--', 'accounts', '--flat', '--directives', `type:${type}`);
}
args.push(...echo(lotsMarker), '--', 'commodities', 'tag:lots');
args.push(...echo(registersMarker));
for (const name of names) {
// Print's acct query is case-insensitive; exact canonical membership is
// checked after decoding, without dropping any counterpart postings.
const query = `acct:${pattern(name).slice(0, -1)}(:|$)`;
args.push('--', 'print', '--round=none', '-Ojson', query, ...echo(originalEnd));
args.push(
'--',
'print',
'--round=none',
'--value=transacted',
'-Ojson',
query,
...echo(transactedEnd)
);
// No separator before the literal pattern: run owns `--` separators.
args.push(
'--',
'aregister',
'--historical',
'--empty',
'-Ojson',
pattern(name),
...echo(registerEnd)
);
}
const { overview, accounts } = await consumeHledger(
args,
{ signal: options.signal },
async (output, check) => {
const reader = new ReportStreamReader(output, check);
const { overview, accounts, metadata } = await reader.prefix(registersMarker, (prefix) =>
decodePreparedPrefix(prefix, names, discovery, plan)
);
for (const account of accounts) {
await writePreparedAccountRows(reader, account, metadata, options.writeRows, check);
}
await reader.eof();
return { overview, accounts };
}
);
// This timestamp describes completion of the whole set, including native exit.
return {
overview: {
...overview,
hledgerVersion: supportedHledgerVersion,
preparedAt: new Date().toISOString()
},
accounts
};
} catch {
// No native diagnostics, paths, financial wire data or nested causes escape.
throw new Error(failureMessage);
}
}
packages/hledger/src/prepare-reports/account-stream.ts
+48
View File
@@ -0,0 +1,48 @@
import type { ReportStreamReader } from '../report-stream.js';
import {
parseAccountRegisterRow,
organizeAccountRow,
type PreparedAccountRow
} from '../transactions/account-rows.js';
import { AccountTransactionExports } from '../transactions/export.js';
import { accountInvestmentContext, type InvestmentMetadata } from '../transactions/investment.js';
import { accountMarkers, type PreparedAccountSummary } from './batch.js';
export type AccountRowWriter = (
accountName: string,
rows: readonly PreparedAccountRow[]
) => Promise<void>;
export async function writePreparedAccountRows(
reader: ReportStreamReader,
account: PreparedAccountSummary,
metadata: InvestmentMetadata,
writeRows: AccountRowWriter,
check: () => void
): Promise<void> {
const exports = new AccountTransactionExports(account.name, account.access === 'ambiguous');
await reader.array('object', async (json) => exports.original(json));
await reader.expect(`\n${accountMarkers.originalEnd}`);
await reader.array('object', async (json) => exports.transacted(json));
await reader.expect(`\n${accountMarkers.transactedEnd}`);
const context = accountInvestmentContext(exports.transactions(), metadata, account.name);
let chunk: PreparedAccountRow[] = [];
const write = async () => {
check();
// Cancellation kills native immediately, but caller-owned writes must settle
// before preparation returns and its staging can safely be discarded.
await writeRows(account.name, chunk);
check();
chunk = [];
};
await reader.array('array', async (json) => {
const tuple = parseAccountRegisterRow(json);
if (account.access === 'ambiguous') return;
const transaction = exports.take(tuple[0].tindex);
chunk.push(organizeAccountRow(tuple, account.name, transaction, context));
account.rowCount = (account.rowCount ?? 0) + 1;
if (chunk.length === 50) await write();
});
await reader.expect(`\n${accountMarkers.registerEnd}`);
if (account.access === 'ready' && (chunk.length > 0 || account.rowCount === 0)) await write();
}
packages/hledger/src/prepare-reports/batch.ts
+228
View File
@@ -0,0 +1,228 @@
import { decodeOverviewReports, planOverviewReports } from '../overview/batch.js';
import { organizeOverviewData } from '../overview/report.js';
import {
planAccountBalances,
decodeAccountBalances,
organizeAccountBalance,
type PreparedBalance
} from '../accounts/balances.js';
import { journalMetadata } from '../journal-stats.js';
import type { InvestmentMetadata } from '../transactions/investment.js';
export interface PreparedAccountSummary {
name: string;
access: 'ready' | 'ambiguous';
rowCount: number | null;
summary: PreparedBalance;
}
const accountMarker = '\n__HLEDGER_PREPARED_ACCOUNTS__\n';
const discoveryEnd = '\n__HLEDGER_PREPARED_DISCOVERY_END__\n';
const findMarker = '\n__HLEDGER_PREPARED_FIND__\n';
export const registersMarker = '\n__HLEDGER_PREPARED_REGISTERS__\n';
export const accountMarkers = {
originalEnd: '\n__HLEDGER_PREPARED_ORIGINAL_END__\n',
transactedEnd: '\n__HLEDGER_PREPARED_TRANSACTED_END__\n',
registerEnd: '\n__HLEDGER_PREPARED_REGISTER_END__\n'
} as const;
const roleQueries = [
['assets', 'A', '\n__HLEDGER_PREPARED_ASSETS__\n'],
['cash', 'C', '\n__HLEDGER_PREPARED_CASH__\n'],
['liabilities', 'L', '\n__HLEDGER_PREPARED_LIABILITIES__\n'],
['income', 'R', '\n__HLEDGER_PREPARED_INCOME__\n'],
['expenses', 'X', '\n__HLEDGER_PREPARED_EXPENSES__\n'],
['gains', 'G', '\n__HLEDGER_PREPARED_GAINS__\n']
] as const;
const lotsMarker = '\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n';
const failureMessage = 'Invalid prepared report batch.';
function echo(marker: string): string[] {
return ['--', 'echo', marker.trimEnd()];
}
function accountNames(text: string): string[] {
if (text === '') return [];
if (!text.endsWith('\n')) throw new Error(failureMessage);
const names = text
.slice(0, -1)
.split('\n')
.map((line) => {
if (!line.startsWith('account ')) throw new Error(failureMessage);
const name = line.slice('account '.length);
// Names are identities, never paths. Reject controls and unrepresentable
// identities rather than normalizing native whitespace or Unicode.
if (!name.trim() || /\p{Cc}/u.test(name)) {
throw new Error(failureMessage);
}
return name;
});
if (new Set(names).size !== names.length) throw new Error(failureMessage);
return names;
}
function commodityNames(text: string): ReadonlySet<string> {
if (text === '') return new Set();
if (!text.endsWith('\n')) throw new Error(failureMessage);
const names = text.slice(0, -1).split('\n');
// Commodities emits raw symbols, not quoted commodity directives.
if (names.some((name) => /\p{Cc}/u.test(name)) || new Set(names).size !== names.length)
throw new Error(failureMessage);
return new Set(names);
}
function splitLast(text: string, marker: string): [string, string] {
const index = text.lastIndexOf(marker);
if (index < 0) throw new Error(failureMessage);
return [text.slice(0, index), text.slice(index + marker.length)];
}
function escapeAccountName(name: string): string {
return name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
export function planDiscoveryCommands(inputFlags: readonly string[]): string[] {
return [
...inputFlags,
'run',
'--',
'stats',
...echo(accountMarker),
'--',
'accounts',
'--flat',
'--directives',
...echo(discoveryEnd)
];
}
export function decodeDiscovery(text: string) {
if (!text.endsWith(discoveryEnd)) throw new Error(failureMessage);
const [stats, namesText] = splitLast(text.slice(0, -discoveryEnd.length), accountMarker);
const names = accountNames(namesText).sort();
const discovery = journalMetadata(stats);
return { names, discovery };
}
export function planPreparedReportCommands(
inputFlags: readonly string[],
discovery: ReturnType<typeof journalMetadata>,
names: readonly string[]
) {
const overviewPlan = planOverviewReports(discovery);
const args = [
...inputFlags,
'run',
'--',
...overviewPlan.commands,
...planAccountBalances(discovery),
...echo(accountMarker),
'--',
'accounts',
'--flat',
'--directives'
];
for (const name of names) {
const exactAccountPattern = `^${escapeAccountName(name)}$`;
args.push(...echo(findMarker), '--', 'accounts', '--find', '--directives', exactAccountPattern);
}
for (const [, type, marker] of roleQueries) {
args.push(...echo(marker), '--', 'accounts', '--flat', '--directives', `type:${type}`);
}
args.push(...echo(lotsMarker), '--', 'commodities', 'tag:lots');
args.push(...echo(registersMarker));
for (const name of names) {
const escapedName = escapeAccountName(name);
const exactAccountPattern = `^${escapedName}$`;
// Print's acct query is case-insensitive; exact canonical membership is
// checked after decoding, without dropping any counterpart postings.
const subtreeQuery = `acct:^${escapedName}(:|$)`;
args.push(
'--',
'print',
'--round=none',
'-Ojson',
subtreeQuery,
...echo(accountMarkers.originalEnd)
);
args.push(
'--',
'print',
'--round=none',
'--value=transacted',
'-Ojson',
subtreeQuery,
...echo(accountMarkers.transactedEnd)
);
// No separator before the literal pattern: run owns `--` separators.
args.push(
'--',
'aregister',
'--historical',
'--empty',
'-Ojson',
exactAccountPattern,
...echo(accountMarkers.registerEnd)
);
}
return { args, overviewPlan };
}
export function decodePreparedPrefix(
prefix: string,
names: string[],
discovery: ReturnType<typeof journalMetadata>,
overviewPlan: ReturnType<typeof planOverviewReports>
) {
// These native query sets are part of the same financial prefix and
// cached journal, never inferred from account names or inherited posting tags.
const [beforeLots, lots] = splitLast(prefix, lotsMarker);
prefix = beforeLots;
const metadata: InvestmentMetadata = {
assets: new Set(),
cash: new Set(),
liabilities: new Set(),
income: new Set(),
expenses: new Set(),
gains: new Set(),
lotCommodities: commodityNames(lots)
};
for (const [role, , marker] of [...roleQueries].reverse()) {
const [before, text] = splitLast(prefix, marker);
metadata[role] = new Set(accountNames(text));
prefix = before;
}
const canonical: string[] = [];
for (let index = names.length - 1; index >= 0; index--) {
const [before, found] = splitLast(prefix, findMarker);
const matches = accountNames(found);
if (matches.length !== 1) throw new Error(failureMessage);
canonical.unshift(matches[0]);
prefix = before;
}
const [financial, actualNames] = splitLast(prefix, accountMarker);
const actual = accountNames(actualNames).sort();
if (actual.length !== names.length || actual.some((name, index) => name !== names[index])) {
throw new Error(failureMessage);
}
const accountReports = decodeAccountBalances(financial, discovery.asOf);
const overview = organizeOverviewData(
decodeOverviewReports(accountReports.overviewBatch, discovery, overviewPlan)
);
// Native --find defines equivalence; JS case folding does not.
const counts = new Map<string, number>();
for (const name of canonical) counts.set(name, (counts.get(name) ?? 0) + 1);
const accounts: PreparedAccountSummary[] = names.map((name, index) => {
const nativeName = canonical[index];
const access = nativeName !== name || counts.get(nativeName) !== 1 ? 'ambiguous' : 'ready';
const rowCount = access === 'ambiguous' ? null : 0;
const summary = organizeAccountBalance(
name,
access,
discovery.asOf,
discovery.baseCurrency,
accountReports
);
return { name, access, rowCount, summary };
});
return { overview, accounts, metadata };
}
packages/hledger/src/prepare-reports/run.integration.test.ts
+118
View File
@@ -0,0 +1,118 @@
import { mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, expect, it } from 'vitest';
import { cashJournal } from '../../fixtures/cash.fixture.js';
import type { PreparedAccountRow } from '../transactions/account-rows.js';
import { prepareReports } from './run.js';
const directories: string[] = [];
afterEach(async () => {
await Promise.all(
directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))
);
});
async function createJournal(contents: string): Promise<string> {
const directory = await mkdtemp(join(tmpdir(), 'prepared-reports-'));
directories.push(directory);
const journal = join(directory, 'main.journal');
await writeFile(journal, contents);
return journal;
}
function collectRows() {
const rowsByAccount = new Map<string, PreparedAccountRow[]>();
return {
rowsByAccount,
async writeRows(name: string, rows: readonly PreparedAccountRow[]) {
const previous = rowsByAccount.get(name) ?? [];
previous.push(...rows);
rowsByAccount.set(name, previous);
}
};
}
it('joins native account context, original annotations and transacted values', async () => {
const journal = await createJournal(cashJournal);
const collected = collectRows();
const prepared = await prepareReports(journal, collected);
const rows = collected.rowsByAccount.get('assets:Cash')!;
expect(rows.find((row) => row.transaction.description === 'Dated payment')).toMatchObject({
accountDate: '2026-01-08',
transaction: { date: '2026-02-01', secondaryDate: '2026-02-03' }
});
const foreign = rows.find((row) => row.transaction.description === 'Foreign currency payment')!;
expect(foreign.accountPostingIndexes).toEqual([0]);
expect(foreign.transaction.postings[1]).toMatchObject({
amounts: [
{
commodity: 'EUR',
quantity: '8.00',
cost: { kind: 'UnitCost', amount: { commodity: 'USD', quantity: '1.25' } }
}
],
transactedAmounts: [{ commodity: 'USD', quantity: '10' }]
});
for (const account of prepared.accounts) {
if (account.access === 'ready') {
expect(collected.rowsByAccount.get(account.name)).toHaveLength(account.rowCount!);
}
}
});
it('keeps one financial view when an included source changes between row callbacks', async () => {
const journal = await createJournal(`commodity USD 1.00
account assets:cash ; type: A
account equity:opening ; type: E
include detail.journal
`);
const detail = join(journal, '..', 'detail.journal');
const contents = (quantity: string) => `2026-04-16 Opening
assets:cash ${quantity} USD
equity:opening
`;
await writeFile(detail, contents('1.00'));
const before = collectRows();
const beforeReports = await prepareReports(journal, before);
const during = collectRows();
let changed = false;
const reports = await prepareReports(journal, {
async writeRows(name, rows) {
await during.writeRows(name, rows);
if (!changed) {
changed = true;
await writeFile(detail, contents('2.00'));
}
}
});
expect(changed).toBe(true);
expect(during.rowsByAccount).toEqual(before.rowsByAccount);
expect({ ...reports.overview, preparedAt: '' }).toEqual({
...beforeReports.overview,
preparedAt: ''
});
const after = await prepareReports(journal, { async writeRows() {} });
expect(after.overview.netWorth).not.toEqual(reports.overview.netWorth);
});
it('withholds both accounts when native selection resolves their identities to the same account', async () => {
const journal = await createJournal(`commodity USD
account assets:case
account assets:Case
account equity
2026-01-01 Collision
assets:case 1 USD
assets:Case 2 USD
equity
`);
const collected = collectRows();
const prepared = await prepareReports(journal, collected);
const collisions = prepared.accounts.filter((account) => account.name.startsWith('assets:'));
expect(collisions).toHaveLength(2);
for (const account of collisions) {
expect(account).toMatchObject({ access: 'ambiguous', rowCount: null });
expect(collected.rowsByAccount.has(account.name)).toBe(false);
}
});
packages/hledger/src/prepare-reports/run.test.ts
+289
View File
@@ -0,0 +1,289 @@
import type { ChildProcessWithoutNullStreams } from 'node:child_process';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { supportedHledgerVersion } from '../checks.js';
import { transactionFixture } from '../transactions/transaction.fixture.js';
import { prepareReports } from './run.js';
const processMock = vi.hoisted(() => ({ spawn: vi.fn() }));
vi.mock('node:child_process', async (importOriginal) => {
const childProcesses = await importOriginal<typeof import('node:child_process')>();
return { ...childProcesses, spawn: processMock.spawn };
});
const realChildProcesses =
await vi.importActual<typeof import('node:child_process')>('node:child_process');
const children = new Map<ChildProcessWithoutNullStreams, Promise<void>>();
const releaseWrites: (() => void)[] = [];
beforeEach(() => {
processMock.spawn.mockReset().mockImplementation(() => {
throw new Error('Unexpected hledger process.');
});
});
afterEach(async () => {
try {
for (const release of releaseWrites.splice(0)) release();
for (const child of children.keys()) {
if (child.exitCode === null && child.signalCode === null) child.kill('SIGKILL');
}
await Promise.all(children.values());
} finally {
children.clear();
vi.restoreAllMocks();
}
});
const versionOutput = `hledger ${supportedHledgerVersion}, SECRET banner\n`;
const journalStats = 'Base currency : none\nLast txn : none\nTxns : 0\nTxns span : to\n';
const accountCatalog = '\n__HLEDGER_PREPARED_ACCOUNTS__\naccount assets:cash\n';
const discoveryOutput = `${journalStats}${accountCatalog}\n__HLEDGER_PREPARED_DISCOVERY_END__\n`;
const rolePrefix =
['ASSETS', 'CASH', 'LIABILITIES', 'INCOME', 'EXPENSES', 'GAINS']
.map(
(role) =>
`\n__HLEDGER_PREPARED_${role}__\n${role === 'ASSETS' || role === 'CASH' ? 'account assets:cash\n' : ''}`
)
.join('') + '\n__HLEDGER_PREPARED_LOT_COMMODITIES__\n';
const financialPrefix = `${journalStats}${accountCatalog}\n__HLEDGER_PREPARED_FIND__\naccount assets:cash\n${rolePrefix}\n__HLEDGER_PREPARED_REGISTERS__\n`;
const originalEnd = '\n\n__HLEDGER_PREPARED_ORIGINAL_END__\n';
const transactedEnd = '\n\n__HLEDGER_PREPARED_TRANSACTED_END__\n';
const registerEnd = '\n\n__HLEDGER_PREPARED_REGISTER_END__\n';
const nativeTransaction = {
...transactionFixture(),
tdate: '2031-01-01',
tdescription: 'λ, test'
};
const accountExports = `${financialPrefix}[${JSON.stringify(nativeTransaction)}]${originalEnd}[${JSON.stringify(nativeTransaction)}]${transactedEnd}`;
const registerRow = JSON.stringify([
nativeTransaction,
nativeTransaction,
false,
['other, 東京'],
[],
[]
]);
function accountProtocol(rowCount: number): string {
const transactions = Array.from({ length: rowCount }, (_, index) => ({
...nativeTransaction,
tindex: index + 1,
tpostings: nativeTransaction.tpostings.map((posting) => ({
...posting,
ptransaction_: String(index + 1)
}))
}));
const prints = JSON.stringify(transactions);
const rows = JSON.stringify(
transactions.map((transaction) => [transaction, transaction, false, ['other, 東京'], [], []])
);
return `${financialPrefix}${prints}${originalEnd}${prints}${transactedEnd}${rows}${registerEnd}`;
}
function startFixtureChild(script: string): ChildProcessWithoutNullStreams {
const child = realChildProcesses.spawn(process.execPath, ['-e', script]);
children.set(child, new Promise<void>((resolve) => child.once('close', () => resolve())));
child.stdin.on('error', () => {});
return child;
}
function queueChildOutput(output: string) {
processMock.spawn.mockImplementationOnce(() => {
const child = startFixtureChild(
`process.stdin.pipe(process.stdout,{end:false});process.stdin.on('end',()=>process.stdout.write('',()=>process.exit(0)));`
);
child.stdin.end(output);
return child;
});
}
function mockPreparationProcesses(
output: string,
{ exitCode = 0, keepOpen = false }: { exitCode?: number; keepOpen?: boolean } = {}
): ChildProcessWithoutNullStreams {
queueChildOutput(versionOutput);
queueChildOutput(discoveryOutput);
const child = startFixtureChild(
`process.stdin.pipe(process.stdout,{end:false});process.stdin.on('end',()=>process.stdout.write('',()=>{${keepOpen ? 'setInterval(()=>{},1000)' : `process.exit(${exitCode})`}}));`
);
processMock.spawn.mockImplementationOnce(() => {
child.stdin.end(output);
return child;
});
return child;
}
async function expectRejected(output: string, exitCode = 0) {
const child = mockPreparationProcesses(output, { exitCode });
const writeRows = vi.fn(async () => {});
const error = await prepareReports('/private/SECRET', { writeRows }).catch(
(error: unknown) => error
);
expect(error).toBeInstanceOf(Error);
expect((error as Error).message).not.toMatch(/SECRET|\/private/);
expect((error as Error).cause).toBeUndefined();
expect(child.exitCode !== null || child.signalCode !== null).toBe(true);
return writeRows;
}
describe('preparation completion and failure', () => {
it('timestamps completion after the final row callback and stores only the checked version', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-01-01T00:00:00Z'));
mockPreparationProcesses(`${accountExports}[${registerRow}]${registerEnd}`);
const result = await prepareReports('/private', {
async writeRows() {
vi.setSystemTime(new Date('2026-01-01T00:02:30Z'));
}
});
expect(result.overview.preparedAt).toBe('2026-01-01T00:02:30.000Z');
expect(result.overview.hledgerVersion).toBe(supportedHledgerVersion);
expect(JSON.stringify(result)).not.toContain('SECRET');
} finally {
vi.useRealTimers();
}
});
it.each([
{ name: 'truncated account output', output: `${accountExports}[${registerRow}` },
{
name: 'crossed export frames',
output: `${accountExports}[${registerRow}]${registerEnd}`.replace(
'__HLEDGER_PREPARED_ORIGINAL_END__',
'__WRONG_END__'
)
}
])('rejects $name without returning a partial report', async ({ output }) => {
await expectRejected(output);
});
it('rejects a late native nonzero exit even after row writes', async () => {
const writes = await expectRejected(accountProtocol(51), 9);
expect(writes).toHaveBeenCalled();
});
it('kills the child and withholds private diagnostics when a row callback fails', async () => {
const child = mockPreparationProcesses(accountProtocol(51), { keepOpen: true });
const error = await prepareReports('/private', {
async writeRows() {
throw new Error('SECRET /private 123.45 USD');
}
}).catch((error: unknown) => error);
expect(error).toBeInstanceOf(Error);
expect((error as Error).message).not.toMatch(/SECRET|\/private|123\.45/);
expect((error as Error).cause).toBeUndefined();
expect(child.signalCode).toBe('SIGKILL');
});
it('kills the child on abort but waits for the active row write before returning', async () => {
const child = mockPreparationProcesses(accountProtocol(50), { keepOpen: true });
const childClosed = new Promise<void>((resolve) => {
child.once('close', () => resolve());
});
const controller = new AbortController();
let enterWrite!: () => void;
const firstWrite = new Promise<void>((resolve) => {
enterWrite = resolve;
});
let releaseWrite!: () => void;
const held = new Promise<void>((resolve) => {
releaseWrite = resolve;
});
releaseWrites.push(releaseWrite);
let writeFinished = false;
let settled = false;
const result = prepareReports('/private', {
signal: controller.signal,
async writeRows() {
enterWrite();
await held;
writeFinished = true;
}
});
void result.then(
() => {
settled = true;
},
() => {
settled = true;
}
);
try {
await firstWrite;
controller.abort();
await childClosed;
expect(child.signalCode).toBe('SIGKILL');
expect(settled).toBe(false);
expect(writeFinished).toBe(false);
} finally {
releaseWrite();
}
await expect(result).rejects.toThrow();
expect(writeFinished).toBe(true);
});
it('kills a blocked financial read when aborted', async () => {
const child = mockPreparationProcesses(accountExports.slice(0, -5), { keepOpen: true });
const controller = new AbortController();
const result = prepareReports('/private', {
signal: controller.signal,
async writeRows() {}
});
await vi.waitFor(() => expect(processMock.spawn).toHaveBeenCalledTimes(3));
controller.abort();
await expect(result).rejects.toThrow();
expect(child.signalCode).toBe('SIGKILL');
});
it('finishes the version-only process before reading any journal', async () => {
let child: ChildProcessWithoutNullStreams;
processMock.spawn.mockImplementationOnce(() => {
child = startFixtureChild(
`process.stdout.write(${JSON.stringify(versionOutput)});setInterval(()=>{},1000);`
);
return child;
});
const controller = new AbortController();
const result = prepareReports('/private', {
signal: controller.signal,
async writeRows() {}
});
await new Promise<void>((resolve) => child!.stdout.once('data', () => resolve()));
controller.abort();
await expect(result).rejects.toThrow();
expect(processMock.spawn).toHaveBeenCalledTimes(1);
expect(processMock.spawn.mock.calls[0][1]).not.toContain('-f');
expect(child!.signalCode).toBe('SIGKILL');
});
});
describe('batch identity and framing', () => {
it('rejects a changed account catalog instead of mixing discovery and financial views', async () => {
await expectRejected(
`${accountExports.replace('account assets:cash\n', 'account assets:other\n')}[]${registerEnd}`
);
});
it('does not confuse stats-path markers with protocol boundaries', async () => {
queueChildOutput(versionOutput);
queueChildOutput(`Main file : /private${accountCatalog}part\n${discoveryOutput}`);
queueChildOutput(
`Main file : /private\n__HLEDGER_PREPARED_REGISTERS__\npart\n${accountExports}[]${registerEnd}`
);
const result = await prepareReports('/private', { async writeRows() {} });
expect(result.accounts[0].rowCount).toBe(0);
});
it('rejects a forged complete protocol in a stats path unless physical EOF also validates', async () => {
await expectRejected(
`${accountExports}[]${registerEnd}rest-of-path\n${accountExports}[]${registerEnd}`
);
});
it('rejects unsafe account identities before starting financial execution', async () => {
queueChildOutput(versionOutput);
queueChildOutput(discoveryOutput.replace('assets:cash', 'assets:\tSECRET'));
await expect(prepareReports('/private', { async writeRows() {} })).rejects.toThrow();
expect(processMock.spawn).toHaveBeenCalledTimes(2);
});
});
packages/hledger/src/prepare-reports/run.ts
+59
View File
@@ -0,0 +1,59 @@
import { getHledgerVersion, supportedHledgerVersion } from '../checks.js';
import type { OverviewData } from '../overview/report.js';
import { consumeHledger, readHledger } from '../process.js';
import { ReportStreamReader } from '../report-stream.js';
import { writePreparedAccountRows, type AccountRowWriter } from './account-stream.js';
import {
decodeDiscovery,
decodePreparedPrefix,
planDiscoveryCommands,
planPreparedReportCommands,
registersMarker,
type PreparedAccountSummary
} from './batch.js';
export async function prepareReports(
journal: string,
options: {
writeRows: AccountRowWriter;
signal?: AbortSignal;
}
): Promise<{ overview: OverviewData; accounts: PreparedAccountSummary[] }> {
try {
options.signal?.throwIfAborted();
await getHledgerVersion((args) => readHledger(args, { signal: options.signal }));
options.signal?.throwIfAborted();
const inputFlags = ['--no-conf', '--lang=en', '--strict', '-f', journal];
const discoveryText = await readHledger(planDiscoveryCommands(inputFlags), {
signal: options.signal
});
options.signal?.throwIfAborted();
const { names, discovery } = decodeDiscovery(discoveryText);
const { args, overviewPlan } = planPreparedReportCommands(inputFlags, discovery, names);
const { overview, accounts } = await consumeHledger(
args,
{ signal: options.signal },
async (output, check) => {
const reader = new ReportStreamReader(output, check);
const { overview, accounts, metadata } = await reader.prefix(registersMarker, (prefix) =>
decodePreparedPrefix(prefix, names, discovery, overviewPlan)
);
for (const account of accounts) {
await writePreparedAccountRows(reader, account, metadata, options.writeRows, check);
}
await reader.eof();
return { overview, accounts };
}
);
return {
overview: {
...overview,
hledgerVersion: supportedHledgerVersion,
preparedAt: new Date().toISOString()
},
accounts
};
} catch {
throw new Error('Prepared transactions could not be read. Check the journal.');
}
}
packages/jobs/README.md
+3 -97
View File
@@ -1,98 +1,4 @@
# @hledger-reports/jobs
# Jobs
Shared job definitions and a caller-owned OpenWorkflow **0.11.0** SQLite connection,
following When's thin shared-jobs boundary. This package owns input/output types,
validation, stored workflow identities and queue initialization—not submission
policy, result waits, native execution or retention implementation.
- [Web](../../apps/web/README.md) owns explicit requests, preparation coordination,
status reads, result waits and HTTP errors.
- [Worker](../../apps/worker/README.md) owns all handlers and retired-job rejection,
including [query job retention](../../apps/worker/README.md#query-job-retention)
and its recurring schedule.
- The SDK owns queue persistence, migrations, leases, retries and scheduling.
## Source layout
```text
src/
index.ts
specs.ts
specs.test.ts
queue/
client.ts
client.test.ts
migration.integration.test.ts
queries/
cost-balance.ts
cost-balance.test.ts
```
`specs.ts` is the shared list of jobs. `queue/` owns connection and SDK compatibility;
`queries/` owns query input/output contracts, not execution. Tests sit beside their
owner, with integration checks named explicitly.
## Definitions
| Export | Stored identity | Input → output |
| ------------------------------ | --------------------------------------- | ---------------------------------- |
| `liveBalanceQuerySpec` | `hledger.balance-query` v2 | `QueryRequest` → `LiveQueryResult` |
| `retiredBalanceQuerySpec` | `hledger.balance-query` v1 | `unknown` → `never` |
| `prepareReportsSpec` | `hledger.prepare-dashboard` v1 | `void` → full `OverviewData` |
| `retiredIncomePreparationSpec` | `hledger.prepare-income` v1 | `unknown` → `never` |
| `queryRetentionSpec` | `hledger.query-retention` (unversioned) | `void` → `{ deleted: number }` |
All five definitions retain **maximumAttempts 1** and declare no schedules.
The worker attaches retention's daily UTC schedule; SDK recurrence requires an
unversioned definition. Retired identities stay registered by the worker so
persisted legacy runs can fail once rather than remain unregistered.
The preparation name/version remain unchanged for stored-queue compatibility.
`QueryRequest` is exactly `{ query, asOf }`. `validateQueryRequest` enforces the
strict request schema, expression limits and valid date boundaries without
coercion. The live definition uses the same validation through Standard Schema.
`LiveQueryResult` combines request context with the existing `CostBalanceReport`
via a type-only hledger import; there is no separate financial runtime schema.
`QueryJobError` exposes fixed safe messages/codes, never private causes.
`CostBalanceWorkflow`, `PrepareReportsWorkflow` and `QueryRetentionWorkflow`
provide the SDK handler types for worker implementations.
## Connection and lifecycle
```ts
import { openJobsClient } from '@hledger-reports/jobs';
const client = await openJobsClient(jobsDirectory);
try {
// Owners use client.workflow and client.backend's public SDK APIs.
} finally {
// First drain requests and stop any worker owned by this caller.
await client.close();
}
```
Each call opens its own migrated backend; there is no global singleton.
`JobsClient` exposes only `workflow`, `backend` and `close()`—no submission or
wait facade. Repeated/concurrent closes join one backend stop, including a
retained failure. Opening and closing errors use generic sanitized messages.
Initialization creates a mode0700 directory and precreates mode0600
`queue.sqlite`, allowing SQLite WAL/SHM files to inherit private permissions.
The directory is trusted/private, not protection against hostile file or symlink
replacement. Financial job inputs/results are private operational data.
Legacy input/result files are not read, migrated or deleted by this package.
## SDK compatibility and tests
Before upgrading, stop web and worker and take a consistent backup of the private
jobs directory. Deploy both with the same SDK version. Startup automatically
migrates SQLite; mixed versions and downgrades are not supported rollback paths.
Restore the pre-upgrade backup if rollback is required.
Tests cover definitions, validation, private storage and lifecycle, genuine SDK
0.9 → 0.11 migration (including original rows and queued legacy/live work).
The worker tests its actual retention schedule, restart and pause behavior; this
package does not maintain a general SDK scheduling or migration-locking suite.
The old SDK alias is test-only. Vitest uses compiled workspace dependencies;
build shared packages first. Application submission and execution tests live
with their web and worker owners.
Shared job contracts and queue connections for web and worker. Web owns requesting
work; worker owns execution. The queue contains private financial inputs and results.
packages/snapshots/README.md
+6 -185
View File
@@ -1,187 +1,8 @@
# @hledger-reports/snapshots
# Saved reports
Server-side saved-report storage, independent of native execution and either
application. Browsing reads saved reports and independent validation evidence only; an absent/empty store returns
`null`, as does a store whose latest completed set has an unsupported format. Preparation is explicitly requested by the inline form in `/settings` and
performed by a worker. Store reads never open journals, run hledger, enqueue work,
watch sources, hash inputs, reconcile state or assess freshness.
Stores complete private report sets and recorded journal checks. Writes are staged;
readers keep one selected set for a consistent request. Checks are independent of
financial publication and do not prove freshness.
## API
- `readReportPublication(snapshotDirectory): Promise<ReportPublication | null>`
- `readOverview(snapshotDirectory): Promise<OverviewData | null>`
- `createReportStage(snapshotDirectory, run, previousAccounts?): Promise<ReportStage>`
- `readAccountTransactions(publication, id): Promise<{ id, name, transactions }>`
- `writeValidation(snapshotDirectory, run, journal, result): Promise<void>`
- `readValidation(snapshotDirectory): Promise<ValidationRecord | null>`
`currentReportFormat` is a private implementation constant (**8**) used by reads
and writes, not a package export or homepage display API. Tests use a local format-8
literal rather than making that constant public.
Pass the snapshots directory. `run` contains `{ id, createdAt: Date }`.
A stage streams contiguous account batches through
`await stage.writeRows(fullNativeName, rows)`, then publishes with
`await stage.publish(overview, accountSummaries, journal)`. New writes require a
nonempty journal identity supplied by the worker. Publication returns **void**;
the worker returns its own prepared overview, not another execution's selected
financial data. Await every operation; discard failed/aborted stages with
`stage.discard()`.
The store retains only account metadata and one open account file, not an
all-history row array. Summaries must match written account names and counts.
Ready empty accounts contain `[]`; ambiguous accounts contain `null`, with null
row counts.
## Publication and reads
Each execution owns a unique hidden stage. One directory rename commits the
**entire** immutable completed set under `.sets/`. Each set contains:
```text
overview.json # { format: 8, journal, overview, accounts }
transactions/<8-character-id>.json # { id, name, transactions }
```
New envelopes contain no source fingerprints, input lists or report versions.
The controlled relative `current` symlink makes completed files convenient for
inspection. Application reads **do not follow current**: they select the greatest
completed request creation key (SDK creation time, then run ID with a request
separator for prefix IDs; execution UUID breaks same-request ties). An older recovered
request cannot supersede a newer request, even when financial or preparation
dates move backward. Alias updates reselect to repair delayed older updates.
`ReportPublication` contains `{ overview, journal, accounts, directory }`, all
nonnullable and describing one complete current-format set.
**Journal and directory are private server-only metadata, never page data.** Keep
that directory pinned for subsequent account reads in the request; the account
reader does not reselect a newer set. `readOverview` returns only the financial
overview.
`StoredAccount` contains `{ id, name, access: 'ready' | 'ambiguous', rowCount, summary }`.
The required `summary` is the native `PreparedBalance`: ambiguous access has
`{ status: 'ambiguous' }`; other undated accounts have `{ status: 'undated' }`;
dated ready accounts have `{ status: 'ready', balances, valuation }`. `balances`
is the raw inclusive native bag; `valuation` contains native end-valued `amounts`
and explicit `unconverted` evidence, including descendant residuals. A dated ready unused
account retains native empty bags, not invented zeros. Storage neither sums nor
values accounts. Summaries live in this catalog, not separate value files;
`transactions/<id>.json` and its full history are unchanged.
Account IDs are random eight-character lowercase alphanumeric identifiers, reused
by **exact full native name** from the previous supported-format catalog. Pass only
`previousPublication.accounts`, and only for the **same journal**. IDs may reset
across the format-8 transition; there is no legacy catalog reader to preserve them.
Collisions are checked against all previous and newly assigned IDs. Names remain unchanged data,
never filesystem paths. There is no separate registry or account-page index.
Every account file contains **all** its rows, with no application-imposed file-size cap.
Each `PreparedAccountRow` retains the library-owned original transaction, postings, comments,
ordered tags, exact amounts and reported metadata alongside account-relative
facts (`accountDate`, `otherAccounts`, `isSplit`, `change`, `balance`) and selected
posting indexes. The original header date remains `transaction.date`. Transaction indexes identify entries within
that prepared set, not durable identities across edits. Transaction source positions
and assertion source positions remain **private** in these files; callers must
explicitly project browser-safe fields, never serialize whole stored rows.
Readers check regular files and stable byte length, validate the catalog/header/count
and required transaction, postings and posting-index containers, and return an ordinary array; callers paginate
after reading. Financial contents and exact quantity strings (including nested
amounts) remain producer-owned and opaque. This is the detail exported by hledger's
account register, **not a verbatim journal archive or a complete investment
cost/basis record**: annotations stripped by hledger cannot be recovered here.
Ordinary JSON serialization rejects BigInt/cycles. Storage neither computes report
dates nor copies input journals. Whole-file reads still require sufficient Node memory;
no capacity guarantee is implied.
## Recorded validation
Validation evidence is operational, independent of financial format **8**; its
own format and workflow are unchanged. The worker writes after standalone strict
checking completes, before financial staging. The
store does not run checks or infer outcomes. `JournalValidationResult` contains
`{ outcome: 'passed' | 'could-not-validate', completedAt: string }`; `ValidationRecord`
contains `{ format: 1, journal, request: { id, createdAt }, result }`.
Each record is immutable:
```text
data/snapshots/.validation/<request-ms>-<run-id>-<completed-ms>-<uuid>.json
```
Both millisecond fields are padded to 16 digits. Selection uses the financial
request creation key including its trailing `/` separator, then completion time
and UUID within the same request. A recovered run can record another check; a delayed
older request cannot supersede a newer request. These are ordering rules, not global
clock or stale-worker fencing guarantees.
Writes use an exclusive private temporary file, close it, then rename it to its
unique final name. Temporary names are ignored. Reads select one record and validate
version, outcome, timestamps and request/completion identity against its filename.
Absent/unsupported evidence returns `null`; malformed or unreadable selected evidence
fails safely. Neither case falls back to an older Passed record. The web compares the
selected journal exactly, withholding other-journal evidence without searching backward,
and serializes only outcome/time. Journal/request identity and paths remain private.
Records are retained without cleanup or a history API/UI; there is no mutable latest
pointer, input hash, freshness proof or financial-data duplication. Atomic rename
prevents partial selection, not power loss; an unwritten observation can be lost. Check
completion and financial preparation times can describe different requests/inputs.
An incomplete check preserves prior evidence; evidence-write failure stops the worker
before financial work. A recorded pass survives later financial failure, including
post-commit alias failure. Evidence-read failure does not hide readable financial files.
The separate read-only native inspection command never writes these records.
## Current format and preservation
Only format-8 envelopes `{ format: 8, journal, overview, accounts }` are read.
The current contract requires saved `hledgerVersion: string`, all four native
period transaction counts and the existing coverage, financial data and account
summaries. The producer checks the engine version; storage validates its string
container, not a native banner. Native semantics belong to the
[hledger library](../hledger/README.md#overview-data).
PR3's format 6, the unlanded intermediate format 7 and all older shapes are
unsupported, including otherwise complete sets. The latest unsupported set returns
`null`, without falling back to an older supported set. Old files remain untouched;
there is no automatic preparation, missing-field adapter, migration or deletion.
Users must explicitly prepare again after
[paired web/worker activation](../../docs/development.md#financial-format-8-activation).
Account IDs may reset; exact-name reuse continues within supported same-journal
publications. `currentReportFormat` remains private to this package.
Malformed JSON or invalid current publications fail with a generic read error.
The store checks required overview header primitives and containers: all four
`ranges` (1/3/6/12, including native totals, categories and a required numeric
`transactionCount` that passes `Number.isSafeInteger` and is at least zero),
`journalCoverage` (safe nonnegative count and nullable date strings),
string `hledgerVersion`, twelve `incomeExpenses.months`, twelve `netWorth.points`, latest `balanceSheet`
sides, and required catalog summary states consistent with access/date. Undated
results require `ranges: null`, `balanceSheet: null`, empty histories and
`netWorth.currency: null`; ambiguous summaries remain ambiguous. It does not parse
deep financial tuples or quantities, upgrade partial data, supply zeros or
recompute producer-owned values. Missing, fractional, negative or unsafe counts in
format 8 are malformed current data, not empty periods. A missing/non-string version is also malformed, not optional metadata; there is
no fallback or legacy adapter. A genuine dated empty period stores native count `0`; an
undated report retains `ranges: null`.
Old `dashboard/*.json` files are neither scanned nor read. Format-2 source
metadata is not adapted. Old `source.json`, `dashboard/inputs.json`, and
transaction/preview directories are untouched: they are neither consulted nor new
write destinations. Explicit preparation can publish a new current set without
rewriting or deleting any old files. There is no source-observation API,
register-view helper, migration or compatibility decoder.
Directories are private **0700**, files **0600**. Final directory/file no-follow
checks reject symlinks; path ancestors must still be trusted, not concurrently
replaced by hostile writers. Errors omit private paths and data. Failed stages
clean only their own hidden directories after closing file handles. Completed sets
are never discarded, including after post-commit alias failures. Failure before the
directory rename retains the previous complete set; failure updating `current` after that commit may
fail the job while the new valid complete set remains selectable. There is no rollback.
Validation writes neither commit nor roll back financial data. All successful
sets and orphaned completions are retained; there is no garbage collection,
migration, history UI or user-data deletion.
`pnpm check`, `lint`, and `test` operate on this package. Production exports compiled
`dist/src`; development resolves source via the root loader.
Reports are derived data, not journal backups. See
[data-preservation guidance](../../docs/development.md#private-data-and-upgrades).