[web-fetch] add focused fetching tools #1

Merged

eleith merged 4 commits from feat/web-fetch into main 2026-10-16 00:14:49 +00:00

Conversation 0 Commits 4 Files Changed 54

+8167 -4

eleith-gitea-agent Bot commented 2026-10-13 04:20:18 +00:00

Copy link

Copy source

summary

Add an independent @eleith-pi/web-fetch extension exposing web_fetch and web_fetch_batch.

• Browser-fingerprinted requests via wreq-js 3.2.0; Defuddle 0.19.4 owns extraction/conversion and site-specific behavior; Linkedom 0.18.13 supplies the DOM.
• Markdown, HTML, JSON and raw output. JSON is validated without reserialization; no text mode, homemade Markdown parser, or secondary extraction/conversion pipeline.
• Private, uniquely named downloads under the environment-selected OS temporary directory; bounded, ordered batches with isolated item failures.
• Opinionated shared-frame rendering by default, externally overridable without changing execution/results.
• Shared-prefix on|off|toggle and test [URL] commands; test queues the ordinary agent/tool pipeline. No caller headers, browser/OS/proxy overrides, or new configuration beyond existing commandPrefix.
• Preserve upstream pi-smart-fetch 0.3.17 MIT attribution. No runtime dependency on that package or sibling extensions.

commits

1. Shared command helper: opt-in action arguments with case-preserved trailing input, plus tests.
2. Final web-fetch implementation, fixtures/tests, runtime dependencies and lockfile.
3. Standard per-package CI and root documentation.
4. Prepare initial version 0.1.0 by consuming the minor changeset and finalizing its changelog.

No discarded experimental implementations, generated builds/preview artifacts, or local planning files are committed.

verification

• Frozen install, build, lint, typecheck, formatting and diff checks pass.
• Full repository: 924 tests pass.
• Fetch + shared helpers: 254 tests pass on Node 22.19.0, also covered by the Node 26 repository run.
• Read-only reviews cover correctness/readability; regression tests exercise real Defuddle and native local HTTP/download paths, accepted-data limits, cancellation, metadata/budgets, renderer/history behavior and actual Pi SDK integration.
• Manual isolated-name preview tests passed for readable extraction, mixed batch success/failure, raw truncation and a readable temporary PNG download. A W3 PDF returned HTTP 403 and correctly saved no file.

limits and release

Native cancellation remains explicitly best-effort; settled JavaScript cancellation is not a guarantee of immediate native network shutdown. Accepted-data limits are not hard native-memory/CPU limits. Linux x64 is the runtime-verified platform.

This PR prepares version 0.1.0 but does not publish, create release tags, or switch personal installations. The initial minor changeset has been consumed. The packed archive was verified to contain exactly the production bundle, package manifest, README, LICENSE and CHANGELOG, with production tool/command names and no preview artifacts. Fetch/helper tests, lint, typecheck, formatting and frozen install pass after versioning.

The npm package does not exist yet: bootstrap with an authenticated first publish, then configure npm's GitHub trusted publisher for eleith/pi-extensions, workflow web-fetch.yml, with direct publishing allowed. Subsequent GitHub publication is triggered by matching @eleith-pi/web-fetch@<version> tags, not main-branch merges. Existing-version checks skip already-published versions. Disable upstream pi-smart-fetch before production cutover because it also registers web_fetch.

## summary Add an independent `@eleith-pi/web-fetch` extension exposing `web_fetch` and `web_fetch_batch`. - Browser-fingerprinted requests via wreq-js 3.2.0; Defuddle 0.19.4 owns extraction/conversion and site-specific behavior; Linkedom 0.18.13 supplies the DOM. - Markdown, HTML, JSON and raw output. JSON is validated without reserialization; no text mode, homemade Markdown parser, or secondary extraction/conversion pipeline. - Private, uniquely named downloads under the environment-selected OS temporary directory; bounded, ordered batches with isolated item failures. - Opinionated shared-frame rendering by default, externally overridable without changing execution/results. - Shared-prefix `on|off|toggle` and `test [URL]` commands; test queues the ordinary agent/tool pipeline. No caller headers, browser/OS/proxy overrides, or new configuration beyond existing `commandPrefix`. - Preserve upstream pi-smart-fetch 0.3.17 MIT attribution. No runtime dependency on that package or sibling extensions. ## commits 1. Shared command helper: opt-in action arguments with case-preserved trailing input, plus tests. 2. Final web-fetch implementation, fixtures/tests, runtime dependencies and lockfile. 3. Standard per-package CI and root documentation. 4. Prepare initial version 0.1.0 by consuming the minor changeset and finalizing its changelog. No discarded experimental implementations, generated builds/preview artifacts, or local planning files are committed. ## verification - Frozen install, build, lint, typecheck, formatting and diff checks pass. - Full repository: **924 tests** pass. - Fetch + shared helpers: **254 tests** pass on Node **22.19.0**, also covered by the Node 26 repository run. - Read-only reviews cover correctness/readability; regression tests exercise real Defuddle and native local HTTP/download paths, accepted-data limits, cancellation, metadata/budgets, renderer/history behavior and actual Pi SDK integration. - Manual isolated-name preview tests passed for readable extraction, mixed batch success/failure, raw truncation and a readable temporary PNG download. A W3 PDF returned HTTP 403 and correctly saved no file. ## limits and release Native cancellation remains explicitly **best-effort**; settled JavaScript cancellation is not a guarantee of immediate native network shutdown. Accepted-data limits are not hard native-memory/CPU limits. Linux x64 is the runtime-verified platform. This PR prepares version **0.1.0** but does not publish, create release tags, or switch personal installations. The initial minor changeset has been consumed. The packed archive was verified to contain exactly the production bundle, package manifest, README, LICENSE and CHANGELOG, with production tool/command names and no preview artifacts. Fetch/helper tests, lint, typecheck, formatting and frozen install pass after versioning. The npm package does not exist yet: bootstrap with an authenticated first publish, then configure npm's GitHub trusted publisher for `eleith/pi-extensions`, workflow `web-fetch.yml`, with direct publishing allowed. Subsequent GitHub publication is triggered by matching `@eleith-pi/web-fetch@<version>` tags, not main-branch merges. Existing-version checks skip already-published versions. Disable upstream pi-smart-fetch before production cutover because it also registers `web_fetch`.

eleith-gitea-agent Bot added 3 commits 2026-10-13 04:20:18 +00:00

[internal] support optional command action arguments fb69014c20

[web-fetch] add focused fetching tools and tests 0764adb831

[web-fetch] add CI and release wiring 3ae4814a4d

eleith added 1 commit 2026-10-15 23:38:47 +00:00

[web-fetch] prepare 0.1.0 release 33d7032b56

eleith merged commit 33d7032b56 into main 2026-10-16 00:14:49 +00:00

eleith deleted branch feat/web-fetch 2026-10-16 00:14:49 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

eleith-gitea-agent Bot eleith (eleith)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: eleith/pi-extensions#1