eleith/miniflux-digest
1
1 Fork 0
Code Issues Pull Requests Releases Activity

gemini/feature/multi-digest-architecture #61

Merged
eleith merged 18 commits from gemini/feature/multi-digest-architecture into main 2025-12-17 23:15:33 +00:00
Conversation 0 Commits 18 Files Changed 34
+1285 -276
eleith commented 2025-12-16 19:17:10 +00:00
Owner
Copy link
Copy source

Feature: Multi-Digest Architecture

Overview

This PR transforms miniflux-digest from a single-task reporter into a robust multi-digest engine. Users can now configure multiple, distinct digests (e.g., "Morning Tech", "Weekly Social") to run in parallel, each with its own schedule, content filters, and presentation logic.

Key Features

  • Multi-Digest Support: Define multiple digests in config.yaml, each with unique titles, schedules, and filters.
  • Priority-Based Filtering: Digests are processed in the order defined. The first digest to match an entry "claims" it, preventing duplicates across digests.
  • Enhanced Filtering: Precise control over content inclusion using:
  • Feed Titles (Exact)
  • Category Titles (Exact)
  • Site URLs (Prefix)
  • Entry URLs (Prefix)
  • Regex Patterns for all the above.
  • Context-Aware AI Categorization:
  • User Mode (Strict): If specific categories are provided in the config, the AI strictly sorts entries into those buckets without creating new primary groups.
  • Auto Mode (Open): If no categories are provided, the AI autonomously determines the best primary groups and sub-groups for the content.
  • Robust Configuration:
  • Strict Validation: The configuration loader now errors on unknown fields, preventing silent misconfiguration.
  • Smart Defaults: Defaults restored for view (defaults to "date") and compress (defaults to true).
  • Logic Checks: Validation ensures necessary fields (e.g., smtp.host, digest.host) are present if features like email are enabled.

Technical Changes

  • Core Refactor: internal/config, internal/filter, internal/manager, internal/processor, and internal/digest have been refactored to support the []DigestConfig structure.
  • Filter Engine: Implemented a Matcher in internal/filter that handles the deterministic matching logic.
  • AI Logic: Updated internal/digest/view/ai.go to support the new strict/auto modes and implemented logic to merge "Uncategorized" leftovers into a single group.
  • Mock Data: Refactored internal/testutil to generate realistic, filterable URLs, enabling accurate preview testing.

Fixes

  • Resolved an issue where feed_urls filtering failed in previews due to missing mock data.
  • Fixed potential file path issues by enforcing non-empty digest slugs during validation.
  • Corrected CONFIG.md to accurately reflect required vs. optional fields and default values.
# Feature: Multi-Digest Architecture ## Overview This PR transforms `miniflux-digest` from a single-task reporter into a robust multi-digest engine. Users can now configure multiple, distinct digests (e.g., "Morning Tech", "Weekly Social") to run in parallel, each with its own schedule, content filters, and presentation logic. ## Key Features * **Multi-Digest Support:** Define multiple digests in `config.yaml`, each with unique titles, schedules, and filters. * **Priority-Based Filtering:** Digests are processed in the order defined. The first digest to match an entry "claims" it, preventing duplicates across digests. * **Enhanced Filtering:** Precise control over content inclusion using: * Feed Titles (Exact) * Category Titles (Exact) * Site URLs (Prefix) * Entry URLs (Prefix) * Regex Patterns for all the above. * **Context-Aware AI Categorization:** * **User Mode (Strict):** If specific categories are provided in the config, the AI strictly sorts entries into those buckets without creating new primary groups. * **Auto Mode (Open):** If no categories are provided, the AI autonomously determines the best primary groups and sub-groups for the content. * **Robust Configuration:** * **Strict Validation:** The configuration loader now errors on unknown fields, preventing silent misconfiguration. * **Smart Defaults:** Defaults restored for `view` (defaults to "date") and `compress` (defaults to `true`). * **Logic Checks:** Validation ensures necessary fields (e.g., `smtp.host`, `digest.host`) are present if features like email are enabled. ## Technical Changes * **Core Refactor:** `internal/config`, `internal/filter`, `internal/manager`, `internal/processor`, and `internal/digest` have been refactored to support the `[]DigestConfig` structure. * **Filter Engine:** Implemented a `Matcher` in `internal/filter` that handles the deterministic matching logic. * **AI Logic:** Updated `internal/digest/view/ai.go` to support the new strict/auto modes and implemented logic to merge "Uncategorized" leftovers into a single group. * **Mock Data:** Refactored `internal/testutil` to generate realistic, filterable URLs, enabling accurate preview testing. ## Fixes * Resolved an issue where `feed_urls` filtering failed in previews due to missing mock data. * Fixed potential file path issues by enforcing non-empty digest slugs during validation. * Corrected `CONFIG.md` to accurately reflect required vs. optional fields and default values.
eleith added 4 commits 2025-12-16 19:17:11 +00:00
build: verify and fix Phase 1 configuration architecture 61e57243e3 
This commit stabilizes the 'Multi-Digest Architecture' feature (Phase 1).

Changes:
- Fix mocks in 'internal/app' and 'internal/testutil' to match new EmailService interface.
- Update 'ProcessDigest' calls in tests to respect new 'Digests' config slice.
- Update 'internal/config' tests to satisfy stricter validation rules (added missing 'host', 'view' fields).
- Verify 'Slugify' utility is present and tested.

Result:
- 'make lint', 'make test', and 'make build' now pass.
- The project is ready for Phase 2 (Filter Engine).
feat: implement Phases 2, 3, and 4 (Filter Engine, Processor Refactor, LLM Categories) dd82382bed 
This commit delivers the core logic for the Multi-Digest Architecture.

Phase 2 (Filter Engine):
- Implemented 'internal/filter' with exact/prefix/regex matching.
- Implemented 'internal/manager' for digest priority assignment.
- Updated 'models.Entry' to include 'SiteURL'.

Phase 3 (Processor Refactor):
- Updated 'ProcessDigest' to filter entries using 'DigestManager'.
- Updated 'main.go' to schedule jobs per digest index.

Phase 4 (LLM & Categorization):
- Added 'DefaultCategories' in 'internal/config'.
- Updated LLM prompts and service flow to respect user-defined or default categories.
- Updated mocks and tests across the board.

The application now supports configuring multiple digests with distinct filters and categories, and correctly assigns entries and prompts the LLM accordingly.
feat: implement Phase 5 (Output Structure) c46f9b4f8b 
Updates the archive generation to support the new folder structure: archives/<digest-slug>/<date>.html.

Changes:
- Updated 'OverviewTemplateData' to include 'DigestTitle' and 'DigestSlug'.
- Updated 'ArchiveService' to use 'DigestSlug' in file paths.
- Updated 'DigestService', 'View', and 'Processor' to propagate digest title/slug.
- Updated tests to match new structure and signatures.
fix(config): enhance validation logic ef94db719d 
This commit improves the configuration validation to prevent common user errors and runtime issues.

Changes:
- internal/config/config.go:
    - Added a check to ensure all digest titles result in unique slugs.
    - Added validation for all regex pattern fields in 'filters'.
    - Added 'validate:"dive"' to the 'Categories' field to ensure custom categories are validated.
- internal/config/config_test.go:
    - Added test cases for duplicate slugs, invalid regexes, and invalid categories.

This ensures that the application fails early with helpful errors if the configuration is invalid.
eleith added 1 commit 2025-12-16 19:28:52 +00:00
docs: restructure configuration documentation 6bf421be87 
This commit improves the documentation strategy by separating the comprehensive reference manual from the example configuration file.

Changes:
- Created 'docs/CONFIG.md': A detailed guide to all configuration options, including the new multi-digest and filtering features, validation rules, and default AI categories.
- Updated 'config.yaml.example': Removed verbose comments to create a clean, copy-paste friendly template.
- Updated 'README.md': Added a prominent link to 'docs/CONFIG.md' in the Setup section.
eleith added 1 commit 2025-12-16 19:49:31 +00:00
remove uneeded comment e2a105c0f5
eleith added 4 commits 2025-12-17 13:51:08 +00:00
use any 04ba0516a4
update preview to work with multiple digests 197f1698bd
make host not required again cfd6921fe6
make config validation disallow extra fields, rename site url for filter ed873f5d34
eleith added 1 commit 2025-12-17 14:42:36 +00:00
generate urls for mock entries f8a6902fdd
eleith added 2 commits 2025-12-17 16:16:03 +00:00
merge uncategorized groups if exist 689c02a7dd
update preview with miniflux to work with multiple digests 9220254157
eleith added 1 commit 2025-12-17 22:16:28 +00:00
refactor: enforce strict user categories in AI view 07870d00ee 
- Remove default category injection in config to distinguish between explicit user categories and auto-mode.
- Update AI prompts:
    - Rename initialGroupingPrompt to openGroupingPrompt.
    - Add strictGroupingPrompt for enforcing user-defined categories.
- Update AI grouping logic:
    - Use strictGroupingPrompt and skip consolidation if categories are provided.
    - Use openGroupingPrompt and perform consolidation if categories are empty.
- Add Uncategorized group merging logic.
- Add tests for strict user mode and merging logic.
eleith added 1 commit 2025-12-17 22:41:29 +00:00
fix: validate non-empty digest slugs and remove fallback c04c352dfa 
- Add validation in config.go to ensure digest titles produce non-empty slugs.
- Remove redundant fallback to 'default' slug in archive.go.
- Update tests to reflect these changes.
eleith added 3 commits 2025-12-17 23:06:57 +00:00
fix: restore default config values and update docs c51f66cc43 
- Restore logic to set default values for SMTP port (587) and Digest Compress (true) in config.
- Change Digest.Compress to *bool to correctly distinguish between unset and false values.
- Update CONFIG.md to reflect the removal of default categories in AI view.
- Fix logic that relied on defaults in processor and preview script.
fix: improve config defaults and validation for robustness 8a98b82095 
- Set default Digest.View to 'date' if unspecified.
- Make Digest.View optional in validation.
- Enforce that Digest.Host and SMTP.Host are present if Digest.Email is configured.
- Update documentation to reflect these changes and correct required/optional statuses.
- Update tests to cover new defaults and validation rules.
remove PROJECT.md
ci/woodpecker/push/build-docker Pipeline was successful
Details
8abbd67af4
eleith merged commit 8abbd67af4 into main 2025-12-17 23:15:33 +00:00
eleith deleted branch gemini/feature/multi-digest-architecture 2025-12-17 23:15:33 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
eleith (eleith)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: eleith/miniflux-digest#61