eleith/miniflux-digest
1
1 Fork 0
Code Issues Pull Requests Releases Activity

gemini/refactor/generalize-chunk-processing #60

Merged
eleith merged 12 commits from gemini/refactor/generalize-chunk-processing into main 2025-09-16 18:35:24 +00:00
Conversation 0 Commits 12 Files Changed 11
+825 -292
eleith commented 2025-09-16 18:33:38 +00:00
Owner
Copy link
Copy source
  • refactor ai.go to have smaller method sizes
  • chunk entries for primary grouping, sub grouping and summary llm jobs
  • generate content with response schema method for llm service
  • process chunks of inputs in llm util
- refactor ai.go to have smaller method sizes - chunk entries for primary grouping, sub grouping and summary llm jobs - generate content with response schema method for llm service - process chunks of inputs in llm util
eleith added 12 commits 2025-09-16 18:33:38 +00:00
refactor(digest): Add generic parallel processing helper 0ae0f0efe2 
Adds a generic llm.ProcessInChunks function to run jobs on data chunks in parallel.

Refactors GroupAIEntries to use this new helper function, replacing its own concurrency management.
refactor(digest): Parallelize summarization and sub-grouping 8ba0024b5b 
Refactors `ProcessPrimaryGroupWithAI` to use the `llm.ProcessInChunks` helper.

For large entry groups, summarization and sub-grouping tasks are now chunked and run in parallel to prevent timeouts.

Tests for this function have been updated to cover the new chunking and failure-handling logic.
test for util llm functions f8c75a7c11
refactor(digest): Rename job entry limit constant 6d14ef5168 
Renames `MaxEntriesForSummarization` to `MaxEntriesPerJob` to reflect its general use in all AI processing jobs.
refactor(ai): Generalize chunk processing dd0779827b 
Remove redundant `processSingleChunk` function and simplify
`ProcessPrimaryGroupWithAI` to always process chunks concurrently. This
improves readability and reduces code duplication.
refactor(ai): Abstract LLM entry preparation and execution c54324a2ac 
Abstract out reusable logic for preparing LLM entries and executing
LLM requests. This improves readability and reduces code duplication.
refactor(ai): Refactor consolidatePrimaryGroups to use executeLLMRequestAndParse 59422b3f96 
Refactor the `consolidatePrimaryGroups` function to use the new
`executeLLMRequestAndParse` function. This improves readability and
reduces code duplication.
refactor(ai): Refactor getSummaryForGroup to use executeLLMRequestAndParse 70dbbd2bb8 
Refactor the `getSummaryForGroup` function to use the new
`executeLLMRequestAndParse` function. This improves readability and
reduces code duplication.
refactor(llm): Move ExecuteLLMRequestAndParse to llm/util.go a5ba746e79 
Move the `ExecuteLLMRequestAndParse` function from
`internal/digest/view/ai.go` to `internal/llm/util.go`. This improves
readability and reduces code duplication.
move helper method in llm service 1f33b2e452
improve prompt 1f403ebd93
refactor large methods in view/ai.go
ci/woodpecker/push/build-docker Pipeline was successful
Details
be5c5c3301
eleith merged commit be5c5c3301 into main 2025-09-16 18:35:24 +00:00
eleith deleted branch gemini/refactor/generalize-chunk-processing 2025-09-16 18:35:24 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
eleith (eleith)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: eleith/miniflux-digest#60