Clinton Billedeaux
cea352b576
Summarize local PDF, Word, and plain text files via a new DocumentService (PdfPig + DocumentFormat.OpenXml), and add a Custom summary mode that uses the user's own instructions as the system prompt. Update the console UI, transcript saving, and README to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
157 lines
6 KiB
Markdown
157 lines
6 KiB
Markdown
# YouTube & Document Summarizer
|
||
|
||
A .NET 10 console app that summarizes YouTube videos *and* local documents —
|
||
text, Markdown, CSV, PDF, Word, OpenDocument, and Excel — using an LLM
|
||
(OpenAI or a local Ollama model). One prompt accepts either a YouTube URL or
|
||
a file path; everything else (summarize, save, display) runs through the
|
||
same pipeline.
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- [.NET 10 SDK](https://dotnet.microsoft.com/download)
|
||
- [**yt-dlp**](https://github.com/yt-dlp/yt-dlp) on your `PATH` — used to fetch
|
||
YouTube metadata and captions. No YouTube API key needed.
|
||
|
||
```bash
|
||
pip install yt-dlp
|
||
```
|
||
|
||
- **Local Ollama** (recommended, free) or an **OpenAI API key**
|
||
|
||
---
|
||
|
||
## Setup
|
||
|
||
```bash
|
||
cd summarize
|
||
|
||
# Edit appsettings.json with your LLM endpoint/key (see Configuration below)
|
||
nano appsettings.json
|
||
|
||
dotnet restore
|
||
dotnet run
|
||
```
|
||
|
||
---
|
||
|
||
## Using it
|
||
|
||
The app loops on a single prompt:
|
||
|
||
```
|
||
Enter a YouTube URL or a local file path (or 'q' to quit):
|
||
```
|
||
|
||
- **YouTube**: paste any `watch?v=...`, `youtu.be/...`, `/shorts/...`, or
|
||
`/embed/...` URL (or a bare 11-character video ID).
|
||
- **Local document**: paste or type a path to an existing file. Supported
|
||
types: `.txt` `.md` `.csv` `.pdf` `.docx` `.odt` `.xlsx`. The kind of input
|
||
is auto-detected — no need to say which one it is.
|
||
- `q` quits.
|
||
|
||
For each input you're asked:
|
||
|
||
1. **Save transcript to file?** — `y/n`, defaults to **yes** on Enter.
|
||
Saves the extracted transcript/document text plus the summary to
|
||
`~/Downloads/transcripts/`.
|
||
2. **Choose summary mode** — an arrow-key list, Standard pre-selected:
|
||
- **Standard** — detailed bullet-point summary (the default; Enter picks it)
|
||
- **Personal Filter** — a 1–2 sentence summary plus an ACT / MONITOR /
|
||
IGNORE relevance verdict against personal priorities (time, finances,
|
||
health, family, service to others)
|
||
- **Custom** — write your own instructions on the spot (multi-line; finish
|
||
with a blank line). Replaces the built-in prompt entirely, for when you
|
||
want something other than a summary or a verdict — a table, an
|
||
extraction task, a specific format, etc.
|
||
|
||
Ctrl+C cancels the current operation without killing the app. The terminal
|
||
tab title tracks progress (`Summarize` while idle, `Summarize <title>` while
|
||
working on something).
|
||
|
||
---
|
||
|
||
## Configuration Reference
|
||
|
||
All settings live in `appsettings.json` (bind to `AppSettings` in
|
||
[AppSettings.cs](AppSettings.cs)):
|
||
|
||
| Key | Description | Default |
|
||
| --- | --- | --- |
|
||
| `LLM:BaseUrl` | API endpoint — `https://api.openai.com/v1` for OpenAI, `http://localhost:11434/v1` for Ollama | `https://api.openai.com/v1` |
|
||
| `LLM:ApiKey` | API key (any non-empty value works for Ollama) | *(empty — required for OpenAI)* |
|
||
| `LLM:Model` | Chat model, e.g. `gpt-4o-mini` (OpenAI) or `qwen3:14b` (Ollama) | `gpt-4o-mini` |
|
||
| `LLM:MaxTokens` | Max tokens in the summary response | `1500` |
|
||
| `LLM:TimeoutSeconds` | Max time to wait per API call | `100` |
|
||
| `Summarizer:ChunkWordLimit` | Word count above which a transcript/document is split into chunks (map-reduce, see below). Must stay comfortably above ~200 — the fixed chunk overlap — or chunking will error on short inputs. | `3000` |
|
||
| `Summarizer:ShowTranscript` | Print the full extracted text before summarizing | `false` |
|
||
|
||
Any value can be overridden with an environment variable using `__` as the
|
||
section separator (handy for CI/containers):
|
||
|
||
```bash
|
||
export LLM__ApiKey="sk-..."
|
||
export LLM__Model="gpt-4o"
|
||
dotnet run
|
||
```
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Program.cs
|
||
│ Main loop → auto-detects YouTube URL vs. local file path
|
||
│
|
||
├── YouTubeService — shells out to yt-dlp
|
||
│ ├── ExtractVideoId() — URL parsing
|
||
│ ├── GetVideoMetadataAsync() — video title/channel/date/duration
|
||
│ └── GetTranscriptAsync() — caption download + VTT/SRT/timedtext parsing
|
||
│
|
||
├── DocumentService — reads a local file
|
||
│ ├── BuildMetadata() — title/type/modified-date from the file
|
||
│ └── ExtractTextAsync() — dispatches by extension:
|
||
│ .txt/.md/.csv → read directly
|
||
│ .pdf → PdfPig
|
||
│ .docx/.xlsx → DocumentFormat.OpenXml
|
||
│ .odt → hand-rolled zip + XML (ODF content.xml)
|
||
│
|
||
│ Both feed the same VideoMetadata / VideoTranscript shapes into:
|
||
│
|
||
├── SummarizerService
|
||
│ ├── SummarizeAsync() — routes to single-pass or chunked, per mode
|
||
│ │ (Standard / Personal Filter / Custom)
|
||
│ ├── SinglePassSummarize() — one LLM call for short inputs
|
||
│ └── ChunkedSummarizeAsync() — map-reduce for long inputs
|
||
│
|
||
├── TranscriptFileService — saves transcript + summary to a .txt file
|
||
│
|
||
└── ConsoleRenderer — all terminal output (Spectre.Console):
|
||
banner, prompts, progress spinners, and
|
||
the final summary panel/grid
|
||
```
|
||
|
||
### Source Quality Transparency
|
||
|
||
The app tracks how the text was obtained and flags it accordingly:
|
||
|
||
| Source | Badge | Warning shown? |
|
||
| --- | --- | --- |
|
||
| Owner-published captions | `✓ Owner-published` | No |
|
||
| Community-contributed captions | `✓ Community captions` | Minor note |
|
||
| Auto-generated captions (ASR) | `~ Auto-generated` | Yes — accuracy caveat |
|
||
| No captions (metadata only) | `✗ Metadata only` | Yes — limited accuracy |
|
||
| Local document | `✓ Local document` | No |
|
||
|
||
### Long-Input Strategy
|
||
|
||
Transcripts/documents exceeding `ChunkWordLimit` words use a **map-reduce**
|
||
approach — split into overlapping chunks (200-word overlap preserves context
|
||
at boundaries), each chunk summarized independently, then combined into one
|
||
coherent result. This applies to every mode, including Custom (the combine
|
||
step gets an auto-generated wrapper prompt asking the model to merge the
|
||
partial responses while still honoring your instructions). Handles
|
||
hour-long videos, long reports, and multi-sheet spreadsheets without hitting
|
||
model context limits.
|