discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens
thecopybookhare-cmd/lingua-miner · source
Snapshot 0 · 2026-11-04T12:47:28Z · Python · MIT
Content fingerprint
993cdf11376ad0d0ae5385b2a713c980631583bec5c05aef20a3b29c1e14ff3aGitHub description at capture
Local Migaku-style flashcard miner — learn languages from video. Word-by-word subtitles, one-click Anki cards, 100% offline.
README at capture
# ⛏️ LinguaMiner [](https://github.com/thecopybookhare-cmd/lingua-miner/actions)  [](https://github.com/thecopybookhare-cmd/lingua-miner/releases/latest)  **Local, Migaku-style flashcard miner — learn languages from the videos you love.** Watch anything with word-by-word interactive subtitles, click a word, and get an Anki card with the audio of the sentence, a video frame, the sentence + neural translation, and the word's lemma, part of speech, dictionary senses and frequency. Everything runs **100% locally** — no accounts, no paid APIs.  *One click on an unknown word → dictionary → a card with the sentence audio, the video frame and the translation, ready for Anki.* **▶️ [Watch the 3-minute walkthrough](https://youtu.be/_Iu7WcnNXAo)** — every feature, narrated. ### And this is the card you get  Front carries the sentence, a short **animated clip** of the moment (not a frozen frame) and the sentence audio. Back adds the word and sentence in your language, plus where it came from and how common the word is. ## Features - 🎙️ **Transcription** with Whisper (fine-tuned Catalan model, or generic large-v3 / small) — or use the video's own `.srt` / YouTube subtitles. - 🖱️ **One-click mining**: click any subtitle word (or drag to select an expression) → editable Anki card with segment audio (ffmpeg-trimmed), video frame, sentence + translation. - 🌍 **Thirteen languages**: Catalan, **Spanish**, French, English, German, European Portuguese, Italian, Dutch, Russian, Mandarin Chinese, Cantonese, **Japanese** and **Korean**. Neural translation runs offline via CTranslate2 (OPUS-MT, or NLLB-200 for pairs OPUS doesn't cover) into **Spanish or English** — pick the base under ⚙️ *Settings → Translate to*. Catalan, French, German, Portuguese, Italian and English translate into either; the rest are English-base only, since no →es model exists for them (or, for Spanish, because it would be translating Spanish into itself). Each language downloads its own translator, spaCy model, Wiktionary glosses and Piper voice on first use. [Requesting a language?](docs/adding-a-language.md) — that page lists what has to exist and where each candidate currently stands. - 🎨 **Migaku-style word states**: red = new · orange = learning · no mark = known · grey = ignored — synced back from your Anki review intervals. A header chip shows the % of the video you already know. - ⭐ **Smart recommendations (i+1)**: a word is highlighted only when it is unknown *and* frequent enough to be worth your time *and* not a proper noun — the frequency floor follows your vocabulary level, so you get the words that pay off, not every name you happen not to recognise. - 🌱 **Bring the vocabulary you already have**: coming from Migaku, jpdb, LingQ or an Anki deck? Upload their export (txt, csv, tsv or json) or point at a deck, and those words start out known instead of red. It never overwrites a status you set yourself, and words the export marks as *learning* or *new* stay that way. - 📺 **Watch online**: paste a YouTube / direct / HLS link and it streams instantly (yt-dlp resolves the best format); cards cut audio + image straight from the stream. Quality selector included. Public broadcasters work too: [3Cat](https://www.3cat.cat/3cat/) for Catalan, [RTVE Play](https://www.rtve.es/play/) for Spanish (series, documentaries and radio, free, no account needed, with the show's own Spanish subtitles when it has them). Some programmes can only be watched from some countries; the app tells you so instead of opening a silent player. - 📖 **Read books and articles too**: import a `.txt` or `.epub` and it becomes a reading session with the same clickable words, the same popup and the same i+1 highlighting — a LingQ-style reader that sends cards to Anki with the sentence read aloud by Piper. Pages remember where you left off, **sentence mode** puts one sentence in front of you with its audio, and turning the page can mark whatever you didn't touch as known — off by default, always telling you how many, always undoable. - 🎧 **Condensed audio**: export an MP3 with *only* the dialogue of an episode (a 50-min show becomes ~20 min) for passive listening on your phone — the immersion-method staple. - 🎙️ **Podcasts & radio too**: audio-only sources work as sessions, not just video. - 📋 **Words panel**: every lemma in the video grouped by frequency band; bulk-mark the N most frequent words of the language as known. - 📖 **Dictionaries, four layers deep**: Wiktionary glosses in your base language (Spanish *or* English), Apertium bilingual senses, the neural translation of the word in context, and **your own dictionaries** — import any StarDict (`.ifo`) or Yomitan (`.zip`) file and its definitions show up in the same popup, Migaku-style. - 🔊 Neural TTS pronunciation (Piper), IPA, conjugation tables, remappable shortcuts, daily DB backups. - 📱 **Installable PWA + share mode**: serve the app on your LAN or [Tailscale](https://tailscale.com) so a friend or your phone can use it in a browser (off by default, full access — share only with people you trust). | Your library | Words panel | |---|---| |  |  | ### The same thing, on books  Import a `.txt` or `.epub` and it becomes a reading session: same colours, same popup, same cards — with the sentence read aloud by a neural voice instead of cut from a soundtrack.  *Sentence mode puts one sentence in front of you, and reads it out loud.* ## Install Everyone installs **their own copy** — nothing to host. You don't need Python or ffmpeg beforehand: the installer brings `uv` (which provides Python) and `static-ffmpeg` covers ffmpeg if it's missing. > **Heads up:** installing means pasting one line into a terminal. That's the > only technical step — after it, LinguaMiner is a normal app you open from > Launchpad or the Start menu. Never used a terminal? See > [the step-by-step guide](docs/install-guide.md). ### One command (clones + installs) **macOS / Linux**: ```bash curl -LsSf https://raw.githubusercontent.com/thecopybookhare-cmd/lingua-miner/main/bootstrap.sh | bash ``` **Windows** — no terminal: download [**LinguaMiner-Setup.cmd**](https://github.com/thecopybookhare-cmd/lingua-miner/releases/latest/download/LinguaMiner-Setup.cmd) and double-click it. Windows will ask once whether to run it (*More info → Run anyway*). Prefer PowerShell? This does the same: ```powershell irm https://raw.githubusercontent.com/thecopybookhare-cmd/lingua-miner/main/bootstrap.ps1 | iex ``` Clones the repo into `~/LinguaMiner` and installs everything. *(Prefer to clone yourself? Use the manual steps below.)* ### Manual (from the project folder) **macOS / Linux** ```bash ./install.sh # installs everything; on macOS also creates LinguaMiner.app ./run.sh # or open LinguaMiner.app (Mac) ``` **Windows** (PowerShell, inside the project folder) ```powershell powershell -ExecutionPolicy Bypass -File install.ps1 .\run.bat ``` The installer creates the venv (Python 3.12) and downloads the translator, dictionary and spaCy model. The Whisper model (~3 GB) downloads itself the first time you transcribe. App data lives in your OS's standard folder (`Application Support` on Mac, `AppData\Roaming` on Windows, `~/.local/share` on Linux). ### Docker No installer and no Python on your machine — one command, then open <http://localhost:8977>: ```bash docker run -d --name linguaminer --restart unless-stopped \ -p 127.0.0.1:8977:8977 \ -v linguaminer-data:/data \ --add-host=host.docker.internal:host-gateway \ ghcr.io/thecopybookhare-cmd/lingua-miner:latest ``` Or `docker compose up -d` with the [`compose.yaml`](compose.yaml) in this repo. Built for both Intel/AMD and ARM (Apple Silicon, Raspberry Pi). - **Your data** — library, queued cards, downloaded models — lives in the `linguaminer-data` volume, so updating the image never touches it. The Whisper model (≈3 GB) downloads into it the first time you transcribe. - **Keep the `127.0.0.1:`** in the port. Inside Docker, anyone who can reach that port has full control of the app. - **Anki** keeps running on your computer as usual; cards reach it through `host.docker.internal`. With Docker Desktop (Mac, Windows) that should be all. On Linux, AnkiConnect only listens on localhost, which the container can't reach: in Anki → Tools → Add-ons → AnkiConnect → Config, set `"webBindAddress": "172.17.0.1"` (Docker's bridge). Don't use `0.0.0.0` — that opens your collection to your whole network. If Anki can't be reached, cards queue up and send themselves once it can. - **Videos you already have:** add `-v ~/Videos:/videos:ro`, or open them from the browser. - **Updating:** `docker pull ghcr.io/thecopybookhare-cmd/lingua-miner:latest`, then remove and recreate the container. The in-app updater doesn't apply here. ### Anki (for the cards) 1. Install [Anki](https://apps.ankiweb.net) 2. In Anki: Tools → Add-ons → Get Add-ons → code `2055492159` (AnkiConnect) 3. Restart Anki and keep it open while you mine If Anki is closed, cards queue up and send themselves when it opens. ## Updating Open ⚙️ **Settings → Check for updates**. It tells you what's new and updates in place. > **If you installed before 5 September 2026, do this once.** Versions up to > 1.33.1 shipped a broken update check: the installer clones with `--depth 1`, > and in a shallow clone the command that counts new commits *fails* instead of > returning zero — so the app told you that you were up to date when you > weren't. Fixing it required shipping a new version, which the broken checker > couldn't fetch. In your LinguaMiner folder (`~/LinguaMiner` if you used the > one-line installer): > > ```bash > git fetch --unshallow origin main > git pull --ff-only origin main > ``` > > After that the in-app updater works on its own. See > [v1.33.2](https://github.com/thecopybookhare-cmd/lingua-miner/releases/tag/v1.33.2). ## Usage **→ [Full walkthrough with clips of every feature](docs/tutorial.md)** On first launch a **Getting started** card walks you through these same three steps and shows what is still downloading. The interface follows your system language (English, Spanish or Catalan) — change it under ⚙️ any time. 1. Open a local file (mp4/mkv/mp3…), or paste a **YouTube / direct / HLS** URL and hit **Watch online**. For offline HD, **Import** downloads with a real progress bar. 2. Hit **Transcribe** (use `small` for a quick test) — or use the video's own subtitles if available. 3. Click any word in the subtitle (or drag-select an expression). 4. Review/edit the card in the popup and press **⏎**. Reading instead of watching? **Open a book** takes a `.txt` or `.epub` and steps 3 and 4 are identical — see [the reader section of the walkthrough](docs/tutorial.md#5-reading-books-and-articles). **Shortcuts (Migaku map):** `A`/`←` previous sentence · `D`/`→` next · `S`/`↓` replay · `Q` mine word under cursor · `⇧Q` open card editor · `1-5` word status · `W` hide subs · `E` dual subtitles · `K` condensed playback · `G` subtitle browser · `P` auto-pause · `F` fullscreen · `space` play/pause · `⏎` send card · `Esc` close. All remappable in ⚙️. ## Troubleshooting | Problem | Fix | |---|---| | Blank white window on launch | The app now tells you what failed instead of showing an empty window. If you still get one, you are on an old version — update, and check `desktop.log` in your app data folder | | "Anki closed" badge | Open Anki with AnkiConnect installed; the queue sends itself | | Video won't play | `.mkv` files are remuxed to mp4 automatically on import | | Empty translations | Run `./install.sh` again (downloads the translator) | | Slow transcription | Pick the `small` model in the selector. A 50-min episode takes a few minutes before the progress bar moves — press *Transcribe* once and let it run | New to all this? The [step-by-step install guide](docs/install-guide.md) covers the same ground with no assumed terminal experience. **Hit a bug?** ⚙️ → *Report a problem* builds a report with your version, your system and the last lines of the log, then opens GitHub with it already written. Nothing is sent on its own — you read it, edit it, and decide. If the app won't even open, its error screen has the same button. ## Architecture FastAPI + SQLite + vanilla JS (no build step). Pieces: [faster-whisper](https://github.com/SYSTRAN/faster-whisper) with [projecte-aina/faster-whisper-large-v3-ca-3catparla](https://huggingface.co/projecte-aina/faster-whisper-large-v3-ca-3catparla), [softcatala/translate-cat-spa](https://huggingface.co/softcatala/translate-cat-spa) and OPUS-MT CTranslate2 models per language, Apertium bilingual dictionaries, spaCy, wordfreq, yt-dlp, Piper TTS, AnkiConnect. Language profiles live in [`app/languages.py`](app/languages.py) — adding a language is mostly adding one entry there. ## Development ```bash uv sync # exact versions from uv.lock, dev tools included uv run ruff check app/ tests/ # lint uv run pytest tests/ # tests ``` Dependencies are pinned in `uv.lock`, and the installers, the updater, Docker and CI all install exactly those versions. To move them forward, run `uv lock --upgrade` (or `uv lock --upgrade-package yt-dlp`) and let CI and the install test run. The weekly YouTube check also tries the newest yt-dlp, so you hear about a breaking release before it reaches the lock. See [CONTRIBUTING.md](CONTRIBUTING.md). CI runs lint + tests on Linux/macOS/Windows on every push, and a separate workflow runs the real installers on all three every week. ## License & fair use Code under the [MIT](LICENSE) license © 2026 thecopybookhare. ⚠️ **Personal and educational use only.** The tool plays third-party content for language study; respect copyright and each platform's terms. Don't redistribute downloaded content.