README.md (8165 bytes)
1 # SubRead for Android 2 3 Times an audiobook against its ebook, entirely on the phone, and writes the 4 `.srt` that [Hoshi Reader](https://github.com/HuangAntimony/Hoshi-Reader-Android)'s 5 read-along uses. Nothing is uploaded. 6 7 It also makes **videos + subtitles which can be played with any video player or 8 uploaded to YouTube**: an `.mp4` of the book's cover and the audio, with the 9 `.srt` beside it under the same name. 10 11 <p> 12 <img src="docs/screenshots/1-pick.png" width="300" alt="The first screen: what the app takes, how it works, and the two files to pick"> 13 <img src="docs/screenshots/2-outputs.png" width="300" alt="The outputs: subtitles, and video + subtitles with its size and frame rate"> 14 </p> 15 16 It is the same method as [subread.space](https://subread.space) and 17 [SubPlz](https://github.com/kanjieater/SubPlz): a small speech model transcribes 18 the audio, roughly; the transcript is aligned against the book; the subtitles 19 take their *timing* from the transcript and their *words* from the book. 20 21 ## Other apps 22 23 A reader or an audiobook player can ask SubRead for the `.srt` of a book and get 24 the file back: it starts SubRead with `space.subread.app.action.ALIGN`, the two 25 files as extras, and reads the subtitles from the result. The other app needs no 26 code of SubRead and no speech model of its own. The contract, with the extras 27 and the answer, is in [docs/intent-api.md](docs/intent-api.md). 28 29 ## Layout 30 31 | Path | What | 32 |---|---| 33 | `core/` | The alignment engine. Plain Kotlin, no Android - builds and tests with only a JDK | 34 | `app/` | The Android app: audio decoding, the whisper.cpp bridge, the long-running job, the UI | 35 | `third_party/whisper.cpp` | Pinned submodule | 36 | `tools/make_golden.py` | Generates test fixtures by running the reference Python implementation | 37 | `tools/make_synthetic_golden.py` | The same, for a language without a recorded transcript: makes one from a public-domain text (en, pt, es, ru fixtures) | 38 39 ## The engine 40 41 `core/` is a port of SubPlz's aligner (`ats.align`, `subplz.align.shift_align`), 42 checked stage by stage against the original's real output on real Whisper-tiny 43 transcripts: every cue across the fixtures comes out identical. 44 45 What is new is `AnchoredAligner`. The reference hands Biopython one chapter at a 46 time and needs gigabytes to do it, so it has to guess first which chapter of the 47 book each chapter of audio is. This aligns the whole book at once instead, by 48 anchoring on stretches unique to both texts and solving exactly only between 49 anchors: a 19-hour audiobook against its full text in a few seconds and a few 50 megabytes. `BookAligner` then uses the alignment itself to leave out text nobody 51 narrated - front matter, notes - instead of matching chapters. 52 53 ```bash 54 ./gradlew :core:test 55 ``` 56 57 ## The video 58 59 The picture of the video does not change, so almost none of it is encoded. 60 The device's H.264 encoder makes one key frame. `H264Still` writes each frame 61 after it by hand: a P slice in which each macroblock is skipped, about ten 62 bytes. One minute of these frames is written again and again with new time 63 stamps. AAC audio is copied; other audio is encoded to AAC once. 64 65 The result for a 10-hour book is about 60 MB of video at 720p and one frame a 66 second. The frame rate adds about 20 bytes for each frame. The size of the 67 picture changes only the key frame, one each minute. The sizes on offer are 68 those the device has an encoder for (1080p in software, more with a hardware 69 encoder). No Android encoder makes 4K or 8K H.264 today. 70 71 ## Building the app 72 73 CI builds it (`.github/workflows/build.yml`): the APK is an artifact of every 74 push to `main`. Locally it needs the Android SDK, NDK 29 and CMake 3.31. Without 75 an SDK, Gradle leaves `:app` out and `:core` still builds. 76 77 The speech model (`app/src/main/assets/models/ggml-tiny-q8_0.bin`, 43 MB, MIT 78 licence) is in the repository. It is the file of the whisper.cpp project, and 79 the workflow checks its hash. It is in git and not downloaded by the build, so 80 that a build needs no network: F-Droid asks for that, and the app itself has no 81 network permission. 82 83 The release key is not in the repository; CI reads it from Actions secrets. 84 85 ## Releases 86 87 Each release is a tag (`vMAJOR.MINOR.PATCH`) and a GitHub release with the signed 88 APK attached. Set `versionName` (the tag without the `v`) and `versionCode` in 89 `app/build.gradle.kts` first, and write `fastlane/metadata/android/en-US/changelogs/<versionCode>.txt`: 90 91 ```bash 92 ./gradlew :app:assembleRelease 93 git tag -a v0.2.0 -m "what changed" && git push origin main --tags 94 gh release create v0.2.0 app/build/outputs/apk/release/app-release.apk --notes "what changed" 95 ``` 96 97 `versionCode` must go up every release, or Android refuses the update. 98 99 Attach the APK two times: as `SubRead-<version>.apk`, and as `SubRead.apk`. Hoshi Reader 100 installs SubRead from `releases/latest/download/SubRead.apk`, a link that needs no version. 101 102 ### Google Play 103 104 ```bash 105 ./gradlew :app:bundleRelease -PplayStore=true 106 ``` 107 108 `-PplayStore=true` leaves the Ko-fi link out. Each other build has it. The texts, 109 the graphics and the answers to the Console's questions are in `play/`. 110 111 ### F-Droid 112 113 The build has what F-Droid asks for: free licences only, no network at build 114 time, no Google dependency list in the APK, and the version as plain numbers in 115 the build file. F-Droid reads the listing from `fastlane/metadata/android/`, and 116 finds new releases by their tags. `fdroid/space.subread.app.yml` is the recipe 117 for a merge request to [fdroiddata](https://gitlab.com/fdroid/fdroiddata). 118 F-Droid signs its builds with its own key, so an install from F-Droid and one 119 from GitHub or Google Play do not update each other. 120 121 ## Requirements 122 123 Android 8+, a 64-bit ARM processor with ARMv8.2 half-precision and dot-product 124 instructions (anything from 2018 on). Transcription runs at a small multiple of 125 real time, so a long book takes hours. The job runs only while the app is open 126 and keeps the screen on. An interrupted job continues from its last finished chunk. 127 128 ## More projects 129 130 - [SubRead](https://subread.space/): read along with an audiobook, in the browser. 131 Also [for Android](https://github.com/equwal/subread-android/releases/latest), 132 [for YouTube](https://github.com/equwal/subread-extension/releases/latest) 133 and [for KOReader](https://github.com/equwal/subread.koplugin). 134 - [SubRead Overlay](https://github.com/equwal/subread-overlay/releases/latest): subtitle lines over any Android media player. 135 - [SubRead Dictionary](https://github.com/equwal/subread-dictionary/releases/latest): a pop-up dictionary for Android that reads Yomitan dictionaries. 136 - [SubRead Anki](https://github.com/equwal/subread-anki): one tap makes an Anki card from any Android app. 137 - [Subrep](https://github.com/equwal/subrep-android/releases/latest): live captions of the sound of your phone. 138 - [Book Simulator](https://booksimulator.com/): a reading room for Aozora Bunko and Project Gutenberg books. 139 - [honjimaku.com](https://honjimaku.com/): subtitles for Japanese audiobooks. 140 - [sbm Sync](https://sbmsync.com/): your bookmarks, the same on every device, 141 with [sbm](https://github.com/equwal/sbm) for dmenu, 142 [sbm for Android](https://github.com/equwal/sbm-android/releases/latest) 143 and the [sbm add-on](https://github.com/equwal/sbm-extension/releases/latest) for Firefox and Chrome. 144 - [Rebind](https://github.com/equwal/rebind/releases): remap the hardware buttons of e-ink readers and Android, 145 with [Ink Recents](https://github.com/equwal/ink-recents/releases/latest), 146 [Ink Dim](https://github.com/equwal/ink-dim/releases/latest) 147 and [Ink Update](https://github.com/equwal/ink-update/releases/latest). 148 - [dickt.store](https://dickt.store/): language-learning tools, flashcards and web toys. 149 - [hentaibun.online](https://hentaibun.online/): learn kanbun and kobun. 150 - [Recently Written](https://recentlywritten.com/): the blog, and a list of [all projects](https://recentlywritten.com/projects.html). 151 152 ## Licence 153 154 AGPL-3.0: see `LICENSE`. You may use, change and host this, and you must give 155 your users the source of what you host. `NOTICE` has the licences of the work 156 this is built on (SubPlz, whisper.cpp, Whisper).