commit 2e030986c2e02ab759b0ffbf77f461b9fdd53b35 equwal <truex@equwal.com> 2026-09-21 16:43:59 -0700 Follow the audio player through SubRead Overlay KOReader has no notification access, so it cannot see the media session of the audio player. SubRead Overlay has that access and gives the state of the player to other apps with a content provider. The plugin reads it over JNI every two seconds and sets its clock from it. Start, pause, the seeks and "Move the audio to this page" control the player. A jump of the player position is treated like a seek, so the search reads the whole book once. A dictionary lookup pauses the player. The player starts again when the last lookup window closes, or on Resume after an external dictionary. The mark is now drawn by the plugin with the line boxes of the cue, the same as a highlight. crengine's own selection draws one line of a range only in vertical text. The end of the cue is found with a search for its tail, so the mark covers the whole cue and not only the anchor. The menu entry is first in the Tools menu, so it is on the first page. "Follow the audio player" in the menu turns the player read off. The plugin then runs its own clock as before.
README.md | 59 +++-- main.lua | 547 ++++++++++++++++++++++++++++++++++++--------- spec/player_state_spec.lua | 60 +++++ spec/run.lua | 2 +- spec/text_spec.lua | 10 + subread/android_player.lua | 114 ++++++++++ subread/player_state.lua | 46 ++++ subread/text.lua | 14 ++ 8 files changed, 725 insertions(+), 127 deletions(-)
diff --git a/README.md b/README.md index 1efa0e9..f7de58a 100644 --- a/README.md +++ b/README.md @@ -40,33 +40,59 @@ If the file is somewhere else, open **Tools → SubRead read-along** and tap the first line to select it with the file browser. The plugin remembers the choice for that book. -## Use it +## Use it on Android, with the audio player + +On Android the plugin reads the position of the audio player, so the book +follows the audio exactly. KOReader has no notification access, so it cannot +see the player itself. The [SubRead Overlay](https://github.com/equwal/subread-overlay) +app has that access and answers for KOReader. + +1. Install SubRead Overlay and give it notification access. Its panel does + not need to be on the screen. +2. Start the audiobook in your audio player (Voice, VLC, Smart AudioBook + Player, or any player with a media session). +3. In KOReader, open **Tools → SubRead read-along → Start read-along**. +4. The plugin reads the player every two seconds and turns the pages with it. + Start, Pause, the seeks and **Move the audio to this page** control the + player. + +A dictionary lookup pauses the player. It plays again when you close the +lookup window, or when you come back from an external dictionary app. + +**Follow the audio player** in the menu turns this off. The plugin then runs +its own clock, as below. + +## Use it with the own clock + +Without SubRead Overlay, or on a device that is not Android, the plugin runs +a clock beside your player. 1. Start the audiobook in your audio player. 2. In KOReader, open **Tools → SubRead read-along → Start read-along**. 3. The clock starts at the time of the first cue on the page you are reading, or at the position you had last time. -4. Use **Sync to this page**, or select the words the narrator is reading and - choose **SubRead: sync here**, to set the clock exactly. +4. Use **Sync the clock to this page**, or select the words the narrator is + reading and choose **SubRead: sync here**, to set the clock exactly. ### Controls -The controls dialog shows the clock, whether it runs, and the cue that is on. +The controls dialog shows the position, whether the audio plays, and the cue +that is on. | Control | What it does | |---|---| | Prev. cue / Next cue | Back or forward one cue | | `-10 s` / `+10 s` | Back or forward ten seconds | -| Start / Pause | Starts or stops the clock | -| Stop | Stops the clock and takes the mark off | -| Sync to this page | Sets the clock to the first cue on the page | -| Go to time… | Sets the clock to a position you type | -| Speed | 0.5 to 3.0, for a player that runs fast | +| Start / Pause | Starts or stops the player, or the own clock | +| Stop | Stops the follow and takes the mark off | +| Move the audio to this page | Seeks the player to the first cue on the page. With the own clock: sets the clock there. | +| Go to time… | Seeks to a position you type | +| Speed | 0.5 to 3.0, for the own clock. The player reports its own speed. | | Offset | Seconds to add to the subtitle times. Use a positive value when the audio file starts with an intro that is not in the book. | -| What time is this page? | Shows the time of the first cue on the page, so you can seek your audio player there | +| What time is this page? | Shows the time of the first cue on the page | -Select text in the book and use **SubRead: sync here** to set the clock to the -cue that holds those words. +Select text in the book and use **SubRead: sync here** to move the audio, or +the clock, to the cue that holds those words. The clock position, the speed and the offset are kept for each book. @@ -103,9 +129,9 @@ The screen is only redrawn when the cue changes. A timer tick on its own never redraws anything. A mark that moves inside the same page uses the light `ui` refresh. A page turn uses the `partial` refresh. -The mark is the same selection that KOReader draws for a full text search hit. -crengine draws it, so it costs no extra widget and it is never written into -your highlights. +The mark is drawn over the page with the same line boxes and the same shade +that KOReader uses for a highlight. It is never written into your +highlights. ## Limits @@ -118,7 +144,8 @@ your highlights. that does not match is skipped and the last place stays. * "What time is this page?" needs a next page, so it does not work on the last page of the book. -* The plugin cannot control your audio player. It only follows the clock. +* Without SubRead Overlay the plugin cannot see or control your audio player. + It only follows its own clock. ## Tests diff --git a/main.lua b/main.lua index 0ccf8a0..14d452e 100644 --- a/main.lua +++ b/main.lua @@ -8,11 +8,16 @@ with the audio player and keeps the place of the narration on the screen. Only crengine documents (EPUB, FB2, TXT, HTML) are supported. PDF and DjVu have no xpointer, so the place cannot be followed. +On Android the plugin reads the audio player through SubRead Overlay, an app +with notification access. See subread/android_player.lua. + @module koplugin.SubRead --]]-- local ButtonDialog = require("ui/widget/buttondialog") local DateTimeWidget = require("ui/widget/datetimewidget") +local Device = require("device") +local DictQuickLookup = require("ui/widget/dictquicklookup") local Dispatcher = require("dispatcher") local InfoMessage = require("ui/widget/infomessage") local Notification = require("ui/widget/notification") @@ -29,6 +34,7 @@ local T = require("ffi/util").template local Clock = require("subread.clock") local Cues = require("subread.cues") +local PlayerState = require("subread.player_state") local Srt = require("subread.srt") local Text = require("subread.text") @@ -60,6 +66,20 @@ local MIN_SLEEP = 0.05 -- book to its end, so this limits the cost when the book misses a stretch. local MISS_BACKOFF_MAX = 16 +-- Seconds between two reads of the audio player. The player reports its +-- position and speed, so the clock is exact between two reads. A read only +-- has to notice a pause, a seek or a new speed. +local PLAYER_POLL = 2 +-- Seconds between two reads while no player has a media session. +local PLAYER_POLL_IDLE = 5 +-- A player position this many seconds away from the clock is a seek. +local PLAYER_JUMP = 3 +-- Seconds between two checks for the end of a dictionary lookup. +local LOOKUP_POLL = 0.5 +-- The lookup window can open late. Give up the wait for it after this many +-- checks and start the player again. +local LOOKUP_WAIT_MAX = 20 + -- Characters of cue text shown in a dialog title. local TITLE_CHARS = 60 -- Characters kept from a selection when the user syncs by selecting text. @@ -71,6 +91,8 @@ local SETTING_FILE = "subread_srt_file" local SETTING_POSITION = "subread_position" local SETTING_SPEED = "subread_speed" local SETTING_OFFSET = "subread_offset" +-- Global setting: follow the audio player through SubRead Overlay. +local SETTING_FOLLOW_PLAYER = "subread_follow_player" local SubRead = WidgetContainer:extend{ name = "subread", @@ -90,10 +112,35 @@ function SubRead:init() self.scheduled = false self.tick_task = function() self:_tick() end + -- The bridge to the audio player. Only Android has one. + self.player = nil + if Device:isAndroid() then + local ok, module = pcall(require, "subread.android_player") + if ok then + self.player = module + else + logger.warn("SubRead: no player bridge:", module) + end + end + self.player_on = false -- true while the follow reads the player + self.player_status = nil -- "no_player" while the player has no media session + self.lookup_paused = false -- true while the player waits for a dictionary lookup + self.lookup_external = false + self.lookup_seen = false + self.lookup_waited = 0 + self.lookup_task = function() self:_checkLookupDone() end + self.skip_player_read = false + + self.mark = nil -- { start xpointer, end xpointer } of the cue on screen + + self:placeInToolsMenu() self.ui.menu:registerToMainMenu(self) if self:isSupported() then self:registerDispatcherActions() self:addToHighlightDialog() + -- ReaderView calls paintTo of each view module after the page + -- (readerview.lua:257). The mark is drawn there. + self.view:registerViewModule("subread", self) end end @@ -102,6 +149,23 @@ function SubRead:isSupported() return self.ui.rolling ~= nil end +--- True when the user wants the follow to read the audio player. +function SubRead:followsPlayer() + return self.player ~= nil and G_reader_settings:nilOrTrue(SETTING_FOLLOW_PLAYER) +end + +--- Puts the menu entry first in the Tools menu, so it is on the first page. +-- ReaderMenu sorts the entries with the cached order module +-- (readermenu.lua:350). An id that is not in the order goes to the end of +-- the menu, which is the second page. +function SubRead:placeInToolsMenu() + local ok, order = pcall(require, "ui/elements/reader_menu_order") + if ok and type(order) == "table" and type(order.tools) == "table" + and not util.arrayContains(order.tools, "subread") then + table.insert(order.tools, 1, "subread") + end +end + function SubRead:registerDispatcherActions() Dispatcher:registerAction("subread_controls", { category = "none", event = "SubReadShowControls", @@ -164,6 +228,7 @@ end function SubRead:onCloseWidget() self:_unschedule() + self:_endLookupWait() self.tick_task = nil end @@ -255,17 +320,102 @@ function SubRead:_now() return time.to_number(UIManager:getElapsedTimeSinceBoot()) end +--[[-- The audio player ]]-- + +--- Reads the state of the audio player through SubRead Overlay. +-- @return the state table of PlayerState.parse, or nil and an error name: +-- "not_installed", "no_notification_access" or "no_player". +function SubRead:_readPlayer() + local line = self.player.query() + if not line then return nil, "not_installed" end + return PlayerState.parse(line) +end + +--- Sets the clock from the audio player. +-- Returns true when the follow can go on with the player. Returns false when +-- the player cannot be reached at all; the follow then runs its own clock. +function SubRead:_syncClockToPlayer(now) + local state, err = self:_readPlayer() + if not state then + if err == "no_player" then + -- The player app is closed, or it has no session yet. Keep the + -- place and read again later. + self.player_status = err + self.clock:pause(now) + return true + end + self.player_on = false + self.player_status = nil + self:showMessage(self:playerProblemText(err)) + return false + end + self.player_status = nil + if state.position then + -- A seek in the player app can go backward. The normal search runs + -- forward from the current page, so treat a jump like a seek that + -- the user asked for here. + local jumped = math.abs(state.position - self.clock:getPosition(now)) > PLAYER_JUMP + self.clock:setSpeed(state.speed, now) + self.clock:seek(state.position, now) + if jumped then self:_manualSeek() end + end + if state.playing then + self.clock:start(now) + else + self.clock:pause(now) + end + return true +end + +function SubRead:playerProblemText(err) + if err == "no_notification_access" then + return _("SubRead Overlay has no notification access, so the audio player cannot be read. Give it the access in its app. Until then the plugin runs its own clock.") + end + return _("SubRead Overlay was not found. Install it and give it notification access, so the plugin can follow the audio player. Until then the plugin runs its own clock.") +end + +--- Sends play or pause to the player when the follow reads it. +function SubRead:_playerCall(method) + if not self.player_on then return end + self.player.call(method) + self.skip_player_read = true +end + +--- Moves the clock, and the player when the follow reads it. +-- The player takes a moment for a seek or a play. The read that comes at +-- once after would bring the old state back, so that one read is skipped. +function SubRead:_seekTo(position, now) + self.clock:seek(position, now) + if self.player_on then + self.player.call("seek", PlayerState.seekArgument(self.clock:getPosition(now))) + self.skip_player_read = true + end +end + +--[[-- Start, pause and stop ]]-- + function SubRead:start() if not self:checkSupported() then return end if not self:loadSubtitles() then return end local now = self:_now() - if self.clock:getPosition(now) <= 0 then - local index = self:cueIndexForCurrentPage() - if index then - self.clock:seekCueTime(self.cues:get(index).start, now) + if self:followsPlayer() then + self.player_on = true + if self:_syncClockToPlayer(now) then + self:_playerCall("play") end end - self.clock:start(now) + if not self.player_on then + if self.clock:getPosition(now) <= 0 then + local index = self:cueIndexForCurrentPage() + if index then + self.clock:seekCueTime(self.cues:get(index).start, now) + end + end + end + -- The player takes a moment to start. The next read confirms the state. + if self.player_status ~= "no_player" then + self.clock:start(now) + end -- The reader may have moved the view since the last pause, so drop the -- lower limit and let the search start at the page that is shown. self.last_xpointer = nil @@ -276,8 +426,15 @@ function SubRead:start() end function SubRead:pause() - self.clock:pause(self:_now()) - self:_unschedule() + local now = self:_now() + self.clock:pause(now) + self:_playerCall("pause") + if self.player_on then + -- Keep the reads, so a start from the player app is seen. + self:_schedule(now, self.clock:getCueTime(now)) + else + self:_unschedule() + end self:saveState() end @@ -285,13 +442,22 @@ function SubRead:stop() if self.clock:isRunning() then self.clock:pause(self:_now()) end + self:_endLookupWait() + self.player_on = false + self.player_status = nil self:_unschedule() - if self:isSupported() and self.ui.document then - self.ui.document:clearSelection() + if self.mark then + self.mark = nil + UIManager:setDirty(self.view.dialog, "ui") end self:saveState() end +--- True when the follow runs: the clock, or the reads of the player. +function SubRead:isActive() + return self.clock:isRunning() or self.player_on +end + function SubRead:toggle() if self.clock:isRunning() then self:pause() @@ -314,7 +480,7 @@ end --- Moves the clock and shows the new place at once. function SubRead:seekAndShow(position) local now = self:_now() - self.clock:seek(position, now) + self:_seekTo(position, now) self:_manualSeek() self:_tick() end @@ -327,7 +493,7 @@ function SubRead:skipCues(count) index = index + count if index < 1 then index = 1 end if index > self.cues:count() then index = self.cues:count() end - self.clock:seekCueTime(self.cues:get(index).start, now) + self:_seekTo(self.cues:get(index).start + self.clock.offset, now) self:_manualSeek() self:_tick() end @@ -347,6 +513,10 @@ function SubRead:_tick() self.scheduled = false if not self.cues then return end local now = self:_now() + if self.player_on and not self.skip_player_read then + self:_syncClockToPlayer(now) + end + self.skip_player_read = false local cue_time = self.clock:getCueTime(now) local index = self.cues:findByTime(cue_time) if index and index ~= self.current_index then @@ -356,11 +526,16 @@ function SubRead:_tick() self:_schedule(now, cue_time) end ---- Sleeps until the next cue starts, or MAX_SLEEP, whichever is first. +--- Sleeps until the next cue starts, or the next read of the player, or +--- MAX_SLEEP, whichever is first. function SubRead:_schedule(now, cue_time) self:_unschedule() - if not self.clock:isRunning() then return end local delay = MAX_SLEEP + if self.player_on then + delay = self.player_status == "no_player" and PLAYER_POLL_IDLE or PLAYER_POLL + elseif not self.clock:isRunning() then + return + end local next_index = self.cues:nextAfter(cue_time) if next_index then local wait = self.clock:realSecondsUntilCueTime(self.cues:get(next_index).start, now) @@ -426,8 +601,12 @@ function SubRead:_locate(index) if hits then for _, hit in ipairs(hits) do if hit.start and self:_isAtOrAfter(floor, hit.start) then - self.located[index] = { hit.start, hit["end"] } - return hit.start, hit["end"] + local pos1 = hit["end"] + if Text.len(anchor) < Text.len(cue.norm) then + pos1 = self:_cueEnd(cue, hit.start, pos1) + end + self.located[index] = { hit.start, pos1 } + return hit.start, pos1 end end end @@ -435,6 +614,37 @@ function SubRead:_locate(index) return nil end +-- The text between the start of a cue and the end of its tail may hold ruby +-- text, so it can be longer than the cue. A hit further away than this +-- factor is a tail that belongs to another line. +local CUE_END_SLACK = 2 + +--- Finds the end of the cue text, so the mark covers the whole line. +-- The anchor is only the first characters of the cue. The tail of the cue is +-- searched forward from the current page, and the first hit after the start +-- that lies close to it gives the end. When the tail is not found, the end +-- of the anchor stays. +function SubRead:_cueEnd(cue, pos0, anchor_end) + local tail = Text.tail(cue.norm) + if not tail then return anchor_end end + local document = self.ui.document + local hits = document:findText(tail, SEARCH_FROM_CURRENT_PAGE, SEARCH_FORWARD, + true, self.view.state.page, false, SEARCH_MAX_HITS, SEARCH_FLAGS) + if not hits then return anchor_end end + local limit = Text.len(cue.norm) * CUE_END_SLACK + Text.TAIL_LENGTH + for _, hit in ipairs(hits) do + if hit.start and hit["end"] and self:_isAtOrAfter(pos0, hit.start) then + local between = document:getTextFromXPointers(pos0, hit["end"], false) + if between and Text.len(Text.normalize(between)) <= limit then + return hit["end"] + end + -- The first hit after the start is already too far. + return anchor_end + end + end + return anchor_end +end + --- Brings the cue on screen and marks it. function SubRead:showCue(index) local cue = self.cues:get(index) @@ -459,30 +669,34 @@ function SubRead:showCue(index) if self.controls_dialog then self:showControls() end end ---- Turns the page if needed and draws the mark. --- The mark is the crengine selection, the same one that KOReader draws for a --- full text search hit (readersearch.lua:946). crengine draws it itself, so --- it costs no extra widget, it survives a page turn, and it is never written --- into the book's annotations. +--- Turns the page if needed and asks for a redraw with the mark. function SubRead:drawCue(pos0, pos1) local document = self.ui.document local was_visible = document:isXPointerInCurrentPage(pos0) if not was_visible then self.ui.rolling:onGotoXPointer(pos0) end - if pos1 then - document:getTextFromXPointers(pos0, pos1, true) - end + self.mark = pos1 and { pos0, pos1 } or nil -- A page turn needs a partial refresh. A mark that moves inside the same -- page only needs the light "ui" refresh. UIManager:setDirty(self.view.dialog, was_visible and "ui" or "partial") end ---- Draws the cue that is on again, after something else used the selection. -function SubRead:redrawCurrentCue() - if not self.current_index then return end - local pos0, pos1 = self:_locate(self.current_index) - if pos0 then self:drawCue(pos0, pos1) end +--- Draws the mark over the page. ReaderView calls this after each paint of +--- the page, so the mark survives a page turn and is never written into the +--- annotations of the book. The boxes are the line segments of the cue on +--- the screen, the same ones that ReaderHighlight draws for a highlight. +--- crengine's own selection is not used: in vertical text it draws one line +--- of a range only. +function SubRead:paintTo(bb, x, y) + if not self.mark then return end + local boxes = self.ui.document:getScreenBoxesFromPositions(self.mark[1], self.mark[2], true) + if not boxes then return end + for _, box in ipairs(boxes) do + -- The same drawing as the "lighten" highlight style + -- (readerview.lua:673). The boxes are in screen coordinates. + bb:darkenRect(box.x, box.y, box.w, box.h, self.view.highlight.lighten_factor) + end end --[[-- Page and time ]]-- @@ -505,11 +719,7 @@ function SubRead:cueIndexForCurrentPage() if not self.cues then return nil end local page_text = self:currentPageText() if not page_text or page_text == "" then return nil end - local index = self.cues:findIndexForText(page_text) - -- getTextFromXPointers with draw_selection false can drop the mark, so - -- put it back. - if self.clock:isRunning() then self:redrawCurrentCue() end - return index + return self.cues:findIndexForText(page_text) end function SubRead:showPageTime() @@ -538,16 +748,98 @@ function SubRead:syncToPage() self:syncToCue(index) end ---- Sets the clock to the start of a cue. +--- Sets the clock to the start of a cue. When the follow reads the player, +--- the audio moves there too. function SubRead:syncToCue(index) local cue = self.cues:get(index) if not cue then return end local now = self:_now() - self.clock:seekCueTime(cue.start, now) + if self:followsPlayer() and not self.player_on then + -- "Move the audio to this page" before a start: begin the follow of + -- the player, so the seek reaches it. + self.player_on = true + self:_syncClockToPlayer(now) + end + local position = cue.start + self.clock.offset + self:_seekTo(position, now) self:_manualSeek() self:_tick() - self:showNotification(T(_("Synced to %1"), - datetime.secondsToClock(cue.start + self.clock.offset, false))) + local clock_text = datetime.secondsToClock(position, false) + self:showNotification(self.player_on and T(_("Audio moved to %1"), clock_text) + or T(_("Synced to %1"), clock_text)) +end + +--- Text of the action that sets the place: it moves the audio when the +--- follow reads the player, else the clock of the plugin. +function SubRead:syncToPageText() + if self:followsPlayer() then + return _("Move the audio to this page") + end + return _("Sync the clock to this page") +end + +--[[-- Dictionary lookup ]]-- + +--- ReaderDictionary sends this before it looks a word up +--- (readerdictionary.lua:1429). The player pauses while the user reads. +function SubRead:onWordLookedUp() + if not self.player_on or self.lookup_paused then return end + local state = self:_readPlayer() + if not state or not state.playing then return end + self:_playerCall("pause") + self.clock:pause(self:_now()) + self.lookup_paused = true + self.lookup_seen = false + self.lookup_waited = 0 + -- An external dictionary is another app. KOReader gets a Resume event + -- when the user comes back (device/android/device.lua:179). + self.lookup_external = Device:canExternalDictLookup() + and G_reader_settings:isTrue("external_dict_lookup") + if not self.lookup_external then + UIManager:scheduleIn(LOOKUP_POLL, self.lookup_task) + end +end + +--- Starts the player again when the last lookup window is closed. +-- DictQuickLookup keeps its open windows in window_list +-- (dictquicklookup.lua:176) and sends no event when the last one closes. +function SubRead:_checkLookupDone() + if not self.lookup_paused then return end + local open = #DictQuickLookup.window_list > 0 + if open then + self.lookup_seen = true + elseif not self.lookup_seen then + -- The lookup runs before the window opens. Wait for the window. + self.lookup_waited = self.lookup_waited + 1 + if self.lookup_waited >= LOOKUP_WAIT_MAX then + self:_resumeAfterLookup() + return + end + end + if open or not self.lookup_seen then + UIManager:scheduleIn(LOOKUP_POLL, self.lookup_task) + return + end + self:_resumeAfterLookup() +end + +function SubRead:_endLookupWait() + UIManager:unschedule(self.lookup_task) + self.lookup_paused = false +end + +function SubRead:_resumeAfterLookup() + self:_endLookupWait() + if not self.player_on then return end + self:_playerCall("play") + self.clock:start(self:_now()) + self:_tick() +end + +function SubRead:onResume() + if self.lookup_paused and self.lookup_external then + self:_resumeAfterLookup() + end end --- Sets the clock from a piece of text that the user selected. @@ -605,8 +897,16 @@ end function SubRead:statusText() local now = self:_now() local clock_text = datetime.secondsToClock(self.clock:getPosition(now), false) - local line = self.clock:isRunning() and T(_("Clock: %1, running"), clock_text) - or T(_("Clock: %1, paused"), clock_text) + local line + if self.player_on and self.player_status == "no_player" then + line = _("No audio player is open") + elseif self.player_on then + line = self.clock:isRunning() and T(_("Player: %1, playing"), clock_text) + or T(_("Player: %1, paused"), clock_text) + else + line = self.clock:isRunning() and T(_("Clock: %1, running"), clock_text) + or T(_("Clock: %1, paused"), clock_text) + end if self.cues and self.current_index then local cue = self.cues:get(self.current_index) if cue then @@ -661,7 +961,7 @@ function SubRead:showControls() end }, }, { - { text = _("Sync to this page"), + { text = self:syncToPageText(), callback = again(function() self:syncToPage() end) }, { text = _("Go to time…"), callback = function() @@ -763,7 +1063,11 @@ function SubRead:onSubReadToggle() if not self:checkSupported() then return true end if not self:loadSubtitles() then return true end self:toggle() - self:showNotification(self.clock:isRunning() and _("Read-along running") or _("Read-along paused")) + if self.player_on then + self:showNotification(self.clock:isRunning() and _("Audio playing") or _("Audio paused")) + else + self:showNotification(self.clock:isRunning() and _("Read-along running") or _("Read-along paused")) + end return true end @@ -783,79 +1087,102 @@ function SubRead:subtitleMenuText() end function SubRead:addToMainMenu(menu_items) + local items = { + { + text_func = function() return self:subtitleMenuText() end, + keep_menu_open = true, + callback = function(touchmenu_instance) + if self:checkSupported() then + self:chooseSubtitleFile(touchmenu_instance) + end + end, + hold_callback = function(touchmenu_instance) + self:setSubtitleFile(nil) + touchmenu_instance:updateItems() + end, + separator = true, + }, + { + text_func = function() + return self:isActive() and _("Read-along controls") + or _("Start read-along") + end, + callback = function(touchmenu_instance) + touchmenu_instance:onClose() + if not self:checkSupported() then return end + if not self:isActive() then + -- The controls would cover the line that the narrator reads. + -- They open with the same menu entry when the user wants them. + self:start() + if self:isActive() then + self:showNotification(_("Read-along started. The controls are in this menu.")) + end + return + end + self:showControls() + end, + }, + { + text_func = function() return self:syncToPageText() end, + callback = function(touchmenu_instance) + touchmenu_instance:onClose() + self:syncToPage() + end, + }, + { + text = _("Go to time…"), + keep_menu_open = true, + callback = function() self:showGoToTime() end, + }, + { + text = _("What time is this page?"), + keep_menu_open = true, + callback = function() self:showPageTime() end, + separator = true, + }, + } + if self.player then + table.insert(items, { + text = _("Follow the audio player"), + help_text = _("Reads the position of the audio player through the SubRead Overlay app, which needs notification access. Start, pause and the seeks then control the player. When off, the plugin runs its own clock."), + checked_func = function() return G_reader_settings:nilOrTrue(SETTING_FOLLOW_PLAYER) end, + callback = function() + G_reader_settings:flipNilOrTrue(SETTING_FOLLOW_PLAYER) + self:stop() + end, + separator = true, + }) + end menu_items.subread = { - -- The key is not in reader_menu_order.lua, so MenuSorter puts the - -- entry where the hint says (menusorter.lua:161-182). + -- placeInToolsMenu puts the id first in the Tools order. The hint is + -- for a user order file that does not list the id + -- (menusorter.lua:161-182). sorting_hint = "tools", text = _("SubRead read-along"), - sub_item_table = { - { - text_func = function() return self:subtitleMenuText() end, - keep_menu_open = true, - callback = function(touchmenu_instance) - if self:checkSupported() then - self:chooseSubtitleFile(touchmenu_instance) - end - end, - hold_callback = function(touchmenu_instance) - self:setSubtitleFile(nil) - touchmenu_instance:updateItems() - end, - separator = true, - }, - { - text_func = function() - return self.clock:isRunning() and _("Read-along controls") - or _("Start read-along") - end, - callback = function(touchmenu_instance) - touchmenu_instance:onClose() - if not self:checkSupported() then return end - if not self.clock:isRunning() then - -- The controls would cover the line that the narrator reads. - -- They open with the same menu entry when the user wants them. - self:start() - if self.clock:isRunning() then - self:showNotification(_("Read-along started. The controls are in this menu.")) - end - return - end - self:showControls() - end, - }, - { - text = _("Go to time…"), - keep_menu_open = true, - callback = function() self:showGoToTime() end, - }, - { - text = _("What time is this page?"), - keep_menu_open = true, - callback = function() self:showPageTime() end, - separator = true, - }, - { - text_func = function() - return T(_("Player speed: %1"), string.format("%.2f", self.clock.speed)) - end, - keep_menu_open = true, - callback = function(touchmenu_instance) - self:showSpeed() - if touchmenu_instance then touchmenu_instance:updateItems() end - end, - }, - { - text_func = function() - return T(_("Audio offset: %1 s"), string.format("%+d", self.clock.offset)) - end, - keep_menu_open = true, - callback = function(touchmenu_instance) - self:showOffset() - if touchmenu_instance then touchmenu_instance:updateItems() end - end, - }, - }, + sub_item_table = items, } + table.insert(items, { + text_func = function() + return T(_("Player speed: %1"), string.format("%.2f", self.clock.speed)) + end, + -- The player reports its speed, so the setting is for the own clock. + enabled_func = function() return not self:followsPlayer() end, + keep_menu_open = true, + callback = function(touchmenu_instance) + self:showSpeed() + if touchmenu_instance then touchmenu_instance:updateItems() end + end, + }) + table.insert(items, { + text_func = function() + return T(_("Audio offset: %1 s"), string.format("%+d", self.clock.offset)) + end, + keep_menu_open = true, + callback = function(touchmenu_instance) + self:showOffset() + if touchmenu_instance then touchmenu_instance:updateItems() end + end, + }) end return SubRead diff --git a/spec/player_state_spec.lua b/spec/player_state_spec.lua new file mode 100644 index 0000000..ccad19c --- /dev/null +++ b/spec/player_state_spec.lua @@ -0,0 +1,60 @@ +local PlayerState = require("subread.player_state") + +describe("PlayerState.parse", function() + it("reads a playing report", function() + local state, err = PlayerState.parse("playing=1;position=96153;speed=1.5;package=de.ph1b.audiobook") + assert.is_nil(err) + assert.is_true(state.playing) + assert.near(96.153, state.position, 1e-9) + assert.equals(1.5, state.speed) + assert.equals("de.ph1b.audiobook", state.package) + end) + + it("reads a paused report", function() + local state = PlayerState.parse("playing=0;position=0;speed=1.0;package=p") + assert.is_false(state.playing) + assert.equals(0, state.position) + end) + + it("gives no position for a player without one", function() + local state = PlayerState.parse("playing=1;position=-1;speed=1.0;package=p") + assert.is_true(state.playing) + assert.is_nil(state.position) + end) + + it("returns the error name of a problem line", function() + local state, err = PlayerState.parse("error=no_player") + assert.is_nil(state) + assert.equals("no_player", err) + state, err = PlayerState.parse("error=no_notification_access") + assert.is_nil(state) + assert.equals("no_notification_access", err) + end) + + it("rejects an empty or broken line", function() + local state, err = PlayerState.parse(nil) + assert.is_nil(state) + assert.equals("empty", err) + state, err = PlayerState.parse("") + assert.equals("empty", err) + state, err = PlayerState.parse("hello") + assert.equals("malformed", err) + state, err = PlayerState.parse("playing=1;position=abc") + assert.equals("malformed", err) + end) + + it("falls back to speed 1 for a missing or zero speed", function() + assert.equals(1.0, PlayerState.parse("playing=1;position=5").speed) + assert.equals(1.0, PlayerState.parse("playing=1;position=5;speed=0").speed) + end) + + it("round trips a position through the seek argument", function() + for _, seconds in ipairs({ 0, 0.001, 1, 96.153, 3600.5, 12345.6789 }) do + local argument = PlayerState.seekArgument(seconds) + assert.is_not_nil(argument:match("^%d+$")) + local back = PlayerState.parse("playing=0;position=" .. argument) + assert.near(seconds, back.position, 0.0005) + end + assert.equals("0", PlayerState.seekArgument(-3)) + end) +end) diff --git a/spec/run.lua b/spec/run.lua index c383bd6..c77d230 100644 --- a/spec/run.lua +++ b/spec/run.lua @@ -38,7 +38,7 @@ function _G.it(name, body) end _G.setup, _G.teardown, _G.before_each, _G.after_each = function(f) f() end, function() end, nil, nil -for _, name in ipairs({ "text", "srt", "cues", "clock" }) do +for _, name in ipairs({ "text", "srt", "cues", "clock", "player_state" }) do dofile(here .. name .. "_spec.lua") end print(string.format("%d passed, %d failed", passed, failed)) diff --git a/spec/text_spec.lua b/spec/text_spec.lua index 2f5cf58..f074310 100644 --- a/spec/text_spec.lua +++ b/spec/text_spec.lua @@ -67,6 +67,16 @@ describe("SubRead text", function() end end) + it("gives the tail of a long cue and nil for a short one", function() + assert.is_nil(Text.tail("short cue")) + assert.is_nil(Text.tail("exactly12chr")) + assert.equals("brown fox jump", Text.tail("The quick brown fox jump", 14)) + local cue = "とりあえず書いてしまってから、その作品が向いてそうな雑誌なり出版社に持ち込む。" + local tail = Text.tail(cue) + assert.equals(12, Text.len(tail)) + assert.equals("誌なり出版社に持ち込む。", tail) + end) + it("ellipsizes long text only", function() assert.equals("short", Text.ellipsize("short", 10)) assert.equals("abcde\226\128\166", Text.ellipsize("abcdefgh", 5)) diff --git a/subread/android_player.lua b/subread/android_player.lua new file mode 100644 index 0000000..093b0b6 --- /dev/null +++ b/subread/android_player.lua @@ -0,0 +1,114 @@ +--[[-- +Bridge to the audio player on Android, through SubRead Overlay. + +KOReader has no notification access, so it cannot see the media session of +the player. SubRead Overlay (space.subread.overlay) has that access and +gives the state of the player to other apps with a content provider. This +module calls that provider over JNI, with the helpers of the KOReader +launcher (android.lua). It must only be loaded on Android. + +Each function returns nil when the provider cannot be reached: SubRead +Overlay is not installed, or the call threw a Java exception. +--]]-- + +local android = require("android") +local ffi = require("ffi") +local logger = require("logger") + +local AndroidPlayer = {} + +-- The content provider of SubRead Overlay. See its PlayerProvider. +AndroidPlayer.URI = "content://space.subread.overlay.player/state" +-- Name of the app that the user must install. +AndroidPlayer.PACKAGE = "space.subread.overlay" + +local NULL = ffi.cast("void *", nil) + +--- Clears a pending Java exception. Returns true when there was one. +-- A pending exception makes the next JNI call abort the process, so each +-- call that can throw is followed by this check. +local function clearException(jni, what) + local env = jni.env + if env[0].ExceptionCheck(env) == ffi.C.JNI_TRUE then + env[0].ExceptionClear(env) + logger.warn("SubRead: Java exception in", what) + return true + end + return false +end + +--- Runs fn(jni, resolver, uri) with the ContentResolver of the activity +--- and the Uri of the provider. Frees the local references after. +local function withResolver(fn) + return android.jni:context(android.app.activity.vm, function(jni) + local env = jni.env + local resolver = jni:callObjectMethod(android.app.activity.clazz, + "getContentResolver", "()Landroid/content/ContentResolver;") + local uri_text = env[0].NewStringUTF(env, AndroidPlayer.URI) + local uri = jni:callStaticObjectMethod("android/net/Uri", "parse", + "(Ljava/lang/String;)Landroid/net/Uri;", uri_text) + local result = fn(jni, resolver, uri) + env[0].DeleteLocalRef(env, uri) + env[0].DeleteLocalRef(env, uri_text) + env[0].DeleteLocalRef(env, resolver) + return result + end) +end + +--- Reads the state line of the player. Returns nil when there is no provider. +function AndroidPlayer.query() + local ok, line = pcall(withResolver, function(jni, resolver, uri) + local env = jni.env + -- ContentResolver.query(Uri, String[], String, String[], String). + -- It returns null when no app has the provider. + local cursor = jni:callObjectMethod(resolver, "query", + "(Landroid/net/Uri;[Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;Ljava/lang/String;)Landroid/database/Cursor;", + uri, NULL, NULL, NULL, NULL) + if clearException(jni, "query") or cursor == nil then return nil end + local line + if jni:callBooleanMethod(cursor, "moveToFirst", "()Z") then + local text = jni:callObjectMethod(cursor, "getString", + "(I)Ljava/lang/String;", ffi.new("int32_t", 0)) + if text ~= nil then + line = jni:to_string(text) + env[0].DeleteLocalRef(env, text) + end + end + jni:callVoidMethod(cursor, "close", "()V") + env[0].DeleteLocalRef(env, cursor) + return line + end) + if not ok then + logger.warn("SubRead: player query failed:", line) + return nil + end + return line +end + +--- Sends a command to the player: "play", "pause", or "seek" with the +--- position in milliseconds as a string. Returns true when the provider +--- took the call. +function AndroidPlayer.call(method, argument) + local ok, done = pcall(withResolver, function(jni, resolver, uri) + local env = jni.env + local method_text = env[0].NewStringUTF(env, method) + local argument_text = argument and env[0].NewStringUTF(env, tostring(argument)) or NULL + -- ContentResolver.call(Uri, String, String, Bundle). It throws when + -- no app has the provider. + local bundle = jni:callObjectMethod(resolver, "call", + "(Landroid/net/Uri;Ljava/lang/String;Ljava/lang/String;Landroid/os/Bundle;)Landroid/os/Bundle;", + uri, method_text, argument_text, NULL) + local failed = clearException(jni, "call " .. method) + if bundle ~= nil then env[0].DeleteLocalRef(env, bundle) end + if argument_text ~= NULL then env[0].DeleteLocalRef(env, argument_text) end + env[0].DeleteLocalRef(env, method_text) + return not failed + end) + if not ok then + logger.warn("SubRead: player call failed:", done) + return false + end + return done +end + +return AndroidPlayer diff --git a/subread/player_state.lua b/subread/player_state.lua new file mode 100644 index 0000000..b77736d --- /dev/null +++ b/subread/player_state.lua @@ -0,0 +1,46 @@ +--[[-- +State line of the audio player, as SubRead Overlay reports it. + +Pure Lua. No KOReader dependency, so it can be unit tested on a PC. + +SubRead Overlay is an Android app with notification access. It reads the +media session of the player and gives one line to other apps: + + playing=1;position=96153;speed=1.0;package=de.ph1b.audiobook + +The position is in milliseconds, for the moment of the query. A problem is +one line `error=<reason>`. +--]]-- + +local PlayerState = {} + +--- Parses a state line. +-- @return a table { playing, position (seconds), speed, package }, or nil +-- and an error name. A report without a position has position nil. +function PlayerState.parse(line) + if type(line) ~= "string" or line == "" then return nil, "empty" end + local fields = {} + for key, value in line:gmatch("([%w_]+)=([^;]*)") do + fields[key] = value + end + if fields.error then return nil, fields.error end + if not fields.playing or not fields.position then return nil, "malformed" end + local position_ms = tonumber(fields.position) + if not position_ms then return nil, "malformed" end + local speed = tonumber(fields.speed) or 1.0 + if speed <= 0 then speed = 1.0 end + return { + playing = fields.playing == "1", + position = position_ms >= 0 and position_ms / 1000 or nil, + speed = speed, + package = fields.package, + } +end + +--- Converts a position in seconds to the milliseconds string of a seek call. +function PlayerState.seekArgument(seconds) + if seconds < 0 then seconds = 0 end + return string.format("%d", math.floor(seconds * 1000 + 0.5)) +end + +return PlayerState diff --git a/subread/text.lua b/subread/text.lua index 648c34a..0e91724 100644 --- a/subread/text.lua +++ b/subread/text.lua @@ -143,6 +143,20 @@ function Text.anchors(s, lengths) return out end +-- Length, in characters, of the tail that finds the end of a cue. +Text.TAIL_LENGTH = 12 + +--- Returns the last characters of a cue text, to find its end in the book. +-- Returns nil when the text is not longer than the tail: the anchor then +-- already covers the whole cue. +function Text.tail(s, length) + length = length or Text.TAIL_LENGTH + local norm = Text.normalize(s) + local count = Text.len(norm) + if count <= length then return nil end + return Text.sub(norm, count - length + 1, count) +end + --- Cuts a string to a maximum number of characters, for a dialog title. function Text.ellipsize(s, max_chars) local norm = Text.normalize(s)