Repository navigation
feat(library): mark search results already in a Calibre library - #1377
Conversation
Searching for a book you already own gives no hint that you own it, so the easiest way to end up with a second copy is to not remember you have the first. Shelfmark now reads a Calibre metadata.db, read only, and marks matching search results and the details dialog with an "In library" badge. It is off by default and lives in Settings, General beside the existing Library URL, with a test button that reports how many books it indexed. Matching goes from most to least confident: a shared external id, then an ISBN in either form, then fuzzy title tokens plus the author surname. The check fails open, so an unreadable database degrades the badge and never blocks a search, and entries are cached for ten minutes with an early refresh when the database file changes. The provider interface takes more libraries than Calibre: add a module with the LibraryProvider shape and list it in all_providers(). Nothing above that function knows which libraries exist.
The provider and the ownership field shipped with the index tests but without their own. These cover the parts most likely to bite: reading a metadata.db on a read-only mount, seeing uncheckpointed WAL writes, falling back to an immutable open and then to a snapshot when the file is locked, failing open when it cannot be read at all, and the fingerprint tracking both the database and its WAL. Plus that search results carry the library field only while a check is enabled.
"Dune" is a subset of "Dune Messiah", so a title-token match claimed the shelf held a book it did not, and the badge then told the reader they owned something they had never bought. A match is now rejected when the shelf title adds a word the search did not ask for. Only real words count, so volume numbers and structural words do not disqualify an entry, and the entry's series and author words are subtracted because shelf titles routinely repeat them: "Alex Cross 25: Cross Kill" is still Cross Kill. A trailing parenthetical is dropped from the compared title for the same reason, since that is where series and edition notes usually sit. Collections are kept rather than rejected, and reported as their own holding so the reader is told the book is inside something larger rather than owned on its own. Ownership therefore reports "owned" or "collection" per format instead of a bare boolean. Measured against a 146 book library: the false positives go, and the books the previous rule found are all still found.
|
Pushed b1e7388, which fixes a false-ownership bug I found after opening this, plus 04b71df adding the provider and search-payload tests the first pass missed. The matcher compared title tokens in one direction only, so a shorter title was a subset of any longer one sharing its words. Against a real 146 book library that meant "Dune" reported as owned when the shelf only held "Dune Messiah", and the same for "It" against "It Chapter Two" and "Children of Dune". For a badge that is a wrong answer on screen; for anything gating an acquisition on it, the book never gets fetched. A match is now rejected when the shelf title adds a word the search did not ask for. Getting that to behave on real data took three constraints:
Collections are kept rather than rejected, and reported as their own holding, so an omnibus badges as "In a collection" rather than claiming the volume is owned outright. Two cases it still gets wrongStating them rather than leaving them to be found:
Both have the same principled fix: Calibre exposes Verification
|
|
Awesome ! |
…v1.4.0) (#283) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/calibrain/shelfmark](https://github.com/calibrain/shelfmark) | minor | `v1.3.15` → `v1.4.0` | --- ### Release Notes <details> <summary>calibrain/shelfmark (ghcr.io/calibrain/shelfmark)</summary> ### [`v1.4.0`](https://github.com/calibrain/shelfmark/releases/tag/v1.4.0) [Compare Source](calibrain/shelfmark@v1.3.15...v1.4.0) ##### What's Changed - build(deps): bump the python-deps group with 2 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1302](calibrain/shelfmark#1302) - fix: keep polling queued Real-Debrid torrents by [@​mvanhorn](https://github.com/mvanhorn) in [#​1303](calibrain/shelfmark#1303) - fix(bypass): keep Anna's Archive's aa\_ddg\_check so clearance replays by [@​jfmlima](https://github.com/jfmlima) in [#​1305](calibrain/shelfmark#1305) - fix(postprocess): attach unmatched chaptered audio files to existing book group ([#​1176](calibrain/shelfmark#1176)) by [@​amasen02](https://github.com/amasen02) in [#​1309](calibrain/shelfmark#1309) - build(deps): bump the gh-actions group with 3 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1310](calibrain/shelfmark#1310) - build(deps): bump the npm-deps group in /src/frontend with 4 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1308](calibrain/shelfmark#1308) - build(deps): bump python from `cae66f2` to `cad9a2c` by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1307](calibrain/shelfmark#1307) - build(deps): bump the docker-base-image-digests group with 2 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1306](calibrain/shelfmark#1306) - Deep-link Search By mode via URL hash by [@​nfvelten](https://github.com/nfvelten) in [#​1311](calibrain/shelfmark#1311) - build(deps): bump the python-deps group across 1 directory with 4 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1327](calibrain/shelfmark#1327) - Keep default filters out of the URL hash by [@​nfvelten](https://github.com/nfvelten) in [#​1314](calibrain/shelfmark#1314) - feat(download): add Blackhole torrent handoff by [@​atirna](https://github.com/atirna) in [#​1312](calibrain/shelfmark#1312) - feat(naming): add {FirstAuthor} template token by [@​viniciuspx](https://github.com/viniciuspx) in [#​1322](calibrain/shelfmark#1322) - fix: share rotating log file handlers by [@​eikopf](https://github.com/eikopf) in [#​1316](calibrain/shelfmark#1316) - fix(prowlarr): skip indexers in Prowlarr failure back-off by [@​jfmlima](https://github.com/jfmlima) in [#​1324](calibrain/shelfmark#1324) - fix: prevent Anna's Archive download countdown resets by preserving browser sessions by [@​broglea](https://github.com/broglea) in [#​1325](calibrain/shelfmark#1325) - build(deps): bump the docker-base-image-digests group with 2 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1328](calibrain/shelfmark#1328) - build(deps-dev): bump the npm-deps group in /src/frontend with 4 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1329](calibrain/shelfmark#1329) - build(deps-dev): bump vitest from 4.1.11 to 5.0.0 in /src/frontend by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1330](calibrain/shelfmark#1330) - fix(irc): search by surname, and rank the answer by author ([#​1331](calibrain/shelfmark#1331)) by [@​Kukkerem](https://github.com/Kukkerem) in [#​1332](calibrain/shelfmark#1332) - fix(irc): rank a surname-only result as partial, not wrong ([#​1332](calibrain/shelfmark#1332)) by [@​calibrain](https://github.com/calibrain) in [#​1334](calibrain/shelfmark#1334) - feat(sources): add Libgen as a direct catalogue search source by [@​klaidliadon](https://github.com/klaidliadon) in [#​1326](calibrain/shelfmark#1326) - refactor: make direct download provider-driven by [@​TomJansen](https://github.com/TomJansen) in [#​1337](calibrain/shelfmark#1337) - fix(sources): restore Direct Download search errors and language matches by [@​calibrain](https://github.com/calibrain) in [#​1339](calibrain/shelfmark#1339) - fix(sources): send a Referer when fetching libgen ads.php pages by [@​klaidliadon](https://github.com/klaidliadon) in [#​1340](calibrain/shelfmark#1340) - Feature: Add Download counts to search result displays and Download sidebar by [@​RoninTech](https://github.com/RoninTech) in [#​1336](calibrain/shelfmark#1336) - Extract archives when zip/rar are enabled as supported formats by [@​funkypenguin](https://github.com/funkypenguin) in [#​1343](calibrain/shelfmark#1343) - fix(download): complete consumed Blackhole handoffs by [@​atirna](https://github.com/atirna) in [#​1345](calibrain/shelfmark#1345) - Add configurable word separator for naming templates by [@​viniciuspx](https://github.com/viniciuspx) in [#​1333](calibrain/shelfmark#1333) - build(deps): bump the python-deps group across 1 directory with 5 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1344](calibrain/shelfmark#1344) - fix: unbreak main and follow up on the Blackhole handoff review by [@​calibrain](https://github.com/calibrain) in [#​1346](calibrain/shelfmark#1346) - build(deps): bump python-socketio from 5.16.4 to 5.17.0 in the python-deps group by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1347](calibrain/shelfmark#1347) - Added the ability to sort direct search results by Most downloads by [@​RoninTech](https://github.com/RoninTech) in [#​1351](calibrain/shelfmark#1351) - feat: Add TorBox client support and settings integration by [@​marcelorodrigo](https://github.com/marcelorodrigo) in [#​1342](calibrain/shelfmark#1342) - refactor: extract the per-source release search out of /api/releases by [@​splitsec2](https://github.com/splitsec2) in [#​1355](calibrain/shelfmark#1355) - build(deps): bump the docker-base-image-digests group with 2 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1348](calibrain/shelfmark#1348) - build(deps): bump the npm-deps group in /src/frontend with 9 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1349](calibrain/shelfmark#1349) - build(deps): bump the gh-actions group with 6 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1350](calibrain/shelfmark#1350) - feat(auth): provision proxy users as non-admin once an admin exists by [@​splitsec2](https://github.com/splitsec2) in [#​1356](calibrain/shelfmark#1356) - fix(users): apply user updates only after the payload validates by [@​splitsec2](https://github.com/splitsec2) in [#​1360](calibrain/shelfmark#1360) - fix(download): default is\_admin to False in the request policy guard by [@​splitsec2](https://github.com/splitsec2) in [#​1358](calibrain/shelfmark#1358) - fix(oidc): reject backslash paths in the return\_to sanitizer by [@​splitsec2](https://github.com/splitsec2) in [#​1359](calibrain/shelfmark#1359) - fix(queue): don't stamp CANCELLED over a finished download by [@​splitsec2](https://github.com/splitsec2) in [#​1361](calibrain/shelfmark#1361) - fix(download): check task ownership before serving queued files by [@​splitsec2](https://github.com/splitsec2) in [#​1357](calibrain/shelfmark#1357) - Feature: Show the AA search result stats by [@​RoninTech](https://github.com/RoninTech) in [#​1362](calibrain/shelfmark#1362) - fix(requests): reject non-object items in the batch endpoint by [@​splitsec2](https://github.com/splitsec2) in [#​1369](calibrain/shelfmark#1369) - fix(http): keep the host of a protocol-relative download link by [@​splitsec2](https://github.com/splitsec2) in [#​1368](calibrain/shelfmark#1368) - fix(googlebooks): page by the capped size, not the raw limit by [@​splitsec2](https://github.com/splitsec2) in [#​1370](calibrain/shelfmark#1370) - fix(deluge): send seeding ratio limit under Deluge's own keys by [@​splitsec2](https://github.com/splitsec2) in [#​1367](calibrain/shelfmark#1367) - feat(auth): static API\_KEY (env) accepted as Bearer or X-Api-Key, cookie or key by [@​gavinmcfall](https://github.com/gavinmcfall) in [#​1366](calibrain/shelfmark#1366) - fix(auth): rename the API\_KEY env var to SHELFMARK\_API\_KEY by [@​calibrain](https://github.com/calibrain) in [#​1374](calibrain/shelfmark#1374) - fix: bypass recordings, welib wrong-md5 links, footer build sha ([#​1364](calibrain/shelfmark#1364)) by [@​calibrain](https://github.com/calibrain) in [#​1373](calibrain/shelfmark#1373) - build(deps): bump the python-deps group with 4 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1375](calibrain/shelfmark#1375) - ci: debounce dev image builds instead of building nightly by [@​calibrain](https://github.com/calibrain) in [#​1376](calibrain/shelfmark#1376) - fix(download): stream a completed book instead of buffering it in RAM - lowering memory needs significantly by [@​splitsec2](https://github.com/splitsec2) in [#​1378](calibrain/shelfmark#1378) - perf(docker): keep the heavy build layers cacheable across builds - save 11minutes per build by [@​splitsec2](https://github.com/splitsec2) in [#​1379](calibrain/shelfmark#1379) - test(auth): stop proxy provisioning tests depending on run order by [@​splitsec2](https://github.com/splitsec2) in [#​1381](calibrain/shelfmark#1381) - feat(search): add a configurable default content type by [@​splitsec2](https://github.com/splitsec2) in [#​1371](calibrain/shelfmark#1371) - feat(library): mark search results already in a Calibre library by [@​splitsec2](https://github.com/splitsec2) in [#​1377](calibrain/shelfmark#1377) - Default to english when no lang is slected by [@​calibrain](https://github.com/calibrain) in [#​1396](calibrain/shelfmark#1396) - fix(auth): fail closed when auth prerequisites are missing ([#​1387](calibrain/shelfmark#1387)) by [@​calibrain](https://github.com/calibrain) in [#​1397](calibrain/shelfmark#1397) - build(deps): bump the gh-actions group with 4 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1395](calibrain/shelfmark#1395) - fix(audiobookbay): reuse the resolved magnet when retrying ([#​1388](calibrain/shelfmark#1388)) by [@​calibrain](https://github.com/calibrain) in [#​1398](calibrain/shelfmark#1398) - build(deps): bump the npm-deps group in /src/frontend with 7 updates by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1394](calibrain/shelfmark#1394) - build(deps): bump seleniumbase from 4.54.9 to 4.54.10 in the python-deps group by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1382](calibrain/shelfmark#1382) - feat: narrator, series and bitrate columns for MyAnonamouse results by [@​adman234](https://github.com/adman234) in [#​1390](calibrain/shelfmark#1390) - build(deps): bump astral-sh/uv from 0.12.16 to 0.12.19 in the docker-base-image-digests group across 1 directory by [@​dependabot](https://github.com/dependabot)\[bot] in [#​1392](calibrain/shelfmark#1392) - fix(mam): keep the session ID on MAM and rerun Prowlarr's exact search by [@​calibrain](https://github.com/calibrain) in [#​1399](calibrain/shelfmark#1399) - fix(ui): contain the result count added in [#​1362](calibrain/shelfmark#1362) by [@​calibrain](https://github.com/calibrain) in [#​1363](calibrain/shelfmark#1363) ##### New Contributors - [@​amasen02](https://github.com/amasen02) made their first contribution in [#​1309](calibrain/shelfmark#1309) - [@​atirna](https://github.com/atirna) made their first contribution in [#​1312](calibrain/shelfmark#1312) - [@​viniciuspx](https://github.com/viniciuspx) made their first contribution in [#​1322](calibrain/shelfmark#1322) - [@​eikopf](https://github.com/eikopf) made their first contribution in [#​1316](calibrain/shelfmark#1316) - [@​broglea](https://github.com/broglea) made their first contribution in [#​1325](calibrain/shelfmark#1325) - [@​klaidliadon](https://github.com/klaidliadon) made their first contribution in [#​1326](calibrain/shelfmark#1326) - [@​TomJansen](https://github.com/TomJansen) made their first contribution in [#​1337](calibrain/shelfmark#1337) - [@​RoninTech](https://github.com/RoninTech) made their first contribution in [#​1336](calibrain/shelfmark#1336) - [@​marcelorodrigo](https://github.com/marcelorodrigo) made their first contribution in [#​1342](calibrain/shelfmark#1342) - [@​splitsec2](https://github.com/splitsec2) made their first contribution in [#​1355](calibrain/shelfmark#1355) - [@​gavinmcfall](https://github.com/gavinmcfall) made their first contribution in [#​1366](calibrain/shelfmark#1366) **Full Changelog**: <calibrain/shelfmark@v1.3.15...v1.4.0> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDUuMiIsInVwZGF0ZWRJblZlciI6IjQ0LjEwNS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19--> Reviewed-on: https://git.greyrock.io/todd/home-ops/pulls/283
Per discussion #1372, where you said you were fine with this specific implementation: check whether metadata.db exists, read it if so, and show a check mark saying the book is already there.
Searching for a book you already own gives no hint that you own it, so the easiest way to end up with a second copy is to not remember you have the first. This reads a Calibre
metadata.db, read only, and marks matching results with an In library badge in the card, list and compact views and in the details dialog.Off by default. It sits in Settings, General beside the existing Library URL, with a test button that reports how many books it indexed. No HTTP call, no token, nothing written back.
Matching runs most to least confident: a shared external id, then an ISBN compared in both ISBN-10 and ISBN-13 form, then fuzzy title tokens plus the author surname. The check fails open, so an unreadable database degrades the badge and never blocks a search, and entries are cached for ten minutes with an early refresh when the file changes, so a large library costs one read rather than one per search.
text_match.pyis new and shared by the index and the provider, so title, author and ISBN matching stays consistent in one place.On the provider interface
library_indextalks only to aLibraryProviderprotocol and knows nothing about Calibre. That is deliberate but it is not speculative generality, it is what let me send you the Calibre half on its own: I run an Audiobookshelf provider on the same interface in my fork, which is where the audiobook side of the badge comes from. I have left that out because it is a new service integration rather than something already in the codebase, which is the line your non-goals draw. Happy to send it separately if you ever want it, and equally happy for the answer to be no.Adding a library is a module with the
LibraryProvidershape plus one line inall_providers().Verification
tests/core/test_library_index.py: id, ISBN and fuzzy matching, per-content-type provider selection, fail-open on provider errors, stale-cache reuse, TTL and fingerprint refresh, per-provider cache isolation, and the test-connection path including unsaved form values.tests/core/test_text_match.py: ISBN variants and token matching.src/frontend/src/tests/libraryBadge.test.tsand the added cases inbookTransformers.test.ts.