Skip to content

feat(recall): optional Jev reranking for icm recall - #478

Open
nilhemdot wants to merge 3 commits into
rtk-ai:developfrom
nilhemdot:feat/recall-jev-rerank
Open

nilhemdot wants to merge 3 commits into
rtk-ai:developfrom
nilhemdot:feat/recall-jev-rerank

Conversation

@nilhemdot

Copy link
Copy Markdown

What

Opt-in remote reranking for icm recall, configured with:

[recall]
reranker = "jev:jev-latest"   # needs TYPESAFE_API_KEY

When set, icm recall over-fetches 3× limit candidates, scores them in one batched request against TypeSafe's Jev API, and keeps the top limit by relevance probability before graph expansion. Empty (the default) leaves recall unchanged.

This ports the Jev reranker from memsearch (zilliztech/memsearch, src/memsearch/jev_reranker.py) with the same prompt, criteria and response validation, so users moving from memsearch keep the same ranking behavior.

Design notes

  • Opt-in, off by default. Enabling it sends the query and candidate memory text to api.typesafe.ai. This is documented in config/default.toml.
  • Fails loudly. A missing key, HTTP error, timeout, or malformed response returns an error instead of silently falling back to unreranked results, matching memsearch.
  • No leaks. Error messages never include the response body (it may quote memory text) or the API key.
  • Strict validation. Answers d0..dN must be exactly present, type == "noul", numeric, and in [0, 1].
  • Hook path untouched. The per-prompt recall_context hook never makes network calls, per CONTRIBUTING's "cheap per-prompt path" rule. Only the explicit icm recall CLI reranks.
  • Existing HTTP client. Uses the ureq client already in icm-cli, so no new dependencies.

Testing

  • cargo fmt --all --check && cargo clippy --workspace --all-targets -- -D warnings && cargo test --workspace: all pass.
  • 6 new unit tests in jev.rs:
    • request shape keeps full content
    • valid scores parse in order
    • invalid scores are rejected (null, bool, string, < 0, > 1)
    • missing, extra, or wrong-type answers are rejected
    • empty candidates need no key
    • config prefix parsing
  • Live, against a real store:
    • Query: "how do I route claude through the proxy".
    • Hybrid top 3 had an irrelevant auto-extracted memory at fix: update README license from MIT to source-available #2.
    • With Jev, the relevant memories scored 0.92 and 0.75, and the irrelevant one dropped out.
    • The run took about 6 s end to end.
    • A missing key gives Set TYPESAFE_API_KEY to enable Jev reranking.
    • A bad key gives Jev reranking failed (HTTP 401).

Note: develop's Cargo.lock has pre-existing version drift (icm-cli 0.10.63 vs 0.10.65), so --locked builds fail on develop. I left it out of this PR.

🤖 Generated with Claude Code

rtk-ai#469)

* refactor(icm-cli): remove non-functional cloud sync, moving to icm-pro

The `icm cloud login/logout/status/push/pull` commands called
/api/icm/* endpoints that were never implemented server-side in
rtk-cloud, so cloud sync has never actually worked from the OSS
binary — `icm cloud status: connected` only reflected a parsed
credentials.json, not a working sync.

Cloud sync is moving to a new private icm-pro distribution (mirrors
how rtk-pro carries RTK Cloud sync while the OSS rtk never has),
where it will be built out with the real server-side routes and a
background sync daemon. write_secret_file (TOCTOU-safe secret file
writer, also used by the web dashboard password file) moves to
config.rs since it has no other cloud-specific dependency.

358 tests pass; ureq stays (used by summarizer.rs's Ollama calls and
upgrade.rs's self-update download, unrelated to cloud sync); rpassword
is dropped (cloud-exclusive).

* fix(icm-cli): gate write_secret_file behind the web feature

It's only called from web.rs (feature = "web", non-default). Since
cloud.rs (default-compiled) was removed, the default clippy build had
no caller left and flagged it as dead code (-D warnings -> hard fail).
@nilhemdot

Copy link
Copy Markdown
Author

Added tests/jev_integration.rs (55f89bb): end-to-end tests that run the real icm binary against a fake Jev server on localhost. They use the same TcpListener pattern as http_api_integration.rs, so no new dependencies. To point at the fake server, I added one small override, ICM_JEV_ENDPOINT.

The tests cover:

  • reorder and truncate by score
  • 3× over-fetch, auth header and request shape
  • a 500 response fails without leaking the key, response body or memory text
  • a missing key fails explicitly
  • icm hook prompt never contacts Jev

As a check, I disabled the rerank sort; the ordering test fails, so it catches that regression. Full gate passes: fmt, clippy -D warnings, and cargo test --workspace (822 passed, 0 failed).

nilhemdot and others added 2 commits September 28, 2026 19:28
Adds a `[recall] reranker` setting: `"jev:<model>"` reranks recall results
with TypeSafe's Jev decision model over `TYPESAFE_API_KEY`, and an empty
value (the default) keeps the existing hybrid ordering, so nothing leaves the
machine unless it is turned on.

Every candidate is scored in one batched request as an independent noul
question, the candidate pool widens to 3x the limit so there is something to
reorder, and Jev's relevance probability replaces the hybrid score before
graph expansion folds in neighbours.

Failures are loud: a missing key, an HTTP error or an unparseable response
aborts the recall rather than silently returning unreranked results. Error
messages never echo the API key or the response body, which can quote
private memory text.

Ported from memsearch's jev_reranker.py (zilliztech/memsearch).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Runs the compiled binary against a fake Jev server on localhost with an
isolated DB, HOME and XDG config. Covers reorder-and-truncate by Jev score,
the 3x over-fetch, the bearer auth header, fail-loud errors that leak neither
the key nor the response body, and that the per-prompt hook makes no network
call.

Linux-only for the same sqlite-vec child-process reason as
http_api_integration.rs.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nilhemdot
nilhemdot force-pushed the feat/recall-jev-rerank branch from 55f89bb to 0431b41 Compare September 28, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants