Skip to content

feat(git): refresh recently used mirrors in the background - #79

Draft
profiling wants to merge 3 commits into
rolandjitsu:mainfrom
profiling:profiling/issue-78-periodic-refresh
Draft

profiling wants to merge 3 commits into
rolandjitsu:mainfrom
profiling:profiling/issue-78-periodic-refresh

Conversation

@profiling

@profiling profiling commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds bounded periodic fetches for recently used existing mirrors, most recent first. Disabled by default (--refresh-interval-seconds=0); enabled intervals must be at least 3600 seconds. Addresses #78.

Refresh uses server credentials or anonymous access, shares request locking and TTL, and applies auth blocking and bounded backoff without delaying request retries. It never clones missing mirrors. Successful cache preparation records client use; background growth is measured without LRU promotion. Rounds do not overlap, stop admission at their deadline, and use upstream cancellation/reaping on shutdown.

Validation: independent feature, integration, coverage, portability and documentation reviews; formatting, clippy with warnings denied, 207 all-features locked tests, and full local LLVM coverage pass. No uncovered changed lines remain in src/git.rs. Tests cover scheduling, credentials, cancellation, disabled gating, successful-use boundaries, growth/eviction races, deadline overflow and panic cleanup.

Dependency: none; #77 is merged and this branch is rebased onto the latest main (0.2.13) as a single feature commit.

@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.86450% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.72%. Comparing base (4e38a2e) to head (371a454).

Files with missing lines Patch % Lines
src/git.rs 99.84% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #79      +/-   ##
==========================================
+ Coverage   98.40%   98.72%   +0.32%     
==========================================
  Files           9        9              
  Lines        5085     6451    +1366     
==========================================
+ Hits         5004     6369    +1365     
- Misses         81       82       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch 2 times, most recently from 4d0ff10 to 12f3469 Compare November 15, 2026 16:11
@profiling
profiling marked this pull request as ready for review November 15, 2026 16:21
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch 2 times, most recently from ddebb35 to 5d66076 Compare November 15, 2026 16:58
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch 2 times, most recently from 0e9b543 to a463b51 Compare November 16, 2026 09:34

@rolandjitsu rolandjitsu left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. This addresses the last roadmap item we had on the list. I've left a few comments.

Also, you should expose the options in the helm chart as well.

Comment thread docs/configuration.md Resolved
Comment thread tests/refresh.rs Resolved
Comment thread src/git.rs Resolved
Comment thread src/git.rs Resolved
Comment thread src/git.rs Resolved
Comment thread src/git.rs Resolved
Comment thread src/git.rs Resolved
Comment thread src/git.rs Resolved
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch 2 times, most recently from a65bf4d to 5a66741 Compare November 17, 2026 05:34
@profiling

profiling commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Pushed 1252765 (rebased onto the latest main) addressing most of it:

  • Helm chart: exposed config.refreshIntervalSeconds (default 0, otherwise at least 3600), config.refreshActiveWindowSeconds and config.maxConcurrentRefreshes, with env wiring in the deployment template, a render-time check on the interval, and rows in the chart README.
  • Docs, assert! and child_token(): replied in the respective threads.

I will follow up separately on the end-to-end integration test and the refresh metrics.

@profiling
profiling marked this pull request as draft November 17, 2026 11:42
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch 2 times, most recently from 7aed966 to 19b3eab Compare November 17, 2026 11:53
@profiling
profiling marked this pull request as ready for review November 17, 2026 11:58
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch from 19b3eab to 885c12d Compare November 17, 2026 21:24
@rolandjitsu

Copy link
Copy Markdown
Owner

Thanks for the review. Pushed 5a66741 (rebased onto the latest main) addressing most of it:

  • Helm chart: exposed config.refreshIntervalSeconds (default 0, otherwise at least 3600), config.refreshActiveWindowSeconds and config.maxConcurrentRefreshes, with env wiring in the deployment template, a render-time check on the interval, and rows in the chart README.
  • Docs, assert! and child_token(): replied in the respective threads.

I will follow up separately on the end-to-end integration test and the refresh metrics.

Thanks. I'll have another look. Though, not sure if GitHub has issues or what's going on because I see 102 commits in the PR:

Screenshot 2026-11-18 at 10 00 39

@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch from 885c12d to 5e4a0d8 Compare November 18, 2026 14:42
@profiling

profiling commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for flagging this, and sorry for the noise. It was not a GitHub issue. The branch history had been rewritten in a way that also changed the commits that main already contains, so GitHub counted them as new commits in this PR.

I rebased the three feature commits onto the current main and force-pushed. The PR now contains only:

  • 1252765 feat(git): refresh recently used mirrors in the background
  • 4ba540f feat(metrics): count background refresh results
  • 371a454 test(refresh): cover scheduled refresh end to end

The existing review threads are intact, and I updated the hashes quoted in my earlier replies to match.

Comment thread src/git.rs Outdated
#[derive(Clone)]
pub struct GitCache {
shutdown: CancellationToken,
refresh_enabled: Arc<AtomicBool>,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, the reason you're using atomics here is because if the server starts as refresh enabled, it may actually not work because the config is incorrect. So it starts as false, but gets set to true when start_refresh() runs?

We already validate the config when the server starts, so do we still need this to be atomic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. The atomic was only there because the refresh settings were not part of the cache config, so start_refresh() had to flip a flag at runtime. The settings are validated at startup and never change, so it is not needed. Removed: whether use is tracked now comes from cfg.refresh.is_some().

Comment thread src/git.rs
refresh_enabled: Arc<AtomicBool>,
owners: TaskTracker,
upstream_timeout: Duration,
cfg: GitConfig,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add a Option<RefreshConfig> to GitConfig, then start_refresh() since you can access the git config in the method w/o needing to pass those values to it in main.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as suggested. GitConfig now has refresh: Option<RefreshConfig> (interval, active window, concurrency), built from the CLI options by Config::refresh_config(). start_refresh() takes no arguments and reads it, and refresh_repo / refresh_candidates read the same settings instead of taking them as parameters. A zero interval or concurrency passed by a library caller counts as disabled, with a warning.

I folded this into the original commits rather than adding a new one, so the hashes changed again: 1252765 (feat(git)), 4ba540f (feat(metrics)), 371a454 (test). The branch is also rebased onto the current main (4e38a2e).

Assisted-by: OpenAI:Codex
Assisted-by: Claude:claude-sonnet-5-5
Add refresh_total{result} (ok, error, auth_rejected, preempted,
skipped) so the refresh success and failure rates are visible apart
from client-triggered fetches. Shutdown is not counted.

An eviction that cancels a refresh fetch no longer counts as an
upstream fetch error.

Assisted-by: Claude:claude-sonnet-5-5
Drive a real file:// upstream through the router, push a new commit,
and assert the mirror advances with no further client request. The
paused clock is held still while real git runs, which would otherwise
auto-advance it into the upstream timeout.

Assisted-by: Claude:claude-sonnet-5-5
@profiling
profiling force-pushed the profiling/issue-78-periodic-refresh branch from 5e4a0d8 to 371a454 Compare November 19, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants