Skip to content

Guard source captures against visitor bot-check tokens - #10830

Merged
MaxGhenis merged 10 commits into
mainfrom
guard-source-capture-tokens
Nov 16, 2026
Merged

MaxGhenis merged 10 commits into
mainfrom
guard-source-capture-tokens

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Guard source captures before contributors publish visitor bot-check data. The check scans every file under source_conflicts/sources/ for bot-manager markers and base64 UUID-dollar-IPv4 tokens. Diagnostics give the path and match kind without captured values.

When a site serves bot-check scripts, contributors should extract pandoc text and include the source URL and extraction method.

Invariants and validation

  • Reject complete base64 UUID-dollar-valid-IPv4 payloads, padded or unpadded, in quotes, scripts, query strings, URL paths, JSON-escaped and percent-encoded URLs, and at either end of a line. Deterministic regressions cover the shortest payload at 59 unpadded or 60 padded characters and the longest at 70 unpadded or 72 padded characters in all 13 page contexts.
  • End a token at + and / as at any byte outside the base64 alphabet, in the raw bytes and after one round of percent-decoding. Hex digits, -, $ and . encode only to base64 letters and digits, so a token never contains + or /.
  • Accept the two tested near misses, each one character short of a token, in every page context: the shortest token without its last character, and the longest without its first.
  • Reject the 12 listed bot-manager markers case-insensitively, including Cloudflare's single-underscore challenge options. Independent regressions check those options without another marker or encoded token.
  • Report only the file path and match kind, once per kind. The nested binary fixture checks the full diagnostic and excludes token values and addresses.
  • Accept benign SVG data, stylesheet versions, citations, a public site id and unrelated base64 data.
  • Scan raw bytes recursively, regardless of file extension or UTF-8 validity.
  • Generate UUIDs, documentation addresses, casing, padding and page contexts with the privacy property test. Generate benign SVG and stylesheet examples with the second property.

Narrowing the base64 window to 64–67 characters before padding fails every complete-token regression for the shortest and longest payloads. Restoring / and + to the token alphabet fails the URL-path, plus-sign and escaped-slash regressions. Dropping the percent-decoded scan fails the percent-encoded URL regressions. Removing the pattern's lookbehind or its lookahead leaves all 146 capture tests passing, so no regression pins the rule that a decoded run has no letter or digit on either side and no further = after it.

test_source_conflicts.py collects 251 tests, 146 of them for this check, and the property file caps each of its two properties at 32 examples. CI's Rest Python shards run both files.

Pull request run 43910613214 at 8043062a4f passed all 37 jobs on GitHub's merge with main 8e1837790b.

Methodology choices

  • Detect the listed bot-manager markers without regard to case, including Cloudflare's __cf_chl and _cf_chl_opt.
  • Flag a base64 token only when its entire decoded payload matches a UUID, a dollar sign and a valid IPv4 address. Scan each capture's raw bytes for tokens, then its bytes after one round of percent-decoding, and match markers in the raw bytes only. IPv6, payloads with extra decoded bytes, tokens with a further = after their padding and tokens joined to other letters or digits, such as after a \u002F escape or in a URL percent-encoded twice, remain outside this check.
  • Check valid IPv4 payloads with and without base64 padding, regardless of address routability. Synthetic fixtures include documentation addresses and the two IPv4 length extremes.
  • Use Python tests for capture tooling. This PR changes no variable, parameter, reform or model setup code.

Microsimulation impact

This PR changes a code-health test file, a property test, capture guidance and a changelog fragment, and no formula, parameter or policy test, so it changes no model output.

impact: none: this PR changes no formula, parameter or policy test

axiom: n/a: this change adds source capture privacy checks and guidance without changing encoded provisions.

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits November 12, 2026 00:50
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 15, 2026 22:46
Hex digits, "-", "$" and "." encode only to base64 letters and digits, so a
UUID-dollar-IPv4 token never contains "+" or "/". The capture guard counted
both as token characters and missed a token beside them: in a URL path, after
a JSON-escaped slash, or after a plus sign. The token class now holds letters
and digits only.

The encoded-token test wraps each boundary token in ten page contexts, and
two near misses, each one character short of a token, pass in every context.
The privacy property samples the same contexts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A percent-encoded URL puts "%2F" or "%3D" beside a token. The escape's hex
digits join the token's letters and digits, so the guard missed the token.
The guard now also scans each capture after one round of percent-decoding and
reports a token found in either form once.

The encoded-token and near-miss tests gain a percent-encoded URL path and
query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State in the capture README which runs the guard decodes, where it finds
tokens, and what it misses: IPv6, payloads with extra decoded bytes, and
tokens joined to other letters or digits. Add a raw query string to the
encoded-token contexts, since the README names it. Join a hard-wrapped
CONTRIBUTING paragraph into one line to match its neighbours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review November 16, 2026 03:52
MaxGhenis and others added 2 commits November 15, 2026 22:54
The README's example of a token joined to letters or digits named a bare
slash, which the guard treats as a boundary. It now names the JSON Unicode
escape for a slash, whose last hex digit joins the token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard matches bot-manager markers in a capture's raw bytes only, and
base64 visitor tokens in the raw bytes and after one round of
percent-decoding. Move the percent-decoding clause in CONTRIBUTING.md and
the guard's docstring so it covers tokens alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis MaxGhenis closed this Nov 16, 2026
@MaxGhenis MaxGhenis reopened this Nov 16, 2026
@MaxGhenis MaxGhenis closed this Nov 16, 2026
@MaxGhenis MaxGhenis reopened this Nov 16, 2026
@MaxGhenis MaxGhenis closed this Nov 16, 2026
@MaxGhenis MaxGhenis reopened this Nov 16, 2026
@MaxGhenis MaxGhenis closed this Nov 16, 2026
@MaxGhenis MaxGhenis reopened this Nov 16, 2026
@MaxGhenis
MaxGhenis merged commit 3d73f27 into main Nov 16, 2026
103 of 118 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

US + core hub merge audit, us#10830 at 8043062a4fb34fe4127b18d96a7db2e8019a64ab (squash): Guard source captures against visitor bot-check tokens.

  • Reviews:
    • The PR has no GitHub reviews. Three independent review rounds ran outside GitHub, with a fix round after each of the first two.
    • The first round, at 924529fc20, ran both test files, which passed, and confirmed that pandoc's plain-text output drops script blocks, as the README says. It scanned the 48 capture files at that head and the captures at the heads of 21 other open PRs, and the check flagged none.
    • It found that every token fixture used a 203.0.113.x address, so no test pinned the edges of the 59 to 70 character window. With the window narrowed to 64 to 67, all 40 capture tests and both properties still passed. It also noted that the check missed IPv6 addresses and payloads with trailing bytes, that __cf_chl didn't match Cloudflare's _cf_chl_opt, and that the changelog fragment sat under the wrong type.
    • The first fix round, at b0d3d8f34b, added token cases for 0.0.0.0, 203.0.113.42 and 255.255.255.255, padded and unpadded, each with a length assertion. It added the _cf_chl_opt marker with two cases of its own, stated the IPv4 scope in the README and renamed the fragment to .added.md.
    • The second round, at b0d3d8f34b, found that a complete token beside / or + passed, because the pattern counted both characters as part of a token. It ran 60 complete-token cases in six page contexts, and the check missed the 30 that sat beside a slash or a plus sign. It recomputed each asserted length by hand and asked to join a hard-wrapped CONTRIBUTING paragraph.
    • The second fix round dropped + and / from the token pattern and added a second scan after one round of percent-decoding. It added 13 page contexts, which raised the complete-token cases from 6 to 78, and 26 near misses. It pointed the property at the same contexts and merged main at aead96b524 without conflict. A check of that round at de294a0341 found that CONTRIBUTING.md and the guard's docstring claimed the percent-decoded scan for markers too, and 8043062a4f corrected both.
    • A second check, at 8043062a4f, ran eight mutants under pytest on the 146 capture tests. The guard as it stood before the round failed 48 tests, restoring + and / failed 42, removing the decoded scan failed 12, narrowing the window failed 52 and allowing at most one = of padding failed 28. Three mutants that only widen the match left all 146 passing: no lookbehind, a lookahead without =, and no lookahead.
    • The third round, scoped to the commits since b0d3d8f34b, approved 8043062a4f with no blocking finding. It built 78 complete tokens and 26 near misses from parts and applied the head's expressions: the check rejected all 78 and passed all 26. It found no literal token and no address of the capturing machine in the diff, and it confirmed that the merge of main kept both sides whole.
    • Its one minor finding covered two description bullets, which linked Cloudflare, RFC Editor and IANA pages. I replaced both with its text.
    • In run 43803397761, which tested this head merged into main at aead96b524, all 38 jobs passed. The Quick Feedback job ran both changed test files: 251 tests passed in test_source_conflicts.py and 2 in the property file. Rest Python shard 8 ran test_source_conflicts.py again, with 251 passed, and shard 3 ran the property file, with 2 passed.
    • The reviews and checks listed corrections to the description, and I made each before merge, except one that the third round replaced. The first round asked the description to name the vendors. The third round's bullet names Cloudflare alone, and the description links the 12 markers in the code. It now links no vendor or standards page and states that the PR changes no model output. It says the check flags a token, where it said the check decodes one, and it scopes the near-miss claim to the two tested strings. I also added that markers match in the raw bytes only, the padding limit and the mutants that no test catches.
  • Methodology choices for maintainers:
    • The check fails a capture that holds any of 12 marker strings anywhere in its raw bytes, in any letter case, and it has no allowance list. An official page whose own text holds one of those strings can't land as a capture until a maintainer narrows the list or adds an allowance.
    • The check decodes each run of 59 to 70 letters and digits, with up to two = of padding. It flags a run only when the whole decoded value reads as a UUID, a dollar sign and a valid IPv4 address. It passes a token built on an IPv6 address, a payload with extra decoded bytes, and a token joined to other letters or digits, as after a Unicode-escaped slash or in a URL percent-encoded twice. The README states these limits.
    • The pattern ends a token at + and /. The hex digits, -, $ and . of a payload encode to 43 base64 characters, all letters and digits, so no token holds either character.
    • The check reads tokens in the raw bytes and again after one round of percent-decoding, and it reads markers in the raw bytes only. A marker with every byte percent-escaped passes. All 12 markers use only letters, _ and -, which Python's quote leaves unescaped.
    • The check flags a token built on any valid IPv4 address and applies no routability filter. The fixtures build their tokens from a documentation address and from 0.0.0.0 and 255.255.255.255, the shortest and longest forms.
    • The check scans every file under source_conflicts/sources/, the README there included, so that README can name no listed marker.
  • Gates checked live at merge:
    • gh pr checks exits 0;
    • the latest Pull request run at the head, 43910613214, passed and tested this head merged into main at 8e1837790b;
    • MERGEABLE;
    • not a draft;
    • no reviewer's latest review requests changes;
    • the semantic-overlap guard: since 8e1837790b, main has changed no area this PR touches. Between the PR's merge base aead96b524 and main at 8e1837790b, main changed 108 files. Of this PR's five files, main changed test_source_conflicts.py alone: Apply New Jersey, Ohio, Oklahoma and Pennsylvania rules for a filer whom another taxpayer can claim as a dependent #10043 added three lines to it, and git merges them with this PR's changes without conflict. Record Louisiana's claimable-filer dependent exemption reading #10833 added eight Louisiana captures under source_conflicts/sources/us-la/, which the new check scans. None holds a listed marker, and each holds one run of 59 or more letters and digits, a 64-character SHA-256 hex digest, whose decoded bytes can't start a UUID. The branch merges into main at 8e1837790b without conflict;
    • the merge pins the head.
  • Impact: none: the PR changes a code-health test file, a property test, capture guidance and a changelog fragment, and no formula, parameter or policy test.
  • Next:
    • No test pins the README's edge rule: three mutants that only widen the match pass all 146 capture tests. A follow-up can add pass cases for a letter before the longest token and for a further = after a padded one.
    • After this merge, the check runs in the CI of every PR that isn't a draft. The first round's scan of 21 open PRs' captures predates the _cf_chl_opt marker, the + and / boundaries and the percent-decoded scan, so each open PR that adds captures should merge main and read its own result.
    • The check misses a token built on an IPv6 address, and its marker list holds 12 strings. An issue can add IPv6 payloads and the markers of other bot managers.
    • The README's scan sentence covers tokens alone, and the third round suggested opening it with "The decoded-token check scans".
  • Axiom: n/a, since the PR changes capture checks and guidance and no provision. No pe-parity issue applies.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant