[janhoy]

🤖 AI assisted 🤖

Adds an advisory, LLM-assisted triage comment on pull requests, modelled on Solr's pr-triage-llm.yml and judgements.json, including the bullet comment format from apache/solr#5308.

When a PR is opened or a draft is marked ready (not on later pushes), a model behind the ASF LLM gateway (llm.apache.org) is asked what the change warrants, the answers are compared with what the PR includes, and one comment is posted with a bullet per judgement, mismatches marked ⚠️. When everything agrees it collapses to one line (a deliberate difference from Solr). It never fails the check and posts nothing if the model gives no usable answer. Changelog, docs, README and test files are hidden from the model, so it judges the change on its merits rather than noticing what the author added.

What it checks

The judgements in .github/pr-triage/judgements.json encode the feedback contributor PRs here have historically received from reviewers and that CI does not catch. From the review threads of ~45 non-committer PRs the recurring asks were, in order: a changelog entry in helm/solr-operator/Chart.yaml (by far the most common, also for controller-only changes), a row in the chart README values table, a new CRD field not wired through or not exposed in the solr chart, tests, Solr 8.11 vs 9.x compatibility, and a note on how the change was tested. Unlike Solr there is no Jira-style judgement.

Judgement	Compared with
Operator chart changelog and its kind	artifacthub.io/changes in helm/solr-operator/Chart.yaml
Solr chart changelog	helm/solr/Chart.yaml (only asked when that chart changed)
Chart README values table	helm/*/README.md (only when values/templates changed)
User documentation	docs/modules/**
Upgrade notes (removed fields, changed defaults, pod spec changes, new RBAC, raised minimum versions)	docs/modules/upgrade-notes/**
Tests	*_test.go, tests/e2e/**
CRD option completeness (wired through, markers, exposed in solr chart)	flag: a true verdict is the finding
Version compatibility (needs a Solr/Kubernetes version newer than the oldest supported)	flag
Testing notes (cluster-dependent change without a description of how it was tested)	flag

For docs, upgrade notes, README rows and tests only a missing item is flagged; an extra one the model deems unnecessary is not. The changelog also flags a wrong kind or an entry for a change not worth one.

Files

• .github/workflows/pr-triage-llm.yaml: Solr's two-job design. pull_request_target so fork PRs can use the secret; the job holding the key has a read-only token and the job that comments never sees the key. No PR code is checked out or run. Skips quietly when the secret is absent.
• hack/pr-triage/pr_triage_llm.py: Solr's script plus project context in the system prompt, applies_to, evidence_text, flag judgements without evidence, surplus_ok, tolerant verdict spelling (not-worthy, Fixed), a rebase notice for branches far behind main, and --debug (the raw model reply in the job log).
• hack/pr-triage/test_pr_triage_llm.py: 67 offline tests, run from check.yaml.
• dev-docs/pr-triage-llm.md: what it does, local use, setup. Linked from the README.

Trying it on an existing PR

Needs Python 3, gh, jq and a personal key for llm.apache.org. From this branch:

export LLM_API_KEY='<your llm.apache.org token>' LLM_MODEL=qwen3.8-27b

triage_pr() {
  local n=$1 repo=apache/solr-operator dir=/tmp/triage-$1
  mkdir -p "$dir"
  gh api "repos/$repo/pulls/$n" --jq '{title, body, head, base, maintainer_can_modify, changed_files}' > "$dir/pr.json"
  gh api --paginate "repos/$repo/pulls/$n/files" --jq '.[] | {filename, status, additions, deletions, patch}' | jq -s . > "$dir/files.json"
  python3 hack/pr-triage/pr_triage_llm.py --config .github/pr-triage/judgements.json \
     --pr "$dir/pr.json" --files "$dir/files.json" --out "$dir/result.json" "${@:2}" > /dev/null
  [ -f "$dir/result.json" ] && jq -r .comment "$dir/result.json"
}

triage_pr 843              # the comment as it would be posted
triage_pr 571 --dry-run    # just the prompt, no model call

Reasons are HTML-entity escaped on purpose (model text cannot inject links or mentions); GitHub renders them normally. Agreeing bullets keep their reason in an HTML comment, visible only in the Markdown source.

Run on seven open PRs (#888, #877, #846, #844, #764, #722, #854), every ⚠️ was a real gap except a debatable solr chart changelog on #877: for example the missing solr chart exposure of hostName in #764, and security as the changelog kind for #844.

Before this is live

• Repository secret ASF_LLM_API_KEY (an automation key from llm.apache.org, via ASF Infra). Optional variable LLM_MODEL.
• Expect to tune the question wording in judgements.json as real comments come in; that is the only knob that usually matters.

[janhoy]

This is the output when run locally against #843 today:

3 possible mismatch(es) between what this change warrants and what the PR includes.

	Warranted?	In PR?	Confidence	Why
⚠️ Changelog entry (solr-operator chart) — missing	added	no entry	high	The PR introduces a new opt-in external addressability method `Gateway` via a new CRD field `spec.solrAddressability.external.gateway` and a new enum value, which is a new capability users must enable.
✅ Helm chart README values table	no	no	high	The changes add RBAC rules and CRD fields but do not add, rename, or change any Helm chart values in `helm/solr-operator` or `helm/solr`.
✅ Documentation (docs/modules)	yes	yes	high	New CRD fields (`gateway`, `backendTLSPolicy`, etc.) and a new addressability method are added, which require documentation in `docs/modules` (e.g., `solr-cloud-crd.adoc`).
⚠️ Upgrade notes — missing	yes	no	high	The operator requires new RBAC permissions for `gateway.networking.k8s.io` resources (`httproutes`, `backendtlspolicies`), which users managing RBAC manually must add, and the dependency upgrade to Gateway API v1.4+ may require installing new CRDs.
✅ Tests	yes	yes	high	Significant new Go logic is added in `controllers/solr_gateway_util.go` and `controllers/util/gateway_util.go` for reconciling HTTPRoutes and BackendTLSPolicies, which warrants unit tests in `controllers/*_test.go`.
✅ New CRD option complete	no	—	medium	The new API fields are wired through to the controllers, have appropriate doc comments and kubebuilder markers, and use pointers for optional fields where needed to distinguish unset from zero values.
⚠️ Solr / Kubernetes version compatibility — please check	yes	—	high	The change depends on Gateway API v1.4.0+ for `BackendTLSPolicy` GA support, which is not available in the base Kubernetes 1.22 support range without additional CRD installation, and the PR description notes Kubernetes 1.23+ is required for Gateway API CRDs.
✅ Manual testing described	no	—	high	The PR description explicitly states that e2e tests were added and manual testing was performed with kgateway on Kubernetes 1.32, verifying backward compatibility and TLS backend connections.

⚠️ mismatch, ❔ uncertain or unanswered, ✅ agrees. Matching changelog, docs and test files were hidden from the model; the PR description may still mention them. File presence does not establish adequacy. Assessed commit 8e05fd973b5721fa9dbafaae1ba9f1afbb8ee5bf.

[dsmiley]

Nice! I suggest you incorporate my latest improvements, which reads better and is more compact, plus it runs for each time the PR moves out of draft (simpler & good).

[janhoy]

PR updated with @dsmiley's latest improvements to the solr workflow. Tuned the assessments on some more read PRs. Here is the new output on the same PR #843 as below (table output):

Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

• ⚠️ Operator chart changelog — missing (suggest added): The PR introduces a new opt-in external addressability method `Gateway` (new CRD fields in api/v1beta1/solrcloud_types.go) and new controller logic to manage HTTPRoute and BackendTLSPolicy resources, which is a new capability for users.
• ✅ Chart README values table: not needed
• ✅ User documentation: included
• ⚠️ Upgrade notes — missing: The operator requires new RBAC permissions for gateway.networking.k8s.io resources (httproutes, backendtlspolicies), which users managing RBAC themselves must add, and the PR upgrades dependencies (Go, Kubernetes libraries, controller-runtime) which may have implications for the operator's runtime environment.
• ✅ Tests: included
• ✅ CRD option completeness: no concern
• ⚠️ Version compatibility — please check: The PR requires Gateway API v1.4.0+ for BackendTLSPolicy GA support and upgrades to Go 1.24.0+ and Kubernetes libraries v0.34.1, which may not be available across the entire supported range (Kubernetes 1.22+).
• ✅ Testing notes: no concern

[janhoy]

@dsmiley Are you able to add the required secret to this repo the same way you did for solr?

[dsmiley]

Your updated output isn't as up to date as my latest PR. In particular is the way changelog is handled... whcih should include the word "type: " next to the proposed type. (see that PR)

[dsmiley]

RE the key, it's simple; I suggest that you do it, not me (I've never touched this repo and it's better to spread the usage amongst us). Go to https://llm.apache.org/ Go and create a key for that model and give it a name to differentiate this key from other keys you might make (e.g. perhaps "SOLR-OPERATOR-PR-TRIAGE"). You'll copy the key itself soon. At the CLI, type:
gh secret get ASF_LLM_API_KEY --repo apache/solr
And then paste in the key when asked.
It's really simple! DM me if you need further help.

[janhoy]

Also backported the choice_label part. Sample output:

Passing:

• ✅ Operator chart changelog: kind: fixed

Recommendation:

• ⚠️ Solr chart changelog — missing (suggest kind: added): The helm/solr chart gains a new value podOptions.envFrom (values.yaml) and a new template branch in _custom_option_helpers.tpl that renders it into the SolrCloud spec.