Skip to content

fix(workflows): evaluate chained comparisons pairwise like Python - #5203

Open
aarjang wants to merge 1 commit into
github:mainfrom
aarjang:fix/5071-chained-comparisons
Open

aarjang wants to merge 1 commit into
github:mainfrom
aarjang:fix/5071-chained-comparisons

Conversation

@aarjang

@aarjang aarjang commented Oct 31, 2026

Copy link
Copy Markdown

Description

Fixes #5071.

The workflow expression evaluator split a comparison at the first operator in _COMPARISON_OPERATORS table order and evaluated the whole remainder as the right operand, so a chained comparison such as {{ 0 <= inputs.n <= 10 }} became 0 <= (inputs.n <= 10) — a number compared against a boolean. Validation accepted the condition and nothing was raised at run time, so an if, while or do-while step silently took the wrong branch (with n = 50 the condition read true and the then branch ran).

This PR makes the evaluator scan the expression left to right for every top-level comparison operator (longest operator first at each position, with the same quote/bracket skipping as _find_top_level) and evaluate the chain pairwise with an implicit and, the way Python and Jinja2 chain comparisons: 0 <= n <= 10 is 0 <= n and n <= 10. This matches the module's documented contract ("a safe Jinja2 subset").

  • Single comparisons keep their previous semantics; operators inside quoted or bracketed operands are still ignored.
  • Mixed chains work the way Python defines them, membership included: a < b == c, a < b in items.
  • A chain binds tighter than and/or, as before.
  • Every operand is evaluated before any pair is compared, so validation probes (_collect_leaves) visit the same leaves regardless of which pair fails first.

Testing

  • New regression test TestExpressions::test_chained_comparisons_evaluate_like_python fails on main exactly as the issue describes ({{ 0 <= inputs.n <= 10 }} is True for n = 50) and passes with this change. It also covers mixed-operator chains, chain-vs-and/or precedence, and operator text inside quoted operands.

  • The reproduction script from [Bug]: Chained comparisons such as 0 <= x <= 10 in workflow expressions silently return the wrong result #5071 now prints the expected values, validate_workflow reports no errors, and the if step runs the outside branch for n = 50.

  • tests/test_workflows.py + tests/workflows: all pass locally (.venv/bin/python -m pytest per CONTRIBUTING).

  • Full suite: 13,649 passed; 11 failures in the script-parity tests (test_*_python_parity.py) are a pre-existing local flake — the identical 11 fail on a clean main checkout on the same machine, and each passes in isolation on both branches.

  • Tested locally with uv run specify --help

  • Ran existing tests with uv sync && uv run pytest

  • Tested with a sample project (if applicable)

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Written with Claude Code (model: Claude Fable 5, default reasoning effort, human-supervised). The agent located the defect, wrote the fix and the regression test, and ran the verification (failing-on-main test, issue repro script, full suite comparison against clean main); I reviewed the diff and the evidence before submitting. The commit carries a matching Assisted-by: trailer.

🤖 Generated with Claude Code

The expression evaluator split a comparison at the first operator in
_COMPARISON_OPERATORS table order and evaluated the whole remainder as
the right operand, so a chained comparison such as `0 <= inputs.n <= 10`
became `0 <= (inputs.n <= 10)` — a number compared against a boolean.
Validation accepted the condition and nothing was raised at run time,
so an `if`, `while` or `do-while` step silently took the wrong branch.

Scan the expression left to right for every top-level comparison
operator (longest operator first at each position, with the same
quote/bracket skipping as _find_top_level) and evaluate the chain
pairwise with an implicit `and`, the way Python and Jinja2 chain
comparisons: `0 <= n <= 10` is `0 <= n and n <= 10`. Single
comparisons keep their previous semantics, and every operand is
evaluated before any pair is compared so validation probes visit the
same leaves regardless of which pair fails first.

Fixes github#5071

Assisted-by: Claude Code (model: Claude Fable 5, supervised)
@aarjang
aarjang requested a review from mnriem as a code owner October 31, 2026 11:01
@mnriem mnriem added the triage-must-have Verdict: high-value, important work for Spec Kit — do first label Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-must-have Verdict: high-value, important work for Spec Kit — do first

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Chained comparisons such as 0 <= x <= 10 in workflow expressions silently return the wrong result

2 participants