Skip to content

879: double-escaping of comparison operators in delimited expressions (part 2) - #884

Merged
lindsay-stevens merged 4 commits into
XLSForm:masterfrom
lindsay-stevens:pyxform-879-part2
Nov 10, 2026
Merged

lindsay-stevens merged 4 commits into
XLSForm:masterfrom
lindsay-stevens:pyxform-879-part2

Conversation

@lindsay-stevens

Copy link
Copy Markdown
Contributor

Follow-up to #881 for #879

Why is this the best possible solution? Were any other approaches considered?

As mentioned in the description for #881, the code underneath Survey.insert_output_values converts back and forth between strings and XML Nodes. This PR is the more involved approach to resolving that, where nodes are created directly and not re-converted. This allows utils.node to be simplified quite a bit.

In the course of doing this I noticed (fixed in 6baa83f) that extra space characters were always being added before and after a sequence of text or mixed text nodes. This would happen whether pretty-print was active or inactive, whereas the PR #681 (which added that behaviour) seemed to be about the pretty-print active case, so in that sense, a bug being fixed here is the addition of extra spaces when pretty-print is inactive (the default). I separated this change from the escaping commits in case this seems to risky, but unfortunately that means the main diff in b482a0b has many whitespace test changes that are reverted in 6baa83f.

What are the regression risks?

The XML escaping issue seems resolved same as #881, so the main risk would be if form clients expected the extra spaces, but it seems like Collect and Web Forms trim them anyway.

Does this change require updates to documentation? If so, please file an issue here and include the link below.

No

Before submitting this PR, please make sure you have:

  • included test cases for core behavior and edge cases in tests
  • run python -m unittest and verified all tests pass
  • run ruff format pyxform tests and ruff check pyxform tests to lint code
  • verified that any code or assets from external sources are properly credited in comments

- new test for double quotes
- include in each test an example of expected escaping of the
  characters when they are in Text or Attribute nodes
- previously the functions called by survey.insert_output_values would
  create strings for nested elements that may be mixed with text. This
  meant that utils.node needed to parse those strings to build the
  Node tree. Now the Element or Text nodes are created directly, so
  that utils.node can be more efficient, and it removes XML parsing from
  the main code path (still exists in xform2json).
- the old behaviour lead to confusion about the correct place and timing
  for escaping XML characters (double quote, ampersand, less than,
  greater than), such as in pyxform/XLSForm#879.
- some tests are adjusted to include spaces around the rendered XML,
  which is a result of utils.DetachableElement.writexml where having
  two or more child nodes (one or more of which is Text) results in
  extra spaces. That is now triggered in more situations due to e.g. the
  functions under survey.insert_output_values returning Text() nodes
  for each parsed segment rather than re-joining. But to avoid changing
  too much implementation at once, the indentation or the joining
  behaviour could be adjusted in a future commit. It seems that Collect
  and Web Forms generally trim leading/trailing whitespace anyway.
- in Survey.itext, rather than sending jr:prefixed strings through the
  expression/variables resolver functions (or formatting the result),
  place the prefix in a Text node and send just the user value through
  insert_output_values().
- originally added in pyxform/XLSForm#681 to address an issue with pretty
  printing XML, however the spaces are added regardless of whether
  pretty printing is active (e.g. indents or newlines are non-blank).
  By default XML pretty print is inactive.
- whether or not pretty printing is active, for XForm purposes it seems
  there is no reason to add these spaces:
  - Collect and Web Forms seem to strip leading/trailing spaces in
    control text
  - the added spaces were not part of the original user XLSForm
  - the ODK XForms spec doesn't mention adding spaces in this context
- seems like it would help avoid confusion around XML formatting in
  tests, particularly since some other parts of pyxform_test_case
  re-parse and re-print (pretty=True) parts of the XML for test
  comparisons. For example in the updated tests the indentation has a
  reduced by a level.

@lognaturel lognaturel left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great not to have to convert from node to strings and back again. Getting rid of extra leading/trailing space is good too.

Will this will need to be a major release? Or are you fairly confident the changed APIs are internal-only?

xpq.body_input_label_text("q1"),
{
" Menu: ",
""" it's "edible"! """,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😂

Comment thread pyxform/utils.py Resolved
@lindsay-stevens
lindsay-stevens merged commit 0619ec2 into XLSForm:master Nov 10, 2026
39 of 42 checks passed
@lindsay-stevens
lindsay-stevens deleted the pyxform-879-part2 branch November 10, 2026 15:08
@lindsay-stevens

Copy link
Copy Markdown
Contributor Author

Will this will need to be a major release? Or are you fairly confident the changed APIs are internal-only?

I think is PR is internal-only, but the next release would probably need to be a major version already due to other recent changes (outlined in #871) such as dropping XLS support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants