[inchang-ing]

Raw HTML blocks which start on a line indented by more than three spaces were not extracted by the HTML block preprocessor, as it could not distinguish such a start tag nested within a containing block (a list item, admonition, etc.) from an indented code block. The block parser then treated the HTML as inline content and split it into separate paragraphs at any blank lines within the raw block.

The preprocessor now extracts such indented raw blocks, removes the indentation of the start tag from the stashed content, and preserves that indentation on the line which holds the placeholder. The block parser can then make the decision based on the resulting block structure: within a containing block the placeholder is parsed as nested content, whereas an indented code block keeps the placeholder in its content. In the latter case, a RawHtmlInCodeTreeprocessor (running before the codehilite treeprocessor) restores the stashed HTML as escaped text, so indented code blocks are rendered exactly as before, whether or not codehilite is enabled.

Fixes #1096

Description

AI Assistance Disclosure

• No AI tools were used in preparing this PR.

• AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

  AI tools were use to assist with the following:

  • Code
  • Code Comments
  • Documentation
  • Changelog Entry
  • Commit Message
  • The Description Above

  Specifically, Claude (Anthropic's LLM, via the WorkBuddy agent) was used to draft the code, tests, documentation, changelog entry, commit messages and this description. All output was reviewed, tested and verified by a human before submission.

  • Other: ______________

Checklist

• This PR follows the contribution guidelines.
• The code follows the Code Style Guide.
• The commit message follows the Commit Message Style Guide.
• I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
• I have added or updated relevant tests.
• I have not requested, and will not request, an automated AI review for this PR.

[waylan]

Please edit your initial comment and check the appropriate boxes (add a x in the brackets like this [x]).

[facelessuser]

I would like to see this tested with md_in_html as well. The testing seems generally sparse for such a potentially big change. I'd like to spend some time more thoroughly vetting this and seeing how it will impact 3rd party plugins as well.

[inchang-ing]

@facelessuser Good point — I've added TestIndentedHtmlBlockswMdInHTML to tests/test_syntax/extensions/test_md_in_html.py, which mirrors the two new regression cases with md_in_html enabled:

• test_indented_code_block_with_blank_lines — the indented block with blank lines still renders as a code block with the extension on (stashed HTML escaped in <pre><code>), identical to the default behavior.
• test_raw_indented_in_list_item — the raw block nested in the list item stays a single <div> inside the <li> rather than being split at blank lines.

Full suite: 1138 tests OK (111 skipped). Also checked the boxes in the PR template per @waylan's request.

[waylan]

You checked AI tools were used and I have disclosed which ones but I do not see where you have disclosed which ones.

[waylan]

I see you force pushed a change. As the Contributing Guide states:

It is generally best not to squash multiple commits and force-push your changes to a pull request. Instead, the maintainers would like to be able to follow the series of commits along with the discussion about those changes as they progress over time. If your pull request is accepted, it will be squashed at that time if deemed appropriate.

There have been multiple failures to follow our contributing guidelines. It is disrespectful for the reviewer's time. Please review the guidelines carefully and follow them fully going forward or this PR may be closed.

[inchang-ing]

Sorry about that — both fair points.

The AI disclosure is now filled in: Claude was used to draft the code, tests, docs, changelog entry, commit messages and the description; I reviewed and verified everything before submitting.

On the force push: understood, it won't happen again. The two fixes pushed since are regular incremental commits, and everything further will be as well.

[facelessuser]

I ran tests in Pymdown Extensions using these changes. I noticed two failures. I need time to investigate why this breaks those tests. I'm not sure this necessarily signals that this could be a breaking change for other extensions, but it's at least something I need time to understand.

[inchang-ing]

Since you mentioned you needed time to work out the two failures, here is what I have from my side — it may save you part of it.

I ran the pymdown-extensions suite against two checkouts of Python-Markdown, everything else identical (same pymdown tree, same Python):

• master (b41e0aa): 13 failed, 840 passed
• this branch: 15 failed, 838 passed
• the difference is exactly two tests, both in tests/test_extensions/test_blocks/test_html.py:
  • TestBlocksHTML::test_html_and_html
  • TestBlocksHTML::test_raw_and_md_in_html

(The 13 shared failures are identical on both, so they look like version drift between your suite and 3.11.2 rather than anything this branch does — I left them out of the analysis.)

In both failing tests the only delta is four spaces of indentation surviving on the line that follows the stashed block:

-this is <span>raw</span> **content**</div>
+    this is <span>raw</span> **content**</div>

Repro, no other extensions involved:

markdown.Markdown(extensions=['pymdownx.blocks.html']).convert('''/// html | div
    markdown: html

    <div>
    **content**
    </div>

    this is <span>raw</span> **content**
///''')

What I have established so far: the lines handed to the Markdown core are byte-identical in both versions, so the divergence is inside HTMLExtractor. On this branch the indented <div> is stashed during preprocessing (inraw_indent=4), which happens before your blocks block processor sees the /// container at all. On master the tag is left alone at that point and only turns into a raw block after the container has been dedented. So on this branch the placeholder carries an indentation your dedent did not account for, and that indentation ends up on the trailing line.

What I have not settled is the right fix, and I would rather implement your call than guess:

1. defer stashing while an enclosing block is still undecided, so extensions keep seeing the container; or
2. keep the placeholder indentation-neutral and move the decision entirely into the tree stage.

Given this is exactly the 3rd-party surface you were worried about, tell me which of the two you would consider acceptable and I will push it with a test.

[facelessuser]

Interesting. I'm not sure which is the best path yet, as I haven't had a chance to dig deeper, but this confirms what I was noticing.

It is worrying if the blocks are altered such that 3rd party extensions are misinterpreting them in the block stage, so I agree a change needs to be made. Hopefully, I'll have a chance soon to look more closely.

[facelessuser]

I've finally cleared my AI PR backlog on a number of other projects. I should be able to devote some time to Python Markdown soon.

[facelessuser]

@inchang-ing I did a deep analysis because this is an area that has big implications. Full disclosure, I used AI to dig into various areas and create a list of tests that should pass, but didn't. I didn't trust that I would catch everything. I also tested things against Pymdown Extensions as well to try and ensure things remained backwards compatible and behaved sanely.

The current implementation is definitely not ready to go as there were numerous areas where things were broken. Extensive testing is a must for something like this.

What I found is that the earlier suggested "container" approach seemed to make the most sense and dramatically simplified the code. And it seems reasonably compatible with Pymdown Extensions (and hopefully others). I have a branch here: https://github.com/facelessuser/markdown/tree/html-indent. I think this may be a good starting point.

If we are going to do this, we have to be careful and do it correctly. I think the approach in the linked branch, which is based on your suggestion, is likely the better approach. I make no assertions that, because I've provided it, it is ready to go. It is a prototype. You could certainly apply it here, or maybe we start a new PR based on the new approach, I'm not sure.

[facelessuser]

I guess elements under blockquotes weren't handled properly, and there was a comment issue; those should be fixed on the branch.

Again, this is just a prototype, and I'm sure more testing to gain confidence would be needed. It is certainly possible we could find some kind of deal-breaker, as I've only spent one evening cobbling this together.

[inchang-ing]

Going with a new PR for the container approach, as you suggested. I did not want to rewrite this branch on top of it: the two commits you pushed are the implementation, and it seemed cleaner to keep them intact with your authorship than to move the diff into a commit of mine under this discussion.

So: inchang-ing/markdown, branch html-indent-container, branched from current master (which is your 0bfd8c3). It contains your two commits unchanged, plus one commit of mine that only adds the changelog entry for #1096, since the contributing guide requires one and neither of your commits touched it. I did not change a line of the code or the tests.

I ran the suite on the result: 1310 tests pass, 111 skipped, and flake8 is clean on the touched files. I also rendered three inputs against the parent commit and against the branch to convince myself the behavior actually moved: a raw block with a blank line in a list item went from <p><div> / <p>second to a single <div>, the same inside a blockquote, and a four-space indented code block with the same markup is byte-identical in both.

New PR: #1688

If you would rather have it open from your fork, say so and I will close mine instead of pushing to it further.

[facelessuser]

It is fine that you run with it.