Repository navigation
Raisetolinalg - #412
Draft
arpitj1 wants to merge 318 commits into
Draft
Raisetolinalg#412arpitj1 wants to merge 318 commits into
arpitj1 wants to merge 318 commits into
Conversation
Collaborator
|
@arpitj1 if you don't mind can you please add me as rev as well? I would like to follow what is happening here! |
arpitj1
marked this pull request as draft
January 13, 2025 19:17
…py if current!=totensor
… a given function
…submap, and a more shophisticated approach taken for tracking current tensors via a tree of regions
The existing pattern only handled iter_args whose result eventually flowed into an affine.store, via a small enumerated set of "addf with invariant load" / "mulf with invariant operand" peepholes. Every other consumer — return, math.sqrt, arith.divf/cmpf, func.call, multi-use, multi-iter_arg, "next loop's upper bound", "outer iter_arg's body" — fell into the "unknown operation type" branch of UseChainAnalysis::analyze, left the iter_args in place, and got rejected by AffineForOpRaising. Add MaterializeAffineIterArgsViaAlloca at lower pattern benefit so it only fires when the existing store-fusion fast path fails: one memref<T> slot per iter_arg, init-store before the loop, affine.load at the top / affine.store at the bottom of the new iter_arg-less body, post-loop affine.load + replaceAllUsesWith on each old result. The transform is consumer-blind — RAUW on a same-typed SSA value lets every downstream use (return / store / call / cmpf / index_cast → loop bound / multi-use ...) keep working unchanged. Also guard the existing single-iter_arg incremental rewrite against numIterArgs > 1, which produced an ill-formed terminator and crashed on multi-iter_arg loops; the alloca fallback now handles those. Add memref::MemRefDialect to the pass's dependent dialects so memref.alloca is registered. Verified on 18 surveyed reduction shapes (ddot/dnrm2/dasum families, multi-iter_arg, nested, multi-use, result-as-bound, scf.if bodies, product, integer counters); all 18 lose iter_args and verify. On AffineForOpRaising the 0-D alloca is lifted via polygeist.submap with the 0-D map (d0) -> () and the loop dim is classified as "reduction" automatically — no downstream changes needed. BLAS pipeline now passes end-to-end on the 9 previously-blocked reduction kernels with no regressions on the 11 previously-passing GEMM/AXPY/SCAL/COPY kernels. Update remove-iter-args.mlir: two formerly-negative tests (test_multiply_after_multiply, test_multiple_uses) now transform via the alloca fallback, so their CHECK lines were rewritten; add 8 new survey-derived tests covering the alloca path (ddot/dnrm2/log_sum/ two_reductions/prod/hist/dist) plus the preserved store-fusion path.
…flag
The existing pass (v1) tracks debufferization state via four interlocking
mutable structures (CurrentSlices, regionTensorTree, pendingIfs,
expandedUserList) plus a flat sorted user list across nested regions.
Whenever the IR shape doesn't match the hardcoded region-transition
expectations the bookkeeping desynchronizes from the IR and the pass
silently aborts, produces invalid SSA across regions, or no-ops.
Concrete failures from the 18-case stress survey (see
notes/polygeist_raise_to_linalg/linalg_debufferize_stress_survey.md):
seven silent crashes, one silently-wrong output, one no-op, six
accidental-correctness cases.
v2 is a region-recursive walk. A single SSA `currentTensor` flows
through a recursive `walkBlock` driven by a `WalkCtx`; the recursion
frame stack IS the region tree. Region-bearing ops (scf.for, scf.if,
affine.for, scf.while) are each handled by one helper that detects
whether the body writes the root, rebuilds the op with one extra
tensor iter_arg / extra branch result, and recurses with the new
entry tensor. There is no flat user list, no pendingIfs, no
per-region tensor tree.
Bug A (dangling user list) — structurally absent: invalidation-
safe iteration over block ops.
Bug B (region-blind load/store) — structurally absent: `currentTensor`
is per-frame local.
Bug C (scf.while + affine.for) — handled symmetrically with the
other region-bearing ops.
Bug D (scf.if no-else double — synthesized else uses
yield) replaceOpWithNewOp against the
builder-inserted default yield.
Bug E (multi-level submap chain — applySubmapInverseChain now builds
composition) intermediate base tensors via
forward submaps and unwinds via
submapInverse innermost-first.
Registered side-by-side with v1; gated by a new `use-recursive` pass
option (default false). Invoke v2 via:
polygeist-opt --linalg-debufferize=use-recursive=true ...
Results vs v1 on the 18-case stress corpus:
v1: 11 RAN (1 silently wrong, several accidental) + 7 crashes
v2: 14 DEBUFFED (all correct) + 4 BAILED (cleanly, where v1 was
either accidentally-correct or no-op) + 0 crashes/verify-fails
Results on the BLAS pipeline:
v1: 19/21 DEBUFFED (dgemm OK, sgemm crashes at debuf)
v2: 21/21 DEBUFFED (incl. sgemm — v2 has none of v1's eraseOp issue)
The pattern requires at least one memory op among the root's
transitive users to fire (prevents the greedy driver from looping on
a `create initT + erase initT` no-op cycle), and bails when the
function body has multiple blocks (cf.br support is a future stage).
Adds memref::MemRefDialect and tensor::TensorDialect as explicit
dependent dialects for the pass — needed by the new tensor.insert /
tensor.extract / memref.copy paths.
Flips the `use-recursive` pass option default from false to true.
v1 remains available via `--linalg-debufferize=use-recursive=false`
for fallback / regression investigation.
Verified against both corpora before flipping the default:
Stress corpus (18 inputs at debufferize_stress/):
v1 (was default): 11 RAN (1 silently wrong) + 7 CRASH
v2 (now default): 14 DEBUFFED + 4 BAILED cleanly + 0 CRASH/VERIFY-FAIL
BLAS pipeline (21 kernels):
v1 (was default): 19/21 DEBUFFED (dgemm OK, sgemm crash)
v2 (now default): 21/21 DEBUFFED (incl. sgemm)
The 4 stress cases that BAIL under v2 (s08 call, s10 multi-block,
s13 aliased allocas via cast, s14 memref.subview) all touch consumer
patterns neither version actually debufferizes. In v1 the IR was
silently left in place or accidentally cleaned up by DCE; in v2 we
return failure explicitly via canHandle. End-state IR is the same.
None of these patterns appear in cgeist output, so the BLAS pipeline
is unaffected.
v1 fallback retains its previous 11-RAN/7-CRASH behavior on the
stress corpus when the flag is set to false.
Previously RaiseToLinalg rejected any affine.for whose lower bound
wasn't a constant. This blocked PolyBench's triangular kernels
(trmm, the inner reductions of cholesky/lu/ludcmp/gramschmidt) where
the natural shape is `for k = i+1 to m { B[i,j] += A[k,i] * B[k,j] }`.
Accept non-constant lower bounds when their operands are loop-invariant
w.r.t. the loop being raised (i.e. defined strictly outside it). Capture
the original lb's AffineMap + operands as a BoundMaskInfo and substitute
lb = 0 for the iteration-domain construction. After the linalg.generic
body has been built, emit a guard before the yield:
%idx = linalg.index 0
%lb_val = affine.apply origLbMap(origLbOperands)
%active = arith.cmpi sge, %idx, %lb_val
%gated = arith.select %active, <new_yield_value>, <old_acc_block_arg>
linalg.yield %gated
The fallback `<old_acc_block_arg>` is the block argument corresponding
to the linalg.generic's output operand — i.e. the existing accumulator
value at this iteration. For the masked-out iterations the body's reads
and multiplications still execute (they're side-effect-free under the
existing body-walker contract), but their contribution is discarded.
Safety: the body-walker already restricts the loop body to AffineLoad
(read-only) + AffineStore (the gated write) + pure arith + nested
linalg.generic / affine.if. Under that contract, gating the yield is
sufficient to mask the iteration — the reads have no observable effect
when discarded.
This is the standard polyhedral "iteration-domain compactification +
in-body guard" approach. A follow-up `--linalg-split-iteration-domain`
pass could later split the masked linalg.generic into multiple
rectangular sub-pieces for production performance; the current fix
prioritizes correctness and uniform coverage of all triangular shapes.
Results:
- trmm: 0 → 1 linalg.generic (inner k-reduction now raised with mask).
- ludcmp: 3 → 4 linalg.generic (inner triangular reduction raised).
- lu: 3 → 2 leftover affine.for (one inner loop absorbed).
- All other PolyBench kernels: unchanged.
- BLAS corpus: 21/21 still DEBUFFED.
- LinalgDebufferize stress corpus: 14/4/0 unchanged.
- End-to-end PolyBench: 29/30 unchanged (gramschmidt's debuf failure
is a separate v2 nested-alloca-dominance issue).
- All existing raise-related lit tests still pass.
Full raise of trmm requires additional work in Groups C ("linalg
generic exists with loads/stores") and E (imperfect nesting). This
commit unblocks Group A only.
Files changed:
- lib/polygeist/Passes/RaiseToLinalg.cpp: ~75 LOC added (helper
`allOperandsAreLoopInvariantWrt`, `BoundMaskInfo` struct,
capture-or-fail block replacing the lb rejection, mask emission
before YieldOp).
A loop body that contains both a linalg.generic and raw load/store ops can't be raised into a single linalg.generic — every op in a linalg body executes for every iteration, but the raised inner-kernel is already shaped as a reduction (or whatever) and the raw ops are per-iteration scalar work. They have different iteration semantics. The structural fix is loop distribution: split such a loop into N sibling loops, each containing a homogeneous chunk (one linalg.generic or one block of raw ops). Each sibling raises independently. Safety: distribution preserves semantics iff the outer loop is parallel (each iteration independent). We carry this info forward by having `AffineParallelToFor` annotate its output affine.for ops with `polygeist.was_parallel` (UnitAttr). The distribution pattern only matches loops with that attribute. The new pattern `DistributeAffineForOnLinalgGeneric`: - Matches affine.for with `polygeist.was_parallel`, no iter_args. - Walks the body, grouping ops by chunk boundaries (each linalg.generic or nested affine.for ends a chunk). - Checks no cross-chunk SSA references (would produce use-before-def in the split). - Clones the loop N times, one per chunk. Registered at higher benefit (2) than AffineForOpRaising (1) in step 3 of the pipeline, so distribute applies first on loops that could be split; raise then handles the resulting homogeneous-body loops. Results vs Group-A-only baseline: - trmm: 1 → 2 linalg.generic, 2 → 1 affine.for. j-loop distributed into (matmul-with-mask) and (alpha-scale). Outer i-loop has a true data dependency (i=0 reads B[k,j] which i=k writes), so affine-parallelize correctly leaves it sequential and distribute refuses to split it — exactly the right outcome. - correlation: 3 → 6 linalg.generic (mean/var/corr stages split). - gesummv: 0 → 1 linalg.generic. - 2mm/3mm/atax/deriche/doitgen: unchanged — affine-parallelize is conservative about the alloca produced by `--remove-iter-args` living above i,j, so it doesn't promote those outer loops to affine.parallel and they don't get `was_parallel`. Orthogonal follow-up. No regressions: - LinalgDebufferize stress corpus: 14 DEBUFFED / 4 BAILED / 0 crash. - BLAS corpus: 21/21 DEBUFFED. - PolyBench end-to-end: 29/30 (gramschmidt's debuf failure is a separate v2 nested-alloca-dominance issue). - All raise-related lit tests still pass. The "Distribute+Raising didn't converge" warning may fire for kernels whose outer loop is correctly sequential (multiple linalg.generics in the body, raise rejects with "More than one linalg generic" each iteration without making progress). The output IR is correct; the warning is cosmetic. Files changed: - lib/polygeist/Passes/RaiseToLinalg.cpp: ~90 LOC added (new pattern, attr annotation, pipeline registration).
…neric The initial Group C distribution used "boundary" chunking: walk body in order, push each op into the current chunk, end chunk on each linalg.generic / nested affine.for. This works when the chunk-anchor is preceded only by its own dependencies, but fails when an independent side-effecting op precedes the anchor. Concrete failure: 2mm's j-loop body is `[affine.store cst, tmp[i,j]; submap; submap; submap; linalg.generic]`. The init store and the reduction-via-submap-chain have NO SSA dependency on each other. With boundary chunking they all land in chunks[0] and distribute returns failure() — 2mm stayed at 2 linalg.generic and 4 leftover affine.for. Switch to anchor-based chunking: - Each side-effecting op (linalg.generic, affine.store, memref.store, nested affine.for) is an "anchor". - Each anchor's chunk = itself + transitive SSA dep closure within body. - Chunks must be disjoint; body order determines emit order. For 2mm now: anchors = [store, linalg]. Store has no body-local deps; linalg has the three submaps. Two disjoint chunks → distribute fires into two sibling j-loops → both raise. Same for i-loop. Result: 4 top-level linalg.generic ops, zero leftover affine.for. Results vs prior boundary-chunking commit (6b20d4b): | Kernel | Before | After (this) | |--------------|-----------|--------------| | 2mm | 2L / 4AF | 4L / 0AF | | 3mm | 3L / 6AF | 6L / 0AF | | gesummv | 1L / 2AF | 5L / 0AF | | correlation | 6L / 6AF | 10L / 2AF | | covariance | 4L / 5AF | 5L / 2AF | | adi | 2L / 7AF | 10L / 5AF | | doitgen | 2L / 3AF | 3L / 2AF | PolyBench fully-raised count (0 leftover affine.for): 5 → **8** kernels (gemm, gemver, mvt, seidel-2d, floyd-warshall, 2mm, 3mm, gesummv). No regressions: - LinalgDebufferize stress corpus: 14/4/0 unchanged. - BLAS corpus: 21/21 DEBUFFED unchanged. - PolyBench end-to-end: 29/30 unchanged. - All 29 PolyBench debuf outputs verify clean. - Existing raise-related lit tests pass.
This is the upper-bound mirror of the Group A fix and is what actually
unblocks the "same memref twice in ins" shape (syrk, syr2k). The "same
memref twice" framing was misleading: the underlying issue was that
syrk's inner `for j = 0; j ≤ i; j++` has a non-constant ub `i+1`,
which makes the inner submap's size depend on i — and when the outer
i-loop later tries to wrap the inner linalg, that size doesn't
dominate the new scope. `recursiveCloneWithDominanceCheck` failed
with "Non-dominating block argument encountered".
The same fix shape as Group A's lb-mask:
1. Capture origUbMap + origUbOperands when ub is non-constant and any
operand is an IV of an enclosing affine.for.
2. Substitute each such outer IV with `(outer.ub - 1)` so the
resulting "effective ub" is the *maximum* value across outer
iterations. Use that for the iteration-domain sizing — the submap
now has an outer-scope-dominating size.
3. Emit a body mask `linalg.index <dim> < origUb(captures)` before
the linalg.yield. Combine with the lb mask via `arith.andi` when
both bounds are non-constant.
4. When extending an inner linalg by prepending an outer iter dim,
walk the cloned body and shift each `linalg.index N` to
`linalg.index N+1`. Critical so the body mask emitted in step 3
keeps referring to the correct iter dim after extension.
Results:
syrk: 2 linalg / 2 affine.for → 2 linalg / 0 affine.for
syr2k: 2 linalg / 2 affine.for → 2 linalg / 0 affine.for
Final form (syrk):
linalg.generic { iter=[parallel, parallel] } outs(C)
body: linalg.index 0 (i), linalg.index 1 (j),
if j < i+1: C *= beta; else C unchanged
linalg.generic { iter=[parallel, reduction, parallel] } ins(A, A) outs(C)
body: same structure with the alpha * A[i,k] * A[j,k] accumulation
gated by j < i+1
PolyBench fully-raised count: 8 → 10 (added syrk, syr2k).
- Stress: 14/4/0 unchanged.
- BLAS: 21/21 unchanged.
- End-to-end PolyBench: 29/30 unchanged.
- All 29 PolyBench debuf outputs verify clean.
- All 3 raise-related lit tests pass.
Files changed:
- lib/polygeist/Passes/RaiseToLinalg.cpp: ~60 LOC added.
…oup E) DistributeAffineForOnLinalgGeneric previously required the outer loop to carry `polygeist.was_parallel`. That ruled out fissioning sequential outer loops even when the chunks shared an intermediate buffer indexed identically by the outer IV (atax/bicg-style). Add a chunksDistributionSafe check that, for each chunk, transitively walks memref accesses (including through one polygeist.submap layer for linalg operands) and records which root-dim positions are bound to the outer IV. For each shared root memref with at least one writer across chunks, the iv-bound root-dim set must be non-empty and identical; otherwise reject. Sibling loops only carry was_parallel forward when the input had it. atax (0/3 -> 4/0) and bicg (0/2 -> 4/0) now fully raise. doitgen/symm/ deriche correctly stay rejected (their shared scratch buffers are iv-free across chunks). No regressions on stress (18/18), BLAS (21/21), or PolyBench end-to-end (29/30).
After Groups A/B/C/E, the main remaining PolyBench gap is the pattern [init scratch; accumulate into scratch; copyback from scratch], where the scratch is a function-scope 0-D memref.alloca hoisted from a scalar iter_arg by --remove-iter-args. The Group E dep-check correctly rejects distribution because the scratch's indexing doesn't bind the outer IV. Add PrivatizeScratchAllocaForLoop: when the alloca is used as per-iteration scratch (first body access is a write, no uses after the loop, all in-loop uses are rewriteable), replace memref<T> with memref<? x T> sized by the loop's trip count and rewrite each in-loop access to address new_alloca[iv]. After this every access is iv-bound at root-dim 0, so the dep-check accepts distribution. The new alloca is placed at the original alloca's block, just before the loop's ancestor that lives there. This keeps it at the outer scope so AffineForOpRaising can lift enclosing loops without dominance failures on size operands. Also fix LinalgDebufferize: its alloca -> tensor.empty rewrite was calling the static-shape-only build overload, crashing on dynamic shapes. Pass allocaOp.getDynamicSizes() through. deriche (1L/1AF -> 4L/0AF) now fully raises. symm (2L/2AF -> 4L/1AF) substantial gain; only the sequential outer i-loop remains (which is correct: i has a cross-iter dep on B). PolyBench fully-raised count 12 -> 13. No regressions on stress (18/18), BLAS (21/21), or PolyBench end-to-end (29/30, gramschmidt pre-existing). doitgen still blocked because its scratch is a function argument, not an alloca; that variant of privatization is more involved and deferred.
The polygeist.submap and polygeist.submapInverse ops produced by raise/ debuf are not in standard MLIR; stock mlir-opt can't lower them. Add a new --lower-polygeist-submap pass with three patterns: ComposeSubmapIntoLinalgGeneric matches a linalg.generic and folds operand-defining submaps into the linalg's indexing_maps. Precondition: the submap has no symbols and every result expression is dim-bearing (allows d_i and d_i+const, rejects pure constants or symbols). Verifies the post-compose indexing_maps still collectively cover every iter dim so shape-to-loops inference stays well-defined. LowerSymbolBearingSubmapToSubview lowers a memref-form submap with symbols or constants in its map to memref.subview, computing offsets/ sizes/strides from the affine map. Handles pure SymbolExpr (fixed offset, rank-reduced), pure ConstantExpr (static offset, rank-reduced), pure DimExpr (identity slice), and d_i + (symbol|constant). Rejects broadcasts (output rank > base rank), multi-level chains, and complex expressions (scaling, mod). LowerSymbolBearingSubmapToExtractSlice is the tensor-form analog emitting tensor.extract_slice. LowerSubmapInverse replaces memref-form submapInverse with its base (in-place mutation semantics) and emits tensor.insert_slice for the tensor form. Also add scripts/correctness/: - lower_smoke_test.sh: raise -> lower-polygeist-submap -> mlir-opt to LLVM dialect across all 30 PolyBench kernels. 17/30 currently OK; 13 partials are shapes not yet covered (broadcasts, chains, stencils). - gemm_e2e.sh + gemm_wrapper.c: end-to-end compile-and-run for gemm. Lowers all the way to LLVM IR, links with a C-ABI wrapper that constructs MLIR memref descriptors, displaces gemm.c's kernel_gemm via objcopy --weaken-symbol, and diffs the dumped output against a pure-clang reference. Outputs match bit-exactly on MINI_DATASET. The e2e gemm result is the first real correctness gate for the raise pipeline (vs. structural smoke testing that only verified IR is well- formed). Generalizing to more kernels needs per-kernel wrapper.c generation plus broadcast-shape lowering in the pass.
# Conflicts: # scripts/correctness/build_ce_viewer.py
Preserve the reference-derived specification suites, generic region and partial matching, manifest-driven library lowering, runtime work, and compiler correctness changes. Retain semantic fixtures and experiment summaries alongside the implementation. Record the local LLVM parser-fix commit. Keep machine access details in a private local profile instead of tracked source or notes. Generated binaries, caches, downloads, and local environment files are excluded. Validation: six focused Polygeist lit tests and the LLVM ControlFlow regression pass with existing binaries; Python, shell, and JSON checks pass. This is a checkpoint, not a complete clean-build or artifact-reproduction validation. Existing whitespace in archived/generated inputs is preserved.
arpitj1
force-pushed
the
raisetolinalg
branch
from
November 13, 2026 00:49
a7deda4 to
2a0317f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some modifications to fuse linalg.generic op with for op