Skip to content

Raisetolinalg - #412

Draft
arpitj1 wants to merge 318 commits into
llvm:raisetolinalgfrom
arpitj1:raisetolinalg
Draft

arpitj1 wants to merge 318 commits into
llvm:raisetolinalgfrom
arpitj1:raisetolinalg

Conversation

@arpitj1

@arpitj1 arpitj1 commented Jun 6, 2024

Copy link
Copy Markdown
Collaborator

Some modifications to fuse linalg.generic op with for op

@chelini

chelini commented Jan 13, 2025

Copy link
Copy Markdown
Collaborator

@arpitj1 if you don't mind can you please add me as rev as well? I would like to follow what is happening here!

@arpitj1
arpitj1 marked this pull request as draft January 13, 2025 19:17
arpitj1 added 25 commits July 31, 2025 19:58
…submap, and a more shophisticated approach taken for tracking current tensors via a tree of regions
The existing pattern only handled iter_args whose result eventually flowed
into an affine.store, via a small enumerated set of "addf with invariant
load" / "mulf with invariant operand" peepholes. Every other consumer —
return, math.sqrt, arith.divf/cmpf, func.call, multi-use, multi-iter_arg,
"next loop's upper bound", "outer iter_arg's body" — fell into the
"unknown operation type" branch of UseChainAnalysis::analyze, left the
iter_args in place, and got rejected by AffineForOpRaising.

Add MaterializeAffineIterArgsViaAlloca at lower pattern benefit so it
only fires when the existing store-fusion fast path fails: one
memref<T> slot per iter_arg, init-store before the loop, affine.load
at the top / affine.store at the bottom of the new iter_arg-less body,
post-loop affine.load + replaceAllUsesWith on each old result. The
transform is consumer-blind — RAUW on a same-typed SSA value lets every
downstream use (return / store / call / cmpf / index_cast → loop bound /
multi-use ...) keep working unchanged.

Also guard the existing single-iter_arg incremental rewrite against
numIterArgs > 1, which produced an ill-formed terminator and crashed
on multi-iter_arg loops; the alloca fallback now handles those.

Add memref::MemRefDialect to the pass's dependent dialects so
memref.alloca is registered.

Verified on 18 surveyed reduction shapes (ddot/dnrm2/dasum families,
multi-iter_arg, nested, multi-use, result-as-bound, scf.if bodies,
product, integer counters); all 18 lose iter_args and verify. On
AffineForOpRaising the 0-D alloca is lifted via polygeist.submap with
the 0-D map (d0) -> () and the loop dim is classified as "reduction"
automatically — no downstream changes needed. BLAS pipeline now passes
end-to-end on the 9 previously-blocked reduction kernels with no
regressions on the 11 previously-passing GEMM/AXPY/SCAL/COPY kernels.

Update remove-iter-args.mlir: two formerly-negative tests
(test_multiply_after_multiply, test_multiple_uses) now transform via
the alloca fallback, so their CHECK lines were rewritten; add 8 new
survey-derived tests covering the alloca path (ddot/dnrm2/log_sum/
two_reductions/prod/hist/dist) plus the preserved store-fusion path.
…flag

The existing pass (v1) tracks debufferization state via four interlocking
mutable structures (CurrentSlices, regionTensorTree, pendingIfs,
expandedUserList) plus a flat sorted user list across nested regions.
Whenever the IR shape doesn't match the hardcoded region-transition
expectations the bookkeeping desynchronizes from the IR and the pass
silently aborts, produces invalid SSA across regions, or no-ops.
Concrete failures from the 18-case stress survey (see
notes/polygeist_raise_to_linalg/linalg_debufferize_stress_survey.md):
seven silent crashes, one silently-wrong output, one no-op, six
accidental-correctness cases.

v2 is a region-recursive walk. A single SSA `currentTensor` flows
through a recursive `walkBlock` driven by a `WalkCtx`; the recursion
frame stack IS the region tree. Region-bearing ops (scf.for, scf.if,
affine.for, scf.while) are each handled by one helper that detects
whether the body writes the root, rebuilds the op with one extra
tensor iter_arg / extra branch result, and recurses with the new
entry tensor. There is no flat user list, no pendingIfs, no
per-region tensor tree.

  Bug A (dangling user list)        — structurally absent: invalidation-
                                      safe iteration over block ops.
  Bug B (region-blind load/store)   — structurally absent: `currentTensor`
                                      is per-frame local.
  Bug C (scf.while + affine.for)    — handled symmetrically with the
                                      other region-bearing ops.
  Bug D (scf.if no-else double      — synthesized else uses
         yield)                        replaceOpWithNewOp against the
                                      builder-inserted default yield.
  Bug E (multi-level submap chain   — applySubmapInverseChain now builds
         composition)                  intermediate base tensors via
                                      forward submaps and unwinds via
                                      submapInverse innermost-first.

Registered side-by-side with v1; gated by a new `use-recursive` pass
option (default false). Invoke v2 via:
  polygeist-opt --linalg-debufferize=use-recursive=true ...

Results vs v1 on the 18-case stress corpus:
  v1: 11 RAN (1 silently wrong, several accidental) + 7 crashes
  v2: 14 DEBUFFED (all correct) + 4 BAILED (cleanly, where v1 was
       either accidentally-correct or no-op) + 0 crashes/verify-fails

Results on the BLAS pipeline:
  v1: 19/21 DEBUFFED (dgemm OK, sgemm crashes at debuf)
  v2: 21/21 DEBUFFED (incl. sgemm — v2 has none of v1's eraseOp issue)

The pattern requires at least one memory op among the root's
transitive users to fire (prevents the greedy driver from looping on
a `create initT + erase initT` no-op cycle), and bails when the
function body has multiple blocks (cf.br support is a future stage).

Adds memref::MemRefDialect and tensor::TensorDialect as explicit
dependent dialects for the pass — needed by the new tensor.insert /
tensor.extract / memref.copy paths.
Flips the `use-recursive` pass option default from false to true.
v1 remains available via `--linalg-debufferize=use-recursive=false`
for fallback / regression investigation.

Verified against both corpora before flipping the default:

  Stress corpus (18 inputs at debufferize_stress/):
    v1 (was default):  11 RAN (1 silently wrong) + 7 CRASH
    v2 (now default):  14 DEBUFFED + 4 BAILED cleanly + 0 CRASH/VERIFY-FAIL

  BLAS pipeline (21 kernels):
    v1 (was default):  19/21 DEBUFFED (dgemm OK, sgemm crash)
    v2 (now default):  21/21 DEBUFFED (incl. sgemm)

The 4 stress cases that BAIL under v2 (s08 call, s10 multi-block,
s13 aliased allocas via cast, s14 memref.subview) all touch consumer
patterns neither version actually debufferizes. In v1 the IR was
silently left in place or accidentally cleaned up by DCE; in v2 we
return failure explicitly via canHandle. End-state IR is the same.
None of these patterns appear in cgeist output, so the BLAS pipeline
is unaffected.

v1 fallback retains its previous 11-RAN/7-CRASH behavior on the
stress corpus when the flag is set to false.
Previously RaiseToLinalg rejected any affine.for whose lower bound
wasn't a constant. This blocked PolyBench's triangular kernels
(trmm, the inner reductions of cholesky/lu/ludcmp/gramschmidt) where
the natural shape is `for k = i+1 to m { B[i,j] += A[k,i] * B[k,j] }`.

Accept non-constant lower bounds when their operands are loop-invariant
w.r.t. the loop being raised (i.e. defined strictly outside it). Capture
the original lb's AffineMap + operands as a BoundMaskInfo and substitute
lb = 0 for the iteration-domain construction. After the linalg.generic
body has been built, emit a guard before the yield:

  %idx = linalg.index 0
  %lb_val = affine.apply origLbMap(origLbOperands)
  %active = arith.cmpi sge, %idx, %lb_val
  %gated  = arith.select %active, <new_yield_value>, <old_acc_block_arg>
  linalg.yield %gated

The fallback `<old_acc_block_arg>` is the block argument corresponding
to the linalg.generic's output operand — i.e. the existing accumulator
value at this iteration. For the masked-out iterations the body's reads
and multiplications still execute (they're side-effect-free under the
existing body-walker contract), but their contribution is discarded.

Safety: the body-walker already restricts the loop body to AffineLoad
(read-only) + AffineStore (the gated write) + pure arith + nested
linalg.generic / affine.if. Under that contract, gating the yield is
sufficient to mask the iteration — the reads have no observable effect
when discarded.

This is the standard polyhedral "iteration-domain compactification +
in-body guard" approach. A follow-up `--linalg-split-iteration-domain`
pass could later split the masked linalg.generic into multiple
rectangular sub-pieces for production performance; the current fix
prioritizes correctness and uniform coverage of all triangular shapes.

Results:
- trmm: 0 → 1 linalg.generic (inner k-reduction now raised with mask).
- ludcmp: 3 → 4 linalg.generic (inner triangular reduction raised).
- lu: 3 → 2 leftover affine.for (one inner loop absorbed).
- All other PolyBench kernels: unchanged.
- BLAS corpus: 21/21 still DEBUFFED.
- LinalgDebufferize stress corpus: 14/4/0 unchanged.
- End-to-end PolyBench: 29/30 unchanged (gramschmidt's debuf failure
  is a separate v2 nested-alloca-dominance issue).
- All existing raise-related lit tests still pass.

Full raise of trmm requires additional work in Groups C ("linalg
generic exists with loads/stores") and E (imperfect nesting). This
commit unblocks Group A only.

Files changed:
- lib/polygeist/Passes/RaiseToLinalg.cpp: ~75 LOC added (helper
  `allOperandsAreLoopInvariantWrt`, `BoundMaskInfo` struct,
  capture-or-fail block replacing the lb rejection, mask emission
  before YieldOp).
A loop body that contains both a linalg.generic and raw load/store
ops can't be raised into a single linalg.generic — every op in a
linalg body executes for every iteration, but the raised inner-kernel
is already shaped as a reduction (or whatever) and the raw ops are
per-iteration scalar work. They have different iteration semantics.

The structural fix is loop distribution: split such a loop into N
sibling loops, each containing a homogeneous chunk (one linalg.generic
or one block of raw ops). Each sibling raises independently.

Safety: distribution preserves semantics iff the outer loop is
parallel (each iteration independent). We carry this info forward by
having `AffineParallelToFor` annotate its output affine.for ops with
`polygeist.was_parallel` (UnitAttr). The distribution pattern only
matches loops with that attribute.

The new pattern `DistributeAffineForOnLinalgGeneric`:
- Matches affine.for with `polygeist.was_parallel`, no iter_args.
- Walks the body, grouping ops by chunk boundaries (each
  linalg.generic or nested affine.for ends a chunk).
- Checks no cross-chunk SSA references (would produce
  use-before-def in the split).
- Clones the loop N times, one per chunk.

Registered at higher benefit (2) than AffineForOpRaising (1) in step 3
of the pipeline, so distribute applies first on loops that could be
split; raise then handles the resulting homogeneous-body loops.

Results vs Group-A-only baseline:
- trmm: 1 → 2 linalg.generic, 2 → 1 affine.for. j-loop distributed
  into (matmul-with-mask) and (alpha-scale). Outer i-loop has a true
  data dependency (i=0 reads B[k,j] which i=k writes), so
  affine-parallelize correctly leaves it sequential and distribute
  refuses to split it — exactly the right outcome.
- correlation: 3 → 6 linalg.generic (mean/var/corr stages split).
- gesummv: 0 → 1 linalg.generic.
- 2mm/3mm/atax/deriche/doitgen: unchanged — affine-parallelize is
  conservative about the alloca produced by `--remove-iter-args`
  living above i,j, so it doesn't promote those outer loops to
  affine.parallel and they don't get `was_parallel`. Orthogonal
  follow-up.

No regressions:
- LinalgDebufferize stress corpus: 14 DEBUFFED / 4 BAILED / 0 crash.
- BLAS corpus: 21/21 DEBUFFED.
- PolyBench end-to-end: 29/30 (gramschmidt's debuf failure is
  a separate v2 nested-alloca-dominance issue).
- All raise-related lit tests still pass.

The "Distribute+Raising didn't converge" warning may fire for kernels
whose outer loop is correctly sequential (multiple linalg.generics in
the body, raise rejects with "More than one linalg generic" each
iteration without making progress). The output IR is correct;
the warning is cosmetic.

Files changed:
- lib/polygeist/Passes/RaiseToLinalg.cpp: ~90 LOC added (new pattern,
  attr annotation, pipeline registration).
…neric

The initial Group C distribution used "boundary" chunking: walk body in
order, push each op into the current chunk, end chunk on each
linalg.generic / nested affine.for. This works when the chunk-anchor
is preceded only by its own dependencies, but fails when an
independent side-effecting op precedes the anchor.

Concrete failure: 2mm's j-loop body is `[affine.store cst, tmp[i,j];
submap; submap; submap; linalg.generic]`. The init store and the
reduction-via-submap-chain have NO SSA dependency on each other. With
boundary chunking they all land in chunks[0] and distribute returns
failure() — 2mm stayed at 2 linalg.generic and 4 leftover affine.for.

Switch to anchor-based chunking:
- Each side-effecting op (linalg.generic, affine.store, memref.store,
  nested affine.for) is an "anchor".
- Each anchor's chunk = itself + transitive SSA dep closure within body.
- Chunks must be disjoint; body order determines emit order.

For 2mm now: anchors = [store, linalg]. Store has no body-local deps;
linalg has the three submaps. Two disjoint chunks → distribute fires
into two sibling j-loops → both raise. Same for i-loop. Result: 4
top-level linalg.generic ops, zero leftover affine.for.

Results vs prior boundary-chunking commit (6b20d4b):

| Kernel       | Before    | After (this) |
|--------------|-----------|--------------|
| 2mm          | 2L / 4AF  | 4L / 0AF     |
| 3mm          | 3L / 6AF  | 6L / 0AF     |
| gesummv      | 1L / 2AF  | 5L / 0AF     |
| correlation  | 6L / 6AF  | 10L / 2AF    |
| covariance   | 4L / 5AF  | 5L / 2AF     |
| adi          | 2L / 7AF  | 10L / 5AF    |
| doitgen      | 2L / 3AF  | 3L / 2AF     |

PolyBench fully-raised count (0 leftover affine.for): 5 → **8**
kernels (gemm, gemver, mvt, seidel-2d, floyd-warshall, 2mm, 3mm,
gesummv).

No regressions:
- LinalgDebufferize stress corpus: 14/4/0 unchanged.
- BLAS corpus: 21/21 DEBUFFED unchanged.
- PolyBench end-to-end: 29/30 unchanged.
- All 29 PolyBench debuf outputs verify clean.
- Existing raise-related lit tests pass.
This is the upper-bound mirror of the Group A fix and is what actually
unblocks the "same memref twice in ins" shape (syrk, syr2k). The "same
memref twice" framing was misleading: the underlying issue was that
syrk's inner `for j = 0; j ≤ i; j++` has a non-constant ub `i+1`,
which makes the inner submap's size depend on i — and when the outer
i-loop later tries to wrap the inner linalg, that size doesn't
dominate the new scope. `recursiveCloneWithDominanceCheck` failed
with "Non-dominating block argument encountered".

The same fix shape as Group A's lb-mask:
1. Capture origUbMap + origUbOperands when ub is non-constant and any
   operand is an IV of an enclosing affine.for.
2. Substitute each such outer IV with `(outer.ub - 1)` so the
   resulting "effective ub" is the *maximum* value across outer
   iterations. Use that for the iteration-domain sizing — the submap
   now has an outer-scope-dominating size.
3. Emit a body mask `linalg.index <dim> < origUb(captures)` before
   the linalg.yield. Combine with the lb mask via `arith.andi` when
   both bounds are non-constant.
4. When extending an inner linalg by prepending an outer iter dim,
   walk the cloned body and shift each `linalg.index N` to
   `linalg.index N+1`. Critical so the body mask emitted in step 3
   keeps referring to the correct iter dim after extension.

Results:

  syrk:  2 linalg / 2 affine.for  →  2 linalg / 0 affine.for
  syr2k: 2 linalg / 2 affine.for  →  2 linalg / 0 affine.for

Final form (syrk):

  linalg.generic { iter=[parallel, parallel] } outs(C)
    body: linalg.index 0 (i), linalg.index 1 (j),
          if j < i+1: C *= beta; else C unchanged

  linalg.generic { iter=[parallel, reduction, parallel] } ins(A, A) outs(C)
    body: same structure with the alpha * A[i,k] * A[j,k] accumulation
          gated by j < i+1

PolyBench fully-raised count: 8 → 10 (added syrk, syr2k).
- Stress: 14/4/0 unchanged.
- BLAS: 21/21 unchanged.
- End-to-end PolyBench: 29/30 unchanged.
- All 29 PolyBench debuf outputs verify clean.
- All 3 raise-related lit tests pass.

Files changed:
- lib/polygeist/Passes/RaiseToLinalg.cpp: ~60 LOC added.
…oup E)

DistributeAffineForOnLinalgGeneric previously required the outer loop to
carry `polygeist.was_parallel`. That ruled out fissioning sequential outer
loops even when the chunks shared an intermediate buffer indexed identically
by the outer IV (atax/bicg-style).

Add a chunksDistributionSafe check that, for each chunk, transitively walks
memref accesses (including through one polygeist.submap layer for linalg
operands) and records which root-dim positions are bound to the outer IV.
For each shared root memref with at least one writer across chunks, the
iv-bound root-dim set must be non-empty and identical; otherwise reject.
Sibling loops only carry was_parallel forward when the input had it.

atax (0/3 -> 4/0) and bicg (0/2 -> 4/0) now fully raise. doitgen/symm/
deriche correctly stay rejected (their shared scratch buffers are iv-free
across chunks). No regressions on stress (18/18), BLAS (21/21), or
PolyBench end-to-end (29/30).
After Groups A/B/C/E, the main remaining PolyBench gap is the pattern
[init scratch; accumulate into scratch; copyback from scratch], where
the scratch is a function-scope 0-D memref.alloca hoisted from a scalar
iter_arg by --remove-iter-args. The Group E dep-check correctly rejects
distribution because the scratch's indexing doesn't bind the outer IV.

Add PrivatizeScratchAllocaForLoop: when the alloca is used as
per-iteration scratch (first body access is a write, no uses after the
loop, all in-loop uses are rewriteable), replace memref<T> with
memref<? x T> sized by the loop's trip count and rewrite each in-loop
access to address new_alloca[iv]. After this every access is iv-bound
at root-dim 0, so the dep-check accepts distribution.

The new alloca is placed at the original alloca's block, just before
the loop's ancestor that lives there. This keeps it at the outer scope
so AffineForOpRaising can lift enclosing loops without dominance
failures on size operands.

Also fix LinalgDebufferize: its alloca -> tensor.empty rewrite was
calling the static-shape-only build overload, crashing on dynamic
shapes. Pass allocaOp.getDynamicSizes() through.

deriche (1L/1AF -> 4L/0AF) now fully raises. symm (2L/2AF -> 4L/1AF)
substantial gain; only the sequential outer i-loop remains (which is
correct: i has a cross-iter dep on B). PolyBench fully-raised count
12 -> 13. No regressions on stress (18/18), BLAS (21/21), or PolyBench
end-to-end (29/30, gramschmidt pre-existing).

doitgen still blocked because its scratch is a function argument, not
an alloca; that variant of privatization is more involved and deferred.
The polygeist.submap and polygeist.submapInverse ops produced by raise/
debuf are not in standard MLIR; stock mlir-opt can't lower them. Add a
new --lower-polygeist-submap pass with three patterns:

ComposeSubmapIntoLinalgGeneric matches a linalg.generic and folds
operand-defining submaps into the linalg's indexing_maps. Precondition:
the submap has no symbols and every result expression is dim-bearing
(allows d_i and d_i+const, rejects pure constants or symbols). Verifies
the post-compose indexing_maps still collectively cover every iter dim
so shape-to-loops inference stays well-defined.

LowerSymbolBearingSubmapToSubview lowers a memref-form submap with
symbols or constants in its map to memref.subview, computing offsets/
sizes/strides from the affine map. Handles pure SymbolExpr (fixed
offset, rank-reduced), pure ConstantExpr (static offset, rank-reduced),
pure DimExpr (identity slice), and d_i + (symbol|constant). Rejects
broadcasts (output rank > base rank), multi-level chains, and complex
expressions (scaling, mod).

LowerSymbolBearingSubmapToExtractSlice is the tensor-form analog
emitting tensor.extract_slice. LowerSubmapInverse replaces memref-form
submapInverse with its base (in-place mutation semantics) and emits
tensor.insert_slice for the tensor form.

Also add scripts/correctness/:
- lower_smoke_test.sh: raise -> lower-polygeist-submap -> mlir-opt to
  LLVM dialect across all 30 PolyBench kernels. 17/30 currently OK;
  13 partials are shapes not yet covered (broadcasts, chains, stencils).
- gemm_e2e.sh + gemm_wrapper.c: end-to-end compile-and-run for gemm.
  Lowers all the way to LLVM IR, links with a C-ABI wrapper that
  constructs MLIR memref descriptors, displaces gemm.c's kernel_gemm
  via objcopy --weaken-symbol, and diffs the dumped output against a
  pure-clang reference. Outputs match bit-exactly on MINI_DATASET.

The e2e gemm result is the first real correctness gate for the raise
pipeline (vs. structural smoke testing that only verified IR is well-
formed). Generalizing to more kernels needs per-kernel wrapper.c
generation plus broadcast-shape lowering in the pass.
# Conflicts:
#	scripts/correctness/build_ce_viewer.py
Preserve the reference-derived specification suites, generic region and partial matching, manifest-driven library lowering, runtime work, and compiler correctness changes. Retain semantic fixtures and experiment summaries alongside the implementation.

Record the local LLVM parser-fix commit. Keep machine access details in a private local profile instead of tracked source or notes. Generated binaries, caches, downloads, and local environment files are excluded.

Validation: six focused Polygeist lit tests and the LLVM ControlFlow regression pass with existing binaries; Python, shell, and JSON checks pass. This is a checkpoint, not a complete clean-build or artifact-reproduction validation. Existing whitespace in archived/generated inputs is preserved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants