fix vgg16 diamonds

updat ops validations

.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md

@@ -0,0 +1,362 @@
+# Graph Compute Batch Physical-Fragment Invariant
+
+## Status
+
+This document is **normative** for Raptor's Spatial graph IR.
+
+Every developer or coding agent modifying Spatial graph construction, graph
+verification, Blueprint handling, or `MergeComputeNodes` must read this file
+after `README.md` and `AGENTS.md`.
+
+`AGENTS.md` must reference this invariant:
+
+```text
+* `.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md`
+```
+
+## Scope
+
+This invariant applies to:
+
+- `spat.graph_compute_batch`;
+- graph-level values produced by it;
+- `tensor.parallel_insert_slice` operations that publish its lane results;
+- `spat.blueprint` operations that describe logical reconstruction;
+- graph analyses and transformations that consume those values;
+- the graph-to-scheduled transition in `MergeComputeNodes`.
+
+It does **not** impose the same representation on:
+
+- `spat.scheduled_compute`;
+- `spat.scheduled_compute_batch`;
+- `pim.core` or `pim.core_batch`;
+- values whose cross-core movement is already represented by explicit
+  `spat.channel_send` and `spat.channel_receive` operations.
+
+Scheduled IR represents execution on assigned cores. Communication and value
+availability there are defined by local SSA forwarding and explicit
+send/receive operations, not by the graph physical-fragment invariant.
+
+## Core invariant
+
+For every result of a `spat.graph_compute_batch` with `N` graph lanes:
+
+1. Every graph lane produces exactly one fragment for that result.
+2. All lanes produce fragments with the same exact ranked tensor type `F`.
+3. The graph result is a physical collection of those fragments with type:
+
+   ```text
+   tensor<N x shape(F) x element-type(F)>
+   ```
+
+   Conceptually, the result is `N × F`: one leading physical fragment-slot
+   dimension followed by the complete per-lane fragment shape.
+4. Physical slot `i` identifies a fragment publication. It does not, by itself,
+   identify a row, column, channel, tile, or any other logical tensor position.
+5. The result type carries no logical reconstruction order.
+
+The leading dimension is therefore a **physical fragment-slot dimension**, not
+a logical tensor dimension.
+
+## Per-lane computation is unrestricted
+
+The invariant constrains the published result representation, not what a lane
+may compute.
+
+A graph lane may:
+
+- read several input slices;
+- perform reductions;
+- add or combine multiple columns;
+- execute matrix/vector operations;
+- produce a fragment that corresponds to any logical region;
+- participate in a multi-stage or logarithmic reduction tree implemented by
+  following `spat.graph_compute` or `spat.graph_compute_batch` operations.
+
+Arithmetic combination is graph computation. `spat.blueprint` is not an
+arithmetic reduction operation.
+
+### Example: `16×4 -> 16×2`
+
+Two graph lanes may compute:
+
+```text
+lane 0: input[:, 0] + input[:, 1] -> tensor<16x1>
+lane 1: input[:, 2] + input[:, 3] -> tensor<16x1>
+```
+
+The physical graph result is:
+
+```text
+tensor<2x16x1>
+```
+
+A Blueprint then maps:
+
+```text
+physical slot 0 -> logical output[:, 0:1]
+physical slot 1 -> logical output[:, 1:2]
+```
+
+and describes the logical result `tensor<16x2>`.
+
+For a larger reduction, following graph compute batches may reduce fragments in
+`ceil(log2(N))` stages. Every intermediate batch still publishes a physical
+`batch × fragment` collection.
+
+## Physical publication inside `spat.graph_compute_batch`
+
+The batch body must publish each lane's fragment into the physical result.
+
+For one result with fragment type `F`, the corresponding
+`tensor.parallel_insert_slice` must insert the fragment into one slot of the
+physical `N × F` destination:
+
+```text
+physical offsets = [slot, 0, 0, ...]
+physical sizes   = [1, shape(F)...]
+physical strides = [1, 1, 1, ...]
+```
+
+The slot may be the graph lane directly or a statically analyzable permutation
+of it. The insertion describes physical slot placement only. It must not use a
+logical output dimension as the physical batch dimension.
+
+For each graph result, the body must contain exactly one physical publication
+per graph lane. Since the body executes once per lane, this normally means one
+`tensor.parallel_insert_slice` operation targeting that result.
+
+## Logical reconstruction
+
+Logical reconstruction is separate from physical publication.
+
+The reconstruction descriptor defines, for every physical fragment slot:
+
+- which physical batch operand owns the fragment;
+- which physical slot contains it;
+- its destination offsets in the logical tensor;
+- its destination sizes;
+- its destination strides;
+- coverage and conflict policy where relevant.
+
+The persistent owner of this information is `spat.blueprint` or an equivalent
+explicit graph-level reconstruction operation.
+
+A logical consumer must not infer reconstruction from the physical tensor type
+or assume that physical slot order equals logical order.
+
+The logical mapping may be arbitrary. For example:
+
+```text
+physical slot 0 -> logical row 13
+physical slot 1 -> logical row 4
+physical slot 2 -> logical row 10
+```
+
+The physical result remains a regular `batch × fragment` tensor.
+
+## Relationship between `parallel_insert_slice` and Blueprint
+
+During graph construction, an algorithm may naturally describe logical
+placement with `tensor.parallel_insert_slice` geometry. Before the graph is in
+its canonical form:
+
+1. that geometry must be separated from physical fragment publication;
+2. the graph batch result must be normalized to `N × F`;
+3. the logical insertion geometry must be transferred to a persistent
+   `spat.blueprint` reconstruction descriptor.
+
+After normalization:
+
+- `parallel_insert_slice` inside `spat.graph_compute_batch` publishes into
+  physical fragment slots;
+- `spat.blueprint` describes reconstruction into the logical tensor.
+
+The original graph operation may be erased only after all reconstruction
+information needed by later stages has a persistent owner.
+
+## Blueprint semantics
+
+Blueprint is placement/reconstruction metadata. It may:
+
+- concatenate fragments;
+- reorder fragments;
+- insert fragments into arbitrary disjoint logical regions;
+- describe complete or partial logical coverage;
+- expose a logical tensor view when materialization is required.
+
+Blueprint must not silently perform arithmetic such as addition, multiplication,
+maximum, or reduction. Such transformations must be represented by following
+`spat.graph_compute` or `spat.graph_compute_batch` operations.
+
+A Blueprint consuming a physical fragment batch must explicitly identify the
+physical source slot for every logical fragment. It must not derive that slot
+from operand order unless that convention is explicitly represented and
+verified.
+
+## Multiple results
+
+A `spat.graph_compute_batch` may have several results.
+
+For each result `r` independently:
+
+- every lane produces one fragment of type `F_r`;
+- the graph result type is `N × F_r`;
+- its physical publication and logical reconstruction descriptor are verified
+  independently.
+
+Different results may use different fragment shapes.
+
+## Graph consumers
+
+A graph consumer of a batch result may:
+
+1. consume fragments directly as physical fragments;
+2. select one or more physical slots in a `spat.deferred_communication` body;
+3. use a Blueprint to obtain or describe a logical reconstruction;
+4. feed fragments to following graph computes or graph compute batches.
+
+A consumer must not treat the leading physical slot dimension as a logical
+model dimension unless an explicit graph operation intentionally performs such
+an interpretation.
+
+All constant selection, slicing, reshaping, concatenation, and other
+compile-time shaping needed for a scheduled consumer must be encoded inside the
+corresponding `spat.deferred_communication` body. Phase 2 must not recover
+missing graph semantics by inspecting consumers after the deferred operation.
+
+## Graph lane, scheduled lane, and physical core are different identities
+
+These concepts must never be conflated:
+
+- **graph lane**: the lane of the original `spat.graph_compute_batch`;
+- **physical fragment slot**: the slot in the graph batch result;
+- **scheduled lane**: one lane of a `spat.scheduled_compute_batch` equivalence
+  class;
+- **physical core**: the core selected by PEFT.
+
+The graph batch body or its Blueprint defines graph-lane-to-fragment-slot and
+fragment-slot-to-logical-region mappings.
+
+PEFT defines graph-instance-to-core placement.
+
+Scheduled communication defines how values move between cores.
+
+## Scheduled IR exclusion
+
+Do not add a verifier requiring `spat.scheduled_compute_batch` results to have
+`laneCount` as their first dimension.
+
+Do not rewrite scheduled values merely to resemble graph physical fragment
+collections.
+
+When lowering graph IR into scheduled IR:
+
+- resolve graph fragments and reconstruction metadata before erasing their
+  graph owners;
+- create local forwarding or `spat.channel_send`/`spat.channel_receive` for
+  cross-core dependencies;
+- allow scheduled result representation to follow the scheduled IR contract;
+- preserve numerical and deadlock correctness.
+
+The graph invariant is an input contract for scheduling, not a scheduled-value
+layout contract.
+
+## Required verifier properties
+
+`spat.graph_compute_batch` verification must establish, for every result:
+
+1. the result is a static or otherwise supported ranked tensor;
+2. result rank is exactly `fragment rank + 1`;
+3. result dimension 0 equals `laneCount`;
+4. every lane publication source has the same exact fragment type;
+5. the physical insertion targets the corresponding result block argument;
+6. physical insertion offsets have the fragment slot in dimension 0;
+7. all remaining physical offsets are zero;
+8. physical sizes are `[1] + fragment shape`;
+9. physical strides are unit;
+10. exactly one publication is defined for each graph result in the per-lane
+    body.
+
+These checks apply only to `spat.graph_compute_batch`, not to
+`spat.scheduled_compute_batch`.
+
+Blueprint verification must establish that every logical reconstruction entry:
+
+- references an existing physical batch operand;
+- references a valid physical fragment slot;
+- maps a fragment compatible with the declared logical slice;
+- stays within logical bounds;
+- follows the declared conflict and coverage policies.
+
+## Invalid representations
+
+The following are invariant violations.
+
+### Logical aggregate returned directly by graph batch
+
+```text
+laneCount = 16
+result     = tensor<1x4x16x16>
+```
+
+with each lane inserting into logical dimension 2.
+
+This is a logical assembly masquerading as a graph batch result. The graph
+result must instead be `16 × per-row-fragment`, and a Blueprint must describe
+placement into `tensor<1x4x16x16>`.
+
+### Physical storage derived from logical destination shape
+
+Code equivalent to:
+
+```cpp
+shape = logicalDestinationType.getShape();
+shape[logicalInsertionDimension] = laneCount;
+```
+
+is invalid.
+
+Physical graph storage must be derived from the per-lane fragment type:
+
+```cpp
+physicalShape = [laneCount] + fragmentType.getShape();
+```
+
+### Reconstruction inferred from result type
+
+It is invalid to assume that physical slot `i` belongs at logical offset `i`.
+The Blueprint or another explicit reconstruction descriptor must state the
+mapping.
+
+### Blueprint used for arithmetic
+
+It is invalid to encode `fragment0 + fragment1` as Blueprint reconstruction.
+Create a following graph compute or graph compute batch for the addition.
+
+## Ownership
+
+- ONNX-to-Spatial lowering owns creation of valid graph fragment batches.
+- Graph canonicalization owns normalization of temporary logical-assembly forms
+  into physical graph batches plus Blueprints.
+- `spat.graph_compute_batch` verifier rejects invalid physical publications.
+- `spat.blueprint` owns persistent logical reconstruction metadata.
+- Deferred communication Phase 1 owns complete consumer-side constant shaping.
+- Merge scheduling consumes this graph contract and introduces explicit
+  communication.
+- Scheduled IR verifiers validate scheduled execution and communication, not
+  the graph fragment representation.
+
+## No repair downstream
+
+If graph IR violates this invariant, fix the graph producer or graph
+canonicalization.
+
+Do not repair an invalid graph batch by:
+
+- guessing a lane dimension in `MergeComputeNodes`;
+- deriving physical storage from a logical destination tensor;
+- inspecting deferred-result users;
+- reconstructing omitted Blueprint data after graph erasure;
+- weakening graph verifiers;
+- imposing the graph representation on scheduled operations.

.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md

@@ -0,0 +1,85 @@
+# Compiler Performance Optimization Invariant
+
+## Scope
+
+This invariant applies to:
+
+- ONNX-to-Spatial lowering;
+- graph and trivial merge;
+- PEFT scheduling;
+- scheduled materialization;
+- deferred communication;
+- Spatial-to-PIM lowering;
+- bufferization;
+- liveness planning;
+- PIM code generation.
+
+## Invariant
+
+A compiler compile-time or memory optimization must not reduce available
+hardware parallelism or worsen theoretical execution time. In the absence of a
+precise runtime model, uncertainty is not permission to accept a possible
+runtime regression. It is forbidden to serialize independent work, reduce
+the number of simultaneously schedulable compute instances, replace parallel
+graph or core batches with sequential loops, introduce recomputation, add
+runtime copies, increase communication or instruction count, or lengthen the
+schedule critical path merely to reduce compiler time or memory usage.
+
+Compiler representations may share analysis, planning, or code-generation
+work only when the emitted execution retains the same independent work,
+schedule semantics, scheduling granularity, and available parallelism.
+
+An optimization is acceptable only when its static runtime proxies are equal or
+better.
+
+## Asymptotic cost
+
+For each new or materially changed compiler algorithm, identify the relevant
+input size and target linear or sublinear time and space. Avoid repeated full-IR
+walks, nested scans, and per-operation recomputation when indexing, caching, or
+a single traversal can express the same behavior.
+
+When linear-or-better complexity is not possible, use the lowest justified
+complexity and report the actual time and space Big-O, the input variable, and
+why a lower bound is not practical. Include that cost in the final report; do
+not hide a superlinear path behind small current test sizes.
+
+## Forbidden optimization trades
+
+Do not:
+
+- use fewer parallel lanes or cores to simplify IR;
+- coarsen lane, core, batch, or scheduling granularity when it may reduce
+  concurrency;
+- scalarize a valid graph or core batch;
+- replace independent operations with one sequential loop;
+- recompute values to reduce storage;
+- add device/device or host/device copies to reduce peak memory;
+- change hardware parameters to make a benchmark easier;
+- reduce communication concurrency by imposing a global serial order;
+- increase emitted instruction count to reduce compiler analysis;
+- move PIM work to the host.
+
+## Required proof
+
+Every performance change must report before and after:
+
+- compiler wall time;
+- compiler peak RSS;
+- IR operation and value counts at the changed stage;
+- logical compute-instance count;
+- graph compute batch and scheduled batch lane counts;
+- PEFT class and core assignment;
+- maximum scheduled critical-path or step count;
+- MVM/VMM and vector instruction counts;
+- send and receive counts and bytes;
+- total emitted instruction count;
+- maximum instructions assigned to one core;
+- number and size of runtime memory copies;
+- final host, weights, logical-core, physical-core, and maximum-core memory;
+- numerical validation.
+
+## Stop rule
+
+If compile time or memory improves but a runtime proxy worsens, stop and reject
+the change.

.gitignore

@@ -11,4 +11,6 @@ build_*
 compile.sh
 pimcomp_utils/*
 
-**/__*
+*.zip
+**/*.zip
+**/__pycache__/

AGENTS.md

@@ -1,4 +1,11 @@
 * Always read the full README.md before doing anything
+
+# Required project invariants
+
+Before modifying the relevant subsystem, read:
+
+* `.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md`
+* `.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md`
 * Build commands:
   * `cmake --build ./build_release`
   * `cmake --build ./build_debug`
@@ -6,6 +13,8 @@
 * Always try the release build first before building with the debug version
 * Use the debug build only when it is useful to obtain a clear stack trace with symbols, inspect names, place breakpoints, or test a small case interactively
 * The debug build is very slow, so use it only on small fast tests such as operation validations, not on network validations
+* Always prepend rtk to shell commands if missing and if rtk is available
+* Always use the repository Python virtual environment (`.venv/bin/python` and its bundled tools) for Python commands
 
 # Core engineering philosophy
 
@@ -163,6 +172,10 @@ Do not refactor when:
 * If a change adds a walk, cache, analysis, or structural traversal, justify why it is needed
 * For hot paths, prefer preserving existing asymptotic behavior unless a better structure is part of the requested change
 * If performance may change, mention the expected impact and suggest a targeted timing check
+* Compile-time and compiler-memory improvements must not trade away hardware
+  parallelism or theoretical runtime. Preserve or improve the static runtime
+  proxies defined in
+  `.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md`.
 
 # Goal-driven execution
 

README.md

@@ -67,20 +67,19 @@ ONNX-MLIR -> Spatial -> Pim (tensor) -> Pim (bufferized) -> PIM artifacts
    Converts tensor-semantics PIM IR into memref-semantics PIM IR using MLIR's
    bufferization interfaces.
 
-5. **Static memory coalescing**
+5. **PIM local-memory planning**
-   (`src/PIM/Dialect/Pim/Transforms/StaticMemoryCoalescing`).
+   (`src/PIM/Dialect/Pim/Transforms/LocalMemoryPlanning`).
-   Reuses compatible local memref allocations inside PIM cores before codegen.
+   Computes whole-core lifetimes, reuses addresses for non-overlapping
-
+   allocations, and records the explicit plan in PIM IR.
-6. **PIM code generation** (`src/PIM/Pass/PimCodegen` and
+6. **PIM verification and code generation** (`src/PIM/Pass/PimCodegen` and
    `src/PIM/Compiler`).
-   Folds host constants, materializes remaining host constants, verifies PIM IR,
+   Verifies the memory plan and other PIM invariants, then emits `.pim` core
-   emits `.pim` core files, writes weights, and writes `memory.bin` /
+   files, weights, and `memory.bin` / `config.json` without rerunning liveness.
-   `config.json`.
 
 Supporting pieces:
 - `src/PIM/Common` - shared IR, filesystem, diagnostics, reports, and utility
   helpers.
-- `src/PIM/Compiler` - PIM compiler options, memory/address planning, binary
+- `src/PIM/Compiler` - PIM compiler options, planned-address materialization, binary
   instruction format, artifact writing, weight emission, and codegen entry
   points.
 - `src/PIM/Conversion/SpatialToGraphviz` - optional Spatial graphviz conversion
@@ -88,32 +87,66 @@ Supporting pieces:
 - `src/PIM/Pass` - pass registration and auxiliary passes.
 - `src/PIM/PimAccelerator.{cpp,hpp}` - ONNX-MLIR accelerator entry point.
 
-## Key compiler options
+## PIM compiler options
 
-Pass these to `onnx-mlir` when compiling for PIM:
+Pass these to `onnx-mlir` when compiling for PIM. These are all Raptor/PIM-specific
+options; `onnx-mlir --help` lists the inherited ONNX-MLIR options.
 
 - `--maccel=PIM` - select the PIM accelerator.
 - `--EmitSpatial`, `--EmitPim`, `--EmitPimBufferized`,
   `--EmitPimCodegen` - stop the PIM pipeline at the requested stage. The PIM
   default is `--EmitPimCodegen`.
 - `--core-count=<N>` - required positive core count for PIM compilation.
-- `--crossbar-size=<N>` - crossbar width/height. Default in code is `2`.
+- `--crossbar-size=<N>` - crossbar width/height. Default in code is `128`.
-- `--crossbar-count=<N>` - crossbars per core. Default in code is `256`.
+- `--crossbar-count=<N>` - crossbars per core. Default in code is `64`.
-- `--pim-merge-scheduler=peft` - merge scheduler. `peft` is the only accepted
+- `--pim-memory-report=<summary|none>` - emit the concise combined memory report
-  value in the current code.
+  under `reports/memory_report.txt`, or disable it. Default is `summary`.
 - `--pim-only-codegen` - assume input is already bufferized PIM IR and only run
   the codegen tail.
 - `--pim-emit-json` - also emit `core_*.json` instruction files alongside
   `core_*.pim`.
+- `--pim-export-spatial-dataflow=<none|spatial1|spatial2|spatial3|spatial4|all>` -
+  control Spatial dataflow CSV reports for the graph, trivially merged graph,
+  scheduled, and realized snapshots under `reports/`. Default is `none`.
+- `--pim-conv-lowering=<auto|legacy|depthwise|packed-im2col|streamed-patch|streamed-packed|output-channel-tiled|input-k-tiled|tiled-2d>` -
+  select the convolution lowering strategy. Default is `auto`.
+- `--pim-conv-im2col-max-elements=<N>` - maximum globally materialized im2col
+  elements per convolution before streaming. Default is `1048576`.
+- `--pim-conv-stream-chunk-positions=<N>` - maximum output positions per
+  streamed convolution chunk. Default is `1024`.
+- `--pim-report-conv-lowering=<true|false>` - emit the bounded convolution
+  lowering report. Default is `true`.
 - `--use-experimental-conv-impl` - use the alternate convolution lowering.
+- `--pim-detect-communication-deadlock` - statically simulate expanded
+  send/receive ordering and reject blocking deadlocks. Default is off.
+- `--pim-materialize-scalar-fanout-global-order` - use the experimental,
+  expensive globally ordered scalar-fanout materializer. Default is off.
+- `--pim-trace-communication-materialization` - emit verbose communication
+  materialization diagnostics and provenance attributes. Default is off.
 - `--ignore-concat-error` - soft-fail a ConcatOp corner case.
 
+## Standard PIM hardware profile
+
+Raptor's standard development and YOLO validation profile is:
+
+| Parameter | Value |
+| --- | ---: |
+| Cores | 144 |
+| Crossbars per core | 64 |
+| Crossbar size | 128 × 128 |
+
+Canonical compiler flags:
+
+`--crossbar-count=64 --crossbar-size=128 --core-count=144`
+
+`--core-count` remains mandatory and must be passed explicitly to the compiler.
+
 Example:
 
 ```bash
 ./build_release/Release/bin/onnx-mlir model.onnx -o /tmp/raptor/model \
   --maccel=PIM --EmitPimCodegen \
-  --crossbar-size=2048 --crossbar-count=256 --core-count=1000
+  --crossbar-count=64 --crossbar-size=128 --core-count=144
 ```
 
 This writes PIM artifacts under `/tmp/raptor/pim/`.
@@ -130,21 +163,32 @@ Python dependencies used by the validation scripts are `numpy`, `onnx`, and
 Per-operation validation from the repository root:
 
 ```bash
-python3 validation/validate.py \
+.venv/bin/python validation/validate.py \
   --raptor-path build_release/Release/bin/onnx-mlir \
   --onnx-include-dir onnx-mlir/include \
-  --core-count 1000
+  --operations-dir validation/operations \
+  --crossbar-count 64 \
+  --crossbar-size 128 \
+  --core-count 144 \
+  --verbose \
+  --raptor-extra-arg=--pim-detect-communication-deadlock \
+  --raptor-extra-arg=--pim-export-spatial-dataflow=none
 ```
 
 Validate one network or a subset by pointing `--operations-dir` at any directory
 containing `.onnx` files:
 
 ```bash
-python3 validation/validate.py \
+.venv/bin/python validation/validate.py \
   --raptor-path build_release/Release/bin/onnx-mlir \
   --onnx-include-dir onnx-mlir/include \
   --operations-dir validation/networks/yolo11n/depth_04 \
-  --crossbar-size 2048 --crossbar-count 256 --core-count 1000
+  --crossbar-count 64 \
+  --crossbar-size 128 \
+  --core-count 144 \
+  --verbose \
+  --raptor-extra-arg=--pim-detect-communication-deadlock \
+  --raptor-extra-arg=--pim-export-spatial-dataflow=none
 ```
 
 Useful validation options:
@@ -167,8 +211,9 @@ Each validation run writes artifacts in the model workspace, for example under
 - `simulation/out.bin` - raw simulator output used for comparison.
 
 The compiler currently dumps dialect snapshots such as `spatial0.mlir`,
-`spatial1_dcp_merged.mlir`, `pim0.mlir`, `pim1_buff.mlir`,
+`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
-`pim2_coalesced.mlir`, and `pim3_folded.mlir` when an output directory is
+`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`, `pim1_buff.mlir`,
+`pim2_folded.mlir`, and `pim3_memory_planned.mlir` when an output directory is
 available.
 
 To rerun the simulator manually with tracing after validation has produced a
@@ -197,7 +242,7 @@ Available operation suites under `validation/operations/`: `add`, `concat`,
 Generated operation tests can be regenerated with:
 
 ```bash
-python3 validation/operations/gen_tests.py
+.venv/bin/python validation/operations/gen_tests.py
 ```
 
 ## Build

backend-simulators/pim/pim-simulator/src/lib/instruction_set/isa.rs

@@ -258,24 +258,23 @@ where
 
     let (memory, crossbars) = core.get_memory_crossbar();
     let crossbar = crossbars.get_mut(group).unwrap();
-    let crossbar_stored_bytes = crossbar.stored_bytes();
-    let crossbar_byte_width = crossbar.width();
-
-    let crossbar_elem_width = crossbar_byte_width / size_of::<M>();
-    ensure!(
-        crossbar_byte_width % size_of::<M>() == 0,
-        "M not divisor of the crosbbar size"
-    );
-
     let crossbar_height = crossbar.height();
-    let crossbar_byte_size = crossbar_byte_width * crossbar_height;
+    let crossbar_stored_bytes = crossbar.stored_bytes();
+    let bytes_per_column = crossbar_height * size_of::<M>();
+    ensure!(bytes_per_column != 0, "crossbar height can not be zero");
+    ensure!(
+        crossbar_stored_bytes % bytes_per_column == 0,
+        "Stored crossbar bytes do not describe an integral number of columns"
+    );
+    let crossbar_elem_width = crossbar_stored_bytes / bytes_per_column;
+    ensure!(crossbar_elem_width != 0, "Crossbar contains no stored columns");
 
     let loads = memory
         .reserve_load(r1_val, crossbar_height * size_of::<F>())?
         .execute_load::<F>()?;
     let load = loads[0];
     let vec: Cow<[M]> = load.up();
-    let matrix = crossbar.load::<M>(crossbar_byte_size)?[0];
+    let matrix = crossbar.load::<M>(crossbar_stored_bytes)?[0];
 
     // --- FAER IMPLEMENTATION ---
 

requirements.txt

@@ -0,0 +1,4 @@
+colorama>=0.4.6,<1
+numpy>=1.26.4,<3
+onnx>=1.17,<2
+-e ./tools/raptor_graph_explorer[dev]

src/PIM/CMakeLists.txt

@@ -117,11 +117,9 @@ add_pim_library(OMPIMAccel
   SpatialOps
   PimOps
   OMONNXToSpatial
-  OMSpatialToGraphviz
   OMSpatialToPim
   OMPimCommon
   OMPimBufferization
-  OMPimMemoryCoalescing
   OMPimHostConstantFolding
   OMPimVerification
   MLIRTensorInferTypeOpInterfaceImpl

src/PIM/Common/CMakeLists.txt

@@ -5,9 +5,14 @@ add_pim_library(OMPimCommon
   IR/ConstantUtils.cpp
   IR/CoreBlockUtils.cpp
   IR/EntryPointUtils.cpp
+  IR/IndexingUtils.cpp
   IR/LoopUtils.cpp
   IR/ShapeUtils.cpp
+  IR/ShapingUtils.cpp
+  IR/StaticIntSequence.cpp
+  IR/StaticIntGrid.cpp
   IR/SubviewUtils.cpp
+  IR/TensorSliceUtils.cpp
   IR/WeightUtils.cpp
   Support/CheckedArithmetic.cpp
   Support/DebugDump.cpp

src/PIM/Common/IR/AddressAnalysis.cpp

@@ -1,3 +1,4 @@
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
@@ -7,6 +8,7 @@
 #include <limits>
 
 #include "src/Accelerators/PIM/Common/IR/AddressAnalysis.hpp"
+#include "src/Accelerators/PIM/Common/IR/AffineUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
 #include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
 
@@ -34,12 +36,25 @@ mlir::Value resolveAlias(mlir::Value value, const StaticValueKnowledge* knowledg
 
 llvm::FailureOr<CompiledIndexExpr> compileIndexValueImpl(mlir::Value value);
 llvm::FailureOr<CompiledAddressExpr> compileContiguousAddressExprImpl(mlir::Value value);
+mlir::Value resolveLoopCarriedAliasImpl(mlir::Value value, const StaticValueKnowledge* knowledge);
 
 template <typename... Args>
 CompiledIndexExpr makeCompiledIndexExpr(Args&&... args) {
   return CompiledIndexExpr(std::make_shared<CompiledIndexExprNode>(std::forward<Args>(args)...));
 }
 
+static mlir::Value resolveForYieldedAliasToInit(mlir::scf::ForOp forOp,
+                                                mlir::Value yieldedValue,
+                                                const StaticValueKnowledge* knowledge) {
+  yieldedValue = resolveLoopCarriedAliasImpl(yieldedValue, knowledge);
+  if (auto blockArgument = mlir::dyn_cast<mlir::BlockArgument>(yieldedValue)) {
+    if (blockArgument.getOwner() == forOp.getBody() && blockArgument.getArgNumber() > 0
+        && static_cast<unsigned>(blockArgument.getArgNumber() - 1) < forOp.getInitArgs().size())
+      return resolveLoopCarriedAliasImpl(forOp.getInitArgs()[blockArgument.getArgNumber() - 1], knowledge);
+  }
+  return yieldedValue;
+}
+
 mlir::Value resolveLoopCarriedAliasImpl(mlir::Value value, const StaticValueKnowledge* knowledge) {
   value = resolveAlias(value, knowledge);
 
@@ -56,6 +71,15 @@ mlir::Value resolveLoopCarriedAliasImpl(mlir::Value value, const StaticValueKnow
         return resolveLoopCarriedAliasImpl(tiedOperand->get(), knowledge);
   }
 
+  if (auto forOp = mlir::dyn_cast<mlir::scf::ForOp>(definingOp)) {
+    auto result = mlir::dyn_cast<mlir::OpResult>(value);
+    if (result) {
+      auto yieldOp = mlir::dyn_cast<mlir::scf::YieldOp>(forOp.getBody()->getTerminator());
+      if (yieldOp && result.getResultNumber() < yieldOp.getNumOperands())
+        return resolveForYieldedAliasToInit(forOp, yieldOp.getOperand(result.getResultNumber()), knowledge);
+    }
+  }
+
   if (auto castOp = mlir::dyn_cast<mlir::memref::CastOp>(definingOp))
     return resolveLoopCarriedAliasImpl(castOp.getSource(), knowledge);
   if (auto collapseOp = mlir::dyn_cast<mlir::memref::CollapseShapeOp>(definingOp))
@@ -369,6 +393,11 @@ llvm::FailureOr<int64_t> resolveIndexValueImpl(mlir::Value value, const StaticVa
   if (!definingOp)
     return mlir::failure();
 
+  if (auto affineApplyOp = mlir::dyn_cast<mlir::affine::AffineApplyOp>(definingOp))
+    return evaluateAffineApply(affineApplyOp, [&](mlir::Value operand) {
+      return resolveIndexValueImpl(operand, knowledge);
+    });
+
   if (auto indexCastOp = mlir::dyn_cast<mlir::arith::IndexCastOp>(definingOp))
     return resolveIndexValueImpl(indexCastOp.getIn(), knowledge);
 
@@ -499,16 +528,25 @@ llvm::FailureOr<ResolvedContiguousAddress> resolveContiguousAddressImpl(mlir::Va
         return mlir::failure();
 
       auto yieldOp = mlir::cast<mlir::scf::YieldOp>(forOp.getBody()->getTerminator());
-      mlir::Value yieldedValue = resolveLoopCarriedAliasImpl(yieldOp.getOperand(result.getResultNumber()), knowledge);
+      value = resolveForYieldedAliasToInit(forOp, yieldOp.getOperand(result.getResultNumber()), knowledge);
-      if (auto blockArgument = mlir::dyn_cast<mlir::BlockArgument>(yieldedValue)) {
-        if (blockArgument.getOwner() == forOp.getBody() && blockArgument.getArgNumber() > 0
-            && static_cast<unsigned>(blockArgument.getArgNumber() - 1) < forOp.getInitArgs().size()) {
-          value = resolveAlias(forOp.getInitArgs()[blockArgument.getArgNumber() - 1], knowledge);
       continue;
     }
-      }
 
-      value = yieldedValue;
+    if (auto ifOp = mlir::dyn_cast<mlir::scf::IfOp>(definingOp)) {
+      auto result = mlir::dyn_cast<mlir::OpResult>(value);
+      if (!result)
+        return mlir::failure();
+
+      auto condition = resolveIndexValueImpl(ifOp.getCondition(), knowledge);
+      if (failed(condition))
+        return mlir::failure();
+
+      mlir::Region& selectedRegion = *condition != 0 ? ifOp.getThenRegion() : ifOp.getElseRegion();
+      auto yieldOp = mlir::dyn_cast<mlir::scf::YieldOp>(selectedRegion.front().getTerminator());
+      if (!yieldOp || result.getResultNumber() >= yieldOp.getNumOperands())
+        return mlir::failure();
+
+      value = resolveLoopCarriedAliasImpl(yieldOp.getOperand(result.getResultNumber()), knowledge);
       continue;
     }
 
@@ -609,17 +647,35 @@ llvm::FailureOr<CompiledAddressExpr> compileContiguousAddressExprImpl(mlir::Valu
         return mlir::failure();
 
       auto yieldOp = mlir::cast<mlir::scf::YieldOp>(forOp.getBody()->getTerminator());
-      mlir::Value yieldedValue = yieldOp.getOperand(result.getResultNumber());
+      value = resolveForYieldedAliasToInit(forOp, yieldOp.getOperand(result.getResultNumber()), nullptr);
-      if (auto blockArgument = mlir::dyn_cast<mlir::BlockArgument>(yieldedValue)) {
-        if (blockArgument.getOwner() == forOp.getBody() && blockArgument.getArgNumber() > 0
-            && static_cast<unsigned>(blockArgument.getArgNumber() - 1) < forOp.getInitArgs().size()) {
-          value = forOp.getInitArgs()[blockArgument.getArgNumber() - 1];
       continue;
     }
-      }
 
-      value = yieldedValue;
+    if (auto ifOp = mlir::dyn_cast<mlir::scf::IfOp>(definingOp)) {
-      continue;
+      auto result = mlir::dyn_cast<mlir::OpResult>(value);
+      if (!result)
+        return mlir::failure();
+
+      auto thenYield = mlir::dyn_cast<mlir::scf::YieldOp>(ifOp.getThenRegion().front().getTerminator());
+      auto elseYield = mlir::dyn_cast<mlir::scf::YieldOp>(ifOp.getElseRegion().front().getTerminator());
+      if (!thenYield || !elseYield || result.getResultNumber() >= thenYield.getNumOperands()
+          || result.getResultNumber() >= elseYield.getNumOperands()) {
+        return mlir::failure();
+      }
+
+      auto thenAddress = compileContiguousAddressExprImpl(thenYield.getOperand(result.getResultNumber()));
+      auto elseAddress = compileContiguousAddressExprImpl(elseYield.getOperand(result.getResultNumber()));
+      if (failed(thenAddress) || failed(elseAddress) || thenAddress->base != elseAddress->base)
+        return mlir::failure();
+
+      auto condition = compileIndexValueImpl(ifOp.getCondition());
+      if (failed(condition))
+        return mlir::failure();
+
+      CompiledIndexExprNode selectExpr;
+      selectExpr.kind = CompiledIndexExprNode::Kind::Select;
+      selectExpr.operands = {*condition, thenAddress->byteOffset, elseAddress->byteOffset};
+      return CompiledAddressExpr {thenAddress->base, makeCompiledIndexExpr(std::move(selectExpr))};
     }
 
     if (auto subviewOp = mlir::dyn_cast<mlir::memref::SubViewOp>(definingOp)) {
@@ -801,7 +857,7 @@ llvm::FailureOr<ResolvedContiguousAddress> CompiledAddressExpr::evaluate(const S
   auto resolvedOffset = byteOffset.evaluate(knowledge);
   if (failed(resolvedOffset))
     return mlir::failure();
-  return ResolvedContiguousAddress {base, *resolvedOffset};
+  return ResolvedContiguousAddress {resolveAlias(base, &knowledge), *resolvedOffset};
 }
 
 } // namespace onnx_mlir

src/PIM/Common/IR/AffineUtils.cpp

@@ -31,7 +31,7 @@ static FailureOr<int64_t> ceilDivSigned(int64_t lhs, int64_t rhs) {
 }
 
 Value createOrFoldAffineApply(
-  RewriterBase& rewriter, Location loc, AffineMap map, ValueRange operands, Operation* constantAnchor) {
+  OpBuilder& builder, Location loc, AffineMap map, ValueRange operands, Operation* constantAnchor) {
   assert(constantAnchor && "expected a valid constant anchor");
   assert(map.getNumResults() == 1 && "affine.apply expects a single-result affine map");
 
@@ -40,54 +40,91 @@ Value createOrFoldAffineApply(
   for (Value operand : operands) {
     std::optional<int64_t> constantValue = matchConstantIndexValue(operand);
     if (!constantValue)
-      return affine::AffineApplyOp::create(rewriter, loc, map, operands).getResult();
+      return affine::AffineApplyOp::create(builder, loc, map, operands).getResult();
-    operandConstants.push_back(rewriter.getIndexAttr(*constantValue));
+    operandConstants.push_back(builder.getIndexAttr(*constantValue));
   }
 
   SmallVector<Attribute> foldedResults;
   if (succeeded(map.constantFold(operandConstants, foldedResults)) && foldedResults.size() == 1)
     if (auto constantResult = dyn_cast<IntegerAttr>(foldedResults.front()))
-      return getOrCreateIndexConstant(rewriter, constantAnchor, constantResult.getInt());
+      return getOrCreateIndexConstant(builder, constantAnchor, constantResult.getInt());
 
-  return affine::AffineApplyOp::create(rewriter, loc, map, operands).getResult();
+  return affine::AffineApplyOp::create(builder, loc, map, operands).getResult();
 }
 
 Value createOrFoldAffineApply(
-  RewriterBase& rewriter, Location loc, AffineExpr expr, ValueRange dims, Operation* constantAnchor) {
+  OpBuilder& builder, Location loc, AffineExpr expr, ValueRange dims, Operation* constantAnchor) {
   AffineMap map = AffineMap::get(/*dimCount=*/dims.size(), /*symbolCount=*/0, expr);
-  return createOrFoldAffineApply(rewriter, loc, map, dims, constantAnchor);
+  return createOrFoldAffineApply(builder, loc, map, dims, constantAnchor);
 }
 
-Value affineMulConst(RewriterBase& rewriter, Location loc, Value value, int64_t multiplier, Operation* constantAnchor) {
+Value affineMulConst(OpBuilder& builder, Location loc, Value value, int64_t multiplier, Operation* constantAnchor) {
   assert(constantAnchor && "expected a valid constant anchor");
   if (multiplier == 0)
-    return getOrCreateIndexConstant(rewriter, constantAnchor, 0);
+    return getOrCreateIndexConstant(builder, constantAnchor, 0);
   if (multiplier == 1)
     return value;
 
-  AffineExpr d0 = getAffineDimExpr(0, rewriter.getContext());
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
-  return createOrFoldAffineApply(rewriter, loc, d0 * multiplier, ValueRange {value}, constantAnchor);
+  return createOrFoldAffineApply(builder, loc, d0 * multiplier, ValueRange {value}, constantAnchor);
 }
 
-Value affineModConst(RewriterBase& rewriter, Location loc, Value value, int64_t divisor, Operation* constantAnchor) {
+Value affineAddConst(OpBuilder& builder, Location loc, Value value, int64_t offset, Operation* constantAnchor) {
+  assert(constantAnchor && "expected a valid constant anchor");
+  if (offset == 0)
+    return value;
+
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
+  return createOrFoldAffineApply(builder, loc, d0 + offset, ValueRange {value}, constantAnchor);
+}
+
+Value affineModConst(OpBuilder& builder, Location loc, Value value, int64_t divisor, Operation* constantAnchor) {
   assert(constantAnchor && "expected a valid constant anchor");
   assert(divisor > 0 && "expected a positive affine.mod divisor");
   if (divisor == 1)
-    return getOrCreateIndexConstant(rewriter, constantAnchor, 0);
+    return getOrCreateIndexConstant(builder, constantAnchor, 0);
 
-  AffineExpr d0 = getAffineDimExpr(0, rewriter.getContext());
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
-  return createOrFoldAffineApply(rewriter, loc, d0 % divisor, ValueRange {value}, constantAnchor);
+  return createOrFoldAffineApply(builder, loc, d0 % divisor, ValueRange {value}, constantAnchor);
 }
 
 Value affineFloorDivConst(
-  RewriterBase& rewriter, Location loc, Value value, int64_t divisor, Operation* constantAnchor) {
+  OpBuilder& builder, Location loc, Value value, int64_t divisor, Operation* constantAnchor) {
   assert(constantAnchor && "expected a valid constant anchor");
   assert(divisor > 0 && "expected a positive affine.floor_div divisor");
   if (divisor == 1)
     return value;
 
-  AffineExpr d0 = getAffineDimExpr(0, rewriter.getContext());
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
-  return createOrFoldAffineApply(rewriter, loc, d0.floorDiv(divisor), ValueRange {value}, constantAnchor);
+  return createOrFoldAffineApply(builder, loc, d0.floorDiv(divisor), ValueRange {value}, constantAnchor);
+}
+
+Value affineAddModConst(
+  OpBuilder& builder, Location loc, Value value, int64_t offset, int64_t divisor, Operation* constantAnchor) {
+  assert(constantAnchor && "expected a valid constant anchor");
+  assert(divisor > 0 && "expected a positive affine.mod divisor");
+  if (divisor == 1)
+    return getOrCreateIndexConstant(builder, constantAnchor, 0);
+
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
+  AffineExpr expr = d0;
+  if (offset != 0)
+    expr = expr + offset;
+  return createOrFoldAffineApply(builder, loc, expr % divisor, ValueRange {value}, constantAnchor);
+}
+
+Value affineAddFloorDivConst(
+  OpBuilder& builder, Location loc, Value value, int64_t offset, int64_t divisor, Operation* constantAnchor) {
+  assert(constantAnchor && "expected a valid constant anchor");
+  assert(divisor > 0 && "expected a positive affine.floor_div divisor");
+  if (divisor == 1)
+    return offset == 0 ? value : affineAddConst(builder, loc, value, offset, constantAnchor);
+
+  AffineExpr d0 = getAffineDimExpr(0, builder.getContext());
+  AffineExpr expr = d0;
+  if (offset != 0)
+    expr = expr + offset;
+  return createOrFoldAffineApply(builder, loc, expr.floorDiv(divisor), ValueRange {value}, constantAnchor);
 }
 
 FailureOr<int64_t> evaluateAffineExpr(AffineExpr expr, ArrayRef<int64_t> dims, ArrayRef<int64_t> symbols) {

src/PIM/Common/IR/AffineUtils.hpp

@@ -11,36 +11,56 @@ namespace onnx_mlir {
 
 using IndexValueResolver = llvm::function_ref<llvm::FailureOr<int64_t>(mlir::Value)>;
 
-mlir::Value createOrFoldAffineApply(mlir::RewriterBase& rewriter,
+mlir::Value createOrFoldAffineApply(mlir::OpBuilder& builder,
                                     mlir::Location loc,
                                     mlir::AffineMap map,
                                     mlir::ValueRange operands,
                                     mlir::Operation* constantAnchor);
 
-mlir::Value createOrFoldAffineApply(mlir::RewriterBase& rewriter,
+mlir::Value createOrFoldAffineApply(mlir::OpBuilder& builder,
                                     mlir::Location loc,
                                     mlir::AffineExpr expr,
                                     mlir::ValueRange dims,
                                     mlir::Operation* constantAnchor);
 
-mlir::Value affineMulConst(mlir::RewriterBase& rewriter,
+mlir::Value affineMulConst(mlir::OpBuilder& builder,
                            mlir::Location loc,
                            mlir::Value value,
                            int64_t multiplier,
                            mlir::Operation* constantAnchor);
 
-mlir::Value affineModConst(mlir::RewriterBase& rewriter,
+mlir::Value affineAddConst(mlir::OpBuilder& builder,
+                           mlir::Location loc,
+                           mlir::Value value,
+                           int64_t offset,
+                           mlir::Operation* constantAnchor);
+
+mlir::Value affineModConst(mlir::OpBuilder& builder,
                            mlir::Location loc,
                            mlir::Value value,
                            int64_t divisor,
                            mlir::Operation* constantAnchor);
 
-mlir::Value affineFloorDivConst(mlir::RewriterBase& rewriter,
+mlir::Value affineFloorDivConst(mlir::OpBuilder& builder,
                                 mlir::Location loc,
                                 mlir::Value value,
                                 int64_t divisor,
                                 mlir::Operation* constantAnchor);
 
+mlir::Value affineAddModConst(mlir::OpBuilder& builder,
+                              mlir::Location loc,
+                              mlir::Value value,
+                              int64_t offset,
+                              int64_t divisor,
+                              mlir::Operation* constantAnchor);
+
+mlir::Value affineAddFloorDivConst(mlir::OpBuilder& builder,
+                                   mlir::Location loc,
+                                   mlir::Value value,
+                                   int64_t offset,
+                                   int64_t divisor,
+                                   mlir::Operation* constantAnchor);
+
 llvm::FailureOr<int64_t>
 evaluateAffineExpr(mlir::AffineExpr expr, llvm::ArrayRef<int64_t> dims, llvm::ArrayRef<int64_t> symbols = {});
 

src/PIM/Common/IR/BatchCoreUtils.cpp

@@ -1,5 +1,6 @@
 #include "src/Accelerators/PIM/Common/IR/BatchCoreUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
+#include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
 
 namespace onnx_mlir {
 
@@ -9,6 +10,65 @@ llvm::SmallVector<int32_t> getBatchCoreIds(pim::PimCoreBatchOp coreBatchOp) {
   return llvm::SmallVector<int32_t>(coreIdsAttr.asArrayRef().begin(), coreIdsAttr.asArrayRef().end());
 }
 
+mlir::FailureOr<std::optional<int32_t>>
+getOptionalScheduledCoreId(spatial::SpatScheduledCompute computeOp, llvm::StringRef fieldName) {
+  auto coreIdAttr = computeOp->getAttrOfType<mlir::IntegerAttr>(onnx_mlir::kCoreIdAttrName);
+  if (!coreIdAttr)
+    return std::optional<int32_t> {};
+  if (coreIdAttr.getInt() < 0) {
+    computeOp.emitOpError() << fieldName << " must be non-negative";
+    return mlir::failure();
+  }
+  auto checkedCoreId = pim::checkedI32(coreIdAttr.getInt(), computeOp, fieldName);
+  if (mlir::failed(checkedCoreId))
+    return mlir::failure();
+  return std::optional<int32_t> {*checkedCoreId};
+}
+
+mlir::FailureOr<int32_t> getRequiredScheduledCoreId(spatial::SpatScheduledCompute computeOp, llvm::StringRef fieldName) {
+  auto coreId = getOptionalScheduledCoreId(computeOp, fieldName);
+  if (mlir::failed(coreId))
+    return mlir::failure();
+  if (!*coreId) {
+    computeOp.emitOpError() << "missing required " << onnx_mlir::kCoreIdAttrName;
+    return mlir::failure();
+  }
+  return **coreId;
+}
+
+mlir::FailureOr<std::optional<llvm::SmallVector<int32_t>>>
+getOptionalScheduledBatchCoreIds(spatial::SpatScheduledComputeBatch computeBatchOp, llvm::StringRef fieldName) {
+  auto coreIdsAttr = computeBatchOp->getAttrOfType<mlir::DenseI32ArrayAttr>(onnx_mlir::kCoreIdsAttrName);
+  if (!coreIdsAttr)
+    return std::optional<llvm::SmallVector<int32_t>> {};
+
+  llvm::SmallVector<int32_t> coreIds;
+  coreIds.reserve(coreIdsAttr.size());
+  for (int32_t coreId : coreIdsAttr.asArrayRef()) {
+    if (coreId < 0) {
+      computeBatchOp.emitOpError() << fieldName << " values must be non-negative";
+      return mlir::failure();
+    }
+    auto checkedCoreId = pim::checkedI32(static_cast<int64_t>(coreId), computeBatchOp, fieldName);
+    if (mlir::failed(checkedCoreId))
+      return mlir::failure();
+    coreIds.push_back(*checkedCoreId);
+  }
+  return std::optional<llvm::SmallVector<int32_t>> {std::move(coreIds)};
+}
+
+mlir::FailureOr<llvm::SmallVector<int32_t>>
+getRequiredScheduledBatchCoreIds(spatial::SpatScheduledComputeBatch computeBatchOp, llvm::StringRef fieldName) {
+  auto coreIds = getOptionalScheduledBatchCoreIds(computeBatchOp, fieldName);
+  if (mlir::failed(coreIds))
+    return mlir::failure();
+  if (!*coreIds) {
+    computeBatchOp.emitOpError() << "missing required " << onnx_mlir::kCoreIdsAttrName;
+    return mlir::failure();
+  }
+  return std::move(**coreIds);
+}
+
 llvm::SmallVector<int32_t> getLaneChunkCoreIds(llvm::ArrayRef<int32_t> coreIds, size_t laneCount, unsigned lane) {
   llvm::SmallVector<int32_t> laneCoreIds;
   laneCoreIds.reserve(coreIds.size() / laneCount);

src/PIM/Common/IR/BatchCoreUtils.hpp

@@ -3,12 +3,26 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 
+#include <optional>
+
 #include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 
 namespace onnx_mlir {
 
 llvm::SmallVector<int32_t> getBatchCoreIds(pim::PimCoreBatchOp coreBatchOp);
 
+mlir::FailureOr<std::optional<int32_t>>
+getOptionalScheduledCoreId(spatial::SpatScheduledCompute computeOp, llvm::StringRef fieldName);
+
+mlir::FailureOr<int32_t> getRequiredScheduledCoreId(spatial::SpatScheduledCompute computeOp, llvm::StringRef fieldName);
+
+mlir::FailureOr<std::optional<llvm::SmallVector<int32_t>>>
+getOptionalScheduledBatchCoreIds(spatial::SpatScheduledComputeBatch computeBatchOp, llvm::StringRef fieldName);
+
+mlir::FailureOr<llvm::SmallVector<int32_t>>
+getRequiredScheduledBatchCoreIds(spatial::SpatScheduledComputeBatch computeBatchOp, llvm::StringRef fieldName);
+
 llvm::SmallVector<int32_t> getLaneChunkCoreIds(llvm::ArrayRef<int32_t> coreIds, size_t laneCount, unsigned lane);
 
 bool isExplicitHostMemCopyOperand(mlir::Operation* op, unsigned operandIndex);

src/PIM/Common/IR/CompactAsmUtils.hpp

@@ -1,6 +1,7 @@
 #ifndef ONNX_MLIR_PIM_COMPACT_ASM_UTILS_HPP
 #define ONNX_MLIR_PIM_COMPACT_ASM_UTILS_HPP
 
+#include "mlir/IR/Builders.h"
 #include "mlir/IR/OpImplementation.h"
 #include "mlir/IR/Value.h"
 #include "mlir/Support/LLVM.h"
@@ -9,8 +10,11 @@
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/Support/LogicalResult.h"
 
+#include <optional>
+
 namespace onnx_mlir {
 namespace compact_asm {
 
@@ -21,6 +25,59 @@ enum class ListDelimiter {
   Paren
 };
 
+struct NumericSuffixRange {
+  StringRef prefix;
+  unsigned first = 0;
+  unsigned last = 0;
+};
+
+inline std::optional<NumericSuffixRange> getNumericSuffixRange(StringRef firstName, StringRef lastName) {
+  auto split = [](StringRef name) -> std::optional<std::pair<StringRef, unsigned>> {
+    size_t suffixStart = name.size();
+    while (suffixStart > 0 && name[suffixStart - 1] >= '0' && name[suffixStart - 1] <= '9')
+      --suffixStart;
+    if (suffixStart == name.size())
+      return std::nullopt;
+
+    unsigned number = 0;
+    if (name.drop_front(suffixStart).getAsInteger(10, number))
+      return std::nullopt;
+    return std::pair(name.take_front(suffixStart), number);
+  };
+
+  auto first = split(firstName);
+  auto last = split(lastName);
+  if (!first || !last || first->first != last->first || first->second > last->second)
+    return std::nullopt;
+  return NumericSuffixRange {first->first, first->second, last->second};
+}
+
+inline StringRef getInternedNumberedName(OpAsmParser& parser, StringRef prefix, unsigned number) {
+  return parser.getBuilder().getStringAttr((Twine(prefix) + Twine(number)).str()).getValue();
+}
+
+inline ParseResult parseOptionalRepeatCount(OpAsmParser& parser, int64_t& repeatCount) {
+  repeatCount = 1;
+  if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
+    return success();
+  }
+
+  const char* tokenStart = parser.getCurrentLocation().getPointer();
+  if (!tokenStart || tokenStart[0] != 'x' || tokenStart[1] < '0' || tokenStart[1] > '9')
+    return success();
+
+  StringRef fusedRepeat;
+  if (failed(parser.parseOptionalKeyword(&fusedRepeat)))
+    return failure();
+  if (!fusedRepeat.consume_front("x") || fusedRepeat.empty()
+      || fusedRepeat.getAsInteger(10, repeatCount) || repeatCount <= 0) {
+    return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
+  }
+  return success();
+}
+
 inline ParseResult parseOpenDelimiter(OpAsmParser& parser, ListDelimiter delimiter) {
   if (delimiter == ListDelimiter::Square)
     return parser.parseLSquare();
@@ -59,10 +116,8 @@ inline ParseResult parseCompressedRepeatedList(OpAsmParser& parser,
       return failure();
 
     int64_t repeatCount = 1;
-    if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parseOptionalRepeatCount(parser, repeatCount))
-      if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return failure();
-        return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-    }
     for (int64_t index = 0; index < repeatCount; ++index)
       entries.push_back(entry);
 
@@ -86,10 +141,8 @@ parseCompressedIntegerEntries(OpAsmParser& parser, ListDelimiter delimiter, Smal
         return failure();
 
       int64_t repeatCount = 1;
-      if (succeeded(parser.parseOptionalKeyword("x"))) {
+      if (parseOptionalRepeatCount(parser, repeatCount))
-        if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+        return failure();
-          return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-      }
       for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
         llvm::append_range(values, subgroup);
     }
@@ -109,10 +162,8 @@ parseCompressedIntegerEntries(OpAsmParser& parser, ListDelimiter delimiter, Smal
             return parser.emitError(parser.getCurrentLocation(), "step after 'by' must be positive");
         }
         int64_t repeatCount = 1;
-        if (succeeded(parser.parseOptionalKeyword("x"))) {
+        if (parseOptionalRepeatCount(parser, repeatCount))
-          if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+          return failure();
-            return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-        }
         if ((last - first) % step != 0) {
           return parser.emitError(parser.getCurrentLocation(),
                                   "range end must be reachable from start using the given step");
@@ -124,10 +175,8 @@ parseCompressedIntegerEntries(OpAsmParser& parser, ListDelimiter delimiter, Smal
       }
       else {
         int64_t repeatCount = 1;
-        if (succeeded(parser.parseOptionalKeyword("x"))) {
+        if (parseOptionalRepeatCount(parser, repeatCount))
-          if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+          return failure();
-            return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-        }
         for (int64_t index = 0; index < repeatCount; ++index)
           values.push_back(static_cast<IntT>(first));
       }
@@ -406,10 +455,8 @@ inline ParseResult parseCompressedTypeSequence(OpAsmParser& parser, SmallVectorI
 
   auto appendType = [&](Type type) -> ParseResult {
     int64_t repeatCount = 1;
-    if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parseOptionalRepeatCount(parser, repeatCount))
-      if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return failure();
-        return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-    }
     for (int64_t index = 0; index < repeatCount; ++index)
       types.push_back(type);
     return success();
@@ -432,18 +479,23 @@ inline ParseResult parseCompressedOperandEntryWithFirst(OpAsmParser& parser,
     OpAsmParser::UnresolvedOperand lastOperand;
     if (parser.parseOperand(lastOperand))
       return failure();
-    if (firstOperand.name != lastOperand.name || firstOperand.number > lastOperand.number)
+    if (firstOperand.name == lastOperand.name && firstOperand.number <= lastOperand.number) {
-      return parser.emitError(parser.getCurrentLocation(), "invalid operand range");
       for (unsigned number = firstOperand.number; number <= lastOperand.number; ++number)
         operands.push_back({firstOperand.location, firstOperand.name, number});
       return success();
     }
 
-  int64_t repeatCount = 1;
+    auto range = getNumericSuffixRange(firstOperand.name, lastOperand.name);
-  if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (!range || firstOperand.number != 0 || lastOperand.number != 0)
-    if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return parser.emitError(parser.getCurrentLocation(), "invalid operand range");
-      return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
+    for (unsigned number = range->first; number <= range->last; ++number)
+      operands.push_back({firstOperand.location, getInternedNumberedName(parser, range->prefix, number), 0});
+    return success();
   }
+
+  int64_t repeatCount = 1;
+  if (parseOptionalRepeatCount(parser, repeatCount))
+    return failure();
   for (int64_t index = 0; index < repeatCount; ++index)
     operands.push_back(firstOperand);
   return success();
@@ -560,10 +612,8 @@ inline ParseResult parseCompressedOrTupleOperandList(OpAsmParser& parser,
       return failure();
 
     int64_t repeatCount = 1;
-    if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parseOptionalRepeatCount(parser, repeatCount))
-      if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return failure();
-        return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-    }
     for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
       llvm::append_range(operands, tupleOperands);
 
@@ -574,11 +624,8 @@ inline ParseResult parseCompressedOrTupleOperandList(OpAsmParser& parser,
       if (parseCompressedOperandSequence(parser, tupleOperands) || parser.parseRParen())
         return failure();
 
-      repeatCount = 1;
+      if (parseOptionalRepeatCount(parser, repeatCount))
-      if (succeeded(parser.parseOptionalKeyword("x"))) {
+        return failure();
-        if (parser.parseInteger(repeatCount) || repeatCount <= 0)
-          return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-      }
       for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
         llvm::append_range(operands, tupleOperands);
     }
@@ -608,10 +655,8 @@ parseCompressedOrTupleTypeList(OpAsmParser& parser, ListDelimiter delimiter, Sma
       return failure();
 
     int64_t repeatCount = 1;
-    if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parseOptionalRepeatCount(parser, repeatCount))
-      if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return failure();
-        return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-    }
     for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
       llvm::append_range(types, tupleTypes);
 
@@ -622,11 +667,8 @@ parseCompressedOrTupleTypeList(OpAsmParser& parser, ListDelimiter delimiter, Sma
       if (parseCompressedTypeSequence(parser, tupleTypes, /*allowEmpty=*/false) || parser.parseRParen())
         return failure();
 
-      repeatCount = 1;
+      if (parseOptionalRepeatCount(parser, repeatCount))
-      if (succeeded(parser.parseOptionalKeyword("x"))) {
+        return failure();
-        if (parser.parseInteger(repeatCount) || repeatCount <= 0)
-          return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-      }
       for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
         llvm::append_range(types, tupleTypes);
     }
@@ -639,10 +681,8 @@ parseCompressedOrTupleTypeList(OpAsmParser& parser, ListDelimiter delimiter, Sma
       return failure();
 
     int64_t repeatCount = 1;
-    if (succeeded(parser.parseOptionalKeyword("x"))) {
+    if (parseOptionalRepeatCount(parser, repeatCount))
-      if (parser.parseInteger(repeatCount) || repeatCount <= 0)
+      return failure();
-        return parser.emitError(parser.getCurrentLocation(), "repeat count after 'x' must be positive");
-    }
     for (int64_t repeat = 0; repeat < repeatCount; ++repeat)
       types.push_back(type);
 
@@ -678,10 +718,8 @@ inline ParseResult parseCompressedArgumentEntryWithFirst(OpAsmParser& parser,
     OpAsmParser::Argument lastArgument;
     if (parser.parseArgument(lastArgument))
       return failure();
-    if (firstArgument.ssaName.name != lastArgument.ssaName.name
+    if (firstArgument.ssaName.name == lastArgument.ssaName.name
-        || firstArgument.ssaName.number > lastArgument.ssaName.number) {
+        && firstArgument.ssaName.number <= lastArgument.ssaName.number) {
-      return parser.emitError(parser.getCurrentLocation(), "invalid argument range");
-    }
       for (unsigned number = firstArgument.ssaName.number; number <= lastArgument.ssaName.number; ++number) {
         OpAsmParser::Argument argument;
         argument.ssaName = {firstArgument.ssaName.location, firstArgument.ssaName.name, number};
@@ -690,6 +728,17 @@ inline ParseResult parseCompressedArgumentEntryWithFirst(OpAsmParser& parser,
       return success();
     }
 
+    auto range = getNumericSuffixRange(firstArgument.ssaName.name, lastArgument.ssaName.name);
+    if (!range || firstArgument.ssaName.number != 0 || lastArgument.ssaName.number != 0)
+      return parser.emitError(parser.getCurrentLocation(), "invalid argument range");
+    for (unsigned number = range->first; number <= range->last; ++number) {
+      OpAsmParser::Argument argument;
+      argument.ssaName = {firstArgument.ssaName.location, getInternedNumberedName(parser, range->prefix, number), 0};
+      arguments.push_back(argument);
+    }
+    return success();
+  }
+
   arguments.push_back(firstArgument);
   return success();
 }

src/PIM/Common/IR/ConstantUtils.cpp

@@ -10,6 +10,32 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
+ConstantPool::ConstantPool(Operation *constantAnchor, OpBuilder &builder)
+  : anchor(constantAnchor), block(getConstantInsertionBlock(constantAnchor)),
+    builder(builder) {
+  for (Operation &op : *block)
+    if (auto constant = dyn_cast<arith::ConstantOp>(&op))
+      cache.try_emplace(
+        std::make_pair(constant.getType(), constant.getValue()),
+        constant.getResult());
+}
+
+Value ConstantPool::getIndex(int64_t value) {
+  return get(builder.getIndexType(), builder.getIndexAttr(value));
+}
+
+Value ConstantPool::get(Type type, Attribute value) {
+  auto key = std::make_pair(type, value);
+  if (Value existing = cache.lookup(key))
+    return existing;
+  OpBuilder::InsertionGuard guard(builder);
+  builder.setInsertionPointToStart(block);
+  Value constant = arith::ConstantOp::create(
+    builder, anchor->getLoc(), type, cast<TypedAttr>(value)).getResult();
+  cache.try_emplace(key, constant);
+  return constant;
+}
+
 static std::optional<int64_t> getIndexConstantValue(arith::ConstantOp constantOp) {
   if (!constantOp.getType().isIndex())
     return std::nullopt;
@@ -49,7 +75,7 @@ Value getOrCreateConstant(OperationFolder& folder, Operation* anchorOp, Attribut
   return folder.getOrCreateConstant(hostBlock, arithDialect, value, type);
 }
 
-Value getOrCreateConstant(RewriterBase& rewriter, Operation* anchorOp, Attribute value, Type type) {
+Value getOrCreateConstant(OpBuilder& builder, Operation* anchorOp, Attribute value, Type type) {
   assert(anchorOp && "expected a valid anchor operation");
   Block* hostBlock = getConstantInsertionBlock(anchorOp);
   for (Operation& op : *hostBlock) {
@@ -59,9 +85,16 @@ Value getOrCreateConstant(RewriterBase& rewriter, Operation* anchorOp, Attribute
     return constantOp.getResult();
   }
 
-  OpBuilder::InsertionGuard guard(rewriter);
+  OpBuilder::InsertionGuard guard(builder);
-  rewriter.setInsertionPointToStart(hostBlock);
+  builder.setInsertionPointToStart(hostBlock);
-  return arith::ConstantOp::create(rewriter, anchorOp->getLoc(), type, cast<TypedAttr>(value)).getResult();
+  return arith::ConstantOp::create(builder, anchorOp->getLoc(), type, cast<TypedAttr>(value)).getResult();
+}
+
+Value createConstantAtHostBlockStart(OpBuilder& builder, Operation* anchorOp, TypedAttr value) {
+  assert(anchorOp && "expected a valid anchor operation");
+  OpBuilder::InsertionGuard guard(builder);
+  builder.setInsertionPointToStart(getConstantInsertionBlock(anchorOp));
+  return arith::ConstantOp::create(builder, anchorOp->getLoc(), value).getResult();
 }
 
 Value getOrCreateConstantLike(OperationFolder& folder, arith::ConstantOp constantOp) {
@@ -73,9 +106,8 @@ Value getOrCreateIndexConstant(OperationFolder& folder, Operation* anchorOp, int
   return getOrCreateConstant(folder, anchorOp, builder.getIndexAttr(value), builder.getIndexType());
 }
 
-Value getOrCreateIndexConstant(RewriterBase& rewriter, Operation* anchorOp, int64_t value) {
+Value getOrCreateIndexConstant(OpBuilder& builder, Operation* anchorOp, int64_t value) {
-  Builder builder(anchorOp->getContext());
+  return getOrCreateConstant(builder, anchorOp, builder.getIndexAttr(value), builder.getIndexType());
-  return getOrCreateConstant(rewriter, anchorOp, builder.getIndexAttr(value), builder.getIndexType());
 }
 
 void hoistAndUniquifyIndexConstants(func::FuncOp funcOp, RewriterBase& rewriter) {

src/PIM/Common/IR/ConstantUtils.hpp

@@ -6,23 +6,42 @@
 #include "mlir/IR/Value.h"
 #include "mlir/Transforms/FoldUtils.h"
 
+#include "llvm/ADT/DenseMap.h"
+
 #include <optional>
 
 namespace onnx_mlir {
 
+class ConstantPool {
+public:
+  ConstantPool(mlir::Operation *constantAnchor, mlir::OpBuilder &builder);
+
+  mlir::Value getIndex(int64_t value);
+  mlir::Value get(mlir::Type type, mlir::Attribute value);
+
+private:
+  mlir::Operation *anchor;
+  mlir::Block *block;
+  mlir::OpBuilder &builder;
+  llvm::DenseMap<std::pair<mlir::Type, mlir::Attribute>, mlir::Value> cache;
+};
+
 mlir::Block* getConstantInsertionBlock(mlir::Operation* anchorOp);
 
 mlir::Value
 getOrCreateConstant(mlir::OperationFolder& folder, mlir::Operation* anchorOp, mlir::Attribute value, mlir::Type type);
 
 mlir::Value
-getOrCreateConstant(mlir::RewriterBase& rewriter, mlir::Operation* anchorOp, mlir::Attribute value, mlir::Type type);
+getOrCreateConstant(mlir::OpBuilder& builder, mlir::Operation* anchorOp, mlir::Attribute value, mlir::Type type);
+
+mlir::Value
+createConstantAtHostBlockStart(mlir::OpBuilder& builder, mlir::Operation* anchorOp, mlir::TypedAttr value);
 
 mlir::Value getOrCreateConstantLike(mlir::OperationFolder& folder, mlir::arith::ConstantOp constantOp);
 
 mlir::Value getOrCreateIndexConstant(mlir::OperationFolder& folder, mlir::Operation* anchorOp, int64_t value);
 
-mlir::Value getOrCreateIndexConstant(mlir::RewriterBase& rewriter, mlir::Operation* anchorOp, int64_t value);
+mlir::Value getOrCreateIndexConstant(mlir::OpBuilder& builder, mlir::Operation* anchorOp, int64_t value);
 
 void hoistAndUniquifyIndexConstants(mlir::func::FuncOp funcOp, mlir::RewriterBase& rewriter);
 

src/PIM/Common/IR/CoreBlockUtils.cpp

@@ -1,8 +1,10 @@
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
 
-#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/STLExtras.h"
 
 #include "src/Accelerators/PIM/Common/IR/CoreBlockUtils.hpp"
 #include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
@@ -10,7 +12,8 @@
 namespace onnx_mlir {
 
 bool isCoreStaticAddressOp(mlir::Operation* op) {
-  if (mlir::isa<mlir::arith::ConstantOp,
+  if (mlir::isa<mlir::affine::AffineApplyOp,
+                mlir::arith::ConstantOp,
                 mlir::arith::AddIOp,
                 mlir::arith::SubIOp,
                 mlir::arith::MulIOp,
@@ -27,78 +30,62 @@ bool isCoreStaticAddressOp(mlir::Operation* op) {
                 mlir::memref::CollapseShapeOp,
                 mlir::memref::ExpandShapeOp>(op))
     return true;
-
+  auto selectOp = mlir::dyn_cast<mlir::arith::SelectOp>(op);
-  if (auto selectOp = mlir::dyn_cast<mlir::arith::SelectOp>(op))
+  return selectOp && selectOp.getType().isIntOrIndex();
-    return selectOp.getType().isIntOrIndex();
-
-  return false;
 }
 
-mlir::LogicalResult
+namespace {
-walkPimCoreBlock(mlir::Block& block,
+
-                 const StaticValueKnowledge& knowledge,
+enum class CoreWalkMode {
-                 llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)> callback) {
+  ExecuteCommunication,
+  StructuralExtremes
+};
+using CoreWalkCallback = llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)>;
+
+static void propagateRegionResults(mlir::ValueRange results, mlir::Region& region, StaticValueKnowledge& knowledge) {
+  if (region.empty())
+    return;
+  auto yield = mlir::cast<mlir::scf::YieldOp>(region.front().getTerminator());
+  for (auto [result, yielded] : llvm::zip(results, yield.getOperands()))
+    knowledge.aliases[result] = resolveLoopCarriedAlias(yielded, knowledge);
+}
+
+static mlir::LogicalResult walkPimCoreBlockImpl(mlir::Block& block,
+                                                const StaticValueKnowledge& initialKnowledge,
+                                                CoreWalkMode mode,
+                                                const PimCoreCommunicationPlan* communicationPlan,
+                                                CoreWalkCallback callback) {
   bool hasFailure = false;
-  for (mlir::Operation& op : block) {
+  StaticValueKnowledge knowledge = initialKnowledge;
+  llvm::StringRef purpose = mode == CoreWalkMode::ExecuteCommunication ? "communication verification" : "verification";
+  llvm::SmallVector<mlir::Operation*, 0> structuralOperations;
+  llvm::ArrayRef<mlir::Operation*> operations;
+  if (communicationPlan) {
+    auto it = communicationPlan->find(&block);
+    if (it != communicationPlan->end())
+      operations = it->second;
+  }
+  else {
+    structuralOperations.reserve(block.getOperations().size());
+    for (mlir::Operation& op : block)
+      structuralOperations.push_back(&op);
+    operations = structuralOperations;
+  }
+
+  for (mlir::Operation* operation : operations) {
+    mlir::Operation& op = *operation;
     if (mlir::isa<pim::PimHaltOp, mlir::scf::YieldOp>(op) || isCoreStaticAddressOp(&op))
       continue;
     if (auto loadOp = mlir::dyn_cast<mlir::memref::LoadOp>(op);
         loadOp && succeeded(resolveIndexValue(loadOp.getResult(), knowledge)))
       continue;
-
     if (auto forOp = mlir::dyn_cast<mlir::scf::ForOp>(op)) {
-      mlir::Block& loopBody = forOp.getRegion().front();
+      auto lower = resolveIndexValue(forOp.getLowerBound(), knowledge);
-      auto lowerBound = resolveIndexValue(forOp.getLowerBound(), knowledge);
+      auto upper = resolveIndexValue(forOp.getUpperBound(), knowledge);
-      auto upperBound = resolveIndexValue(forOp.getUpperBound(), knowledge);
       auto step = resolveIndexValue(forOp.getStep(), knowledge);
-      if (failed(lowerBound) || failed(upperBound) || failed(step) || *step <= 0) {
+      if (failed(lower) || failed(upper) || failed(step)
-        forOp.emitOpError("requires statically evaluable scf.for bounds for PIM codegen");
+          || (mode == CoreWalkMode::ExecuteCommunication && *step <= 0)) {
-        hasFailure = true;
+        forOp.emitOpError() << "requires statically evaluable scf.for bounds for PIM " << purpose;
-        continue;
-      }
-
-      llvm::SmallVector<mlir::Value> iterValues(forOp.getInitArgs().begin(), forOp.getInitArgs().end());
-      for (int64_t inductionValue = *lowerBound; inductionValue < *upperBound; inductionValue += *step) {
-        StaticValueKnowledge loopKnowledge = knowledge;
-        loopKnowledge.indexValues[forOp.getInductionVar()] = inductionValue;
-        for (auto [iterArg, iterValue] : llvm::zip_equal(forOp.getRegionIterArgs(), iterValues))
-          loopKnowledge.aliases[iterArg] = iterValue;
-
-        if (failed(walkPimCoreBlock(loopBody, loopKnowledge, callback)))
-          hasFailure = true;
-
-        auto yieldOp = mlir::cast<mlir::scf::YieldOp>(loopBody.getTerminator());
-        for (auto [index, yieldedValue] : llvm::enumerate(yieldOp.getOperands()))
-          iterValues[index] = resolveLoopCarriedAlias(yieldedValue, loopKnowledge);
-      }
-      continue;
-    }
-
-    if (failed(callback(op, knowledge)))
-      hasFailure = true;
-  }
-  return mlir::success(!hasFailure);
-}
-
-mlir::LogicalResult walkPimCoreBlockStructurally(
-  mlir::Block& block,
-  const StaticValueKnowledge& knowledge,
-  llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)> callback) {
-  bool hasFailure = false;
-  for (mlir::Operation& op : block) {
-    if (mlir::isa<pim::PimHaltOp, mlir::scf::YieldOp>(op) || isCoreStaticAddressOp(&op))
-      continue;
-    if (auto loadOp = mlir::dyn_cast<mlir::memref::LoadOp>(op);
-        loadOp && succeeded(resolveIndexValue(loadOp.getResult(), knowledge)))
-      continue;
-
-    if (auto forOp = mlir::dyn_cast<mlir::scf::ForOp>(op)) {
-      mlir::Block& loopBody = forOp.getRegion().front();
-      auto lowerBound = resolveIndexValue(forOp.getLowerBound(), knowledge);
-      auto upperBound = resolveIndexValue(forOp.getUpperBound(), knowledge);
-      auto step = resolveIndexValue(forOp.getStep(), knowledge);
-      if (failed(lowerBound) || failed(upperBound) || failed(step)) {
-        forOp.emitOpError("requires statically evaluable scf.for bounds for PIM verification");
         hasFailure = true;
         continue;
       }
@@ -108,30 +95,117 @@ mlir::LogicalResult walkPimCoreBlockStructurally(
         continue;
       }
 
-      llvm::SmallVector<int64_t, 2> samples;
+      mlir::Block& body = forOp.getRegion().front();
-      if (*lowerBound < *upperBound) {
+      llvm::SmallVector<mlir::Value> iterValues(forOp.getInitArgs().begin(), forOp.getInitArgs().end());
-        samples.push_back(*lowerBound);
+      auto visitIteration = [&](int64_t induction, bool carryValues) {
-        int64_t last = *lowerBound + ((*upperBound - 1 - *lowerBound) / *step) * *step;
-        if (last != *lowerBound)
-          samples.push_back(last);
-      }
-
-      for (int64_t inductionValue : samples) {
         StaticValueKnowledge loopKnowledge = knowledge;
-        loopKnowledge.indexValues[forOp.getInductionVar()] = inductionValue;
+        loopKnowledge.indexValues[forOp.getInductionVar()] = induction;
-        for (auto [iterArg, iterValue] : llvm::zip_equal(forOp.getRegionIterArgs(), forOp.getInitArgs()))
+        for (auto [index, iterArg] : llvm::enumerate(forOp.getRegionIterArgs()))
-          loopKnowledge.aliases[iterArg] = iterValue;
+          loopKnowledge.aliases[iterArg] = carryValues ? iterValues[index] : forOp.getInitArgs()[index];
+        hasFailure |= failed(walkPimCoreBlockImpl(body, loopKnowledge, mode, communicationPlan, callback));
+        auto yield = mlir::cast<mlir::scf::YieldOp>(body.getTerminator());
+        for (auto [index, yielded] : llvm::enumerate(yield.getOperands()))
+          iterValues[index] = resolveLoopCarriedAlias(yielded, loopKnowledge);
+      };
 
-        if (failed(walkPimCoreBlockStructurally(loopBody, loopKnowledge, callback)))
+      if (mode == CoreWalkMode::ExecuteCommunication) {
-          hasFailure = true;
+        for (int64_t induction = *lower; induction < *upper; induction += *step)
+          visitIteration(induction, true);
       }
+      else if (*lower < *upper) {
+        visitIteration(*lower, false);
+        int64_t last = *lower + ((*upper - 1 - *lower) / *step) * *step;
+        if (last != *lower)
+          visitIteration(last, false);
+      }
+      for (auto [result, value] : llvm::zip(forOp.getResults(), iterValues))
+        knowledge.aliases[result] = value;
       continue;
     }
 
-    if (failed(callback(op, knowledge)))
+    if (auto ifOp = mlir::dyn_cast<mlir::scf::IfOp>(op)) {
+      auto condition = resolveIndexValue(ifOp.getCondition(), knowledge);
+      if (failed(condition)) {
+        ifOp.emitOpError() << "requires statically evaluable scf.if condition for PIM " << purpose;
         hasFailure = true;
+        continue;
+      }
+      mlir::Region& selected = *condition != 0 ? ifOp.getThenRegion() : ifOp.getElseRegion();
+      if (mode == CoreWalkMode::ExecuteCommunication) {
+        hasFailure |= !selected.empty()
+                   && failed(walkPimCoreBlockImpl(selected.front(), knowledge, mode, communicationPlan, callback));
+      }
+      else {
+        for (mlir::Region* region : {&ifOp.getThenRegion(), &ifOp.getElseRegion()})
+          hasFailure |= !region->empty()
+                     && failed(walkPimCoreBlockImpl(region->front(), knowledge, mode, communicationPlan, callback));
+      }
+      propagateRegionResults(ifOp.getResults(), selected, knowledge);
+      continue;
+    }
+
+    if (auto switchOp = mlir::dyn_cast<mlir::scf::IndexSwitchOp>(op)) {
+      auto selector = resolveIndexValue(switchOp.getArg(), knowledge);
+      if (failed(selector)) {
+        switchOp.emitOpError() << "requires a statically evaluable scf.index_switch selector for PIM " << purpose;
+        hasFailure = true;
+        continue;
+      }
+      mlir::Region* selected = &switchOp.getDefaultRegion();
+      for (auto [caseValue, caseRegion] : llvm::zip(switchOp.getCases(), switchOp.getCaseRegions()))
+        if (caseValue == *selector) {
+          selected = &caseRegion;
+          break;
+        }
+      for (mlir::Region& region : switchOp->getRegions()) {
+        if (mode == CoreWalkMode::ExecuteCommunication && &region != selected)
+          continue;
+        hasFailure |= failed(walkPimCoreBlockImpl(region.front(), knowledge, mode, communicationPlan, callback));
+      }
+      propagateRegionResults(switchOp.getResults(), *selected, knowledge);
+      continue;
+    }
+
+    if (mode != CoreWalkMode::ExecuteCommunication || mlir::isa<pim::PimSendOp, pim::PimReceiveOp>(op))
+      hasFailure |= failed(callback(op, knowledge));
   }
   return mlir::success(!hasFailure);
 }
 
+} // namespace
+
+PimCoreCommunicationPlan buildPimCoreCommunicationPlan(mlir::Block& block) {
+  llvm::DenseSet<mlir::Operation*> communicationAncestors;
+  block.walk([&](mlir::Operation* op) {
+    if (!mlir::isa<pim::PimSendOp, pim::PimReceiveOp>(op))
+      return;
+    for (mlir::Operation* parent = op->getParentOp(); parent; parent = parent->getParentOp())
+      communicationAncestors.insert(parent);
+  });
+
+  PimCoreCommunicationPlan plan;
+  block.walk([&](mlir::Operation* op) {
+    bool isCommunication = mlir::isa<pim::PimSendOp, pim::PimReceiveOp>(op);
+    bool isControlFlow = mlir::isa<mlir::scf::ForOp, mlir::scf::IfOp, mlir::scf::IndexSwitchOp>(op);
+    if (isCommunication || (isControlFlow && (op->getNumResults() != 0 || communicationAncestors.contains(op))))
+      plan[op->getBlock()].push_back(op);
+  });
+  return plan;
+}
+
+mlir::LogicalResult walkPimCoreCommunicationBlock(
+  mlir::Block& block,
+  const PimCoreCommunicationPlan& plan,
+  const StaticValueKnowledge& knowledge,
+  llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)> callback) {
+  return walkPimCoreBlockImpl(block, knowledge, CoreWalkMode::ExecuteCommunication, &plan, callback);
+}
+
+mlir::LogicalResult walkPimCoreBlockStructurally(
+  mlir::Block& block,
+  const StaticValueKnowledge& knowledge,
+  llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)> callback) {
+  return walkPimCoreBlockImpl(block, knowledge, CoreWalkMode::StructuralExtremes, nullptr, callback);
+}
+
 } // namespace onnx_mlir

src/PIM/Common/IR/CoreBlockUtils.hpp

@@ -3,21 +3,28 @@
 #include "mlir/IR/Block.h"
 #include "mlir/Support/LogicalResult.h"
 
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
+#include "llvm/ADT/SmallVector.h"
 
 #include "src/Accelerators/PIM/Common/IR/AddressAnalysis.hpp"
 
 namespace onnx_mlir {
 
+using PimCoreCommunicationPlan = llvm::DenseMap<mlir::Block*, llvm::SmallVector<mlir::Operation*, 8>>;
+
 /// Returns true for ops in a `pim.core` body that only participate in static
 /// address or index computation and therefore do not emit PIM instructions.
 bool isCoreStaticAddressOp(mlir::Operation* op);
 
-/// Walks a `pim.core` body, statically unrolling nested `scf.for` loops when
+/// Walks a `pim.core` body's communication stream, statically unrolling
-/// their bounds are known and invoking `callback` only on instruction-emitting
+/// control flow that contains send/receive operations and invoking `callback`
-/// operations.
+/// on those operations in execution order.
-mlir::LogicalResult
+PimCoreCommunicationPlan buildPimCoreCommunicationPlan(mlir::Block& block);
-walkPimCoreBlock(mlir::Block& block,
+
+mlir::LogicalResult walkPimCoreCommunicationBlock(
+  mlir::Block& block,
+  const PimCoreCommunicationPlan& plan,
   const StaticValueKnowledge& knowledge,
   llvm::function_ref<mlir::LogicalResult(mlir::Operation&, const StaticValueKnowledge&)> callback);
 

src/PIM/Common/IR/IndexingUtils.cpp

@@ -1,6 +1,6 @@
 #include <algorithm>
 
-#include "IndexingUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/IndexingUtils.hpp"
 
 using namespace mlir;
 

src/PIM/Common/IR/ShapeUtils.cpp

@@ -1,6 +1,9 @@
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/ErrorHandling.h"
 
+#include <functional>
+
 #include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
 
 namespace onnx_mlir {
@@ -163,4 +166,80 @@ bool isContiguousSubviewWithDynamicOffsets(llvm::ArrayRef<int64_t> sourceShape,
   return true;
 }
 
+bool hasStaticPositiveShape(llvm::ArrayRef<int64_t> shape) {
+  return llvm::all_of(shape, [](int64_t dim) { return dim > 0; });
+}
+
+bool hasStaticPositiveShape(mlir::RankedTensorType type) {
+  return type.hasStaticShape() && hasStaticPositiveShape(type.getShape());
+}
+
+int64_t getStaticShapeElementCount(llvm::ArrayRef<int64_t> shape) {
+  return std::accumulate(shape.begin(), shape.end(), int64_t {1}, std::multiplies<int64_t> {});
+}
+
+llvm::SmallVector<int64_t> permuteShape(llvm::ArrayRef<int64_t> shape, llvm::ArrayRef<int64_t> permutation) {
+  llvm::SmallVector<int64_t> permutedShape;
+  permutedShape.reserve(permutation.size());
+  for (int64_t axis : permutation)
+    permutedShape.push_back(shape[axis]);
+  return permutedShape;
+}
+
+llvm::SmallVector<int64_t> invertPermutation(llvm::ArrayRef<int64_t> permutation) {
+  llvm::SmallVector<int64_t> inversePermutation(permutation.size());
+  for (auto [newIndex, oldIndex] : llvm::enumerate(permutation))
+    inversePermutation[oldIndex] = static_cast<int64_t>(newIndex);
+  return inversePermutation;
+}
+
+mlir::FailureOr<llvm::SmallVector<int64_t>>
+getTransposePermutationChecked(std::optional<mlir::ArrayAttr> permAttr, int64_t rank) {
+  llvm::SmallVector<int64_t> permutation;
+  if (!permAttr) {
+    permutation.reserve(rank);
+    for (int64_t dim = rank - 1; dim >= 0; --dim)
+      permutation.push_back(dim);
+    return permutation;
+  }
+
+  if (static_cast<int64_t>(permAttr->size()) != rank)
+    return mlir::failure();
+
+  permutation.reserve(permAttr->size());
+  llvm::SmallVector<bool> seen(rank, false);
+  for (mlir::IntegerAttr attr : permAttr->getAsRange<mlir::IntegerAttr>()) {
+    int64_t axis = attr.getInt();
+    if (axis < 0 || axis >= rank || seen[axis])
+      return mlir::failure();
+    seen[axis] = true;
+    permutation.push_back(axis);
+  }
+  return permutation;
+}
+
+llvm::SmallVector<mlir::OpFoldResult> getStaticIndexAttrs(mlir::Builder& builder, llvm::ArrayRef<int64_t> values) {
+  llvm::SmallVector<mlir::OpFoldResult> attrs;
+  attrs.reserve(values.size());
+  for (int64_t value : values)
+    attrs.push_back(builder.getIndexAttr(value));
+  return attrs;
+}
+
+llvm::SmallVector<mlir::OpFoldResult> getUnitStrides(mlir::PatternRewriter& rewriter, int64_t rank) {
+  return llvm::SmallVector<mlir::OpFoldResult>(rank, rewriter.getIndexAttr(1));
+}
+
+llvm::SmallVector<mlir::OpFoldResult> getZeroOffsets(mlir::PatternRewriter& rewriter, int64_t rank) {
+  return llvm::SmallVector<mlir::OpFoldResult>(rank, rewriter.getIndexAttr(0));
+}
+
+llvm::SmallVector<mlir::OpFoldResult> getStaticSizes(mlir::PatternRewriter& rewriter, llvm::ArrayRef<int64_t> shape) {
+  llvm::SmallVector<mlir::OpFoldResult> sizes;
+  sizes.reserve(shape.size());
+  for (int64_t dim : shape)
+    sizes.push_back(rewriter.getIndexAttr(dim));
+  return sizes;
+}
+
 } // namespace onnx_mlir

src/PIM/Common/IR/ShapeUtils.hpp

@@ -2,15 +2,23 @@
 
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/Value.h"
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallVector.h"
 
 #include <cstddef>
+#include <optional>
+#include <type_traits>
+#include <utility>
 
 namespace onnx_mlir {
 
+using HSliceId = size_t;
+using CoreId = size_t;
+
 llvm::SmallVector<int64_t> computeRowMajorStrides(llvm::ArrayRef<int64_t> shape);
 
 llvm::SmallVector<int64_t>
@@ -36,4 +44,69 @@ bool isContiguousSubviewWithDynamicOffsets(llvm::ArrayRef<int64_t> sourceShape,
                                            llvm::ArrayRef<int64_t> staticSizes,
                                            llvm::ArrayRef<int64_t> staticStrides);
 
+template <class A, class B, class C = std::common_type_t<A, B>>
+constexpr C ceilIntegerDivide(A a, B b) {
+  static_assert(std::is_integral_v<A>, "A must be an integer type");
+  static_assert(std::is_integral_v<B>, "B must be an integer type");
+  C ac = static_cast<C>(a);
+  C bc = static_cast<C>(b);
+  return 1 + (ac - 1) / bc;
+}
+
+template <class A, class B, class C = std::common_type_t<A, B>>
+constexpr std::pair<C, C> ceilIntegerDivideWithRemainder(A a, B b) {
+  static_assert(std::is_integral_v<A>, "A must be an integer type");
+  static_assert(std::is_integral_v<B>, "B must be an integer type");
+  C ac = static_cast<C>(a);
+  C bc = static_cast<C>(b);
+  return {ceilIntegerDivide(ac, bc), ac % bc};
+}
+
+template <class T>
+bool isVectorShape(mlir::ArrayRef<T> shape) {
+  return shape.size() == 2 && (shape[0] == 1 || shape[1] == 1);
+}
+
+template <class T>
+bool isMatrixShape(mlir::ArrayRef<T> shape) {
+  return shape.size() == 2;
+}
+
+template <class T>
+bool isHVectorShape(mlir::ArrayRef<T> shape) {
+  return shape.size() == 2 && shape[0] == 1;
+}
+
+inline auto getTensorShape(mlir::Value tensor) {
+  return mlir::cast<mlir::RankedTensorType>(tensor.getType()).getShape();
+}
+
+inline bool haveSameStaticShape(mlir::Value lhs, mlir::Value rhs) {
+  auto lhsType = mlir::dyn_cast<mlir::RankedTensorType>(lhs.getType());
+  auto rhsType = mlir::dyn_cast<mlir::RankedTensorType>(rhs.getType());
+  return lhsType && rhsType && lhsType.hasStaticShape() && rhsType.hasStaticShape()
+      && lhsType.getShape() == rhsType.getShape();
+}
+
+bool hasStaticPositiveShape(mlir::ArrayRef<int64_t> shape);
+
+bool hasStaticPositiveShape(mlir::RankedTensorType type);
+
+int64_t getStaticShapeElementCount(mlir::ArrayRef<int64_t> shape);
+
+llvm::SmallVector<int64_t> permuteShape(mlir::ArrayRef<int64_t> shape, mlir::ArrayRef<int64_t> permutation);
+
+llvm::SmallVector<int64_t> invertPermutation(mlir::ArrayRef<int64_t> permutation);
+
+mlir::FailureOr<llvm::SmallVector<int64_t>> getTransposePermutationChecked(std::optional<mlir::ArrayAttr> permAttr,
+                                                                           int64_t rank);
+
+llvm::SmallVector<mlir::OpFoldResult> getStaticIndexAttrs(mlir::Builder& builder, llvm::ArrayRef<int64_t> values);
+
+llvm::SmallVector<mlir::OpFoldResult> getUnitStrides(mlir::PatternRewriter& rewriter, int64_t rank);
+
+llvm::SmallVector<mlir::OpFoldResult> getZeroOffsets(mlir::PatternRewriter& rewriter, int64_t rank);
+
+llvm::SmallVector<mlir::OpFoldResult> getStaticSizes(mlir::PatternRewriter& rewriter, llvm::ArrayRef<int64_t> shape);
+
 } // namespace onnx_mlir

src/PIM/Common/IR/ShapingUtils.cpp

@@ -0,0 +1,39 @@
+#include "mlir/Dialect/Linalg/IR/Linalg.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+
+#include "ShapingUtils.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+#include "src/Dialect/ONNX/ONNXOps.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+
+bool isShapingOnlyOp(Operation *op) {
+  return isa<tensor::CastOp,
+             tensor::CollapseShapeOp,
+             tensor::ExpandShapeOp,
+             tensor::ExtractSliceOp,
+             tensor::InsertSliceOp,
+             tensor::ConcatOp,
+             tensor::EmptyOp,
+             tensor::ExtractOp,
+             tensor::InsertOp,
+             tensor::SplatOp,
+             linalg::TransposeOp,
+             ONNXTransposeOp,
+             spatial::SpatConcatOp,
+             spatial::SpatExtractRowsOp>(op);
+}
+
+bool isPureIndexComputationOp(Operation *op) {
+  if (op->getNumRegions() != 0 || op->getNumResults() == 0 || op->hasTrait<OpTrait::IsTerminator>()
+      || !isMemoryEffectFree(op))
+    return false;
+  auto isIndexOrInteger = [](Type type) { return type.isIndex() || isa<IntegerType>(type); };
+  return llvm::all_of(op->getOperandTypes(), isIndexOrInteger)
+      && llvm::all_of(op->getResultTypes(), isIndexOrInteger);
+}
+
+} // namespace onnx_mlir

src/PIM/Common/IR/ShapingUtils.hpp

@@ -0,0 +1,13 @@
+#pragma once
+
+namespace mlir {
+class Operation;
+}
+
+namespace onnx_mlir {
+
+bool isShapingOnlyOp(mlir::Operation *op);
+
+bool isPureIndexComputationOp(mlir::Operation *op);
+
+} // namespace onnx_mlir

src/PIM/Common/IR/StaticIntGrid.cpp

@@ -0,0 +1,223 @@
+#include "StaticIntGrid.hpp"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+
+#include "AffineUtils.hpp"
+#include "ConstantUtils.hpp"
+#include <algorithm>
+#include <limits>
+
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static std::optional<size_t> cellCount(size_t rows, size_t columns) {
+  if (!rows || !columns ||
+      rows > static_cast<size_t>(std::numeric_limits<int64_t>::max()) / columns)
+    return std::nullopt;
+  return rows * columns;
+}
+
+static bool checkedFlatIndex(
+    size_t row, size_t columns, size_t column, size_t &flat) {
+  bool overflow;
+  flat = llvm::SaturatingMultiplyAdd(row, columns, column, &overflow);
+  return !overflow &&
+         flat <= static_cast<size_t>(std::numeric_limits<int64_t>::max());
+}
+
+static bool affineValue(int64_t base, int64_t rowStep, int64_t columnStep,
+                        size_t row, size_t column, int64_t &result) {
+  if (row > static_cast<size_t>(std::numeric_limits<int64_t>::max()) ||
+      column > static_cast<size_t>(std::numeric_limits<int64_t>::max()))
+    return false;
+  int64_t rowValue, columnValue;
+  return !llvm::MulOverflow(rowStep, static_cast<int64_t>(row), rowValue)
+      && !llvm::MulOverflow(columnStep, static_cast<int64_t>(column),
+                           columnValue)
+      && !llvm::AddOverflow(base, rowValue, result)
+      && !llvm::AddOverflow(result, columnValue, result);
+}
+
+} // namespace
+
+FailureOr<StaticIntGrid> StaticIntGrid::fromSequences(
+    ArrayRef<StaticIntSequence> input, bool columnsInput,
+    int64_t sparseBase) {
+  if (input.empty() || !input.front().size())
+    return failure();
+  size_t rowCount = columnsInput ? input.front().size() : input.size();
+  size_t columnCount = columnsInput ? input.size() : input.front().size();
+  auto cells = cellCount(rowCount, columnCount);
+  if (!cells ||
+      llvm::any_of(input, [&](const StaticIntSequence &sequence) {
+        return sequence.size() != input.front().size();
+      }))
+    return failure();
+  StaticIntGrid result(rowCount, columnCount, input.front().valueAt(0));
+  if (llvm::all_equal(input)) {
+    if (input.front().getKind() == StaticIntSequenceKind::Uniform)
+      return result;
+    result.kind = columnsInput ? Kind::ActionOnly : Kind::LaneOnly;
+    result.values = input.front();
+    return result;
+  }
+  SmallVector<int64_t> outerBases;
+  for (const StaticIntSequence &sequence : input)
+    outerBases.push_back(sequence.valueAt(0));
+  if (llvm::all_of(input, [](const StaticIntSequence &sequence) {
+        return sequence.getKind() == StaticIntSequenceKind::Uniform;
+      })) {
+    result.kind = columnsInput ? Kind::LaneOnly : Kind::ActionOnly;
+    result.values = StaticIntSequence::fromValues(outerBases);
+    return result;
+  }
+  auto innerStep = input.front().getAffineStep();
+  StaticIntSequence bases = StaticIntSequence::fromValues(outerBases);
+  auto outerStep = bases.getAffineStep();
+  if (innerStep && outerStep &&
+      llvm::all_of(input, [&](const StaticIntSequence &sequence) {
+        return sequence.getAffineStep() == innerStep;
+      }))
+    return affine2D(result.base,
+      columnsInput ? *innerStep : *outerStep,
+      columnsInput ? *outerStep : *innerStep, rowCount, columnCount);
+  SmallVector<int64_t> values;
+  values.reserve(*cells);
+  for (size_t row = 0; row < rowCount; ++row)
+    for (size_t column = 0; column < columnCount; ++column)
+      values.push_back(columnsInput ? input[column].valueAt(row)
+                                    : input[row].valueAt(column));
+  result.values = StaticIntSequence::fromValues(values);
+  for (size_t index = 0; index < *cells; ++index)
+    if (values[index] != sparseBase)
+      result.overrideKeys.push_back(static_cast<int64_t>(index));
+  if (result.overrideKeys.size() <= *cells / 4) {
+    result.kind = Kind::SparseLaneOverrides;
+    result.base = sparseBase;
+  } else {
+    result.kind = Kind::Dense;
+    result.overrideKeys.clear();
+  }
+  return result;
+}
+
+FailureOr<StaticIntGrid> StaticIntGrid::fromRows(
+    ArrayRef<StaticIntSequence> rows) {
+  if (rows.empty() || !rows.front().size())
+    return failure();
+  return fromSequences(rows, false, rows.front().valueAt(0));
+}
+
+FailureOr<StaticIntGrid> StaticIntGrid::fromColumns(
+    size_t rowCount, ArrayRef<StaticIntSequence> columnSequences,
+    int64_t defaultValue) {
+  if (!cellCount(rowCount, columnSequences.size()))
+    return failure();
+  SmallVector<StaticIntSequence> padded;
+  padded.reserve(columnSequences.size());
+  for (const StaticIntSequence &sequence : columnSequences) {
+    if (sequence.size() > rowCount)
+      return failure();
+    if (sequence.size() == rowCount) {
+      padded.push_back(sequence);
+      continue;
+    }
+    SmallVector<int64_t> values(rowCount, defaultValue);
+    for (size_t row = 0; row < sequence.size(); ++row)
+      values[row] = sequence.valueAt(row);
+    padded.push_back(StaticIntSequence::fromValues(values));
+  }
+  return fromSequences(padded, true, defaultValue);
+}
+
+FailureOr<StaticIntGrid> StaticIntGrid::affine2D(
+    int64_t base, int64_t rowStep, int64_t columnStep,
+    size_t rows, size_t columns) {
+  int64_t last;
+  if (!cellCount(rows, columns) ||
+      !affineValue(base, rowStep, columnStep, rows - 1, columns - 1, last))
+    return failure();
+  StaticIntGrid result(rows, columns, base);
+  result.kind = rowStep || columnStep ? Kind::Affine : Kind::Uniform;
+  result.rowStep = rowStep;
+  result.columnStep = columnStep;
+  return result;
+}
+
+FailureOr<StaticIntGrid> StaticIntGrid::laneIntervals(
+    size_t columns, ArrayRef<std::pair<size_t, size_t>> intervals,
+    int64_t insideValue, int64_t outsideValue) {
+  if (!columns)
+    return failure();
+  SmallVector<int64_t> values(columns, outsideValue);
+  for (auto [begin, end] : intervals) {
+    if (begin > end || end > columns)
+      return failure();
+    std::fill(values.begin() + begin, values.begin() + end, insideValue);
+  }
+  StaticIntSequence row = StaticIntSequence::fromValues(values);
+  return fromRows(ArrayRef<StaticIntSequence>(row));
+}
+
+int64_t StaticIntGrid::valueAt(size_t row, size_t column) const {
+  assert(row < rows && column < columns);
+  if (kind == Kind::Uniform)
+    return base;
+  if (kind == Kind::ActionOnly)
+    return values->valueAt(row);
+  if (kind == Kind::LaneOnly)
+    return values->valueAt(column);
+  if (kind == Kind::Affine) {
+    int64_t result;
+    bool valid = affineValue(base, rowStep, columnStep, row, column, result);
+    assert(valid);
+    return result;
+  }
+  size_t flat;
+  bool valid = checkedFlatIndex(row, columns, column, flat);
+  assert(valid);
+  if (kind == Kind::SparseLaneOverrides) {
+    auto found = llvm::lower_bound(overrideKeys, static_cast<int64_t>(flat));
+    if (found == overrideKeys.end() || *found != static_cast<int64_t>(flat))
+      return base;
+  }
+  assert(kind == Kind::Dense || kind == Kind::SparseLaneOverrides);
+  return values->valueAt(flat);
+}
+
+Value StaticIntGrid::emitLookup(Value row, Value column,
+                                Operation *constantAnchor,
+                                ConstantPool &constants, OpBuilder &builder,
+                                Location loc) const {
+  if (kind == Kind::Uniform)
+    return constants.getIndex(base);
+  if (kind == Kind::ActionOnly || kind == Kind::LaneOnly)
+    return emitStaticIntLookup(
+      *values, kind == Kind::ActionOnly ? row : column,
+      constantAnchor, constants, builder, loc);
+  Value flat = affineMulConst(builder, loc, row, columns, constantAnchor);
+  flat = arith::AddIOp::create(builder, loc, flat, column);
+  if (kind == Kind::Affine) {
+    Value rowValue = affineMulConst(
+      builder, loc, row, rowStep, constantAnchor);
+    Value columnValue = affineMulConst(
+      builder, loc, column, columnStep, constantAnchor);
+    Value result = arith::AddIOp::create(builder, loc, rowValue, columnValue);
+    return affineAddConst(builder, loc, result, base, constantAnchor);
+  }
+  return emitStaticIntLookup(
+    *values, flat, constantAnchor, constants, builder, loc);
+}
+
+OpFoldResult StaticIntGrid::emitFoldedLookup(
+    Value row, Value column, Operation *constantAnchor,
+    ConstantPool &constants, OpBuilder &builder, Location loc) const {
+  return kind == Kind::Uniform
+       ? OpFoldResult(builder.getIndexAttr(base))
+       : OpFoldResult(emitLookup(
+           row, column, constantAnchor, constants, builder, loc));
+}
+
+} // namespace onnx_mlir

src/PIM/Common/IR/StaticIntGrid.hpp

@@ -0,0 +1,61 @@
+#pragma once
+
+#include "StaticIntSequence.hpp"
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+
+#include <utility>
+
+namespace onnx_mlir {
+
+class ConstantPool;
+
+class StaticIntGrid {
+public:
+  static mlir::FailureOr<StaticIntGrid> fromColumns(
+    size_t rows, llvm::ArrayRef<StaticIntSequence> columns,
+    int64_t defaultValue);
+  static mlir::FailureOr<StaticIntGrid> fromRows(
+    llvm::ArrayRef<StaticIntSequence> rows);
+  static mlir::FailureOr<StaticIntGrid> affine2D(
+    int64_t base, int64_t rowStep, int64_t columnStep,
+    size_t rows, size_t columns);
+  static mlir::FailureOr<StaticIntGrid> laneIntervals(
+    size_t columns,
+    llvm::ArrayRef<std::pair<size_t, size_t>> intervals,
+    int64_t insideValue, int64_t outsideValue);
+
+  int64_t valueAt(size_t row, size_t column) const;
+
+  mlir::Value emitLookup(mlir::Value row, mlir::Value column,
+                         mlir::Operation *constantAnchor,
+                         ConstantPool &constants, mlir::OpBuilder &builder,
+                         mlir::Location loc) const;
+  mlir::OpFoldResult emitFoldedLookup(
+    mlir::Value row, mlir::Value column, mlir::Operation *constantAnchor,
+    ConstantPool &constants, mlir::OpBuilder &builder,
+    mlir::Location loc) const;
+
+private:
+  enum class Kind { Uniform, ActionOnly, LaneOnly, Affine,
+                    SparseLaneOverrides, Dense };
+
+  StaticIntGrid(size_t rows, size_t columns, int64_t base)
+      : rows(rows), columns(columns), base(base) {}
+
+  static mlir::FailureOr<StaticIntGrid> fromSequences(
+    llvm::ArrayRef<StaticIntSequence> sequences, bool columns,
+    int64_t sparseBase);
+
+  Kind kind = Kind::Uniform;
+  size_t rows = 0;
+  size_t columns = 0;
+  int64_t base = 0;
+  int64_t rowStep = 0;
+  int64_t columnStep = 0;
+  llvm::SmallVector<int64_t> overrideKeys;
+  std::optional<StaticIntSequence> values;
+};
+
+} // namespace onnx_mlir

src/PIM/Common/IR/StaticIntSequence.cpp

@@ -0,0 +1,539 @@
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
+
+#include <limits>
+
+#include "AffineUtils.hpp"
+#include "ConstantUtils.hpp"
+#include "StaticIntSequence.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static bool getAffineValue(int64_t base, int64_t step, size_t index,
+                           int64_t &value) {
+  if (index > static_cast<size_t>(std::numeric_limits<int64_t>::max()))
+    return false;
+  int64_t scaled;
+  return !llvm::MulOverflow(static_cast<int64_t>(index), step, scaled)
+      && !llvm::AddOverflow(base, scaled, value);
+}
+
+static FailureOr<SmallVector<int64_t>> getI64Values(Operation *op,
+                                                     StringRef name) {
+  Attribute attr = op->getAttr(name);
+  if (!attr)
+    return op->emitOpError() << "is missing " << name << " metadata",
+           failure();
+  if (auto scalar = dyn_cast<IntegerAttr>(attr))
+    return SmallVector<int64_t> {scalar.getInt()};
+  if (auto array = dyn_cast<DenseI64ArrayAttr>(attr))
+    return SmallVector<int64_t>(array.asArrayRef());
+  auto elements = dyn_cast<DenseIntElementsAttr>(attr);
+  auto type = elements ? dyn_cast<RankedTensorType>(elements.getType())
+                       : RankedTensorType();
+  if (!elements || !type || type.getRank() != 1
+      || !type.getElementType().isInteger(64))
+    return op->emitOpError() << "has invalid " << name << " metadata",
+           failure();
+  SmallVector<int64_t> values;
+  values.reserve(elements.getNumElements());
+  for (APInt value : elements.getValues<APInt>())
+    values.push_back(value.getSExtValue());
+  return values;
+}
+
+} // namespace
+
+StaticIntSequence StaticIntSequence::uniform(int64_t value, size_t count) {
+  assert(count != 0 && "empty static integer sequence");
+  StaticIntSequence result;
+  result.kind = StaticIntSequenceKind::Uniform;
+  result.count = count;
+  result.base = value;
+  return result;
+}
+
+StaticIntSequence StaticIntSequence::affine(int64_t base, int64_t step,
+                                             size_t count) {
+  assert(count != 0 && "empty static integer sequence");
+  int64_t last;
+  assert(getAffineValue(base, step, count - 1, last)
+         && "overflowing static affine sequence");
+  if (count == 1 || step == 0)
+    return uniform(base, count);
+  StaticIntSequence result;
+  result.kind = StaticIntSequenceKind::Affine;
+  result.count = count;
+  result.base = base;
+  result.step = step;
+  return result;
+}
+
+StaticIntSequence StaticIntSequence::runLengthEncoded(
+  ArrayRef<int64_t> runs, size_t count) {
+  assert(count != 0 && runs.size() % 2 == 0
+         && "invalid run-length encoded sequence");
+  StaticIntSequence result;
+  result.kind = StaticIntSequenceKind::RunLengthEncoded;
+  result.count = count;
+  result.data.assign(runs);
+  return result;
+}
+
+StaticIntSequence StaticIntSequence::fromValues(ArrayRef<int64_t> values) {
+  assert(!values.empty() && "empty static integer sequence");
+  if (llvm::all_equal(values))
+    return uniform(values.front(), values.size());
+  int64_t step;
+  bool isAffine = !llvm::SubOverflow(values[1], values[0], step);
+  for (size_t index = 1; isAffine && index < values.size(); ++index) {
+    int64_t difference;
+    isAffine = !llvm::SubOverflow(values[index], values[index - 1],
+                                  difference)
+        && difference == step;
+  }
+  if (isAffine)
+    return affine(values.front(), step, values.size());
+
+  SmallVector<int64_t> runs;
+  for (int64_t value : values) {
+    if (!runs.empty() && runs[runs.size() - 2] == value) {
+      ++runs.back();
+      continue;
+    }
+    runs.push_back(value);
+    runs.push_back(1);
+  }
+  if (runs.size() < values.size())
+    return runLengthEncoded(runs, values.size());
+  StaticIntSequence result;
+  result.kind = StaticIntSequenceKind::Dense;
+  result.count = values.size();
+  result.data.assign(values);
+  return result;
+}
+
+int64_t StaticIntSequence::valueAt(size_t index) const {
+  assert(index < count && "static integer sequence index out of range");
+  if (kind == StaticIntSequenceKind::Uniform)
+    return base;
+  if (kind == StaticIntSequenceKind::Affine) {
+    int64_t value;
+    bool valid = getAffineValue(base, step, index, value);
+    assert(valid && "overflowing static affine sequence");
+    return value;
+  }
+  if (kind == StaticIntSequenceKind::Dense)
+    return data[index];
+  for (size_t run = 0; run < data.size(); run += 2) {
+    size_t length = static_cast<size_t>(data[run + 1]);
+    if (index < length)
+      return data[run];
+    index -= length;
+  }
+  llvm_unreachable("malformed run-length encoded sequence");
+}
+
+std::optional<size_t> StaticIntSequence::find(int64_t value, size_t begin,
+                                              size_t length) const {
+  assert(begin <= count && length <= count - begin
+         && "invalid static integer sequence search");
+  if (length == 0)
+    return std::nullopt;
+  size_t end = begin + length;
+  if (kind == StaticIntSequenceKind::Uniform)
+    return value == base ? std::optional<size_t>(begin) : std::nullopt;
+  if (kind == StaticIntSequenceKind::Affine) {
+    int64_t delta;
+    if (llvm::SubOverflow(value, base, delta) || delta % step != 0)
+      return std::nullopt;
+    int64_t index = delta / step;
+    return index >= static_cast<int64_t>(begin)
+            && index < static_cast<int64_t>(end)
+        ? std::optional<size_t>(index)
+        : std::nullopt;
+  }
+  if (kind == StaticIntSequenceKind::Dense) {
+    ArrayRef<int64_t> selected = ArrayRef(data).slice(begin, length);
+    auto found = llvm::find(selected, value);
+    return found == selected.end()
+        ? std::nullopt
+        : std::optional<size_t>(begin + (found - selected.begin()));
+  }
+  size_t runBegin = 0;
+  for (size_t run = 0; run < data.size(); run += 2) {
+    size_t runEnd = runBegin + static_cast<size_t>(data[run + 1]);
+    if (runEnd > begin && runBegin < end && data[run] == value)
+      return std::max(begin, runBegin);
+    if (runBegin >= end)
+      break;
+    runBegin = runEnd;
+  }
+  return std::nullopt;
+}
+
+StaticIntSequence StaticIntSequence::slice(size_t begin, size_t length) const {
+  assert(length != 0 && begin <= count - length && "invalid sequence slice");
+  if (kind == StaticIntSequenceKind::Uniform)
+    return uniform(base, length);
+  if (kind == StaticIntSequenceKind::Affine)
+    return affine(valueAt(begin), step, length);
+  if (kind == StaticIntSequenceKind::Dense)
+    return fromValues(ArrayRef(data).slice(begin, length));
+  SmallVector<int64_t> runs;
+  size_t end = begin + length;
+  forEachEqualRun([&](int64_t value, size_t runBegin, size_t runCount) {
+    size_t selectedBegin = std::max(begin, runBegin);
+    size_t selectedEnd = std::min(end, runBegin + runCount);
+    if (selectedBegin >= selectedEnd)
+      return;
+    if (!runs.empty() && runs[runs.size() - 2] == value)
+      runs.back() += selectedEnd - selectedBegin;
+    else {
+      runs.push_back(value);
+      runs.push_back(selectedEnd - selectedBegin);
+    }
+  });
+  if (runs.size() == 2)
+    return uniform(runs.front(), length);
+  if (runs.size() < length)
+    return runLengthEncoded(runs, length);
+  SmallVector<int64_t> values;
+  for (size_t run = 0; run < runs.size(); run += 2)
+    values.append(runs[run + 1], runs[run]);
+  return fromValues(values);
+}
+
+StaticIntSequence StaticIntSequence::remap(ArrayRef<unsigned> indices) const {
+  assert(!indices.empty() && "empty static integer sequence remap");
+  SmallVector<int64_t> values;
+  values.reserve(indices.size());
+  for (unsigned index : indices)
+    values.push_back(valueAt(index));
+  return fromValues(values);
+}
+
+bool StaticIntSequence::operator==(const StaticIntSequence& other) const {
+  return kind == other.kind && count == other.count && base == other.base
+      && step == other.step && data == other.data;
+}
+
+llvm::hash_code StaticIntSequence::hash() const {
+  return llvm::hash_combine(kind, count, base, step,
+                            llvm::hash_combine_range(data.begin(), data.end()));
+}
+
+void StaticIntSequence::forEachEqualRun(
+  llvm::function_ref<void(int64_t, size_t, size_t)> callback) const {
+  if (kind == StaticIntSequenceKind::Uniform) {
+    callback(base, 0, count);
+    return;
+  }
+  if (kind == StaticIntSequenceKind::RunLengthEncoded) {
+    size_t begin = 0;
+    for (size_t run = 0; run < data.size(); run += 2) {
+      size_t runCount = static_cast<size_t>(data[run + 1]);
+      callback(data[run], begin, runCount);
+      begin += runCount;
+    }
+    return;
+  }
+  size_t begin = 0;
+  while (begin < count) {
+    int64_t value = valueAt(begin);
+    size_t end = begin + 1;
+    while (end < count && valueAt(end) == value)
+      ++end;
+    callback(value, begin, end - begin);
+    begin = end;
+  }
+}
+
+void StaticIntSequenceChain::append(const StaticIntSequence &sequence,
+                                    size_t begin, size_t length) {
+  assert(length != 0 && begin <= sequence.size() - length
+         && "invalid static integer sequence chain slice");
+  if (!slices.empty()) {
+    StaticIntSequenceSlice &last = slices.back();
+    if (last.sequence == &sequence && last.begin + last.count == begin) {
+      last.count += length;
+      count += length;
+      return;
+    }
+    auto affinePart = [](const StaticIntSequenceSlice &slice,
+                         int64_t &base, int64_t &step) {
+      base = slice.sequence->valueAt(slice.begin);
+      if (slice.count == 1) {
+        step = 0;
+        return true;
+      }
+      return !llvm::SubOverflow(slice.sequence->valueAt(slice.begin + 1),
+                                base, step)
+          && (slice.sequence->kind == StaticIntSequenceKind::Uniform
+              || slice.sequence->kind == StaticIntSequenceKind::Affine);
+    };
+    StaticIntSequenceSlice next {&sequence, begin, length};
+    int64_t leftBase, leftStep, rightBase, rightStep, expected;
+    if (affinePart(last, leftBase, leftStep)
+        && affinePart(next, rightBase, rightStep)
+        && (last.count == 1 || length == 1 || leftStep == rightStep)) {
+      int64_t step = last.count == 1 ? rightStep : leftStep;
+      if (getAffineValue(leftBase, step, last.count, expected)
+          && expected == rightBase) {
+        owned.push_back(std::make_unique<StaticIntSequence>(
+          StaticIntSequence::affine(leftBase, step, last.count + length)));
+        last = {owned.back().get(), 0, last.count + length};
+        count += length;
+        return;
+      }
+    }
+  }
+  slices.push_back({&sequence, begin, length});
+  count += length;
+}
+
+void StaticIntSequenceChain::append(StaticIntSequence sequence) {
+  size_t length = sequence.size();
+  owned.push_back(std::make_unique<StaticIntSequence>(std::move(sequence)));
+  append(*owned.back(), 0, length);
+}
+
+int64_t StaticIntSequenceChain::valueAt(size_t index) const {
+  assert(index < count && "static integer sequence chain index out of range");
+  for (const StaticIntSequenceSlice &slice : slices) {
+    if (index < slice.count)
+      return slice.sequence->valueAt(slice.begin + index);
+    index -= slice.count;
+  }
+  llvm_unreachable("malformed static integer sequence chain");
+}
+
+void StaticIntSequenceChain::forEachSegment(llvm::function_ref<void(
+  const StaticIntSequence &, size_t, size_t)> callback) const {
+  for (const StaticIntSequenceSlice &slice : slices)
+    callback(*slice.sequence, slice.begin, slice.count);
+}
+
+void StaticIntSequenceChain::forEachEqualRun(
+  llvm::function_ref<void(int64_t, size_t, size_t)> callback) const {
+  std::optional<int64_t> pendingValue;
+  size_t pendingBegin = 0, pendingCount = 0, chainBegin = 0;
+  auto flush = [&] {
+    if (pendingValue)
+      callback(*pendingValue, pendingBegin, pendingCount);
+  };
+  for (const StaticIntSequenceSlice &slice : slices) {
+    size_t sliceEnd = slice.begin + slice.count;
+    slice.sequence->forEachEqualRun(
+      [&](int64_t value, size_t runBegin, size_t runCount) {
+        size_t begin = std::max(slice.begin, runBegin);
+        size_t end = std::min(sliceEnd, runBegin + runCount);
+        if (begin >= end)
+          return;
+        size_t selectedCount = end - begin;
+        size_t globalBegin = chainBegin + begin - slice.begin;
+        if (pendingValue && *pendingValue == value
+            && pendingBegin + pendingCount == globalBegin) {
+          pendingCount += selectedCount;
+          return;
+        }
+        flush();
+        pendingValue = value;
+        pendingBegin = globalBegin;
+        pendingCount = selectedCount;
+      });
+    chainBegin += slice.count;
+  }
+  flush();
+}
+
+StaticIntSequence StaticIntSequenceChain::canonicalize() const {
+  assert(count != 0 && "empty static integer sequence chain");
+  int64_t first = valueAt(0);
+  bool uniform = true;
+  forEachEqualRun([&](int64_t value, size_t, size_t) {
+    uniform &= value == first;
+  });
+  if (uniform)
+    return StaticIntSequence::uniform(first, count);
+
+  int64_t step = 0, previous = first;
+  bool affine = true, haveStep = false;
+  size_t position = 0;
+  forEachSegment([&](const StaticIntSequence &sequence, size_t begin,
+                     size_t length) {
+    if (!affine)
+      return;
+    for (size_t index = 0; index < length; ++index) {
+      int64_t value = sequence.valueAt(begin + index);
+      if (position++ == 0) {
+        previous = value;
+        continue;
+      }
+      if (!haveStep) {
+        affine = !llvm::SubOverflow(value, previous, step);
+        haveStep = true;
+      } else if (haveStep) {
+        int64_t difference;
+        affine = !llvm::SubOverflow(value, previous, difference)
+            && difference == step;
+      }
+      previous = value;
+      if (!affine)
+        break;
+    }
+  });
+  if (affine && haveStep)
+    return StaticIntSequence::affine(first, step, count);
+
+  SmallVector<int64_t> runs;
+  forEachEqualRun([&](int64_t value, size_t, size_t runCount) {
+    runs.push_back(value);
+    runs.push_back(runCount);
+  });
+  if (runs.size() < count)
+    return StaticIntSequence::runLengthEncoded(runs, count);
+  SmallVector<int64_t> values;
+  values.reserve(count);
+  for (size_t run = 0; run < runs.size(); run += 2)
+    values.append(runs[run + 1], runs[run]);
+  return StaticIntSequence::fromValues(values);
+}
+
+int64_t StaticIntSequenceChainCursor::value() const {
+  assert(!done() && "static integer sequence chain cursor is done");
+  const StaticIntSequenceSlice &current = chain.slices[slice];
+  return current.sequence->valueAt(current.begin + offset);
+}
+
+void StaticIntSequenceChainCursor::advance() {
+  assert(!done() && "static integer sequence chain cursor is done");
+  if (++offset != chain.slices[slice].count)
+    return;
+  offset = 0;
+  ++slice;
+}
+
+void setStaticIntSequenceAttr(Operation *op, StringRef name,
+                              const StaticIntSequence &sequence,
+                              size_t logicalCount) {
+  assert(sequence.size() == logicalCount && logicalCount != 0
+         && "invalid static integer metadata count");
+  SmallVector<int64_t> values;
+  StringRef encoding;
+  switch (sequence.kind) {
+  case StaticIntSequenceKind::Uniform:
+    encoding = "uniform";
+    values.push_back(sequence.base);
+    break;
+  case StaticIntSequenceKind::Affine:
+    encoding = "affine";
+    values = {sequence.base, sequence.step};
+    break;
+  case StaticIntSequenceKind::RunLengthEncoded:
+    encoding = "rle";
+    values = sequence.data;
+    break;
+  case StaticIntSequenceKind::Dense:
+    encoding = "dense";
+    values = sequence.data;
+    break;
+  }
+  OpBuilder builder(op);
+  auto type = RankedTensorType::get(
+    {static_cast<int64_t>(values.size())}, builder.getI64Type());
+  op->setAttr(name, DenseIntElementsAttr::get(type, values));
+  if (sequence.kind != StaticIntSequenceKind::Dense)
+    op->setAttr((name + "_encoding").str(), builder.getStringAttr(encoding));
+}
+
+FailureOr<StaticIntSequence> getStaticIntSequenceAttr(
+  Operation *op, StringRef name, size_t logicalCount) {
+  if (logicalCount == 0)
+    return op->emitOpError() << "has zero logical count for " << name,
+           failure();
+  auto values = getI64Values(op, name);
+  if (failed(values))
+    return failure();
+  auto encoding = op->getAttrOfType<StringAttr>((name + "_encoding").str());
+  if (!encoding) {
+    if (values->size() != logicalCount)
+      return op->emitOpError() << "has invalid dense " << name << " count",
+             failure();
+    return StaticIntSequence::fromValues(*values);
+  }
+  if (encoding.getValue() == "uniform") {
+    if (values->size() != 1)
+      return op->emitOpError() << "has invalid uniform " << name,
+             failure();
+    return StaticIntSequence::uniform(values->front(), logicalCount);
+  }
+  if (encoding.getValue() == "affine") {
+    int64_t last;
+    if (values->size() != 2
+        || !getAffineValue((*values)[0], (*values)[1], logicalCount - 1, last))
+      return op->emitOpError() << "has invalid affine " << name,
+             failure();
+    return StaticIntSequence::affine((*values)[0], (*values)[1], logicalCount);
+  }
+  if (encoding.getValue() == "rle") {
+    size_t count = 0;
+    if (values->empty() || values->size() % 2 != 0)
+      return op->emitOpError() << "has invalid RLE " << name, failure();
+    for (size_t index = 1; index < values->size(); index += 2) {
+      if ((*values)[index] <= 0
+          || static_cast<uint64_t>((*values)[index]) > logicalCount - count)
+        return op->emitOpError() << "has invalid RLE " << name, failure();
+      count += (*values)[index];
+    }
+    if (count != logicalCount)
+      return op->emitOpError() << "has mismatched RLE " << name << " count",
+             failure();
+    return StaticIntSequence::runLengthEncoded(*values, count);
+  }
+  if (encoding.getValue() == "dense") {
+    if (values->size() != logicalCount)
+      return op->emitOpError() << "has invalid dense " << name << " count",
+             failure();
+    return StaticIntSequence::fromValues(*values);
+  }
+  return op->emitOpError() << "has unknown " << name << " encoding",
+         failure();
+}
+
+Value emitStaticIntLookup(const StaticIntSequence& sequence, Value position,
+                          Operation* constantAnchor,
+                          ConstantPool& constants, OpBuilder& builder,
+                          Location loc) {
+  if (sequence.getKind() == StaticIntSequenceKind::Uniform)
+    return constants.getIndex(sequence.valueAt(0));
+  if (sequence.getKind() == StaticIntSequenceKind::Affine) {
+    Value scaled = affineMulConst(builder, loc, position,
+                                  sequence.valueAt(1) - sequence.valueAt(0),
+                                  constantAnchor);
+    return affineAddConst(builder, loc, scaled, sequence.valueAt(0),
+                          constantAnchor);
+  }
+  SmallVector<int64_t> values;
+  values.reserve(sequence.size());
+  sequence.forEachEqualRun([&](int64_t value, size_t, size_t count) {
+    values.append(count, value);
+  });
+  auto type = RankedTensorType::get(
+    {static_cast<int64_t>(values.size())}, builder.getI64Type());
+  Value table = constants.get(type,
+    DenseElementsAttr::get(type, ArrayRef<int64_t>(values)));
+  Value selected = tensor::ExtractOp::create(
+    builder, loc, table, ValueRange {position});
+  return arith::IndexCastOp::create(
+    builder, loc, builder.getIndexType(), selected);
+}
+
+} // namespace onnx_mlir

src/PIM/Common/IR/StaticIntSequence.hpp

@@ -0,0 +1,123 @@
+#pragma once
+
+#include "mlir/IR/Builders.h"
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/FunctionExtras.h"
+#include "llvm/ADT/Hashing.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <memory>
+#include <optional>
+
+namespace onnx_mlir {
+
+class ConstantPool;
+
+enum class StaticIntSequenceKind {
+  Uniform,
+  Affine,
+  RunLengthEncoded,
+  Dense
+};
+
+class StaticIntSequence {
+public:
+  static StaticIntSequence fromValues(llvm::ArrayRef<int64_t> values);
+  static StaticIntSequence uniform(int64_t value, size_t count);
+  static StaticIntSequence affine(int64_t base, int64_t step, size_t count);
+
+  size_t size() const { return count; }
+  int64_t valueAt(size_t index) const;
+  std::optional<size_t> find(int64_t value, size_t begin, size_t length) const;
+  StaticIntSequence slice(size_t begin, size_t count) const;
+  StaticIntSequence remap(llvm::ArrayRef<unsigned> indices) const;
+  StaticIntSequenceKind getKind() const { return kind; }
+  std::optional<int64_t> getAffineStep() const {
+    if (kind == StaticIntSequenceKind::Uniform)
+      return 0;
+    return kind == StaticIntSequenceKind::Affine
+         ? std::optional<int64_t>(step) : std::nullopt;
+  }
+
+  bool operator==(const StaticIntSequence& other) const;
+  llvm::hash_code hash() const;
+
+  void forEachEqualRun(
+    llvm::function_ref<void(int64_t, size_t, size_t)> callback) const;
+
+private:
+  friend class StaticIntSequenceChain;
+  friend void setStaticIntSequenceAttr(mlir::Operation *, llvm::StringRef,
+                                       const StaticIntSequence &, size_t);
+  friend mlir::FailureOr<StaticIntSequence>
+  getStaticIntSequenceAttr(mlir::Operation *, llvm::StringRef, size_t);
+
+  static StaticIntSequence runLengthEncoded(
+    llvm::ArrayRef<int64_t> runs, size_t count);
+  StaticIntSequenceKind kind = StaticIntSequenceKind::Dense;
+  size_t count = 0;
+  int64_t base = 0;
+  int64_t step = 0;
+  llvm::SmallVector<int64_t> data;
+};
+
+struct StaticIntSequenceSlice {
+  const StaticIntSequence *sequence = nullptr;
+  size_t begin = 0;
+  size_t count = 0;
+};
+
+class StaticIntSequenceChain {
+public:
+  void append(const StaticIntSequence &sequence, size_t begin, size_t count);
+  void append(StaticIntSequence sequence);
+  size_t size() const { return count; }
+  int64_t valueAt(size_t index) const;
+  void forEachSegment(llvm::function_ref<void(
+    const StaticIntSequence &, size_t, size_t)> callback) const;
+  void forEachEqualRun(
+    llvm::function_ref<void(int64_t, size_t, size_t)> callback) const;
+  StaticIntSequence canonicalize() const;
+
+private:
+  friend class StaticIntSequenceChainCursor;
+  llvm::SmallVector<StaticIntSequenceSlice> slices;
+  llvm::SmallVector<std::unique_ptr<StaticIntSequence>> owned;
+  size_t count = 0;
+};
+
+class StaticIntSequenceChainCursor {
+public:
+  explicit StaticIntSequenceChainCursor(const StaticIntSequenceChain &chain)
+    : chain(chain) {}
+
+  bool done() const { return slice == chain.slices.size(); }
+  int64_t value() const;
+  void advance();
+
+private:
+  const StaticIntSequenceChain &chain;
+  size_t slice = 0;
+  size_t offset = 0;
+};
+
+void setStaticIntSequenceAttr(mlir::Operation *op, llvm::StringRef name,
+                              const StaticIntSequence &sequence,
+                              size_t logicalCount);
+
+mlir::FailureOr<StaticIntSequence>
+getStaticIntSequenceAttr(mlir::Operation *op, llvm::StringRef name,
+                         size_t logicalCount);
+
+mlir::Value emitStaticIntLookup(const StaticIntSequence& sequence,
+                                mlir::Value position,
+                                mlir::Operation* constantAnchor,
+                                ConstantPool& constants,
+                                mlir::OpBuilder& builder,
+                                mlir::Location loc);
+
+} // namespace onnx_mlir

src/PIM/Common/IR/TensorSliceUtils.cpp

@@ -0,0 +1,133 @@
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+
+#include "src/Accelerators/PIM/Common/IR/TensorSliceUtils.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+
+Value extractAxisSlice(
+  PatternRewriter& rewriter, Location loc, Value source, int64_t axis, int64_t offset, int64_t size) {
+  auto sourceType = cast<RankedTensorType>(source.getType());
+  SmallVector<int64_t> resultShape(sourceType.getShape());
+  resultShape[axis] = size;
+  auto resultType = RankedTensorType::get(resultShape, sourceType.getElementType(), sourceType.getEncoding());
+
+  SmallVector<OpFoldResult> offsets = getZeroOffsets(rewriter, sourceType.getRank());
+  SmallVector<OpFoldResult> sizes = getStaticSizes(rewriter, sourceType.getShape());
+  offsets[axis] = rewriter.getIndexAttr(offset);
+  sizes[axis] = rewriter.getIndexAttr(size);
+  return tensor::ExtractSliceOp::create(
+           rewriter, loc, resultType, source, offsets, sizes, getUnitStrides(rewriter, sourceType.getRank()))
+    .getResult();
+}
+
+Value extractStaticSliceOrIdentity(OpBuilder& rewriter,
+                                   Location loc,
+                                   Value source,
+                                   RankedTensorType resultType,
+                                   ArrayRef<OpFoldResult> offsets,
+                                   ArrayRef<OpFoldResult> sizes,
+                                   ArrayRef<OpFoldResult> strides) {
+  auto sourceType = cast<RankedTensorType>(source.getType());
+  size_t rank = static_cast<size_t>(sourceType.getRank());
+
+  bool isIdentitySlice =
+    sourceType == resultType && sourceType.hasStaticShape() && offsets.size() == rank && sizes.size() == rank
+    && strides.size() == rank;
+  if (isIdentitySlice) {
+    ArrayRef<int64_t> sourceShape = sourceType.getShape();
+    for (auto [dim, offset, size, stride] : llvm::zip_equal(sourceShape, offsets, sizes, strides)) {
+      std::optional<int64_t> staticOffset = mlir::getConstantIntValue(offset);
+      std::optional<int64_t> staticSize = mlir::getConstantIntValue(size);
+      std::optional<int64_t> staticStride = mlir::getConstantIntValue(stride);
+      if (!staticOffset || !staticSize || !staticStride || *staticOffset != 0 || *staticSize != dim
+          || *staticStride != 1) {
+        isIdentitySlice = false;
+        break;
+      }
+    }
+  }
+
+  if (isIdentitySlice)
+    return source;
+
+  return rewriter.createOrFold<tensor::ExtractSliceOp>(
+    loc, resultType, source, offsets, sizes, strides);
+}
+
+Value insertStaticSlice(
+  PatternRewriter& rewriter, Location loc, Value source, Value dest, ArrayRef<OpFoldResult> offsets) {
+  auto sourceType = cast<RankedTensorType>(source.getType());
+  return tensor::InsertSliceOp::create(rewriter,
+                                       loc,
+                                       source,
+                                       dest,
+                                       offsets,
+                                       getStaticSizes(rewriter, sourceType.getShape()),
+                                       getUnitStrides(rewriter, sourceType.getRank()))
+    .getResult();
+}
+
+Value extractMixedSliceOrIdentity(OpBuilder &rewriter,
+                                  Location loc,
+                                  Value source,
+                                  RankedTensorType resultType,
+                                  const MixedSliceGeometry &geometry) {
+  return extractStaticSliceOrIdentity(rewriter, loc, source, resultType,
+                                      geometry.offsets, geometry.sizes,
+                                      geometry.strides);
+}
+
+Value insertMixedSlice(OpBuilder &builder, Location loc, Value source,
+                       Value dest, const MixedSliceGeometry &geometry) {
+  SmallVector<OpFoldResult> sizes(geometry.sizes);
+  auto sourceType = dyn_cast<RankedTensorType>(source.getType());
+  auto destType = dyn_cast<RankedTensorType>(dest.getType());
+  if (sourceType && destType && sourceType.hasStaticShape()
+      && sourceType.getRank() == destType.getRank()) {
+    sizes.clear();
+    for (int64_t dimension : sourceType.getShape())
+      sizes.push_back(builder.getIndexAttr(dimension));
+  }
+  return tensor::InsertSliceOp::create(builder, loc, source, dest,
+                                       geometry.offsets, sizes,
+                                       geometry.strides);
+}
+
+FailureOr<Value> addLeadingUnitTensorDimension(OpBuilder& builder, Location loc, Value value) {
+  auto type = dyn_cast<RankedTensorType>(value.getType());
+  if (!type || !type.hasStaticShape())
+    return failure();
+  SmallVector<int64_t> shape {1};
+  llvm::append_range(shape, type.getShape());
+  auto resultType = RankedTensorType::get(shape, type.getElementType(), type.getEncoding());
+  SmallVector<ReassociationIndices> reassociation;
+  if (type.getRank() != 0) {
+    reassociation.push_back({0, 1});
+    for (int64_t dim = 1; dim < type.getRank(); ++dim)
+      reassociation.push_back({dim + 1});
+  }
+  return tensor::ExpandShapeOp::create(builder, loc, resultType, value, reassociation).getResult();
+}
+
+FailureOr<Value> removeLeadingUnitTensorDimension(
+  OpBuilder& builder, Location loc, Value value, RankedTensorType resultType) {
+  if (value.getType() == resultType)
+    return value;
+  auto type = dyn_cast<RankedTensorType>(value.getType());
+  if (!type || !resultType || !type.hasStaticShape() || !resultType.hasStaticShape()
+      || type.getRank() != resultType.getRank() + 1 || type.getDimSize(0) != 1
+      || type.getElementType() != resultType.getElementType()
+      || !llvm::equal(type.getShape().drop_front(), resultType.getShape()))
+    return failure();
+  SmallVector<ReassociationIndices> reassociation;
+  if (resultType.getRank() != 0) {
+    reassociation.push_back({0, 1});
+    for (int64_t dim = 1; dim < resultType.getRank(); ++dim)
+      reassociation.push_back({dim + 1});
+  }
+  return tensor::CollapseShapeOp::create(builder, loc, resultType, value, reassociation).getResult();
+}
+
+} // namespace onnx_mlir

src/PIM/Common/IR/TensorSliceUtils.hpp

@@ -0,0 +1,52 @@
+#pragma once
+
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/IR/ValueRange.h"
+#include "mlir/Transforms/DialectConversion.h"
+
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+
+namespace onnx_mlir {
+
+struct MixedSliceGeometry {
+  llvm::SmallVector<mlir::OpFoldResult> offsets;
+  llvm::SmallVector<mlir::OpFoldResult> sizes;
+  llvm::SmallVector<mlir::OpFoldResult> strides;
+};
+
+mlir::Value extractAxisSlice(
+  mlir::PatternRewriter& rewriter, mlir::Location loc, mlir::Value source, int64_t axis, int64_t offset, int64_t size);
+
+mlir::Value extractStaticSliceOrIdentity(mlir::OpBuilder& rewriter,
+                                         mlir::Location loc,
+                                         mlir::Value source,
+                                         mlir::RankedTensorType resultType,
+                                         llvm::ArrayRef<mlir::OpFoldResult> offsets,
+                                         llvm::ArrayRef<mlir::OpFoldResult> sizes,
+                                         llvm::ArrayRef<mlir::OpFoldResult> strides);
+
+mlir::Value insertStaticSlice(mlir::PatternRewriter& rewriter,
+                              mlir::Location loc,
+                              mlir::Value source,
+                              mlir::Value dest,
+                              llvm::ArrayRef<mlir::OpFoldResult> offsets);
+
+mlir::Value extractMixedSliceOrIdentity(mlir::OpBuilder &rewriter,
+                                        mlir::Location loc,
+                                        mlir::Value source,
+                                        mlir::RankedTensorType resultType,
+                                        const MixedSliceGeometry &geometry);
+
+mlir::Value insertMixedSlice(mlir::OpBuilder &builder,
+                             mlir::Location loc,
+                             mlir::Value source,
+                             mlir::Value dest,
+                             const MixedSliceGeometry &geometry);
+
+mlir::FailureOr<mlir::Value>
+addLeadingUnitTensorDimension(mlir::OpBuilder& builder, mlir::Location loc, mlir::Value value);
+
+mlir::FailureOr<mlir::Value> removeLeadingUnitTensorDimension(
+  mlir::OpBuilder& builder, mlir::Location loc, mlir::Value value, mlir::RankedTensorType resultType);
+
+} // namespace onnx_mlir

src/PIM/Common/IR/WeightUtils.cpp

@@ -146,17 +146,17 @@ void walkPimMvmVmmWeightUses(mlir::Operation* root, llvm::function_ref<void(mlir
   });
 }
 
-std::optional<unsigned> resolveWeightIndex(mlir::Operation* weightOwner, mlir::Value weight) {
+std::optional<unsigned> resolveWeightIndex(mlir::Operation* coreLikeOp, mlir::Value weight) {
   weight = stripMemRefAddressingOps(weight);
 
-  if (auto coreOp = mlir::dyn_cast_or_null<pim::PimCoreOp>(weightOwner)) {
+  if (auto coreOp = mlir::dyn_cast_or_null<pim::PimCoreOp>(coreLikeOp)) {
     for (unsigned weightIndex = 0; weightIndex < coreOp.getWeights().size(); ++weightIndex)
       if (coreOp.getWeightArgument(weightIndex) == weight)
         return weightIndex;
     return std::nullopt;
   }
 
-  if (auto coreBatchOp = mlir::dyn_cast_or_null<pim::PimCoreBatchOp>(weightOwner)) {
+  if (auto coreBatchOp = mlir::dyn_cast_or_null<pim::PimCoreBatchOp>(coreLikeOp)) {
     for (unsigned weightIndex = 0; weightIndex < coreBatchOp.getWeights().size(); ++weightIndex)
       if (coreBatchOp.getWeightArgument(weightIndex) == weight)
         return weightIndex;
@@ -167,7 +167,7 @@ std::optional<unsigned> resolveWeightIndex(mlir::Operation* weightOwner, mlir::V
 }
 
 llvm::FailureOr<ResolvedWeightView>
-resolveWeightView(mlir::Operation* weightOwner, mlir::Value weight, const StaticValueKnowledge& knowledge) {
+resolveWeightView(mlir::Operation* coreLikeOp, mlir::Value weight, const StaticValueKnowledge& knowledge) {
   llvm::SmallVector<mlir::Operation*> viewOps;
   mlir::Value current = weight;
 
@@ -179,7 +179,7 @@ resolveWeightView(mlir::Operation* weightOwner, mlir::Value weight, const Static
 
     if (auto defOp = current.getDefiningOp()) {
       if (auto getGlobalOp = mlir::dyn_cast<mlir::memref::GetGlobalOp>(defOp)) {
-        auto moduleOp = weightOwner ? weightOwner->getParentOfType<mlir::ModuleOp>() : mlir::ModuleOp {};
+        auto moduleOp = coreLikeOp ? coreLikeOp->getParentOfType<mlir::ModuleOp>() : mlir::ModuleOp {};
         auto globalOp = lookupGlobalForGetGlobal(moduleOp, getGlobalOp);
         if (!globalOp || !globalOp.getInitialValue())
           return mlir::failure();
@@ -297,15 +297,15 @@ resolveWeightView(mlir::Operation* weightOwner, mlir::Value weight, const Static
       continue;
     }
 
-    auto weightIndex = resolveWeightIndex(weightOwner, current);
+    auto weightIndex = resolveWeightIndex(coreLikeOp, current);
     if (!weightIndex)
       return mlir::failure();
 
-    if (auto coreOp = mlir::dyn_cast_or_null<pim::PimCoreOp>(weightOwner)) {
+    if (auto coreOp = mlir::dyn_cast_or_null<pim::PimCoreOp>(coreLikeOp)) {
       current = coreOp.getWeights()[*weightIndex];
       continue;
     }
-    if (auto coreBatchOp = mlir::dyn_cast_or_null<pim::PimCoreBatchOp>(weightOwner)) {
+    if (auto coreBatchOp = mlir::dyn_cast_or_null<pim::PimCoreBatchOp>(coreLikeOp)) {
       current = coreBatchOp.getWeights()[*weightIndex];
       continue;
     }

src/PIM/Common/IR/WeightUtils.hpp

@@ -45,9 +45,9 @@ bool hasOnlySpatialMvmVmmWeightUses(mlir::Value value);
 /// passes can identify globals that must remain weight-backed.
 void walkPimMvmVmmWeightUses(mlir::Operation* root, llvm::function_ref<void(mlir::OpOperand&)> callback);
 
-std::optional<unsigned> resolveWeightIndex(mlir::Operation* weightOwner, mlir::Value weight);
+std::optional<unsigned> resolveWeightIndex(mlir::Operation* coreLikeOp, mlir::Value weight);
 llvm::FailureOr<ResolvedWeightView>
-resolveWeightView(mlir::Operation* weightOwner, mlir::Value weight, const StaticValueKnowledge& knowledge = {});
+resolveWeightView(mlir::Operation* coreLikeOp, mlir::Value weight, const StaticValueKnowledge& knowledge = {});
 
 template <typename CoreLikeOpTy>
 llvm::SmallVector<unsigned, 8> getUsedWeightIndices(CoreLikeOpTy coreLikeOp) {

src/PIM/Common/LabeledList.hpp

@@ -1,315 +0,0 @@
-#pragma once
-
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/ilist_node.h"
-#include "llvm/ADT/simple_ilist.h"
-
-#include <cassert>
-#include <iterator>
-#include <limits>
-#include <type_traits>
-
-namespace onnx_mlir {
-
-template <typename NodeT>
-class LabeledList;
-
-template <typename NodeT>
-class LabeledListNode : public llvm::ilist_node<NodeT> {
-  friend class LabeledList<NodeT>;
-
-public:
-  using Label = uint64_t;
-
-  LabeledListNode() = default;
-  LabeledListNode(const LabeledListNode&) = delete;
-  LabeledListNode(LabeledListNode&&) = default;
-  LabeledListNode& operator=(LabeledListNode&&) = delete;
-
-  ~LabeledListNode() { assert(owner_ == nullptr && "destroying a linked LabeledListNode"); }
-
-  bool isLinked() const { return owner_ != nullptr; }
-  Label getOrderLabel() const { return label; }
-
-  friend bool operator<(const LabeledListNode& lft, const LabeledListNode& rgt) { return lft.label < rgt.label; }
-
-private:
-  const void* owner_ = nullptr;
-  Label label = 0;
-};
-
-template <typename NodeT>
-class LabeledList {
-
-  using Label = typename NodeT::Label;
-
-  static constexpr Label kLowerSentinel = 0;
-  static constexpr Label kUpperSentinel = std::numeric_limits<Label>::max();
-  static constexpr Label kRelabelGap = 2;
-
-public:
-  using List = llvm::simple_ilist<NodeT>;
-  using Iterator = typename List::iterator;
-  using RIterator = typename List::reverse_iterator;
-  using ConstIterator = typename List::const_iterator;
-
-  LabeledList() = default;
-  LabeledList(const LabeledList&) = delete;
-  LabeledList& operator=(const LabeledList&) = delete;
-  LabeledList(LabeledList&&) = delete;
-  LabeledList& operator=(LabeledList&&) = delete;
-
-  ~LabeledList() { clear(); }
-
-  bool empty() const { return size_ == 0; }
-  size_t size() const { return size_; }
-
-  NodeT* front() { return empty() ? nullptr : &nodes_.front(); }
-  const NodeT* front() const { return empty() ? nullptr : &nodes_.front(); }
-
-  NodeT* back() { return empty() ? nullptr : &nodes_.back(); }
-  const NodeT* back() const { return empty() ? nullptr : &nodes_.back(); }
-
-  static NodeT* previous(NodeT* node) {
-    if (!node || !owner(node))
-      return nullptr;
-    auto* list = owner(node);
-    auto it = node->getIterator();
-    if (it == list->nodes_.begin())
-      return nullptr;
-    return &*std::prev(it);
-  }
-
-  static const NodeT* previous(const NodeT* node) {
-    if (!node || !owner(node))
-      return nullptr;
-    const auto* list = owner(node);
-    auto it = const_cast<NodeT*>(node)->getIterator();
-    if (it == list->nodes_.begin())
-      return nullptr;
-    return &*std::prev(it);
-  }
-
-  static NodeT* next(NodeT* node) {
-    if (!node || !owner(node))
-      return nullptr;
-    auto* list = owner(node);
-    auto it = std::next(node->getIterator());
-    if (it == list->nodes_.end())
-      return nullptr;
-    return &*it;
-  }
-
-  static const NodeT* next(const NodeT* node) {
-    if (!node || !owner(node))
-      return nullptr;
-    const auto* list = owner(node);
-    auto it = std::next(const_cast<NodeT*>(node)->getIterator());
-    if (it == list->nodes_.end())
-      return nullptr;
-    return &*it;
-  }
-
-  bool contains(const NodeT* node) const { return node && node->owner_ == this; }
-
-  Label getOrderLabel(const NodeT* node) const {
-    assert(contains(node) && "node must belong to this list");
-    return node->label;
-  }
-
-  bool comesBefore(const NodeT* lhs, const NodeT* rhs) const {
-    assert(contains(lhs) && contains(rhs) && "nodes must belong to this list");
-    return lhs->label < rhs->label;
-  }
-
-  void pushFront(NodeT* node) { insertBefore(front(), node); }
-
-  void pushBack(NodeT* node) { insertBefore(nullptr, node); }
-
-  void insertBefore(NodeT* nextNode, NodeT* node) {
-    assert(node && "cannot insert a null node");
-    assert(!node->owner_ && "node is already linked");
-    assert(nextNode == nullptr || contains(nextNode));
-
-    Iterator nextIt = nextNode ? getIteratorFor(nextNode) : nodes_.end();
-    nodes_.insert(nextIt, *node);
-    node->owner_ = this;
-    ++size_;
-    assignLabel(getIteratorFor(node));
-  }
-
-  void insertAfter(NodeT* prevNode, NodeT* node) {
-    assert(prevNode == nullptr || contains(prevNode));
-    if (prevNode == nullptr)
-      insertBefore(front(), node);
-    else
-      insertBefore(next(prevNode), node);
-  }
-
-  void remove(NodeT* node) {
-    assert(contains(node) && "node must belong to this list");
-    nodes_.remove(*node);
-    node->owner_ = nullptr;
-    node->label = 0;
-    --size_;
-  }
-
-  void moveBefore(NodeT* node, NodeT* nextNode) {
-    assert(contains(node) && "node must belong to this list");
-    assert(nextNode == nullptr || contains(nextNode));
-
-    Iterator nodeIt = getIteratorFor(node);
-    Iterator nextIt = nextNode ? getIteratorFor(nextNode) : nodes_.end();
-    if (nodeIt == nextIt || std::next(nodeIt) == nextIt)
-      return;
-
-    nodes_.splice(nextIt, nodes_, nodeIt);
-    assignLabel(getIteratorFor(node));
-  }
-
-  void moveAfter(NodeT* node, NodeT* prevNode) {
-    assert(contains(node) && "node must belong to this list");
-    assert(prevNode == nullptr || contains(prevNode));
-
-    Iterator nextIt = prevNode ? std::next(getIteratorFor(prevNode)) : nodes_.begin();
-    if (getIteratorFor(node) == nextIt)
-      return;
-    moveBefore(node, nextIt == nodes_.end() ? nullptr : &*nextIt);
-  }
-
-  void clear() {
-    while (!nodes_.empty()) {
-      NodeT* node = &nodes_.front();
-      node->owner_ = nullptr;
-      node->label = 0;
-      nodes_.remove(*node);
-    }
-    size_ = 0;
-  }
-
-  Iterator begin() { return nodes_.begin(); }
-  Iterator end() { return nodes_.end(); }
-
-  RIterator rbegin() { return nodes_.rbegin(); }
-  RIterator rend() { return nodes_.rend(); }
-
-private:
-  static const LabeledList* owner(const NodeT* node) { return static_cast<const LabeledList*>(node->owner_); }
-  static LabeledList* owner(NodeT* node) { return static_cast<LabeledList*>(const_cast<void*>(node->owner_)); }
-
-  static Label lowerLabel(const NodeT* node) { return node ? node->label : kLowerSentinel; }
-  static Label upperLabel(const NodeT* node) { return node ? node->label : kUpperSentinel; }
-
-  static Label labelGap(Label lower, Label upper) {
-    assert(lower < upper && "labels must be strictly ordered");
-    return upper - lower;
-  }
-
-  static bool hasMidpoint(Label lower, Label upper) { return labelGap(lower, upper) > 1; }
-
-  static bool hasRelabelSlack(Label lower, Label upper, size_t nodeCount) {
-    Label gap = labelGap(lower, upper);
-    return gap / static_cast<Label>(nodeCount + 1) >= kRelabelGap;
-  }
-
-  Iterator getIteratorFor(NodeT* node) { return node->getIterator(); }
-  ConstIterator getiteratorFor(const NodeT* node) const { return node->getIterator(); }
-
-  NodeT* previousNode(Iterator it) {
-    if (it == nodes_.begin())
-      return nullptr;
-    return &*std::prev(it);
-  }
-
-  const NodeT* previousNode(ConstIterator it) const {
-    if (it == nodes_.begin())
-      return nullptr;
-    return &*std::prev(it);
-  }
-
-  NodeT* nextNode(Iterator it) {
-    ++it;
-    if (it == nodes_.end())
-      return nullptr;
-    return &*it;
-  }
-
-  const NodeT* nextNode(ConstIterator it) const {
-    ++it;
-    if (it == nodes_.end())
-      return nullptr;
-    return &*it;
-  }
-
-  void assignLabel(Iterator it) {
-    Label lower = lowerLabel(previousNode(it));
-    Label upper = upperLabel(nextNode(it));
-    if (hasMidpoint(lower, upper)) {
-      (*it).label = lower + static_cast<Label>(labelGap(lower, upper) / 2);
-      return;
-    }
-
-    relabelAround(it);
-  }
-
-  void relabelAround(Iterator center) {
-    size_t targetCount = 1;
-    while (true) {
-      Iterator left = center;
-      Iterator right = center;
-      size_t actualCount = 1;
-      expandWindow(center, targetCount, left, right, actualCount);
-
-      Label lower = lowerLabel(previousNode(left));
-      Label upper = upperLabel(nextNode(right));
-      if (hasRelabelSlack(lower, upper, actualCount)) {
-        relabelWindow(left, actualCount, lower, upper);
-        return;
-      }
-
-      if (left == nodes_.begin() && nextNode(right) == nullptr) {
-        assert(hasRelabelSlack(lower, upper, actualCount) && "label space exhausted");
-        relabelWindow(left, actualCount, lower, upper);
-        return;
-      }
-
-      targetCount *= 2;
-    }
-  }
-
-  void expandWindow(Iterator center, size_t targetCount, Iterator& left, Iterator& right, size_t& actualCount) {
-    left = center;
-    right = center;
-    actualCount = 1;
-
-    while (actualCount < targetCount && (left != nodes_.begin() || nextNode(right) != nullptr)) {
-      if (left != nodes_.begin()) {
-        --left;
-        ++actualCount;
-        if (actualCount == targetCount)
-          break;
-      }
-      if (nextNode(right) != nullptr) {
-        ++right;
-        ++actualCount;
-      }
-    }
-  }
-
-  void relabelWindow(Iterator left, size_t nodeCount, Label lower, Label upper) {
-    assert(nodeCount > 0 && "relabel window must not be empty");
-    Label step = labelGap(lower, upper) / static_cast<Label>(nodeCount + 1);
-    assert(step >= 1 && "relabel step must be positive");
-
-    Iterator it = left;
-    for (size_t index = 1; index <= nodeCount; ++index) {
-      (*it).label = lower + step * index;
-      ++it;
-    }
-  }
-
-  List nodes_;
-  size_t size_ = 0;
-};
-
-} // namespace onnx_mlir

src/PIM/Common/PimCommon.hpp

@@ -15,15 +15,29 @@
 #include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/CoreBlockUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/EntryPointUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/IndexingUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/WeightUtils.hpp"
 #include "src/Accelerators/PIM/Common/Support/DebugDump.hpp"
 #include "src/Accelerators/PIM/Common/Support/FileSystemUtils.hpp"
 #include "src/Compiler/CompilerOptions.hpp"
 
+#include <cstdint>
+#include <array>
+#include <limits>
+
 namespace onnx_mlir {
 
 inline constexpr llvm::StringLiteral kCoreIdAttrName = "coreId";
 inline constexpr llvm::StringLiteral kCoreIdsAttrName = "coreIds";
+inline constexpr llvm::StringLiteral kLocalMemoryAddressAttrName = "pim.local_memory_address";
+inline constexpr llvm::StringLiteral kLocalMemorySizeAttrName = "pim.local_memory_size";
+inline constexpr std::array<llvm::StringLiteral, 4> kRemovedLocalMemoryPlanAttrNames = {
+  "pim.local_memory_slot",
+  "pim.local_memory_slot_size",
+  "pim.local_memory_fallback_count",
+  "pim.local_memory_nested_single_use_count",
+};
+inline constexpr size_t kPimLocalMemoryAddressLimit = static_cast<size_t>(std::numeric_limits<int32_t>::max());
 
 } // namespace onnx_mlir

src/PIM/Common/Support/DebugDump.cpp

@@ -7,18 +7,26 @@
 
 namespace onnx_mlir {
 
-void dumpModule(mlir::ModuleOp moduleOp, const std::string& name) {
+std::fstream openDialectDumpFileWithExtension(const std::string& name, llvm::StringRef destination,  llvm::StringRef extension) {
   std::string outputDir = getOutputDir();
   if (outputDir.empty())
+    return {};
+
+  std::string dialectsDir = (outputDir + destination).str();
+  createDirectory(dialectsDir);
+  return std::fstream(dialectsDir + "/" + name + "." + extension.str(), std::ios::out);
+}
+
+void dumpModule(mlir::ModuleOp moduleOp, const std::string& name, bool assumeVerified) {
+  std::fstream file = openDialectDumpFileWithExtension(name, "/dialects", "mlir");
+  if (!file.is_open())
     return;
 
-  std::string dialectsDir = outputDir + "/dialects";
-  createDirectory(dialectsDir);
-
-  std::fstream file(dialectsDir + "/" + name + ".mlir", std::ios::out);
   llvm::raw_os_ostream os(file);
   mlir::OpPrintingFlags flags;
   flags.elideLargeElementsAttrs().enableDebugInfo(true, false);
+  if (assumeVerified)
+    flags.assumeVerified();
   moduleOp.print(os, flags);
   os.flush();
   file.close();

src/PIM/Common/Support/DebugDump.hpp

@@ -1,13 +1,18 @@
 #pragma once
 
 #include "mlir/IR/BuiltinOps.h"
+#include "llvm/ADT/StringRef.h"
 
+#include <fstream>
 #include <string>
 
 namespace onnx_mlir {
 
 /// Emits a MLIR snapshot under the current compiler output
 /// directory for pass-level debugging.
-void dumpModule(mlir::ModuleOp moduleOp, const std::string& name);
+void dumpModule(mlir::ModuleOp moduleOp, const std::string& name, bool assumeVerified = false);
+
+/// Opens a file under the same dialect dump directory used by dumpModule.
+std::fstream openDialectDumpFileWithExtension(const std::string& name,llvm::StringRef destination = "/dialects",  llvm::StringRef extension = "mlir");
 
 } // namespace onnx_mlir

src/PIM/Common/Support/ReportUtils.cpp

@@ -17,6 +17,20 @@ std::fstream openReportFileWithExtension(const std::string& name, llvm::StringRe
 
 std::fstream openReportFile(const std::string& name) { return openReportFileWithExtension(name, "txt"); }
 
+std::fstream openAppendedReportFileWithExtension(const std::string& name, llvm::StringRef extension) {
+  std::string outputDir = getOutputDir();
+  if (outputDir.empty())
+    return {};
+
+  std::string reportsDir = outputDir + "/reports";
+  createDirectory(reportsDir);
+  return std::fstream(reportsDir + "/" + name + "." + extension.str(), std::ios::out | std::ios::app);
+}
+
+std::fstream openAppendedReportFile(const std::string& name) {
+  return openAppendedReportFileWithExtension(name, "txt");
+}
+
 std::string formatReportMemory(uint64_t bytes) {
   const char* units[] = {"B", "KB", "MB", "GB", "TB", "PB", "EB"};
   int i = 0;

src/PIM/Common/Support/ReportUtils.hpp

@@ -12,6 +12,8 @@ namespace onnx_mlir {
 
 std::fstream openReportFile(const std::string& name);
 std::fstream openReportFileWithExtension(const std::string& name, llvm::StringRef extension);
+std::fstream openAppendedReportFile(const std::string& name);
+std::fstream openAppendedReportFileWithExtension(const std::string& name, llvm::StringRef extension);
 std::string formatReportMemory(uint64_t bytes);
 
 struct ReportField {

src/PIM/Compiler/CMakeLists.txt

@@ -17,7 +17,7 @@ add_pim_library(OMPimCompilerUtils
   PimCompilerUtils.cpp
   PimArtifactWriter.cpp
   PimCodeGen.cpp
-  PimMemoryLiveness.cpp
+  PimCoreProgram.cpp
   PimWeightEmitter.cpp
 
   EXCLUDE_FROM_OM_LIBS
@@ -26,11 +26,12 @@ add_pim_library(OMPimCompilerUtils
   ${PIM_COMPILER_INCLUDE_DIRS}
 
   LINK_LIBS PUBLIC
+  MLIRAffineToStandard
   OMPimCompilerOptions
   OMPimCommon
   OMPimBufferization
-  OMPimMemoryCoalescing
   OMPimHostConstantFolding
+  OMPimLocalMemoryPlanning
   OMPimVerification
   OMPimPasses
   OMONNXToSpatial

src/PIM/Compiler/PimArtifactWriter.cpp

@@ -7,7 +7,7 @@
 #include <algorithm>
 #include <cassert>
 #include <cstring>
-#include <vector>
+#include <limits>
 
 #include "src/Accelerators/PIM/Common/IR/WeightUtils.hpp"
 #include "src/Accelerators/PIM/Compiler/PimArtifactWriter.hpp"
@@ -20,6 +20,65 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
+PimInstructionWriter::PimInstructionWriter(raw_pwrite_stream& stream)
+: stream(stream), buffer(kBufferSize) {
+  pim_binary::writeHeader(stream);
+}
+
+PimInstructionWriter::PimInstructionWriter(raw_fd_ostream& stream)
+: PimInstructionWriter(static_cast<raw_pwrite_stream&>(stream)) {
+  fileStream = &stream;
+}
+
+bool PimInstructionWriter::consumeStreamError() {
+  if (!fileStream || !fileStream->has_error())
+    return false;
+  fileStream->clear_error();
+  hasFailure = true;
+  return true;
+}
+
+LogicalResult PimInstructionWriter::flushBuffer() {
+  if (hasFailure)
+    return failure();
+  if (bufferedBytes != 0) {
+    stream.write(buffer.data(), bufferedBytes);
+    bufferedBytes = 0;
+  }
+  return consumeStreamError() ? failure() : success();
+}
+
+LogicalResult PimInstructionWriter::append(const pim_binary::InstructionRecord& record) {
+  if (hasFailure || finalized || count == std::numeric_limits<uint32_t>::max()) {
+    hasFailure = true;
+    return failure();
+  }
+
+  if (bufferedBytes + pim_binary::kRecordSize > buffer.size() && failed(flushBuffer()))
+    return failure();
+
+  pim_binary::EncodedInstruction encoded = pim_binary::encodeInstructionRecord(record);
+  std::memcpy(buffer.data() + bufferedBytes, encoded.data(), encoded.size());
+  bufferedBytes += encoded.size();
+  ++count;
+  return success();
+}
+
+LogicalResult PimInstructionWriter::finalize() {
+  if (finalized)
+    return failure();
+  finalized = true;
+  if (hasFailure || failed(flushBuffer()))
+    return failure();
+
+  stream.flush();
+  if (consumeStreamError())
+    return failure();
+  pim_binary::patchInstructionCount(stream, count);
+  stream.flush();
+  return consumeStreamError() ? failure() : success();
+}
+
 OnnxMlirCompilerErrorCodes
 writeMemoryBinary(ModuleOp moduleOp, func::FuncOp funcOp, PimAcceleratorMemory& memory, StringRef outputDirPath) {
   auto memoryFilePath = (outputDirPath + "/memory.bin").str();

src/PIM/Compiler/PimArtifactWriter.hpp

@@ -2,16 +2,45 @@
 
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/IR/BuiltinOps.h"
+#include "mlir/Support/LogicalResult.h"
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/JSON.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <vector>
 
 #include "onnx-mlir/Compiler/OMCompilerTypes.h"
+#include "src/Accelerators/PIM/Compiler/PimBinaryFormat.hpp"
 
 namespace onnx_mlir {
 
 class PimAcceleratorMemory;
 
+class PimInstructionWriter {
+public:
+  explicit PimInstructionWriter(llvm::raw_pwrite_stream& stream);
+  explicit PimInstructionWriter(llvm::raw_fd_ostream& stream);
+
+  mlir::LogicalResult append(const pim_binary::InstructionRecord& record);
+  mlir::LogicalResult finalize();
+
+private:
+  static constexpr size_t kBufferSize = 256 * 1024;
+  mlir::LogicalResult flushBuffer();
+  bool consumeStreamError();
+
+  llvm::raw_pwrite_stream& stream;
+  llvm::raw_fd_ostream* fileStream = nullptr;
+  std::vector<char> buffer;
+  size_t bufferedBytes = 0;
+  uint32_t count = 0;
+  bool finalized = false;
+  bool hasFailure = false;
+};
+
 OnnxMlirCompilerErrorCodes writeMemoryBinary(mlir::ModuleOp moduleOp,
                                              mlir::func::FuncOp funcOp,
                                              PimAcceleratorMemory& memory,

src/PIM/Compiler/PimBinaryFormat.hpp

@@ -1,5 +1,6 @@
 #pragma once
 
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/JSON.h"
@@ -53,6 +54,14 @@ enum class Opcode : uint32_t {
   sync = 32,
 };
 
+inline constexpr size_t kOpcodeCount = static_cast<size_t>(Opcode::sync) + 1;
+inline constexpr std::array<llvm::StringLiteral, kOpcodeCount> kOpcodeNames = {
+  "nop",   "sldi", "sld",   "sadd",  "ssub", "smul", "saddi", "smuli", "setbw", "mvmul", "vvadd",
+  "vvsub", "vvmul", "vvdmul", "vvmax", "vvsll", "vvsra", "vavg", "vrelu", "vtanh", "vsigm", "vsoftmax",
+  "vmv",   "vrsu", "vrsl",  "ld",    "st",   "lldi", "lmv",   "send",  "recv",  "wait",  "sync",
+};
+static_assert(kOpcodeNames.size() == kOpcodeCount);
+
 struct InstructionRecord {
   Opcode opcode = Opcode::nop;
   uint8_t rd = 0;
@@ -64,14 +73,29 @@ struct InstructionRecord {
   uint8_t flags = 0;
 };
 
+using EncodedInstruction = std::array<char, kRecordSize>;
+static_assert(kRecordSize == 20);
+static_assert(sizeof(EncodedInstruction) == kRecordSize);
+
+inline EncodedInstruction encodeInstructionRecord(const InstructionRecord& record) {
+  EncodedInstruction encoded = {};
+  encoded[0] = static_cast<char>(static_cast<uint8_t>(record.opcode));
+  encoded[1] = static_cast<char>(record.rd);
+  encoded[2] = static_cast<char>(record.r1);
+  encoded[3] = static_cast<char>(record.flags);
+  llvm::support::endian::write32le(encoded.data() + 4, static_cast<uint32_t>(record.r2OrImm));
+  llvm::support::endian::write32le(encoded.data() + 8, static_cast<uint32_t>(record.generic1));
+  llvm::support::endian::write32le(encoded.data() + 12, static_cast<uint32_t>(record.generic2));
+  llvm::support::endian::write32le(encoded.data() + 16, static_cast<uint32_t>(record.generic3));
+  return encoded;
+}
+
 inline void writeUint32LE(llvm::raw_ostream& os, uint32_t value) {
   std::array<char, sizeof(uint32_t)> bytes;
   llvm::support::endian::write32le(bytes.data(), value);
   os.write(bytes.data(), bytes.size());
 }
 
-inline void writeInt32LE(llvm::raw_ostream& os, int32_t value) { writeUint32LE(os, static_cast<uint32_t>(value)); }
-
 inline void writeHeader(llvm::raw_ostream& os) {
   os.write(kMagic, sizeof(kMagic));
   writeUint32LE(os, kVersion);
@@ -84,17 +108,6 @@ inline void patchInstructionCount(llvm::raw_pwrite_stream& os, uint32_t instruct
   os.pwrite(bytes.data(), bytes.size(), kCountOffset);
 }
 
-inline void writeInstructionRecord(llvm::raw_ostream& os, const InstructionRecord& record) {
-  os << static_cast<char>(static_cast<uint8_t>(record.opcode));
-  os << static_cast<char>(record.rd);
-  os << static_cast<char>(record.r1);
-  os << static_cast<char>(record.flags);
-  writeInt32LE(os, record.r2OrImm);
-  writeInt32LE(os, record.generic1);
-  writeInt32LE(os, record.generic2);
-  writeInt32LE(os, record.generic3);
-}
-
 inline int32_t toI32(int64_t value) { return onnx_mlir::pim::checkedI32OrCrash(value, "binary field"); }
 
 inline uint8_t toU8(int64_t value) {
@@ -107,113 +120,64 @@ inline int32_t getOptionalInt(const llvm::json::Object& object, llvm::StringRef
   return defaultValue;
 }
 
+struct InstructionJsonFormat {
+  bool rd;
+  bool r1;
+  bool offset;
+  llvm::StringLiteral r2;
+  llvm::StringLiteral generic1;
+  llvm::StringLiteral generic2;
+  llvm::StringLiteral generic3;
+};
+
+inline constexpr std::array<InstructionJsonFormat, kOpcodeCount> kInstructionJsonFormats = {{
+  {false, false, false, "",     "",     "",      ""    }, // nop
+  {true,  false, false, "imm",  "",     "",      ""    }, // sldi
+  {true,  true,  true,  "",     "",     "",      ""    }, // sld
+  {true,  true,  false, "rs2",  "",     "",      ""    }, // sadd
+  {true,  true,  false, "rs2",  "",     "",      ""    }, // ssub
+  {true,  true,  false, "rs2",  "",     "",      ""    }, // smul
+  {true,  true,  false, "imm",  "",     "",      ""    }, // saddi
+  {true,  true,  false, "imm",  "",     "",      ""    }, // smuli
+  {false, false, false, "",     "ibiw", "obiw",  ""    }, // setbw
+  {true,  true,  false, "mbiw", "relu", "group", ""    }, // mvmul
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvadd
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvsub
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvmul
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvdmul
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvmax
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvsll
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vvsra
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vavg
+  {true,  true,  true,  "",     "",     "",      "len" }, // vrelu
+  {true,  true,  true,  "",     "",     "",      "len" }, // vtanh
+  {true,  true,  true,  "",     "",     "",      "len" }, // vsigm
+  {true,  true,  true,  "",     "",     "",      "len" }, // vsoftmax
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vmv
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vrsu
+  {true,  true,  true,  "rs2",  "",     "",      "len" }, // vrsl
+  {true,  true,  true,  "",     "",     "",      "size"}, // ld
+  {true,  true,  true,  "",     "",     "",      "size"}, // st
+  {true,  false, true,  "imm",  "",     "",      "len" }, // lldi
+  {true,  true,  true,  "",     "",     "",      "len" }, // lmv
+  {true,  false, true,  "core", "",     "",      "size"}, // send
+  {true,  false, true,  "core", "",     "",      "size"}, // recv
+  {false, false, false, "",     "",     "",      ""    }, // wait
+  {false, false, false, "",     "",     "",      ""    }, // sync
+}};
+static_assert(kInstructionJsonFormats.size() == kOpcodeCount);
+
 inline Opcode opcodeFromString(llvm::StringRef opName) {
-  if (opName == "nop")
+  for (auto [index, name] : llvm::enumerate(kOpcodeNames))
-    return Opcode::nop;
+    if (opName == name)
-  if (opName == "sldi")
+      return static_cast<Opcode>(index);
-    return Opcode::sldi;
-  if (opName == "sld")
-    return Opcode::sld;
-  if (opName == "sadd")
-    return Opcode::sadd;
-  if (opName == "ssub")
-    return Opcode::ssub;
-  if (opName == "smul")
-    return Opcode::smul;
-  if (opName == "saddi")
-    return Opcode::saddi;
-  if (opName == "smuli")
-    return Opcode::smuli;
-  if (opName == "setbw")
-    return Opcode::setbw;
-  if (opName == "mvmul")
-    return Opcode::mvmul;
-  if (opName == "vvadd")
-    return Opcode::vvadd;
-  if (opName == "vvsub")
-    return Opcode::vvsub;
-  if (opName == "vvmul")
-    return Opcode::vvmul;
-  if (opName == "vvdmul")
-    return Opcode::vvdmul;
-  if (opName == "vvmax")
-    return Opcode::vvmax;
-  if (opName == "vvsll")
-    return Opcode::vvsll;
-  if (opName == "vvsra")
-    return Opcode::vvsra;
-  if (opName == "vavg")
-    return Opcode::vavg;
-  if (opName == "vrelu")
-    return Opcode::vrelu;
-  if (opName == "vtanh")
-    return Opcode::vtanh;
-  if (opName == "vsigm")
-    return Opcode::vsigm;
-  if (opName == "vsoftmax")
-    return Opcode::vsoftmax;
-  if (opName == "vmv")
-    return Opcode::vmv;
-  if (opName == "vrsu")
-    return Opcode::vrsu;
-  if (opName == "vrsl")
-    return Opcode::vrsl;
-  if (opName == "ld")
-    return Opcode::ld;
-  if (opName == "st")
-    return Opcode::st;
-  if (opName == "lldi")
-    return Opcode::lldi;
-  if (opName == "lmv")
-    return Opcode::lmv;
-  if (opName == "send")
-    return Opcode::send;
-  if (opName == "recv")
-    return Opcode::recv;
-  if (opName == "wait")
-    return Opcode::wait;
-  if (opName == "sync")
-    return Opcode::sync;
   llvm_unreachable("Unsupported PIM binary opcode");
 }
 
 inline llvm::StringRef opcodeToString(Opcode opcode) {
-  switch (opcode) {
+  size_t index = static_cast<size_t>(opcode);
-  case Opcode::nop:      return "nop";
+  assert(index < kOpcodeNames.size() && "Unsupported PIM binary opcode");
-  case Opcode::sldi:     return "sldi";
+  return kOpcodeNames[index];
-  case Opcode::sld:      return "sld";
-  case Opcode::sadd:     return "sadd";
-  case Opcode::ssub:     return "ssub";
-  case Opcode::smul:     return "smul";
-  case Opcode::saddi:    return "saddi";
-  case Opcode::smuli:    return "smuli";
-  case Opcode::setbw:    return "setbw";
-  case Opcode::mvmul:    return "mvmul";
-  case Opcode::vvadd:    return "vvadd";
-  case Opcode::vvsub:    return "vvsub";
-  case Opcode::vvmul:    return "vvmul";
-  case Opcode::vvdmul:   return "vvdmul";
-  case Opcode::vvmax:    return "vvmax";
-  case Opcode::vvsll:    return "vvsll";
-  case Opcode::vvsra:    return "vvsra";
-  case Opcode::vavg:     return "vavg";
-  case Opcode::vrelu:    return "vrelu";
-  case Opcode::vtanh:    return "vtanh";
-  case Opcode::vsigm:    return "vsigm";
-  case Opcode::vsoftmax: return "vsoftmax";
-  case Opcode::vmv:      return "vmv";
-  case Opcode::vrsu:     return "vrsu";
-  case Opcode::vrsl:     return "vrsl";
-  case Opcode::ld:       return "ld";
-  case Opcode::st:       return "st";
-  case Opcode::lldi:     return "lldi";
-  case Opcode::lmv:      return "lmv";
-  case Opcode::send:     return "send";
-  case Opcode::recv:     return "recv";
-  case Opcode::wait:     return "wait";
-  case Opcode::sync:     return "sync";
-  }
-  llvm_unreachable("Unsupported PIM binary opcode");
 }
 
 inline InstructionRecord makeInstructionRecord(const llvm::json::Object& instruction) {
@@ -221,43 +185,25 @@ inline InstructionRecord makeInstructionRecord(const llvm::json::Object& instruc
   std::optional<llvm::StringRef> opName = instruction.getString("op");
   assert(opName && "Missing op field in PIM instruction");
   record.opcode = opcodeFromString(*opName);
+  const auto& format = kInstructionJsonFormats[static_cast<size_t>(record.opcode)];
+  if (format.rd)
     record.rd = toU8(getOptionalInt(instruction, "rd"));
+  if (format.r1)
     record.r1 = toU8(getOptionalInt(instruction, "rs1"));
-
+  if (!format.r2.empty())
-  switch (record.opcode) {
+    record.r2OrImm = getOptionalInt(instruction, format.r2);
-  case Opcode::sldi:
+  if (!format.generic1.empty())
-  case Opcode::saddi:
+    record.generic1 = getOptionalInt(instruction, format.generic1);
-  case Opcode::smuli:
+  if (!format.generic2.empty())
-  case Opcode::lldi:  record.r2OrImm = getOptionalInt(instruction, "imm"); break;
+    record.generic2 = getOptionalInt(instruction, format.generic2);
-  case Opcode::mvmul:
+  if (format.offset) {
-    record.r2OrImm = getOptionalInt(instruction, "mbiw");
-    record.generic1 = getOptionalInt(instruction, "relu");
-    record.generic2 = getOptionalInt(instruction, "group");
-    break;
-  case Opcode::setbw:
-    record.generic1 = getOptionalInt(instruction, "ibiw");
-    record.generic2 = getOptionalInt(instruction, "obiw");
-    break;
-  case Opcode::send:
-  case Opcode::recv:
-    record.r2OrImm = getOptionalInt(instruction, "core");
-    record.generic3 = getOptionalInt(instruction, "size");
-    break;
-  default: record.r2OrImm = getOptionalInt(instruction, "rs2"); break;
-  }
-
-  if (record.opcode != Opcode::mvmul && record.opcode != Opcode::setbw) {
     if (auto* offsetValue = instruction.getObject("offset")) {
       record.generic1 = getOptionalInt(*offsetValue, "offset_select");
       record.generic2 = getOptionalInt(*offsetValue, "offset_value");
     }
   }
-
+  if (!format.generic3.empty())
-  if (instruction.get("len"))
+    record.generic3 = getOptionalInt(instruction, format.generic3);
-    record.generic3 = getOptionalInt(instruction, "len");
-  else if (instruction.get("size") && record.opcode != Opcode::send && record.opcode != Opcode::recv)
-    record.generic3 = getOptionalInt(instruction, "size");
-
   return record;
 }
 
@@ -271,98 +217,21 @@ inline llvm::json::Object makeInstructionJson(const InstructionRecord& record) {
     offset["offset_value"] = offsetValue;
     instruction["offset"] = std::move(offset);
   };
-
+  const auto& format = kInstructionJsonFormats[static_cast<size_t>(record.opcode)];
-  switch (record.opcode) {
+  if (format.rd)
-  case Opcode::sldi:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["imm"] = record.r2OrImm;
-    break;
-  case Opcode::sld:
     instruction["rd"] = static_cast<int64_t>(record.rd);
+  if (format.r1)
     instruction["rs1"] = static_cast<int64_t>(record.r1);
+  if (!format.r2.empty())
+    instruction[format.r2] = record.r2OrImm;
+  if (!format.generic1.empty())
+    instruction[format.generic1] = record.generic1;
+  if (!format.generic2.empty())
+    instruction[format.generic2] = record.generic2;
+  if (format.offset)
     addOffset(record.generic1, record.generic2);
-    break;
+  if (!format.generic3.empty())
-  case Opcode::sadd:
+    instruction[format.generic3] = record.generic3;
-  case Opcode::ssub:
-  case Opcode::smul:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    instruction["rs2"] = record.r2OrImm;
-    break;
-  case Opcode::saddi:
-  case Opcode::smuli:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    instruction["imm"] = record.r2OrImm;
-    break;
-  case Opcode::setbw:
-    instruction["ibiw"] = record.generic1;
-    instruction["obiw"] = record.generic2;
-    break;
-  case Opcode::mvmul:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    instruction["mbiw"] = record.r2OrImm;
-    instruction["relu"] = record.generic1;
-    instruction["group"] = record.generic2;
-    break;
-  case Opcode::vvadd:
-  case Opcode::vvsub:
-  case Opcode::vvmul:
-  case Opcode::vvdmul:
-  case Opcode::vvmax:
-  case Opcode::vvsll:
-  case Opcode::vvsra:
-  case Opcode::vavg:
-  case Opcode::vmv:
-  case Opcode::vrsu:
-  case Opcode::vrsl:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    instruction["rs2"] = record.r2OrImm;
-    addOffset(record.generic1, record.generic2);
-    instruction["len"] = record.generic3;
-    break;
-  case Opcode::vrelu:
-  case Opcode::vtanh:
-  case Opcode::vsigm:
-  case Opcode::vsoftmax:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    addOffset(record.generic1, record.generic2);
-    instruction["len"] = record.generic3;
-    break;
-  case Opcode::ld:
-  case Opcode::st:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    addOffset(record.generic1, record.generic2);
-    instruction["size"] = record.generic3;
-    break;
-  case Opcode::lldi:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["imm"] = record.r2OrImm;
-    addOffset(record.generic1, record.generic2);
-    instruction["len"] = record.generic3;
-    break;
-  case Opcode::lmv:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["rs1"] = static_cast<int64_t>(record.r1);
-    addOffset(record.generic1, record.generic2);
-    instruction["len"] = record.generic3;
-    break;
-  case Opcode::send:
-  case Opcode::recv:
-    instruction["rd"] = static_cast<int64_t>(record.rd);
-    instruction["core"] = record.r2OrImm;
-    addOffset(record.generic1, record.generic2);
-    instruction["size"] = record.generic3;
-    break;
-  case Opcode::wait:
-  case Opcode::sync:
-  case Opcode::nop:  break;
-  }
-
   return instruction;
 }
 

src/PIM/Compiler/PimCodeGen.hpp

@@ -9,6 +9,7 @@
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/raw_os_ostream.h"
 
+#include <array>
 #include <fstream>
 #include <limits>
 #include <optional>
@@ -23,21 +24,15 @@
 
 namespace onnx_mlir {
 
+struct CompiledCoreProgram;
+struct CompiledTransposePlan;
+class PimInstructionWriter;
+
 struct MemEntry {
   size_t address;
   size_t size;
 };
 
-struct PhysicalSlotInfo {
-  size_t id = 0;
-  size_t address = 0;
-  size_t size = 0;
-};
-
-struct MemoryPlanArtifacts {
-  std::string textReport;
-};
-
 struct MemoryValueKey {
   mlir::Value value;
   std::optional<unsigned> lane;
@@ -45,15 +40,29 @@ struct MemoryValueKey {
   bool operator==(const MemoryValueKey& other) const { return value == other.value && lane == other.lane; }
 };
 
+struct CompiledLocalMemoryEntry {
+  mlir::Value value;
+  MemEntry memory;
+};
+
+struct CompiledCoreMemoryPlan {
+  llvm::SmallVector<CompiledLocalMemoryEntry, 32> entries;
+  uint64_t logicalAllocationCount = 0;
+  uint64_t logicalBytes = 0;
+  size_t arenaSize = 0;
+};
+
 struct MemoryReportRow {
-  uint64_t numAlloca = 0;
+  uint64_t hostObjectCount = 0;
-  uint64_t sizeAlloca = 0;
+  uint64_t hostBytes = 0;
-  uint64_t numGlobal = 0;
+  uint64_t logicalLocalAllocationCount = 0;
-  uint64_t sizeGlobal = 0;
+  uint64_t logicalLocalBytes = 0;
+  uint64_t physicalLocalBytes = 0;
 
   bool operator==(const MemoryReportRow& other) const {
-    return numAlloca == other.numAlloca && sizeAlloca == other.sizeAlloca && numGlobal == other.numGlobal
+    return hostObjectCount == other.hostObjectCount && hostBytes == other.hostBytes
-        && sizeGlobal == other.sizeGlobal;
+        && logicalLocalAllocationCount == other.logicalLocalAllocationCount
+        && logicalLocalBytes == other.logicalLocalBytes && physicalLocalBytes == other.physicalLocalBytes;
   }
 };
 
@@ -80,36 +89,30 @@ struct MemoryReportEntry {
   uint64_t id = 0;
   llvm::SmallVector<int32_t, 8> coreIds;
   MemoryReportRow row;
-  uint64_t totalAllocaCount = 0;
-  uint64_t totalAllocaBytes = 0;
 };
 
 class PimMemory {
   llvm::SmallVector<PendingMemEntry, 32> memEntries;
-  llvm::SmallVector<PhysicalSlotInfo, 32> localPhysicalSlots;
   llvm::SmallDenseMap<MemoryValueKey, MemEntry, 32>& globalMemEntriesMap;
   llvm::SmallDenseMap<MemoryValueKey, MemEntry, 32> ownedMemEntriesMap;
   MemoryReportRow reportRow;
-  MemoryPlanArtifacts livenessArtifacts;
+  std::optional<size_t> localArenaSize;
 
   size_t minAlignment = 4;
   size_t firstAvailableAddress = 0;
-  size_t nextPhysicalSlotId = 0;
 
   MemEntry* gatherMemEntry(mlir::Value value, std::optional<unsigned> lane = std::nullopt);
+  size_t allocateAddress(size_t size, const MemoryValueKey& key);
   void allocateGatheredMemory();
   void allocateMemoryForValue(const MemoryValueKey& key, MemEntry& memEntry, MemoryReportKind reportKind);
-  PhysicalSlotInfo allocatePhysicalSlot(size_t slotSize, const MemoryValueKey& key);
 
 public:
   PimMemory(llvm::SmallDenseMap<MemoryValueKey, MemEntry, 32>& globalMemEntriesMap)
   : globalMemEntriesMap(globalMemEntriesMap) {}
 
   void allocateHost(mlir::ModuleOp moduleOp, mlir::func::FuncOp funcOp);
-  void allocateCore(mlir::Operation* op, std::optional<unsigned> lane = std::nullopt);
+  void allocateCore(const CompiledCoreMemoryPlan& plan, std::optional<unsigned> lane = std::nullopt);
   MemoryReportRow getReportRow() const;
-  const MemoryPlanArtifacts& getLivenessArtifacts() const { return livenessArtifacts; }
-  void remove(mlir::Value val);
 
   size_t getFirstAvailableAddress() const { return firstAvailableAddress; }
   MemEntry getMemEntry(const MemoryValueKey& key) const;
@@ -130,12 +133,9 @@ private:
   mutable llvm::DenseMap<mlir::Value, CompiledAddressExpr> compiledAddressExprs;
 
 public:
-  PimAcceleratorMemory()
+  PimAcceleratorMemory();
-  : hostMem(memEntriesMap), fileReport(openReportFile("memory_report")) {}
+  PimAcceleratorMemory(
-  PimAcceleratorMemory(const llvm::SmallDenseMap<MemoryValueKey, MemEntry, 32>& initialMemEntries, bool enableReport)
+    const llvm::SmallDenseMap<MemoryValueKey, MemEntry, 32>& initialMemEntries, bool enableReport);
-  : memEntriesMap(initialMemEntries),
-    hostMem(memEntriesMap),
-    fileReport(enableReport ? openReportFile("memory_report") : std::fstream()) {}
 
   PimMemory& getOrCreateDeviceMem(size_t id);
 
@@ -145,19 +145,16 @@ public:
   llvm::FailureOr<int64_t> getIndexValue(mlir::Value value, const StaticValueKnowledge& knowledge = {}) const;
   void reportHost();
   void recordCoreReport(size_t coreId, const MemoryReportRow& row);
-  void recordBatchReport(uint64_t batchId,
+  void recordBatchReport(
-                         llvm::ArrayRef<int32_t> coreIds,
+    uint64_t batchId, llvm::ArrayRef<int32_t> coreIds, const MemoryReportRow& perCoreRow);
-                         const MemoryReportRow& perCoreRow,
-                         uint64_t totalAllocaCount,
-                         uint64_t totalAllocaBytes);
   void setTotalWeightBytes(uint64_t bytes) { totalWeightBytes = bytes; }
   void flushReport();
-  void clean(mlir::Operation* op);
 };
 
 struct CoreEmissionJob {
   mlir::Operation* coreLikeOp = nullptr;
-  size_t originalCoreId = 0;
+  const CompiledCoreProgram* program = nullptr;
+  const CompiledCoreMemoryPlan* memoryPlan = nullptr;
   size_t emittedCoreId = 0;
   llvm::SmallVector<unsigned, 4> lanes;
   std::optional<uint64_t> batchReportId;
@@ -165,11 +162,11 @@ struct CoreEmissionJob {
 
 class PimCodeGen {
   PimAcceleratorMemory& memory;
-  llvm::raw_fd_ostream& coreBinaryStream;
+  PimInstructionWriter& instructionWriter;
   llvm::raw_fd_ostream* coreJsonStream;
   const llvm::DenseMap<size_t, size_t>& emittedCoreIds;
   std::optional<unsigned> batchLane;
-  mutable uint32_t emittedInstructionCount = 0;
+  mutable std::array<std::optional<int32_t>, 256> scalarRegisterValues = {};
 
   size_t addressOf(mlir::Value value, const StaticValueKnowledge& knowledge) const {
     return memory.getValueAddress(value, knowledge, batchLane);
@@ -177,31 +174,46 @@ class PimCodeGen {
   size_t remapCoreId(size_t coreId) const;
 
   void emitInstruction(const pim_binary::InstructionRecord& instruction) const;
+  void updateScalarRegisterCache(const pim_binary::InstructionRecord& instruction) const;
 
+  void genSetRegisterImmediate(uint8_t registerNumber, int32_t immediate) const;
   void genSetRegisterImmediateUnsigned(size_t registerNumber, size_t immediate) const;
   void setupRd(size_t rdAddress, size_t rdOffset) const;
   void setupRdRs1(size_t rdAddress, size_t rdOffset, size_t rs1Address, size_t rs1Offset) const;
   void setupRdRs1Rs2(
     size_t rdAddress, size_t rdOffset, size_t rs1Address, size_t rs1Offset, size_t rs2Address, size_t rs2Offset) const;
 
-  void emitMemCopyOp(mlir::StringRef opName,
+  void emitMemCopyOp(pim_binary::Opcode opcode,
                      size_t rdAddr,
                      size_t rdOffset,
                      size_t rs1Addr,
                      size_t rs1Offset,
                      size_t size,
                      mlir::StringRef sizeFieldName = "size") const;
-  void emitCommunicationOp(mlir::StringRef opName, size_t bufferAddr, size_t coreId, size_t size) const;
+  void emitCommunicationOp(pim_binary::Opcode opcode, size_t bufferAddr, size_t coreId, size_t size) const;
   void emitMvmOp(size_t groupId, size_t rdAddr, size_t rdOffset, size_t rs1Addr, size_t rs1Offset) const;
 
 public:
+  void emitBinaryVectorOp(pim_binary::Opcode opcode,
+                          mlir::Value output,
+                          mlir::Value lhs,
+                          mlir::Value rhs,
+                          size_t byteSize,
+                          const StaticValueKnowledge& knowledge) const;
+  void emitUnaryVectorOp(pim_binary::Opcode opcode,
+                         mlir::Value output,
+                         mlir::Value input,
+                         size_t byteSize,
+                         const StaticValueKnowledge& knowledge,
+                         int32_t r2OrImm = 0,
+                         int32_t generic1 = 0) const;
+
   PimCodeGen(PimAcceleratorMemory& memory,
-             llvm::raw_fd_ostream& coreBinary,
+             PimInstructionWriter& instructionWriter,
              llvm::raw_fd_ostream* coreJson,
              const llvm::DenseMap<size_t, size_t>& emittedCoreIds)
-  : memory(memory), coreBinaryStream(coreBinary), coreJsonStream(coreJson), emittedCoreIds(emittedCoreIds) {}
+  : memory(memory), instructionWriter(instructionWriter), coreJsonStream(coreJson), emittedCoreIds(emittedCoreIds) {}
 
-  uint32_t getEmittedInstructionCount() const { return emittedInstructionCount; }
   void setBatchLane(std::optional<unsigned> lane) { batchLane = lane; }
   llvm::FailureOr<int64_t> indexOf(mlir::Value value, const StaticValueKnowledge& knowledge) const {
     return memory.getIndexValue(value, knowledge);
@@ -218,18 +230,7 @@ public:
   template <typename MVMTy>
   void codeGenMVMLikeOp(size_t mvmId, MVMTy mvmLikeOp, bool transposeMatrix, const StaticValueKnowledge& knowledge);
 
-  void codeGenVVAddOp(pim::PimVVAddOp vvaddOp, const StaticValueKnowledge& knowledge) const;
+  void codeGenTransposeOp(const CompiledTransposePlan& plan, const StaticValueKnowledge& knowledge) const;
-  void codeGenVVSubOp(pim::PimVVSubOp vvsubOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVVMulOp(pim::PimVVMulOp vvmulOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVVMaxOp(pim::PimVVMaxOp vvmaxOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVVDMulOp(pim::PimVVDMulOp vvdmulOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVAvgOp(pim::PimVAvgOp vavgOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVReluOp(pim::PimVReluOp vreluOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVTanhOp(pim::PimVTanhOp vtanhOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVSigmOp(pim::PimVSigmOp vsigmOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenVSoftmaxOp(pim::PimVSoftmaxOp vsoftmaxOp, const StaticValueKnowledge& knowledge) const;
-  void codeGetGlobalOp(mlir::memref::GetGlobalOp getGlobalOp, const StaticValueKnowledge& knowledge) const;
-  void codeGenTransposeOp(pim::PimTransposeOp transposeOp, const StaticValueKnowledge& knowledge) const;
 };
 
 OnnxMlirCompilerErrorCodes compileToPimCode(mlir::ModuleOp& moduleOpRef, std::string& outputDirName);

src/PIM/Compiler/PimCompilerOptions.cpp

@@ -15,21 +15,53 @@ llvm::cl::opt<PimEmissionTargetType> pimEmissionTarget(
   llvm::cl::init(EmitPimCodegen),
   llvm::cl::cat(OnnxMlirOptions));
 
-llvm::cl::opt<PimMergeSchedulerType>
-  pimMergeScheduler("pim-merge-scheduler",
-                    llvm::cl::desc("Scheduler used by the Spatial merge-compute-nodes pass"),
-                    llvm::cl::values(clEnumValN(MergeSchedulerPeft, "peft", "Use PEFT scheduling")),
-                    llvm::cl::init(MergeSchedulerPeft),
-                    llvm::cl::cat(OnnxMlirOptions));
-
 llvm::cl::opt<PimMemoryReportLevel> pimMemoryReport(
   "pim-memory-report",
   llvm::cl::desc("Emit a human-readable PIM memory planning report"),
   llvm::cl::values(clEnumValN(PimMemoryReportNone, "none", "Do not emit any PIM memory planning report")),
+  llvm::cl::values(clEnumValN(PimMemoryReportSummary, "summary", "Emit a concise PIM memory summary")),
+  llvm::cl::init(PimMemoryReportSummary),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<PimConvLoweringType> pimConvLowering(
+  "pim-conv-lowering",
+  llvm::cl::desc("Convolution lowering strategy for PIM"),
+  llvm::cl::values(clEnumValN(PimConvLoweringAuto, "auto", "Select the Conv lowering strategy automatically")),
+  llvm::cl::values(clEnumValN(PimConvLoweringLegacy, "legacy", "Use the legacy explicit-im2col Conv lowering")),
+  llvm::cl::values(clEnumValN(PimConvLoweringDepthwise, "depthwise", "Force the depthwise-specialized Conv lowering")),
   llvm::cl::values(
-    clEnumValN(PimMemoryReportSummary, "summary", "Emit a concise slot reuse report with key offenders")),
+    clEnumValN(PimConvLoweringPackedIm2Col, "packed-im2col", "Use explicit im2col with packed multi-position GEMM")),
-  llvm::cl::values(clEnumValN(PimMemoryReportFull, "full", "Emit the full detailed PIM memory planning report")),
+  llvm::cl::values(clEnumValN(PimConvLoweringStreamedPatch,
-  llvm::cl::init(PimMemoryReportNone),
+                              "streamed-patch",
+                              "Use streamed/chunked im2col rows without multi-position packing")),
+  llvm::cl::values(clEnumValN(PimConvLoweringStreamedPacked,
+                              "streamed-packed",
+                              "Use streamed/chunked im2col rows with packed multi-position GEMM")),
+  llvm::cl::values(clEnumValN(PimConvLoweringOutputChannelTiled,
+                              "output-channel-tiled",
+                              "Force Conv lowering that relies on Gemm output-channel tiling")),
+  llvm::cl::values(
+    clEnumValN(PimConvLoweringInputKTiled, "input-k-tiled", "Force Conv lowering that relies on Gemm K tiling")),
+  llvm::cl::values(clEnumValN(PimConvLoweringTiled2D,
+                              "tiled-2d",
+                              "Force Conv lowering that relies on Gemm 2D K/C tiling")),
+  llvm::cl::init(PimConvLoweringAuto),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<PimSpatialDataflowExportType> pimExportSpatialDataflow(
+  "pim-export-spatial-dataflow",
+  llvm::cl::desc("Emit Gephi-importable CSV dataflow reports for Spatial pipeline snapshots"),
+  llvm::cl::values(clEnumValN(SpatialDataflowExportNone, "none", "Do not emit Spatial dataflow CSV reports")),
+  llvm::cl::values(
+    clEnumValN(SpatialDataflowExportSpatial1, "spatial1", "Emit spatial1 graph dataflow CSV reports")),
+  llvm::cl::values(
+    clEnumValN(SpatialDataflowExportSpatial2, "spatial2", "Emit spatial2 trivially merged graph dataflow CSV reports")),
+  llvm::cl::values(
+    clEnumValN(SpatialDataflowExportSpatial3, "spatial3", "Emit spatial3 scheduled dataflow CSV reports")),
+  llvm::cl::values(
+    clEnumValN(SpatialDataflowExportSpatial4, "spatial4", "Emit spatial4 realized dataflow CSV reports")),
+  llvm::cl::values(clEnumValN(SpatialDataflowExportAll, "all", "Emit all Spatial dataflow CSV reports")),
+  llvm::cl::init(SpatialDataflowExportNone),
   llvm::cl::cat(OnnxMlirOptions));
 
 llvm::cl::opt<bool>
@@ -38,27 +70,56 @@ llvm::cl::opt<bool>
                  llvm::cl::init(false),
                  llvm::cl::cat(OnnxMlirOptions));
 
-llvm::cl::opt<bool>
-  pimDisableMemoryCoalescing("pim-disable-memory-coalescing",
-                             llvm::cl::desc("Skip the PIM memory coalescing pass (developer diagnostic option)"),
-                             llvm::cl::init(false),
-                             llvm::cl::cat(OnnxMlirOptions));
-
 llvm::cl::opt<bool> useExperimentalConvImpl("use-experimental-conv-impl",
                                             llvm::cl::desc("Use experimental implementation for convolution"),
                                             llvm::cl::init(false),
                                             llvm::cl::cat(OnnxMlirOptions));
 
+llvm::cl::opt<uint64_t> pimConvIm2colMaxElements(
+  "pim-conv-im2col-max-elements",
+  llvm::cl::desc("Maximum number of im2col elements to materialize globally for one Conv before streaming/chunking"),
+  llvm::cl::init(1ull << 20),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<uint64_t> pimConvStreamChunkPositions(
+  "pim-conv-stream-chunk-positions",
+  llvm::cl::desc("Maximum number of Conv output positions to materialize in one streamed chunk"),
+  llvm::cl::init(1024),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<bool> pimReportConvLowering("pim-report-conv-lowering",
+                                          llvm::cl::desc("Emit a bounded Conv lowering report"),
+                                          llvm::cl::init(true),
+                                          llvm::cl::cat(OnnxMlirOptions));
+
 llvm::cl::opt<bool> pimEmitJson("pim-emit-json",
                                 llvm::cl::desc("Also emit per-core JSON instruction files alongside binary .pim files"),
                                 llvm::cl::init(false),
                                 llvm::cl::cat(OnnxMlirOptions));
 
-llvm::cl::opt<size_t>
+llvm::cl::opt<bool> pimDetectCommunicationDeadlock(
-  crossbarSize("crossbar-size", llvm::cl::desc("Width and height of a single crossbar"), llvm::cl::init(2));
+  "pim-detect-communication-deadlock",
+  llvm::cl::desc("Expensively simulate the statically expanded PIM send/receive order at verification time and fail if a blocking communication deadlock is found"),
+  llvm::cl::init(false),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<bool> pimMaterializeScalarFanoutGlobalOrder(
+  "pim-materialize-scalar-fanout-global-order",
+  llvm::cl::desc("Experimental expensive materializer mode: emit scalar-source fanout as globally ordered communication events instead of all-send fanout loops"),
+  llvm::cl::init(false),
+  llvm::cl::cat(OnnxMlirOptions));
+
+llvm::cl::opt<bool> pimTraceCommunicationMaterialization(
+  "pim-trace-communication-materialization",
+  llvm::cl::desc("Emit verbose materializer-time diagnostics and provenance attributes for every Spatial communication op"),
+  llvm::cl::init(false),
+  llvm::cl::cat(OnnxMlirOptions));
 
 llvm::cl::opt<size_t>
-  crossbarCountInCore("crossbar-count", llvm::cl::desc("Number of crossbars in each core"), llvm::cl::init(256));
+  crossbarSize("crossbar-size", llvm::cl::desc("Width and height of a single crossbar"), llvm::cl::init(128));
+
+llvm::cl::opt<size_t>
+  crossbarCountInCore("crossbar-count", llvm::cl::desc("Number of crossbars in each core"), llvm::cl::init(64));
 
 llvm::cl::opt<long> coresCount("core-count",
                                llvm::cl::desc("Number of cores in the chip. Required for PIM compilation."),

src/PIM/Compiler/PimCompilerOptions.hpp

@@ -20,29 +20,51 @@ typedef enum {
   EmitPimCodegen = 3
 } PimEmissionTargetType;
 
-typedef enum {
-  MergeSchedulerPeft = 0,
-} PimMergeSchedulerType;
-
 typedef enum {
   PimMemoryReportNone = 0,
   PimMemoryReportSummary = 1,
-  PimMemoryReportFull = 2,
 } PimMemoryReportLevel;
 
+typedef enum {
+  PimConvLoweringAuto = 0,
+  PimConvLoweringLegacy = 1,
+  PimConvLoweringDepthwise = 2,
+  PimConvLoweringPackedIm2Col = 3,
+  PimConvLoweringStreamedPatch = 4,
+  PimConvLoweringStreamedPacked = 5,
+  PimConvLoweringOutputChannelTiled = 6,
+  PimConvLoweringInputKTiled = 7,
+  PimConvLoweringTiled2D = 8,
+} PimConvLoweringType;
+
+typedef enum {
+  SpatialDataflowExportNone = 0,
+  SpatialDataflowExportSpatial1 = 1,
+  SpatialDataflowExportSpatial2 = 2,
+  SpatialDataflowExportSpatial3 = 3,
+  SpatialDataflowExportSpatial4 = 4,
+  SpatialDataflowExportAll = 5,
+} PimSpatialDataflowExportType;
+
 extern llvm::cl::OptionCategory OnnxMlirOptions;
 extern llvm::cl::opt<PimEmissionTargetType> pimEmissionTarget;
-extern llvm::cl::opt<PimMergeSchedulerType> pimMergeScheduler;
 extern llvm::cl::opt<PimMemoryReportLevel> pimMemoryReport;
+extern llvm::cl::opt<PimConvLoweringType> pimConvLowering;
+extern llvm::cl::opt<PimSpatialDataflowExportType> pimExportSpatialDataflow;
 
 extern llvm::cl::opt<bool> pimOnlyCodegen;
-extern llvm::cl::opt<bool> pimDisableMemoryCoalescing;
 extern llvm::cl::opt<bool> useExperimentalConvImpl;
 extern llvm::cl::opt<bool> pimEmitJson;
+extern llvm::cl::opt<bool> pimReportConvLowering;
+extern llvm::cl::opt<bool> pimDetectCommunicationDeadlock;
+extern llvm::cl::opt<bool> pimMaterializeScalarFanoutGlobalOrder;
+extern llvm::cl::opt<bool> pimTraceCommunicationMaterialization;
 
 extern llvm::cl::opt<size_t> crossbarSize;
 extern llvm::cl::opt<size_t> crossbarCountInCore;
 extern llvm::cl::opt<long> coresCount;
+extern llvm::cl::opt<uint64_t> pimConvIm2colMaxElements;
+extern llvm::cl::opt<uint64_t> pimConvStreamChunkPositions;
 
 bool hasExplicitPimCoreCount();
 void verifyExplicitPimCoreCount();

src/PIM/Compiler/PimCompilerUtils.cpp

@@ -1,3 +1,4 @@
+#include "mlir/Conversion/AffineToStandard/AffineToStandard.h"
 #include "mlir/Transforms/Passes.h"
 
 #include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
@@ -20,7 +21,9 @@ void addPassesPim(OwningOpRef<ModuleOp>& module,
   verifyExplicitPimCoreCount();
 
   if (pimOnlyCodegen) {
-    // Skip all the lowering passes and directly generate code for PIM.
+    pm.addPass(createPimLocalMemoryPlanningPass());
+    pm.addPass(createPimVerificationPass());
+    pm.addPass(createEmitPimCodePass());
     return;
   }
 
@@ -29,6 +32,9 @@ void addPassesPim(OwningOpRef<ModuleOp>& module,
 
   if (pimEmissionTarget >= EmitSpatial) {
     pm.addPass(createONNXToSpatialPass());
+    pm.addPass(createSpatialLayoutPlanningPass());
+    pm.addPass(createLowerSpatialPlansPass());
+    pm.addPass(createTrivialGraphComputeMergePass());
     pm.addPass(createMergeComputeNodesPass());
     pm.addPass(createMessagePass("Onnx lowered to Spatial"));
   }
@@ -44,10 +50,11 @@ void addPassesPim(OwningOpRef<ModuleOp>& module,
   }
 
   if (pimEmissionTarget >= EmitPimCodegen) {
+    pm.addPass(mlir::createLowerAffinePass());
     pm.addPass(createPimHostConstantFoldingPass());
     pm.addPass(createMessagePass("Pim host constants folded"));
-    if (!pimDisableMemoryCoalescing)
+    pm.addPass(createPimLocalMemoryPlanningPass());
-      pm.addPass(createPimMemoryCoalescingPass());
+    pm.addPass(createMessagePass("Pim local memory planned"));
     pm.addPass(createPimVerificationPass());
     pm.addPass(createMessagePass("Pim verified"));
     pm.addPass(createEmitPimCodePass());

src/PIM/Compiler/PimCoreProgram.cpp

@@ -0,0 +1,209 @@
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/BuiltinTypes.h"
+
+#include "llvm/ADT/STLExtras.h"
+
+#include "src/Accelerators/PIM/Common/IR/CoreBlockUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
+#include "src/Accelerators/PIM/Compiler/PimCoreProgram.hpp"
+#include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
+
+using namespace llvm;
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static FailureOr<CompiledCoreOpKind> classifyCompiledCoreOpKind(Operation& op) {
+  if (isa<pim::PimMemCopyHostToDevOp>(op)) return CompiledCoreOpKind::Load;
+  if (isa<pim::PimMemCopyDevToHostOp>(op)) return CompiledCoreOpKind::Store;
+  if (isa<pim::PimMemCopyOp>(op)) return CompiledCoreOpKind::Lmv;
+  if (isa<pim::PimReceiveOp>(op)) return CompiledCoreOpKind::Receive;
+  if (isa<pim::PimSendOp>(op)) return CompiledCoreOpKind::Send;
+  if (isa<pim::PimConcatOp>(op)) return CompiledCoreOpKind::Concat;
+  if (isa<pim::PimVMMOp>(op)) return CompiledCoreOpKind::Vmm;
+  if (isa<pim::PimTransposeOp>(op)) return CompiledCoreOpKind::Transpose;
+  if (isa<pim::PimVVAddOp>(op)) return CompiledCoreOpKind::VVAdd;
+  if (isa<pim::PimVVSubOp>(op)) return CompiledCoreOpKind::VVSub;
+  if (isa<pim::PimVVMulOp>(op)) return CompiledCoreOpKind::VVMul;
+  if (isa<pim::PimVVMaxOp>(op)) return CompiledCoreOpKind::VVMax;
+  if (isa<pim::PimVVDMulOp>(op)) return CompiledCoreOpKind::VVDMul;
+  if (isa<pim::PimVAvgOp>(op)) return CompiledCoreOpKind::VAvg;
+  if (isa<pim::PimVReluOp>(op)) return CompiledCoreOpKind::VRelu;
+  if (isa<pim::PimVTanhOp>(op)) return CompiledCoreOpKind::VTanh;
+  if (isa<pim::PimVSigmOp>(op)) return CompiledCoreOpKind::VSigm;
+  if (isa<pim::PimVSoftmaxOp>(op)) return CompiledCoreOpKind::VSoftmax;
+  return failure();
+}
+
+static bool isStoragePreservingTranspose(ArrayRef<size_t> sourceShape, ArrayRef<int64_t> permutation) {
+  SmallVector<unsigned> sourceNonUnitDims;
+  SmallVector<unsigned> destinationSourceNonUnitDims;
+  for (auto [dim, size] : llvm::enumerate(sourceShape))
+    if (size != 1)
+      sourceNonUnitDims.push_back(dim);
+  for (int64_t sourceDim : permutation)
+    if (sourceShape[sourceDim] != 1)
+      destinationSourceNonUnitDims.push_back(static_cast<unsigned>(sourceDim));
+  return sourceNonUnitDims == destinationSourceNonUnitDims;
+}
+
+static FailureOr<CompiledTransposePlan> compileTransposePlan(pim::PimTransposeOp transposeOp) {
+  auto sourceType = cast<ShapedType>(transposeOp.getInput().getType());
+  ArrayRef<int64_t> sourceShape = sourceType.getShape();
+  size_t rank = sourceShape.size();
+  CompiledTransposePlan plan;
+  plan.source = transposeOp.getInput();
+  plan.destination = transposeOp.getOutputBuffer();
+  plan.elementBytes = getElementTypeSizeInBytes(sourceType.getElementType());
+  auto totalElements = pim::checkedSize(sourceType.getNumElements(), transposeOp, "transpose elements");
+  if (failed(totalElements)) return failure();
+  plan.totalElements = *totalElements;
+  auto totalBytes = pim::checkedMul(plan.totalElements, plan.elementBytes, transposeOp, "transpose byte size");
+  if (failed(totalBytes)) return failure();
+  plan.totalBytes = *totalBytes;
+
+  SmallVector<int64_t> permutation = map_to_vector(transposeOp.getPermutation().getAsRange<IntegerAttr>(),
+                                                   [](IntegerAttr attr) { return attr.getInt(); });
+  if (permutation.size() != rank) {
+    transposeOp.emitOpError("requires permutation rank to match source rank for PIM codegen");
+    return failure();
+  }
+
+  SmallVector<size_t> destinationShape(rank);
+  plan.destinationStrides.assign(rank, 1);
+  plan.destinationDimensionForSource.assign(rank, 0);
+  plan.destinationRewinds.assign(rank, 0);
+  SmallVector<bool> seenSourceDimensions(rank, false);
+  for (size_t dim = 0; dim < rank; ++dim) {
+    auto size = pim::checkedSize(sourceShape[dim], transposeOp, "transpose source dimension");
+    if (failed(size)) return failure();
+    plan.sourceShape.push_back(*size);
+  }
+  for (auto [destinationDim, sourceDim] : llvm::enumerate(permutation)) {
+    if (sourceDim < 0 || static_cast<size_t>(sourceDim) >= rank || seenSourceDimensions[sourceDim]) {
+      transposeOp.emitOpError("requires a valid permutation containing each source dimension exactly once");
+      return failure();
+    }
+    seenSourceDimensions[sourceDim] = true;
+    destinationShape[destinationDim] = plan.sourceShape[sourceDim];
+    plan.destinationDimensionForSource[sourceDim] = destinationDim;
+  }
+  for (size_t dim = rank; dim > 1; --dim) {
+    auto stride = pim::checkedMul(
+      plan.destinationStrides[dim - 1], destinationShape[dim - 1], transposeOp, "transpose destination stride");
+    if (failed(stride)) return failure();
+    plan.destinationStrides[dim - 2] = *stride;
+  }
+  for (size_t sourceDim = 0; sourceDim < rank; ++sourceDim) {
+    auto rewind = pim::checkedMul(plan.sourceShape[sourceDim],
+                                  plan.destinationStrides[plan.destinationDimensionForSource[sourceDim]],
+                                  transposeOp,
+                                  "transpose destination rewind");
+    if (failed(rewind)) return failure();
+    plan.destinationRewinds[sourceDim] = *rewind;
+  }
+  plan.storagePreserving = isStoragePreservingTranspose(plan.sourceShape, permutation);
+  return plan;
+}
+
+static LogicalResult compileCoreEmissionPlan(Block& block, SmallVectorImpl<CompiledCoreNode>& plan) {
+  for (Operation& op : block) {
+    if (isa<pim::PimHaltOp, scf::YieldOp, memref::GetGlobalOp>(op) || isCoreStaticAddressOp(&op))
+      continue;
+    if (auto loadOp = dyn_cast<memref::LoadOp>(op); loadOp && succeeded(compileIndexExpr(loadOp.getResult())))
+      continue;
+
+    if (auto forOp = dyn_cast<scf::ForOp>(op)) {
+      auto lower = compileIndexExpr(forOp.getLowerBound());
+      auto upper = compileIndexExpr(forOp.getUpperBound());
+      auto step = compileIndexExpr(forOp.getStep());
+      if (failed(lower) || failed(upper) || failed(step)) {
+        forOp.emitOpError("requires statically evaluable scf.for bounds for PIM codegen");
+        return failure();
+      }
+      CompiledCoreNode node;
+      node.kind = CompiledCoreNode::Kind::Loop;
+      node.op = forOp;
+      node.lowerBound = *lower;
+      node.upperBound = *upper;
+      node.step = *step;
+      node.loopBody = std::make_unique<SmallVector<CompiledCoreNode, 8>>();
+      if (failed(compileCoreEmissionPlan(forOp.getRegion().front(), *node.loopBody))) return failure();
+      plan.push_back(std::move(node));
+      continue;
+    }
+    if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
+      auto condition = compileIndexExpr(ifOp.getCondition());
+      if (failed(condition)) {
+        ifOp.emitOpError("requires statically evaluable scf.if condition for PIM codegen");
+        return failure();
+      }
+      CompiledCoreNode node;
+      node.kind = CompiledCoreNode::Kind::If;
+      node.op = ifOp;
+      node.condition = *condition;
+      node.thenBody = std::make_unique<SmallVector<CompiledCoreNode, 8>>();
+      node.elseBody = std::make_unique<SmallVector<CompiledCoreNode, 8>>();
+      if (failed(compileCoreEmissionPlan(ifOp.getThenRegion().front(), *node.thenBody))) return failure();
+      if (!ifOp.getElseRegion().empty()
+          && failed(compileCoreEmissionPlan(ifOp.getElseRegion().front(), *node.elseBody)))
+        return failure();
+      plan.push_back(std::move(node));
+      continue;
+    }
+    if (auto switchOp = dyn_cast<scf::IndexSwitchOp>(op)) {
+      auto selector = compileIndexExpr(switchOp.getArg());
+      if (failed(selector)) {
+        switchOp.emitOpError("requires a statically evaluable scf.index_switch selector for PIM codegen");
+        return failure();
+      }
+      CompiledCoreNode node;
+      node.kind = CompiledCoreNode::Kind::IndexSwitch;
+      node.op = switchOp;
+      node.condition = *selector;
+      llvm::append_range(node.caseValues, switchOp.getCases());
+      for (Region& region : switchOp.getCaseRegions()) {
+        auto body = std::make_unique<SmallVector<CompiledCoreNode, 8>>();
+        if (failed(compileCoreEmissionPlan(region.front(), *body))) return failure();
+        node.caseBodies.push_back(std::move(body));
+      }
+      node.defaultBody = std::make_unique<SmallVector<CompiledCoreNode, 8>>();
+      if (failed(compileCoreEmissionPlan(switchOp.getDefaultRegion().front(), *node.defaultBody))) return failure();
+      plan.push_back(std::move(node));
+      continue;
+    }
+
+    auto opKind = classifyCompiledCoreOpKind(op);
+    if (failed(opKind)) {
+      InFlightDiagnostic diagnostic = op.emitError() << "unsupported codegen for op '" << op.getName() << "'";
+      if (auto coreOp = op.getParentOfType<pim::PimCoreOp>())
+        diagnostic << " inside pim.core " << coreOp.getCoreId();
+      else if (auto batchOp = op.getParentOfType<pim::PimCoreBatchOp>())
+        diagnostic << " inside pim.core_batch with laneCount " << batchOp.getLaneCount();
+      return failure();
+    }
+    CompiledCoreNode node;
+    node.op = &op;
+    node.opKind = *opKind;
+    if (*opKind == CompiledCoreOpKind::Transpose) {
+      auto transposePlan = compileTransposePlan(cast<pim::PimTransposeOp>(op));
+      if (failed(transposePlan)) return failure();
+      node.transposePlan = *transposePlan;
+    }
+    plan.push_back(std::move(node));
+  }
+  return success();
+}
+
+} // namespace
+
+LogicalResult compileCoreProgram(Operation* coreLikeOp, CompiledCoreProgram& program) {
+  Block& block = isa<pim::PimCoreOp>(coreLikeOp) ? cast<pim::PimCoreOp>(coreLikeOp).getBody().front()
+                                                 : cast<pim::PimCoreBatchOp>(coreLikeOp).getBody().front();
+  return compileCoreEmissionPlan(block, program.nodes);
+}
+
+} // namespace onnx_mlir

src/PIM/Compiler/PimCoreProgram.hpp

@@ -0,0 +1,74 @@
+#pragma once
+
+#include "mlir/IR/Operation.h"
+#include "mlir/Support/LogicalResult.h"
+
+#include "llvm/ADT/SmallVector.h"
+
+#include <memory>
+#include <optional>
+
+#include "src/Accelerators/PIM/Common/IR/AddressAnalysis.hpp"
+
+namespace onnx_mlir {
+
+struct CompiledTransposePlan {
+  mlir::Value source;
+  mlir::Value destination;
+  size_t elementBytes = 0;
+  size_t totalElements = 0;
+  size_t totalBytes = 0;
+  llvm::SmallVector<size_t> sourceShape;
+  llvm::SmallVector<size_t> destinationStrides;
+  llvm::SmallVector<unsigned> destinationDimensionForSource;
+  llvm::SmallVector<size_t> destinationRewinds;
+  bool storagePreserving = false;
+};
+
+enum class CompiledCoreOpKind : uint8_t {
+  Load,
+  Store,
+  Lmv,
+  Receive,
+  Send,
+  Concat,
+  Vmm,
+  Transpose,
+  VVAdd,
+  VVSub,
+  VVMul,
+  VVMax,
+  VVDMul,
+  VAvg,
+  VRelu,
+  VTanh,
+  VSigm,
+  VSoftmax
+};
+
+struct CompiledCoreNode {
+  enum class Kind : uint8_t { Op, Loop, If, IndexSwitch };
+
+  Kind kind = Kind::Op;
+  mlir::Operation* op = nullptr;
+  CompiledCoreOpKind opKind = CompiledCoreOpKind::Load;
+  CompiledIndexExpr lowerBound;
+  CompiledIndexExpr upperBound;
+  CompiledIndexExpr step;
+  CompiledIndexExpr condition;
+  std::unique_ptr<llvm::SmallVector<CompiledCoreNode, 8>> loopBody;
+  std::unique_ptr<llvm::SmallVector<CompiledCoreNode, 8>> thenBody;
+  std::unique_ptr<llvm::SmallVector<CompiledCoreNode, 8>> elseBody;
+  llvm::SmallVector<int64_t> caseValues;
+  llvm::SmallVector<std::unique_ptr<llvm::SmallVector<CompiledCoreNode, 8>>> caseBodies;
+  std::unique_ptr<llvm::SmallVector<CompiledCoreNode, 8>> defaultBody;
+  std::optional<CompiledTransposePlan> transposePlan;
+};
+
+struct CompiledCoreProgram {
+  llvm::SmallVector<CompiledCoreNode, 32> nodes;
+};
+
+mlir::LogicalResult compileCoreProgram(mlir::Operation* coreLikeOp, CompiledCoreProgram& program);
+
+} // namespace onnx_mlir

src/PIM/Compiler/PimMemoryLiveness.cpp

@@ -1,733 +0,0 @@
-#include "mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h"
-#include "mlir/Dialect/MemRef/IR/MemRef.h"
-#include "mlir/Dialect/SCF/IR/SCF.h"
-#include "mlir/IR/Value.h"
-#include "mlir/Interfaces/DestinationStyleOpInterface.h"
-
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/SmallPtrSet.h"
-#include "llvm/Support/raw_ostream.h"
-
-#include <numeric>
-#include <string>
-#include <tuple>
-#include <utility>
-
-#include "Common/Support/CheckedArithmetic.hpp"
-#include "Common/Support/ReportUtils.hpp"
-#include "src/Accelerators/PIM/Common/PimCommon.hpp"
-#include "src/Accelerators/PIM/Compiler/PimMemoryLiveness.hpp"
-#include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
-
-using namespace llvm;
-using namespace mlir;
-using namespace onnx_mlir;
-
-namespace {
-
-static std::optional<unsigned> getLaneForMemoryValue(mlir::Value value, std::optional<unsigned> lane) {
-  if (!lane)
-    return std::nullopt;
-  auto allocOp = value.getDefiningOp<memref::AllocOp>();
-  if (!allocOp || !allocOp->getParentOfType<pim::PimCoreBatchOp>())
-    return std::nullopt;
-  return lane;
-}
-
-static MemoryValueKey getMemoryValueKey(mlir::Value value, std::optional<unsigned> lane = std::nullopt) {
-  return {value, getLaneForMemoryValue(value, lane)};
-}
-
-struct MemoryTouchInterval {
-  uint64_t start = 0;
-  uint64_t end = 0;
-  Operation* startOp = nullptr;
-  Operation* endOp = nullptr;
-  Operation* firstTouchOp = nullptr;
-  Operation* lastTouchOp = nullptr;
-  uint64_t firstTouchPosition = 0;
-  uint64_t lastTouchPosition = 0;
-  bool hasRuntimeUse = false;
-  bool startUsedAllocFallback = false;
-  bool endUsedFallback = false;
-  bool escapesLoop = false;
-  std::string fallbackReason;
-  llvm::SmallVector<std::string, 8> aliasesFollowed;
-};
-
-struct OperationOrdering {
-  llvm::DenseMap<Operation*, uint64_t> position;
-  llvm::DenseMap<Operation*, uint64_t> subtreeEnd;
-  uint64_t nextPosition = 0;
-};
-
-static std::string printValueToString(mlir::Value value) {
-  std::string text;
-  llvm::raw_string_ostream os(text);
-  value.print(os);
-  os.flush();
-  return text;
-}
-
-static std::string printOperationToString(Operation* op) {
-  if (!op)
-    return "<none>";
-  std::string text;
-  llvm::raw_string_ostream os(text);
-  op->print(os);
-  os.flush();
-  return text;
-}
-
-static std::string printLocationToString(Location loc) {
-  std::string text;
-  llvm::raw_string_ostream os(text);
-  loc.print(os);
-  os.flush();
-  return text;
-}
-
-static std::string collapseWhitespace(StringRef text) {
-  std::string out;
-  out.reserve(text.size());
-  bool lastWasSpace = false;
-  for (char c : text) {
-    bool isSpace = c == ' ' || c == '\n' || c == '\t' || c == '\r';
-    if (isSpace) {
-      if (!lastWasSpace && !out.empty())
-        out.push_back(' ');
-      lastWasSpace = true;
-      continue;
-    }
-    out.push_back(c);
-    lastWasSpace = false;
-  }
-  return out;
-}
-
-static std::string abbreviate(StringRef text, size_t maxLen) {
-  if (text.size() <= maxLen)
-    return text.str();
-  return (text.take_front(maxLen - 3) + "...").str();
-}
-
-static std::string summarizeValue(mlir::Value value, size_t maxLen = 72) {
-  return abbreviate(collapseWhitespace(printValueToString(value)), maxLen);
-}
-
-static std::string summarizeOperation(Operation* op, size_t maxLen = 96) {
-  if (!op)
-    return "<none>";
-  std::string prefix = op->getName().getStringRef().str();
-  std::string full = collapseWhitespace(printOperationToString(op));
-  if (full == prefix)
-    return prefix;
-  return abbreviate(prefix + " :: " + full, maxLen);
-}
-
-static std::string summarizeLocation(Location loc, size_t maxLen = 88) {
-  return abbreviate(collapseWhitespace(printLocationToString(loc)), maxLen);
-}
-
-static void assignOperationOrdering(Operation* op, OperationOrdering& ordering) {
-  uint64_t position = ordering.nextPosition++;
-  ordering.position[op] = position;
-  uint64_t end = position;
-  for (Region& region : op->getRegions())
-    for (Block& block : region)
-      for (Operation& nestedOp : block) {
-        assignOperationOrdering(&nestedOp, ordering);
-        end = std::max(end, ordering.subtreeEnd.lookup(&nestedOp));
-      }
-  ordering.subtreeEnd[op] = end;
-}
-
-static OperationOrdering buildOperationOrdering(Operation* coreLikeOp) {
-  OperationOrdering ordering;
-  if (!coreLikeOp || coreLikeOp->getNumRegions() != 1 || coreLikeOp->getRegion(0).empty())
-    return ordering;
-
-  for (Operation& op : coreLikeOp->getRegion(0).front())
-    assignOperationOrdering(&op, ordering);
-  return ordering;
-}
-
-static bool isSupportedAliasOp(Operation* op) {
-  return isa<memref::SubViewOp, memref::CastOp, memref::CollapseShapeOp, memref::ExpandShapeOp>(op);
-}
-
-static bool isRuntimeMemoryTouchOp(Operation* op) {
-  return isa<pim::PimMemCopyHostToDevOp,
-             pim::PimMemCopyDevToHostOp,
-             pim::PimMemCopyOp,
-             pim::PimReceiveOp,
-             pim::PimSendOp,
-             pim::PimConcatOp,
-             pim::PimVMMOp,
-             pim::PimTransposeOp,
-             pim::PimVVAddOp,
-             pim::PimVVSubOp,
-             pim::PimVVMulOp,
-             pim::PimVVMaxOp,
-             pim::PimVVDMulOp,
-             pim::PimVAvgOp,
-             pim::PimVReluOp,
-             pim::PimVTanhOp,
-             pim::PimVSigmOp,
-             pim::PimVSoftmaxOp>(op);
-}
-
-static bool isIgnoredLivenessUser(Operation* op) {
-  return isSupportedAliasOp(op) || isa<scf::ForOp, scf::YieldOp, memref::DeallocOp>(op) || isCoreStaticAddressOp(op);
-}
-
-static bool isWithin(mlir::Value value, Region* region) {
-  if (!region)
-    return false;
-  if (auto blockArg = dyn_cast<BlockArgument>(value))
-    return blockArg.getOwner()->getParent() == region;
-  if (Operation* definingOp = value.getDefiningOp())
-    return definingOp->getParentRegion() == region || region->isAncestor(definingOp->getParentRegion());
-  return false;
-}
-
-static bool isNestedAllocation(Operation* coreLikeOp, memref::AllocOp allocOp) {
-  if (!coreLikeOp || coreLikeOp->getNumRegions() != 1 || coreLikeOp->getRegion(0).empty())
-    return false;
-  return allocOp->getBlock() != &coreLikeOp->getRegion(0).front();
-}
-
-static void addFallbackReason(std::string& reason, StringRef newReason) {
-  if (newReason.empty())
-    return;
-  if (!reason.empty())
-    reason += "; ";
-  reason += newReason.str();
-}
-
-static void appendAliasDescription(llvm::SmallVectorImpl<std::string>& aliases, mlir::Value value) {
-  std::string text = printValueToString(value);
-  if (!llvm::is_contained(aliases, text))
-    aliases.push_back(std::move(text));
-}
-
-struct OrderedTouchRange {
-  uint64_t start = 0;
-  uint64_t end = 0;
-  Operation* startOp = nullptr;
-  Operation* endOp = nullptr;
-  bool escapedLoop = false;
-};
-
-static OrderedTouchRange
-getEffectiveTouchRange(mlir::Value definingValue, Operation* user, const OperationOrdering& ordering) {
-  OrderedTouchRange range {ordering.position.lookup(user), ordering.position.lookup(user), user, user, false};
-  for (Operation* current = user; current; current = current->getParentOp()) {
-    auto forOp = dyn_cast<scf::ForOp>(current);
-    if (!forOp || isWithin(definingValue, &forOp.getRegion()))
-      continue;
-    range.start = std::min(range.start, ordering.position.lookup(forOp));
-    range.end = std::max(range.end, ordering.subtreeEnd.lookup(forOp));
-    range.startOp = forOp;
-    range.endOp = forOp;
-    range.escapedLoop = true;
-  }
-  return range;
-}
-
-static MemoryTouchInterval
-computeMemoryTouchInterval(memref::AllocOp allocOp, const OperationOrdering& ordering, uint64_t fallbackEnd) {
-  MemoryTouchInterval interval;
-  interval.start = ordering.position.lookup(allocOp);
-  interval.end = interval.start;
-  interval.startOp = allocOp;
-  interval.endOp = allocOp;
-
-  SmallPtrSet<mlir::Value, 16> visitedValues;
-  SmallPtrSet<Operation*, 32> visitedUsers;
-  SmallVector<mlir::Value> pendingValues;
-  pendingValues.push_back(allocOp.getResult());
-  auto parentLoop = allocOp->getParentOfType<scf::ForOp>();
-
-  while (!pendingValues.empty()) {
-    mlir::Value value = pendingValues.pop_back_val();
-    if (!visitedValues.insert(value).second)
-      continue;
-
-    for (Operation* user : value.getUsers()) {
-      if (!visitedUsers.insert(user).second)
-        continue;
-
-      if (isSupportedAliasOp(user)) {
-        for (mlir::Value result : user->getResults()) {
-          pendingValues.push_back(result);
-          appendAliasDescription(interval.aliasesFollowed, result);
-        }
-      }
-
-      if (auto dpsOp = dyn_cast<DestinationStyleOpInterface>(user)) {
-        for (OpResult result : user->getResults()) {
-          OpOperand* tiedOperand = dpsOp.getTiedOpOperand(result);
-          if (!tiedOperand || tiedOperand->get() != value)
-            continue;
-          pendingValues.push_back(result);
-          appendAliasDescription(interval.aliasesFollowed, result);
-        }
-      }
-
-      if (auto forOp = dyn_cast<scf::ForOp>(user)) {
-        for (auto [index, initArg] : llvm::enumerate(forOp.getInitArgs())) {
-          if (initArg != value)
-            continue;
-          pendingValues.push_back(forOp.getRegionIterArgs()[index]);
-          pendingValues.push_back(forOp.getResult(index));
-          appendAliasDescription(interval.aliasesFollowed, forOp.getRegionIterArgs()[index]);
-          appendAliasDescription(interval.aliasesFollowed, forOp.getResult(index));
-          if (parentLoop && forOp != parentLoop)
-            interval.escapesLoop = true;
-        }
-      }
-
-      if (auto yieldOp = dyn_cast<scf::YieldOp>(user)) {
-        auto forOp = dyn_cast<scf::ForOp>(yieldOp->getParentOp());
-        if (!forOp) {
-          addFallbackReason(interval.fallbackReason, "yield without scf.for parent");
-        }
-        else {
-          for (auto [index, operand] : llvm::enumerate(yieldOp.getOperands())) {
-            if (operand != value)
-              continue;
-            pendingValues.push_back(forOp.getResult(index));
-            appendAliasDescription(interval.aliasesFollowed, forOp.getResult(index));
-            if (parentLoop && forOp == parentLoop)
-              interval.escapesLoop = true;
-          }
-        }
-      }
-
-      if (isRuntimeMemoryTouchOp(user)) {
-        uint64_t touchPosition = ordering.position.lookup(user);
-        if (!interval.hasRuntimeUse || touchPosition < interval.firstTouchPosition) {
-          interval.firstTouchPosition = touchPosition;
-          interval.firstTouchOp = user;
-        }
-        if (!interval.hasRuntimeUse || touchPosition > interval.lastTouchPosition) {
-          interval.lastTouchPosition = touchPosition;
-          interval.lastTouchOp = user;
-        }
-
-        OrderedTouchRange range = getEffectiveTouchRange(allocOp.getResult(), user, ordering);
-        interval.escapesLoop |= range.escapedLoop;
-        if (!interval.hasRuntimeUse) {
-          interval.start = range.start;
-          interval.end = range.end;
-          interval.startOp = range.startOp;
-          interval.endOp = range.endOp;
-          interval.hasRuntimeUse = true;
-        }
-        else {
-          if (range.start < interval.start) {
-            interval.start = range.start;
-            interval.startOp = range.startOp;
-          }
-          if (range.end > interval.end) {
-            interval.end = range.end;
-            interval.endOp = range.endOp;
-          }
-        }
-        continue;
-      }
-
-      if (isIgnoredLivenessUser(user))
-        continue;
-
-      addFallbackReason(interval.fallbackReason, "unhandled user op");
-      interval.endUsedFallback = true;
-    }
-  }
-
-  if (!interval.hasRuntimeUse) {
-    interval.startUsedAllocFallback = true;
-    interval.endUsedFallback = true;
-    interval.start = ordering.position.lookup(allocOp);
-    interval.end = fallbackEnd;
-    interval.startOp = allocOp;
-    interval.endOp = allocOp->getParentOp();
-    interval.firstTouchPosition = interval.start;
-    interval.lastTouchPosition = interval.end;
-    addFallbackReason(interval.fallbackReason, "no runtime memory touch");
-    return interval;
-  }
-
-  if (interval.endUsedFallback) {
-    interval.end = std::max(interval.end, fallbackEnd);
-    interval.endOp = allocOp->getParentOp();
-  }
-
-  return interval;
-}
-
-static FailureOr<size_t> getAllocSizeBytes(memref::AllocOp allocOp) {
-  auto type = dyn_cast<ShapedType>(allocOp.getType());
-  if (!type)
-    return failure();
-  auto checkedBytes = pim::getCheckedShapedTypeSizeInBytes(type, allocOp, "memory allocation byte size");
-  if (failed(checkedBytes))
-    return failure();
-  return pim::checkedSize(*checkedBytes, allocOp, "memory allocation byte size");
-}
-
-static bool intervalsOverlap(const LocalAllocInterval& lhs, const LocalAllocInterval& rhs) {
-  return !(lhs.end < rhs.start || rhs.end < lhs.start);
-}
-
-static uint64_t getSlotLogicalBytes(const PlannedPhysicalSlot& slot, ArrayRef<LocalAllocInterval> intervals) {
-  uint64_t slotLogicalBytes = 0;
-  for (size_t intervalIndex : slot.intervalIndices)
-    slotLogicalBytes += intervals[intervalIndex].size;
-  return slotLogicalBytes;
-}
-
-} // namespace
-
-SmallVector<LocalAllocInterval, 0> onnx_mlir::buildLocalAllocIntervals(Operation* coreLikeOp,
-                                                                       std::optional<unsigned> lane) {
-  SmallVector<LocalAllocInterval, 0> intervals;
-  OperationOrdering ordering = buildOperationOrdering(coreLikeOp);
-  if (ordering.position.empty())
-    return intervals;
-
-  uint64_t fallbackEnd = ordering.nextPosition == 0 ? 0 : ordering.nextPosition - 1;
-  size_t nextIntervalId = 0;
-  coreLikeOp->walk([&](memref::AllocOp allocOp) {
-    auto checkedSize = getAllocSizeBytes(allocOp);
-    if (failed(checkedSize)) {
-      llvm::errs() << "Failed to compute local allocation size for value: ";
-      allocOp.getResult().print(llvm::errs());
-      llvm::errs() << "\n";
-      llvm_unreachable("Failed to compute local allocation size");
-    }
-
-    MemoryTouchInterval touchInterval = computeMemoryTouchInterval(allocOp, ordering, fallbackEnd);
-    LocalAllocInterval interval;
-    interval.id = nextIntervalId++;
-    interval.alloc = allocOp;
-    interval.key = getMemoryValueKey(allocOp.getResult(), lane);
-    interval.start = touchInterval.start;
-    interval.end = touchInterval.end;
-    interval.size = *checkedSize;
-    interval.startOp = touchInterval.startOp;
-    interval.endOp = touchInterval.endOp;
-    interval.firstTouchOp = touchInterval.firstTouchOp;
-    interval.lastTouchOp = touchInterval.lastTouchOp;
-    interval.firstTouchPosition = touchInterval.firstTouchPosition;
-    interval.lastTouchPosition = touchInterval.lastTouchPosition;
-    interval.startUsedAllocFallback = touchInterval.startUsedAllocFallback;
-    interval.endUsedFallback = touchInterval.endUsedFallback;
-    interval.hasRuntimeUse = touchInterval.hasRuntimeUse;
-    interval.insideNestedRegion = isNestedAllocation(coreLikeOp, allocOp);
-    interval.escapesLoop = touchInterval.escapesLoop;
-    interval.fallbackReason = std::move(touchInterval.fallbackReason);
-    interval.aliasesFollowed = std::move(touchInterval.aliasesFollowed);
-    intervals.push_back(std::move(interval));
-  });
-
-  return intervals;
-}
-
-SmallVector<PlannedPhysicalSlot, 0> onnx_mlir::planPhysicalSlots(MutableArrayRef<LocalAllocInterval> intervals) {
-  SmallVector<PlannedPhysicalSlot, 0> slots;
-  SmallVector<size_t> intervalOrder(intervals.size());
-  std::iota(intervalOrder.begin(), intervalOrder.end(), 0);
-  llvm::stable_sort(intervalOrder, [&](size_t lhsIndex, size_t rhsIndex) {
-    const LocalAllocInterval& lhs = intervals[lhsIndex];
-    const LocalAllocInterval& rhs = intervals[rhsIndex];
-    if (lhs.size != rhs.size)
-      return lhs.size > rhs.size;
-    if (lhs.start != rhs.start)
-      return lhs.start < rhs.start;
-    if (lhs.end != rhs.end)
-      return lhs.end < rhs.end;
-    return lhs.id < rhs.id;
-  });
-
-  for (size_t intervalIndex : intervalOrder) {
-    LocalAllocInterval& interval = intervals[intervalIndex];
-    PlannedPhysicalSlot* bestSlot = nullptr;
-    auto bestKey = std::tuple<size_t, size_t, size_t, size_t>(std::numeric_limits<size_t>::max(),
-                                                              std::numeric_limits<size_t>::max(),
-                                                              std::numeric_limits<size_t>::max(),
-                                                              std::numeric_limits<size_t>::max());
-
-    for (size_t slotIndex = 0; slotIndex < slots.size(); ++slotIndex) {
-      PlannedPhysicalSlot& slot = slots[slotIndex];
-      bool compatible = true;
-      for (size_t otherIndex : slot.intervalIndices) {
-        if (intervalsOverlap(interval, intervals[otherIndex])) {
-          compatible = false;
-          break;
-        }
-      }
-      if (!compatible)
-        continue;
-
-      size_t resultingSize = std::max(slot.requiredSize, interval.size);
-      size_t growth = resultingSize - slot.requiredSize;
-      auto candidateKey =
-        std::tuple<size_t, size_t, size_t, size_t>(growth, resultingSize, slot.intervalIndices.size(), slot.id);
-      if (candidateKey < bestKey) {
-        bestKey = candidateKey;
-        bestSlot = &slot;
-      }
-    }
-
-    if (!bestSlot) {
-      slots.push_back({slots.size(), interval.size, interval.size, 0, {intervalIndex}});
-      interval.slotPlanIndex = slots.size() - 1;
-      interval.physicalSlotId = slots.back().id;
-      interval.physicalSlotSize = slots.back().requiredSize;
-      continue;
-    }
-
-    bestSlot->requiredSize = std::max(bestSlot->requiredSize, interval.size);
-    bestSlot->size = bestSlot->requiredSize;
-    bestSlot->intervalIndices.push_back(intervalIndex);
-    interval.slotPlanIndex = static_cast<size_t>(bestSlot - slots.data());
-    interval.physicalSlotId = bestSlot->id;
-    interval.physicalSlotSize = bestSlot->requiredSize;
-  }
-
-  return slots;
-}
-
-MemoryPlanArtifacts onnx_mlir::buildMemoryPlanArtifacts(Operation* coreLikeOp,
-                                                        std::optional<unsigned> lane,
-                                                        ArrayRef<LocalAllocInterval> intervals,
-                                                        ArrayRef<PlannedPhysicalSlot> slots,
-                                                        size_t addressLimit,
-                                                        PimMemoryReportLevel reportLevel) {
-  MemoryPlanArtifacts artifacts;
-
-  uint64_t totalLogicalBytes = 0;
-  uint64_t totalPhysicalBytes = 0;
-  uint64_t fallbackIntervals = 0;
-  uint64_t noRuntimeTouchIntervals = 0;
-  uint64_t reusedAllocations = 0;
-  uint64_t nestedIntervals = 0;
-  uint64_t loopEscapingIntervals = 0;
-  size_t largestLogicalAllocation = 0;
-  size_t largestPhysicalSlot = 0;
-  size_t maximumAssignedAddress = 0;
-
-  for (const LocalAllocInterval& interval : intervals) {
-    totalLogicalBytes += interval.size;
-    largestLogicalAllocation = std::max(largestLogicalAllocation, interval.size);
-    maximumAssignedAddress = std::max(maximumAssignedAddress, interval.assignedAddress + interval.physicalSlotSize);
-    if (interval.startUsedAllocFallback || interval.endUsedFallback)
-      ++fallbackIntervals;
-    if (!interval.hasRuntimeUse)
-      ++noRuntimeTouchIntervals;
-    if (interval.insideNestedRegion)
-      ++nestedIntervals;
-    if (interval.escapesLoop)
-      ++loopEscapingIntervals;
-  }
-  for (const PlannedPhysicalSlot& slot : slots) {
-    totalPhysicalBytes += slot.size;
-    largestPhysicalSlot = std::max(largestPhysicalSlot, slot.size);
-    if (slot.intervalIndices.size() > 1)
-      reusedAllocations += slot.intervalIndices.size() - 1;
-  }
-
-  uint64_t savedBytes = totalLogicalBytes >= totalPhysicalBytes ? totalLogicalBytes - totalPhysicalBytes : 0;
-  double savedPercent =
-    totalLogicalBytes == 0 ? 0.0 : 100.0 * static_cast<double>(savedBytes) / static_cast<double>(totalLogicalBytes);
-
-  raw_string_ostream os(artifacts.textReport);
-  os << "=== PIM Memory Liveness Report ===\n";
-  os << "Op: " << coreLikeOp->getName() << "\n";
-  if (lane)
-    os << "Lane: " << *lane << "\n";
-  os << "Summary:\n";
-  os << "  logical allocation bytes:  " << formatReportMemory(totalLogicalBytes) << " (" << totalLogicalBytes << ")\n";
-  os << "  physical allocation bytes: " << formatReportMemory(totalPhysicalBytes) << " (" << totalPhysicalBytes
-     << ")\n";
-  os << "  saved bytes:               " << formatReportMemory(savedBytes) << " (" << savedBytes << ")\n";
-  os << "  saved percent: " << format("%.2f%%", savedPercent) << "\n";
-  os << "  intervals: " << intervals.size() << "\n";
-  os << "  physical slots: " << slots.size() << "\n";
-  os << "  reused allocations: " << reusedAllocations << "\n";
-  os << "  fallback intervals: " << fallbackIntervals << "\n";
-  os << "  intervals with no runtime memory touch: " << noRuntimeTouchIntervals << "\n";
-  os << "  nested allocations: " << nestedIntervals << "\n";
-  os << "  loop-escaping allocations: " << loopEscapingIntervals << "\n";
-  os << "  largest logical allocation: " << largestLogicalAllocation << "\n";
-  os << "  largest physical slot: " << largestPhysicalSlot << "\n";
-  os << "  address limit: " << addressLimit << "\n";
-  os << "  peak physical memory: " << formatReportMemory(maximumAssignedAddress) << " (" << maximumAssignedAddress
-     << ")\n";
-  os << "  maximum assigned address: " << maximumAssignedAddress << "\n";
-
-  os << "\nHow To Read:\n";
-  os << "  `summary` only shows the strongest reuse cases and the worst offenders.\n";
-  os << "  Use `--pim-memory-report=full` when you need the complete slot-by-slot and interval-by-interval dump.\n";
-  os << "  Large single-use slots, fallback intervals, and nested single-use allocations are the best places\n";
-  os << "  to inspect if allocations should be moved, sunk, or made easier to coalesce earlier in the pipeline.\n";
-
-  SmallVector<const PlannedPhysicalSlot*> reusedSlots;
-  SmallVector<const PlannedPhysicalSlot*> singleUseSlots;
-  for (const PlannedPhysicalSlot& slot : slots)
-    if (slot.intervalIndices.size() > 1)
-      reusedSlots.push_back(&slot);
-    else
-      singleUseSlots.push_back(&slot);
-
-  llvm::stable_sort(reusedSlots, [&](const PlannedPhysicalSlot* lhs, const PlannedPhysicalSlot* rhs) {
-    uint64_t lhsLogicalBytes = getSlotLogicalBytes(*lhs, intervals);
-    uint64_t rhsLogicalBytes = getSlotLogicalBytes(*rhs, intervals);
-    if (lhs->intervalIndices.size() != rhs->intervalIndices.size())
-      return lhs->intervalIndices.size() > rhs->intervalIndices.size();
-    if (lhsLogicalBytes != rhsLogicalBytes)
-      return lhsLogicalBytes > rhsLogicalBytes;
-    if (lhs->size != rhs->size)
-      return lhs->size > rhs->size;
-    return lhs->id < rhs->id;
-  });
-  llvm::stable_sort(singleUseSlots, [&](const PlannedPhysicalSlot* lhs, const PlannedPhysicalSlot* rhs) {
-    if (lhs->size != rhs->size)
-      return lhs->size > rhs->size;
-    return lhs->id < rhs->id;
-  });
-
-  constexpr size_t kSummaryReuseLimit = 6;
-  constexpr size_t kSummaryOffenderLimit = 10;
-
-  os << "\nBest Reuse:\n";
-  if (reusedSlots.empty()) {
-    os << "  no slots were shared by multiple intervals\n";
-  }
-  else {
-    for (const PlannedPhysicalSlot* slot : ArrayRef(reusedSlots).take_front(kSummaryReuseLimit)) {
-      uint64_t slotLogicalBytes = getSlotLogicalBytes(*slot, intervals);
-      os << "  slot #" << slot->id << "  addr=" << slot->address << "  size=" << formatReportMemory(slot->size)
-         << "  intervals=" << slot->intervalIndices.size() << "  logical_sum=" << formatReportMemory(slotLogicalBytes)
-         << "\n";
-      for (size_t intervalIndex : slot->intervalIndices) {
-        const LocalAllocInterval& interval = intervals[intervalIndex];
-        os << "    #" << interval.id << "  [" << interval.start << "," << interval.end << "]"
-           << "  logical=" << formatReportMemory(interval.size)
-           << "  first=" << summarizeOperation(interval.firstTouchOp, 40)
-           << "  last=" << summarizeOperation(interval.lastTouchOp, 40) << "\n";
-      }
-    }
-  }
-
-  os << "\nTop Offenders:\n";
-  bool printedAttention = false;
-  for (const PlannedPhysicalSlot* slot : ArrayRef(singleUseSlots).take_front(kSummaryOffenderLimit)) {
-    const LocalAllocInterval& interval = intervals[slot->intervalIndices.front()];
-    printedAttention = true;
-    os << "  slot #" << slot->id << " is single-use"
-       << "  size=" << formatReportMemory(slot->size) << "  interval=#" << interval.id
-       << "  value=" << summarizeValue(interval.key.value, 56) << "\n";
-    os << "    first=" << summarizeOperation(interval.firstTouchOp, 40)
-       << "  last=" << summarizeOperation(interval.lastTouchOp, 40)
-       << "  nested=" << (interval.insideNestedRegion ? "yes" : "no")
-       << "  escapes_loop=" << (interval.escapesLoop ? "yes" : "no") << "\n";
-  }
-  size_t fallbackPrinted = 0;
-  for (const LocalAllocInterval& interval : intervals) {
-    if (!(interval.startUsedAllocFallback || interval.endUsedFallback) || fallbackPrinted >= kSummaryOffenderLimit)
-      continue;
-    printedAttention = true;
-    ++fallbackPrinted;
-    os << "  fallback interval #" << interval.id << "  size=" << formatReportMemory(interval.size)
-       << "  value=" << summarizeValue(interval.key.value, 56) << "\n";
-    os << "    reason: " << (interval.fallbackReason.empty() ? "<none>" : interval.fallbackReason) << "\n";
-  }
-  size_t nestedPrinted = 0;
-  for (const LocalAllocInterval& interval : intervals) {
-    if (nestedPrinted >= kSummaryOffenderLimit)
-      break;
-    if (!(interval.insideNestedRegion && slots[interval.slotPlanIndex].intervalIndices.size() == 1))
-      continue;
-    printedAttention = true;
-    ++nestedPrinted;
-    os << "  nested single-use interval #" << interval.id << "  slot #" << interval.physicalSlotId
-       << "  size=" << formatReportMemory(interval.size) << "  value=" << summarizeValue(interval.key.value, 56)
-       << "\n";
-    os << "    hint: move or sink this alloc inside the nested region if the IR allows it.\n";
-  }
-  if (!printedAttention)
-    os << "  no obvious blockers detected in this core\n";
-
-  if (reportLevel == PimMemoryReportFull) {
-    os << "\nSlot Reuse:\n";
-    for (const PlannedPhysicalSlot& slot : slots) {
-      uint64_t slotLogicalBytes = getSlotLogicalBytes(slot, intervals);
-      os << "  slot #" << slot.id << "  addr=" << slot.address << "  size=" << formatReportMemory(slot.size) << " ("
-         << slot.size << ")"
-         << "  intervals=" << slot.intervalIndices.size() << "  logical_sum=" << formatReportMemory(slotLogicalBytes)
-         << "\n";
-      for (size_t intervalIndex : slot.intervalIndices) {
-        const LocalAllocInterval& interval = intervals[intervalIndex];
-        mlir::Value allocValue = interval.key.value;
-        os << "    [" << interval.start << "," << interval.end << "]"
-           << "  #" << interval.id << "  logical=" << formatReportMemory(interval.size)
-           << "  nested=" << (interval.insideNestedRegion ? "yes" : "no")
-           << "  escapes_loop=" << (interval.escapesLoop ? "yes" : "no")
-           << "  first=" << summarizeOperation(interval.firstTouchOp, 48)
-           << "  last=" << summarizeOperation(interval.lastTouchOp, 48) << "\n";
-        os << "      value=" << summarizeValue(allocValue) << "\n";
-      }
-    }
-  }
-
-  if (reportLevel == PimMemoryReportFull) {
-    os << "\nInterval Details:\n";
-    for (const LocalAllocInterval& interval : intervals) {
-      const PlannedPhysicalSlot& slot = slots[interval.slotPlanIndex];
-      mlir::Value allocValue = interval.key.value;
-      Operation* definingOp = allocValue.getDefiningOp();
-      os << "  #" << interval.id << "  slot=" << slot.id << "  live=[" << interval.start << "," << interval.end << "]"
-         << "  logical=" << formatReportMemory(interval.size)
-         << "  slot_size=" << formatReportMemory(interval.physicalSlotSize) << "  addr=" << interval.assignedAddress
-         << "\n";
-      os << "    value=" << summarizeValue(allocValue, 88) << "\n";
-      os << "    type=" << allocValue.getType() << "\n";
-      os << "    loc="
-         << summarizeLocation(definingOp ? definingOp->getLoc() : UnknownLoc::get(coreLikeOp->getContext())) << "\n";
-      os << "    nested=" << (interval.insideNestedRegion ? "yes" : "no")
-         << "  escapes_loop=" << (interval.escapesLoop ? "yes" : "no")
-         << "  start_fallback=" << (interval.startUsedAllocFallback ? "yes" : "no")
-         << "  end_fallback=" << (interval.endUsedFallback ? "yes" : "no") << "\n";
-      os << "    first_use=" << summarizeOperation(interval.firstTouchOp) << " @" << interval.firstTouchPosition
-         << "\n";
-      os << "    last_use=" << summarizeOperation(interval.lastTouchOp) << " @" << interval.lastTouchPosition << "\n";
-      os << "    slot_peers=";
-      bool first = true;
-      for (size_t otherIndex : slot.intervalIndices) {
-        if (intervals[otherIndex].id == interval.id)
-          continue;
-        if (!first)
-          os << ", ";
-        os << "#" << intervals[otherIndex].id;
-        first = false;
-      }
-      if (first)
-        os << "<none>";
-      os << "\n";
-      if (!interval.fallbackReason.empty())
-        os << "    fallback_reason=" << interval.fallbackReason << "\n";
-      if (!interval.aliasesFollowed.empty()) {
-        os << "    aliases_followed=" << interval.aliasesFollowed.size() << "\n";
-        for (const std::string& alias : interval.aliasesFollowed)
-          os << "      - " << abbreviate(collapseWhitespace(alias), 108) << "\n";
-      }
-    }
-  }
-  os.flush();
-
-  return artifacts;
-}

src/PIM/Compiler/PimMemoryLiveness.hpp

@@ -1,63 +0,0 @@
-#pragma once
-
-#include "mlir/Dialect/MemRef/IR/MemRef.h"
-
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/SmallVector.h"
-
-#include <limits>
-#include <optional>
-#include <string>
-
-#include "src/Accelerators/PIM/Compiler/PimCodeGen.hpp"
-#include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
-
-namespace onnx_mlir {
-
-struct LocalAllocInterval {
-  size_t id = 0;
-  mlir::memref::AllocOp alloc;
-  MemoryValueKey key;
-  uint64_t start = 0;
-  uint64_t end = 0;
-  size_t size = 0;
-  mlir::Operation* startOp = nullptr;
-  mlir::Operation* endOp = nullptr;
-  mlir::Operation* firstTouchOp = nullptr;
-  mlir::Operation* lastTouchOp = nullptr;
-  uint64_t firstTouchPosition = 0;
-  uint64_t lastTouchPosition = 0;
-  bool startUsedAllocFallback = false;
-  bool endUsedFallback = false;
-  bool hasRuntimeUse = false;
-  bool insideNestedRegion = false;
-  bool escapesLoop = false;
-  std::string fallbackReason;
-  llvm::SmallVector<std::string, 8> aliasesFollowed;
-  size_t slotPlanIndex = std::numeric_limits<size_t>::max();
-  size_t physicalSlotId = std::numeric_limits<size_t>::max();
-  size_t assignedAddress = 0;
-  size_t physicalSlotSize = 0;
-};
-
-struct PlannedPhysicalSlot {
-  size_t id = std::numeric_limits<size_t>::max();
-  size_t requiredSize = 0;
-  size_t size = 0;
-  size_t address = 0;
-  llvm::SmallVector<size_t, 8> intervalIndices;
-};
-
-llvm::SmallVector<LocalAllocInterval, 0> buildLocalAllocIntervals(mlir::Operation* coreLikeOp,
-                                                                  std::optional<unsigned> lane);
-
-llvm::SmallVector<PlannedPhysicalSlot, 0> planPhysicalSlots(llvm::MutableArrayRef<LocalAllocInterval> intervals);
-
-MemoryPlanArtifacts buildMemoryPlanArtifacts(mlir::Operation* coreLikeOp,
-                                             std::optional<unsigned> lane,
-                                             llvm::ArrayRef<LocalAllocInterval> intervals,
-                                             llvm::ArrayRef<PlannedPhysicalSlot> slots,
-                                             size_t addressLimit,
-                                             PimMemoryReportLevel reportLevel);
-
-} // namespace onnx_mlir

src/PIM/Conversion/CMakeLists.txt

@@ -1,3 +1,2 @@
 add_subdirectory(ONNXToSpatial)
-add_subdirectory(SpatialToGraphviz)
 add_subdirectory(SpatialToPim)

src/PIM/Conversion/ONNXToSpatial/CMakeLists.txt

@@ -10,6 +10,7 @@ add_pim_library(OMONNXToSpatial
   Patterns/Post.cpp
   Patterns/GeneratedConversion.cpp
   Patterns/Math/Conv.cpp
+  Patterns/Math/ConvGeometry.cpp
   Patterns/Math/Elementwise.cpp
   Patterns/Math/Gemm.cpp
   Patterns/Math/MatMul.cpp
@@ -19,6 +20,7 @@ add_pim_library(OMONNXToSpatial
   Patterns/NN/Sigmoid.cpp
   Patterns/NN/Softmax.cpp
   Patterns/Tensor/Concat.cpp
+  Patterns/Tensor/Flatten.cpp
   Patterns/Tensor/Gather.cpp
   Patterns/Tensor/Resize.cpp
   Patterns/Tensor/Reshape.cpp
@@ -26,9 +28,13 @@ add_pim_library(OMONNXToSpatial
   Patterns/Tensor/Split.cpp
   Patterns/Tensor/Transpose.cpp
   ONNXToSpatialPass.cpp
+  SpatialLayoutPlanningPass.cpp
+  LowerSpatialPlansPass.cpp
   Common/AttributeUtils.cpp
+  Common/BiasAddUtils.cpp
   Common/ComputeRegionBuilder.cpp
-  Common/IndexingUtils.cpp
+  Common/MatrixProductLowering.cpp
+  Common/RowStripLayoutUtils.cpp
   Common/ShapeTilingUtils.cpp
   Common/WeightMaterialization.cpp
 

src/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.cpp

@@ -0,0 +1,112 @@
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinTypes.h"
+
+#include "llvm/ADT/SmallVector.h"
+
+#include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+
+LogicalResult isSupportedBiasAddShape(RankedTensorType biasType, RankedTensorType resultType) {
+  if (!biasType || !resultType || !biasType.hasStaticShape() || !resultType.hasStaticShape())
+    return failure();
+  if (resultType.getRank() != 4)
+    return failure();
+  if (biasType.getElementType() != resultType.getElementType())
+    return failure();
+
+  const int64_t channels = resultType.getDimSize(1);
+  ArrayRef<int64_t> shape = biasType.getShape();
+  if (shape.empty())
+    return success();
+  if (shape.size() == 1)
+    return success(shape[0] == channels);
+  if (shape.size() == 2)
+    return success(shape[0] == 1 && shape[1] == channels);
+  if (shape.size() == 4)
+    return success(shape[0] == 1 && shape[1] == channels && shape[2] == 1 && shape[3] == 1);
+  return failure();
+}
+
+FailureOr<SmallVector<Attribute>> getBiasChannelValues(DenseElementsAttr denseAttr, RankedTensorType resultType) {
+  auto biasType = dyn_cast<RankedTensorType>(denseAttr.getType());
+  if (!biasType || failed(isSupportedBiasAddShape(biasType, resultType)))
+    return failure();
+
+  const int64_t channels = resultType.getDimSize(1);
+  if (denseAttr.isSplat()) {
+    return SmallVector<Attribute>(channels, denseAttr.getSplatValue<Attribute>());
+  }
+
+  SmallVector<Attribute> flattened(denseAttr.getValues<Attribute>());
+  if (biasType.getRank() == 1)
+    return flattened;
+  if (biasType.getRank() == 2)
+    return flattened;
+
+  SmallVector<Attribute> channelValues;
+  channelValues.reserve(channels);
+  const int64_t channelStride = biasType.getDimSize(2) * biasType.getDimSize(3);
+  for (int64_t channel = 0; channel < channels; ++channel)
+    channelValues.push_back(flattened[channel * channelStride]);
+  return channelValues;
+}
+
+bool isSupportedBiasAddValue(Value bias, RankedTensorType resultType, DenseElementsAttr* denseAttr) {
+  auto attr = getHostConstDenseElementsAttr(bias);
+  if (!attr)
+    return false;
+  auto biasType = dyn_cast<RankedTensorType>(attr.getType());
+  if (!biasType || failed(isSupportedBiasAddShape(biasType, resultType)))
+    return false;
+  if (failed(getBiasChannelValues(attr, resultType)))
+    return false;
+  if (denseAttr)
+    *denseAttr = attr;
+  return true;
+}
+
+FailureOr<BiasAddPlanCandidate> classifyBiasAddPlanCandidate(Value lhs, Value rhs, RankedTensorType resultType) {
+  auto lhsType = dyn_cast<RankedTensorType>(lhs.getType());
+  auto rhsType = dyn_cast<RankedTensorType>(rhs.getType());
+  if (!lhsType || !rhsType)
+    return failure();
+  if (lhsType == resultType && isSupportedBiasAddValue(rhs, resultType))
+    return BiasAddPlanCandidate {lhs, rhs};
+  if (rhsType == resultType && isSupportedBiasAddValue(lhs, resultType))
+    return BiasAddPlanCandidate {rhs, lhs};
+  return failure();
+}
+
+FailureOr<Value>
+materializeDenseBiasAddTensor(Value bias, RankedTensorType resultType, RewriterBase& rewriter, Location loc) {
+  DenseElementsAttr denseAttr;
+  if (!isSupportedBiasAddValue(bias, resultType, &denseAttr))
+    return failure();
+
+  FailureOr<SmallVector<Attribute>> channelValues = getBiasChannelValues(denseAttr, resultType);
+  if (failed(channelValues))
+    return failure();
+
+  SmallVector<Attribute> resultValues;
+  resultValues.reserve(resultType.getNumElements());
+  const int64_t batches = resultType.getDimSize(0);
+  const int64_t channels = resultType.getDimSize(1);
+  const int64_t height = resultType.getDimSize(2);
+  const int64_t width = resultType.getDimSize(3);
+  for (int64_t n = 0; n < batches; ++n)
+    for (int64_t c = 0; c < channels; ++c)
+      for (int64_t h = 0; h < height; ++h)
+        for (int64_t w = 0; w < width; ++w)
+          resultValues.push_back((*channelValues)[c]);
+
+  auto resultAttr = DenseElementsAttr::get(resultType, resultValues);
+  return getOrCreateConstant(rewriter, rewriter.getInsertionBlock()->getParentOp(), resultAttr, resultType);
+}
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp

@@ -0,0 +1,30 @@
+#pragma once
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Support/LogicalResult.h"
+
+namespace onnx_mlir {
+
+struct BiasAddPlanCandidate {
+  mlir::Value data;
+  mlir::Value bias;
+};
+
+mlir::LogicalResult isSupportedBiasAddShape(mlir::RankedTensorType biasType, mlir::RankedTensorType resultType);
+bool isSupportedBiasAddValue(mlir::Value bias,
+                             mlir::RankedTensorType resultType,
+                             mlir::DenseElementsAttr* denseAttr = nullptr);
+mlir::FailureOr<llvm::SmallVector<mlir::Attribute>>
+getBiasChannelValues(mlir::DenseElementsAttr denseAttr, mlir::RankedTensorType resultType);
+mlir::FailureOr<BiasAddPlanCandidate> classifyBiasAddPlanCandidate(mlir::Value lhs,
+                                                                   mlir::Value rhs,
+                                                                   mlir::RankedTensorType resultType);
+mlir::FailureOr<mlir::Value> materializeDenseBiasAddTensor(mlir::Value bias,
+                                                           mlir::RankedTensorType resultType,
+                                                           mlir::RewriterBase& rewriter,
+                                                           mlir::Location loc);
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/Common.hpp

@@ -2,8 +2,9 @@
 
 #include "AttributeUtils.hpp"
 #include "ComputeRegionBuilder.hpp"
-#include "IndexingUtils.hpp"
+#include "MatrixProductLowering.hpp"
 #include "ShapeTilingUtils.hpp"
 #include "WeightMaterialization.hpp"
+#include "src/Accelerators/PIM/Common/IR/TensorSliceUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"

src/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.cpp

@@ -9,7 +9,54 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
-Value sumTensors(ArrayRef<Value> tensors, ConversionPatternRewriter& rewriter) {
+FailureOr<Value> createFragmentAssemblyBlueprint(Value physicalBatch,
+                                                 RankedTensorType logicalType,
+                                                 ArrayRef<FragmentAssemblyEntry> entries,
+                                                 StringRef physicalLayout,
+                                                 StringRef indexMap,
+                                                 PatternRewriter& rewriter,
+                                                 Location loc) {
+  auto physicalType = dyn_cast<RankedTensorType>(physicalBatch.getType());
+  if (!physicalType || !physicalType.hasStaticShape() || !logicalType || !logicalType.hasStaticShape()
+      || physicalType.getRank() != logicalType.getRank() + 1 || entries.empty())
+    return emitError(loc, "invalid static physical batch for fragment assembly"), failure();
+
+  const int64_t rank = logicalType.getRank();
+  const int64_t laneCount = physicalType.getDimSize(0);
+  if (laneCount <= 0)
+    return emitError(loc, "fragment assembly requires at least one physical source slot"), failure();
+  const int64_t fragmentElements = physicalType.getNumElements() / laneCount;
+  SmallVector<int64_t> operandIndices(entries.size(), 0), sourceSlots, sourceOffsets, offsets, sizes,
+    strides(entries.size() * rank, 1);
+  for (const FragmentAssemblyEntry& entry : entries) {
+    if (entry.sourceSlot < 0 || entry.sourceSlot >= laneCount || entry.sourceOffset < 0
+        || entry.destinationOffsets.size() != static_cast<size_t>(rank)
+        || entry.sizes.size() != static_cast<size_t>(rank))
+      return emitError(loc, "invalid fragment assembly entry"), failure();
+    int64_t entryElements = 1;
+    for (int64_t dim = 0; dim < rank; ++dim) {
+      if (entry.destinationOffsets[dim] < 0 || entry.sizes[dim] <= 0
+          || entry.destinationOffsets[dim] + entry.sizes[dim] > logicalType.getDimSize(dim))
+        return emitError(loc, "fragment assembly entry exceeds the logical tensor"), failure();
+      entryElements *= entry.sizes[dim];
+    }
+    if (entry.sourceOffset + entryElements > fragmentElements)
+      return emitError(loc, "fragment assembly entry exceeds its physical source slot"), failure();
+    sourceSlots.push_back(entry.sourceSlot);
+    sourceOffsets.push_back(entry.sourceOffset);
+    llvm::append_range(offsets, entry.destinationOffsets);
+    llvm::append_range(sizes, entry.sizes);
+  }
+  return spatial::SpatBlueprintOp::create(rewriter, loc, logicalType, physicalBatch, ValueRange {},
+    rewriter.getStringAttr("nchw"), rewriter.getStringAttr(physicalLayout),
+    rewriter.getDenseI64ArrayAttr(offsets), rewriter.getDenseI64ArrayAttr(sizes),
+    rewriter.getStringAttr(indexMap), rewriter.getStringAttr("fragment_assembly"),
+    rewriter.getDenseI64ArrayAttr(operandIndices), rewriter.getDenseI64ArrayAttr(sourceSlots),
+    rewriter.getDenseI64ArrayAttr(sourceOffsets), rewriter.getDenseI64ArrayAttr(strides),
+    rewriter.getStringAttr("disjoint"), rewriter.getStringAttr("complete")).getOutput();
+}
+
+Value sumTensors(ArrayRef<Value> tensors, PatternRewriter& rewriter) {
   if (tensors.size() == 1)
     return tensors[0];
 

src/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.hpp

@@ -13,11 +13,19 @@
 #include <utility>
 
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
+#include "src/Accelerators/PIM/Common/IR/TensorSliceUtils.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 
 namespace onnx_mlir {
 
+struct FragmentAssemblyEntry {
+  int64_t sourceSlot;
+  int64_t sourceOffset;
+  llvm::SmallVector<int64_t, 4> destinationOffsets;
+  llvm::SmallVector<int64_t, 4> sizes;
+};
+
 namespace detail {
 
 inline mlir::ValueRange getBlockArgs(mlir::Block* block) { return mlir::ValueRange(block->getArguments()); }
@@ -60,6 +68,56 @@ struct SpatComputeBatchBodyArgs {
   mlir::ValueRange outputs;
 };
 
+inline mlir::SmallVector<mlir::Type> getGraphComputeBlockArgTypes(mlir::ValueRange weights, mlir::ValueRange inputs) {
+  mlir::SmallVector<mlir::Type> blockArgTypes;
+  blockArgTypes.reserve(weights.size() + inputs.size());
+  for (mlir::Value weight : weights)
+    blockArgTypes.push_back(weight.getType());
+  for (mlir::Value input : inputs)
+    blockArgTypes.push_back(input.getType());
+  return blockArgTypes;
+}
+
+inline mlir::SmallVector<mlir::Location> getGraphComputeBlockArgLocs(mlir::Location defaultLoc,
+                                                                     mlir::ValueRange weights,
+                                                                     mlir::ValueRange inputs) {
+  mlir::SmallVector<mlir::Location> blockArgLocs;
+  blockArgLocs.reserve(weights.size() + inputs.size());
+  for (mlir::Value weight : weights)
+    blockArgLocs.push_back(weight.getLoc());
+  for (mlir::Value input : inputs)
+    blockArgLocs.push_back(input.getLoc());
+  return blockArgLocs;
+}
+
+inline mlir::SmallVector<mlir::Type> getGraphComputeBatchBlockArgTypes(mlir::OpBuilder& builder,
+                                                                       mlir::TypeRange resultTypes,
+                                                                       mlir::ValueRange weights,
+                                                                       mlir::ValueRange inputs) {
+  mlir::SmallVector<mlir::Type> blockArgTypes {builder.getIndexType()};
+  blockArgTypes.reserve(1 + weights.size() + inputs.size() + resultTypes.size());
+  for (mlir::Value weight : weights)
+    blockArgTypes.push_back(weight.getType());
+  for (mlir::Value input : inputs)
+    blockArgTypes.push_back(input.getType());
+  llvm::append_range(blockArgTypes, resultTypes);
+  return blockArgTypes;
+}
+
+inline mlir::SmallVector<mlir::Location> getGraphComputeBatchBlockArgLocs(mlir::Location defaultLoc,
+                                                                          mlir::TypeRange resultTypes,
+                                                                          mlir::ValueRange weights,
+                                                                          mlir::ValueRange inputs) {
+  mlir::SmallVector<mlir::Location> blockArgLocs {defaultLoc};
+  blockArgLocs.reserve(1 + weights.size() + inputs.size() + resultTypes.size());
+  for (mlir::Value weight : weights)
+    blockArgLocs.push_back(weight.getLoc());
+  for (mlir::Value input : inputs)
+    blockArgLocs.push_back(input.getLoc());
+  blockArgLocs.append(resultTypes.size(), defaultLoc);
+  return blockArgLocs;
+}
+
 } // namespace detail
 
 template <typename RewriterT>
@@ -87,26 +145,43 @@ inline mlir::Value createSpatConcat(RewriterT& rewriter, mlir::Location loc, int
   return spatial::SpatConcatOp::create(rewriter, loc, outputType, rewriter.getI64IntegerAttr(axis), inputs).getOutput();
 }
 
-/// Builds a `spat.compute` with a fixed number of SSA inputs and erases it if
+template <typename RewriterT>
+spatial::SpatGraphCompute createEmptySpatGraphCompute(RewriterT& rewriter,
+                                                      mlir::Location loc,
+                                                      mlir::TypeRange resultTypes,
+                                                      mlir::ValueRange weights,
+                                                      mlir::ValueRange inputs,
+                                                      mlir::TypeRange blockArgTypes,
+                                                      llvm::ArrayRef<mlir::Location> blockArgLocs) {
+  auto computeOp = spatial::SpatGraphCompute::create(rewriter, loc, resultTypes, weights, inputs);
+  rewriter.createBlock(&computeOp.getBody(), computeOp.getBody().end(), blockArgTypes, blockArgLocs);
+  rewriter.setInsertionPointToStart(&computeOp.getBody().front());
+  return computeOp;
+}
+
+template <typename RewriterT>
+spatial::SpatGraphCompute createEmptySpatGraphCompute(RewriterT& rewriter,
+                                                      mlir::Location loc,
+                                                      mlir::TypeRange resultTypes,
+                                                      mlir::ValueRange weights,
+                                                      mlir::ValueRange inputs) {
+  auto blockArgTypes = detail::getGraphComputeBlockArgTypes(weights, inputs);
+  auto blockArgLocs = detail::getGraphComputeBlockArgLocs(loc, weights, inputs);
+  return createEmptySpatGraphCompute(rewriter, loc, resultTypes, weights, inputs, blockArgTypes, blockArgLocs);
+}
+
+/// Builds a `spat.graph_compute` with a fixed number of SSA inputs and erases it if
 /// the body callback reports failure.
 template <size_t NumInputs, typename RewriterT, typename BodyFn>
-auto createSpatCompute(RewriterT& rewriter,
+auto createSpatGraphCompute(RewriterT& rewriter,
                             mlir::Location loc,
                             mlir::TypeRange resultTypes,
                             mlir::ValueRange weights,
                             mlir::ValueRange inputs,
                             BodyFn&& body) {
   assert(inputs.size() == NumInputs && "NumInputs must match the number of input values");
-  auto computeOp = spatial::SpatCompute::create(rewriter, loc, resultTypes, weights, inputs);
+  auto computeOp = createEmptySpatGraphCompute(rewriter, loc, resultTypes, weights, inputs);
-
+  auto* block = &computeOp.getBody().front();
-  auto* block = new mlir::Block();
-  for (mlir::Value weight : weights)
-    block->addArgument(weight.getType(), loc);
-  for (mlir::Value input : inputs)
-    block->addArgument(input.getType(), loc);
-
-  computeOp.getBody().push_back(block);
-  rewriter.setInsertionPointToStart(block);
 
   using BodyResult = detail::InvokeWithBlockArgsResultT<std::decay_t<BodyFn>, std::make_index_sequence<NumInputs>>;
   if constexpr (std::is_same_v<BodyResult, void>) {
@@ -124,32 +199,24 @@ auto createSpatCompute(RewriterT& rewriter,
     if (mlir::failed(bodyResult)) {
       rewriter.setInsertionPointAfter(computeOp);
       rewriter.eraseOp(computeOp);
-      return mlir::FailureOr<spatial::SpatCompute>(mlir::failure());
+      return mlir::FailureOr<spatial::SpatGraphCompute>(mlir::failure());
     }
     rewriter.setInsertionPointAfter(computeOp);
-    return mlir::FailureOr<spatial::SpatCompute>(computeOp);
+    return mlir::FailureOr<spatial::SpatGraphCompute>(computeOp);
   }
 }
 
-/// Builds a `spat.compute` whose body consumes the block arguments as a single
+/// Builds a `spat.graph_compute` whose body consumes the block arguments as a single
 /// `ValueRange`, which is convenient for variadic reductions/concats.
 template <typename RewriterT, typename BodyFn>
-auto createSpatCompute(RewriterT& rewriter,
+auto createSpatGraphCompute(RewriterT& rewriter,
                             mlir::Location loc,
                             mlir::TypeRange resultTypes,
                             mlir::ValueRange weights,
                             mlir::ValueRange inputs,
                             BodyFn&& body) {
-  auto computeOp = spatial::SpatCompute::create(rewriter, loc, resultTypes, weights, inputs);
+  auto computeOp = createEmptySpatGraphCompute(rewriter, loc, resultTypes, weights, inputs);
-
+  auto* block = &computeOp.getBody().front();
-  auto* block = new mlir::Block();
-  for (mlir::Value weight : weights)
-    block->addArgument(weight.getType(), loc);
-  for (mlir::Value input : inputs)
-    block->addArgument(input.getType(), loc);
-
-  computeOp.getBody().push_back(block);
-  rewriter.setInsertionPointToStart(block);
 
   using BodyResult = detail::InvokeWithValueRangeResultT<std::decay_t<BodyFn>>;
   if constexpr (std::is_same_v<BodyResult, void>) {
@@ -163,50 +230,60 @@ auto createSpatCompute(RewriterT& rewriter,
     if (mlir::failed(bodyResult)) {
       rewriter.setInsertionPointAfter(computeOp);
       rewriter.eraseOp(computeOp);
-      return mlir::FailureOr<spatial::SpatCompute>(mlir::failure());
+      return mlir::FailureOr<spatial::SpatGraphCompute>(mlir::failure());
     }
     rewriter.setInsertionPointAfter(computeOp);
-    return mlir::FailureOr<spatial::SpatCompute>(computeOp);
+    return mlir::FailureOr<spatial::SpatGraphCompute>(computeOp);
   }
 }
 
+template <typename RewriterT>
+auto createEmptySpatGraphComputeBatch(RewriterT& rewriter,
+                                      mlir::Location loc,
+                                      mlir::TypeRange resultTypes,
+                                      int64_t laneCount,
+                                      mlir::ValueRange weights,
+                                      mlir::ValueRange inputs,
+                                      mlir::TypeRange blockArgTypes,
+                                      llvm::ArrayRef<mlir::Location> blockArgLocs) {
+  if (laneCount <= 0 || laneCount > std::numeric_limits<int32_t>::max())
+    return mlir::FailureOr<spatial::SpatGraphComputeBatch>(mlir::failure());
+
+  auto laneCountAttr = pim::getCheckedI32Attr(rewriter, loc, laneCount, "spatial compute_batch lane count");
+  if (mlir::failed(laneCountAttr))
+    return mlir::FailureOr<spatial::SpatGraphComputeBatch>(mlir::failure());
+
+  auto batchOp = spatial::SpatGraphComputeBatch::create(rewriter, loc, resultTypes, *laneCountAttr, weights, inputs);
+  rewriter.createBlock(&batchOp.getBody(), batchOp.getBody().end(), blockArgTypes, blockArgLocs);
+  rewriter.setInsertionPointToStart(&batchOp.getBody().front());
+  return mlir::FailureOr<spatial::SpatGraphComputeBatch>(batchOp);
+}
+
+template <typename RewriterT>
+auto createEmptySpatGraphComputeBatch(RewriterT& rewriter,
+                                      mlir::Location loc,
+                                      mlir::TypeRange resultTypes,
+                                      int64_t laneCount,
+                                      mlir::ValueRange weights,
+                                      mlir::ValueRange inputs) {
+  auto blockArgTypes = detail::getGraphComputeBatchBlockArgTypes(rewriter, resultTypes, weights, inputs);
+  auto blockArgLocs = detail::getGraphComputeBatchBlockArgLocs(loc, resultTypes, weights, inputs);
+  return createEmptySpatGraphComputeBatch(
+    rewriter, loc, resultTypes, laneCount, weights, inputs, blockArgTypes, blockArgLocs);
+}
+
 template <typename RewriterT, typename BodyFn>
-auto createSpatComputeBatch(RewriterT& rewriter,
+auto createSpatGraphComputeBatch(RewriterT& rewriter,
                                  mlir::Location loc,
                                  mlir::TypeRange resultTypes,
                                  int64_t laneCount,
                                  mlir::ValueRange weights,
                                  mlir::ValueRange inputs,
                                  BodyFn&& body) {
-  if (laneCount <= 0 || laneCount > std::numeric_limits<int32_t>::max())
+  auto batchOp = createEmptySpatGraphComputeBatch(rewriter, loc, resultTypes, laneCount, weights, inputs);
-    return mlir::FailureOr<spatial::SpatComputeBatch>(mlir::failure());
+  if (failed(batchOp))
-
+    return mlir::FailureOr<spatial::SpatGraphComputeBatch>(mlir::failure());
-  auto laneCountAttr = pim::getCheckedI32Attr(rewriter, loc, laneCount, "spatial compute_batch lane count");
+  auto* block = &(*batchOp).getBody().front();
-  if (mlir::failed(laneCountAttr))
-    return mlir::FailureOr<spatial::SpatComputeBatch>(mlir::failure());
-
-  auto batchOp = spatial::SpatComputeBatch::create(rewriter, loc, resultTypes, *laneCountAttr, weights, inputs);
-
-  mlir::SmallVector<mlir::Type> blockArgTypes {rewriter.getIndexType()};
-  mlir::SmallVector<mlir::Location> blockArgLocs {loc};
-  blockArgTypes.reserve(1 + weights.size() + inputs.size() + resultTypes.size());
-  blockArgLocs.reserve(1 + weights.size() + inputs.size() + resultTypes.size());
-  for (mlir::Value weight : weights) {
-    blockArgTypes.push_back(weight.getType());
-    blockArgLocs.push_back(weight.getLoc());
-  }
-  for (mlir::Value input : inputs) {
-    blockArgTypes.push_back(input.getType());
-    blockArgLocs.push_back(input.getLoc());
-  }
-  for (mlir::Type resultType : resultTypes) {
-    blockArgTypes.push_back(resultType);
-    blockArgLocs.push_back(loc);
-  }
-
-  auto* block =
-    rewriter.createBlock(&batchOp.getBody(), batchOp.getBody().end(), mlir::TypeRange(blockArgTypes), blockArgLocs);
-  rewriter.setInsertionPointToStart(block);
 
   detail::SpatComputeBatchBodyArgs args {
     block->getArgument(0),
@@ -217,21 +294,54 @@ auto createSpatComputeBatch(RewriterT& rewriter,
   using BodyResult = std::invoke_result_t<BodyFn, detail::SpatComputeBatchBodyArgs>;
   if constexpr (std::is_same_v<BodyResult, void>) {
     std::forward<BodyFn>(body)(args);
-    rewriter.setInsertionPointAfter(batchOp);
+    rewriter.setInsertionPointAfter(*batchOp);
-    return mlir::FailureOr<spatial::SpatComputeBatch>(batchOp);
+    return batchOp;
   }
   else {
     auto bodyResult = std::forward<BodyFn>(body)(args);
     if (mlir::failed(bodyResult)) {
-      rewriter.setInsertionPointAfter(batchOp);
+      rewriter.setInsertionPointAfter(*batchOp);
-      rewriter.eraseOp(batchOp);
+      rewriter.eraseOp(*batchOp);
-      return mlir::FailureOr<spatial::SpatComputeBatch>(mlir::failure());
+      return mlir::FailureOr<spatial::SpatGraphComputeBatch>(mlir::failure());
     }
-    rewriter.setInsertionPointAfter(batchOp);
+    rewriter.setInsertionPointAfter(*batchOp);
-    return mlir::FailureOr<spatial::SpatComputeBatch>(batchOp);
+    return batchOp;
   }
 }
 
+template <size_t NumInputs, typename RewriterT, typename BodyFn>
+auto createSpatCompute(RewriterT& rewriter,
+                       mlir::Location loc,
+                       mlir::TypeRange resultTypes,
+                       mlir::ValueRange weights,
+                       mlir::ValueRange inputs,
+                       BodyFn&& body) {
+  return createSpatGraphCompute<NumInputs>(
+    rewriter, loc, resultTypes, weights, inputs, std::forward<BodyFn>(body));
+}
+
+template <typename RewriterT, typename BodyFn>
+auto createSpatCompute(RewriterT& rewriter,
+                       mlir::Location loc,
+                       mlir::TypeRange resultTypes,
+                       mlir::ValueRange weights,
+                       mlir::ValueRange inputs,
+                       BodyFn&& body) {
+  return createSpatGraphCompute(rewriter, loc, resultTypes, weights, inputs, std::forward<BodyFn>(body));
+}
+
+template <typename RewriterT, typename BodyFn>
+auto createSpatComputeBatch(RewriterT& rewriter,
+                            mlir::Location loc,
+                            mlir::TypeRange resultTypes,
+                            int64_t laneCount,
+                            mlir::ValueRange weights,
+                            mlir::ValueRange inputs,
+                            BodyFn&& body) {
+  return createSpatGraphComputeBatch(
+    rewriter, loc, resultTypes, laneCount, weights, inputs, std::forward<BodyFn>(body));
+}
+
 inline void createParallelInsertSliceIntoBatchOutput(mlir::PatternRewriter& rewriter,
                                                      mlir::Location loc,
                                                      mlir::Value source,
@@ -244,6 +354,46 @@ inline void createParallelInsertSliceIntoBatchOutput(mlir::PatternRewriter& rewr
   mlir::tensor::ParallelInsertSliceOp::create(rewriter, loc, source, dest, offsets, sizes, strides);
 }
 
+inline void publishGraphBatchPhysicalFragment(mlir::PatternRewriter& rewriter,
+                                              mlir::Location loc,
+                                              mlir::Value fragment,
+                                              mlir::Value output,
+                                              mlir::Value physicalSlot) {
+  auto fragmentType = mlir::cast<mlir::RankedTensorType>(fragment.getType());
+  mlir::SmallVector<mlir::OpFoldResult> offsets {physicalSlot};
+  mlir::SmallVector<mlir::OpFoldResult> sizes {rewriter.getIndexAttr(1)};
+  mlir::SmallVector<mlir::OpFoldResult> strides {rewriter.getIndexAttr(1)};
+  for (int64_t dim : fragmentType.getShape()) {
+    offsets.push_back(rewriter.getIndexAttr(0));
+    sizes.push_back(rewriter.getIndexAttr(dim));
+    strides.push_back(rewriter.getIndexAttr(1));
+  }
+  createParallelInsertSliceIntoBatchOutput(rewriter, loc, fragment, output, offsets, sizes, strides);
+}
+
+inline mlir::FailureOr<mlir::Value>
+extractGraphBatchPhysicalFragment(mlir::PatternRewriter& rewriter,
+                                  mlir::Location loc,
+                                  mlir::Value physicalBatch,
+                                  mlir::OpFoldResult slot,
+                                  mlir::RankedTensorType fragmentType) {
+  if (fragmentType.getRank() == 0)
+    return mlir::failure();
+  auto physicalType = mlir::dyn_cast<mlir::RankedTensorType>(physicalBatch.getType());
+  if (!physicalType || physicalType.getRank() != fragmentType.getRank() + 1)
+    return mlir::failure();
+  mlir::SmallVector<mlir::OpFoldResult> offsets {slot};
+  mlir::SmallVector<mlir::OpFoldResult> sizes {rewriter.getIndexAttr(1)};
+  mlir::SmallVector<mlir::OpFoldResult> strides {rewriter.getIndexAttr(1)};
+  for (int64_t dim : fragmentType.getShape()) {
+    offsets.push_back(rewriter.getIndexAttr(0));
+    sizes.push_back(rewriter.getIndexAttr(dim));
+    strides.push_back(rewriter.getIndexAttr(1));
+  }
+  return extractMixedSliceOrIdentity(
+    rewriter, loc, physicalBatch, fragmentType, {offsets, sizes, strides});
+}
+
 template <typename BodyFn>
 mlir::Value materializeOrComputeUnary(mlir::Value input,
                                       mlir::RankedTensorType resultType,
@@ -262,6 +412,14 @@ mlir::Value materializeOrComputeUnary(mlir::Value input,
   return computeOp.getResult(0);
 }
 
-mlir::Value sumTensors(mlir::ArrayRef<mlir::Value> tensors, mlir::ConversionPatternRewriter& rewriter);
+mlir::Value sumTensors(mlir::ArrayRef<mlir::Value> tensors, mlir::PatternRewriter& rewriter);
+
+mlir::FailureOr<mlir::Value> createFragmentAssemblyBlueprint(mlir::Value physicalBatch,
+                                                             mlir::RankedTensorType logicalType,
+                                                             llvm::ArrayRef<FragmentAssemblyEntry> entries,
+                                                             llvm::StringRef physicalLayout,
+                                                             llvm::StringRef indexMap,
+                                                             mlir::PatternRewriter& rewriter,
+                                                             mlir::Location loc);
 
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/MatrixProductLowering.cpp

@@ -0,0 +1,73 @@
+#include "MatrixProductLowering.hpp"
+
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+
+Value createZeroPaddedTensor(Value value, RankedTensorType resultType, PatternRewriter& rewriter, Location loc) {
+  auto sourceType = cast<RankedTensorType>(value.getType());
+  SmallVector<OpFoldResult> lowPads(sourceType.getRank(), rewriter.getIndexAttr(0));
+  SmallVector<OpFoldResult> highPads;
+  highPads.reserve(sourceType.getRank());
+  for (auto [sourceDim, resultDim] : llvm::zip(sourceType.getShape(), resultType.getShape()))
+    highPads.push_back(rewriter.getIndexAttr(resultDim - sourceDim));
+
+  auto padOp = tensor::PadOp::create(rewriter, loc, resultType, value, lowPads, highPads);
+  auto* padBlock = new Block();
+  for (int64_t i = 0; i < sourceType.getRank(); ++i)
+    padBlock->addArgument(rewriter.getIndexType(), loc);
+  padOp.getRegion().push_back(padBlock);
+  rewriter.setInsertionPointToStart(padBlock);
+  auto zero = getOrCreateConstant(
+    rewriter, padOp.getOperation(), rewriter.getZeroAttr(sourceType.getElementType()), sourceType.getElementType());
+  tensor::YieldOp::create(rewriter, loc, zero);
+  rewriter.setInsertionPointAfter(padOp);
+  return padOp.getResult();
+}
+
+Value createPaddedInputCompute(Value input,
+                               RankedTensorType paddedInputType,
+                               PatternRewriter& rewriter,
+                               Location loc) {
+  auto inputType = cast<RankedTensorType>(input.getType());
+  if (inputType == paddedInputType)
+    return input;
+
+  auto producer = inputType.getRank() == 2 && paddedInputType.getRank() == 2
+    ? input.getDefiningOp<spatial::SpatGraphComputeBatch>()
+    : spatial::SpatGraphComputeBatch();
+  auto inputFragmentType = producer
+    ? spatial::getGraphBatchFragmentType(inputType, producer.getLaneCount())
+    : FailureOr<RankedTensorType>(failure());
+  auto paddedFragmentType = producer
+    ? spatial::getGraphBatchFragmentType(paddedInputType, producer.getLaneCount())
+    : FailureOr<RankedTensorType>(failure());
+  if (producer && succeeded(inputFragmentType) && succeeded(paddedFragmentType)) {
+    auto batch = createSpatComputeBatch(rewriter, loc, TypeRange {paddedInputType}, producer.getLaneCount(), {}, input,
+      [&](detail::SpatComputeBatchBodyArgs args) -> LogicalResult {
+        auto fragment = extractGraphBatchPhysicalFragment(
+          rewriter, loc, args.inputs.front(), args.lane, *inputFragmentType);
+        if (failed(fragment))
+          return failure();
+        Value padded = createZeroPaddedTensor(*fragment, *paddedFragmentType, rewriter, loc);
+        publishGraphBatchPhysicalFragment(rewriter, loc, padded, args.outputs.front(), args.lane);
+        return success();
+      });
+    if (succeeded(batch))
+      return batch->getResult(0);
+  }
+
+  auto computeOp = createSpatCompute<1>(rewriter, loc, TypeRange {paddedInputType}, {}, input, [&](Value computeInput) {
+    Value paddedInput = createZeroPaddedTensor(computeInput, paddedInputType, rewriter, loc);
+    spatial::SpatYieldOp::create(rewriter, loc, paddedInput);
+  });
+  return computeOp.getResult(0);
+}
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/MatrixProductLowering.hpp

@@ -0,0 +1,20 @@
+#pragma once
+
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Location.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Transforms/DialectConversion.h"
+
+namespace onnx_mlir {
+
+mlir::Value createZeroPaddedTensor(mlir::Value value,
+                                   mlir::RankedTensorType resultType,
+                                   mlir::PatternRewriter& rewriter,
+                                   mlir::Location loc);
+
+mlir::Value createPaddedInputCompute(mlir::Value input,
+                                     mlir::RankedTensorType paddedInputType,
+                                     mlir::PatternRewriter& rewriter,
+                                     mlir::Location loc);
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.cpp

@@ -0,0 +1,231 @@
+#include "llvm/ADT/SmallVector.h"
+
+#include "src/Accelerators/PIM/Common/IR/AffineUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+#include "src/Dialect/ONNX/ONNXOps.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+
+FailureOr<RowStripPhysicalValue> describeRowStripPhysicalValue(Value storage, RankedTensorType logicalType) {
+  auto storageType = dyn_cast<RankedTensorType>(storage.getType());
+  if (!storageType || !storageType.hasStaticShape() || !logicalType || !logicalType.hasStaticShape()
+      || storageType.getRank() != 5 || logicalType.getRank() != 4 || logicalType.getDimSize(0) != 1
+      || storageType.getElementType() != logicalType.getElementType()
+      || storageType.getDimSize(1) != 1 || storageType.getDimSize(2) <= 0
+      || storageType.getDimSize(3) != 1 || storageType.getDimSize(4) != logicalType.getDimSize(3))
+    return failure();
+  const int64_t tilesPerRow = ceilIntegerDivide(logicalType.getDimSize(1), storageType.getDimSize(2));
+  if (storageType.getDimSize(0) != logicalType.getDimSize(2) * tilesPerRow)
+    return failure();
+  return RowStripPhysicalValue {storage, logicalType,
+    RankedTensorType::get(storageType.getShape().drop_front(), storageType.getElementType(), storageType.getEncoding()),
+    tilesPerRow};
+}
+
+RankedTensorType getRowStripFragmentType(RankedTensorType logicalType) {
+  return RankedTensorType::get({logicalType.getDimSize(0), logicalType.getDimSize(1), 1, logicalType.getDimSize(3)},
+                               logicalType.getElementType(),
+                               logicalType.getEncoding());
+}
+
+RankedTensorType getRowStripStorageType(RankedTensorType logicalType) {
+  return spatial::getGraphBatchPhysicalResultType(logicalType.getDimSize(2), getRowStripFragmentType(logicalType));
+}
+
+std::pair<SmallVector<int64_t>, SmallVector<int64_t>> buildRowStripMetadata(RankedTensorType type) {
+  SmallVector<int64_t> offsets;
+  SmallVector<int64_t> sizes;
+  const int64_t channels = type.getDimSize(1);
+  const int64_t height = type.getDimSize(2);
+  const int64_t width = type.getDimSize(3);
+  offsets.reserve(height * 4);
+  sizes.reserve(height * 4);
+  for (int64_t row = 0; row < height; ++row) {
+    offsets.append({0, 0, row, 0});
+    sizes.append({1, channels, 1, width});
+  }
+  return {offsets, sizes};
+}
+
+Value extractRowStripFragment(Value storage,
+                              RankedTensorType logicalType,
+                              OpFoldResult row,
+                              PatternRewriter& rewriter,
+                              Location loc) {
+  return *extractGraphBatchPhysicalFragment(rewriter, loc, storage, row, getRowStripFragmentType(logicalType));
+}
+
+void insertRowStripFragment(Value fragment,
+                            Value output,
+                            RankedTensorType logicalType,
+                            OpFoldResult row,
+                            PatternRewriter& rewriter,
+                            Location loc) {
+  assert(fragment.getType() == getRowStripFragmentType(logicalType));
+  assert(output.getType() == getRowStripStorageType(logicalType));
+  auto slot = dyn_cast<Value>(row);
+  assert(slot && "row-strip graph publication requires a dynamic physical slot");
+  publishGraphBatchPhysicalFragment(rewriter, loc, fragment, output, slot);
+}
+
+FailureOr<Value> createPerChannelConstantFragment(DenseElementsAttr denseAttr,
+                                                  RankedTensorType fragmentType,
+                                                  PatternRewriter& rewriter) {
+  FailureOr<SmallVector<Attribute>> channelValues = getBiasChannelValues(denseAttr, fragmentType);
+  if (failed(channelValues))
+    return failure();
+
+  SmallVector<Attribute> values;
+  values.reserve(fragmentType.getNumElements());
+  for (int64_t n = 0; n < fragmentType.getDimSize(0); ++n)
+    for (int64_t channel = 0; channel < fragmentType.getDimSize(1); ++channel)
+      for (int64_t h = 0; h < fragmentType.getDimSize(2); ++h)
+        for (int64_t w = 0; w < fragmentType.getDimSize(3); ++w)
+          values.push_back((*channelValues)[channel]);
+
+  auto attr = DenseElementsAttr::get(fragmentType, values);
+  return getOrCreateConstant(rewriter, rewriter.getInsertionBlock()->getParentOp(), attr, fragmentType);
+}
+
+FailureOr<Value> createRowStripStorageFromRows(Value rows,
+                                               RankedTensorType logicalType,
+                                               PatternRewriter& rewriter,
+                                               Location loc) {
+  auto rowsType = dyn_cast<RankedTensorType>(rows.getType());
+  if (!rowsType || !rowsType.hasStaticShape() || rowsType.getRank() != 2)
+    return failure();
+  if (!logicalType || !logicalType.hasStaticShape() || logicalType.getRank() != 4)
+    return failure();
+  if (logicalType.getDimSize(0) != 1)
+    return failure();
+  if (rowsType.getElementType() != logicalType.getElementType())
+    return failure();
+
+  const int64_t channels = logicalType.getDimSize(1);
+  const int64_t height = logicalType.getDimSize(2);
+  const int64_t width = logicalType.getDimSize(3);
+  if (rowsType.getDimSize(0) != height * width)
+    return failure();
+  if (rowsType.getDimSize(1) != channels)
+    return failure();
+
+  auto rowSliceType = RankedTensorType::get({width, channels}, logicalType.getElementType(), rowsType.getEncoding());
+  auto channelWidthType = RankedTensorType::get({channels, width}, logicalType.getElementType(), rowsType.getEncoding());
+  auto fragmentType = getRowStripFragmentType(logicalType);
+  auto storageType = getRowStripStorageType(logicalType);
+  auto batchOp = createSpatComputeBatch(
+    rewriter, loc, TypeRange {storageType}, height, {}, ValueRange {rows}, [&](detail::SpatComputeBatchBodyArgs args) {
+      Operation* anchorOp = rewriter.getInsertionBlock()->getParentOp();
+      Value rowStart = affineMulConst(rewriter, loc, args.lane, width, anchorOp);
+      SmallVector<OpFoldResult> rowOffsets {rowStart, rewriter.getIndexAttr(0)};
+      SmallVector<OpFoldResult> rowSizes {rewriter.getIndexAttr(width), rewriter.getIndexAttr(channels)};
+      Value rowSlice = tensor::ExtractSliceOp::create(
+        rewriter, loc, rowSliceType, args.inputs.front(), rowOffsets, rowSizes, getUnitStrides(rewriter, 2));
+      Value channelWidth = ONNXTransposeOp::create(
+        rewriter, loc, channelWidthType, rowSlice, rewriter.getI64ArrayAttr({1, 0})).getResult();
+      Value fragment = tensor::ExpandShapeOp::create(
+        rewriter, loc, fragmentType, channelWidth, SmallVector<ReassociationIndices> {{0, 1}, {2, 3}});
+      insertRowStripFragment(fragment, args.outputs.front(), logicalType, args.lane, rewriter, loc);
+      return success();
+    });
+  if (failed(batchOp))
+    return failure();
+  return batchOp->getResult(0);
+}
+
+FailureOr<Value> createRowStripAssemblyBlueprint(const RowStripPhysicalValue& value,
+                                                 PatternRewriter& rewriter,
+                                                 Location loc) {
+  SmallVector<FragmentAssemblyEntry> entries;
+  const int64_t tileChannels = value.fragmentType.getDimSize(1);
+  for (int64_t row = 0; row < value.logicalType.getDimSize(2); ++row)
+    for (int64_t tile = 0; tile < value.tilesPerRow; ++tile) {
+      const int64_t channelOffset = tile * tileChannels;
+      entries.push_back({row * value.tilesPerRow + tile, 0, {0, channelOffset, row, 0},
+        {1, std::min(tileChannels, value.logicalType.getDimSize(1) - channelOffset), 1,
+         value.logicalType.getDimSize(3)}});
+    }
+  return createFragmentAssemblyBlueprint(value.storage, value.logicalType, entries, "nchw_row_strip",
+                                         kRowStripIndexMap, rewriter, loc);
+}
+
+FailureOr<Value> applyRowStripRelu(const RowStripPhysicalValue& value, PatternRewriter& rewriter, Location loc) {
+  auto storageType = cast<RankedTensorType>(value.storage.getType());
+  const int64_t laneCount = storageType.getDimSize(0);
+  auto batchOp = createSpatComputeBatch(rewriter,
+                                        loc,
+                                        TypeRange {storageType},
+                                        laneCount,
+                                        {},
+                                        ValueRange {value.storage},
+                                        [&](detail::SpatComputeBatchBodyArgs args) {
+                                          FailureOr<Value> fragment = extractGraphBatchPhysicalFragment(
+                                            rewriter, loc, args.inputs.front(), args.lane, value.fragmentType);
+                                          if (failed(fragment)) return failure();
+                                          Value relu = spatial::SpatReluOp::create(
+                                            rewriter, loc, value.fragmentType, *fragment).getResult();
+                                          publishGraphBatchPhysicalFragment(
+                                            rewriter, loc, relu, args.outputs.front(), args.lane);
+                                          return success();
+                                        });
+  if (failed(batchOp))
+    return failure();
+  return batchOp->getResult(0);
+}
+
+FailureOr<Value> applyRowStripBiasAdd(const RowStripPhysicalValue& value,
+                                     Value bias,
+                                     PatternRewriter& rewriter,
+                                     Location loc) {
+  DenseElementsAttr denseAttr;
+  if (!isSupportedBiasAddValue(bias, value.logicalType, &denseAttr))
+    return failure();
+  FailureOr<SmallVector<Attribute>> channelValues = getBiasChannelValues(denseAttr, value.logicalType);
+  if (failed(channelValues)) return failure();
+  auto storageType = cast<RankedTensorType>(value.storage.getType());
+  auto biasStorageType = spatial::getGraphBatchPhysicalResultType(value.tilesPerRow, value.fragmentType);
+  SmallVector<Attribute> biasValues(
+    biasStorageType.getNumElements(), cast<Attribute>(rewriter.getZeroAttr(value.fragmentType.getElementType())));
+  const int64_t tileChannels = value.fragmentType.getDimSize(1);
+  const int64_t width = value.fragmentType.getDimSize(3);
+  for (int64_t channel = 0; channel < value.logicalType.getDimSize(1); ++channel)
+    for (int64_t w = 0; w < width; ++w)
+      biasValues[((channel / tileChannels) * tileChannels + channel % tileChannels) * width + w] =
+        (*channelValues)[channel];
+  Value biasStorage = getOrCreateConstant(rewriter, rewriter.getInsertionBlock()->getParentOp(),
+    DenseElementsAttr::get(biasStorageType, biasValues), biasStorageType);
+  const int64_t laneCount = storageType.getDimSize(0);
+  auto batchOp = createSpatComputeBatch(rewriter,
+                                        loc,
+                                        TypeRange {storageType},
+                                        laneCount,
+                                        {},
+                                        ValueRange {value.storage, biasStorage},
+                                        [&](detail::SpatComputeBatchBodyArgs args) {
+                                          Operation* anchorOp = rewriter.getInsertionBlock()->getParentOp();
+                                          FailureOr<Value> fragment = extractGraphBatchPhysicalFragment(
+                                            rewriter, loc, args.inputs[0], args.lane, value.fragmentType);
+                                          Value tile = affineModConst(rewriter, loc, args.lane,
+                                            value.tilesPerRow, anchorOp);
+                                          FailureOr<Value> constant = extractGraphBatchPhysicalFragment(
+                                            rewriter, loc, args.inputs[1], tile, value.fragmentType);
+                                          if (failed(fragment) || failed(constant)) return failure();
+                                          Value added = spatial::SpatVAddOp::create(
+                                            rewriter, loc, value.fragmentType, *fragment, *constant).getResult();
+                                          publishGraphBatchPhysicalFragment(
+                                            rewriter, loc, added, args.outputs.front(), args.lane);
+                                          return success();
+                                        });
+  if (failed(batchOp))
+    return failure();
+  return batchOp->getResult(0);
+}
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.hpp

@@ -0,0 +1,69 @@
+#pragma once
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/PatternMatch.h"
+
+namespace onnx_mlir {
+
+inline constexpr llvm::StringLiteral kRowStripIndexMap = "nchw_row_strip_fragments";
+
+struct RowStripPhysicalValue {
+  mlir::Value storage;
+  mlir::RankedTensorType logicalType;
+  mlir::RankedTensorType fragmentType;
+  int64_t tilesPerRow;
+};
+
+mlir::FailureOr<RowStripPhysicalValue> describeRowStripPhysicalValue(mlir::Value storage,
+                                                                     mlir::RankedTensorType logicalType);
+
+std::pair<llvm::SmallVector<int64_t>, llvm::SmallVector<int64_t>>
+buildRowStripMetadata(mlir::RankedTensorType type);
+
+mlir::RankedTensorType getRowStripFragmentType(mlir::RankedTensorType logicalType);
+
+mlir::RankedTensorType getRowStripStorageType(mlir::RankedTensorType logicalType);
+
+llvm::SmallVector<mlir::OpFoldResult> buildRowStripFragmentOffsets(mlir::PatternRewriter& rewriter,
+                                                                   mlir::OpFoldResult row);
+
+llvm::SmallVector<mlir::OpFoldResult> buildRowStripFragmentSizes(mlir::PatternRewriter& rewriter,
+                                                                 mlir::RankedTensorType logicalType);
+
+mlir::Value extractRowStripFragment(mlir::Value storage,
+                                    mlir::RankedTensorType logicalType,
+                                    mlir::OpFoldResult row,
+                                    mlir::PatternRewriter& rewriter,
+                                    mlir::Location loc);
+
+void insertRowStripFragment(mlir::Value fragment,
+                            mlir::Value output,
+                            mlir::RankedTensorType logicalType,
+                            mlir::OpFoldResult row,
+                            mlir::PatternRewriter& rewriter,
+                            mlir::Location loc);
+
+mlir::FailureOr<mlir::Value> createPerChannelConstantFragment(mlir::DenseElementsAttr denseAttr,
+                                                              mlir::RankedTensorType fragmentType,
+                                                              mlir::PatternRewriter& rewriter);
+
+mlir::FailureOr<mlir::Value> createRowStripStorageFromRows(mlir::Value rows,
+                                                           mlir::RankedTensorType logicalType,
+                                                           mlir::PatternRewriter& rewriter,
+                                                           mlir::Location loc);
+
+mlir::FailureOr<mlir::Value> createRowStripAssemblyBlueprint(const RowStripPhysicalValue& value,
+                                                               mlir::PatternRewriter& rewriter,
+                                                               mlir::Location loc);
+
+mlir::FailureOr<mlir::Value> applyRowStripRelu(const RowStripPhysicalValue& value,
+                                               mlir::PatternRewriter& rewriter,
+                                               mlir::Location loc);
+
+mlir::FailureOr<mlir::Value> applyRowStripBiasAdd(const RowStripPhysicalValue& value,
+                                                  mlir::Value bias,
+                                                  mlir::PatternRewriter& rewriter,
+                                                  mlir::Location loc);
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/ShapeTilingUtils.cpp

@@ -3,9 +3,6 @@
 
 #include "llvm/ADT/SmallVector.h"
 
-#include <functional>
-
-#include "IndexingUtils.hpp"
 #include "ShapeTilingUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
 #include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
@@ -15,75 +12,8 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
-bool hasStaticPositiveShape(ArrayRef<int64_t> shape) {
-  return llvm::all_of(shape, [](int64_t dim) { return dim > 0; });
-}
-
-bool hasStaticPositiveShape(RankedTensorType type) {
-  return type.hasStaticShape() && hasStaticPositiveShape(type.getShape());
-}
-
-int64_t getStaticShapeElementCount(ArrayRef<int64_t> shape) {
-  return std::accumulate(shape.begin(), shape.end(), int64_t {1}, std::multiplies<int64_t> {});
-}
-
-SmallVector<int64_t> permuteShape(ArrayRef<int64_t> shape, ArrayRef<int64_t> permutation) {
-  SmallVector<int64_t> permutedShape;
-  permutedShape.reserve(permutation.size());
-  for (int64_t axis : permutation)
-    permutedShape.push_back(shape[axis]);
-  return permutedShape;
-}
-
-SmallVector<int64_t> invertPermutation(ArrayRef<int64_t> permutation) {
-  SmallVector<int64_t> inversePermutation(permutation.size());
-  for (auto [newIndex, oldIndex] : llvm::enumerate(permutation))
-    inversePermutation[oldIndex] = static_cast<int64_t>(newIndex);
-  return inversePermutation;
-}
-
-FailureOr<SmallVector<int64_t>> getTransposePermutationChecked(std::optional<ArrayAttr> permAttr, int64_t rank) {
-  SmallVector<int64_t> permutation;
-  if (!permAttr) {
-    permutation.reserve(rank);
-    for (int64_t dim = rank - 1; dim >= 0; --dim)
-      permutation.push_back(dim);
-    return permutation;
-  }
-
-  if (static_cast<int64_t>(permAttr->size()) != rank)
-    return failure();
-
-  permutation.reserve(permAttr->size());
-  SmallVector<bool> seen(rank, false);
-  for (IntegerAttr attr : permAttr->getAsRange<IntegerAttr>()) {
-    int64_t axis = attr.getInt();
-    if (axis < 0 || axis >= rank || seen[axis])
-      return failure();
-    seen[axis] = true;
-    permutation.push_back(axis);
-  }
-  return permutation;
-}
-
-SmallVector<OpFoldResult> getUnitStrides(PatternRewriter& rewriter, int64_t rank) {
-  return SmallVector<OpFoldResult>(rank, rewriter.getIndexAttr(1));
-}
-
-SmallVector<OpFoldResult> getZeroOffsets(PatternRewriter& rewriter, int64_t rank) {
-  return SmallVector<OpFoldResult>(rank, rewriter.getIndexAttr(0));
-}
-
-SmallVector<OpFoldResult> getStaticSizes(PatternRewriter& rewriter, ArrayRef<int64_t> shape) {
-  SmallVector<OpFoldResult> sizes;
-  sizes.reserve(shape.size());
-  for (int64_t dim : shape)
-    sizes.push_back(rewriter.getIndexAttr(dim));
-  return sizes;
-}
-
 SmallVector<Value> sliceTensor(
-  const Value& tensorToSlice, size_t axis, int64_t sliceSize, ConversionPatternRewriter& rewriter, Location loc) {
+  const Value& tensorToSlice, size_t axis, int64_t sliceSize, PatternRewriter& rewriter, Location loc) {
   ArrayRef<long> shape = getTensorShape(tensorToSlice);
   assert("Invalid axis" && axis < shape.size());
 
@@ -129,7 +59,7 @@ SmallVector<Value> sliceTensor(
 }
 
 SmallVector<Value>
-sliceVector(const Value& vectorToSlice, int64_t sliceSize, ConversionPatternRewriter& rewriter, Location loc) {
+sliceVector(const Value& vectorToSlice, int64_t sliceSize, PatternRewriter& rewriter, Location loc) {
   ArrayRef<long> shape = getTensorShape(vectorToSlice);
   assert("Not a vector" && isVectorShape(shape));
   size_t axis = shape[0] != 1 ? 0 : 1;
@@ -137,7 +67,7 @@ sliceVector(const Value& vectorToSlice, int64_t sliceSize, ConversionPatternRewr
 }
 
 DenseMap<CoreId, SmallVector<Value>>
-sliceVectorPerCrossbarPerCore(const Value& vectorToSlice, ConversionPatternRewriter& rewriter, Location loc) {
+sliceVectorPerCrossbarPerCore(const Value& vectorToSlice, PatternRewriter& rewriter, Location loc) {
   SmallVector<Value> slices = sliceVector(vectorToSlice, crossbarSize, rewriter, loc);
   DenseMap<CoreId, SmallVector<Value>> slicesPerCore;
   for (size_t sliceId = 0; sliceId < slices.size(); sliceId++) {
@@ -147,33 +77,4 @@ sliceVectorPerCrossbarPerCore(const Value& vectorToSlice, ConversionPatternRewri
   return slicesPerCore;
 }
 
-Value extractAxisSlice(
-  PatternRewriter& rewriter, Location loc, Value source, int64_t axis, int64_t offset, int64_t size) {
-  auto sourceType = cast<RankedTensorType>(source.getType());
-  SmallVector<int64_t> resultShape(sourceType.getShape());
-  resultShape[axis] = size;
-  auto resultType = RankedTensorType::get(resultShape, sourceType.getElementType(), sourceType.getEncoding());
-
-  SmallVector<OpFoldResult> offsets = getZeroOffsets(rewriter, sourceType.getRank());
-  SmallVector<OpFoldResult> sizes = getStaticSizes(rewriter, sourceType.getShape());
-  offsets[axis] = rewriter.getIndexAttr(offset);
-  sizes[axis] = rewriter.getIndexAttr(size);
-  return tensor::ExtractSliceOp::create(
-           rewriter, loc, resultType, source, offsets, sizes, getUnitStrides(rewriter, sourceType.getRank()))
-    .getResult();
-}
-
-Value insertStaticSlice(
-  PatternRewriter& rewriter, Location loc, Value source, Value dest, ArrayRef<OpFoldResult> offsets) {
-  auto sourceType = cast<RankedTensorType>(source.getType());
-  return tensor::InsertSliceOp::create(rewriter,
-                                       loc,
-                                       source,
-                                       dest,
-                                       offsets,
-                                       getStaticSizes(rewriter, sourceType.getShape()),
-                                       getUnitStrides(rewriter, sourceType.getRank()))
-    .getResult();
-}
-
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/ShapeTilingUtils.hpp

@@ -1,114 +1,31 @@
 #pragma once
 
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
-#include "mlir/IR/BuiltinTypes.h"
-#include "mlir/IR/Value.h"
 #include "mlir/IR/ValueRange.h"
 #include "mlir/Transforms/DialectConversion.h"
 
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/SmallVector.h"
 
-#include <cassert>
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
-#include <cstddef>
-#include <optional>
-#include <type_traits>
-#include <utility>
 
 namespace onnx_mlir {
 
-using HSliceId = size_t;
-using CoreId = size_t;
-
-template <class A, class B, class C = std::common_type_t<A, B>>
-constexpr C ceilIntegerDivide(A a, B b) {
-  static_assert(std::is_integral_v<A>, "A must be an integer type");
-  static_assert(std::is_integral_v<B>, "B must be an integer type");
-  C ac = static_cast<C>(a);
-  C bc = static_cast<C>(b);
-  return 1 + (ac - 1) / bc;
-}
-
-template <class A, class B, class C = std::common_type_t<A, B>>
-constexpr std::pair<C, C> ceilIntegerDivideWithRemainder(A a, B b) {
-  static_assert(std::is_integral_v<A>, "A must be an integer type");
-  static_assert(std::is_integral_v<B>, "B must be an integer type");
-  C ac = static_cast<C>(a);
-  C bc = static_cast<C>(b);
-  return {ceilIntegerDivide(ac, bc), ac % bc};
-}
-
-template <class T>
-bool isVectorShape(mlir::ArrayRef<T> shape) {
-  return shape.size() == 2 && (shape[0] == 1 || shape[1] == 1);
-}
-
-template <class T>
-bool isMatrixShape(mlir::ArrayRef<T> shape) {
-  return shape.size() == 2;
-}
-
-template <class T>
-bool isHVectorShape(mlir::ArrayRef<T> shape) {
-  return shape.size() == 2 && shape[0] == 1;
-}
-
-inline auto getTensorShape(mlir::Value tensor) {
-  return mlir::cast<mlir::RankedTensorType>(tensor.getType()).getShape();
-}
-
-inline bool haveSameStaticShape(mlir::Value lhs, mlir::Value rhs) {
-  auto lhsType = mlir::dyn_cast<mlir::RankedTensorType>(lhs.getType());
-  auto rhsType = mlir::dyn_cast<mlir::RankedTensorType>(rhs.getType());
-  return lhsType && rhsType && lhsType.hasStaticShape() && rhsType.hasStaticShape()
-      && lhsType.getShape() == rhsType.getShape();
-}
-
-bool hasStaticPositiveShape(mlir::ArrayRef<int64_t> shape);
-
-bool hasStaticPositiveShape(mlir::RankedTensorType type);
-
-int64_t getStaticShapeElementCount(mlir::ArrayRef<int64_t> shape);
-
-llvm::SmallVector<int64_t> permuteShape(mlir::ArrayRef<int64_t> shape, mlir::ArrayRef<int64_t> permutation);
-
-llvm::SmallVector<int64_t> invertPermutation(mlir::ArrayRef<int64_t> permutation);
-
-mlir::FailureOr<llvm::SmallVector<int64_t>> getTransposePermutationChecked(std::optional<mlir::ArrayAttr> permAttr,
-                                                                           int64_t rank);
-
-llvm::SmallVector<mlir::OpFoldResult> getUnitStrides(mlir::PatternRewriter& rewriter, int64_t rank);
-
-llvm::SmallVector<mlir::OpFoldResult> getZeroOffsets(mlir::PatternRewriter& rewriter, int64_t rank);
-
-llvm::SmallVector<mlir::OpFoldResult> getStaticSizes(mlir::PatternRewriter& rewriter, mlir::ArrayRef<int64_t> shape);
-
 /// Slices a statically shaped tensor along one axis into contiguous pieces of
 /// at most `sliceSize` elements.
 llvm::SmallVector<mlir::Value> sliceTensor(const mlir::Value& tensorToSlice,
                                            size_t axis,
                                            int64_t sliceSize,
-                                           mlir::ConversionPatternRewriter& rewriter,
+                                           mlir::PatternRewriter& rewriter,
                                            mlir::Location loc);
 
 llvm::SmallVector<mlir::Value> sliceVector(const mlir::Value& vectorToSlice,
                                            int64_t sliceSize,
-                                           mlir::ConversionPatternRewriter& rewriter,
+                                           mlir::PatternRewriter& rewriter,
                                            mlir::Location loc);
 
 /// Partitions one logical vector into per-core crossbar-sized slices using the
 /// current PIM target geometry.
 llvm::DenseMap<CoreId, llvm::SmallVector<mlir::Value>> sliceVectorPerCrossbarPerCore(
-  const mlir::Value& vectorToSlice, mlir::ConversionPatternRewriter& rewriter, mlir::Location loc);
+  const mlir::Value& vectorToSlice, mlir::PatternRewriter& rewriter, mlir::Location loc);
-
-mlir::Value extractAxisSlice(
-  mlir::PatternRewriter& rewriter, mlir::Location loc, mlir::Value source, int64_t axis, int64_t offset, int64_t size);
-
-mlir::Value insertStaticSlice(mlir::PatternRewriter& rewriter,
-                              mlir::Location loc,
-                              mlir::Value source,
-                              mlir::Value dest,
-                              llvm::ArrayRef<mlir::OpFoldResult> offsets);
 
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Common/WeightMaterialization.cpp

@@ -19,9 +19,11 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
-bool isWeightLikeComputeOperand(Value value) {
+static bool isWeightMaterializationValue(Value value, bool requireMatrixShape) {
   auto rankedType = dyn_cast<RankedTensorType>(value.getType());
-  if (!rankedType || !isMatrixShape(rankedType.getShape()))
+  if (!rankedType)
+    return false;
+  if (requireMatrixShape && !isMatrixShape(rankedType.getShape()))
     return false;
 
   llvm::SmallPtrSet<Operation*, 8> visited;
@@ -29,8 +31,14 @@ bool isWeightLikeComputeOperand(Value value) {
   while (auto* definingOp = value.getDefiningOp()) {
     if (!visited.insert(definingOp).second)
       return false;
-    if (isa<arith::ConstantOp, ONNXConstantOp>(definingOp) || hasWeightAlways(definingOp))
+    if (isa<arith::ConstantOp, ONNXConstantOp>(definingOp) || hasWeightAlways(definingOp)) {
+      auto sourceType = dyn_cast<RankedTensorType>(value.getType());
+      if (!sourceType)
+        return false;
+      if (requireMatrixShape && !isMatrixShape(sourceType.getShape()))
+        return false;
       return true;
+    }
 
     if (auto extractSliceOp = dyn_cast<tensor::ExtractSliceOp>(definingOp)) {
       value = extractSliceOp.getSource();
@@ -55,6 +63,8 @@ bool isWeightLikeComputeOperand(Value value) {
   return false;
 }
 
+bool isWeightLikeComputeOperand(Value value) { return isWeightMaterializationValue(value, /*requireMatrixShape=*/true); }
+
 FailureOr<Value> materializeWeightLikeValueInBlock(Value value, IRRewriter& rewriter, IRMapping& mapper) {
   if (auto mapped = mapper.lookupOrNull(value))
     return cast<Value>(mapped);
@@ -91,7 +101,7 @@ FailureOr<Value> materializeWeightLikeValueInBlock(Value value, IRRewriter& rewr
       continue;
     }
 
-    if (isWeightLikeComputeOperand(operand)) {
+    if (isWeightMaterializationValue(operand, /*requireMatrixShape=*/false)) {
       auto clonedOperand = materializeWeightLikeValueInBlock(operand, rewriter, mapper);
       if (failed(clonedOperand))
         return failure();

src/PIM/Conversion/ONNXToSpatial/CompileTime.cpp

@@ -9,10 +9,12 @@
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 
+#include <cstring>
 #include <utility>
 
 #include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ShapingUtils.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
@@ -21,24 +23,7 @@ using namespace mlir;
 
 namespace onnx_mlir {
 
-namespace {
+FailureOr<DenseElementsAttr> transposeDenseElementsAttr(DenseElementsAttr denseAttr, ArrayRef<int64_t> perms) {
-
-static bool hasStaticUnitStrides(tensor::ExtractSliceOp extractSliceOp) {
-  return llvm::all_of(extractSliceOp.getStaticStrides(), [](int64_t stride) { return stride == 1; });
-}
-
-static bool hasConstantIndices(tensor::ExtractOp extractOp) {
-  return llvm::all_of(extractOp.getIndices(), [](Value index) { return matchConstantIndexValue(index).has_value(); });
-}
-
-static bool isStaticTensorResult(Operation* op) {
-  return llvm::all_of(op->getResultTypes(), [](Type type) {
-    auto shapedType = dyn_cast<ShapedType>(type);
-    return shapedType && shapedType.hasStaticShape();
-  });
-}
-
-static FailureOr<DenseElementsAttr> transposeDenseElements(DenseElementsAttr denseAttr, ArrayRef<int64_t> perms) {
   auto tensorType = dyn_cast<RankedTensorType>(denseAttr.getType());
   if (!tensorType)
     return failure();
@@ -59,7 +44,45 @@ static FailureOr<DenseElementsAttr> transposeDenseElements(DenseElementsAttr den
 
   auto transposedType = RankedTensorType::get(transposedShape, tensorType.getElementType(), tensorType.getEncoding());
   if (denseAttr.isSplat())
-    return DenseElementsAttr::get(transposedType, denseAttr.getSplatValue<Attribute>());
+    return DenseElementsAttr::getFromRawBuffer(transposedType, denseAttr.getRawData());
+
+  const unsigned elementBitWidth = tensorType.getElementTypeBitWidth();
+  const ArrayRef<char> inputData = denseAttr.getRawData();
+  if (elementBitWidth % 8 == 0) {
+    const size_t elementBytes = elementBitWidth / 8;
+    const size_t expectedBytes = denseAttr.getNumElements() * elementBytes;
+    if (inputData.size() == expectedBytes) {
+      SmallVector<char> transposedData(expectedBytes);
+      if (rank == 2 && perms[0] == 1 && perms[1] == 0) {
+        const int64_t rows = tensorType.getDimSize(0);
+        const int64_t columns = tensorType.getDimSize(1);
+        for (int64_t row = 0; row < rows; ++row)
+          for (int64_t column = 0; column < columns; ++column)
+            std::memcpy(transposedData.data() + (column * rows + row) * elementBytes,
+                        inputData.data() + (row * columns + column) * elementBytes,
+                        elementBytes);
+        return DenseElementsAttr::getFromRawBuffer(transposedType, transposedData);
+      }
+
+      SmallVector<int64_t> originalStrides = computeRowMajorStrides(tensorType.getShape());
+      SmallVector<int64_t> transposedStrides = computeRowMajorStrides(transposedShape);
+      SmallVector<int64_t> originalIndices(rank);
+      for (int64_t linearIndex = 0; linearIndex < tensorType.getNumElements(); ++linearIndex) {
+        int64_t remaining = linearIndex;
+        for (int64_t dim = 0; dim < rank; ++dim) {
+          originalIndices[dim] = originalStrides.empty() ? 0 : remaining / originalStrides[dim];
+          remaining = originalStrides.empty() ? 0 : remaining % originalStrides[dim];
+        }
+        int64_t transposedLinearIndex = 0;
+        for (int64_t dim = 0; dim < rank; ++dim)
+          transposedLinearIndex += originalIndices[perms[dim]] * transposedStrides[dim];
+        std::memcpy(transposedData.data() + transposedLinearIndex * elementBytes,
+                    inputData.data() + linearIndex * elementBytes,
+                    elementBytes);
+      }
+      return DenseElementsAttr::getFromRawBuffer(transposedType, transposedData);
+    }
+  }
 
   SmallVector<Attribute> originalValues(denseAttr.getValues<Attribute>());
   SmallVector<Attribute> transposedValues(originalValues.size());
@@ -84,16 +107,30 @@ static FailureOr<DenseElementsAttr> transposeDenseElements(DenseElementsAttr den
   return DenseElementsAttr::get(transposedType, transposedValues);
 }
 
+namespace {
+
+static bool hasStaticUnitStrides(tensor::ExtractSliceOp extractSliceOp) {
+  return llvm::all_of(extractSliceOp.getStaticStrides(), [](int64_t stride) { return stride == 1; });
+}
+
+static bool hasConstantIndices(tensor::ExtractOp extractOp) {
+  return llvm::all_of(extractOp.getIndices(), [](Value index) { return matchConstantIndexValue(index).has_value(); });
+}
+
+static bool isStaticTensorResult(Operation* op) {
+  return llvm::all_of(op->getResultTypes(), [](Type type) {
+    auto shapedType = dyn_cast<ShapedType>(type);
+    return shapedType && shapedType.hasStaticShape();
+  });
+}
+
 static FailureOr<DenseElementsAttr> reshapeDenseElements(DenseElementsAttr denseAttr, RankedTensorType resultType) {
   auto sourceType = dyn_cast<RankedTensorType>(denseAttr.getType());
-  if (!sourceType || !resultType || sourceType.getNumElements() != resultType.getNumElements())
+  if (!sourceType || !resultType || sourceType.getNumElements() != resultType.getNumElements()
+      || sourceType.getElementType() != resultType.getElementType())
     return failure();
 
-  if (denseAttr.isSplat())
+  return DenseElementsAttr::getFromRawBuffer(resultType, denseAttr.getRawData());
-    return DenseElementsAttr::get(resultType, denseAttr.getSplatValue<Attribute>());
-
-  SmallVector<Attribute> values(denseAttr.getValues<Attribute>());
-  return DenseElementsAttr::get(resultType, values);
 }
 
 static FailureOr<DenseElementsAttr> extractSliceDenseElements(DenseElementsAttr denseAttr,
@@ -161,7 +198,7 @@ static DenseElementsAttr getHostConstantDenseElementsAttrImpl(Value value, llvm:
     perm.reserve(transposeOp.getPermAttr().size());
     for (IntegerAttr attr : transposeOp.getPermAttr().getAsRange<IntegerAttr>())
       perm.push_back(attr.getInt());
-    auto transposedAttr = transposeDenseElements(inputAttr, perm);
+    auto transposedAttr = transposeDenseElementsAttr(inputAttr, perm);
     return succeeded(transposedAttr) ? *transposedAttr : nullptr;
   }
 
@@ -171,7 +208,7 @@ static DenseElementsAttr getHostConstantDenseElementsAttrImpl(Value value, llvm:
       return nullptr;
 
     SmallVector<int64_t> perm(transposeOp.getPermutation().begin(), transposeOp.getPermutation().end());
-    auto transposedAttr = transposeDenseElements(inputAttr, perm);
+    auto transposedAttr = transposeDenseElementsAttr(inputAttr, perm);
     return succeeded(transposedAttr) ? *transposedAttr : nullptr;
   }
 
@@ -219,6 +256,9 @@ getCompileTimeSourceImpl(Operation* op, llvm::SmallPtrSetImpl<Operation*>& visit
 
   chainLength += 1;
 
+  if (!isShapingOnlyOp(op))
+    return std::nullopt;
+
   if (auto extractOp = dyn_cast<tensor::ExtractOp>(op))
     return hasConstantIndices(extractOp)
            ? getCompileTimeSourceImpl(extractOp.getTensor().getDefiningOp(), visited, chainLength)

src/PIM/Conversion/ONNXToSpatial/CompileTime.hpp

@@ -4,6 +4,8 @@
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/Value.h"
 
+#include "llvm/ADT/ArrayRef.h"
+
 namespace onnx_mlir {
 
 struct CompileTimeSource {
@@ -19,4 +21,7 @@ bool isCompileTimeOp(mlir::Operation* op);
 
 mlir::DenseElementsAttr getHostConstDenseElementsAttr(mlir::Value value);
 
+mlir::FailureOr<mlir::DenseElementsAttr> transposeDenseElementsAttr(
+  mlir::DenseElementsAttr denseAttr, llvm::ArrayRef<int64_t> permutation);
+
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/LowerSpatialPlansPass.cpp

@@ -0,0 +1,522 @@
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/Linalg/IR/Linalg.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/DialectConversion.h"
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallPtrSet.h"
+
+#include "Conversion/ONNXToSpatial/ONNXToSpatialVerifier.hpp"
+#include "mlir/Transforms/Passes.h"
+#include "src/Accelerators/PIM/Common/PimCommon.hpp"
+#include "src/Accelerators/PIM/Common/Support/DebugDump.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/PlanLowering.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/Transforms/MergeComputeNodes/SpatialDataflowCsvExporter.hpp"
+#include "src/Accelerators/PIM/Pass/PIMPasses.h"
+#include "src/Dialect/ONNX/ONNXOps.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static constexpr StringLiteral kDenseLayout = "dense_nchw";
+static constexpr StringLiteral kRowStripLayout = "nchw_row_strip";
+
+static FailureOr<RowStripPhysicalValue> getRowStripValue(llvm::DenseMap<Value, RowStripPhysicalValue>& rowStripValues,
+                                                         Value value) {
+  auto it = rowStripValues.find(value);
+  if (it == rowStripValues.end())
+    return failure();
+  return it->second;
+}
+
+static FailureOr<RowStripPhysicalValue> buildRowStripValue(spatial::SpatBlueprintOp blueprint,
+                                                           Value storage) {
+  auto logicalType = dyn_cast<RankedTensorType>(blueprint.getOutput().getType());
+  if (!logicalType)
+    return blueprint.emitOpError("requires ranked logical output type"), failure();
+  if (blueprint.getIndexMap() != kRowStripIndexMap)
+    return blueprint.emitOpError("requires the canonical row-strip index map"), failure();
+  FailureOr<RowStripPhysicalValue> value = describeRowStripPhysicalValue(storage, logicalType);
+  if (failed(value))
+    return blueprint.emitOpError("requires physical row-strip fragment storage"), failure();
+  return *value;
+}
+
+static FailureOr<Value>
+lowerRowStripRelu(const RowStripPhysicalValue& input, spatial::SpatReluPlanOp planOp, PatternRewriter& rewriter) {
+  return applyRowStripRelu(input, rewriter, planOp.getLoc());
+}
+
+static FailureOr<Value> lowerRowStripBiasAdd(const RowStripPhysicalValue& input,
+                                             spatial::SpatBiasAddPlanOp planOp,
+                                             PatternRewriter& rewriter) {
+  return applyRowStripBiasAdd(input, planOp.getBias(), rewriter, planOp.getLoc());
+}
+
+static FailureOr<Value>
+materializeRowStripToDense(const RowStripPhysicalValue& rowStripValue, Location loc, PatternRewriter& rewriter) {
+  if (rowStripValue.logicalType.getRank() != 4 || !rowStripValue.logicalType.hasStaticShape())
+    return failure();
+  return createRowStripAssemblyBlueprint(rowStripValue, rewriter, loc);
+}
+
+static FailureOr<Value> lowerDenseBatchBiasAdd(Value input, Value bias, RankedTensorType resultType,
+                                               PatternRewriter& rewriter, Location loc) {
+  auto producer = input.getDefiningOp<spatial::SpatGraphComputeBatch>();
+  auto inputType = dyn_cast<RankedTensorType>(input.getType());
+  auto biasType = dyn_cast<RankedTensorType>(bias.getType());
+  if (!producer || !inputType || !biasType || !inputType.hasStaticShape() || !biasType.hasStaticShape()
+      || !resultType.hasStaticShape() || inputType.getDimSize(0) != producer.getLaneCount()
+      || biasType.getDimSize(0) != producer.getLaneCount() || resultType.getDimSize(0) != producer.getLaneCount())
+    return failure();
+  auto inputFragmentType = spatial::getGraphBatchFragmentType(inputType, producer.getLaneCount());
+  auto outputFragmentType = spatial::getGraphBatchFragmentType(resultType, producer.getLaneCount());
+  if (failed(inputFragmentType) || failed(outputFragmentType) || inputFragmentType->getRank() != biasType.getRank()
+      || inputFragmentType->getDimSize(0) != 1 || inputFragmentType->getShape().drop_front() != biasType.getShape().drop_front()
+      || inputFragmentType->getRank() != outputFragmentType->getRank() + 1)
+    return failure();
+  for (auto [inputDim, outputDim] : llvm::zip(inputFragmentType->getShape().drop_front(), outputFragmentType->getShape()))
+    if (outputDim > inputDim)
+      return failure();
+
+  auto batch = createSpatComputeBatch(rewriter, loc, TypeRange {resultType}, producer.getLaneCount(), {}, ValueRange {input, bias},
+    [&](detail::SpatComputeBatchBodyArgs args) -> LogicalResult {
+      FailureOr<Value> fragment = extractGraphBatchPhysicalFragment(rewriter, loc, args.inputs[0], args.lane, *inputFragmentType);
+      if (failed(fragment))
+        return failure();
+      MixedSliceGeometry biasSlice;
+      for (int64_t dim : inputFragmentType->getShape()) {
+        biasSlice.offsets.push_back(biasSlice.offsets.empty() ? OpFoldResult(args.lane) : rewriter.getIndexAttr(0));
+        biasSlice.sizes.push_back(rewriter.getIndexAttr(dim));
+        biasSlice.strides.push_back(rewriter.getIndexAttr(1));
+      }
+      Value biasFragment = extractMixedSliceOrIdentity(rewriter, loc, args.inputs[1], *inputFragmentType, biasSlice);
+      if (!biasFragment)
+        return failure();
+      Value added = spatial::SpatVAddOp::create(rewriter, loc, *inputFragmentType, *fragment, biasFragment);
+      MixedSliceGeometry outputSlice;
+      outputSlice.offsets.assign(inputFragmentType->getRank(), rewriter.getIndexAttr(0));
+      outputSlice.sizes.push_back(rewriter.getIndexAttr(1));
+      outputSlice.strides.assign(inputFragmentType->getRank(), rewriter.getIndexAttr(1));
+      for (int64_t dim : outputFragmentType->getShape())
+        outputSlice.sizes.push_back(rewriter.getIndexAttr(dim));
+      Value output = extractMixedSliceOrIdentity(rewriter, loc, added, *outputFragmentType, outputSlice);
+      if (!output)
+        return failure();
+      publishGraphBatchPhysicalFragment(rewriter, loc, output, args.outputs.front(), args.lane);
+      return success();
+    });
+  if (failed(batch))
+    return failure();
+  return batch->getResult(0);
+}
+
+struct LowerSpatialPlansPass final : PassWrapper<LowerSpatialPlansPass, OperationPass<ModuleOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(LowerSpatialPlansPass)
+
+  StringRef getArgument() const override { return "lower-spatial-plans"; }
+  StringRef getDescription() const override { return "Lower selected Spatial planning ops to low-level Spatial IR."; }
+
+  void runOnOperation() override {
+    ModuleOp moduleOp = getOperation();
+    MLIRContext* ctx = moduleOp.getContext();
+    auto entryFunc = getPimEntryFunc(moduleOp);
+    if (failed(entryFunc)) {
+      moduleOp.emitError("failed to locate the PIM entry function during LowerSpatialPlans");
+      signalPassFailure();
+      return;
+    }
+    func::FuncOp funcOp = *entryFunc;
+    PatternRewriter rewriter(ctx);
+    llvm::DenseMap<Value, RowStripPhysicalValue> rowStripValues;
+    llvm::SmallPtrSet<Operation*, 16> eraseAfterLowering;
+    auto verifyLogicalPhase = [&](StringRef stage) -> bool {
+      if (succeeded(verifyLogicalSpatialGraphInvariants(*entryFunc)))
+        return true;
+      moduleOp.emitError() << "logical Spatial graph verification failed " << stage;
+      signalPassFailure();
+      return false;
+    };
+
+    if (!verifyLogicalPhase("at the start of LowerSpatialPlans"))
+      return;
+    for (Operation& op : llvm::make_early_inc_range(funcOp.getBody().front())) {
+      if (auto planOp = dyn_cast<spatial::SpatConv2DPlanOp>(&op)) {
+        FailureOr<RowStripPhysicalValue> rowStripInput = getRowStripValue(rowStripValues, planOp.getInput());
+        auto rowStripBlueprint = llvm::find_if(planOp.getResult().getUsers(), [](Operation* user) {
+          auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(user);
+          return blueprint && blueprint.getPhysicalLayout() == kRowStripLayout;
+        });
+        if (rowStripBlueprint != planOp.getResult().getUsers().end()) {
+          rewriter.setInsertionPoint(planOp);
+          std::optional<Value> physicalInput;
+          if (succeeded(rowStripInput))
+            physicalInput = rowStripInput->storage;
+          FailureOr<Value> lowered = lowerSelectedConv2DPlan(
+            planOp,
+            physicalInput,
+            /*emitRowStripLayout=*/true,
+            rewriter);
+          if (failed(lowered)) {
+            planOp.emitOpError("failed to lower selected row-strip Spatial Conv plan");
+            signalPassFailure();
+            return;
+          }
+          auto blueprint = cast<spatial::SpatBlueprintOp>(*rowStripBlueprint);
+          FailureOr<RowStripPhysicalValue> rowStripValue = buildRowStripValue(blueprint, *lowered);
+          if (failed(rowStripValue)) {
+            signalPassFailure();
+            return;
+          }
+          rowStripValues[blueprint.getResult()] = *rowStripValue;
+          eraseAfterLowering.insert(planOp);
+          eraseAfterLowering.insert(blueprint);
+          continue;
+        }
+        rewriter.setInsertionPoint(planOp);
+        FailureOr<Value> lowered =
+          lowerSelectedConv2DPlan(planOp, std::nullopt, /*emitRowStripLayout=*/false, rewriter);
+        if (failed(lowered)) {
+          planOp.emitOpError("failed to lower selected Spatial Conv plan");
+          signalPassFailure();
+          return;
+        }
+        rewriter.replaceOp(planOp, *lowered);
+        continue;
+      }
+
+      if (auto planOp = dyn_cast<spatial::SpatReluPlanOp>(&op)) {
+        if (succeeded(getRowStripValue(rowStripValues, planOp.getInput()))) {
+          auto outputBlueprint = llvm::find_if(planOp.getResult().getUsers(), [](Operation* user) {
+            auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(user);
+            return blueprint && blueprint.getPhysicalLayout() == kRowStripLayout;
+          });
+          if (outputBlueprint == planOp.getResult().getUsers().end()) {
+            planOp.emitOpError("row-strip Relu plan requires a row-strip blueprint result");
+            signalPassFailure();
+            return;
+          }
+
+          FailureOr<RowStripPhysicalValue> input = getRowStripValue(rowStripValues, planOp.getInput());
+          rewriter.setInsertionPoint(planOp);
+          FailureOr<Value> lowered = lowerRowStripRelu(*input, planOp, rewriter);
+          if (failed(lowered)) {
+            planOp.emitOpError("failed to lower selected row-strip Spatial Relu plan");
+            signalPassFailure();
+            return;
+          }
+          auto blueprint = cast<spatial::SpatBlueprintOp>(*outputBlueprint);
+          FailureOr<RowStripPhysicalValue> output = buildRowStripValue(blueprint, *lowered);
+          if (failed(output)) {
+            signalPassFailure();
+            return;
+          }
+          rowStripValues[blueprint.getResult()] = *output;
+          eraseAfterLowering.insert(planOp);
+          eraseAfterLowering.insert(blueprint);
+          continue;
+        }
+
+        rewriter.setInsertionPoint(planOp);
+        auto computeOp = createSpatCompute<1>(
+          rewriter, planOp.getLoc(), planOp.getOutput().getType(), {}, planOp.getInput(), [&](Value x) {
+            auto relu = spatial::SpatReluOp::create(rewriter, planOp.getLoc(), planOp.getOutput().getType(), x);
+            spatial::SpatYieldOp::create(rewriter, planOp.getLoc(), relu.getResult());
+          });
+        rewriter.replaceOp(planOp, computeOp.getResults());
+        continue;
+      }
+      if (auto planOp = dyn_cast<spatial::SpatMaxPool2DPlanOp>(&op)) {
+        auto outputBlueprint = llvm::find_if(planOp.getResult().getUsers(), [](Operation* user) {
+          auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(user);
+          return blueprint && blueprint.getPhysicalLayout() == kRowStripLayout;
+        });
+        if (outputBlueprint == planOp.getResult().getUsers().end()) {
+          planOp.emitOpError("selected MaxPool plan requires a row-strip blueprint result");
+          signalPassFailure();
+          return;
+        }
+
+        FailureOr<RowStripPhysicalValue> input = getRowStripValue(rowStripValues, planOp.getInput());
+        rewriter.setInsertionPoint(planOp);
+        std::optional<Value> physicalInput;
+        if (succeeded(input)) {
+          if (input->tilesPerRow == 1) {
+            physicalInput = input->storage;
+          }
+          else {
+            FailureOr<Value> denseInput = materializeRowStripToDense(*input, planOp.getLoc(), rewriter);
+            if (failed(denseInput)) {
+              planOp.emitOpError("failed to materialize tiled row-strip input for MaxPool");
+              signalPassFailure();
+              return;
+            }
+            planOp.getInputMutable().assign(*denseInput);
+          }
+        }
+        FailureOr<Value> lowered = lowerSelectedMaxPool2DPlan(
+          planOp, physicalInput, rewriter);
+        if (failed(lowered)) {
+          planOp.emitOpError("failed to lower selected row-strip Spatial MaxPool plan");
+          signalPassFailure();
+          return;
+        }
+        auto blueprint = cast<spatial::SpatBlueprintOp>(*outputBlueprint);
+        FailureOr<RowStripPhysicalValue> output = buildRowStripValue(blueprint, *lowered);
+        if (failed(output)) {
+          signalPassFailure();
+          return;
+        }
+        rowStripValues[blueprint.getResult()] = *output;
+        eraseAfterLowering.insert(planOp);
+        eraseAfterLowering.insert(blueprint);
+        continue;
+      }
+      if (auto planOp = dyn_cast<spatial::SpatBiasAddPlanOp>(&op)) {
+        if (succeeded(getRowStripValue(rowStripValues, planOp.getInput()))) {
+          auto outputBlueprint = llvm::find_if(planOp.getResult().getUsers(), [](Operation* user) {
+            auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(user);
+            return blueprint && blueprint.getPhysicalLayout() == kRowStripLayout;
+          });
+          if (outputBlueprint == planOp.getResult().getUsers().end()) {
+            planOp.emitOpError("row-strip bias_add plan requires a row-strip blueprint result");
+            signalPassFailure();
+            return;
+          }
+
+          FailureOr<RowStripPhysicalValue> input = getRowStripValue(rowStripValues, planOp.getInput());
+          rewriter.setInsertionPoint(planOp);
+          FailureOr<Value> lowered = lowerRowStripBiasAdd(*input, planOp, rewriter);
+          if (failed(lowered)) {
+            planOp.emitOpError("failed to lower selected row-strip Spatial bias_add plan");
+            signalPassFailure();
+            return;
+          }
+          auto blueprint = cast<spatial::SpatBlueprintOp>(*outputBlueprint);
+          FailureOr<RowStripPhysicalValue> output = buildRowStripValue(blueprint, *lowered);
+          if (failed(output)) {
+            signalPassFailure();
+            return;
+          }
+          rowStripValues[blueprint.getResult()] = *output;
+          eraseAfterLowering.insert(planOp);
+          eraseAfterLowering.insert(blueprint);
+          continue;
+        }
+
+        auto resultType = dyn_cast<RankedTensorType>(planOp.getOutput().getType());
+        if (!resultType) {
+          planOp.emitOpError("requires ranked output type");
+          signalPassFailure();
+          return;
+        }
+        rewriter.setInsertionPoint(planOp);
+        FailureOr<Value> denseBias = materializeDenseBiasAddTensor(planOp.getBias(), resultType, rewriter, planOp.getLoc());
+        if (failed(denseBias)) {
+          planOp.emitOpError("failed to materialize dense Conv-style bias");
+          signalPassFailure();
+          return;
+        }
+        if (planOp.getInput().getDefiningOp<spatial::SpatGraphComputeBatch>()) {
+          FailureOr<Value> lowered = lowerDenseBatchBiasAdd(planOp.getInput(), *denseBias, resultType, rewriter, planOp.getLoc());
+          if (succeeded(lowered)) {
+            rewriter.replaceOp(planOp, *lowered);
+            continue;
+          }
+        }
+        auto computeOp = createSpatCompute<2>(rewriter,
+                                              planOp.getLoc(),
+                                              planOp.getOutput().getType(),
+                                              {},
+                                              ValueRange {planOp.getInput(), *denseBias},
+                                              [&](Value x, Value y) {
+                                                auto added = spatial::SpatVAddOp::create(
+                                                  rewriter, planOp.getLoc(), planOp.getOutput().getType(), x, y);
+                                                spatial::SpatYieldOp::create(rewriter, planOp.getLoc(), added.getResult());
+                                              });
+        rewriter.replaceOp(planOp, computeOp.getResults());
+        continue;
+      }
+      if (auto materializeOp = dyn_cast<spatial::SpatMaterializeLayoutOp>(&op)) {
+        if (materializeOp.getSourcePhysicalLayout() == kDenseLayout
+            && materializeOp.getTargetPhysicalLayout() == kDenseLayout) {
+          rewriter.replaceOp(materializeOp, materializeOp.getInput());
+          continue;
+        }
+        if (materializeOp.getSourcePhysicalLayout() != kRowStripLayout
+            || materializeOp.getTargetPhysicalLayout() != kDenseLayout) {
+          materializeOp.emitOpError("non-dense materialize_layout lowering is not supported yet");
+          signalPassFailure();
+          return;
+        }
+        FailureOr<RowStripPhysicalValue> rowStripValue = getRowStripValue(rowStripValues, materializeOp.getInput());
+        if (failed(rowStripValue)) {
+          materializeOp.emitOpError("expected a row-strip blueprint input during row-strip materialization");
+          signalPassFailure();
+          return;
+        }
+        rewriter.setInsertionPoint(materializeOp);
+        FailureOr<Value> dense = materializeRowStripToDense(*rowStripValue, materializeOp.getLoc(), rewriter);
+        if (failed(dense)) {
+          materializeOp.emitOpError("failed to materialize selected row-strip layout back to dense NCHW");
+          signalPassFailure();
+          return;
+        }
+        rewriter.replaceOp(materializeOp, *dense);
+        continue;
+      }
+      if (auto blueprintOp = dyn_cast<spatial::SpatBlueprintOp>(&op)) {
+        if (std::optional<StringRef> mode = blueprintOp.getMode(); mode && *mode == "fragment_assembly")
+          continue;
+        if (blueprintOp.getPhysicalLayout() == kDenseLayout) {
+          rewriter.replaceOp(blueprintOp, blueprintOp.getInput());
+          continue;
+        }
+        if (blueprintOp.getPhysicalLayout() != kRowStripLayout) {
+          blueprintOp.emitOpError("non-dense blueprint lowering is not supported yet");
+          signalPassFailure();
+          return;
+        }
+        if (!eraseAfterLowering.contains(blueprintOp)) {
+          blueprintOp.emitOpError("unhandled row-strip blueprint remained during LowerSpatialPlans");
+          signalPassFailure();
+          return;
+        }
+      }
+    }
+    bool erasedAny = true;
+    while (erasedAny) {
+      erasedAny = false;
+      for (Operation& op : llvm::make_early_inc_range(funcOp.getBody().front())) {
+        if (!eraseAfterLowering.contains(&op))
+          continue;
+        if (!op.use_empty())
+          continue;
+        eraseAfterLowering.erase(&op);
+        rewriter.eraseOp(&op);
+        erasedAny = true;
+      }
+    }
+    if (!eraseAfterLowering.empty()) {
+      for (Operation& op : funcOp.getBody().front())
+        if (eraseAfterLowering.contains(&op))
+          op.emitOpError("selected row-strip planning op could not be fully eliminated during LowerSpatialPlans");
+      signalPassFailure();
+      return;
+    }
+    ConversionTarget helperTarget(*ctx);
+    helperTarget.addLegalDialect<spatial::SpatialDialect,
+                                 tensor::TensorDialect,
+                                 linalg::LinalgDialect,
+                                 affine::AffineDialect,
+                                 arith::ArithDialect,
+                                 scf::SCFDialect,
+                                 func::FuncDialect>();
+    helperTarget.addLegalOp<spatial::SpatGraphCompute, spatial::SpatGraphComputeBatch>();
+    helperTarget.addIllegalOp<ONNXGemmOp, ONNXTransposeOp>();
+    helperTarget.markOpRecursivelyLegal<spatial::SpatGraphCompute, spatial::SpatGraphComputeBatch>();
+
+    RewritePatternSet helperPatterns(ctx);
+    populateGemmPatterns(helperPatterns, ctx);
+    populateTransposePatterns(helperPatterns, ctx);
+    FrozenRewritePatternSet frozenHelperPatterns(
+      std::move(helperPatterns));
+    SmallVector<Operation*> topLevelHelperOps;
+    funcOp.walk([&](Operation* op) {
+      if (isa<spatial::SpatGraphCompute,
+              spatial::SpatGraphComputeBatch>(op))
+        return WalkResult::skip();
+      if (isa<ONNXGemmOp, ONNXTransposeOp>(op))
+        topLevelHelperOps.push_back(op);
+      return WalkResult::advance();
+    });
+    for (Operation *helper : topLevelHelperOps) {
+      if (failed(applyPartialConversion(
+            helper, helperTarget, frozenHelperPatterns))) {
+        moduleOp.emitError("failed to lower helper ONNX ops emitted by selected Spatial plan lowering");
+        signalPassFailure();
+        return;
+      }
+    }
+    ConversionTarget nestedHelperTarget(*ctx);
+    nestedHelperTarget.addLegalDialect<spatial::SpatialDialect,
+                                       tensor::TensorDialect,
+                                       linalg::LinalgDialect,
+                                       affine::AffineDialect,
+                                       arith::ArithDialect,
+                                       scf::SCFDialect,
+                                       func::FuncDialect>();
+    nestedHelperTarget.addIllegalOp<ONNXGemmOp, ONNXTransposeOp>();
+    SmallVector<Operation*> computeLikeOps;
+    funcOp.walk([&](Operation* op) {
+      if (isa<spatial::SpatGraphCompute, spatial::SpatGraphComputeBatch>(op))
+        computeLikeOps.push_back(op);
+    });
+    for (Operation* op : computeLikeOps) {
+      if (failed(applyFullConversion(
+            op, nestedHelperTarget, frozenHelperPatterns))) {
+        op->emitOpError("failed to lower nested helper ONNX ops emitted by selected Spatial plan lowering");
+        signalPassFailure();
+        return;
+      }
+    }
+    if (!verifyLogicalPhase("after nested helper conversions"))
+      return;
+    bool hasIllegalOps = false;
+    moduleOp.walk([&](Operation* op) {
+      if (isa<ONNXEntryPointOp>(op))
+        return;
+      if (auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(op)) {
+        if (std::optional<StringRef> mode = blueprint.getMode(); mode && *mode == "fragment_assembly")
+          return;
+        op->emitOpError("planning blueprint must not remain after LowerSpatialPlans");
+        hasIllegalOps = true;
+      } else if (isa<spatial::SpatConv2DPlanOp,
+                     spatial::SpatBiasAddPlanOp,
+                     spatial::SpatReluPlanOp,
+                     spatial::SpatMaxPool2DPlanOp,
+                     spatial::SpatMaterializeLayoutOp>(op)
+          || op->getDialect()->getNamespace() == "onnx") {
+        op->emitOpError("operation must not remain after LowerSpatialPlans");
+        hasIllegalOps = true;
+      }
+    });
+
+    PassManager canonicalizationPM(ctx);
+    canonicalizationPM.addPass(createCanonicalizerPass());
+    if (failed(canonicalizationPM.run(moduleOp)))
+      moduleOp.emitWarning("failed to run LowerSpatialPlansPass canonicalization; continuing");
+
+    if (hasIllegalOps) {
+      signalPassFailure();
+    } else {
+      dumpModule(moduleOp, "spatial1_graph");
+      spatial::SpatialDataflowExportStage exportMode = spatial::getSpatialDataflowExportStage();
+      if (spatial::shouldExportSpatialDataflowStage(exportMode, spatial::SpatialDataflowExportStage::Spatial1)
+          && failed(spatial::exportSpatialDataflowCsvGraph(funcOp, "spatial1_graph"))) {
+        signalPassFailure();
+        return;
+      }
+    }
+
+    if (!verifyLogicalPhase("at the end of LowerSpatialPlans"))
+      return;
+  }
+};
+
+} // namespace
+
+std::unique_ptr<Pass> createLowerSpatialPlansPass() { return std::make_unique<LowerSpatialPlansPass>(); }
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/ONNXToSpatialPass.cpp

@@ -13,11 +13,13 @@
 
 #include "Common/Common.hpp"
 #include "Common/PimCommon.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/ONNXToSpatialVerifier.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
+#include "ONNXToSpatialVerifier.hpp"
 
 using namespace mlir;
 
@@ -41,10 +43,18 @@ struct ONNXToSpatialPass : PassWrapper<ONNXToSpatialPass, OperationPass<ModuleOp
 static void populateEmptyFunction(func::FuncOp funcOp) {
   IRRewriter rewriter(funcOp.getContext());
   IRMapping mapper;
-  SmallVector<spatial::SpatCompute> computes(funcOp.getOps<spatial::SpatCompute>());
+  SmallVector<spatial::SpatGraphCompute> computes(funcOp.getOps<spatial::SpatGraphCompute>());
-  SmallVector<spatial::SpatComputeBatch> computeBatches(funcOp.getOps<spatial::SpatComputeBatch>());
+  SmallVector<spatial::SpatGraphComputeBatch> computeBatches(funcOp.getOps<spatial::SpatGraphComputeBatch>());
-  if (!computes.empty() || !computeBatches.empty())
+  SmallVector<spatial::SpatConv2DPlanOp> convPlans(funcOp.getOps<spatial::SpatConv2DPlanOp>());
+  SmallVector<spatial::SpatBiasAddPlanOp> biasAddPlans(funcOp.getOps<spatial::SpatBiasAddPlanOp>());
+  SmallVector<spatial::SpatReluPlanOp> reluPlans(funcOp.getOps<spatial::SpatReluPlanOp>());
+  SmallVector<spatial::SpatMaxPool2DPlanOp> maxPoolPlans(funcOp.getOps<spatial::SpatMaxPool2DPlanOp>());
+  SmallVector<spatial::SpatBlueprintOp> blueprints(funcOp.getOps<spatial::SpatBlueprintOp>());
+  SmallVector<spatial::SpatMaterializeLayoutOp> materializers(funcOp.getOps<spatial::SpatMaterializeLayoutOp>());
+  if (!computes.empty() || !computeBatches.empty() || !convPlans.empty() || !biasAddPlans.empty() || !reluPlans.empty()
+      || !maxPoolPlans.empty() || !blueprints.empty() || !materializers.empty()) {
     return;
+  }
 
   auto returnOp = cast<func::ReturnOp>(funcOp.getFunctionBody().front().getTerminator());
   rewriter.setInsertionPoint(returnOp);
@@ -58,16 +68,16 @@ static void populateEmptyFunction(func::FuncOp funcOp) {
     sourceLocs.push_back(source.getLoc());
   }
 
-  auto newCompute = spatial::SpatCompute::create(
+  auto newCompute = createEmptySpatGraphCompute(
-    rewriter, returnOp.getLoc(), returnOp.getOperandTypes(), funcOp.getArguments(), {}, {});
+    rewriter, returnOp.getLoc(), returnOp.getOperandTypes(), {}, funcOp.getArguments(), sourceTypes, sourceLocs);
-  auto* newBlock = rewriter.createBlock(&newCompute.getBody(), newCompute.getBody().end(), sourceTypes, sourceLocs);
+  auto* newBlock = &newCompute.getBody().front();
   for (auto [blockArg, computeArg] : llvm::zip(newBlock->getArguments(), newCompute.getOperands()))
     mapper.map(computeArg, blockArg);
   newCompute.getProperties().setOperandSegmentSizes({0, static_cast<int>(sourceTypes.size())});
 
   rewriter.setInsertionPointToEnd(newBlock);
   for (Operation& op : funcOp.getOps())
-    if (!isa<spatial::SpatCompute, func::ReturnOp>(&op))
+    if (!isa<spatial::SpatGraphCompute, func::ReturnOp>(&op))
       rewriter.clone(op, mapper);
 
   auto yield = spatial::SpatYieldOp::create(rewriter, funcOp.getLoc(), returnOp.getOperands());
@@ -75,7 +85,7 @@ static void populateEmptyFunction(func::FuncOp funcOp) {
     yield.setOperand(i, mapper.lookupOrDefault(yield.getOperand(i)));
 
   for (Operation& op : llvm::make_early_inc_range(funcOp.getOps()))
-    if (!isa<spatial::SpatCompute, func::ReturnOp>(&op)) {
+    if (!isa<spatial::SpatGraphCompute, func::ReturnOp>(&op)) {
       op.dropAllUses();
       rewriter.eraseOp(&op);
     }
@@ -96,7 +106,7 @@ void ONNXToSpatialPass::runOnOperation() {
                             affine::AffineDialect,
                             arith::ArithDialect,
                             scf::SCFDialect>();
-  preTarget.addIllegalOp<ONNXConstantOp, ONNXFlattenOp>();
+  preTarget.addIllegalOp<ONNXConstantOp>();
 
   RewritePatternSet prePatterns(ctx);
   populatePrePatterns(prePatterns, ctx);
@@ -135,6 +145,7 @@ void ONNXToSpatialPass::runOnOperation() {
   target.addIllegalOp<ONNXSigmoidOp>();
   target.addIllegalOp<ONNXSoftmaxOp>();
   target.addIllegalOp<ONNXConcatOp>();
+  target.addIllegalOp<ONNXFlattenOp>();
   target.addIllegalOp<ONNXGatherOp>();
   target.addIllegalOp<ONNXReshapeOp>();
   target.addIllegalOp<ONNXResizeOp>();
@@ -152,6 +163,11 @@ void ONNXToSpatialPass::runOnOperation() {
     return;
   }
 
+    if (failed(verifyLogicalSpatialGraphInvariants(*entryFunc))) {
+      moduleOp.emitError("logical Spatial graph verification failed after ONNX conversion");
+      signalPassFailure();
+      return;
+    }
   ConversionTarget earlyPostTarget(*ctx);
   earlyPostTarget.addLegalDialect<spatial::SpatialDialect,
                                   ONNXDialect,
@@ -161,13 +177,13 @@ void ONNXToSpatialPass::runOnOperation() {
                                   arith::ArithDialect,
                                   scf::SCFDialect>();
 
-  PassManager cleanupPM(ctx);
-  cleanupPM.addPass(createCanonicalizerPass());
-  if (failed(cleanupPM.run(moduleOp)))
-    moduleOp.emitWarning("failed to run ONNX-to-Spatial canonicalization cleanup; continuing");
-
   annotateWeightsConstants(*entryFunc);
 
+    if (failed(verifyLogicalSpatialGraphInvariants(*entryFunc))) {
+      moduleOp.emitError("logical Spatial graph verification failed after weight annotation");
+      signalPassFailure();
+      return;
+    }
   ConversionTarget postTarget(*ctx);
   postTarget.addLegalDialect<spatial::SpatialDialect,
                              ONNXDialect,
@@ -176,11 +192,16 @@ void ONNXToSpatialPass::runOnOperation() {
                              affine::AffineDialect,
                              arith::ArithDialect,
                              scf::SCFDialect>();
-  postTarget.addDynamicallyLegalOp<spatial::SpatCompute>(
+  postTarget.addDynamicallyLegalOp<spatial::SpatGraphCompute>(
-    [](spatial::SpatCompute computeOp) { return !requiresPostRewrite(computeOp); });
+    [](spatial::SpatGraphCompute computeOp) { return !requiresPostRewrite(computeOp); });
-  postTarget.addDynamicallyLegalOp<spatial::SpatComputeBatch>(
+  postTarget.addDynamicallyLegalOp<spatial::SpatGraphComputeBatch>(
-    [](spatial::SpatComputeBatch computeOp) { return !requiresPostRewrite(computeOp); });
+    [](spatial::SpatGraphComputeBatch computeOp) { return !requiresPostRewrite(computeOp); });
 
+    if (failed(verifyLogicalSpatialGraphInvariants(*entryFunc))) {
+      moduleOp.emitError("logical Spatial graph verification failed before post rewrites");
+      signalPassFailure();
+      return;
+    }
   RewritePatternSet postPatterns(ctx);
   populatePostPatterns(postPatterns, ctx);
   if (failed(applyPartialConversion(*entryFunc, postTarget, std::move(postPatterns)))) {
@@ -191,8 +212,18 @@ void ONNXToSpatialPass::runOnOperation() {
 
   populateEmptyFunction(*entryFunc);
 
+  PassManager canonicalizationPM(ctx);
+  canonicalizationPM.addPass(createCanonicalizerPass());
+  if (failed(canonicalizationPM.run(moduleOp)))
+    moduleOp.emitWarning("failed to run ONNXToSpatial canonicalization; continuing");
+
   dumpModule(moduleOp, "spatial0");
 
+  if (failed(verifyLogicalSpatialGraphInvariants(*entryFunc))) {
+      moduleOp.emitError("logical Spatial graph verification failed after ONNX-to-Spatial");
+      signalPassFailure();
+      return;
+    }
   if (failed(verifyONNXToSpatial(*entryFunc))) {
     moduleOp.emitError("ONNX-to-Spatial host legality verification failed");
     signalPassFailure();

src/PIM/Conversion/ONNXToSpatial/ONNXToSpatialVerifier.cpp

@@ -1,4 +1,6 @@
 #include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 
 #include "Common/IR/WeightUtils.hpp"
@@ -13,6 +15,8 @@ namespace onnx_mlir {
 
 namespace {
 
+constexpr StringLiteral kPhaseMarker = "phase-check";
+
 void checkWeightUseChains(func::FuncOp func, pim::CappedDiagnosticReporter& diagnostics) {
   func.walk([&](Operation* op) {
     if (!hasWeightAlways(op))
@@ -23,134 +27,206 @@ void checkWeightUseChains(func::FuncOp func, pim::CappedDiagnosticReporter& diag
         continue;
 
       diagnostics.report(op, [&](Operation* illegalOp) {
-        illegalOp->emitOpError(
+        illegalOp->emitOpError()
-          "weight-marked values may only flow through static view/slice helper chains into Spatial VMM weights");
+          << kPhaseMarker
+          << " weight-marked values may only flow through static view/slice helper chains into Spatial VMM weights";
       });
       return;
     }
   });
 }
 
-Region* getParentRegion(Value value) {
+bool isRegionOrAncestorOf(Region& region, Region* candidate) {
-  if (auto blockArg = dyn_cast<BlockArgument>(value))
+  return candidate && (&region == candidate || region.isAncestor(candidate));
-    return blockArg.getOwner()->getParent();
-  if (Operation* definingOp = value.getDefiningOp())
-    return definingOp->getParentRegion();
-  return nullptr;
 }
 
-bool isDefinedInsideRegion(Value value, Region& region) {
+bool isValueDefinedInsideRegion(Value value, Region& region) {
-  Region* parentRegion = getParentRegion(value);
+  if (auto blockArg = dyn_cast<BlockArgument>(value))
-  return parentRegion && (&region == parentRegion || region.isAncestor(parentRegion));
+    return isRegionOrAncestorOf(region, blockArg.getOwner()->getParent());
+  if (Operation* definingOp = value.getDefiningOp())
+    return isRegionOrAncestorOf(region, definingOp->getParentRegion());
+  return false;
+}
+
+bool isLegalExternalCapture(Value value, Region& region) {
+  if (isValueDefinedInsideRegion(value, region))
+    return true;
+
+  Operation* definingOp = value.getDefiningOp();
+  return definingOp && definingOp->hasTrait<OpTrait::ConstantLike>();
+}
+
+bool isRecordedDeferredCommunicationSource(Operation* op, Value value) {
+  auto transfer = dyn_cast<spatial::SpatDeferredCommunicationOp>(op);
+  return transfer && llvm::is_contained(transfer.getSources(), value);
+}
+
+template <typename ComputeOpTy>
+void verifyComputeBodyCaptures(ComputeOpTy compute, StringRef kind, pim::CappedDiagnosticReporter& diagnostics) {
+  Region& body = compute.getBody();
+  body.walk([&](Operation* nestedOp) {
+    for (OpOperand& operand : nestedOp->getOpOperands()) {
+      Value value = operand.get();
+      if (isLegalExternalCapture(value, body) || isRecordedDeferredCommunicationSource(nestedOp, value))
+        continue;
+
+      Operation* definingOp = value.getDefiningOp();
+      diagnostics.report(compute.getOperation(), [&](Operation* illegalOp) {
+        InFlightDiagnostic diag =
+          illegalOp->emitOpError() << kPhaseMarker << " " << kind << " body captures non-constant external operand #"
+                                   << operand.getOperandNumber() << " used by " << nestedOp->getName().getStringRef();
+        diag << " (type " << value.getType() << ")";
+        if (definingOp)
+          diag.attachNote(definingOp->getLoc()) << "defining op is " << definingOp->getName().getStringRef();
+        else if (auto blockArg = dyn_cast<BlockArgument>(value)) {
+          if (Operation* owner = blockArg.getOwner()->getParentOp())
+            diag.attachNote(owner->getLoc())
+              << "external block argument belongs to " << owner->getName().getStringRef();
+        }
+      });
+    }
+  });
 }
 
 bool isLegalHostBackedValue(Value value) {
   Operation* definingOp = value.getDefiningOp();
   if (!definingOp)
     return isa<BlockArgument>(value);
-
-  if (isa<spatial::SpatChannelReceiveOp>(definingOp))
-    return false;
-
   return definingOp->getDialect()->getNamespace() != "spat";
 }
 
-LogicalResult verifyComputeLikeInputs(Operation* computeLikeOp,
+bool isScheduledPhase1Value(Value value) {
-                                      ValueRange inputs,
+  Operation* definingOp = value.getDefiningOp();
+  return isa_and_nonnull<spatial::SpatScheduledCompute, spatial::SpatScheduledComputeBatch>(definingOp);
+}
+
+template <typename ComputeOpTy>
+void verifyScheduledInputs(ComputeOpTy compute,
                            bool allowChannelReceiveInputs,
                            StringRef kind,
                            pim::CappedDiagnosticReporter& diagnostics) {
-  for (auto [inputIndex, input] : llvm::enumerate(inputs)) {
+  for (auto [inputIndex, input] : llvm::enumerate(compute.getInputs())) {
-    unsigned currentInputIndex = inputIndex;
+    size_t currentInputIndex = inputIndex;
     Operation* definingOp = input.getDefiningOp();
     if (allowChannelReceiveInputs && isa_and_nonnull<spatial::SpatChannelReceiveOp>(definingOp))
       continue;
+    if (isScheduledPhase1Value(input))
+      continue;
     if (isLegalHostBackedValue(input))
       continue;
 
-    diagnostics.report(computeLikeOp, [&](Operation* illegalOp) {
+    diagnostics.report(compute.getOperation(), [&](Operation* illegalOp) {
-      InFlightDiagnostic diagnostic = illegalOp->emitOpError()
+      InFlightDiagnostic diag = illegalOp->emitOpError()
-                                   << kind << " input #" << currentInputIndex
+                                << kPhaseMarker << " " << kind << " input #" << currentInputIndex
-                                   << (allowChannelReceiveInputs ? " must come from the host or an explicit "
+                                << (allowChannelReceiveInputs ? " must come from the host or explicit spat.channel_receive"
-                                                                   "spat.channel_receive"
                                                               : " must come from the host");
       if (definingOp)
-        diagnostic.attachNote(definingOp->getLoc()) << "illegal Spatial producer is " << definingOp->getName();
+        diag.attachNote(definingOp->getLoc()) << "illegal producer is " << definingOp->getName().getStringRef();
     });
-    return failure();
   }
-  return success();
 }
 
-void verifyNoExternalTensorCaptures(Operation* ownerOp,
+template <typename ComputeOpTy>
-                                    Region& region,
+void verifyNoNestedFragmentAssemblyBlueprints(ComputeOpTy compute,
-                                    StringRef kind,
                                                   pim::CappedDiagnosticReporter& diagnostics) {
-  region.walk([&](Operation* op) {
+  compute.getBody().walk([&](spatial::SpatBlueprintOp blueprint) {
-    for (OpOperand& operand : op->getOpOperands()) {
+    std::optional<StringRef> mode = blueprint.getMode();
-      Value value = operand.get();
+    if (!mode || *mode != "fragment_assembly")
-      if (!isa<TensorType>(value.getType()))
+      return;
-        continue;
+    diagnostics.report(blueprint.getOperation(), [&](Operation* illegalOp) {
-      if (isDefinedInsideRegion(value, region) || isa<BlockArgument>(value))
+      illegalOp->emitOpError("fragment assembly blueprint must be host-level after merge materialization");
-        continue;
+    });
-
-      Operation* definingOp = value.getDefiningOp();
-      if (definingOp && definingOp->hasTrait<OpTrait::ConstantLike>())
-        continue;
-
-      diagnostics.report(ownerOp, [&](Operation* illegalOp) {
-        InFlightDiagnostic diagnostic = illegalOp->emitOpError() << kind << " body may not capture external tensor "
-                                                                 << "values";
-        diagnostic.attachNote(op->getLoc())
-          << "tensor operand #" << operand.getOperandNumber() << " is defined outside the compute body by "
-          << (definingOp ? definingOp->getName().getStringRef() : StringRef("<block argument>"));
   });
 }
+
+void verifyLogicalTopLevelOps(func::FuncOp funcOp, pim::CappedDiagnosticReporter& diagnostics) {
+  for (Operation& op : funcOp.getOps()) {
+    if (isa<func::ReturnOp,
+            spatial::SpatGraphCompute,
+            spatial::SpatGraphComputeBatch,
+            spatial::SpatConv2DPlanOp,
+            spatial::SpatBiasAddPlanOp,
+            spatial::SpatReluPlanOp,
+            spatial::SpatMaxPool2DPlanOp,
+            spatial::SpatBlueprintOp,
+            spatial::SpatMaterializeLayoutOp>(&op)) {
+      continue;
+    }
+    if (isa<spatial::SpatScheduledCompute, spatial::SpatScheduledComputeBatch>(&op)) {
+      diagnostics.report(&op, [&](Operation* illegalOp) {
+        illegalOp->emitOpError() << kPhaseMarker << " scheduled Spatial compute op is not allowed in logical graph phase";
       });
+      continue;
+    }
+    if (isa<spatial::SpatChannelReceiveOp, spatial::SpatChannelSendOp>(&op)) {
+      diagnostics.report(&op, [&](Operation* illegalOp) {
+        illegalOp->emitOpError() << kPhaseMarker
+                                 << " explicit channel communication is not expected before merge materialization";
+      });
+      continue;
+    }
+    if (isCompileTimeOp(&op))
+      continue;
+
+    diagnostics.report(&op, [&](Operation* illegalOp) {
+      illegalOp->emitOpError()
+        << kPhaseMarker << " non-foldable top-level runtime op remains in logical Spatial graph; lower it inside spat.graph_compute";
+    });
+  }
+}
+
+void verifyScheduledTopLevelOps(func::FuncOp funcOp, pim::CappedDiagnosticReporter& diagnostics) {
+  for (Operation& op : funcOp.getOps()) {
+    if (isa<spatial::SpatChannelSendOp, spatial::SpatChannelReceiveOp>(&op)) {
+      diagnostics.report(&op, [&](Operation* illegalOp) {
+        illegalOp->emitOpError() << kPhaseMarker << " real channel communication is not allowed in scheduled phase 1";
+      });
+    }
+  }
 }
 
 } // namespace
 
-LogicalResult verifyONNXToSpatial(func::FuncOp funcOp) {
+LogicalResult verifyNoComputeBodyCaptures(func::FuncOp funcOp) {
   pim::CappedDiagnosticReporter diagnostics;
-
+  for (auto compute : funcOp.getOps<spatial::SpatGraphCompute>())
-  for (Operation& op : funcOp.getOps()) {
+    verifyComputeBodyCaptures(compute, "graph_compute", diagnostics);
-    if (isa<func::ReturnOp, spatial::SpatCompute, spatial::SpatComputeBatch>(&op))
+  for (auto batch : funcOp.getOps<spatial::SpatGraphComputeBatch>())
-      continue;
+    verifyComputeBodyCaptures(batch, "graph_compute_batch", diagnostics);
-    if (isCompileTimeOp(&op))
+  for (auto compute : funcOp.getOps<spatial::SpatScheduledCompute>())
-      continue;
+    verifyComputeBodyCaptures(compute, "scheduled_compute", diagnostics);
-
+  for (auto batch : funcOp.getOps<spatial::SpatScheduledComputeBatch>())
-    diagnostics.report(&op, [](Operation* illegalOp) {
+    verifyComputeBodyCaptures(batch, "scheduled_compute_batch", diagnostics);
-      illegalOp->emitOpError(
+  diagnostics.emitSuppressedSummary(funcOp, "compute body capture verification failed");
-        "non-foldable top-level runtime op remains after ONNX-to-Spatial; lower it inside spat.compute");
-    });
-  }
-  checkWeightUseChains(funcOp, diagnostics);
-  diagnostics.emitSuppressedSummary(funcOp, "ONNX-to-Spatial verification failed");
-
   return success(!diagnostics.hasFailure());
 }
 
-LogicalResult verifySpatialCommunicationInvariants(func::FuncOp funcOp) {
+LogicalResult verifyONNXToSpatial(func::FuncOp funcOp) { return verifyLogicalSpatialGraphInvariants(funcOp); }
+
+LogicalResult verifyLogicalSpatialGraphInvariants(func::FuncOp funcOp) {
   pim::CappedDiagnosticReporter diagnostics;
-
+  verifyLogicalTopLevelOps(funcOp, diagnostics);
-  for (auto computeOp : funcOp.getOps<spatial::SpatCompute>()) {
+  checkWeightUseChains(funcOp, diagnostics);
-    (void) verifyComputeLikeInputs(
+  if (failed(verifyNoComputeBodyCaptures(funcOp)))
-      computeOp.getOperation(), computeOp.getInputs(), /*allowChannelReceiveInputs=*/true, "spat.compute", diagnostics);
+    return failure();
-    verifyNoExternalTensorCaptures(computeOp.getOperation(), computeOp.getBody(), "spat.compute", diagnostics);
+  diagnostics.emitSuppressedSummary(funcOp, "logical Spatial graph verification failed");
+  return success(!diagnostics.hasFailure());
 }
 
-  for (auto computeBatchOp : funcOp.getOps<spatial::SpatComputeBatch>()) {
+LogicalResult verifyScheduledSpatialInvariants(func::FuncOp funcOp) {
-    (void) verifyComputeLikeInputs(computeBatchOp.getOperation(),
+  pim::CappedDiagnosticReporter diagnostics;
-                                   computeBatchOp.getInputs(),
+  verifyScheduledTopLevelOps(funcOp, diagnostics);
-                                   /*allowChannelReceiveInputs=*/false,
+  for (auto compute : funcOp.getOps<spatial::SpatScheduledCompute>()) {
-                                   "spat.compute_batch",
+    verifyScheduledInputs(compute, /*allowChannelReceiveInputs=*/true, "spat.scheduled_compute", diagnostics);
-                                   diagnostics);
+    verifyNoNestedFragmentAssemblyBlueprints(compute, diagnostics);
-    verifyNoExternalTensorCaptures(
-      computeBatchOp.getOperation(), computeBatchOp.getBody(), "spat.compute_batch", diagnostics);
   }
-
+  for (auto batch : funcOp.getOps<spatial::SpatScheduledComputeBatch>()) {
-  diagnostics.emitSuppressedSummary(funcOp, "Spatial communication invariant verification failed");
+    verifyScheduledInputs(batch, /*allowChannelReceiveInputs=*/false, "spat.scheduled_compute_batch", diagnostics);
+    verifyNoNestedFragmentAssemblyBlueprints(batch, diagnostics);
+  }
+  if (failed(verifyNoComputeBodyCaptures(funcOp)))
+    return failure();
+  diagnostics.emitSuppressedSummary(funcOp, "scheduled Spatial verification failed");
   return success(!diagnostics.hasFailure());
 }
 

src/PIM/Conversion/ONNXToSpatial/ONNXToSpatialVerifier.hpp

@@ -6,6 +6,8 @@
 namespace onnx_mlir {
 
 mlir::LogicalResult verifyONNXToSpatial(mlir::func::FuncOp funcOp);
-mlir::LogicalResult verifySpatialCommunicationInvariants(mlir::func::FuncOp funcOp);
+mlir::LogicalResult verifyNoComputeBodyCaptures(mlir::func::FuncOp funcOp);
+mlir::LogicalResult verifyLogicalSpatialGraphInvariants(mlir::func::FuncOp funcOp);
+mlir::LogicalResult verifyScheduledSpatialInvariants(mlir::func::FuncOp funcOp);
 
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns.cpp

@@ -19,6 +19,7 @@ void populateConversionPatterns(RewritePatternSet& patterns, MLIRContext* ctx) {
   populateSigmoidPatterns(patterns, ctx);
   populateSoftmaxPatterns(patterns, ctx);
   populateConcatPatterns(patterns, ctx);
+  populateFlattenPatterns(patterns, ctx);
   populateGatherPatterns(patterns, ctx);
   populateResizePatterns(patterns, ctx);
   populateReshapePatterns(patterns, ctx);

src/PIM/Conversion/ONNXToSpatial/Patterns.hpp

@@ -26,6 +26,7 @@ void populateReluPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext*
 void populateSigmoidPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateSoftmaxPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateConcatPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
+void populateFlattenPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateGatherPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateResizePatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateReshapePatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
@@ -33,8 +34,8 @@ void populateSlicePatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext*
 void populateSplitPatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 void populateTransposePatterns(mlir::RewritePatternSet& patterns, mlir::MLIRContext* ctx);
 
-bool requiresPostRewrite(spatial::SpatCompute computeOp);
+bool requiresPostRewrite(spatial::SpatGraphCompute computeOp);
-bool requiresPostRewrite(spatial::SpatComputeBatch computeOp);
+bool requiresPostRewrite(spatial::SpatGraphComputeBatch computeOp);
 void annotateWeightsConstants(mlir::func::FuncOp funcOp);
 
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/ConvGeometry.cpp

@@ -0,0 +1,77 @@
+#include "ConvGeometry.hpp"
+
+#include <algorithm>
+
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
+
+namespace onnx_mlir {
+
+bool isDepthwiseConv(int64_t group, int64_t numChannelsIn, int64_t numChannelsOut, int64_t numChannelsInPerGroup) {
+  return group == numChannelsIn && numChannelsInPerGroup == 1 && numChannelsOut % group == 0;
+}
+
+ConvGeometry buildConvGeometry(const ConvLoweringState& state) {
+  ConvGeometry geo {
+    state.batchSize,
+    state.numChannelsIn,
+    state.xHeight,
+    state.xWidth,
+    state.numChannelsOut,
+    state.wHeight,
+    state.wWidth,
+    state.outHeight,
+    state.outWidth,
+    state.group,
+    state.numChannelsInPerGroup,
+    state.numChannelsOutPerGroup,
+    state.numChannelsInPerGroup * state.wHeight * state.wWidth,
+    state.numChannelsOutPerGroup,
+    state.batchSize * state.outHeight * state.outWidth,
+    static_cast<int64_t>(crossbarSize.getValue()),
+    1,
+    0,
+    state.hasBias,
+    isDepthwiseConv(state.group, state.numChannelsIn, state.numChannelsOut, state.numChannelsInPerGroup),
+  };
+  geo.pack = std::max<int64_t>(1, geo.xbarSize / std::max<int64_t>(geo.k, geo.c));
+  geo.im2colElements = static_cast<uint64_t>(std::max<int64_t>(0, geo.p)) * static_cast<uint64_t>(std::max<int64_t>(0, geo.k));
+  return geo;
+}
+
+uint64_t chooseStreamChunkPositions(const ConvGeometry& geo, int64_t packFactor) {
+  const uint64_t patchElements = static_cast<uint64_t>(std::max<int64_t>(1, geo.k));
+  uint64_t chunkPositions = std::max<uint64_t>(1, pimConvIm2colMaxElements / patchElements);
+  chunkPositions = std::min<uint64_t>(chunkPositions, static_cast<uint64_t>(std::max<int64_t>(1, geo.p)));
+  chunkPositions = std::min<uint64_t>(chunkPositions, std::max<uint64_t>(1, pimConvStreamChunkPositions));
+
+  if (packFactor > 1 && chunkPositions > static_cast<uint64_t>(packFactor)) {
+    chunkPositions -= chunkPositions % static_cast<uint64_t>(packFactor);
+    chunkPositions = std::max<uint64_t>(chunkPositions, static_cast<uint64_t>(packFactor));
+  }
+  return std::max<uint64_t>(1, chunkPositions);
+}
+
+RowInterval computeConvInputRowsForOutputRows(RowInterval outputRows, const ConvLoweringState& state) {
+  const int64_t rawBegin = outputRows.begin * state.strideHeight - state.padHeightBegin;
+  const int64_t rawEnd =
+    (outputRows.end - 1) * state.strideHeight - state.padHeightBegin + state.dilationHeight * (state.wHeight - 1) + 1;
+  return {std::max<int64_t>(0, rawBegin), std::min<int64_t>(state.xHeight, rawEnd)};
+}
+
+ConvRowDemand buildConvRowDemand(RowInterval outputRows, const ConvLoweringState& state) {
+  ConvRowDemand demand;
+  demand.outputRows = outputRows;
+  demand.neededInputRows = computeConvInputRowsForOutputRows(outputRows, state);
+  demand.acquiredInputRows = demand.neededInputRows;
+
+  const int64_t rawBegin = outputRows.begin * state.strideHeight - state.padHeightBegin;
+  const int64_t rawEnd =
+    (outputRows.end - 1) * state.strideHeight - state.padHeightBegin + state.dilationHeight * (state.wHeight - 1) + 1;
+  demand.topHaloRows = std::max<int64_t>(0, -rawBegin);
+  demand.bottomHaloRows = std::max<int64_t>(0, rawEnd - state.xHeight);
+  demand.acquiredInputRows = demand.neededInputRows;
+  return demand;
+}
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/ConvGeometry.hpp

@@ -0,0 +1,86 @@
+#pragma once
+
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Value.h"
+
+#include <cstdint>
+
+namespace onnx_mlir {
+
+struct ConvLoweringState {
+  mlir::Value x;
+  mlir::Value w;
+  mlir::Value b;
+  mlir::RankedTensorType xType;
+  mlir::RankedTensorType wType;
+  mlir::RankedTensorType outType;
+  int64_t batchSize;
+  int64_t numChannelsIn;
+  int64_t xHeight;
+  int64_t xWidth;
+  int64_t numChannelsOut;
+  int64_t wHeight;
+  int64_t wWidth;
+  int64_t outHeight;
+  int64_t outWidth;
+  int64_t group;
+  int64_t numChannelsInPerGroup;
+  int64_t numChannelsOutPerGroup;
+  int64_t padHeightBegin;
+  int64_t padHeightEnd;
+  int64_t padWidthBegin;
+  int64_t padWidthEnd;
+  int64_t strideHeight;
+  int64_t strideWidth;
+  int64_t dilationHeight;
+  int64_t dilationWidth;
+  bool hasBias;
+};
+
+struct ConvGeometry {
+  int64_t batchSize;
+  int64_t numChannelsIn;
+  int64_t xHeight;
+  int64_t xWidth;
+  int64_t numChannelsOut;
+  int64_t wHeight;
+  int64_t wWidth;
+  int64_t outHeight;
+  int64_t outWidth;
+  int64_t group;
+  int64_t numChannelsInPerGroup;
+  int64_t numChannelsOutPerGroup;
+  int64_t k;
+  int64_t c;
+  int64_t p;
+  int64_t xbarSize;
+  int64_t pack;
+  uint64_t im2colElements;
+  bool hasBias;
+  bool isDepthwise;
+};
+
+struct RowInterval {
+  int64_t begin = 0;
+  int64_t end = 0;
+};
+
+struct ConvRowDemand {
+  RowInterval outputRows;
+  RowInterval neededInputRows;
+  RowInterval acquiredInputRows;
+  int64_t topHaloRows = 0;
+  int64_t bottomHaloRows = 0;
+};
+
+bool isDepthwiseConv(int64_t group, int64_t numChannelsIn, int64_t numChannelsOut, int64_t numChannelsInPerGroup);
+
+ConvGeometry buildConvGeometry(const ConvLoweringState& state);
+
+uint64_t chooseStreamChunkPositions(const ConvGeometry& geo, int64_t packFactor);
+
+RowInterval computeConvInputRowsForOutputRows(RowInterval outputRows, const ConvLoweringState& state);
+
+ConvRowDemand buildConvRowDemand(RowInterval outputRows, const ConvLoweringState& state);
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/Elementwise.cpp

@@ -5,7 +5,7 @@
 
 #include "llvm/ADT/SmallVector.h"
 
-#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
@@ -47,38 +47,28 @@ static FailureOr<Value> materializeBroadcastedConstantTensor(Value value,
     return failure();
 
   const int64_t rankOffset = static_cast<int64_t>(resultShape.size() - sourceShape.size());
-  for (int64_t i = 0; i < static_cast<int64_t>(resultShape.size()); ++i) {
-    const int64_t sourceIndex = i - rankOffset;
-    const int64_t sourceDim = sourceIndex < 0 ? 1 : sourceShape[sourceIndex];
-    const int64_t resultDim = resultShape[i];
-    if (sourceDim != 1 && sourceDim != resultDim)
-      return failure();
-  }
-
-  SmallVector<Attribute> sourceValues(denseAttr.getValues<Attribute>());
   SmallVector<int64_t> sourceStrides = computeRowMajorStrides(sourceShape);
   SmallVector<int64_t> resultStrides = computeRowMajorStrides(resultShape);
-
+  SmallVector<Attribute> sourceValues(denseAttr.getValues<Attribute>());
   SmallVector<Attribute> resultValues;
   resultValues.reserve(resultType.getNumElements());
 
   for (int64_t flatIndex = 0; flatIndex < resultType.getNumElements(); ++flatIndex) {
     int64_t remaining = flatIndex;
     int64_t sourceFlatIndex = 0;
-
     for (int64_t i = 0; i < static_cast<int64_t>(resultShape.size()); ++i) {
       const int64_t resultIndex = resultStrides.empty() ? 0 : remaining / resultStrides[i];
       remaining = resultStrides.empty() ? 0 : remaining % resultStrides[i];
-
       const int64_t sourceIndex = i - rankOffset;
       if (sourceIndex < 0)
         continue;
-
       const int64_t sourceDim = sourceShape[sourceIndex];
+      const int64_t resultDim = resultShape[i];
+      if (sourceDim != 1 && sourceDim != resultDim)
+        return failure();
       const int64_t mappedIndex = sourceDim == 1 ? 0 : resultIndex;
       sourceFlatIndex += mappedIndex * sourceStrides[sourceIndex];
     }
-
     resultValues.push_back(sourceValues[sourceFlatIndex]);
   }
 
@@ -106,7 +96,7 @@ static FailureOr<Value> materializeReciprocalTensor(Value value,
   if (failed(broadcastedValue))
     return failure();
 
-  auto denseAttr = dyn_cast<DenseFPElementsAttr>(getDenseConstantAttr(*broadcastedValue));
+  auto denseAttr = dyn_cast<DenseFPElementsAttr>(getHostConstDenseElementsAttr(*broadcastedValue));
   if (!denseAttr)
     return failure();
 
@@ -185,10 +175,45 @@ struct DivToSpatialCompute : OpConversionPattern<ONNXDivOp> {
   }
 };
 
+struct AddToSpatialCompute : OpConversionPattern<ONNXAddOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(ONNXAddOp op, ONNXAddOpAdaptor adaptor, ConversionPatternRewriter& rewriter) const override {
+    auto resultType = dyn_cast<RankedTensorType>(op.getResult().getType());
+    if (!resultType || !resultType.hasStaticShape())
+      return failure();
+
+    FailureOr<BiasAddPlanCandidate> candidate =
+      classifyBiasAddPlanCandidate(adaptor.getA(), adaptor.getB(), resultType);
+    if (succeeded(candidate)) {
+      auto plan = spatial::SpatBiasAddPlanOp::create(
+        rewriter, op.getLoc(), resultType, candidate->data, candidate->bias, rewriter.getStringAttr("nchw"));
+      rewriter.replaceOp(op, plan.getResult());
+      return success();
+    }
+
+    auto lhs = prepareElementwiseOperand(adaptor.getA(), resultType, rewriter, op.getLoc());
+    if (failed(lhs))
+      return failure();
+    auto rhs = prepareElementwiseOperand(adaptor.getB(), resultType, rewriter, op.getLoc());
+    if (failed(rhs))
+      return failure();
+
+    auto computeOp =
+      createSpatCompute<2>(rewriter, op.getLoc(), resultType, {}, ValueRange {*lhs, *rhs}, [&](Value x, Value y) {
+        auto loweredOp = spatial::SpatVAddOp::create(rewriter, op.getLoc(), resultType, x, y);
+        spatial::SpatYieldOp::create(rewriter, op.getLoc(), loweredOp.getResult());
+      });
+    rewriter.replaceOp(op, computeOp);
+    return success();
+  }
+};
+
 } // namespace
 
 void populateElementwisePatterns(RewritePatternSet& patterns, MLIRContext* ctx) {
-  patterns.add<BinaryElementwiseToSpatialCompute<ONNXAddOp, spatial::SpatVAddOp>>(ctx);
+  patterns.add<AddToSpatialCompute>(ctx);
   patterns.add<BinaryElementwiseToSpatialCompute<ONNXSubOp, spatial::SpatVSubOp>>(ctx);
   patterns.add<BinaryElementwiseToSpatialCompute<ONNXMulOp, spatial::SpatVMulOp>>(ctx);
   patterns.add<DivToSpatialCompute>(ctx);

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/Gemm.cpp

@@ -17,6 +17,7 @@
 #include "src/Accelerators/PIM/Common/IR/AffineUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/TensorSliceUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Accelerators/PIM/Common/Support/Diagnostics.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
@@ -87,28 +88,6 @@ static Value createGemmBatchHOffset(Value lane,
                                  rewriter.getInsertionBlock()->getParentOp());
 }
 
-static Value
-createZeroPaddedTensor(Value value, RankedTensorType resultType, ConversionPatternRewriter& rewriter, Location loc) {
-  auto sourceType = cast<RankedTensorType>(value.getType());
-  SmallVector<OpFoldResult> lowPads(sourceType.getRank(), rewriter.getIndexAttr(0));
-  SmallVector<OpFoldResult> highPads;
-  highPads.reserve(sourceType.getRank());
-  for (auto [sourceDim, resultDim] : llvm::zip(sourceType.getShape(), resultType.getShape()))
-    highPads.push_back(rewriter.getIndexAttr(resultDim - sourceDim));
-
-  auto padOp = tensor::PadOp::create(rewriter, loc, resultType, value, lowPads, highPads);
-  auto* padBlock = new Block();
-  for (int64_t i = 0; i < sourceType.getRank(); ++i)
-    padBlock->addArgument(rewriter.getIndexType(), loc);
-  padOp.getRegion().push_back(padBlock);
-  rewriter.setInsertionPointToStart(padBlock);
-  auto zero = getOrCreateConstant(
-    rewriter, padOp.getOperation(), rewriter.getZeroAttr(sourceType.getElementType()), sourceType.getElementType());
-  tensor::YieldOp::create(rewriter, loc, zero);
-  rewriter.setInsertionPointAfter(padOp);
-  return padOp.getResult();
-}
-
 static FailureOr<Value> materializePaddedConstantMatrix(Value value,
                                                         RankedTensorType resultType,
                                                         ConversionPatternRewriter& rewriter,
@@ -232,22 +211,6 @@ static Value extractATile(
   return tensor::ExtractSliceOp::create(rewriter, loc, aTileType, a, offsets, sizes, strides).getResult();
 }
 
-static Value createPaddedInputCompute(Value input,
-                                      RankedTensorType paddedInputType,
-                                      ConversionPatternRewriter& rewriter,
-                                      Location loc) {
-  auto inputType = cast<RankedTensorType>(input.getType());
-  if (inputType == paddedInputType)
-    return input;
-
-  auto computeOp = createSpatCompute<1>(rewriter, loc, TypeRange {paddedInputType}, {}, input, [&](Value computeInput) {
-    Value paddedInput = createZeroPaddedTensor(computeInput, paddedInputType, rewriter, loc);
-    spatial::SpatYieldOp::create(rewriter, loc, paddedInput);
-  });
-
-  return computeOp.getResult(0);
-}
-
 static FailureOr<spatial::SpatComputeBatch> createVmmBatch(Value a,
                                                            Value b,
                                                            RankedTensorType aType,
@@ -285,15 +248,11 @@ static FailureOr<spatial::SpatComputeBatch> createVmmBatch(Value a,
       SmallVector<OpFoldResult> bSizes {rewriter.getIndexAttr(crossbarSize.getValue()),
                                         rewriter.getIndexAttr(crossbarSize.getValue())};
       SmallVector<OpFoldResult> unitStrides = getUnitStrides(rewriter, 2);
-      Value bTile =
+      Value bTile = extractStaticSliceOrIdentity(
-        tensor::ExtractSliceOp::create(rewriter, loc, bTileType, args.weights.front(), bOffsets, bSizes, unitStrides)
+        rewriter, loc, args.weights.front(), bTileType, bOffsets, bSizes, unitStrides);
-          .getResult();
       Value piece = spatial::SpatVMMOp::create(rewriter, loc, pieceType, bTile, aTile).getResult();
 
-      SmallVector<OpFoldResult> pieceOffsets {args.lane, rewriter.getIndexAttr(0)};
+      publishGraphBatchPhysicalFragment(rewriter, loc, piece, args.outputs.front(), args.lane);
-      SmallVector<OpFoldResult> pieceSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(crossbarSize.getValue())};
-      createParallelInsertSliceIntoBatchOutput(
-        rewriter, loc, piece, args.outputs.front(), pieceOffsets, pieceSizes, unitStrides);
     });
   if (failed(batchOp))
     return failure();
@@ -440,11 +399,7 @@ static FailureOr<spatial::SpatComputeBatch> createVvdmulBatch(Value a,
       Value bVector = extractDynamicGemmBColumn(args.inputs[1], column, vectorType, rewriter, loc);
       Value scalar = spatial::SpatVVDMulOp::create(rewriter, loc, scalarType, aVector, bVector).getResult();
 
-      SmallVector<OpFoldResult> outputOffsets {args.lane, rewriter.getIndexAttr(0)};
+      publishGraphBatchPhysicalFragment(rewriter, loc, scalar, args.outputs.front(), args.lane);
-      SmallVector<OpFoldResult> scalarSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
-      SmallVector<OpFoldResult> unitStrides = getUnitStrides(rewriter, 2);
-      createParallelInsertSliceIntoBatchOutput(
-        rewriter, loc, scalar, args.outputs.front(), outputOffsets, scalarSizes, unitStrides);
     });
   if (failed(batchOp))
     return failure();
@@ -486,15 +441,14 @@ static FailureOr<spatial::SpatCompute> createDynamicGemmOutputCompute(Value scal
         Value row = createDynamicGemmBatchRow(lane, numOutCols, rewriter, nestedLoc);
         Value column =
           onnx_mlir::affineModConst(rewriter, nestedLoc, lane, numOutCols, rewriter.getInsertionBlock()->getParentOp());
-        SmallVector<OpFoldResult> scalarOffsets {lane, rewriter.getIndexAttr(0)};
         SmallVector<OpFoldResult> scalarSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
         SmallVector<OpFoldResult> unitStrides {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
-        Value scalar = tensor::ExtractSliceOp::create(
+        FailureOr<Value> scalar = extractGraphBatchPhysicalFragment(rewriter, nestedLoc, pieces, lane, scalarType);
-                         rewriter, nestedLoc, scalarType, pieces, scalarOffsets, scalarSizes, unitStrides)
+        if (failed(scalar))
-                         .getResult();
+          return failure();
         if (alpha != 1.0f) {
           Value alphaTensor = createScalarTensorConstant(scalarType, alpha, rewriter, nestedLoc);
-          scalar = spatial::SpatVMulOp::create(rewriter, nestedLoc, scalarType, scalar, alphaTensor).getResult();
+          *scalar = spatial::SpatVMulOp::create(rewriter, nestedLoc, scalarType, *scalar, alphaTensor).getResult();
         }
         if (biasArg) {
           Value biasScalar =
@@ -504,11 +458,11 @@ static FailureOr<spatial::SpatCompute> createDynamicGemmOutputCompute(Value scal
             biasScalar =
               spatial::SpatVMulOp::create(rewriter, nestedLoc, scalarType, biasScalar, betaTensor).getResult();
           }
-          scalar = spatial::SpatVAddOp::create(rewriter, nestedLoc, scalarType, scalar, biasScalar).getResult();
+          *scalar = spatial::SpatVAddOp::create(rewriter, nestedLoc, scalarType, *scalar, biasScalar).getResult();
         }
         SmallVector<OpFoldResult> outputOffsets {row, column};
         Value outputNext =
-          tensor::InsertSliceOp::create(rewriter, nestedLoc, scalar, outputAcc, outputOffsets, scalarSizes, unitStrides)
+          tensor::InsertSliceOp::create(rewriter, nestedLoc, *scalar, outputAcc, outputOffsets, scalarSizes, unitStrides)
             .getResult();
         yielded.push_back(outputNext);
         return success();
@@ -544,14 +498,13 @@ static Value extractReductionPiece(Value partialPiecesArg,
                                    int64_t numOutRows,
                                    ConversionPatternRewriter& rewriter,
                                    Location loc) {
-  SmallVector<OpFoldResult> unitStrides {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
+  SmallVector<OpFoldResult> unitStrides {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
-  SmallVector<OpFoldResult> pieceSizes {rewriter.getIndexAttr(numOutRows),
+  SmallVector<OpFoldResult> pieceSizes {rewriter.getIndexAttr(numOutRows), rewriter.getIndexAttr(1), rewriter.getIndexAttr(crossbarSize.getValue())};
-                                        rewriter.getIndexAttr(crossbarSize.getValue())};
   SmallVector<OpFoldResult> pieceOffsets {
-    createPartialGroupOffset(hSlice, kSlice, numKSlices, numOutRows, rewriter, loc), rewriter.getIndexAttr(0)};
+    createPartialGroupOffset(hSlice, kSlice, numKSlices, numOutRows, rewriter, loc), rewriter.getIndexAttr(0), rewriter.getIndexAttr(0)};
-  return tensor::ExtractSliceOp::create(
+  return extractMixedSliceOrIdentity(
-           rewriter, loc, pieceType, partialPiecesArg, pieceOffsets, pieceSizes, unitStrides)
+    rewriter, loc, partialPiecesArg, pieceType,
-    .getResult();
+    {pieceOffsets, pieceSizes, unitStrides});
 }
 
 static Value reducePartialPiecesForHSlice(Value partialPiecesArg,
@@ -769,7 +722,7 @@ LogicalResult GemmToSpatialComputes::matchAndRewrite(ONNXGemmOp gemmOp,
       return failure();
     }
 
-    auto scalarPiecesType = RankedTensorType::get({laneCount64, 1}, outType.getElementType());
+    auto scalarPiecesType = spatial::getGraphBatchPhysicalResultType(laneCount64, RankedTensorType::get({1, 1}, outType.getElementType()));
     auto batchOp = createVvdmulBatch(a, b, aType, bType, scalarPiecesType, outType, rewriter, loc);
     if (failed(batchOp))
       return failure();
@@ -841,8 +794,8 @@ LogicalResult GemmToSpatialComputes::matchAndRewrite(ONNXGemmOp gemmOp,
     return failure();
   }
 
-  auto partialPiecesType =
+  auto partialPiecesType = spatial::getGraphBatchPhysicalResultType(
-    RankedTensorType::get({laneCount64, static_cast<int64_t>(crossbarSize.getValue())}, outType.getElementType());
+    laneCount64, RankedTensorType::get({1, static_cast<int64_t>(crossbarSize.getValue())}, outType.getElementType()));
   auto batchOp =
     createVmmBatch(a, b, aType, paddedBType, partialPiecesType, numOutRows, numKSlices, numOutHSlices, rewriter, loc);
   if (failed(batchOp))

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/MatMul.cpp

@@ -9,6 +9,7 @@
 
 #include "src/Accelerators/PIM/Common/IR/AffineUtils.hpp"
 #include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/TensorSliceUtils.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
@@ -255,42 +256,6 @@ static Value transposeLastTwoDims(Value value, PatternRewriter& rewriter, Locati
   return createONNXTranspose(resultType, {0, 2, 1});
 }
 
-static Value createZeroPaddedTensor(Value value, RankedTensorType resultType, PatternRewriter& rewriter, Location loc) {
-  auto sourceType = cast<RankedTensorType>(value.getType());
-  SmallVector<OpFoldResult> lowPads(sourceType.getRank(), rewriter.getIndexAttr(0));
-  SmallVector<OpFoldResult> highPads;
-  highPads.reserve(sourceType.getRank());
-  for (auto [sourceDim, resultDim] : llvm::zip(sourceType.getShape(), resultType.getShape()))
-    highPads.push_back(rewriter.getIndexAttr(resultDim - sourceDim));
-
-  auto padOp = tensor::PadOp::create(rewriter, loc, resultType, value, lowPads, highPads);
-  auto* padBlock = new Block();
-  for (int64_t i = 0; i < sourceType.getRank(); ++i)
-    padBlock->addArgument(rewriter.getIndexType(), loc);
-  padOp.getRegion().push_back(padBlock);
-  rewriter.setInsertionPointToStart(padBlock);
-  auto zero = getOrCreateConstant(
-    rewriter, padOp.getOperation(), rewriter.getZeroAttr(sourceType.getElementType()), sourceType.getElementType());
-  tensor::YieldOp::create(rewriter, loc, zero);
-  rewriter.setInsertionPointAfter(padOp);
-  return padOp.getResult();
-}
-
-static Value createPaddedBatchedInputCompute(Value input,
-                                             RankedTensorType paddedInputType,
-                                             PatternRewriter& rewriter,
-                                             Location loc) {
-  auto inputType = cast<RankedTensorType>(input.getType());
-  if (inputType == paddedInputType)
-    return input;
-
-  auto computeOp = createSpatCompute<1>(rewriter, loc, TypeRange {paddedInputType}, {}, input, [&](Value computeInput) {
-    Value paddedInput = createZeroPaddedTensor(computeInput, paddedInputType, rewriter, loc);
-    spatial::SpatYieldOp::create(rewriter, loc, paddedInput);
-  });
-  return computeOp.getResult(0);
-}
-
 static FailureOr<Value> materializePaddedBatchedWeight(Value value,
                                                        ArrayRef<int64_t> sourceBatchShape,
                                                        ArrayRef<int64_t> targetBatchShape,
@@ -335,22 +300,14 @@ static Value extractBatchedATile(Value a,
                                  RankedTensorType aTileType,
                                  PatternRewriter& rewriter,
                                  Location loc) {
-  auto aSliceType = RankedTensorType::get({1, 1, aTileType.getDimSize(1)}, aTileType.getElementType());
   Value sourceBatchIndex =
     mapOutputBatchIndexToSourceBatchIndex(outputBatchIndex, sourceBatchShape, outputBatchShape, rewriter, loc);
   SmallVector<OpFoldResult> offsets {OpFoldResult(sourceBatchIndex), row, kOffset};
   SmallVector<OpFoldResult> sizes {
     rewriter.getIndexAttr(1), rewriter.getIndexAttr(1), rewriter.getIndexAttr(aTileType.getDimSize(1))};
-  auto slice =
+  return extractMixedSliceOrIdentity(
-    tensor::ExtractSliceOp::create(rewriter, loc, aSliceType, a, offsets, sizes, getUnitStrides(rewriter, 3));
+    rewriter, loc, a, aTileType,
-  return tensor::CollapseShapeOp::create(rewriter,
+    {offsets, sizes, getUnitStrides(rewriter, 3)});
-                                         loc,
-                                         aTileType,
-                                         slice,
-                                         SmallVector<ReassociationIndices> {
-                                           {0, 1},
-                                           {2}
-  });
 }
 
 static Value extractBatchedBTile(Value b,
@@ -362,24 +319,15 @@ static Value extractBatchedBTile(Value b,
                                  RankedTensorType bTileType,
                                  PatternRewriter& rewriter,
                                  Location loc) {
-  auto bSliceType =
-    RankedTensorType::get({1, bTileType.getDimSize(0), bTileType.getDimSize(1)}, bTileType.getElementType());
   Value sourceBatchIndex =
     mapOutputBatchIndexToSourceBatchIndex(outputBatchIndex, sourceBatchShape, outputBatchShape, rewriter, loc);
   SmallVector<OpFoldResult> offsets {OpFoldResult(sourceBatchIndex), kOffset, hOffset};
   SmallVector<OpFoldResult> sizes {rewriter.getIndexAttr(1),
                                    rewriter.getIndexAttr(bTileType.getDimSize(0)),
                                    rewriter.getIndexAttr(bTileType.getDimSize(1))};
-  auto slice =
+  return extractMixedSliceOrIdentity(
-    tensor::ExtractSliceOp::create(rewriter, loc, bSliceType, b, offsets, sizes, getUnitStrides(rewriter, 3));
+    rewriter, loc, b, bTileType,
-  return tensor::CollapseShapeOp::create(rewriter,
+    {offsets, sizes, getUnitStrides(rewriter, 3)});
-                                         loc,
-                                         bTileType,
-                                         slice,
-                                         SmallVector<ReassociationIndices> {
-                                           {0, 1},
-                                           {2}
-  });
 }
 
 static Value getBatchLaneIndex(
@@ -434,10 +382,7 @@ static FailureOr<spatial::SpatComputeBatch> createBatchedVmmBatch(Value a,
         args.weights.front(), bBatchShape, outputBatchShape, batch, kOffset, hOffset, bTileType, rewriter, loc);
       Value piece = spatial::SpatVMMOp::create(rewriter, loc, pieceType, bTile, aTile).getResult();
 
-      SmallVector<OpFoldResult> pieceOffsets {args.lane, rewriter.getIndexAttr(0)};
+      publishGraphBatchPhysicalFragment(rewriter, loc, piece, args.outputs.front(), args.lane);
-      SmallVector<OpFoldResult> pieceSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(crossbarSize.getValue())};
-      createParallelInsertSliceIntoBatchOutput(
-        rewriter, loc, piece, args.outputs.front(), pieceOffsets, pieceSizes, getUnitStrides(rewriter, 2));
     });
   if (failed(batchOp))
     return failure();
@@ -487,22 +432,14 @@ static Value extractDynamicBatchedRowVector(Value matrix,
                                             RankedTensorType vectorType,
                                             PatternRewriter& rewriter,
                                             Location loc) {
-  auto rowSliceType = RankedTensorType::get({1, 1, vectorType.getDimSize(1)}, vectorType.getElementType());
   Value sourceBatchIndex =
     mapOutputBatchIndexToSourceBatchIndex(outputBatchIndex, sourceBatchShape, outputBatchShape, rewriter, loc);
   SmallVector<OpFoldResult> offsets {OpFoldResult(sourceBatchIndex), row, rewriter.getIndexAttr(0)};
   SmallVector<OpFoldResult> sizes {
     rewriter.getIndexAttr(1), rewriter.getIndexAttr(1), rewriter.getIndexAttr(vectorType.getDimSize(1))};
-  auto rowSlice =
+  return extractMixedSliceOrIdentity(
-    tensor::ExtractSliceOp::create(rewriter, loc, rowSliceType, matrix, offsets, sizes, getUnitStrides(rewriter, 3));
+    rewriter, loc, matrix, vectorType,
-  return tensor::CollapseShapeOp::create(rewriter,
+    {offsets, sizes, getUnitStrides(rewriter, 3)});
-                                         loc,
-                                         vectorType,
-                                         rowSlice,
-                                         SmallVector<ReassociationIndices> {
-                                           {0, 1},
-                                           {2}
-  });
 }
 
 static FailureOr<spatial::SpatComputeBatch> createBatchedVvdmulBatch(Value a,
@@ -542,10 +479,7 @@ static FailureOr<spatial::SpatComputeBatch> createBatchedVvdmulBatch(Value a,
       Value bVector = extractDynamicBatchedBColumn(
         args.inputs[1], bBatchShape, outputBatchShape, batch, column, vectorType, rewriter, loc);
       Value scalar = spatial::SpatVVDMulOp::create(rewriter, loc, scalarType, aVector, bVector).getResult();
-      SmallVector<OpFoldResult> outputOffsets {args.lane, rewriter.getIndexAttr(0)};
+      publishGraphBatchPhysicalFragment(rewriter, loc, scalar, args.outputs.front(), args.lane);
-      SmallVector<OpFoldResult> scalarSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
-      createParallelInsertSliceIntoBatchOutput(
-        rewriter, loc, scalar, args.outputs.front(), outputOffsets, scalarSizes, getUnitStrides(rewriter, 2));
     });
   if (failed(batchOp))
     return failure();
@@ -561,7 +495,6 @@ static FailureOr<Value> createBatchedDynamicOutputCompute(Value scalarPieces,
   const int64_t numOutRows = outType.getDimSize(1);
   const int64_t numOutCols = outType.getDimSize(2);
   auto scalarType = RankedTensorType::get({1, 1}, outType.getElementType());
-  auto outputScalarType = RankedTensorType::get({1, 1, 1}, outType.getElementType());
 
   auto computeOp = createSpatCompute<1>(
     rewriter, loc, TypeRange {outType}, {}, ValueRange {scalarPieces}, [&](Value pieces) -> LogicalResult {
@@ -584,24 +517,15 @@ static FailureOr<Value> createBatchedDynamicOutputCompute(Value scalarPieces,
           Value batchLane = affineModConst(rewriter, nestedLoc, lane, numOutRows * numOutCols, anchorOp);
           Value row = affineFloorDivConst(rewriter, nestedLoc, batchLane, numOutCols, anchorOp);
           Value column = affineModConst(rewriter, nestedLoc, batchLane, numOutCols, anchorOp);
-          SmallVector<OpFoldResult> scalarOffsets {lane, rewriter.getIndexAttr(0)};
+          FailureOr<Value> scalar = extractGraphBatchPhysicalFragment(rewriter, nestedLoc, pieces, lane, scalarType);
-          SmallVector<OpFoldResult> scalarSizes {rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
+          if (failed(scalar))
-          Value scalar = tensor::ExtractSliceOp::create(
+            return failure();
-            rewriter, nestedLoc, scalarType, pieces, scalarOffsets, scalarSizes, getUnitStrides(rewriter, 2));
-          Value expanded = tensor::ExpandShapeOp::create(rewriter,
-                                                         nestedLoc,
-                                                         outputScalarType,
-                                                         scalar,
-                                                         SmallVector<ReassociationIndices> {
-                                                           {0},
-                                                           {1, 2}
-          });
           SmallVector<OpFoldResult> outputOffsets {batch, row, column};
           SmallVector<OpFoldResult> outputSizes = {
             rewriter.getIndexAttr(1), rewriter.getIndexAttr(1), rewriter.getIndexAttr(1)};
           Value next =
             tensor::InsertSliceOp::create(
-              rewriter, nestedLoc, expanded, outputAcc, outputOffsets, outputSizes, getUnitStrides(rewriter, 3))
+              rewriter, nestedLoc, *scalar, outputAcc, outputOffsets, outputSizes, getUnitStrides(rewriter, 3))
               .getResult();
           yielded.push_back(next);
           return success();
@@ -632,10 +556,11 @@ static Value extractBatchedReductionPiece(Value partialPiecesArg,
   Value kOffset = getOrCreateIndexConstant(rewriter, rewriter.getInsertionBlock()->getParentOp(), kSlice * numOutRows);
   Value batchAndHSlice = arith::AddIOp::create(rewriter, loc, batchOffset, hOffset);
   Value pieceOffset = arith::AddIOp::create(rewriter, loc, batchAndHSlice, kOffset);
-  SmallVector<OpFoldResult> offsets {pieceOffset, rewriter.getIndexAttr(0)};
+  SmallVector<OpFoldResult> offsets {pieceOffset, rewriter.getIndexAttr(0), rewriter.getIndexAttr(0)};
-  SmallVector<OpFoldResult> sizes {rewriter.getIndexAttr(numOutRows), rewriter.getIndexAttr(crossbarSize.getValue())};
+  SmallVector<OpFoldResult> sizes {rewriter.getIndexAttr(numOutRows), rewriter.getIndexAttr(1), rewriter.getIndexAttr(crossbarSize.getValue())};
-  return tensor::ExtractSliceOp::create(
+  return extractMixedSliceOrIdentity(
-    rewriter, loc, pieceType, partialPiecesArg, offsets, sizes, getUnitStrides(rewriter, 2));
+    rewriter, loc, partialPiecesArg, pieceType,
+    {offsets, sizes, getUnitStrides(rewriter, 3)});
 }
 
 static Value reduceBatchedPartialPiecesForHSlice(Value partialPiecesArg,
@@ -682,8 +607,6 @@ static FailureOr<Value> createBatchedReductionCompute(Value partialPieces,
       const int64_t numOutHSlices = ceilIntegerDivide(outType.getDimSize(2), crossbarSize.getValue());
       auto pieceType = RankedTensorType::get({numOutRows, static_cast<int64_t>(crossbarSize.getValue())},
                                              partialPiecesType.getElementType());
-      auto outputSliceType = RankedTensorType::get({1, numOutRows, static_cast<int64_t>(crossbarSize.getValue())},
-                                                   partialPiecesType.getElementType());
 
       Value outputInit =
         tensor::EmptyOp::create(rewriter, loc, paddedOutType.getShape(), paddedOutType.getElementType()).getResult();
@@ -713,14 +636,6 @@ static FailureOr<Value> createBatchedReductionCompute(Value partialPieces,
               Value outputAcc = hIterArgs.front();
               Value reduced = reduceBatchedPartialPiecesForHSlice(
                 partialPiecesArg, batch, hSlice, pieceType, numKSlices, numOutHSlices, numOutRows, rewriter, hLoc);
-              Value expandedReduced = tensor::ExpandShapeOp::create(rewriter,
-                                                                    hLoc,
-                                                                    outputSliceType,
-                                                                    reduced,
-                                                                    SmallVector<ReassociationIndices> {
-                                                                      {0, 1},
-                                                                      {2}
-              });
               Value hOffset = affineMulConst(
                 rewriter, hLoc, hSlice, crossbarSize.getValue(), rewriter.getInsertionBlock()->getParentOp());
               SmallVector<OpFoldResult> outputOffsets {batch, rewriter.getIndexAttr(0), hOffset};
@@ -729,7 +644,7 @@ static FailureOr<Value> createBatchedReductionCompute(Value partialPieces,
                                                      rewriter.getIndexAttr(crossbarSize.getValue())};
               Value next =
                 tensor::InsertSliceOp::create(
-                  rewriter, hLoc, expandedReduced, outputAcc, outputOffsets, outputSizes, getUnitStrides(rewriter, 3))
+                  rewriter, hLoc, reduced, outputAcc, outputOffsets, outputSizes, getUnitStrides(rewriter, 3))
                   .getResult();
               hYielded.push_back(next);
               return success();
@@ -950,7 +865,10 @@ struct MatMulToGemm : OpRewritePattern<ONNXMatMulOp> {
 
   LogicalResult matchAndRewrite(ONNXMatMulOp matmulOp, PatternRewriter& rewriter) const override {
     auto shapeInfo = analyzeMatMulShape(matmulOp);
-    if (failed(shapeInfo) || shapeInfo->lhsWasVector || shapeInfo->rhsWasVector || !shapeInfo->outputBatchShape.empty())
+    if (failed(shapeInfo) || shapeInfo->lhsWasVector || shapeInfo->rhsWasVector)
+      return failure();
+
+    if (!shapeInfo->outputBatchShape.empty())
       return failure();
 
     Location loc = matmulOp.getLoc();
@@ -991,9 +909,19 @@ struct MatMulToGemm : OpRewritePattern<ONNXMatMulOp> {
       gemmResult =
         ONNXTransposeOp::create(rewriter, loc, shapeInfo->outType, gemmResult, rewriter.getI64ArrayAttr({1, 0}))
           .getResult();
+
+    if (shapeInfo->outputBatchShape.empty()) {
       rewriter.replaceOp(matmulOp, gemmResult);
       return success();
     }
+
+    auto directOutType =
+      RankedTensorType::get({1, shapeInfo->m, shapeInfo->n}, shapeInfo->outType.getElementType(), shapeInfo->outType.getEncoding());
+    Value batchedResult = ensureBatchedTensor(gemmResult, /*batchSize=*/1, shapeInfo->m, shapeInfo->n, rewriter, loc);
+    Value finalResult = finalizeNormalizedMatMulResult(batchedResult, directOutType, *shapeInfo, rewriter, loc);
+    rewriter.replaceOp(matmulOp, finalResult);
+    return success();
+  }
 };
 
 struct MatMulBatchedToSpatialComputes : OpRewritePattern<ONNXMatMulOp> {
@@ -1040,10 +968,10 @@ struct MatMulBatchedToSpatialComputes : OpRewritePattern<ONNXMatMulOp> {
       auto paddedRhs =
         materializePaddedBatchedWeight(plan.rhs, plan.rhsBatchShape, plan.outputBatchShape, paddedRhsType, rewriter);
       if (succeeded(paddedRhs)) {
-        Value paddedLhs = createPaddedBatchedInputCompute(plan.lhs, paddedLhsType, rewriter, loc);
+        Value paddedLhs = createPaddedInputCompute(plan.lhs, paddedLhsType, rewriter, loc);
         const int64_t laneCount = plan.batch * plan.m * numKSlices * numOutHSlices;
-        auto partialPiecesType = RankedTensorType::get({laneCount, static_cast<int64_t>(crossbarSize.getValue())},
+        auto partialPiecesType = spatial::getGraphBatchPhysicalResultType(
-                                                       shapeInfo->outType.getElementType());
+          laneCount, RankedTensorType::get({1, static_cast<int64_t>(crossbarSize.getValue())}, shapeInfo->outType.getElementType()));
         auto batchOp = createBatchedVmmBatch(paddedLhs,
                                              *paddedRhs,
                                              paddedLhsType,
@@ -1084,7 +1012,8 @@ struct MatMulBatchedToSpatialComputes : OpRewritePattern<ONNXMatMulOp> {
       }
     }
     const int64_t laneCount = plan.batch * plan.m * plan.n;
-    auto scalarPiecesType = RankedTensorType::get({laneCount, 1}, shapeInfo->outType.getElementType());
+    auto scalarPiecesType = spatial::getGraphBatchPhysicalResultType(
+      laneCount, RankedTensorType::get({1, 1}, shapeInfo->outType.getElementType()));
     auto batchOp = createBatchedVvdmulBatch(plan.lhs,
                                             plan.lhsBatchShape,
                                             plan.rhs,

src/PIM/Conversion/ONNXToSpatial/Patterns/Math/ReduceMean.cpp

@@ -5,7 +5,6 @@
 #include "llvm/ADT/SmallVector.h"
 
 #include <algorithm>
-#include <numeric>
 #include <optional>
 #include <type_traits>
 
@@ -122,14 +121,6 @@ static RankedTensorType getKeepdimsType(RankedTensorType inputType, Type element
   return RankedTensorType::get(shape, elementType, inputType.getEncoding());
 }
 
-static RankedTensorType getCompactKeptType(RankedTensorType inputType, Type elementType, ArrayRef<bool> reducedAxes) {
-  SmallVector<int64_t> shape;
-  for (auto [dim, isReduced] : llvm::zip_equal(inputType.getShape(), reducedAxes))
-    if (!isReduced)
-      shape.push_back(dim);
-  return RankedTensorType::get(shape, elementType, inputType.getEncoding());
-}
-
 static RankedTensorType getReducedSliceType(RankedTensorType inputType, ArrayRef<bool> reducedAxes) {
   SmallVector<int64_t> shape;
   shape.reserve(inputType.getRank());
@@ -139,9 +130,7 @@ static RankedTensorType getReducedSliceType(RankedTensorType inputType, ArrayRef
 }
 
 static RankedTensorType getLanePackedKeepdimsType(int64_t laneCount, RankedTensorType leafType) {
-  SmallVector<int64_t> shape(leafType.getShape().begin(), leafType.getShape().end());
+  return spatial::getGraphBatchPhysicalResultType(laneCount, leafType);
-  shape.front() = laneCount;
-  return RankedTensorType::get(shape, leafType.getElementType(), leafType.getEncoding());
 }
 
 static SmallVector<int64_t> getKeptAxes(ArrayRef<bool> reducedAxes) {
@@ -191,12 +180,9 @@ static FailureOr<Value> buildReduceMeanKeepdimsBatch(Value input,
 
   SmallVector<OpFoldResult> sliceOffsets;
   SmallVector<OpFoldResult> sliceSizes;
-  SmallVector<OpFoldResult> insertOffsets;
-  SmallVector<OpFoldResult> insertSizes(inputType.getRank(), rewriter.getIndexAttr(1));
   SmallVector<OpFoldResult> unitStrides = getUnitStrides(rewriter, inputType.getRank());
   sliceOffsets.reserve(inputType.getRank());
   sliceSizes.reserve(inputType.getRank());
-  insertOffsets.reserve(inputType.getRank());
 
   auto batchOp =
     createSpatComputeBatch(rewriter,
@@ -209,7 +195,6 @@ static FailureOr<Value> buildReduceMeanKeepdimsBatch(Value input,
                              size_t keptAxisIndex = 0;
                              sliceOffsets.clear();
                              sliceSizes.clear();
-                             insertOffsets.clear();
                              for (auto [axis, isReduced] : llvm::enumerate(reducedAxes)) {
                                if (isReduced) {
                                  sliceOffsets.push_back(rewriter.getIndexAttr(0));
@@ -224,72 +209,90 @@ static FailureOr<Value> buildReduceMeanKeepdimsBatch(Value input,
                                sliceSizes.push_back(rewriter.getIndexAttr(1));
                              }
 
-                             insertOffsets.push_back(args.lane);
-                             insertOffsets.append(inputType.getRank() - 1, rewriter.getIndexAttr(0));
-
                              Value slice = tensor::ExtractSliceOp::create(
                                rewriter, loc, sliceType, args.inputs.front(), sliceOffsets, sliceSizes, unitStrides);
                              Value reduced = spatial::SpatVAvgOp::create(rewriter, loc, leafType, slice).getResult();
-                             createParallelInsertSliceIntoBatchOutput(
+                             publishGraphBatchPhysicalFragment(rewriter, loc, reduced, args.outputs.front(), args.lane);
-                               rewriter, loc, reduced, args.outputs.front(), insertOffsets, insertSizes, unitStrides);
                            });
   if (failed(batchOp))
     return failure();
   return (*batchOp).getResult(0);
 }
 
-static Value buildKeepdimsFromLanePackedBatch(Value batchValue,
+static FailureOr<Value> buildReduceMeanKeepdimsBlueprint(
-                                              RankedTensorType keepdimsType,
+    Value batchValue, RankedTensorType keepdimsType,
-                                              RankedTensorType compactKeptType,
+    ArrayRef<bool> reducedAxes, ConversionPatternRewriter& rewriter,
-                                              ArrayRef<bool> reducedAxes,
-                                              ConversionPatternRewriter& rewriter,
     Location loc) {
-  auto batchType = cast<RankedTensorType>(batchValue.getType());
+  auto batchType = dyn_cast<RankedTensorType>(batchValue.getType());
-  if (batchType == keepdimsType)
+  int64_t rank = keepdimsType.getRank();
-    return batchValue;
+  if (!batchType || !batchType.hasStaticShape()
+      || !keepdimsType.hasStaticShape()
+      || static_cast<int64_t>(reducedAxes.size()) != rank
+      || batchType.getRank() != rank + 1
+      || batchType.getElementType() != keepdimsType.getElementType())
+    return failure();
 
-  SmallVector<ReassociationIndices> collapseToFlat {{}};
+  int64_t laneCount = 1;
-  for (int64_t axis = 0; axis < batchType.getRank(); ++axis)
+  SmallVector<int64_t> keptAxes;
-    collapseToFlat.front().push_back(axis);
+  SmallVector<int64_t> keptAxisStrides;
+  for (auto [axis, isReduced] : llvm::enumerate(reducedAxes)) {
+    int64_t dim = keepdimsType.getDimSize(axis);
+    if (dim <= 0 || (isReduced && dim != 1))
+      return failure();
+    if (!isReduced)
+      keptAxes.push_back(axis);
+  }
+  keptAxisStrides.resize(keptAxes.size(), 1);
+  for (int64_t index = static_cast<int64_t>(keptAxes.size()) - 1;
+       index >= 0; --index) {
+    keptAxisStrides[index] = laneCount;
+    int64_t dim = keepdimsType.getDimSize(keptAxes[index]);
+    if (laneCount > std::numeric_limits<int64_t>::max() / dim)
+      return failure();
+    laneCount *= dim;
+  }
+  if (batchType.getDimSize(0) != laneCount
+      || llvm::any_of(batchType.getShape().drop_front(),
+                      [](int64_t dim) { return dim != 1; }))
+    return failure();
 
-  SmallVector<ReassociationIndices> expandFlatToCompact(1);
+  SmallVector<int64_t> operandIndices(laneCount, 0);
-  for (int64_t axis = 0; axis < compactKeptType.getRank(); ++axis)
+  SmallVector<int64_t> sourceSlots;
-    expandFlatToCompact.front().push_back(axis);
+  SmallVector<int64_t> sourceOffsets(laneCount, 0);
-
+  SmallVector<int64_t> fragmentOffsets;
-  SmallVector<ReassociationIndices> expandCompactToKeepdims;
+  sourceSlots.reserve(laneCount);
-  ReassociationIndices pendingLeadingReducedAxes;
+  fragmentOffsets.reserve(laneCount * rank);
+  for (int64_t lane = 0; lane < laneCount; ++lane) {
+    sourceSlots.push_back(lane);
+    size_t keptAxisIndex = 0;
     for (auto [axis, isReduced] : llvm::enumerate(reducedAxes)) {
       if (isReduced) {
-      if (expandCompactToKeepdims.empty())
+        fragmentOffsets.push_back(0);
-        pendingLeadingReducedAxes.push_back(axis);
-      else
-        expandCompactToKeepdims.back().push_back(axis);
         continue;
       }
-
+      int64_t dim = keepdimsType.getDimSize(axis);
-    expandCompactToKeepdims.emplace_back();
+      fragmentOffsets.push_back(
-    auto& group = expandCompactToKeepdims.back();
+        (lane / keptAxisStrides[keptAxisIndex]) % dim);
-    group.append(pendingLeadingReducedAxes.begin(), pendingLeadingReducedAxes.end());
+      ++keptAxisIndex;
-    pendingLeadingReducedAxes.clear();
-    group.push_back(axis);
     }
-  if (!pendingLeadingReducedAxes.empty())
+  }
-    expandCompactToKeepdims.back().append(pendingLeadingReducedAxes.begin(), pendingLeadingReducedAxes.end());
+  SmallVector<int64_t> fragmentSizes(fragmentOffsets.size(), 1);
-
+  SmallVector<int64_t> fragmentStrides(fragmentOffsets.size(), 1);
-  auto reshapeCompute =
+  return spatial::SpatBlueprintOp::create(
-    createSpatCompute<1>(rewriter, loc, TypeRange {keepdimsType}, {}, ValueRange {batchValue}, [&](Value input) {
+           rewriter, loc, keepdimsType, batchValue, ValueRange {},
-      auto flatType =
+           rewriter.getStringAttr("nchw"),
-        RankedTensorType::get({batchType.getDimSize(0)}, batchType.getElementType(), batchType.getEncoding());
+           rewriter.getStringAttr("fragmented"),
-      Value flat = tensor::CollapseShapeOp::create(rewriter, loc, flatType, input, collapseToFlat);
+           rewriter.getDenseI64ArrayAttr(fragmentOffsets),
-      Value compact = flat;
+           rewriter.getDenseI64ArrayAttr(fragmentSizes),
-      if (compactKeptType != flatType)
+           rewriter.getStringAttr("reduce_mean_keepdims_fragments"),
-        compact = tensor::ExpandShapeOp::create(rewriter, loc, compactKeptType, flat, expandFlatToCompact);
+           rewriter.getStringAttr("fragment_assembly"),
-      Value keepdims = compact;
+           rewriter.getDenseI64ArrayAttr(operandIndices),
-      if (keepdimsType != compactKeptType)
+           rewriter.getDenseI64ArrayAttr(sourceSlots),
-        keepdims = tensor::ExpandShapeOp::create(rewriter, loc, keepdimsType, compact, expandCompactToKeepdims);
+           rewriter.getDenseI64ArrayAttr(sourceOffsets),
-      spatial::SpatYieldOp::create(rewriter, loc, keepdims);
+           rewriter.getDenseI64ArrayAttr(fragmentStrides),
-    });
+           rewriter.getStringAttr("disjoint"),
-  return reshapeCompute.getResult(0);
+           rewriter.getStringAttr("complete"))
+    .getOutput();
 }
 
 static SmallVector<ReassociationIndices> buildCollapseReassociation(ArrayRef<bool> reducedAxes) {
@@ -366,26 +369,36 @@ struct ReduceMeanToSpatialCompute : OpConversionPattern<ReduceMeanOp> {
 
     Location loc = reduceMeanOp.getLoc();
     RankedTensorType leafType = getAllOnesType(inputType, resultType.getElementType());
-    RankedTensorType compactKeptType = getCompactKeptType(inputType, resultType.getElementType(), reducedAxes);
     RankedTensorType keepdimsType = getKeepdimsType(inputType, resultType.getElementType(), reducedAxes);
     int64_t laneCount = 1;
-    for (int64_t dim : compactKeptType.getShape())
+    for (auto [dim, isReduced] : llvm::zip_equal(keepdimsType.getShape(), reducedAxes)) {
+      if (isReduced)
+        continue;
+      if (dim <= 0 || laneCount > std::numeric_limits<int32_t>::max() / dim)
+        return rewriter.notifyMatchFailure(
+          reduceMeanOp, "ReduceMean physical lane count is not representable");
       laneCount *= dim;
+    }
     RankedTensorType batchType = getLanePackedKeepdimsType(laneCount, leafType);
 
     auto lanePackedKeepdims =
       buildReduceMeanKeepdimsBatch(adaptor.getData(), reducedAxes, batchType, leafType, rewriter, loc);
     if (failed(lanePackedKeepdims))
       return failure();
-    Value reducedKeepdims =
+    auto reducedKeepdims = buildReduceMeanKeepdimsBlueprint(
-      buildKeepdimsFromLanePackedBatch(*lanePackedKeepdims, keepdimsType, compactKeptType, reducedAxes, rewriter, loc);
+      *lanePackedKeepdims, keepdimsType, reducedAxes, rewriter, loc);
+    if (failed(reducedKeepdims))
+      return rewriter.notifyMatchFailure(
+        reduceMeanOp,
+        "cannot build physical-fragment ReduceMean keepdims reconstruction");
 
     if (semantics->keepdims != 0) {
-      rewriter.replaceOp(reduceMeanOp, reducedKeepdims);
+      rewriter.replaceOp(reduceMeanOp, *reducedKeepdims);
       return success();
     }
 
-    Value reduced = squeezeReducedAxes(reducedKeepdims, resultType, reducedAxes, rewriter, loc);
+    Value reduced = squeezeReducedAxes(
+      *reducedKeepdims, resultType, reducedAxes, rewriter, loc);
     rewriter.replaceOp(reduceMeanOp, reduced);
     return success();
   }

src/PIM/Conversion/ONNXToSpatial/Patterns/NN/Pool.cpp

@@ -16,6 +16,8 @@
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/PlanLowering.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
 
@@ -24,7 +26,7 @@ using namespace mlir;
 namespace onnx_mlir {
 namespace {
 
-static Value materializeTileTensor(ConversionPatternRewriter& rewriter, Location loc, Value tile) {
+static Value materializeTileTensor(PatternRewriter& rewriter, Location loc, Value tile) {
   auto tileType = cast<RankedTensorType>(tile.getType());
   Value empty = tensor::EmptyOp::create(rewriter, loc, tileType.getShape(), tileType.getElementType());
   return insertStaticSlice(rewriter, loc, tile, empty, getZeroOffsets(rewriter, tileType.getRank()));
@@ -228,6 +230,23 @@ struct PoolToSpatialComputeBase : public OpConversionPattern<PoolOp> {
       }
     }
 
+    if constexpr (std::is_same_v<PoolOp, ONNXMaxPoolSingleOutOp>) {
+      if (batchSize == 1) {
+        auto plan = spatial::SpatMaxPool2DPlanOp::create(
+          rewriter,
+          loc,
+          outType,
+          x,
+          rewriter.getDenseI64ArrayAttr({kernelHeight, kernelWidth}),
+          rewriter.getDenseI64ArrayAttr({padTop, padLeft, padBottom, padRight}),
+          rewriter.getDenseI64ArrayAttr({strideHeight, strideWidth}),
+          rewriter.getDenseI64ArrayAttr({dilationHeight, dilationWidth}),
+          rewriter.getStringAttr("nchw"));
+        rewriter.replaceOp(poolOp, plan.getResult());
+        return success();
+      }
+    }
+
     const int64_t xbarSize = static_cast<int64_t>(crossbarSize.getValue());
     const int64_t channelTileCount = (channels + xbarSize - 1) / xbarSize;
     const int64_t outputPatchCount = batchSize * outputHeight * outputWidth;
@@ -396,6 +415,220 @@ struct PoolToSpatialCompute<ONNXAveragePoolOp>
 
 } // namespace
 
+LogicalResult canLowerMaxPoolPlanToRowStrip(spatial::SpatMaxPool2DPlanOp planOp) {
+  auto inputType = dyn_cast<RankedTensorType>(planOp.getInput().getType());
+  auto outputType = dyn_cast<RankedTensorType>(planOp.getOutput().getType());
+  if (!inputType || !outputType || !inputType.hasStaticShape() || !outputType.hasStaticShape())
+    return failure();
+  if (inputType.getRank() != 4 || outputType.getRank() != 4 || inputType.getDimSize(0) != 1
+      || outputType.getDimSize(0) != 1 || inputType.getDimSize(1) != outputType.getDimSize(1))
+    return failure();
+  if (llvm::any_of(planOp.getKernelShape(), [](int64_t value) { return value <= 0; })
+      || llvm::any_of(planOp.getStrides(), [](int64_t value) { return value <= 0; })
+      || llvm::any_of(planOp.getDilations(), [](int64_t value) { return value <= 0; }))
+    return failure();
+  return success();
+}
+
+static Value createClampedPoolIndexTable(PatternRewriter& rewriter,
+                                         Operation* anchorOp,
+                                         int64_t outputSize,
+                                         int64_t kernelSize,
+                                         int64_t stride,
+                                         int64_t dilation,
+                                         int64_t padBegin,
+                                         int64_t inputSize) {
+  auto tableType = RankedTensorType::get({outputSize * kernelSize}, rewriter.getIndexType());
+  SmallVector<Attribute> values;
+  values.reserve(tableType.getNumElements());
+  for (int64_t output = 0; output < outputSize; ++output)
+    for (int64_t kernel = 0; kernel < kernelSize; ++kernel)
+      values.push_back(rewriter.getIndexAttr(
+        std::clamp(output * stride + kernel * dilation - padBegin, int64_t {0}, inputSize - 1)));
+  return getOrCreateConstant(rewriter, anchorOp, DenseElementsAttr::get(tableType, values), tableType);
+}
+
+static Value extractPoolIndex(PatternRewriter& rewriter,
+                              Location loc,
+                              Operation* anchorOp,
+                              Value table,
+                              Value outputIndex,
+                              int64_t kernelIndex,
+                              int64_t kernelSize) {
+  Value tableIndex = arith::MulIOp::create(
+    rewriter, loc, outputIndex, getOrCreateIndexConstant(rewriter, anchorOp, kernelSize));
+  if (kernelIndex != 0)
+    tableIndex = arith::AddIOp::create(
+      rewriter, loc, tableIndex, getOrCreateIndexConstant(rewriter, anchorOp, kernelIndex));
+  return tensor::ExtractOp::create(rewriter, loc, table, tableIndex);
+}
+
+FailureOr<Value> lowerSelectedMaxPool2DPlan(spatial::SpatMaxPool2DPlanOp planOp,
+                                            std::optional<Value> rowStripInput,
+                                            PatternRewriter& rewriter) {
+  if (failed(canLowerMaxPoolPlanToRowStrip(planOp)))
+    return failure();
+
+  Location loc = planOp.getLoc();
+  auto inputType = cast<RankedTensorType>(planOp.getInput().getType());
+  auto outputType = cast<RankedTensorType>(planOp.getOutput().getType());
+  const int64_t channels = inputType.getDimSize(1);
+  const int64_t inputHeight = inputType.getDimSize(2);
+  const int64_t inputWidth = inputType.getDimSize(3);
+  const int64_t outputHeight = outputType.getDimSize(2);
+  const int64_t outputWidth = outputType.getDimSize(3);
+  const int64_t kernelHeight = planOp.getKernelShape()[0];
+  const int64_t kernelWidth = planOp.getKernelShape()[1];
+  Value input = rowStripInput.value_or(planOp.getInput());
+  auto actualInputType = dyn_cast<RankedTensorType>(input.getType());
+  const bool physicalInput = actualInputType == getRowStripStorageType(inputType);
+  if (!physicalInput && actualInputType != inputType)
+    return failure();
+
+  Operation* anchorOp = rewriter.getInsertionBlock()->getParentOp();
+  Value rowTable = createClampedPoolIndexTable(rewriter,
+                                               anchorOp,
+                                               outputHeight,
+                                               kernelHeight,
+                                               planOp.getStrides()[0],
+                                               planOp.getDilations()[0],
+                                               planOp.getPads()[0],
+                                               inputHeight);
+  Value columnTable = createClampedPoolIndexTable(rewriter,
+                                                  anchorOp,
+                                                  outputWidth,
+                                                  kernelWidth,
+                                                  planOp.getStrides()[1],
+                                                  planOp.getDilations()[1],
+                                                  planOp.getPads()[1],
+                                                  inputWidth);
+  auto inputFragmentType = getRowStripFragmentType(inputType);
+  auto outputFragmentType = getRowStripFragmentType(outputType);
+  auto outputStorageType = getRowStripStorageType(outputType);
+  auto tileType = RankedTensorType::get({1, channels, 1, 1}, outputType.getElementType());
+  auto batch = createSpatComputeBatch(
+    rewriter,
+    loc,
+    TypeRange {outputStorageType},
+    outputHeight,
+    {},
+    ValueRange {input},
+    [&](detail::SpatComputeBatchBodyArgs args) -> LogicalResult {
+      SmallVector<Value> inputRows;
+      inputRows.reserve(kernelHeight);
+      for (int64_t kernelRow = 0; kernelRow < kernelHeight; ++kernelRow) {
+        Value sourceRow =
+          extractPoolIndex(rewriter, loc, anchorOp, rowTable, args.lane, kernelRow, kernelHeight);
+        if (physicalInput) {
+          inputRows.push_back(
+            extractRowStripFragment(args.inputs.front(), inputType, sourceRow, rewriter, loc));
+        }
+        else {
+          SmallVector<OpFoldResult> offsets {
+            rewriter.getIndexAttr(0), rewriter.getIndexAttr(0), sourceRow, rewriter.getIndexAttr(0)};
+          inputRows.push_back(tensor::ExtractSliceOp::create(rewriter,
+                                                            loc,
+                                                            inputFragmentType,
+                                                            args.inputs.front(),
+                                                            offsets,
+                                                            SmallVector<OpFoldResult> {rewriter.getIndexAttr(1),
+                                                                                       rewriter.getIndexAttr(channels),
+                                                                                       rewriter.getIndexAttr(1),
+                                                                                       rewriter.getIndexAttr(inputWidth)},
+                                                            getUnitStrides(rewriter, 4)));
+        }
+      }
+
+      auto windowType = RankedTensorType::get(
+        {1, channels, kernelHeight, inputWidth}, inputType.getElementType(), inputType.getEncoding());
+      Value window = tensor::EmptyOp::create(
+        rewriter, loc, windowType.getShape(), windowType.getElementType());
+      for (int64_t kernelRow = 0; kernelRow < kernelHeight; ++kernelRow) {
+        SmallVector<OpFoldResult> offsets {rewriter.getIndexAttr(0),
+                                           rewriter.getIndexAttr(0),
+                                           rewriter.getIndexAttr(kernelRow),
+                                           rewriter.getIndexAttr(0)};
+        window = tensor::InsertSliceOp::create(rewriter,
+                                               loc,
+                                               inputRows[kernelRow],
+                                               window,
+                                               offsets,
+                                               SmallVector<OpFoldResult> {rewriter.getIndexAttr(1),
+                                                                          rewriter.getIndexAttr(channels),
+                                                                          rewriter.getIndexAttr(1),
+                                                                          rewriter.getIndexAttr(inputWidth)},
+                                               getUnitStrides(rewriter, 4));
+      }
+
+      Value outputInit = tensor::EmptyOp::create(
+        rewriter, loc, outputFragmentType.getShape(), outputFragmentType.getElementType());
+      Operation* bodyAnchor = rewriter.getInsertionBlock()->getParentOp();
+      Value c0 = getOrCreateIndexConstant(rewriter, bodyAnchor, 0);
+      Value c1 = getOrCreateIndexConstant(rewriter, bodyAnchor, 1);
+      Value cOutputWidth = getOrCreateIndexConstant(rewriter, bodyAnchor, outputWidth);
+      auto outputLoop = buildNormalizedScfFor(
+        rewriter,
+        loc,
+        c0,
+        cOutputWidth,
+        c1,
+        ValueRange {outputInit},
+        [&](OpBuilder&, Location nestedLoc, Value outputColumn, ValueRange iterArgs, SmallVectorImpl<Value>& yielded) {
+          Value reduced;
+          for (int64_t kernelRow = 0; kernelRow < kernelHeight; ++kernelRow) {
+            for (int64_t kernelColumn = 0; kernelColumn < kernelWidth; ++kernelColumn) {
+              Value sourceColumn = extractPoolIndex(rewriter,
+                                                    nestedLoc,
+                                                    bodyAnchor,
+                                                    columnTable,
+                                                    outputColumn,
+                                                    kernelColumn,
+                                                    kernelWidth);
+              SmallVector<OpFoldResult> offsets {
+                rewriter.getIndexAttr(0),
+                rewriter.getIndexAttr(0),
+                rewriter.getIndexAttr(kernelRow),
+                sourceColumn};
+              Value point = tensor::ExtractSliceOp::create(rewriter,
+                                                           nestedLoc,
+                                                           tileType,
+                                                           window,
+                                                           offsets,
+                                                           SmallVector<OpFoldResult> {rewriter.getIndexAttr(1),
+                                                                                      rewriter.getIndexAttr(channels),
+                                                                                      rewriter.getIndexAttr(1),
+                                                                                      rewriter.getIndexAttr(1)},
+                                                           getUnitStrides(rewriter, 4));
+              reduced = reduced ? spatial::SpatVMaxOp::create(rewriter, nestedLoc, tileType, reduced, point).getResult()
+                                : materializeTileTensor(rewriter, nestedLoc, point);
+            }
+          }
+          SmallVector<OpFoldResult> outputOffsets {
+            rewriter.getIndexAttr(0), rewriter.getIndexAttr(0), rewriter.getIndexAttr(0), outputColumn};
+          Value updated = tensor::InsertSliceOp::create(rewriter,
+                                                        nestedLoc,
+                                                        reduced,
+                                                        iterArgs.front(),
+                                                        outputOffsets,
+                                                        SmallVector<OpFoldResult> {rewriter.getIndexAttr(1),
+                                                                                   rewriter.getIndexAttr(channels),
+                                                                                   rewriter.getIndexAttr(1),
+                                                                                   rewriter.getIndexAttr(1)},
+                                                        getUnitStrides(rewriter, 4));
+          yielded.push_back(updated);
+          return success();
+        });
+      if (failed(outputLoop))
+        return failure();
+      insertRowStripFragment(
+        outputLoop->results.front(), args.outputs.front(), outputType, args.lane, rewriter, loc);
+      return success();
+    });
+  if (failed(batch))
+    return failure();
+  return batch->getResult(0);
+}
+
 void populatePoolPatterns(RewritePatternSet& patterns, MLIRContext* ctx) {
   patterns.insert<PoolToSpatialCompute<ONNXMaxPoolSingleOutOp>>(ctx);
   patterns.insert<PoolToSpatialCompute<ONNXAveragePoolOp>>(ctx);

src/PIM/Conversion/ONNXToSpatial/Patterns/NN/Relu.cpp

@@ -16,12 +16,9 @@ struct ReluToSpatialCompute : OpConversionPattern<ONNXReluOp> {
   matchAndRewrite(ONNXReluOp reluOp, ONNXReluOpAdaptor adaptor, ConversionPatternRewriter& rewriter) const override {
     Location loc = reluOp.getLoc();
     Type resultType = reluOp.getResult().getType();
-    constexpr size_t numInputs = 1;
+    auto reluPlan = spatial::SpatReluPlanOp::create(
-    auto computeOp = createSpatCompute<numInputs>(rewriter, loc, resultType, {}, adaptor.getX(), [&](Value x) {
+      rewriter, loc, resultType, adaptor.getX(), rewriter.getStringAttr("nchw"));
-      auto spatReluOp = spatial::SpatReluOp::create(rewriter, loc, resultType, x);
+    rewriter.replaceOp(reluOp, reluPlan.getResult());
-      spatial::SpatYieldOp::create(rewriter, loc, spatReluOp.getResult());
-    });
-    rewriter.replaceOp(reluOp, computeOp);
     return success();
   }
 };

src/PIM/Conversion/ONNXToSpatial/Patterns/Post.cpp

@@ -10,6 +10,7 @@
 
 #include "src/Accelerators/PIM/Common/IR/WeightUtils.hpp"
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/ComputeRegionBuilder.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/WeightMaterialization.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
@@ -118,18 +119,16 @@ static LogicalResult mapPromotedInputArguments(ComputeOpTy compute,
 }
 
 // Promotes foldable helper chains from runtime inputs to weights to avoid artificial compute inputs.
-struct PromoteWeightLikeComputeInputsPattern : OpRewritePattern<spatial::SpatCompute> {
+struct PromoteWeightLikeComputeInputsPattern : OpRewritePattern<spatial::SpatGraphCompute> {
-  using OpRewritePattern<spatial::SpatCompute>::OpRewritePattern;
+  using OpRewritePattern<spatial::SpatGraphCompute>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(spatial::SpatCompute compute, PatternRewriter& rewriter) const override {
+  LogicalResult matchAndRewrite(spatial::SpatGraphCompute compute, PatternRewriter& rewriter) const override {
     auto promoted = computePromotedOperands(compute);
     if (failed(promoted))
       return rewriter.notifyMatchFailure(compute, "no weight-like inputs to promote");
     Block& oldBlock = compute.getBody().front();
 
     rewriter.setInsertionPointAfter(compute);
-    auto newCompute = spatial::SpatCompute::create(
-      rewriter, compute.getLoc(), compute.getResultTypes(), promoted->newWeights, promoted->newInputs);
     SmallVector<Type> newBlockArgTypes;
     SmallVector<Location> newBlockArgLocs;
     for (Value weight : promoted->newWeights) {
@@ -138,10 +137,14 @@ struct PromoteWeightLikeComputeInputsPattern : OpRewritePattern<spatial::SpatCom
     }
     llvm::append_range(newBlockArgTypes, promoted->newInputTypes);
     llvm::append_range(newBlockArgLocs, promoted->newInputLocs);
-    auto* newBlock = rewriter.createBlock(
+    auto newCompute = createEmptySpatGraphCompute(rewriter,
-      &newCompute.getBody(), newCompute.getBody().end(), TypeRange(newBlockArgTypes), newBlockArgLocs);
+                                                  compute.getLoc(),
-    newCompute.getProperties().setOperandSegmentSizes(
+                                                  compute.getResultTypes(),
-      {static_cast<int>(promoted->newWeights.size()), static_cast<int>(promoted->newInputs.size())});
+                                                  promoted->newWeights,
+                                                  promoted->newInputs,
+                                                  TypeRange(newBlockArgTypes),
+                                                  newBlockArgLocs);
+    auto* newBlock = &newCompute.getBody().front();
     rewriter.setInsertionPointToStart(newBlock);
 
     IRRewriter bodyRewriter(rewriter.getContext());
@@ -182,10 +185,10 @@ struct PromoteWeightLikeComputeInputsPattern : OpRewritePattern<spatial::SpatCom
 };
 
 // Promotes foldable batch helper chains to weights while preserving compact compute_batch IR.
-struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::SpatComputeBatch> {
+struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::SpatGraphComputeBatch> {
-  using OpRewritePattern<spatial::SpatComputeBatch>::OpRewritePattern;
+  using OpRewritePattern<spatial::SpatGraphComputeBatch>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(spatial::SpatComputeBatch compute, PatternRewriter& rewriter) const override {
+  LogicalResult matchAndRewrite(spatial::SpatGraphComputeBatch compute, PatternRewriter& rewriter) const override {
     auto promoted = computePromotedOperands(compute);
     if (failed(promoted))
       return rewriter.notifyMatchFailure(compute, "no weight-like batch inputs to promote");
@@ -193,12 +196,6 @@ struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::Sp
 
     rewriter.setInsertionPointAfter(compute);
 
-    auto laneCountAttr = pim::getCheckedI32Attr(
-      rewriter, compute, static_cast<uint64_t>(compute.getLaneCount()), "promoted compute_batch lane count");
-    if (failed(laneCountAttr))
-      return failure();
-    auto newCompute = spatial::SpatComputeBatch::create(
-      rewriter, compute.getLoc(), compute.getResultTypes(), *laneCountAttr, promoted->newWeights, promoted->newInputs);
     auto laneArg = compute.getLaneArgument();
     if (!laneArg)
       return rewriter.notifyMatchFailure(compute, "missing compute_batch lane block argument");
@@ -223,23 +220,30 @@ struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::Sp
       newBlockArgLocs.push_back(outputArg->getLoc());
     }
 
-    auto* newBlock = rewriter.createBlock(
+    auto newCompute = createEmptySpatGraphComputeBatch(rewriter,
-      &newCompute.getBody(), newCompute.getBody().end(), TypeRange(newBlockArgTypes), newBlockArgLocs);
+                                                       compute.getLoc(),
-    newCompute.getProperties().setOperandSegmentSizes(
+                                                       compute.getResultTypes(),
-      {static_cast<int>(promoted->newWeights.size()), static_cast<int>(promoted->newInputs.size())});
+                                                       compute.getLaneCount(),
+                                                       promoted->newWeights,
+                                                       promoted->newInputs,
+                                                       TypeRange(newBlockArgTypes),
+                                                       newBlockArgLocs);
+    if (failed(newCompute))
+      return failure();
+    auto* newBlock = &(*newCompute).getBody().front();
     rewriter.setInsertionPointToStart(newBlock);
 
     IRRewriter bodyRewriter(rewriter.getContext());
     bodyRewriter.setInsertionPointToStart(newBlock);
 
     IRMapping mapper;
-    auto newLaneArg = newCompute.getLaneArgument();
+    auto newLaneArg = (*newCompute).getLaneArgument();
     if (!newLaneArg)
       return rewriter.notifyMatchFailure(compute, "missing rewritten compute_batch lane block argument");
     mapper.map(*laneArg, *newLaneArg);
     for (auto [weightIndex, weight] : llvm::enumerate(compute.getWeights())) {
       auto oldWeightArg = compute.getWeightArgument(weightIndex);
-      auto newWeightArg = newCompute.getWeightArgument(weightIndex);
+      auto newWeightArg = (*newCompute).getWeightArgument(weightIndex);
       if (!oldWeightArg || !newWeightArg)
         return rewriter.notifyMatchFailure(compute, "missing compute_batch weight block argument during rewrite");
       mapper.map(*oldWeightArg, *newWeightArg);
@@ -249,7 +253,7 @@ struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::Sp
           *promoted,
           bodyRewriter,
           mapper,
-          [&](size_t index) { return newCompute.getInputArgument(index); },
+          [&](size_t index) { return (*newCompute).getInputArgument(index); },
           rewriter)))
       return failure();
     for (auto resultIndex : llvm::seq<size_t>(0, compute.getNumResults())) {
@@ -263,7 +267,7 @@ struct PromoteWeightLikeComputeBatchInputsPattern : OpRewritePattern<spatial::Sp
     for (Operation& op : oldBlock)
       rewriter.clone(op, mapper);
 
-    rewriter.replaceOp(compute, newCompute.getResults());
+    rewriter.replaceOp(compute, (*newCompute).getResults());
     return success();
   }
 };
@@ -281,8 +285,8 @@ void annotateWeightsConstants(func::FuncOp funcOp) {
   });
 }
 
-bool requiresPostRewrite(spatial::SpatCompute computeOp) { return hasPromotableWeightLikeInputs(computeOp); }
+bool requiresPostRewrite(spatial::SpatGraphCompute computeOp) { return hasPromotableWeightLikeInputs(computeOp); }
 
-bool requiresPostRewrite(spatial::SpatComputeBatch computeOp) { return hasPromotableWeightLikeInputs(computeOp); }
+bool requiresPostRewrite(spatial::SpatGraphComputeBatch computeOp) { return hasPromotableWeightLikeInputs(computeOp); }
 
 } // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns/Tensor/Flatten.cpp

@@ -0,0 +1,112 @@
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/Transforms/DialectConversion.h"
+
+#include "llvm/ADT/SmallVector.h"
+
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+#include "src/Dialect/ONNX/ONNXOps.hpp"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static FailureOr<int64_t> normalizeFlattenAxis(int64_t axis, int64_t rank) {
+  int64_t normalizedAxis = axis < 0 ? rank + axis : axis;
+  if (normalizedAxis < 0 || normalizedAxis > rank)
+    return failure();
+  return normalizedAxis;
+}
+
+static int64_t product(ArrayRef<int64_t> values) {
+  int64_t result = 1;
+  for (int64_t value : values)
+    result *= value;
+  return result;
+}
+
+static SmallVector<ReassociationIndices> getCollapseTo1DReassociation(int64_t rank) {
+  SmallVector<ReassociationIndices> reassociation(1);
+  reassociation.front().reserve(rank);
+  for (int64_t dim = 0; dim < rank; ++dim)
+    reassociation.front().push_back(dim);
+  return reassociation;
+}
+
+static SmallVector<ReassociationIndices> getExpandFrom1DReassociation(int64_t rank) {
+  SmallVector<ReassociationIndices> reassociation(1);
+  reassociation.front().reserve(rank);
+  for (int64_t dim = 0; dim < rank; ++dim)
+    reassociation.front().push_back(dim);
+  return reassociation;
+}
+
+static Value buildFlatten(Value input,
+                          RankedTensorType sourceType,
+                          RankedTensorType resultType,
+                          int64_t axis,
+                          ConversionPatternRewriter& rewriter,
+                          Location loc) {
+  if (sourceType == resultType)
+    return input;
+
+  if (axis > 0 && axis < sourceType.getRank()) {
+    SmallVector<ReassociationIndices> reassociation(2);
+    for (int64_t dim = 0; dim < axis; ++dim)
+      reassociation[0].push_back(dim);
+    for (int64_t dim = axis; dim < sourceType.getRank(); ++dim)
+      reassociation[1].push_back(dim);
+    return tensor::CollapseShapeOp::create(rewriter, loc, resultType, input, reassociation);
+  }
+
+  Value flattened = input;
+  if (sourceType.getRank() != 1) {
+    auto flatType = RankedTensorType::get({sourceType.getNumElements()}, sourceType.getElementType());
+    flattened = tensor::CollapseShapeOp::create(
+      rewriter, loc, flatType, flattened, getCollapseTo1DReassociation(sourceType.getRank()));
+  }
+  return tensor::ExpandShapeOp::create(
+    rewriter, loc, resultType, flattened, getExpandFrom1DReassociation(resultType.getRank()));
+}
+
+struct Flatten : OpConversionPattern<ONNXFlattenOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult matchAndRewrite(ONNXFlattenOp flattenOp,
+                                ONNXFlattenOpAdaptor adaptor,
+                                ConversionPatternRewriter& rewriter) const override {
+    auto sourceType = dyn_cast<RankedTensorType>(adaptor.getInput().getType());
+    auto resultType = dyn_cast<RankedTensorType>(flattenOp.getOperation()->getResult(0).getType());
+    if (!sourceType || !resultType || !sourceType.hasStaticShape() || !resultType.hasStaticShape())
+      return failure();
+    if (!hasStaticPositiveShape(sourceType) || !hasStaticPositiveShape(resultType) || resultType.getRank() != 2)
+      return failure();
+
+    auto axis = normalizeFlattenAxis(flattenOp.getAxis(), sourceType.getRank());
+    if (failed(axis))
+      return failure();
+
+    int64_t outerDim = product(sourceType.getShape().take_front(*axis));
+    int64_t innerDim = product(sourceType.getShape().drop_front(*axis));
+    if (resultType.getShape()[0] != outerDim || resultType.getShape()[1] != innerDim)
+      return failure();
+
+    auto replaceWithFlatten = [&](auto build) -> LogicalResult {
+      Value flattened = materializeOrComputeUnary(adaptor.getInput(), resultType, rewriter, flattenOp.getLoc(), build);
+      rewriter.replaceOp(flattenOp, flattened);
+      return success();
+    };
+
+    return replaceWithFlatten([&](Value input) {
+      return buildFlatten(input, sourceType, resultType, *axis, rewriter, flattenOp.getLoc());
+    });
+  }
+};
+
+} // namespace
+
+void populateFlattenPatterns(RewritePatternSet& patterns, MLIRContext* ctx) { patterns.add<Flatten>(ctx); }
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/Patterns/Tensor/Transpose.cpp

@@ -5,7 +5,7 @@
 
 #include "llvm/ADT/SmallVector.h"
 
-#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/CompileTime.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Patterns.hpp"
 #include "src/Dialect/ONNX/ONNXOps.hpp"
@@ -52,35 +52,12 @@ static FailureOr<Value> materializeTransposedConstant(Value input,
     return failure();
   }
 
-  if (denseAttr.isSplat())
+  auto transposedAttr = transposeDenseElementsAttr(denseAttr, permutation);
+  if (failed(transposedAttr) || transposedAttr->getType() != resultType)
+    return failure();
   return getOrCreateConstant(rewriter,
                              rewriter.getInsertionBlock()->getParentOp(),
-                               DenseElementsAttr::get(resultType, denseAttr.getSplatValue<Attribute>()),
+                             *transposedAttr,
-                               resultType);
-
-  SmallVector<Attribute> inputValues(denseAttr.getValues<Attribute>());
-  SmallVector<Attribute> resultValues(inputValues.size());
-  SmallVector<int64_t> inputStrides = computeRowMajorStrides(inputType.getShape());
-  SmallVector<int64_t> resultStrides = computeRowMajorStrides(resultType.getShape());
-  SmallVector<int64_t> inputIndices(inputType.getRank(), 0);
-
-  for (auto [linearIndex, value] : llvm::enumerate(inputValues)) {
-    int64_t remaining = static_cast<int64_t>(linearIndex);
-    for (int64_t dim = 0; dim < inputType.getRank(); ++dim) {
-      inputIndices[dim] = inputStrides.empty() ? 0 : remaining / inputStrides[dim];
-      remaining = inputStrides.empty() ? 0 : remaining % inputStrides[dim];
-    }
-
-    int64_t resultLinearIndex = 0;
-    for (int64_t dim = 0; dim < resultType.getRank(); ++dim)
-      resultLinearIndex += inputIndices[permutation[dim]] * resultStrides[dim];
-
-    resultValues[resultLinearIndex] = value;
-  }
-
-  return getOrCreateConstant(rewriter,
-                             rewriter.getInsertionBlock()->getParentOp(),
-                             DenseElementsAttr::get(resultType, resultValues),
                              resultType);
 }
 

src/PIM/Conversion/ONNXToSpatial/PlanLowering.hpp

@@ -0,0 +1,28 @@
+#pragma once
+
+#include <optional>
+
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Support/LogicalResult.h"
+
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+
+namespace onnx_mlir {
+
+mlir::FailureOr<mlir::Value>
+lowerSelectedConv2DPlan(spatial::SpatConv2DPlanOp planOp,
+                        std::optional<mlir::Value> rowStripInput,
+                        bool emitRowStripLayout,
+                        mlir::PatternRewriter& rewriter);
+
+mlir::LogicalResult canLowerConvPlanToRowStrip(spatial::SpatConv2DPlanOp planOp);
+mlir::LogicalResult canConsumeAndProduceRowStrip(spatial::SpatConv2DPlanOp planOp);
+
+mlir::LogicalResult canLowerMaxPoolPlanToRowStrip(spatial::SpatMaxPool2DPlanOp planOp);
+
+mlir::FailureOr<mlir::Value>
+lowerSelectedMaxPool2DPlan(spatial::SpatMaxPool2DPlanOp planOp,
+                           std::optional<mlir::Value> rowStripInput,
+                           mlir::PatternRewriter& rewriter);
+
+} // namespace onnx_mlir

src/PIM/Conversion/ONNXToSpatial/SpatialLayoutPlanningPass.cpp

@@ -0,0 +1,259 @@
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Pass/Pass.h"
+
+#include "llvm/ADT/DenseMap.h"
+
+#include "Conversion/ONNXToSpatial/ONNXToSpatialVerifier.hpp"
+#include "src/Accelerators/PIM/Common/PimCommon.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/Common/RowStripLayoutUtils.hpp"
+#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/PlanLowering.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
+#include "src/Accelerators/PIM/Pass/PIMPasses.h"
+
+using namespace mlir;
+
+namespace onnx_mlir {
+namespace {
+
+static constexpr StringLiteral kLogicalLayout = "nchw";
+static constexpr StringLiteral kDenseLayout = "dense_nchw";
+static constexpr StringLiteral kRowStripLayout = "nchw_row_strip";
+
+enum class SelectedLayout {
+  DenseNchw,
+  NchwRowStrip,
+};
+
+static SelectedLayout getSelectedLayout(llvm::DenseMap<Value, SelectedLayout>& layouts, Value value) {
+  auto it = layouts.find(value);
+  return it == layouts.end() ? SelectedLayout::DenseNchw : it->second;
+}
+
+static bool usesSelectedRowStrip(Operation* user, llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  if (auto reluPlan = dyn_cast<spatial::SpatReluPlanOp>(user))
+    return getSelectedLayout(layouts, reluPlan.getResult()) == SelectedLayout::NchwRowStrip;
+  if (auto biasAddPlan = dyn_cast<spatial::SpatBiasAddPlanOp>(user))
+    return getSelectedLayout(layouts, biasAddPlan.getResult()) == SelectedLayout::NchwRowStrip;
+  if (auto convPlan = dyn_cast<spatial::SpatConv2DPlanOp>(user))
+    return getSelectedLayout(layouts, convPlan.getResult()) == SelectedLayout::NchwRowStrip;
+  if (auto maxPoolPlan = dyn_cast<spatial::SpatMaxPool2DPlanOp>(user))
+    return getSelectedLayout(layouts, maxPoolPlan.getResult()) == SelectedLayout::NchwRowStrip;
+  return false;
+}
+
+static bool allUsersCanHandleRowStrip(Value value, llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  for (Operation* user : value.getUsers()) {
+    if (usesSelectedRowStrip(user, layouts))
+      continue;
+    // Dense-only users must be materialized explicitly.
+    continue;
+  }
+  return true;
+}
+
+static bool canConsumeRowStripAsUser(Operation* user) {
+  if (isa<spatial::SpatReluPlanOp>(user))
+    return true;
+  if (auto biasAddPlan = dyn_cast<spatial::SpatBiasAddPlanOp>(user)) {
+    auto resultType = dyn_cast<RankedTensorType>(biasAddPlan.getOutput().getType());
+    return resultType && isSupportedBiasAddValue(biasAddPlan.getBias(), resultType);
+  }
+  if (auto convPlan = dyn_cast<spatial::SpatConv2DPlanOp>(user))
+    return succeeded(canConsumeAndProduceRowStrip(convPlan));
+  if (auto maxPoolPlan = dyn_cast<spatial::SpatMaxPool2DPlanOp>(user))
+    return succeeded(canLowerMaxPoolPlanToRowStrip(maxPoolPlan));
+  return false;
+}
+
+static bool hasRowStripConsumer(Value value) {
+  for (Operation* user : value.getUsers())
+    if (canConsumeRowStripAsUser(user))
+      return true;
+  return false;
+}
+
+static bool canSelectConvRowStrip(spatial::SpatConv2DPlanOp convPlan,
+                                  llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  SelectedLayout inputLayout = getSelectedLayout(layouts, convPlan.getInput());
+  if (inputLayout == SelectedLayout::NchwRowStrip)
+    return succeeded(canConsumeAndProduceRowStrip(convPlan));
+  return succeeded(canLowerConvPlanToRowStrip(convPlan));
+}
+
+static SelectedLayout chooseConvLayout(spatial::SpatConv2DPlanOp convPlan,
+                                       llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  if (!canSelectConvRowStrip(convPlan, layouts))
+    return SelectedLayout::DenseNchw;
+  if (!allUsersCanHandleRowStrip(convPlan.getResult(), layouts))
+    return SelectedLayout::DenseNchw;
+  return SelectedLayout::NchwRowStrip;
+}
+
+static SelectedLayout chooseReluLayout(spatial::SpatReluPlanOp reluPlan,
+                                       llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  if (getSelectedLayout(layouts, reluPlan.getInput()) != SelectedLayout::NchwRowStrip)
+    return SelectedLayout::DenseNchw;
+  if (!hasRowStripConsumer(reluPlan.getResult()))
+    return SelectedLayout::DenseNchw;
+  if (!allUsersCanHandleRowStrip(reluPlan.getResult(), layouts))
+    return SelectedLayout::DenseNchw;
+  return SelectedLayout::NchwRowStrip;
+}
+
+static SelectedLayout chooseBiasAddLayout(spatial::SpatBiasAddPlanOp biasAddPlan,
+                                          llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  if (getSelectedLayout(layouts, biasAddPlan.getInput()) != SelectedLayout::NchwRowStrip)
+    return SelectedLayout::DenseNchw;
+  auto resultType = dyn_cast<RankedTensorType>(biasAddPlan.getOutput().getType());
+  if (!resultType || !isSupportedBiasAddValue(biasAddPlan.getBias(), resultType))
+    return SelectedLayout::DenseNchw;
+  if (!hasRowStripConsumer(biasAddPlan.getResult()))
+    return SelectedLayout::DenseNchw;
+  if (!allUsersCanHandleRowStrip(biasAddPlan.getResult(), layouts))
+    return SelectedLayout::DenseNchw;
+  return SelectedLayout::NchwRowStrip;
+}
+
+static SelectedLayout chooseMaxPoolLayout(spatial::SpatMaxPool2DPlanOp maxPoolPlan) {
+  return succeeded(canLowerMaxPoolPlanToRowStrip(maxPoolPlan)) ? SelectedLayout::NchwRowStrip
+                                                               : SelectedLayout::DenseNchw;
+}
+
+static spatial::SpatBlueprintOp insertRowStripBlueprint(IRRewriter& rewriter, Value value) {
+  auto outputType = cast<RankedTensorType>(value.getType());
+  auto [offsets, sizes] = buildRowStripMetadata(outputType);
+  return spatial::SpatBlueprintOp::create(rewriter,
+                                              value.getLoc(),
+                                              outputType,
+                                              value,
+                                              ValueRange {},
+                                              rewriter.getStringAttr(kLogicalLayout),
+                                              rewriter.getStringAttr(kRowStripLayout),
+                                              rewriter.getDenseI64ArrayAttr(offsets),
+                                              rewriter.getDenseI64ArrayAttr(sizes),
+                                              rewriter.getStringAttr(kRowStripIndexMap),
+                                              nullptr,
+                                              nullptr,
+                                              nullptr,
+                                              nullptr,
+                                              nullptr,
+                                              nullptr,
+                                              nullptr);
+}
+
+static void materializeDenseUses(IRRewriter& rewriter,
+                                 Value layoutValue,
+                                 llvm::DenseMap<Value, SelectedLayout>& layouts) {
+  SmallVector<OpOperand*> denseUses;
+  for (OpOperand& use : layoutValue.getUses()) {
+    if (usesSelectedRowStrip(use.getOwner(), layouts))
+      continue;
+    denseUses.push_back(&use);
+  }
+
+  for (OpOperand* use : denseUses) {
+    Operation* owner = use->getOwner();
+    rewriter.setInsertionPoint(owner);
+    auto materialized = spatial::SpatMaterializeLayoutOp::create(rewriter,
+                                                                 owner->getLoc(),
+                                                                 use->get().getType(),
+                                                                 use->get(),
+                                                                 rewriter.getStringAttr(kLogicalLayout),
+                                                                 rewriter.getStringAttr(kRowStripLayout),
+                                                                 rewriter.getStringAttr(kDenseLayout));
+    use->set(materialized.getResult());
+  }
+}
+
+struct SpatialLayoutPlanningPass final : PassWrapper<SpatialLayoutPlanningPass, OperationPass<ModuleOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(SpatialLayoutPlanningPass)
+
+  StringRef getArgument() const override { return "spatial-layout-planning"; }
+  StringRef getDescription() const override { return "Select conservative Spatial layouts and insert reconciliation barriers."; }
+
+  void runOnOperation() override {
+    auto entryFunc = getPimEntryFunc(getOperation());
+    if (failed(entryFunc)) {
+      getOperation().emitError("failed to locate the PIM entry function during Spatial layout planning");
+      signalPassFailure();
+      return;
+    }
+
+    func::FuncOp funcOp = *entryFunc;
+    IRRewriter rewriter(&getContext());
+    llvm::DenseMap<Value, SelectedLayout> layouts;
+
+    bool changed = true;
+    while (changed) {
+      changed = false;
+      for (Operation& op : llvm::make_early_inc_range(funcOp.getBody().front())) {
+        if (auto convPlan = dyn_cast<spatial::SpatConv2DPlanOp>(&op)) {
+          SelectedLayout selected = chooseConvLayout(convPlan, layouts);
+          if (layouts[convPlan.getResult()] != selected) {
+            layouts[convPlan.getResult()] = selected;
+            changed = true;
+          }
+          continue;
+        }
+        if (auto reluPlan = dyn_cast<spatial::SpatReluPlanOp>(&op)) {
+          SelectedLayout selected = chooseReluLayout(reluPlan, layouts);
+          if (layouts[reluPlan.getResult()] != selected) {
+            layouts[reluPlan.getResult()] = selected;
+            changed = true;
+          }
+          continue;
+        }
+        if (auto biasAddPlan = dyn_cast<spatial::SpatBiasAddPlanOp>(&op)) {
+          SelectedLayout selected = chooseBiasAddLayout(biasAddPlan, layouts);
+          if (layouts[biasAddPlan.getResult()] != selected) {
+            layouts[biasAddPlan.getResult()] = selected;
+            changed = true;
+          }
+          continue;
+        }
+        if (auto maxPoolPlan = dyn_cast<spatial::SpatMaxPool2DPlanOp>(&op)) {
+          SelectedLayout selected = chooseMaxPoolLayout(maxPoolPlan);
+          if (layouts[maxPoolPlan.getResult()] != selected) {
+            layouts[maxPoolPlan.getResult()] = selected;
+            changed = true;
+          }
+          continue;
+        }
+      }
+    }
+
+    for (Operation& op : llvm::make_early_inc_range(funcOp.getBody().front())) {
+      Value producedValue;
+      if (auto convPlan = dyn_cast<spatial::SpatConv2DPlanOp>(&op))
+        producedValue = convPlan.getResult();
+      else if (auto biasAddPlan = dyn_cast<spatial::SpatBiasAddPlanOp>(&op))
+        producedValue = biasAddPlan.getResult();
+      else if (auto reluPlan = dyn_cast<spatial::SpatReluPlanOp>(&op))
+        producedValue = reluPlan.getResult();
+      else if (auto maxPoolPlan = dyn_cast<spatial::SpatMaxPool2DPlanOp>(&op))
+        producedValue = maxPoolPlan.getResult();
+      else
+        continue;
+
+      if (getSelectedLayout(layouts, producedValue) != SelectedLayout::NchwRowStrip)
+        continue;
+
+      rewriter.setInsertionPointAfter(&op);
+      auto blueprint = insertRowStripBlueprint(rewriter, producedValue);
+      rewriter.replaceAllUsesExcept(producedValue, blueprint.getResult(), blueprint);
+      materializeDenseUses(rewriter, blueprint.getResult(), layouts);
+    }
+    if (failed(verifyLogicalSpatialGraphInvariants(*entryFunc))) {
+      getOperation().emitError("logical Spatial graph verification failed after SpatialLayoutPlanning");
+      signalPassFailure();
+    }
+  }
+};
+
+} // namespace
+
+std::unique_ptr<Pass> createSpatialLayoutPlanningPass() { return std::make_unique<SpatialLayoutPlanningPass>(); }
+
+} // namespace onnx_mlir

src/PIM/Conversion/SpatialToGraphviz/CMakeLists.txt

@@ -1,17 +0,0 @@
-add_onnx_mlir_rewriter(SpatialToGraphviz)
-
-add_pim_library(OMSpatialToGraphviz
-  SpatialToGraphviz.cpp
-
-  EXCLUDE_FROM_OM_LIBS
-
-  LINK_LIBS PUBLIC
-  MLIRTosaDialect
-  OMCompilerOptions
-  OMPimCommon
-  OMONNXOps
-  SpatialOps
-
-  ACCEL_INCLUDE_DIRS PRIVATE
-  ${PIM_GENERATED_INCLUDE_DIRS}
-)

src/PIM/Conversion/SpatialToGraphviz/SpatialToGraphviz.cpp

@@ -1,259 +0,0 @@
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
-#include "mlir/Dialect/Tosa/IR/TosaOps.h"
-#include "mlir/IR/Block.h"
-#include "mlir/IR/Diagnostics.h"
-#include "mlir/IR/Value.h"
-#include "mlir/Pass/Pass.h"
-#include "mlir/Support/LLVM.h"
-#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
-
-#include "llvm/Support/Casting.h"
-#include "llvm/Support/Format.h"
-
-#include "src/Accelerators/PIM/Common/PimCommon.hpp"
-#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
-#include "src/Accelerators/PIM/Pass/PIMPasses.h"
-#include "src/Dialect/ONNX/ONNXOps.hpp"
-
-#define FORMAT_OPERATION(op) 'x' << llvm::format_hex_no_prefix(reinterpret_cast<size_t>(op), 0)
-#define FORMAT_ARGUMENT(computeOpPointer, argumentNum) llvm::format("Arg_%p_%u", computeOpPointer, argumentNum)
-
-using namespace mlir;
-
-namespace onnx_mlir {
-
-namespace {
-
-struct SpatialToGraphvizPass : public PassWrapper<SpatialToGraphvizPass, OperationPass<ModuleOp>> {
-
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(SpatialToGraphvizPass)
-
-  StringRef getArgument() const override { return "convert-spatial-to-graphviz"; }
-
-  StringRef getDescription() const override { return "Lower ONNX ops to Spatial ops."; }
-
-  SpatialToGraphvizPass(raw_ostream& os = llvm::errs())
-  : os(os) {}
-  SpatialToGraphvizPass(const SpatialToGraphvizPass& pass)
-  : SpatialToGraphvizPass(pass.os) {}
-  void runOnOperation() final;
-
-private:
-  raw_ostream& os;
-
-  /**
-   * Draws the subgraph for a given spatial::SpatCompute, including:
-   * 1. Input nodes (block arguments)
-   * 2. Operations
-   * 3. Edges between yield (output) and its users
-   *
-   * @param op The spatial::SpatCompute to draw the subgraph for.
-   * @param computeNum The number of the compute operation.
-   */
-  void drawComputeOpSubgraph(spatial::SpatCompute op, size_t computeNum) {
-    os << "\tsubgraph cluster" << computeNum << " {\n\t\tlabel=\"Compute" << computeNum << "\";\n"
-       << "\t\tstyle=filled;\n"
-       << "\t\tcolor=lightblue;\n";
-
-    Block& block = op.getBody().front();
-
-    // Inputs
-    size_t inputNum = 0;
-    for (BlockArgument& input : block.getArguments()) {
-
-      auto fromOp = FORMAT_ARGUMENT(op.getOperation(), inputNum);
-
-      os << "\t\t" << fromOp << " [label=\"Arg" << inputNum << "\",shape=box];\n";
-      for (auto userOp : input.getUsers())
-        os << "\t\t" << fromOp << " -> " << FORMAT_OPERATION(userOp) << ";\n";
-      inputNum++;
-    }
-
-    // Iterate operations
-    for (auto& childOp : block.getOperations()) {
-      os << "\t\t" << FORMAT_OPERATION(&childOp) << " [label=\"" << childOp.getName() << "\"];\n";
-
-      drawEdgesFromOpToItsUsers(&childOp);
-    }
-
-    os << "\t}\n";
-
-    // Draw edges from the yield to the users of this computeOp
-    Operation* yieldOp = block.getTerminator();
-    if (!isa<spatial::SpatYieldOp>(yieldOp)) {
-      yieldOp->emitError("Terminator of block must be YieldOp ???");
-      signalPassFailure();
-      return;
-    }
-
-    for (auto computeOpResult : op->getResults()) {
-      for (auto& computeOpUse : computeOpResult.getUses()) {
-        auto toOp = FORMAT_ARGUMENT(computeOpUse.getOwner(), computeOpUse.getOperandNumber());
-        os << "\t" << FORMAT_OPERATION(yieldOp) << " -> " << toOp << ";\n";
-      }
-    }
-  }
-
-  /**
-   * @brief Draws the subgraph for a concatOp.
-   *
-   * This function draws a subgraph for a concatOp. The subgraph consists of a
-   * node for each input of the concatOp, as well as an output node. Edges are
-   * created from the output node to each user of the concatOp.
-   *
-   * @param concatOp The concatOp for which the subgraph is drawn.
-   * @param concatOpNum The number of the concatOp.
-   */
-  void drawConcatOpSubgraph(Operation* concatOp, size_t concatOpNum) {
-    os << "\tsubgraph clusterconcat" << concatOpNum << " {\n\t\tlabel=\"ConcatOp" << concatOpNum << "\";\n"
-       << "\t\tstyle=filled;\n"
-       << "\t\tcolor=orange;\n";
-
-    // Inputs
-    size_t inputNum = 0;
-    for (Value input : concatOp->getOperands()) {
-      auto fromOp = FORMAT_ARGUMENT(concatOp, inputNum);
-
-      os << "\t\t" << fromOp << " [label=\"Input" << inputNum << "\"];\n";
-      for (auto userOp : input.getUsers())
-        os << "\t\t" << fromOp << " -> " << FORMAT_OPERATION(userOp) << ";\n";
-      inputNum++;
-    }
-
-    // Output
-    os << "\t\t" << FORMAT_OPERATION(concatOp) << " [label=Out];\n";
-
-    os << "\t}\n";
-
-    // Edges from output to users
-
-    for (auto& computeOpUse : concatOp->getResult(0).getUses()) {
-      os << "\t" << FORMAT_OPERATION(concatOp) << " -> "
-         << FORMAT_ARGUMENT(computeOpUse.getOwner(), computeOpUse.getOperandNumber()) << ";\n";
-    }
-  }
-
-  /**
-   * Draws the ExtractSliceOp in the graph visualization.
-   *
-   * This function takes a tensor::ExtractSliceOp and adds the corresponding
-   * node and edges to the graph visualization. It creates a node with the
-   * label as the static offsets attribute of the sliceOp, and connects it to
-   * the compute operations that use the result of the sliceOp.
-   *
-   * @param sliceOp The tensor::ExtractSliceOp to be drawn in the graph
-   * visualization.
-   */
-  void drawExtractSliceOp(tensor::ExtractSliceOp sliceOp) {
-    auto nodeId = FORMAT_ARGUMENT(sliceOp.getOperation(), 0);
-    os << "\t" << nodeId << " [label=\"Slice: ";
-    sliceOp.getStaticOffsetsAttr().print(os);
-    os << "\",color=lawngreen];\n";
-
-    for (auto& computeOpUse : sliceOp.getResult().getUses()) {
-      os << "\t" << nodeId << " -> " << FORMAT_ARGUMENT(computeOpUse.getOwner(), computeOpUse.getOperandNumber())
-         << ";\n";
-    }
-  }
-
-  void drawBiasTileOp(tensor::ExtractSliceOp sliceOp) {
-    auto nodeId = FORMAT_ARGUMENT(sliceOp.getOperation(), 0);
-    os << "\t" << nodeId << " [label=\"Bias: ";
-    sliceOp.getStaticOffsetsAttr().print(os);
-    os << "\",color=lightpink];\n";
-
-    for (auto user : sliceOp.getResult().getUsers())
-      os << "\t" << nodeId << " -> " << FORMAT_OPERATION(user) << ";\n";
-  }
-
-  /**
-   * Draws edges from the given operation to its users.
-   *
-   * @param fromOp The operation from which the edges are drawn.
-   */
-  void drawEdgesFromOpToItsUsers(mlir::Operation* fromOp) {
-    for (auto result : fromOp->getResults())
-      for (auto userOp : result.getUsers())
-        os << "\t\t" << FORMAT_OPERATION(fromOp) << " -> " << FORMAT_OPERATION(userOp) << ";\n";
-  }
-
-  /**
-   * Draws input node and edges for the given `funcOp`.
-   *
-   * @param funcOp The `funcOp` for which to draw input nodes and edges.
-   */
-  void drawInputNodesAndEdges(func::FuncOp& funcOp) {
-    os << "\tinput [label=\"Module Input\",color=green];\n";
-
-    size_t funcOpArgNum = 0;
-    for (BlockArgument& arg : funcOp.getArguments()) {
-
-      for (auto& useOp : arg.getUses()) {
-        os << "\tinput -> " << FORMAT_ARGUMENT(useOp.getOwner(), useOp.getOperandNumber()) << "[label=" << funcOpArgNum
-           << "];\n";
-      }
-      funcOpArgNum++;
-    }
-  }
-};
-
-void SpatialToGraphvizPass::runOnOperation() {
-  ModuleOp module = getOperation();
-
-  auto entryFunc = getPimEntryFunc(module);
-  if (failed(entryFunc)) {
-    module.emitError("failed to locate the PIM entry function for Spatial graph visualization");
-    signalPassFailure();
-    return;
-  }
-  func::FuncOp func = *entryFunc;
-
-  os << "digraph G {\n"
-     << "\tnode [style=filled,color=white];\n";
-
-  size_t computeNum = 0;
-  size_t concatNum = 0;
-
-  // Iterate over the ComputeOps within FuncOp:
-  // 1. Print their subgraph
-  // 2. Print the edges from its inputs to its outputs
-  for (Operation& op : func.getOps()) {
-    if (auto computeOp = dyn_cast<spatial::SpatCompute>(op)) {
-      drawComputeOpSubgraph(computeOp, computeNum++);
-    }
-    else if (auto concatOp = dyn_cast<tensor::ConcatOp>(op)) {
-      drawConcatOpSubgraph(concatOp, concatNum++);
-    }
-    else if (auto extractSliceOp = dyn_cast<tensor::ExtractSliceOp>(op)) {
-      auto producerOp = extractSliceOp->getOperand(0).getDefiningOp();
-      if (producerOp) {
-        // Skip extractSliceOp if producer is constant weights (ONNXConstantOp)
-        if (llvm::isa<ONNXConstantOp>(producerOp))
-          continue;
-        // If produced by tosa::ReshapeOp (i.e. it is a bias tile) connect
-        // directly to its user, which is not a ComputeOp argument.
-        if (llvm::isa<tosa::ReshapeOp>(producerOp)) {
-          drawBiasTileOp(extractSliceOp);
-          continue;
-        }
-      }
-
-      drawExtractSliceOp(extractSliceOp);
-    }
-  }
-
-  // Draw input node, and edges to it users
-  drawInputNodesAndEdges(func);
-
-  // Draw output node (use the return Operation - argument number=0 - as nodeId)
-  auto returnOp = func.getBody().front().getTerminator();
-  os << '\t' << FORMAT_ARGUMENT(returnOp, 0) << " [label=\"Module Output\",color=green];\n";
-
-  os << "}\n";
-}
-
-} // namespace
-
-std::unique_ptr<Pass> createSpatialToGraphvizPass() { return std::make_unique<SpatialToGraphvizPass>(); }
-
-} // namespace onnx_mlir

src/PIM/Conversion/SpatialToPim/BatchCoreLoweringPatterns.cpp

@@ -2,13 +2,16 @@
 #include "mlir/Dialect/Bufferization/IR/Bufferization.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/IRMapping.h"
 #include "mlir/IR/Matchers.h"
+#include <limits>
 
 #include "Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "Conversion/SpatialToPim/SpatialToPimPass.hpp"
 #include "src/Accelerators/PIM/Common/IR/BatchCoreUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Common.hpp"
@@ -26,22 +29,9 @@ static bool isUsedOnlyAsExplicitHostOperand(Value value) {
   });
 }
 
-static FailureOr<SmallVector<int32_t>> getPimCoreIdsForBatchOp(spatial::SpatComputeBatch computeBatchOp,
+static bool isUsedOnlyByExtractSlices(Value value) {
-                                                               size_t& fallbackCoreId) {
+  return !value.use_empty()
-  if (auto coreIdsAttr = computeBatchOp->getAttrOfType<DenseI32ArrayAttr>(onnx_mlir::kCoreIdsAttrName))
+         && llvm::all_of(value.getUsers(), [](Operation* user) { return isa<tensor::ExtractSliceOp>(user); });
-    return SmallVector<int32_t>(coreIdsAttr.asArrayRef().begin(), coreIdsAttr.asArrayRef().end());
-
-  SmallVector<int32_t> coreIds;
-  coreIds.reserve(static_cast<size_t>(computeBatchOp.getLaneCount()));
-  for (uint32_t lane = 0; lane < computeBatchOp.getLaneCount(); ++lane) {
-    auto checkedCoreId =
-      pim::checkedI32(static_cast<uint64_t>(fallbackCoreId), computeBatchOp, "fallback spatial compute_batch core id");
-    if (failed(checkedCoreId))
-      return failure();
-    coreIds.push_back(*checkedCoreId);
-    ++fallbackCoreId;
-  }
-  return coreIds;
 }
 
 static FailureOr<unsigned> getDirectReturnOperandIndex(OpResult result) {
@@ -54,6 +44,194 @@ static FailureOr<unsigned> getDirectReturnOperandIndex(OpResult result) {
   return result.getUses().begin()->getOperandNumber();
 }
 
+static FailureOr<SmallVector<FragmentAssemblyCopy, 8>>
+collectFragmentAssemblyCopiesFromBlueprint(spatial::SpatBlueprintOp blueprint,
+                                          IRMapping& mapper,
+                                          int64_t lane,
+                                          unsigned hostTargetIndex,
+                                          Value fixedSource = {}) {
+  SmallVector<FragmentAssemblyCopy, 8> copies;
+  auto resultType = dyn_cast<RankedTensorType>(blueprint.getOutput().getType());
+  if (!resultType || !resultType.hasStaticShape())
+    return blueprint.emitOpError("fragment assembly lowering requires static ranked tensor results");
+
+  std::optional<ArrayRef<int64_t>> operandIndicesAttr = blueprint.getFragmentOperandIndices();
+  std::optional<ArrayRef<int64_t>> sourceSlotsAttr = blueprint.getFragmentSourceSlots();
+  std::optional<ArrayRef<int64_t>> fragmentStridesAttr = blueprint.getFragmentStrides();
+  if (!operandIndicesAttr || !sourceSlotsAttr || !fragmentStridesAttr)
+    return blueprint.emitOpError(
+      "fragment assembly lowering requires explicit operand indices, source slots, and unit strides");
+
+  ArrayRef<int64_t> operandIndices = *operandIndicesAttr;
+  ArrayRef<int64_t> sourceSlots = *sourceSlotsAttr;
+  std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = blueprint.getFragmentSourceOffsets();
+  if (!sourceOffsetsAttr)
+    return blueprint.emitOpError("fragment assembly lowering requires explicit source offsets");
+  ArrayRef<int64_t> sourceOffsets = *sourceOffsetsAttr;
+  ArrayRef<int64_t> flatOffsets = blueprint.getFragmentOffsets();
+  ArrayRef<int64_t> flatSizes = blueprint.getFragmentSizes();
+  ArrayRef<int64_t> flatStrides = *fragmentStridesAttr;
+  int64_t rank = resultType.getRank();
+
+  SmallVector<Value> fragmentOperands {blueprint.getInput()};
+  llvm::append_range(fragmentOperands, blueprint.getFragments());
+  if (failed(validateFragmentAssemblyMetadata(blueprint,
+                                              rank,
+                                              fragmentOperands.size(),
+                                              operandIndices,
+                                              sourceOffsets,
+                                              flatOffsets,
+                                              flatSizes,
+                                              flatStrides)))
+    return failure();
+
+  SmallVector<int64_t> hostStrides = computeRowMajorStrides(resultType.getShape());
+  for (int64_t fragmentIndex = 0; fragmentIndex < static_cast<int64_t>(operandIndices.size()); ++fragmentIndex) {
+    Value source = fixedSource ? fixedSource : mapper.lookupOrDefault(fragmentOperands[operandIndices[fragmentIndex]]);
+    auto sourceType = dyn_cast<RankedTensorType>(source.getType());
+    if (!sourceType || !sourceType.hasStaticShape())
+      return blueprint.emitOpError("fragment assembly lowering requires static ranked tensor operands");
+
+    size_t elementSize = getElementTypeSizeInBytes(sourceType.getElementType());
+    SmallVector<int64_t, 4> fragmentOffsets;
+    SmallVector<int64_t, 4> fragmentSizes;
+    for (int64_t dim = 0; dim < rank; ++dim) {
+      int64_t flatIndex = fragmentIndex * rank + dim;
+      if (flatStrides[flatIndex] != 1)
+        return blueprint.emitOpError("fragment assembly lowering only supports unit strides");
+      fragmentOffsets.push_back(flatOffsets[flatIndex]);
+      fragmentSizes.push_back(flatSizes[flatIndex]);
+    }
+
+    if (failed(forEachContiguousDestinationChunk(
+          resultType.getShape(),
+          fragmentOffsets,
+          fragmentSizes,
+          [&](ArrayRef<int64_t> chunkOffsets, int64_t relativeSourceOffset, int64_t chunkElements) -> LogicalResult {
+            int64_t hostElementOffset = 0;
+            for (auto [dim, offset] : llvm::enumerate(chunkOffsets))
+              hostElementOffset += offset * hostStrides[dim];
+
+            FragmentAssemblyCopy copy;
+            copy.source = source;
+            copy.sourceType = sourceType;
+            copy.hostTargetIndex = hostTargetIndex;
+            copy.lane = lane;
+            copy.sourceByteOffset =
+              (getFragmentAssemblySourceElementOffset(
+                 sourceType, sourceSlots[fragmentIndex], sourceOffsets[fragmentIndex])
+               + relativeSourceOffset)
+              * static_cast<int64_t>(elementSize);
+            copy.hostByteOffset = hostElementOffset * static_cast<int64_t>(elementSize);
+            copy.byteSize = chunkElements * static_cast<int64_t>(elementSize);
+            copies.push_back(copy);
+            return success();
+          })))
+      return failure();
+  }
+
+  return copies;
+}
+
+static FailureOr<SmallVector<FragmentAssemblyCopy, 8>>
+collectTopLevelFragmentAssemblyCopies(OpResult result, RankedTensorType packedResultType, uint32_t laneCount) {
+  SmallVector<FragmentAssemblyCopy, 8> copies;
+  if (!packedResultType.hasStaticShape() || laneCount == 0)
+    return failure();
+
+  int64_t packedElementCount = packedResultType.getNumElements();
+  if (packedElementCount % static_cast<int64_t>(laneCount) != 0)
+    return failure();
+  int64_t payloadElementCount = packedElementCount / static_cast<int64_t>(laneCount);
+  size_t elementSize = getElementTypeSizeInBytes(packedResultType.getElementType());
+
+  for (OpOperand& use : result.getUses()) {
+    auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(use.getOwner());
+    if (!blueprint || blueprint->getParentOp() != blueprint->getParentOfType<func::FuncOp>())
+      return failure();
+    std::optional<StringRef> mode = blueprint.getMode();
+    std::optional<ArrayRef<int64_t>> operandIndicesAttr = blueprint.getFragmentOperandIndices();
+    std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = blueprint.getFragmentSourceOffsets();
+    std::optional<ArrayRef<int64_t>> sourceSlotsAttr = blueprint.getFragmentSourceSlots();
+    if (!mode || *mode != "fragment_assembly" || !operandIndicesAttr || !sourceOffsetsAttr || !sourceSlotsAttr)
+      return failure();
+    if (!blueprint.getOutput().hasOneUse() || !isa<func::ReturnOp>(*blueprint.getOutput().getUsers().begin()))
+      return failure();
+
+    auto hostResultType = dyn_cast<RankedTensorType>(blueprint.getOutput().getType());
+    std::optional<ArrayRef<int64_t>> stridesAttr = blueprint.getFragmentStrides();
+    if (!hostResultType || !hostResultType.hasStaticShape() || !stridesAttr)
+      return failure();
+
+    ArrayRef<int64_t> operandIndices = *operandIndicesAttr;
+    ArrayRef<int64_t> sourceOffsets = *sourceOffsetsAttr;
+    ArrayRef<int64_t> sourceSlots = *sourceSlotsAttr;
+    if (sourceSlots.size() != operandIndices.size())
+      return failure();
+    ArrayRef<int64_t> flatOffsets = blueprint.getFragmentOffsets();
+    ArrayRef<int64_t> flatSizes = blueprint.getFragmentSizes();
+    ArrayRef<int64_t> flatStrides = *stridesAttr;
+    int64_t rank = hostResultType.getRank();
+    unsigned returnIndex = blueprint.getOutput().getUses().begin()->getOperandNumber();
+    SmallVector<Value> fragmentOperands {blueprint.getInput()};
+    llvm::append_range(fragmentOperands, blueprint.getFragments());
+    if (failed(validateFragmentAssemblyMetadata(blueprint,
+                                                rank,
+                                                fragmentOperands.size(),
+                                                operandIndices,
+                                                sourceOffsets,
+                                                flatOffsets,
+                                                flatSizes,
+                                                flatStrides)))
+      return failure();
+    SmallVector<int64_t> hostStrides = computeRowMajorStrides(hostResultType.getShape());
+    for (int64_t fragmentIndex = 0; fragmentIndex < static_cast<int64_t>(operandIndices.size()); ++fragmentIndex) {
+      if (operandIndices[fragmentIndex] != static_cast<int64_t>(use.getOperandNumber()))
+        continue;
+
+      int64_t sourceElementOffset = getFragmentAssemblySourceElementOffset(
+        packedResultType, sourceSlots[fragmentIndex], sourceOffsets[fragmentIndex]);
+      int64_t lane = sourceElementOffset / payloadElementCount;
+      if (lane < 0 || lane >= static_cast<int64_t>(laneCount))
+        return failure();
+      SmallVector<int64_t, 4> fragmentOffsets;
+      SmallVector<int64_t, 4> fragmentSizes;
+      for (int64_t dim = 0; dim < rank; ++dim) {
+        int64_t flatIndex = fragmentIndex * rank + dim;
+        if (flatStrides[flatIndex] != 1)
+          return failure();
+        fragmentOffsets.push_back(flatOffsets[flatIndex]);
+        fragmentSizes.push_back(flatSizes[flatIndex]);
+      }
+
+      if (failed(forEachContiguousDestinationChunk(
+            hostResultType.getShape(),
+            fragmentOffsets,
+            fragmentSizes,
+            [&](ArrayRef<int64_t> chunkOffsets, int64_t relativeSourceOffset, int64_t chunkElements) -> LogicalResult {
+              int64_t hostElementOffset = 0;
+              for (auto [dim, offset] : llvm::enumerate(chunkOffsets))
+                hostElementOffset += offset * hostStrides[dim];
+
+              FragmentAssemblyCopy copy;
+              copy.source = result;
+              copy.sourceType = packedResultType;
+              copy.hostTargetIndex = returnIndex;
+              copy.lane = lane;
+              copy.sourceByteOffset =
+                ((sourceElementOffset % payloadElementCount) + relativeSourceOffset) * static_cast<int64_t>(elementSize);
+              copy.hostByteOffset = hostElementOffset * static_cast<int64_t>(elementSize);
+              copy.byteSize = chunkElements * static_cast<int64_t>(elementSize);
+              copies.push_back(copy);
+              return success();
+            })))
+        return failure();
+    }
+  }
+
+  return copies;
+}
+
 static Value createScaledIndexValue(IRRewriter& rewriter, Location loc, Value base, int64_t scale) {
   if (scale == 1)
     return base;
@@ -63,25 +241,32 @@ static Value createScaledIndexValue(IRRewriter& rewriter, Location loc, Value ba
 }
 
 static Value createHostTargetOffset(IRRewriter& rewriter,
-                                    tensor::ParallelInsertSliceOp insertSlice,
+                                    Location loc,
                                     ShapedType destinationType,
+                                    ArrayRef<OpFoldResult> mixedOffsets,
+                                    ArrayRef<int64_t> additionalOffsets,
                                     IRMapping& mapper) {
   int64_t elementBytes = static_cast<int64_t>(getElementTypeSizeInBytes(destinationType.getElementType()));
   SmallVector<int64_t> strides = computeRowMajorStrides(destinationType.getShape());
 
   Value totalOffset;
-  Location loc = insertSlice.getLoc();
+  for (auto [dim, offset] : llvm::enumerate(mixedOffsets)) {
-  for (auto [dim, offset] : llvm::enumerate(insertSlice.getMixedOffsets())) {
     int64_t scale = strides[dim] * elementBytes;
     Value scaledOffset;
     if (auto attr = dyn_cast<Attribute>(offset)) {
       auto intAttr = dyn_cast<IntegerAttr>(attr);
       assert(intAttr && "expected integer offset attribute");
-      scaledOffset =
+      scaledOffset = getOrCreateIndexConstant(rewriter,
-        getOrCreateIndexConstant(rewriter, rewriter.getInsertionBlock()->getParentOp(), intAttr.getInt() * scale);
+                                              rewriter.getInsertionBlock()->getParentOp(),
-    }
+                                              (intAttr.getInt() + additionalOffsets[dim]) * scale);
-    else {
+    } else {
       scaledOffset = createScaledIndexValue(rewriter, loc, mapper.lookupOrDefault(cast<Value>(offset)), scale);
+      if (additionalOffsets[dim] != 0) {
+        Value staticOffset = getOrCreateIndexConstant(rewriter,
+                                                      rewriter.getInsertionBlock()->getParentOp(),
+                                                      additionalOffsets[dim] * scale);
+        scaledOffset = arith::AddIOp::create(rewriter, loc, scaledOffset, staticOffset).getResult();
+      }
     }
 
     totalOffset =
@@ -93,9 +278,22 @@ static Value createHostTargetOffset(IRRewriter& rewriter,
   return totalOffset;
 }
 
+static Value createHostTargetOffset(IRRewriter& rewriter,
+                                    tensor::ParallelInsertSliceOp insertSlice,
+                                    ShapedType destinationType,
+                                    IRMapping& mapper) {
+  SmallVector<int64_t> zeroOffsets(destinationType.getRank(), 0);
+  return createHostTargetOffset(rewriter,
+                                insertSlice.getLoc(),
+                                destinationType,
+                                insertSlice.getMixedOffsets(),
+                                zeroOffsets,
+                                mapper);
+}
+
 } // namespace
 
-LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatComputeBatch computeBatchOp,
+LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatScheduledComputeBatch computeBatchOp,
                                                             IRRewriter& rewriter) {
   Location loc = computeBatchOp.getLoc();
   Block& oldBlock = computeBatchOp.getBody().front();
@@ -110,7 +308,7 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
       "resultful compute_batch lowering currently requires a spat.in_parallel terminator");
   }
 
-  auto coreIds = getPimCoreIdsForBatchOp(computeBatchOp, coreId);
+  auto coreIds = getRequiredScheduledBatchCoreIds(computeBatchOp, "spatial compute_batch core id");
   if (failed(coreIds))
     return failure();
   SmallVector<Value> batchWeights(computeBatchOp.getWeights().begin(), computeBatchOp.getWeights().end());
@@ -130,14 +328,32 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
   coreBatchOp->setAttr(onnx_mlir::kCoreIdsAttrName, rewriter.getDenseI32ArrayAttr(*coreIds));
 
   SmallVector<unsigned> returnOperandIndices;
+  SmallVector<SmallVector<FragmentAssemblyCopyRun, 1>, 4> fragmentAssemblyRunsByResult;
   if (computeBatchOp.getNumResults() != 0) {
-    returnOperandIndices.resize(computeBatchOp.getNumResults());
+    returnOperandIndices.resize(computeBatchOp.getNumResults(), std::numeric_limits<unsigned>::max());
+    fragmentAssemblyRunsByResult.resize(computeBatchOp.getNumResults());
     for (auto [resultIndex, result] : llvm::enumerate(computeBatchOp.getResults())) {
+      if (result.use_empty())
+        continue;
       FailureOr<unsigned> returnOperandIndex = getDirectReturnOperandIndex(cast<OpResult>(result));
-      if (failed(returnOperandIndex))
+      if (succeeded(returnOperandIndex)) {
-        return computeBatchOp.emitOpError(
-          "resultful compute_batch lowering currently requires each result to be used directly by func.return");
         returnOperandIndices[resultIndex] = *returnOperandIndex;
+        continue;
+      }
+
+      auto resultType = dyn_cast<RankedTensorType>(result.getType());
+      if (!resultType || !resultType.hasStaticShape())
+        return computeBatchOp.emitOpError(
+          "resultful compute_batch publication lowering requires static ranked tensor results");
+      FailureOr<SmallVector<FragmentAssemblyCopy, 8>> fragmentAssemblyCopies =
+        collectTopLevelFragmentAssemblyCopies(cast<OpResult>(result), resultType, computeBatchOp.getLaneCount());
+      if (failed(fragmentAssemblyCopies))
+        return computeBatchOp.emitOpError("failed to collect top-level fragment assembly copies for compute_batch result");
+      FailureOr<SmallVector<FragmentAssemblyCopyRun, 8>> fragmentAssemblyRuns =
+        groupFragmentAssemblyCopyRuns(*fragmentAssemblyCopies, computeBatchOp.getLaneCount());
+      if (failed(fragmentAssemblyRuns))
+        return computeBatchOp.emitOpError("failed to group top-level fragment assembly copies into regular runs");
+      fragmentAssemblyRunsByResult[resultIndex].assign(fragmentAssemblyRuns->begin(), fragmentAssemblyRuns->end());
     }
   }
 
@@ -152,6 +368,7 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
     rewriter.createBlock(&coreBatchOp.getBody(), coreBatchOp.getBody().end(), TypeRange(blockArgTypes), blockArgLocs);
 
   IRMapping mapper;
+  SmallPtrSet<Value, 4> hostResidentTensors;
   rewriter.setInsertionPointToStart(newBlock);
   auto oldLaneArg = computeBatchOp.getLaneArgument();
   if (!oldLaneArg)
@@ -195,6 +412,23 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
     if (isa<spatial::SpatYieldOp>(op))
       continue;
 
+    // Cloning a region-bearing operation may leave the rewriter inside that
+    // region. Every old-block operation is lowered at the core-batch body
+    // boundary.
+    rewriter.setInsertionPointToEnd(newBlock);
+
+    if (auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(op)) {
+      std::optional<StringRef> modeAttr = blueprint.getMode();
+      if (modeAttr && *modeAttr == "fragment_assembly") {
+        for (Operation* user : blueprint.getOutput().getUsers()) {
+          if (!isa<tensor::ParallelInsertSliceOp>(user))
+            return blueprint.emitOpError(
+              "fragment assembly blueprint lowering expects only tensor.parallel_insert_slice users");
+        }
+        continue;
+      }
+    }
+
     if (auto parallelOp = dyn_cast<spatial::SpatInParallelOp>(op)) {
       auto firstOutputArg = computeBatchOp.getOutputArgument(0);
       if (!firstOutputArg)
@@ -211,10 +445,75 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
         unsigned resultIndex = outputArg.getArgNumber() - firstOutputArg->getArgNumber();
         if (resultIndex >= returnOperandIndices.size())
           return insertSlice.emitOpError("result index out of range while lowering host batch output");
+        bool hasDirectReturn = returnOperandIndices[resultIndex] != std::numeric_limits<unsigned>::max();
+        bool hasFragmentAssembly = resultIndex < fragmentAssemblyRunsByResult.size()
+                                   && !fragmentAssemblyRunsByResult[resultIndex].empty();
+        if (!hasDirectReturn && !hasFragmentAssembly)
+          continue;
 
         Value mappedSource = mapper.lookup(insertSlice.getSource());
+
+        if (hasFragmentAssembly) {
+          BlockArgument laneArg = coreBatchOp.getLaneArgument();
+          auto mappedSourceType = dyn_cast<ShapedType>(mappedSource.getType());
+          if (!mappedSourceType || !mappedSourceType.hasStaticShape())
+            return insertSlice.emitOpError("fragment assembly batch lowering requires a static ranked lane-local source");
+          DenseMap<unsigned, Value> updatedOutputs;
+          for (const FragmentAssemblyCopyRun& run : fragmentAssemblyRunsByResult[resultIndex]) {
+            Value outputTensor = updatedOutputs.lookup(run.hostTargetIndex);
+            if (!outputTensor)
+              outputTensor = outputTensors[run.hostTargetIndex](rewriter, insertSlice.getLoc());
+            FragmentAssemblyCopyRun mappedRun = run;
+            mappedRun.source = mappedSource;
+            FailureOr<Value> updated =
+              emitFragmentAssemblyCopyRuns(rewriter,
+                                           insertSlice.getLoc(),
+                                           ArrayRef<FragmentAssemblyCopyRun> {mappedRun},
+                                           outputTensor,
+                                           coreBatchOp.getOperation(),
+                                           laneArg);
+            if (failed(updated))
+              return failure();
+            updatedOutputs[run.hostTargetIndex] = *updated;
+          }
+          continue;
+        }
+
         Value hostTarget = getOrCreateHostOutputTensor(resultIndex, insertSlice.getLoc());
         auto hostTargetType = cast<ShapedType>(hostTarget.getType());
+        if (auto blueprint =
+              insertSlice.getSource().getDefiningOp<spatial::SpatBlueprintOp>()) {
+          std::optional<StringRef> modeAttr = blueprint.getMode();
+          if (modeAttr && *modeAttr == "fragment_assembly") {
+            FailureOr<SmallVector<FragmentAssemblyCopy, 8>> fragmentAssemblyCopies =
+              collectFragmentAssemblyCopiesFromBlueprint(blueprint, mapper, /*lane=*/0, /*hostTargetIndex=*/0);
+            if (failed(fragmentAssemblyCopies))
+              return failure();
+            FailureOr<SmallVector<FragmentAssemblyCopyRun, 8>> fragmentAssemblyRuns =
+              groupFragmentAssemblyCopyRuns(*fragmentAssemblyCopies, /*laneCount=*/1);
+            if (failed(fragmentAssemblyRuns))
+              return failure();
+            SmallVector<int64_t> zeroOffsets(hostTargetType.getRank(), 0);
+            Value baseHostOffset = createHostTargetOffset(rewriter,
+                                                         blueprint.getLoc(),
+                                                         hostTargetType,
+                                                         insertSlice.getMixedOffsets(),
+                                                         zeroOffsets,
+                                                         mapper);
+            FailureOr<Value> updatedHostTarget = emitFragmentAssemblyCopyRuns(rewriter,
+                                                                              blueprint.getLoc(),
+                                                                              *fragmentAssemblyRuns,
+                                                                              hostTarget,
+                                                                              coreBatchOp.getOperation(),
+                                                                              std::nullopt,
+                                                                              baseHostOffset);
+            if (failed(updatedHostTarget))
+              return failure();
+            hostOutputTensors[resultIndex] = *updatedHostTarget;
+            continue;
+          }
+        }
+
         Value hostTargetOffset = createHostTargetOffset(rewriter, insertSlice, hostTargetType, mapper);
         Value zeroOffset = getOrCreateIndexConstant(rewriter, coreBatchOp.getOperation(), 0);
         auto sizeAttr = getTensorSizeInBytesAttr(rewriter, coreBatchOp.getOperation(), mappedSource);
@@ -236,8 +535,10 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
       if (isa_and_present<memref::GetGlobalOp>(toTensorOp.getBuffer().getDefiningOp())) {
         Operation* cloned = rewriter.clone(op, mapper);
         auto clonedTensor = cloned->getResult(0);
-        if (isUsedOnlyAsExplicitHostOperand(toTensorOp.getResult())) {
+        if (isUsedOnlyAsExplicitHostOperand(toTensorOp.getResult())
+            || isUsedOnlyByExtractSlices(toTensorOp.getResult())) {
           mapper.map(toTensorOp.getResult(), clonedTensor);
+          hostResidentTensors.insert(toTensorOp.getResult());
           continue;
         }
         auto clonedType = cast<ShapedType>(clonedTensor.getType());
@@ -255,6 +556,28 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
       }
     }
 
+    if (auto extractSlice = dyn_cast<tensor::ExtractSliceOp>(op);
+        extractSlice && hostResidentTensors.contains(extractSlice.getSource())) {
+      Operation* cloned = rewriter.clone(op, mapper);
+      Value hostSlice = cloned->getResult(0);
+      auto outputBuffer = createEmptyTensorFromShaped(rewriter, loc, cast<ShapedType>(hostSlice.getType()));
+      Value zeroOffset = getOrCreateIndexConstant(rewriter, coreBatchOp.getOperation(), 0);
+      auto sizeAttr = getTensorSizeInBytesAttr(rewriter, coreBatchOp.getOperation(), hostSlice);
+      if (failed(sizeAttr))
+        return failure();
+      auto copied = pim::PimMemCopyHostToDevOp::create(rewriter,
+                                                       loc,
+                                                       outputBuffer.getType(),
+                                                       zeroOffset,
+                                                       zeroOffset,
+                                                       outputBuffer,
+                                                       hostSlice,
+                                                       *sizeAttr)
+                      .getOutput();
+      mapper.map(extractSlice.getResult(), copied);
+      continue;
+    }
+
     for (auto [operandIndex, operand] : llvm::enumerate(op.getOperands())) {
       if (!isa<TensorType>(operand.getType()) || mapper.contains(operand))
         continue;
@@ -264,9 +587,15 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeBatchOp(spatial::SpatCompute
       Operation* definingOp = operand.getDefiningOp();
       if (definingOp && definingOp->getBlock() == &oldBlock)
         continue;
+      if (definingOp && definingOp->hasTrait<OpTrait::ConstantLike>())
+        continue;
 
-      return computeBatchOp.emitOpError(
+      InFlightDiagnostic diagnostic =
-        "expected external tensor communication to be materialized in Spatial before batch lowering");
+        computeBatchOp.emitOpError("expected external tensor communication to be materialized in Spatial before batch lowering");
+      diagnostic << " while cloning nested op '" << op.getName() << "' tensor operand #" << operandIndex;
+      if (definingOp)
+        diagnostic << " from external producer '" << definingOp->getName() << "'";
+      return diagnostic;
     }
 
     Operation* cloned = rewriter.clone(op, mapper);

src/PIM/Conversion/SpatialToPim/Common.cpp

@@ -1,11 +1,20 @@
 #include "mlir/IR/ValueRange.h"
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+
 #include "llvm/ADT/STLExtras.h"
 
 #include <cassert>
+#include <limits>
 
 #include "Common.hpp"
+#include "src/Accelerators/PIM/Common/IR/AffineUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
+#include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
 
 using namespace llvm;
 using namespace mlir;
@@ -51,6 +60,23 @@ bool hasLaterUserInBlock(mlir::Value value, Operation* operation) {
   return false;
 }
 
+static bool isTensorView(mlir::Value value) {
+  return isa_and_nonnull<tensor::CastOp,
+                         tensor::CollapseShapeOp,
+                         tensor::ExpandShapeOp,
+                         tensor::ExtractSliceOp,
+                         tensor::ReshapeOp>(value.getDefiningOp());
+}
+
+static bool isLoopCarriedOutput(mlir::Value operand, Operation* operation) {
+  auto argument = dyn_cast<BlockArgument>(operand);
+  if (!argument || argument.getArgNumber() == 0 || operation->getBlock() != argument.getOwner())
+    return false;
+  auto loop = dyn_cast_or_null<scf::ForOp>(argument.getOwner()->getParentOp());
+  return loop && cast<scf::YieldOp>(loop.getBody()->getTerminator())
+                   .getOperand(argument.getArgNumber() - 1) == operation->getResult(0);
+}
+
 mlir::Value getBestOutputTensorFromOperandsOrAllocate(RewriterBase& rewriter, Operation* operation) {
   assert("Only support operations with a single result" && operation->getNumResults() == 1);
   mlir::Value result = operation->getResult(0);
@@ -59,7 +85,11 @@ mlir::Value getBestOutputTensorFromOperandsOrAllocate(RewriterBase& rewriter, Op
 
   SmallVector<mlir::Value> operands = getOpOperandsSortedByUses(operation);
   auto validOperands = make_filter_range(operands, [operation, resultType](mlir::Value operand) {
-    return operand.getType() == resultType && !hasLaterUserInBlock(operand, operation);
+    return operand.getType() == resultType
+      && (!isa<BlockArgument>(operand) || isLoopCarriedOutput(operand, operation))
+      && !operand.getDefiningOp<arith::ConstantOp>()
+      && !isTensorView(operand)
+      && !hasLaterUserInBlock(operand, operation);
   });
   auto bestOperand = validOperands.begin();
 
@@ -72,4 +102,496 @@ mlir::Value getBestOutputTensorFromOperandsOrAllocate(RewriterBase& rewriter, Op
     rewriter, operation->getLoc(), resultShapedType.getShape(), resultShapedType.getElementType());
 }
 
+LogicalResult validateFragmentAssemblyMetadata(spatial::SpatBlueprintOp blueprint,
+                                               int64_t resultRank,
+                                               size_t operandCount,
+                                               ArrayRef<int64_t> operandIndices,
+                                               ArrayRef<int64_t> sourceOffsets,
+                                               ArrayRef<int64_t> flatOffsets,
+                                               ArrayRef<int64_t> flatSizes,
+                                               ArrayRef<int64_t> flatStrides) {
+  if (operandIndices.size() != sourceOffsets.size())
+    return blueprint.emitOpError("fragment assembly operand index and source offset counts must match");
+  if (flatOffsets.size() != flatSizes.size())
+    return blueprint.emitOpError("fragment assembly offset and size arrays must have matching lengths");
+  if (flatStrides.size() != flatOffsets.size())
+    return blueprint.emitOpError("fragment assembly stride and offset arrays must have matching lengths");
+  if (flatOffsets.size() != operandIndices.size() * static_cast<size_t>(resultRank))
+    return blueprint.emitOpError("fragment assembly metadata must provide one rank-sized offset/size/stride tuple per fragment");
+
+  for (auto [fragmentIndex, operandIndex] : llvm::enumerate(operandIndices)) {
+    if (operandIndex < 0 || operandIndex >= static_cast<int64_t>(operandCount))
+      return blueprint.emitOpError("fragment assembly operand index is out of range");
+    if (sourceOffsets[fragmentIndex] < 0)
+      return blueprint.emitOpError("fragment assembly source offsets must be nonnegative");
+  }
+
+  return success();
+}
+
+static SmallVector<int64_t, 4> expandFlatElementIndex(int64_t flatIndex, ArrayRef<int64_t> shape) {
+  SmallVector<int64_t, 4> indices(shape.size(), 0);
+  for (int64_t dim = static_cast<int64_t>(shape.size()) - 1; dim >= 0; --dim) {
+    indices[dim] = flatIndex % shape[dim];
+    flatIndex /= shape[dim];
+  }
+  return indices;
+}
+
+FailureOr<SmallVector<int64_t, 4>>
+getStaticSliceOffsetsForElementOffset(Operation* anchor,
+                                      ShapedType sourceType,
+                                      ArrayRef<int64_t> fragmentShape,
+                                      int64_t sourceElementOffset,
+                                      StringRef fieldName) {
+  if (!sourceType.hasStaticShape())
+    return (anchor->emitOpError() << fieldName << " requires a static source shape"), failure();
+  if (sourceElementOffset < 0)
+    return (anchor->emitOpError() << fieldName << " requires a nonnegative source element offset"), failure();
+  if (sourceType.getRank() != static_cast<int64_t>(fragmentShape.size()))
+    return (anchor->emitOpError() << fieldName << " requires fragment rank to match source rank"), failure();
+
+  int64_t sourceElementCount = sourceType.getNumElements();
+  int64_t fragmentElementCount = 1;
+  for (int64_t dim = 0; dim < sourceType.getRank(); ++dim) {
+    if (fragmentShape[dim] < 0)
+      return (anchor->emitOpError() << fieldName << " requires nonnegative fragment sizes"), failure();
+    fragmentElementCount *= fragmentShape[dim];
+  }
+  if (sourceElementOffset + fragmentElementCount > sourceElementCount)
+    return (anchor->emitOpError() << fieldName << " exceeds the source tensor bounds"), failure();
+
+  SmallVector<int64_t, 4> sliceOffsets = expandFlatElementIndex(sourceElementOffset, sourceType.getShape());
+  for (int64_t dim = 0; dim < sourceType.getRank(); ++dim) {
+    if (sliceOffsets[dim] + fragmentShape[dim] > sourceType.getDimSize(dim))
+      return (anchor->emitOpError() << fieldName << " does not describe a valid unit-stride slice"), failure();
+  }
+  return sliceOffsets;
+}
+
+LogicalResult
+forEachContiguousDestinationChunk(ArrayRef<int64_t> destShape,
+                                  ArrayRef<int64_t> baseOffsets,
+                                  ArrayRef<int64_t> sizes,
+                                  llvm::function_ref<LogicalResult(ArrayRef<int64_t>, int64_t, int64_t)> callback) {
+  int64_t rank = static_cast<int64_t>(sizes.size());
+  int64_t suffixStart = rank - 1;
+  while (suffixStart > 0 && sizes[suffixStart] == destShape[suffixStart])
+    --suffixStart;
+  if (sizes[suffixStart] == destShape[suffixStart] && suffixStart == 0)
+    suffixStart = 0;
+  else
+    ++suffixStart;
+
+  int64_t chunkElements = 1;
+  for (int64_t dim = suffixStart; dim < rank; ++dim)
+    chunkElements *= sizes[dim];
+
+  SmallVector<int64_t, 4> prefixExtents(sizes.begin(), sizes.begin() + suffixStart);
+  SmallVector<int64_t, 4> current(prefixExtents.size(), 0);
+  int64_t sourceChunkOrdinal = 0;
+
+  auto visit = [&](auto&& visit, int64_t dim) -> LogicalResult {
+    if (dim == static_cast<int64_t>(prefixExtents.size())) {
+      SmallVector<int64_t, 4> chunkOffsets(baseOffsets.begin(), baseOffsets.end());
+      for (int64_t prefixDim = 0; prefixDim < static_cast<int64_t>(current.size()); ++prefixDim)
+        chunkOffsets[prefixDim] += current[prefixDim];
+      if (failed(callback(chunkOffsets, sourceChunkOrdinal * chunkElements, chunkElements)))
+        return failure();
+      ++sourceChunkOrdinal;
+      return success();
+    }
+
+    for (int64_t index = 0; index < prefixExtents[dim]; ++index) {
+      current[dim] = index;
+      if (failed(visit(visit, dim + 1)))
+        return failure();
+    }
+    return success();
+  };
+
+  if (prefixExtents.empty())
+    return callback(baseOffsets, 0, chunkElements);
+  return visit(visit, 0);
+}
+
+int64_t getFragmentAssemblySourceElementOffset(RankedTensorType sourceType,
+                                               int64_t sourceSlot,
+                                               int64_t sourceOffset) {
+  assert(sourceType.getRank() > 0 && sourceType.hasStaticShape()
+         && "fragment assembly source must have a static leading slot dimension");
+  return sourceSlot * (sourceType.getNumElements() / sourceType.getDimSize(0)) + sourceOffset;
+}
+
+static mlir::Value
+createSteppedOffset(OpBuilder& builder, Location loc, mlir::Value start, mlir::Value index,
+                    int64_t stepBytes, Operation *constantAnchor) {
+  if (stepBytes == 0)
+    return start;
+  return createOrFoldAffineApply(
+    builder, loc, builder.getAffineDimExpr(0) + builder.getAffineDimExpr(1) * stepBytes,
+    ValueRange {start, index}, constantAnchor);
+}
+
+static mlir::Value createIndexedOffset(OpBuilder& builder,
+                                       Location loc,
+                                       mlir::Value indexArg,
+                                       ArrayRef<int64_t> values,
+                                       Operation *constantAnchor) {
+  assert(!values.empty() && "expected lane-indexed values");
+  if (llvm::all_of(values.drop_front(), [&](int64_t value) { return value == values.front(); }))
+    return getOrCreateIndexConstant(builder, constantAnchor, values.front());
+
+  if (values.size() >= 2) {
+    int64_t step = values[1] - values[0];
+    bool arithmetic = llvm::all_of(llvm::seq<size_t>(2, values.size()), [&](size_t index) {
+      return values[index] == values.front() + static_cast<int64_t>(index) * step;
+    });
+    if (arithmetic) {
+      return createOrFoldAffineApply(
+        builder, loc, builder.getAffineDimExpr(0) * step + values.front(),
+        ValueRange {indexArg}, constantAnchor);
+    }
+  }
+
+  RankedTensorType tableType = RankedTensorType::get(
+    {static_cast<int64_t>(values.size())}, builder.getI64Type());
+  DenseElementsAttr tableAttr = DenseElementsAttr::get(tableType, values);
+  mlir::Value table = getOrCreateConstant(builder, constantAnchor, tableAttr, tableType);
+  mlir::Value selected = tensor::ExtractOp::create(builder, loc, table, ValueRange {indexArg});
+  return arith::IndexCastOp::create(builder, loc, builder.getIndexType(), selected).getResult();
+}
+
+struct FragmentAssemblyCopyRunFamily {
+  FragmentAssemblyCopyRun prototype;
+  SmallVector<int64_t, 8> sourceRunStartDeltas;
+  SmallVector<int64_t, 8> hostRunStartDeltas;
+};
+
+static bool computeUniformRunStartDelta(ArrayRef<int64_t> prototypeStarts,
+                                        ArrayRef<int64_t> runStarts,
+                                        int64_t& delta) {
+  if (prototypeStarts.size() != runStarts.size() || prototypeStarts.empty())
+    return false;
+
+  delta = runStarts.front() - prototypeStarts.front();
+  return llvm::all_of(llvm::zip_equal(prototypeStarts, runStarts), [&](auto pair) {
+    auto [prototypeStart, runStart] = pair;
+    return runStart - prototypeStart == delta;
+  });
+}
+
+static bool canMergeFragmentAssemblyCopyRunIntoFamily(const FragmentAssemblyCopyRunFamily& family,
+                                                      const FragmentAssemblyCopyRun& run,
+                                                      int64_t& sourceRunStartDelta,
+                                                      int64_t& hostRunStartDelta) {
+  const FragmentAssemblyCopyRun& prototype = family.prototype;
+  if (prototype.source != run.source || prototype.sourceType != run.sourceType
+      || prototype.hostTargetIndex != run.hostTargetIndex || prototype.count != run.count
+      || prototype.sourceStepBytes != run.sourceStepBytes || prototype.hostStepBytes != run.hostStepBytes
+      || prototype.byteSize != run.byteSize)
+    return false;
+
+  if (!computeUniformRunStartDelta(prototype.sourceStartBytesByLane, run.sourceStartBytesByLane, sourceRunStartDelta))
+    return false;
+  return computeUniformRunStartDelta(prototype.hostStartBytesByLane, run.hostStartBytesByLane, hostRunStartDelta);
+}
+
+static SmallVector<FragmentAssemblyCopyRunFamily, 8>
+groupFragmentAssemblyCopyRunFamilies(ArrayRef<FragmentAssemblyCopyRun> runs) {
+  auto compareRunStarts = [](ArrayRef<int64_t> lhs, ArrayRef<int64_t> rhs) {
+    return std::lexicographical_compare(lhs.begin(), lhs.end(), rhs.begin(), rhs.end());
+  };
+
+  SmallVector<FragmentAssemblyCopyRun, 8> sortedRuns(runs.begin(), runs.end());
+  llvm::sort(sortedRuns, [&](const FragmentAssemblyCopyRun& lhs, const FragmentAssemblyCopyRun& rhs) {
+    if (lhs.hostTargetIndex != rhs.hostTargetIndex)
+      return lhs.hostTargetIndex < rhs.hostTargetIndex;
+    if (lhs.source != rhs.source)
+      return lhs.source.getAsOpaquePointer() < rhs.source.getAsOpaquePointer();
+    if (lhs.byteSize != rhs.byteSize)
+      return lhs.byteSize < rhs.byteSize;
+    if (lhs.count != rhs.count)
+      return lhs.count < rhs.count;
+    if (lhs.sourceStepBytes != rhs.sourceStepBytes)
+      return lhs.sourceStepBytes < rhs.sourceStepBytes;
+    if (lhs.hostStepBytes != rhs.hostStepBytes)
+      return lhs.hostStepBytes < rhs.hostStepBytes;
+    if (compareRunStarts(lhs.sourceStartBytesByLane, rhs.sourceStartBytesByLane))
+      return true;
+    if (compareRunStarts(rhs.sourceStartBytesByLane, lhs.sourceStartBytesByLane))
+      return false;
+    return compareRunStarts(lhs.hostStartBytesByLane, rhs.hostStartBytesByLane);
+  });
+
+  SmallVector<FragmentAssemblyCopyRunFamily, 8> families;
+  for (const FragmentAssemblyCopyRun& run : sortedRuns) {
+    int64_t sourceRunStartDelta = 0;
+    int64_t hostRunStartDelta = 0;
+    if (!families.empty()
+        && canMergeFragmentAssemblyCopyRunIntoFamily(
+          families.back(), run, sourceRunStartDelta, hostRunStartDelta)) {
+      families.back().sourceRunStartDeltas.push_back(sourceRunStartDelta);
+      families.back().hostRunStartDeltas.push_back(hostRunStartDelta);
+      continue;
+    }
+
+    FragmentAssemblyCopyRunFamily family;
+    family.prototype = run;
+    family.sourceRunStartDeltas.push_back(0);
+    family.hostRunStartDeltas.push_back(0);
+    families.push_back(std::move(family));
+  }
+
+  return families;
+}
+
+FailureOr<SmallVector<FragmentAssemblyCopyRun, 8>>
+groupFragmentAssemblyCopyRuns(ArrayRef<FragmentAssemblyCopy> copies, uint32_t laneCount) {
+  if (laneCount == 0)
+    return failure();
+
+  struct LaneLocalCopyRun {
+    FragmentAssemblyCopyRun run;
+    int64_t lane = 0;
+  };
+
+  SmallVector<FragmentAssemblyCopy, 8> sortedCopies(copies.begin(), copies.end());
+  llvm::sort(sortedCopies, [](const FragmentAssemblyCopy& lhs, const FragmentAssemblyCopy& rhs) {
+    if (lhs.hostTargetIndex != rhs.hostTargetIndex)
+      return lhs.hostTargetIndex < rhs.hostTargetIndex;
+    if (lhs.source != rhs.source)
+      return lhs.source.getAsOpaquePointer() < rhs.source.getAsOpaquePointer();
+    if (lhs.lane != rhs.lane)
+      return lhs.lane < rhs.lane;
+    if (lhs.byteSize != rhs.byteSize)
+      return lhs.byteSize < rhs.byteSize;
+    if (lhs.sourceByteOffset != rhs.sourceByteOffset)
+      return lhs.sourceByteOffset < rhs.sourceByteOffset;
+    return lhs.hostByteOffset < rhs.hostByteOffset;
+  });
+
+  SmallVector<LaneLocalCopyRun, 8> laneRuns;
+  for (const FragmentAssemblyCopy& copy : sortedCopies) {
+    if (copy.lane < 0 || copy.lane >= static_cast<int64_t>(laneCount))
+      return failure();
+
+    if (!laneRuns.empty()) {
+      LaneLocalCopyRun& laneRun = laneRuns.back();
+      FragmentAssemblyCopyRun& run = laneRun.run;
+      if (run.source == copy.source && run.sourceType == copy.sourceType
+          && run.hostTargetIndex == copy.hostTargetIndex && laneRun.lane == copy.lane && run.byteSize == copy.byteSize
+          && run.sourceStartBytesByLane.size() == 1 && run.hostStartBytesByLane.size() == 1) {
+        int64_t previousSourceOffset = run.sourceStartBytesByLane.front() + (run.count - 1) * run.sourceStepBytes;
+        int64_t previousHostOffset = run.hostStartBytesByLane.front() + (run.count - 1) * run.hostStepBytes;
+        int64_t sourceDelta = copy.sourceByteOffset - previousSourceOffset;
+        int64_t hostDelta = copy.hostByteOffset - previousHostOffset;
+        if (run.count == 1) {
+          run.sourceStepBytes = sourceDelta;
+          run.hostStepBytes = hostDelta;
+          ++run.count;
+          continue;
+        }
+        if (run.sourceStepBytes == sourceDelta && run.hostStepBytes == hostDelta) {
+          ++run.count;
+          continue;
+        }
+      }
+    }
+
+    LaneLocalCopyRun laneRun;
+    laneRun.run.source = copy.source;
+    laneRun.run.sourceType = copy.sourceType;
+    laneRun.run.hostTargetIndex = copy.hostTargetIndex;
+    laneRun.run.count = 1;
+    laneRun.run.byteSize = copy.byteSize;
+    laneRun.run.sourceStartBytesByLane.push_back(copy.sourceByteOffset);
+    laneRun.run.hostStartBytesByLane.push_back(copy.hostByteOffset);
+    laneRun.lane = copy.lane;
+    laneRuns.push_back(std::move(laneRun));
+  }
+
+  SmallVector<FragmentAssemblyCopyRun, 8> mergedRuns;
+  for (const LaneLocalCopyRun& laneRun : laneRuns) {
+    size_t laneIndex = static_cast<size_t>(laneRun.lane);
+    auto mergedIt = llvm::find_if(mergedRuns, [&](const FragmentAssemblyCopyRun& run) {
+      return run.source == laneRun.run.source && run.sourceType == laneRun.run.sourceType
+             && run.hostTargetIndex == laneRun.run.hostTargetIndex && run.count == laneRun.run.count
+             && run.byteSize == laneRun.run.byteSize && run.sourceStepBytes == laneRun.run.sourceStepBytes
+             && run.hostStepBytes == laneRun.run.hostStepBytes && laneIndex < run.sourceStartBytesByLane.size()
+             && run.sourceStartBytesByLane[laneIndex] == std::numeric_limits<int64_t>::min();
+    });
+
+    if (mergedIt == mergedRuns.end()) {
+      FragmentAssemblyCopyRun merged = laneRun.run;
+      merged.sourceStartBytesByLane.assign(laneCount, std::numeric_limits<int64_t>::min());
+      merged.hostStartBytesByLane.assign(laneCount, std::numeric_limits<int64_t>::min());
+      merged.sourceStartBytesByLane[laneIndex] = laneRun.run.sourceStartBytesByLane.front();
+      merged.hostStartBytesByLane[laneIndex] = laneRun.run.hostStartBytesByLane.front();
+      mergedRuns.push_back(std::move(merged));
+      continue;
+    }
+
+    mergedIt->sourceStartBytesByLane[laneIndex] = laneRun.run.sourceStartBytesByLane.front();
+    mergedIt->hostStartBytesByLane[laneIndex] = laneRun.run.hostStartBytesByLane.front();
+  }
+
+  for (const FragmentAssemblyCopyRun& run : mergedRuns) {
+    if (llvm::any_of(run.sourceStartBytesByLane,
+                     [](int64_t value) { return value == std::numeric_limits<int64_t>::min(); }))
+      return failure();
+    if (llvm::any_of(run.hostStartBytesByLane,
+                     [](int64_t value) { return value == std::numeric_limits<int64_t>::min(); }))
+      return failure();
+  }
+
+  return mergedRuns;
+}
+
+static FailureOr<mlir::Value> emitFragmentAssemblyCopyRun(OpBuilder& builder,
+                                                          Location loc,
+                                                          const FragmentAssemblyCopyRun& run,
+                                                          mlir::Value hostTarget,
+                                                          Operation* anchor,
+                                                          std::optional<mlir::Value> laneArg,
+                                                          mlir::Value baseHostOffset,
+                                                          mlir::Value sourceRunStartDelta = {},
+                                                          mlir::Value hostRunStartDelta = {}) {
+  auto sizeAttr = pim::getCheckedI32Attr(builder, anchor, run.byteSize, "fragment assembly host copy byte size");
+  if (failed(sizeAttr))
+    return failure();
+
+  mlir::Value hostStart;
+  mlir::Value sourceStart;
+  if (laneArg) {
+    hostStart = createIndexedOffset(builder, loc, *laneArg, run.hostStartBytesByLane, anchor);
+    sourceStart = createIndexedOffset(builder, loc, *laneArg, run.sourceStartBytesByLane, anchor);
+  } else {
+    hostStart = getOrCreateIndexConstant(builder, anchor, run.hostStartBytesByLane.front());
+    sourceStart = getOrCreateIndexConstant(builder, anchor, run.sourceStartBytesByLane.front());
+  }
+
+  if (hostRunStartDelta)
+    hostStart = arith::AddIOp::create(builder, loc, hostStart, hostRunStartDelta).getResult();
+  if (sourceRunStartDelta)
+    sourceStart = arith::AddIOp::create(builder, loc, sourceStart, sourceRunStartDelta).getResult();
+  if (baseHostOffset)
+    hostStart = arith::AddIOp::create(builder, loc, baseHostOffset, hostStart).getResult();
+
+  if (run.count == 1) {
+    return pim::PimMemCopyDevToHostOp::create(builder,
+                                              loc,
+                                              hostTarget.getType(),
+                                              hostStart,
+                                              sourceStart,
+                                              hostTarget,
+                                              run.source,
+                                              *sizeAttr)
+      .getOutput();
+  }
+
+  mlir::Value lowerBound = getOrCreateIndexConstant(builder, anchor, 0);
+  mlir::Value upperBound = getOrCreateIndexConstant(builder, anchor, run.count);
+  mlir::Value step = getOrCreateIndexConstant(builder, anchor, 1);
+  FailureOr<NormalizedLoopResult> loop = buildNormalizedScfFor(
+    builder,
+    loc,
+    lowerBound,
+    upperBound,
+    step,
+    ValueRange {hostTarget},
+    [&](OpBuilder& loopBuilder,
+        Location bodyLoc,
+        mlir::Value flatIndex,
+        ValueRange iterArgs,
+        SmallVectorImpl<mlir::Value>& yielded) {
+      mlir::Value hostOffset = createSteppedOffset(
+        loopBuilder, bodyLoc, hostStart, flatIndex, run.hostStepBytes, anchor);
+      mlir::Value sourceOffset =
+        createSteppedOffset(loopBuilder, bodyLoc, sourceStart, flatIndex, run.sourceStepBytes, anchor);
+      mlir::Value copied =
+        pim::PimMemCopyDevToHostOp::create(loopBuilder,
+                                           bodyLoc,
+                                           iterArgs.front().getType(),
+                                           hostOffset,
+                                           sourceOffset,
+                                           iterArgs.front(),
+                                           run.source,
+                                           *sizeAttr)
+          .getOutput();
+      yielded.push_back(copied);
+      return success();
+    });
+  if (failed(loop))
+    return failure();
+  return loop->results.front();
+}
+
+static FailureOr<mlir::Value> emitFragmentAssemblyCopyRunFamily(OpBuilder& builder,
+                                                                Location loc,
+                                                                const FragmentAssemblyCopyRunFamily& family,
+                                                                mlir::Value hostTarget,
+                                                                Operation* anchor,
+                                                                std::optional<mlir::Value> laneArg,
+                                                                mlir::Value baseHostOffset) {
+  if (family.sourceRunStartDeltas.size() == 1)
+    return emitFragmentAssemblyCopyRun(
+      builder, loc, family.prototype, hostTarget, anchor, laneArg, baseHostOffset);
+
+  mlir::Value lowerBound = getOrCreateIndexConstant(builder, anchor, 0);
+  mlir::Value upperBound = getOrCreateIndexConstant(builder, anchor, family.sourceRunStartDeltas.size());
+  mlir::Value step = getOrCreateIndexConstant(builder, anchor, 1);
+  FailureOr<NormalizedLoopResult> outerLoop = buildNormalizedScfFor(
+    builder,
+    loc,
+    lowerBound,
+    upperBound,
+    step,
+    ValueRange {hostTarget},
+    [&](OpBuilder& loopBuilder,
+        Location bodyLoc,
+        mlir::Value runIndex,
+        ValueRange iterArgs,
+        SmallVectorImpl<mlir::Value>& yielded) {
+      mlir::Value sourceRunStartDelta =
+        createIndexedOffset(loopBuilder, bodyLoc, runIndex, family.sourceRunStartDeltas, anchor);
+      mlir::Value hostRunStartDelta =
+        createIndexedOffset(loopBuilder, bodyLoc, runIndex, family.hostRunStartDeltas, anchor);
+      FailureOr<mlir::Value> copied = emitFragmentAssemblyCopyRun(loopBuilder,
+                                                                  bodyLoc,
+                                                                  family.prototype,
+                                                                  iterArgs.front(),
+                                                                  anchor,
+                                                                  laneArg,
+                                                                  baseHostOffset,
+                                                                  sourceRunStartDelta,
+                                                                  hostRunStartDelta);
+      if (failed(copied))
+        return failure();
+      yielded.push_back(*copied);
+      return success();
+    });
+  if (failed(outerLoop))
+    return failure();
+  return outerLoop->results.front();
+}
+
+FailureOr<mlir::Value> emitFragmentAssemblyCopyRuns(IRRewriter& rewriter,
+                                                    Location loc,
+                                                    ArrayRef<FragmentAssemblyCopyRun> runs,
+                                                    mlir::Value hostTarget,
+                                                    Operation* anchor,
+                                                    std::optional<mlir::Value> laneArg,
+                                                    mlir::Value baseHostOffset) {
+  for (const FragmentAssemblyCopyRunFamily& family : groupFragmentAssemblyCopyRunFamilies(runs)) {
+    FailureOr<mlir::Value> updatedHostTarget =
+      emitFragmentAssemblyCopyRunFamily(rewriter, loc, family, hostTarget, anchor, laneArg, baseHostOffset);
+    if (failed(updatedHostTarget))
+      return failure();
+    hostTarget = *updatedHostTarget;
+  }
+
+  return hostTarget;
+}
+
 } // namespace onnx_mlir

src/PIM/Conversion/SpatialToPim/Common.hpp

@@ -1,10 +1,23 @@
 #pragma once
 
+#include <optional>
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/STLFunctionalExtras.h"
+
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/Value.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Support/LogicalResult.h"
 
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 
+namespace onnx_mlir::spatial {
+class SpatBlueprintOp;
+}
+
 namespace onnx_mlir {
 
 mlir::FailureOr<mlir::IntegerAttr>
@@ -29,6 +42,66 @@ mlir::SmallVector<mlir::Value> getOpOperandsSortedByUses(mlir::Operation* operat
 
 mlir::Value getBestOutputTensorFromOperandsOrAllocate(mlir::RewriterBase& rewriter, mlir::Operation* operation);
 
+mlir::LogicalResult validateFragmentAssemblyMetadata(onnx_mlir::spatial::SpatBlueprintOp blueprint,
+                                                     int64_t resultRank,
+                                                     size_t operandCount,
+                                                     llvm::ArrayRef<int64_t> operandIndices,
+                                                     llvm::ArrayRef<int64_t> sourceOffsets,
+                                                     llvm::ArrayRef<int64_t> flatOffsets,
+                                                     llvm::ArrayRef<int64_t> flatSizes,
+                                                     llvm::ArrayRef<int64_t> flatStrides);
+
+mlir::FailureOr<mlir::SmallVector<int64_t, 4>>
+getStaticSliceOffsetsForElementOffset(mlir::Operation* anchor,
+                                      mlir::ShapedType sourceType,
+                                      llvm::ArrayRef<int64_t> fragmentShape,
+                                      int64_t sourceElementOffset,
+                                      llvm::StringRef fieldName);
+
+mlir::LogicalResult
+forEachContiguousDestinationChunk(llvm::ArrayRef<int64_t> destShape,
+                                  llvm::ArrayRef<int64_t> baseOffsets,
+                                  llvm::ArrayRef<int64_t> sizes,
+                                  llvm::function_ref<mlir::LogicalResult(llvm::ArrayRef<int64_t>, int64_t, int64_t)>
+                                    callback);
+
+int64_t getFragmentAssemblySourceElementOffset(mlir::RankedTensorType sourceType,
+                                               int64_t sourceSlot,
+                                               int64_t sourceOffset);
+
+struct FragmentAssemblyCopy {
+  mlir::Value source;
+  mlir::RankedTensorType sourceType;
+  unsigned hostTargetIndex = 0;
+  int64_t lane = 0;
+  int64_t sourceByteOffset = 0;
+  int64_t hostByteOffset = 0;
+  int64_t byteSize = 0;
+};
+
+struct FragmentAssemblyCopyRun {
+  mlir::Value source;
+  mlir::RankedTensorType sourceType;
+  unsigned hostTargetIndex = 0;
+  int64_t count = 0;
+  int64_t sourceStepBytes = 0;
+  int64_t hostStepBytes = 0;
+  int64_t byteSize = 0;
+  mlir::SmallVector<int64_t, 8> sourceStartBytesByLane;
+  mlir::SmallVector<int64_t, 8> hostStartBytesByLane;
+};
+
+mlir::FailureOr<mlir::SmallVector<FragmentAssemblyCopyRun, 8>>
+groupFragmentAssemblyCopyRuns(llvm::ArrayRef<FragmentAssemblyCopy> copies, uint32_t laneCount = 1);
+
+mlir::FailureOr<mlir::Value> emitFragmentAssemblyCopyRuns(mlir::IRRewriter& rewriter,
+                                                          mlir::Location loc,
+                                                          llvm::ArrayRef<FragmentAssemblyCopyRun> runs,
+                                                          mlir::Value hostTarget,
+                                                          mlir::Operation* anchor,
+                                                          std::optional<mlir::Value> laneArg = std::nullopt,
+                                                          mlir::Value baseHostOffset = {});
+
 inline mlir::tensor::EmptyOp
 createEmptyTensorFromShaped(mlir::IRRewriter& rewriter, mlir::Location loc, mlir::ShapedType shapedType) {
   return mlir::tensor::EmptyOp::create(rewriter, loc, shapedType.getShape(), shapedType.getElementType());

src/PIM/Conversion/SpatialToPim/ComputeLikeRegionUtils.cpp

@@ -17,10 +17,10 @@ std::optional<unsigned> getDirectComputeLikeInputIndex(Operation* owner, unsigne
     return operandNumber - inputBegin;
   };
 
-  if (auto compute = dyn_cast<spatial::SpatCompute>(owner))
+  if (auto compute = dyn_cast<spatial::SpatScheduledCompute>(owner))
     return getInputIndex(owner, compute.getInputs().size());
 
-  if (auto computeBatch = dyn_cast<spatial::SpatComputeBatch>(owner))
+  if (auto computeBatch = dyn_cast<spatial::SpatScheduledComputeBatch>(owner))
     return getInputIndex(owner, computeBatch.getInputs().size());
 
   return std::nullopt;
@@ -32,13 +32,13 @@ void replaceAndEraseDirectComputeLikeInput(PatternRewriter& rewriter,
                                            Value replacement) {
   Block& body = owner->getRegion(0).front();
   BlockArgument bodyArgument;
-  if (auto compute = dyn_cast<spatial::SpatCompute>(owner)) {
+  if (auto compute = dyn_cast<spatial::SpatScheduledCompute>(owner)) {
     auto computeArg = compute.getInputArgument(inputIndex);
     assert(computeArg && "expected compute input block argument");
     bodyArgument = *computeArg;
   }
   else {
-    auto batchArg = cast<spatial::SpatComputeBatch>(owner).getInputArgument(inputIndex);
+    auto batchArg = cast<spatial::SpatScheduledComputeBatch>(owner).getInputArgument(inputIndex);
     assert(batchArg && "expected compute_batch input block argument");
     bodyArgument = *batchArg;
   }
@@ -46,10 +46,10 @@ void replaceAndEraseDirectComputeLikeInput(PatternRewriter& rewriter,
 
   rewriter.startOpModification(owner);
   bodyArgument.replaceAllUsesWith(replacement);
-  if (auto compute = dyn_cast<spatial::SpatCompute>(owner))
+  if (auto compute = dyn_cast<spatial::SpatScheduledCompute>(owner))
     compute.getInputsMutable().erase(inputIndex);
   else
-    cast<spatial::SpatComputeBatch>(owner).getInputsMutable().erase(inputIndex);
+    cast<spatial::SpatScheduledComputeBatch>(owner).getInputsMutable().erase(inputIndex);
   body.eraseArgument(bodyArgIndex);
   rewriter.finalizeOpModification(owner);
 }

src/PIM/Conversion/SpatialToPim/CoreLoweringPatterns.cpp

@@ -1,6 +1,7 @@
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Tosa/IR/TosaOps.h"
 #include "mlir/IR/IRMapping.h"
@@ -8,6 +9,10 @@
 
 #include "Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "Conversion/SpatialToPim/SpatialToPimPass.hpp"
+#include "src/Accelerators/PIM/Common/IR/BatchCoreUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ConstantUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
+#include "src/Accelerators/PIM/Common/IR/ShapingUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Common.hpp"
@@ -30,6 +35,95 @@ static bool isChannelUseChainOp(Operation* op) {
              pim::PimTransposeOp>(op);
 }
 
+static FailureOr<Value> lowerFragmentAssemblyBlueprint(IRRewriter& rewriter,
+                                                       spatial::SpatBlueprintOp blueprint,
+                                                       IRMapping& mapping) {
+  auto resultType = dyn_cast<ShapedType>(blueprint.getOutput().getType());
+  if (!resultType || !resultType.hasStaticShape())
+    return blueprint.emitOpError("fragment assembly lowering requires a static ranked tensor result");
+
+  std::optional<StringRef> modeAttr = blueprint.getMode();
+  std::optional<ArrayRef<int64_t>> operandIndicesAttr = blueprint.getFragmentOperandIndices();
+  std::optional<ArrayRef<int64_t>> sourceSlotsAttr = blueprint.getFragmentSourceSlots();
+  std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = blueprint.getFragmentSourceOffsets();
+  std::optional<ArrayRef<int64_t>> fragmentStridesAttr = blueprint.getFragmentStrides();
+  if (!modeAttr || *modeAttr != "fragment_assembly" || !operandIndicesAttr || !sourceSlotsAttr
+      || !sourceOffsetsAttr || !fragmentStridesAttr)
+    return blueprint.emitOpError("fragment assembly lowering requires explicit fragment metadata");
+
+  ArrayRef<int64_t> operandIndices = *operandIndicesAttr;
+  ArrayRef<int64_t> sourceSlots = *sourceSlotsAttr;
+  ArrayRef<int64_t> sourceOffsets = *sourceOffsetsAttr;
+  ArrayRef<int64_t> flatOffsets = blueprint.getFragmentOffsets();
+  ArrayRef<int64_t> flatSizes = blueprint.getFragmentSizes();
+  ArrayRef<int64_t> flatStrides = *fragmentStridesAttr;
+  int64_t rank = resultType.getRank();
+
+  SmallVector<Value> fragmentOperands {blueprint.getInput()};
+  llvm::append_range(fragmentOperands, blueprint.getFragments());
+  if (failed(validateFragmentAssemblyMetadata(blueprint,
+                                              rank,
+                                              fragmentOperands.size(),
+                                              operandIndices,
+                                              sourceOffsets,
+                                              flatOffsets,
+                                              flatSizes,
+                                              flatStrides)))
+    return failure();
+
+  SmallVector<int64_t> hostStrides = computeRowMajorStrides(resultType.getShape());
+  SmallVector<FragmentAssemblyCopy, 8> copies;
+  for (int64_t fragmentIndex = 0; fragmentIndex < static_cast<int64_t>(operandIndices.size()); ++fragmentIndex) {
+    int64_t operandIndex = operandIndices[fragmentIndex];
+
+    SmallVector<int64_t, 4> fragmentOffsets;
+    SmallVector<int64_t, 4> fragmentSizes;
+    for (int64_t dim = 0; dim < rank; ++dim) {
+      int64_t flatIndex = fragmentIndex * rank + dim;
+      if (flatStrides[flatIndex] != 1)
+        return blueprint.emitOpError("fragment assembly lowering only supports unit strides");
+      fragmentOffsets.push_back(flatOffsets[flatIndex]);
+      fragmentSizes.push_back(flatSizes[flatIndex]);
+    }
+
+    Value source = mapping.lookupOrDefault(fragmentOperands[operandIndex]);
+    auto sourceType = dyn_cast<RankedTensorType>(source.getType());
+    if (!sourceType || !sourceType.hasStaticShape())
+      return blueprint.emitOpError("fragment assembly lowering requires static ranked tensor operands");
+    size_t elementSize = getElementTypeSizeInBytes(sourceType.getElementType());
+    if (failed(forEachContiguousDestinationChunk(
+          resultType.getShape(),
+          fragmentOffsets,
+          fragmentSizes,
+          [&](ArrayRef<int64_t> chunkOffsets, int64_t relativeSourceOffset, int64_t chunkElements) -> LogicalResult {
+            int64_t hostElementOffset = 0;
+            for (auto [dim, offset] : llvm::enumerate(chunkOffsets))
+              hostElementOffset += offset * hostStrides[dim];
+
+            FragmentAssemblyCopy copy;
+            copy.source = source;
+            copy.sourceType = sourceType;
+            copy.sourceByteOffset =
+              (getFragmentAssemblySourceElementOffset(
+                 sourceType, sourceSlots[fragmentIndex], sourceOffsets[fragmentIndex])
+               + relativeSourceOffset)
+              * static_cast<int64_t>(elementSize);
+            copy.hostByteOffset = hostElementOffset * static_cast<int64_t>(elementSize);
+            copy.byteSize = chunkElements * static_cast<int64_t>(elementSize);
+            copies.push_back(copy);
+            return success();
+          })))
+      return failure();
+  }
+
+  Value currentOutput = createEmptyTensorFromShaped(rewriter, blueprint.getLoc(), resultType);
+  FailureOr<SmallVector<FragmentAssemblyCopyRun, 8>> runs = groupFragmentAssemblyCopyRuns(copies);
+  if (failed(runs))
+    return failure();
+  return emitFragmentAssemblyCopyRuns(
+    rewriter, blueprint.getLoc(), *runs, currentOutput, blueprint.getOperation());
+}
+
 static void
 cloneMappedHelperOperands(Operation* op, IRMapping& mapping, IRRewriter& rewriter, OperationFolder& constantFolder) {
   for (Value operand : op->getOperands()) {
@@ -55,18 +149,7 @@ cloneMappedHelperOperands(Operation* op, IRMapping& mapping, IRRewriter& rewrite
   }
 }
 
-static FailureOr<int32_t> getPimCoreIdForComputeOp(spatial::SpatCompute computeOp, size_t& fallbackCoreId) {
+static LogicalResult collectHelperComputeChain(spatial::SpatScheduledCompute computeOp,
-  if (auto spatialCoreIdAttr = computeOp->getAttrOfType<IntegerAttr>(onnx_mlir::kCoreIdAttrName))
-    return pim::checkedI32(spatialCoreIdAttr.getInt(), computeOp, "spatial compute core id");
-  auto checkedCoreId =
-    pim::checkedI32(static_cast<uint64_t>(fallbackCoreId), computeOp, "fallback spatial compute core id");
-  if (failed(checkedCoreId))
-    return failure();
-  ++fallbackCoreId;
-  return *checkedCoreId;
-}
-
-static LogicalResult collectHelperComputeChain(spatial::SpatCompute computeOp,
                                                SmallVectorImpl<Operation*>& helperChain,
                                                bool requireReturnUse = true) {
   if (computeOp.getInputs().size() != 1 || computeOp.getNumResults() != 1)
@@ -104,16 +187,79 @@ static LogicalResult collectHelperComputeChain(spatial::SpatCompute computeOp,
   return success();
 }
 
-static bool inlineInputlessHelperComputeForWeightLikeUsers(spatial::SpatCompute computeOp,
+static bool isHostMaterializableHelperOp(Operation* op) {
+  if (isa<spatial::SpatYieldOp>(op))
+    return true;
+  if (isa<arith::ConstantOp>(op) || op->hasTrait<OpTrait::ConstantLike>())
+    return true;
+  if (auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(op)) {
+    std::optional<StringRef> mode = blueprint.getMode();
+    return mode && *mode == "fragment_assembly";
+  }
+  return isShapingOnlyOp(op) || isPureIndexComputationOp(op);
+}
+
+static FailureOr<DenseMap<Value, Attribute>>
+analyzeHostMaterializableHelper(spatial::SpatScheduledCompute computeOp) {
+  DenseMap<Value, Attribute> folded;
+  for (auto [weightIndex, weight] : llvm::enumerate(computeOp.getWeights())) {
+    auto argument = computeOp.getWeightArgument(weightIndex);
+    if (!argument)
+      return failure();
+    Attribute constant;
+    if (matchPattern(weight, m_Constant(&constant)))
+      folded[*argument] = constant;
+  }
+  Block& block = computeOp.getBody().front();
+  for (Operation& op : block) {
+    if (!isHostMaterializableHelperOp(&op))
+      return failure();
+    if (isa<spatial::SpatYieldOp, spatial::SpatBlueprintOp>(op)
+        || (isShapingOnlyOp(&op) && !isPureIndexComputationOp(&op)))
+      continue;
+    if (isa<arith::ConstantOp>(op) || op.hasTrait<OpTrait::ConstantLike>()) {
+      for (Value result : op.getResults()) {
+        Attribute constant;
+        if (!matchPattern(result, m_Constant(&constant)))
+          return failure();
+        folded[result] = constant;
+      }
+      continue;
+    }
+    if (!isPureIndexComputationOp(&op) || op.getNumRegions() != 0)
+      return failure();
+    SmallVector<Attribute> operands;
+    for (Value operand : op.getOperands()) {
+      auto it = folded.find(operand);
+      if (it == folded.end())
+        return failure();
+      operands.push_back(it->second);
+    }
+    SmallVector<OpFoldResult> results;
+    if (failed(op.fold(operands, results))
+        || results.size() != op.getNumResults())
+      return failure();
+    for (auto [result, foldResult] : llvm::zip(op.getResults(), results)) {
+      auto attribute = dyn_cast<Attribute>(foldResult);
+      if (!attribute)
+        return failure();
+      folded[result] = attribute;
+    }
+  }
+  return folded;
+}
+
+static bool inlineInputlessHelperComputeForWeightLikeUsers(spatial::SpatScheduledCompute computeOp,
                                                            IRRewriter& rewriter,
                                                            OperationFolder& constantFolder) {
   if (!computeOp.getInputs().empty() || computeOp.getNumResults() != 1)
     return false;
+  if (computeOp.getResult(0).use_empty())
+    return false;
   if (!llvm::all_of(computeOp.getResult(0).getUsers(), [](Operation* user) {
-        return isa<spatial::SpatCompute, spatial::SpatComputeBatch, pim::PimCoreOp, pim::PimCoreBatchOp>(user);
+        return isa<spatial::SpatScheduledCompute, spatial::SpatScheduledComputeBatch, pim::PimCoreOp, pim::PimCoreBatchOp>(user);
       }))
     return false;
-
   Block& block = computeOp.getBody().front();
   if (block.getNumArguments() != computeOp.getWeights().size())
     return false;
@@ -121,6 +267,9 @@ static bool inlineInputlessHelperComputeForWeightLikeUsers(spatial::SpatCompute
   auto yieldOp = dyn_cast<spatial::SpatYieldOp>(block.getTerminator());
   if (!yieldOp || yieldOp.getNumOperands() != 1)
     return false;
+  auto folded = analyzeHostMaterializableHelper(computeOp);
+  if (failed(folded))
+    return false;
 
   rewriter.setInsertionPoint(computeOp);
   IRMapping mapping;
@@ -131,6 +280,31 @@ static bool inlineInputlessHelperComputeForWeightLikeUsers(spatial::SpatCompute
     mapping.map(*weightArg, weight);
   }
   for (Operation& op : block.without_terminator()) {
+    if (auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(op)) {
+      std::optional<StringRef> modeAttr = blueprint.getMode();
+      if (modeAttr && *modeAttr == "fragment_assembly") {
+        auto lowered = lowerFragmentAssemblyBlueprint(rewriter, blueprint, mapping);
+        if (failed(lowered))
+          return false;
+        mapping.map(blueprint.getOutput(), *lowered);
+        continue;
+      }
+    }
+
+    if (isa<arith::ConstantOp>(op) || op.hasTrait<OpTrait::ConstantLike>()
+        || isPureIndexComputationOp(&op)) {
+      for (Value result : op.getResults()) {
+        auto it = folded->find(result);
+        if (it == folded->end())
+          return false;
+        mapping.map(
+          result,
+          getOrCreateConstant(constantFolder, computeOp, it->second,
+                              result.getType()));
+      }
+      continue;
+    }
+
     cloneMappedHelperOperands(&op, mapping, rewriter, constantFolder);
     Operation* clonedOp = rewriter.clone(op, mapping);
     for (auto [originalResult, newResult] : llvm::zip(op.getResults(), clonedOp->getResults()))
@@ -145,7 +319,7 @@ static bool inlineInputlessHelperComputeForWeightLikeUsers(spatial::SpatCompute
 
 } // namespace
 
-LogicalResult raptor::SpatialToPimPass::lowerComputeOp(spatial::SpatCompute computeOp,
+LogicalResult raptor::SpatialToPimPass::lowerComputeOp(spatial::SpatScheduledCompute computeOp,
                                                        IRRewriter& rewriter,
                                                        OperationFolder& constantFolder) {
   Location loc = computeOp->getLoc();
@@ -214,7 +388,7 @@ LogicalResult raptor::SpatialToPimPass::lowerComputeOp(spatial::SpatCompute comp
   if (!computeOp.getWeights().empty())
     computeWeights.append(computeOp.getWeights().begin(), computeOp.getWeights().end());
   rewriter.setInsertionPointAfter(computeOp);
-  auto checkedCoreId = getPimCoreIdForComputeOp(computeOp, coreId);
+  auto checkedCoreId = getRequiredScheduledCoreId(computeOp, "spatial compute core id");
   if (failed(checkedCoreId))
     return failure();
   auto coreIdAttr = pim::getCheckedI32Attr(rewriter, computeOp, static_cast<int64_t>(*checkedCoreId), "pim core id");

src/PIM/Conversion/SpatialToPim/Patterns.cpp

@@ -1,6 +1,10 @@
+#include "mlir/Transforms/DialectConversion.h"
+#include "src/Accelerators/PIM/Common/PimCommon.hpp"
+#include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Common.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Patterns.hpp"
 #include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
+#include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 
 using namespace mlir;
 
@@ -11,6 +15,92 @@ namespace raptor {
 
 } // namespace raptor
 
+struct LowerFragmentAssemblyBlueprintPattern
+  : OpConversionPattern<spatial::SpatBlueprintOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult matchAndRewrite(spatial::SpatBlueprintOp op,
+                                OpAdaptor adaptor,
+                                ConversionPatternRewriter& rewriter) const override {
+    std::optional<StringRef> modeAttr = op.getMode();
+    if (!modeAttr || *modeAttr != "fragment_assembly")
+      return failure();
+
+    auto resultType = dyn_cast<ShapedType>(op.getOutput().getType());
+    if (!resultType || !resultType.hasStaticShape())
+      return op.emitOpError("fragment assembly lowering requires a static ranked tensor result");
+
+    std::optional<ArrayRef<int64_t>> operandIndicesAttr = op.getFragmentOperandIndices();
+    std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = op.getFragmentSourceOffsets();
+    std::optional<ArrayRef<int64_t>> fragmentStridesAttr = op.getFragmentStrides();
+    if (!operandIndicesAttr || !sourceOffsetsAttr || !fragmentStridesAttr)
+      return op.emitOpError("fragment assembly lowering requires explicit fragment metadata");
+
+    ArrayRef<int64_t> operandIndices = *operandIndicesAttr;
+    ArrayRef<int64_t> sourceOffsets = *sourceOffsetsAttr;
+    ArrayRef<int64_t> flatOffsets = op.getFragmentOffsets();
+    ArrayRef<int64_t> flatSizes = op.getFragmentSizes();
+    ArrayRef<int64_t> flatStrides = *fragmentStridesAttr;
+    int64_t rank = resultType.getRank();
+
+    SmallVector<Value> fragmentOperands {adaptor.getInput()};
+    llvm::append_range(fragmentOperands, adaptor.getFragments());
+    if (failed(validateFragmentAssemblyMetadata(
+          op, rank, fragmentOperands.size(), operandIndices, sourceOffsets, flatOffsets, flatSizes, flatStrides)))
+      return failure();
+
+    Value currentOutput =
+      tensor::EmptyOp::create(rewriter, op.getLoc(), resultType.getShape(), resultType.getElementType()).getResult();
+    for (int64_t fragmentIndex = 0; fragmentIndex < static_cast<int64_t>(operandIndices.size()); ++fragmentIndex) {
+      int64_t operandIndex = operandIndices[fragmentIndex];
+
+      SmallVector<int64_t, 4> fragmentOffsets;
+      for (int64_t dim = 0; dim < rank; ++dim) {
+        int64_t flatIndex = fragmentIndex * rank + dim;
+        if (flatStrides[flatIndex] != 1)
+          return op.emitOpError("fragment assembly lowering only supports unit strides");
+        fragmentOffsets.push_back(flatOffsets[flatIndex]);
+      }
+
+      Value source = fragmentOperands[operandIndex];
+      auto sourceType = dyn_cast<RankedTensorType>(source.getType());
+      if (!sourceType || !sourceType.hasStaticShape())
+        return op.emitOpError("fragment assembly lowering requires static ranked tensor operands");
+
+      SmallVector<int64_t, 4> fragmentShape;
+      fragmentShape.reserve(rank);
+      for (int64_t dim = 0; dim < rank; ++dim)
+        fragmentShape.push_back(flatSizes[fragmentIndex * rank + dim]);
+
+      Value fragment = source;
+      if (llvm::to_vector(sourceType.getShape()) != fragmentShape || sourceOffsets[fragmentIndex] != 0) {
+        FailureOr<SmallVector<int64_t, 4>> extractOffsets = getStaticSliceOffsetsForElementOffset(
+          op, sourceType, fragmentShape, sourceOffsets[fragmentIndex], "fragment assembly source slice");
+        if (failed(extractOffsets))
+          return failure();
+        fragment = tensor::ExtractSliceOp::create(rewriter,
+                                                  op.getLoc(),
+                                                  source,
+                                                  getStaticIndexAttrs(rewriter, *extractOffsets),
+                                                  getStaticIndexAttrs(rewriter, fragmentShape),
+                                                  getUnitStrides(rewriter, rank));
+      }
+
+      currentOutput = tensor::InsertSliceOp::create(rewriter,
+                                                    op.getLoc(),
+                                                    fragment,
+                                                    currentOutput,
+                                                    getStaticIndexAttrs(rewriter, fragmentOffsets),
+                                                    getStaticIndexAttrs(rewriter, fragmentShape),
+                                                    getUnitStrides(rewriter, rank))
+                        .getResult();
+    }
+
+    rewriter.replaceOp(op, currentOutput);
+    return success();
+  }
+};
+
 void populateInitialPatterns(RewritePatternSet& patterns) {
   raptor::populateWithGenerated(patterns);
   populateTransposeLoweringPatterns(patterns);
@@ -19,6 +109,7 @@ void populateInitialPatterns(RewritePatternSet& patterns) {
 void populateCoreBodyPatterns(RewritePatternSet& patterns) {
   raptor::populateWithGenerated(patterns);
   populateTransposeLoweringPatterns(patterns);
+  patterns.add<LowerFragmentAssemblyBlueprintPattern>(patterns.getContext());
 }
 
 } // namespace onnx_mlir

src/PIM/Conversion/SpatialToPim/Patterns/ChannelLowering.cpp

@@ -1,7 +1,8 @@
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
-
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Common.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Patterns.hpp"
+#include "src/Accelerators/PIM/Common/IR/ShapeUtils.hpp"
 #include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
 #include "src/Accelerators/PIM/Dialect/Spatial/SpatialOps.hpp"
 
@@ -10,6 +11,38 @@ using namespace mlir;
 namespace onnx_mlir {
 namespace {
 
+static void copyRaptorDebugAttrs(Operation* source, Operation* target) {
+  for (NamedAttribute attr : source->getAttrs()) {
+    StringRef name = attr.getName().strref();
+    if (name.starts_with("raptor."))
+      target->setAttr(attr.getName(), attr.getValue());
+  }
+}
+
+static Value createDestinationByteOffset(PatternRewriter& rewriter,
+                                         tensor::InsertSliceOp insert) {
+  auto destinationType = cast<RankedTensorType>(insert.getDestType());
+  SmallVector<int64_t> strides = computeRowMajorStrides(destinationType.getShape());
+  int64_t elementBytes = getElementTypeSizeInBytes(destinationType.getElementType());
+  Value total = arith::ConstantIndexOp::create(rewriter, insert.getLoc(), 0);
+  for (auto [dimension, offset] : llvm::enumerate(insert.getMixedOffsets())) {
+    int64_t scale = strides[dimension] * elementBytes;
+    Value component;
+    if (auto attribute = dyn_cast<Attribute>(offset)) {
+      component = arith::ConstantIndexOp::create(
+        rewriter, insert.getLoc(), cast<IntegerAttr>(attribute).getInt() * scale);
+    } else {
+      component = cast<Value>(offset);
+      if (scale != 1)
+        component = arith::MulIOp::create(
+          rewriter, insert.getLoc(), component,
+          arith::ConstantIndexOp::create(rewriter, insert.getLoc(), scale));
+    }
+    total = arith::AddIOp::create(rewriter, insert.getLoc(), total, component);
+  }
+  return total;
+}
+
 struct ChannelSendLowering : OpRewritePattern<spatial::SpatChannelSendOp> {
   using OpRewritePattern::OpRewritePattern;
 
@@ -17,7 +50,8 @@ struct ChannelSendLowering : OpRewritePattern<spatial::SpatChannelSendOp> {
     auto sizeAttr = getTensorSizeInBytesAttr(rewriter, op.getOperation(), op.getInput());
     if (failed(sizeAttr))
       return failure();
-    pim::PimSendOp::create(rewriter, op.getLoc(), op.getInput(), *sizeAttr, op.getTargetCoreId());
+    auto send = pim::PimSendOp::create(rewriter, op.getLoc(), op.getInput(), *sizeAttr, op.getTargetCoreId());
+    copyRaptorDebugAttrs(op.getOperation(), send.getOperation());
     rewriter.eraseOp(op);
     return success();
   }
@@ -31,18 +65,45 @@ struct ChannelReceiveLowering : OpRewritePattern<spatial::SpatChannelReceiveOp>
       rewriter.eraseOp(op);
       return success();
     }
-    auto outputType = cast<ShapedType>(op.getResult().getType());
+    auto outputType = cast<RankedTensorType>(op.getResult().getType());
+    tensor::InsertSliceOp destinationInsert;
+    if (op->hasOneUse()) {
+      auto insert = dyn_cast<tensor::InsertSliceOp>(*op->getUsers().begin());
+      auto destinationType = insert
+        ? dyn_cast<RankedTensorType>(insert.getDestType()) : RankedTensorType();
+      if (insert && insert.getSource() == op.getOutput()
+          && insert.getSourceType() == outputType
+          && insert->getBlock() == op->getBlock() && destinationType
+          && destinationType.hasStaticShape()
+          && isContiguousSubviewWithDynamicOffsets(
+            destinationType.getShape(), insert.getMixedOffsets(),
+            insert.getStaticSizes(), insert.getStaticStrides()))
+        destinationInsert = insert;
+    }
     Value outputBuffer =
       tensor::EmptyOp::create(rewriter, op.getLoc(), outputType.getShape(), outputType.getElementType()).getResult();
     auto sizeAttr = getTensorSizeInBytesAttr(rewriter, op.getOperation(), op.getResult());
     if (failed(sizeAttr))
       return failure();
-    Value received = pim::PimReceiveOp::create(
+    auto receive = pim::PimReceiveOp::create(
-                       rewriter, op.getLoc(), op.getResult().getType(), outputBuffer, *sizeAttr, op.getSourceCoreId())
+      rewriter, op.getLoc(), op.getResult().getType(), outputBuffer, *sizeAttr, op.getSourceCoreId());
-                       .getOutput();
+    copyRaptorDebugAttrs(op.getOperation(), receive.getOperation());
+    Value received = receive.getOutput();
+    if (!destinationInsert) {
       rewriter.replaceOp(op, received);
       return success();
     }
+
+    rewriter.setInsertionPoint(destinationInsert);
+    Value targetOffset = createDestinationByteOffset(rewriter, destinationInsert);
+    Value zero = arith::ConstantIndexOp::create(rewriter, op.getLoc(), 0);
+    auto copy = pim::PimMemCopyOp::create(
+      rewriter, op.getLoc(), destinationInsert.getDestType(), targetOffset, zero,
+      destinationInsert.getDest(), received, *sizeAttr);
+    rewriter.replaceOp(destinationInsert, copy.getOutput());
+    rewriter.eraseOp(op);
+    return success();
+  }
 };
 
 struct ExtractRowsLowering : OpRewritePattern<spatial::SpatExtractRowsOp> {

src/PIM/Conversion/SpatialToPim/Patterns/GlobalTensorMaterialization.cpp

@@ -59,7 +59,7 @@ struct MoveExtractSliceIntoCompute final : OpRewritePattern<mlir::tensor::Extrac
       return failure();
 
     for (auto& uses : extractSliceOp->getUses()) {
-      if (isa<spatial::SpatCompute>(uses.getOwner())) {
+      if (isa<spatial::SpatScheduledCompute>(uses.getOwner())) {
         if (!getDirectComputeLikeInputIndex(uses.getOwner(), uses.getOperandNumber()))
           return failure();
       }
@@ -72,7 +72,7 @@ struct MoveExtractSliceIntoCompute final : OpRewritePattern<mlir::tensor::Extrac
 
     for (auto& uses : llvm::make_early_inc_range(extractSliceOp->getUses())) {
 
-      if (auto spatCompute = dyn_cast<spatial::SpatCompute>(uses.getOwner())) {
+      if (auto spatCompute = dyn_cast<spatial::SpatScheduledCompute>(uses.getOwner())) {
         auto inputIndex = getDirectComputeLikeInputIndex(spatCompute, uses.getOperandNumber());
         if (!inputIndex)
           return failure();
@@ -92,7 +92,7 @@ struct MoveExtractSliceIntoCompute final : OpRewritePattern<mlir::tensor::Extrac
         replaceAndEraseDirectComputeLikeInput(
           rewriter, spatCompute.getOperation(), *inputIndex, mapSpatToExtract[spatCompute.getOperation()]);
       }
-      else if (auto spatComputeBatch = dyn_cast<spatial::SpatComputeBatch>(uses.getOwner())) {
+      else if (auto spatComputeBatch = dyn_cast<spatial::SpatScheduledComputeBatch>(uses.getOwner())) {
         auto inputIndex = getDirectComputeLikeInputIndex(spatComputeBatch, uses.getOperandNumber());
         if (!inputIndex)
           return failure();
@@ -114,7 +114,7 @@ struct MoveExtractSliceIntoCompute final : OpRewritePattern<mlir::tensor::Extrac
       }
       else {
         {
-          if (auto spatCompute = uses.getOwner()->getParentOfType<spatial::SpatCompute>()) {
+          if (auto spatCompute = uses.getOwner()->getParentOfType<spatial::SpatScheduledCompute>()) {
             rewriter.setInsertionPoint(&spatCompute.getBody().front().front());
             if (!mapSpatToExtract.contains(spatCompute.getOperation())) {
               auto newExtractSlice = rewriter.clone(*extractSliceOp.getOperation());
@@ -125,7 +125,7 @@ struct MoveExtractSliceIntoCompute final : OpRewritePattern<mlir::tensor::Extrac
             uses.set(mapSpatToExtract[spatCompute.getOperation()]);
             rewriter.finalizeOpModification(spatCompute.getOperation());
           }
-          else if (auto spatComputeBatch = uses.getOwner()->getParentOfType<spatial::SpatComputeBatch>()) {
+          else if (auto spatComputeBatch = uses.getOwner()->getParentOfType<spatial::SpatScheduledComputeBatch>()) {
             rewriter.setInsertionPoint(&spatComputeBatch.getBody().front().front());
             if (!mapSpatToExtract.contains(spatComputeBatch.getOperation())) {
               auto newExtractSlice = rewriter.clone(*extractSliceOp.getOperation());
@@ -179,7 +179,7 @@ struct FuncOpArgToGlobalMemoryPattern final : OpRewritePattern<mlir::func::FuncO
 
       for (auto& argUses : llvm::make_early_inc_range(arg.getUses())) {
         auto argUser = argUses.getOwner();
-        if (auto spatCompute = dyn_cast<spatial::SpatCompute>(argUser)) {
+        if (auto spatCompute = dyn_cast<spatial::SpatScheduledCompute>(argUser)) {
           auto inputIndex = getDirectComputeLikeInputIndex(spatCompute, argUses.getOperandNumber());
           if (!inputIndex)
             return failure();
@@ -191,7 +191,7 @@ struct FuncOpArgToGlobalMemoryPattern final : OpRewritePattern<mlir::func::FuncO
 
           replaceAndEraseDirectComputeLikeInput(rewriter, spatCompute.getOperation(), BBArgIndex, toTensor);
         }
-        else if (auto spatComputeBatch = dyn_cast<spatial::SpatComputeBatch>(argUser)) {
+        else if (auto spatComputeBatch = dyn_cast<spatial::SpatScheduledComputeBatch>(argUser)) {
           auto inputIndex = getDirectComputeLikeInputIndex(spatComputeBatch, argUses.getOperandNumber());
           if (!inputIndex)
             return failure();

src/PIM/Conversion/SpatialToPim/ReturnPathNormalization.cpp

@@ -2,6 +2,7 @@
 #include "mlir/Dialect/Bufferization/IR/Bufferization.h"
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Tosa/IR/TosaOps.h"
 #include "mlir/IR/BuiltinOps.h"
@@ -11,6 +12,7 @@
 
 #include "Conversion/ONNXToSpatial/Common/Common.hpp"
 #include "Conversion/SpatialToPim/SpatialToPimPass.hpp"
+#include "src/Accelerators/PIM/Common/IR/LoopUtils.hpp"
 #include "src/Accelerators/PIM/Common/PimCommon.hpp"
 #include "src/Accelerators/PIM/Common/Support/CheckedArithmetic.hpp"
 #include "src/Accelerators/PIM/Conversion/SpatialToPim/Common.hpp"
@@ -86,7 +88,7 @@ getCheckedByteOffset(int64_t elementOffset, size_t elementSize, Operation* ancho
   return pim::checkedCast<int64_t>(*byteOffset, anchor, fieldName);
 }
 
-static LogicalResult collectHelperComputeChain(spatial::SpatCompute computeOp,
+static LogicalResult collectHelperComputeChain(spatial::SpatScheduledCompute computeOp,
                                                SmallVectorImpl<Operation*>& helperChain) {
   if (computeOp.getInputs().size() != 1 || computeOp.getNumResults() != 1)
     return failure();
@@ -149,6 +151,40 @@ static std::optional<ReturnUseInfo> analyzeReturnUse(Value value) {
   };
 }
 
+static FailureOr<SmallVector<std::pair<spatial::SpatBlueprintOp, size_t>, 4>>
+analyzeTopLevelFragmentAssemblyUses(Value value) {
+  SmallVector<std::pair<spatial::SpatBlueprintOp, size_t>, 4> uses;
+  for (OpOperand& use : value.getUses()) {
+    auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(use.getOwner());
+    if (!blueprint || blueprint->getParentOp() != blueprint->getParentOfType<func::FuncOp>())
+      return failure();
+    std::optional<StringRef> mode = blueprint.getMode();
+    if (!mode || *mode != "fragment_assembly")
+      return failure();
+    if (!blueprint.getOutput().hasOneUse() || !isa<func::ReturnOp>(*blueprint.getOutput().getUsers().begin()))
+      return failure();
+    std::optional<ArrayRef<int64_t>> operandIndicesAttr = blueprint.getFragmentOperandIndices();
+    std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = blueprint.getFragmentSourceOffsets();
+    std::optional<ArrayRef<int64_t>> stridesAttr = blueprint.getFragmentStrides();
+    auto resultType = dyn_cast<RankedTensorType>(blueprint.getOutput().getType());
+    if (!operandIndicesAttr || !sourceOffsetsAttr || !stridesAttr || !resultType || !resultType.hasStaticShape())
+      return failure();
+    SmallVector<Value> fragmentOperands {blueprint.getInput()};
+    llvm::append_range(fragmentOperands, blueprint.getFragments());
+    if (failed(validateFragmentAssemblyMetadata(blueprint,
+                                                resultType.getRank(),
+                                                fragmentOperands.size(),
+                                                *operandIndicesAttr,
+                                                *sourceOffsetsAttr,
+                                                blueprint.getFragmentOffsets(),
+                                                blueprint.getFragmentSizes(),
+                                                *stridesAttr)))
+      return failure();
+    uses.emplace_back(blueprint, use.getOperandNumber());
+  }
+  return uses;
+}
+
 static std::optional<ConcatReturnUseInfo> analyzeConcatReturnUse(Value value) {
   auto getConcatResult = [](Operation* op) -> Value {
     if (auto tensorConcat = dyn_cast<tensor::ConcatOp>(op))
@@ -212,7 +248,7 @@ static std::optional<ConcatReturnUseInfo> analyzeConcatReturnUse(Value value) {
   }
 
   SmallVector<Operation*> helperChain;
-  if (auto helperCompute = dyn_cast<spatial::SpatCompute>(currentUser)) {
+  if (auto helperCompute = dyn_cast<spatial::SpatScheduledCompute>(currentUser)) {
     if (helperCompute.getInputs().size() != 1 || helperCompute.getInputs().front() != currentValue)
       return std::nullopt;
 
@@ -559,6 +595,119 @@ raptor::SpatialToPimPass::ReturnPathLoweringResult raptor::SpatialToPimPass::low
     }
   }
 
+  FailureOr<SmallVector<std::pair<spatial::SpatBlueprintOp, size_t>, 4>> fragmentAssemblyUses =
+    analyzeTopLevelFragmentAssemblyUses(producedValue);
+  if (succeeded(fragmentAssemblyUses)) {
+    auto sourceType = dyn_cast<RankedTensorType>(storedValue.getType());
+    if (!sourceType || !sourceType.hasStaticShape()) {
+      producerOp->emitOpError("fragment assembly publication requires a static ranked tensor source");
+      return ReturnPathLoweringResult::Failure;
+    }
+
+    size_t elementSize = getElementTypeSizeInBytes(sourceType.getElementType());
+    for (auto [blueprint, operandNumber] : *fragmentAssemblyUses) {
+      rewriter.setInsertionPointAfterValue(storedValue);
+      std::optional<ArrayRef<int64_t>> operandIndicesAttr = blueprint.getFragmentOperandIndices();
+      std::optional<ArrayRef<int64_t>> sourceSlotsAttr = blueprint.getFragmentSourceSlots();
+      std::optional<ArrayRef<int64_t>> sourceOffsetsAttr = blueprint.getFragmentSourceOffsets();
+      std::optional<ArrayRef<int64_t>> stridesAttr = blueprint.getFragmentStrides();
+      if (!operandIndicesAttr || !sourceSlotsAttr || !sourceOffsetsAttr || !stridesAttr) {
+        blueprint.emitOpError(
+          "fragment assembly lowering requires explicit operand, source-slot, source-offset, and stride metadata");
+        return ReturnPathLoweringResult::Failure;
+      }
+
+      size_t returnIndex = blueprint.getOutput().getUses().begin()->getOperandNumber();
+      Value outputTensor = outputTensors[returnIndex](rewriter, loc);
+      auto outputType = dyn_cast<RankedTensorType>(outputTensor.getType());
+      auto resultType = dyn_cast<RankedTensorType>(blueprint.getOutput().getType());
+      if (!outputType || !resultType || !resultType.hasStaticShape()) {
+        blueprint.emitOpError("fragment assembly lowering requires static ranked host outputs");
+        return ReturnPathLoweringResult::Failure;
+      }
+
+      ArrayRef<int64_t> operandIndices = *operandIndicesAttr;
+      ArrayRef<int64_t> sourceSlots = *sourceSlotsAttr;
+      ArrayRef<int64_t> sourceOffsets = *sourceOffsetsAttr;
+      ArrayRef<int64_t> flatOffsets = blueprint.getFragmentOffsets();
+      ArrayRef<int64_t> flatSizes = blueprint.getFragmentSizes();
+      ArrayRef<int64_t> flatStrides = *stridesAttr;
+      int64_t rank = resultType.getRank();
+      if (failed(validateFragmentAssemblyMetadata(blueprint,
+                                                  rank,
+                                                  1 + blueprint.getFragments().size(),
+                                                  operandIndices,
+                                                  sourceOffsets,
+                                                  flatOffsets,
+                                                  flatSizes,
+                                                  flatStrides)))
+        return ReturnPathLoweringResult::Failure;
+      SmallVector<FragmentAssemblyCopy, 8> copies;
+      for (int64_t fragmentIndex = 0; fragmentIndex < static_cast<int64_t>(operandIndices.size()); ++fragmentIndex) {
+        if (operandIndices[fragmentIndex] != static_cast<int64_t>(operandNumber))
+          continue;
+
+        SmallVector<int64_t, 4> fragmentOffsets;
+        SmallVector<int64_t, 4> fragmentSizes;
+        for (int64_t dim = 0; dim < rank; ++dim) {
+          int64_t flatIndex = fragmentIndex * rank + dim;
+          if (flatStrides[flatIndex] != 1) {
+            blueprint.emitOpError("fragment assembly lowering only supports unit strides");
+            return ReturnPathLoweringResult::Failure;
+          }
+          fragmentOffsets.push_back(flatOffsets[flatIndex]);
+          fragmentSizes.push_back(flatSizes[flatIndex]);
+        }
+
+        bool failedChunk = false;
+        if (failed(forEachContiguousDestinationChunk(
+          outputType.getShape(),
+          fragmentOffsets,
+          fragmentSizes,
+          [&](ArrayRef<int64_t> chunkOffsets, int64_t relativeSourceOffset, int64_t chunkElements) -> LogicalResult {
+            auto hostOffset =
+              getCheckedByteOffset(computeFlatElementIndex(chunkOffsets, outputType.getShape()),
+                                   elementSize,
+                                   producerOp,
+                                   "fragment assembly host offset");
+            int64_t sourceElementOffset = getFragmentAssemblySourceElementOffset(
+              sourceType, sourceSlots[fragmentIndex], sourceOffsets[fragmentIndex]);
+            auto sourceOffset = getCheckedByteOffset(sourceElementOffset + relativeSourceOffset,
+                                                     elementSize,
+                                                     producerOp,
+                                                     "fragment assembly source offset");
+            auto fragmentBytes =
+              getCheckedByteOffset(chunkElements, elementSize, producerOp, "fragment assembly host copy byte size");
+            if (failed(hostOffset) || failed(sourceOffset) || failed(fragmentBytes)) {
+              failedChunk = true;
+              return failure();
+            }
+            FragmentAssemblyCopy copy;
+            copy.source = storedValue;
+            copy.sourceType = sourceType;
+            copy.hostByteOffset = *hostOffset;
+            copy.sourceByteOffset = *sourceOffset;
+            copy.byteSize = *fragmentBytes;
+            copies.push_back(copy);
+            return success();
+          })))
+          failedChunk = true;
+        if (failedChunk)
+          return ReturnPathLoweringResult::Failure;
+      }
+      FailureOr<SmallVector<FragmentAssemblyCopyRun, 8>> runs = groupFragmentAssemblyCopyRuns(copies);
+      if (failed(runs))
+        return ReturnPathLoweringResult::Failure;
+      FailureOr<Value> updatedOutput =
+        emitFragmentAssemblyCopyRuns(rewriter, blueprint.getLoc(), *runs, outputTensor, producerOp);
+      if (failed(updatedOutput))
+        return ReturnPathLoweringResult::Failure;
+      outputTensor = *updatedOutput;
+      markOpToRemove(blueprint.getOperation());
+    }
+    return ReturnPathLoweringResult::Handled;
+  }
+
   if (auto concatReturnUse = analyzeConcatReturnUse(producedValue)) {
     size_t elementSize = getElementTypeSizeInBytes(storedTensorType.getElementType());
     auto storedByteSize =
@@ -643,7 +792,7 @@ raptor::SpatialToPimPass::ReturnPathLoweringResult raptor::SpatialToPimPass::low
 }
 
 raptor::SpatialToPimPass::ReturnPathLoweringResult raptor::SpatialToPimPass::lowerComputeResultReturnPath(
-  spatial::SpatCompute computeOp, OpResult result, Value yieldValue, IRRewriter& rewriter) {
+  spatial::SpatScheduledCompute computeOp, OpResult result, Value yieldValue, IRRewriter& rewriter) {
   return lowerProducedValueReturnPath(computeOp.getOperation(), result, yieldValue, rewriter);
 }
 
@@ -656,7 +805,7 @@ void raptor::SpatialToPimPass::replaceReturnWithOutputBuffers(func::ReturnOp ret
     if (!isExclusivelyOwnedByReturnChain && op->hasOneUse()) {
       Operation* onlyUser = *op->getUsers().begin();
       isExclusivelyOwnedByReturnChain =
-        isa<func::ReturnOp, tensor::ConcatOp, spatial::SpatConcatOp, pim::PimConcatOp, spatial::SpatCompute>(onlyUser)
+        isa<func::ReturnOp, tensor::ConcatOp, spatial::SpatConcatOp, pim::PimConcatOp, spatial::SpatScheduledCompute>(onlyUser)
         || isReturnHelperChainOp(onlyUser);
     }
     if (!isExclusivelyOwnedByReturnChain)
@@ -669,7 +818,17 @@ void raptor::SpatialToPimPass::replaceReturnWithOutputBuffers(func::ReturnOp ret
       return;
     }
 
-    if (auto computeOp = dyn_cast<spatial::SpatCompute>(op)) {
+    if (auto blueprint = dyn_cast<spatial::SpatBlueprintOp>(op)) {
+      std::optional<StringRef> mode = blueprint.getMode();
+      if (mode && *mode == "fragment_assembly") {
+        markOpToRemove(blueprint.getOperation());
+        for (Value operand : blueprint->getOperands())
+          markOwnedReturnChain(operand.getDefiningOp(), markOwnedReturnChain);
+        return;
+      }
+    }
+
+    if (auto computeOp = dyn_cast<spatial::SpatScheduledCompute>(op)) {
       markOpToRemove(computeOp);
       if (!computeOp.getInputs().empty())
         for (Value input : computeOp.getInputs())

src/PIM/Conversion/SpatialToPim/SpatialToPim.td

@@ -12,7 +12,7 @@ include "src/Accelerators/PIM/Dialect/Pim/Pim.td"
 def spatToPimVMM : Pat<
   (SpatVMMOp:$srcOpRes $weight, $vector),
   (PimVMMOp $weight, $vector,
-    (NativeCodeCall<"onnx_mlir::getBestOutputTensorFromOperandsOrAllocate($_builder, $0.getDefiningOp())"> $srcOpRes))
+    (NativeCodeCall<"tensor::EmptyOp::create($_builder, $_loc, cast<ShapedType>($0.getType()).getShape(), cast<ShapedType>($0.getType()).getElementType())"> $srcOpRes))
 >;
 
 def spatToPimVVDMul : Pat<