nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

rework documentation
Validate Operations / validate-operations (push) Failing after 3h10m55s
Details

Browse Source 
add plot generation scripts
adjust validation tools
add up-to-date results
This commit is contained in:
NiccoloN
2026-10-19 16:34:01 +02:00
parent 9053cc5e8b
commit 034c197260
33 changed files with 3077 additions and 1798 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+5
View File
@@ -153,6 +153,11 @@ Do not refactor when:
* When your changes create orphans, remove imports, variables, functions, or files made unused by your change
* Every changed line should trace directly to the requested fix, a required cleanup, or a justified reuse/refactor decision
# Documentation
* Do not create or retain agent change reports, investigation logs, debugging histories, temporary benchmark audits, or task-completion summaries in repository documentation. Report these findings in the final response instead.
* Documentation should describe the current system, its contracts, usage, and reproducible methodology. Document lasting limitations where relevant, without narrating how they were discovered or fixed.
# Diagnostics and verification
* Use existing bounded diagnostic mechanisms for pass-level verification or codegen failures
README.md
+58 -321
View File
@@ -1,215 +1,64 @@
# Raptor
Raptor is a domain-specific MLIR compiler for neural networks in ONNX format,
targeting in-memory computing / processing-in-memory (PIM) architectures. It
extends ONNX-MLIR with a PIM accelerator and progressively lowers ONNX-MLIR
through custom MLIR dialects to simulator artifacts.
Raptor is an MLIR compiler for ONNX neural networks targeting processing-in-memory
(PIM) architectures. It extends ONNX-MLIR and emits per-core `.pim` binaries,
`memory.bin`, `config.json`, and weights for the simulators under
`backend-simulators/pim/`. Optional JSON instructions use `--pim-emit-json`.
The current target is the Pim simulator stack under `backend-simulators/pim`.
Raptor emits binary per-core `.pim` instruction files by default, plus
`memory.bin`, `config.json`, and weight binaries. It can also emit per-core JSON
instruction files with `--pim-emit-json`.
The target has shared host memory, PIM cores with ReRAM crossbars, explicit
inter-core communication, and no hardware branches or loops. Lowering,
scheduling, and memory planning keep repeated work compact until codegen.
## Overview
## Build
PIM architectures perform most computation directly in memory. The supported
target models a chip with:
- shared host memory,
- multiple PIM cores,
- ReRAM crossbars for vector-matrix / matrix-vector work,
- explicit communication between cores,
- no hardware branch or loop support in emitted simulator code.
Initialize submodules:
Because repeated work such as convolutions is eventually made explicit, emitted
instruction counts can grow quickly. Most compiler work therefore focuses on
lowering, scheduling, memory layout, and code-generation optimizations.
### Targets and simulators
- `backend-simulators/pim/pim-simulator` is the in-tree Rust functional
simulator used by validation. It reads Raptor's `pim/` artifact directory and
compares simulator output against native ONNX-MLIR execution.
- `backend-simulators/pim/pimsim-nn` contains the non-functional Pimsim
simulator used internally by validation for latency, power, and energy.
The helper scripts in `pimcomp_utils/` are for comparison with Pimcomp and
contain local paths; treat them as local utilities, not portable workflows.
## Compilation pipeline
The PIM sources live under `src/PIM` and tests under `test/PIM`. CMake exposes
them to ONNX-MLIR through generated shim directories under
`onnx-mlir/src/Accelerators/PIM` and `onnx-mlir/test/accelerators/PIM`.
High-level lowering flow:
```
ONNX-MLIR -> Spatial -> Pim (tensor) -> Pim (bufferized) -> Pim artifacts
```bash
git submodule update --init --recursive
```
1. **ONNX -> Spatial** (`src/PIM/Conversion/ONNXToSpatial`).
Lowers supported ONNX ops into the `spat` dialect
(`src/PIM/Dialect/Spatial`). Conversion patterns are split by op family under
`Patterns/{Math,NN,Tensor}` and currently cover Conv, Gemm, MatMul,
elementwise Add/Mul/Div, ReduceMean, pooling, Relu, Sigmoid, Softmax,
Concat, Gather, Reshape, Resize, and Split.
The compiler-layer target adapter supplies the target-neutral
`SpatialTargetResources`. Layout-aware plan ops advertise typed alternatives
through the Spatial layout interface; the layout planner records the
selected layout and explicit materialization edges. `LowerSpatialPlans`
then pattern-lowers those selected plans. Contraction and Conv lowering
keep semantic problems, target-dependent plans, and IR materializers in
separate layers. Passes and their invariant/layout analyses live under
`Passes/Transforms` and `Passes/Analyses`.
Follow [ONNX-MLIR's LLVM/MLIR build instructions](onnx-mlir/docs/BuildOnLinuxOSX.md).
The [CI workflow](.github/workflows/validate_operations.yml) records dependency
versions and setup, including CMake and Protobuf. Rust stable is required for
the functional simulator. Python requirements include the Graph Explorer and
need Python 3.13+; create the repository environment if absent, then install:
2. **Merge, schedule, and realize Spatial communication**
(`src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes`).
`TrivialGraphComputeMerge` performs local graph merging. One
`ScheduleAndRealizeSpatial` pass then owns scheduling, intermediate
verification, communication realization, and final verification. Supporting
scheduling code lives under `MergeComputeNodes/Scheduling`.
```bash
python3.13 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
```
3. **Spatial -> Pim** (`src/PIM/Conversion/SpatialToPim`).
Lowers Spatial operations to the `pim` dialect (`src/PIM/Dialect/Pim`),
including `pim.core`, `pim.core_batch`, communication, tensor packing, global
tensor materialization, and return-path normalization.
If compatible Protobuf libraries are unavailable locally:
4. **Bufferization** (`src/PIM/Dialect/Pim/Passes/Transforms/Bufferization`).
`PimBufferizationPreparation` establishes writable destinations without
duplicating the one-shot copy analysis, `PimOneShotBufferization` runs
MLIR's one-shot analysis,
`PimMemoryNormalization` forwards/removes redundant copies and normalizes
addressable accesses, and `PimBufferizationVerification` checks tensor
absence, contiguity, and copy address spaces.
```bash
git clone --depth 1 --branch v34.0 https://github.com/protocolbuffers/protobuf
cmake -S protobuf -B protobuf/build -G Ninja \
-DCMAKE_BUILD_TYPE=Release -Dprotobuf_BUILD_TESTS=OFF
cmake --build protobuf/build
sudo cmake --install protobuf/build
```
5. **Pim local-memory planning**
(`src/PIM/Dialect/Pim/Passes/Transforms/LocalMemoryPlanning`).
Computes whole-core lifetimes, reuses addresses for non-overlapping
allocations, and records the explicit plan in Pim IR. Reusable lifetime
analysis lives under `src/PIM/Dialect/Pim/Passes/Analyses`.
6. **Pim verification and code generation** (`src/PIM/Passes/PimCodegen` and
`src/PIM/Compiler`).
Verifies the memory plan and other Pim invariants, then emits `.pim` core
files, weights, and `memory.bin` / `config.json` without rerunning liveness.
Configure and build Raptor, adjusting `MLIR_DIR` to your LLVM build:
Supporting pieces:
- `src/PIM/Common` - shared IR, filesystem, diagnostics, reports, and utility
helpers.
- `src/PIM/Compiler` - Pim compiler options, planned-address materialization, binary
instruction format, artifact writing, weight emission, and codegen entry
points.
- `src/PIM/Conversion/SpatialToGraphviz` - optional Spatial graphviz conversion
pass.
- `src/PIM/Passes` - pass registration and auxiliary passes.
- `src/PIM/PimAccelerator.{cpp,hpp}` - ONNX-MLIR accelerator entry point.
```bash
cmake -S . -B build_release -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DONNX_MLIR_ACCELERATORS=PIM \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DMLIR_DIR="$PWD/onnx-mlir/llvm-project/build_release/lib/cmake/mlir"
cmake --build ./build_release
```
## Pim compiler options
Use release builds first. For debugging, configure `build_debug` with
`-DCMAKE_BUILD_TYPE=Debug` and the matching MLIR package, then run
`cmake --build ./build_debug`. Use debug only for small cases requiring symbols.
Optional `mold` linking uses `-fuse-ld=mold` in
`CMAKE_{EXE,SHARED,MODULE}_LINKER_FLAGS`.
Pass these to `onnx-mlir` when compiling for Pim. These are all Raptor/Pim-specific
options; `onnx-mlir --help` lists the inherited ONNX-MLIR options.
Always build through `cmake --build`; invoking `ninja` directly bypasses CMake
configuration and generated shim updates.
- `--maccel=PIM` - select the Pim accelerator. Default: no Pim accelerator.
- `--EmitSpatial`, `--EmitPim`, `--EmitPimBufferized`,
`--EmitPimCodegen` - stop the Pim pipeline at the requested stage. Default:
`--EmitPimCodegen` for Pim compilation.
- `--core-count=<N>` - required positive core count for Pim compilation.
Default: none; this option is required.
- `--crossbar-size=<N>` - required positive crossbar width/height for Pim
compilation. Default: none; this option is required.
- `--crossbar-count=<N>` - required positive crossbar count per core for Pim
compilation. Default: none; this option is required.
- `--pipeline=<N>` - number of throughput pipeline stages; `1` preserves
latency scheduling. Default: `1`.
- `--pim-target-config=<PATH>` - optional Pim target configuration used by the
target adapter to construct the target-neutral Spatial scheduling cost and
topology model. Resource values must match the explicit core/crossbar flags.
Default: empty; use the built-in target model.
- `--pim-memory-report=<summary|none>` - emit the concise combined memory report
under `reports/memory_report.txt`, or disable it. Default: `summary`.
- `--pim-only-codegen` - assume input is already bufferized Pim IR and only run
the codegen tail. Default: off.
- `--pim-disable-synchronization` - omit generated `wait` and `sync`
instructions for performance ablation. Default: off.
- `--pim-disable-spatial-planning` - select the first, trivial DenseNCHW layout
alternative for every Spatial plan operation, disabling cost-based layout
planning while leaving ONNX rewrites and graph-compute merging enabled.
Default: off.
### Spatial layout plan variants
Spatial plan operations advertise alternatives as an exact combination of
operand physical layouts and one result physical layout. Every plan operation
has the default `DenseNCHW -> DenseNCHW` alternative. The planner can select
the following additional variants when the operation, tensor shapes, and
target resources make them legal:
| Physical layout or plan | Meaning and current use |
|---|---|
| `DenseNCHW` | Ordinary dense NCHW storage. This is the first alternative and the one selected by `--pim-disable-spatial-planning`. |
| `NHWCRowStrip` | Row-strip storage for NCHW logical tensors: spatial rows are processed as channel vectors. This enables row-strip lowering through compatible chains. |
| `Fragmented` | Fragmented physical input accepted by `Flatten`, which reassembles it to dense NCHW. It is not currently selected as a plan result. |
| `NCHWRowStrip` | A Spatial IR layout enum value reserved for NCHW-oriented row strips; current layout-capability implementations do not advertise it as a plan alternative. |
The operation-specific non-trivial alternatives are:
| Plan operation | Additional alternatives beyond dense NCHW |
|---|---|
| `Conv2D` | Dense input to row-strip output, or row-strip input to row-strip output when the target-dependent Conv lowering supports it. |
| `Flatten` | Fragmented input to dense output, or row-strip input to dense output when legal. |
| `Relu` | Row-strip input to row-strip output. |
| `SiLU` | Row-strip input to row-strip output, with a stronger intrinsic cost preference than the generic row-strip variant. |
| `ResizeNearest` | Row-strip input to row-strip output when its lowering is legal. |
| `MaxPool2D` | Dense input to row-strip output, or row-strip input to row-strip output. |
| `GlobalAveragePool` | Dense input to row-strip output, or row-strip input to row-strip output. |
| `BiasAdd` | Row-strip data input plus a dense bias input to row-strip output when the bias shape is supported. |
| `Add` | All data inputs row-strip to row-strip output. |
| `Concat` | All inputs row-strip to row-strip output. |
Cost-based planning scores intrinsic alternative cost, operand layout
mismatches, and downstream incompatibility, then iterates in alternating
forward and reverse operation order until the bounded analysis converges.
Function results are required to remain `DenseNCHW`; explicit materialization
operations reconcile layout mismatches at boundaries. With
`--pim-disable-spatial-planning`, the pass still runs and records a valid plan,
but chooses the first dense alternative for every plan operation. Later graph
compute merging is unchanged, so elementwise operations such as `Relu` remain
separate from neighboring parallel operations and can create fan-out/fan-in
diamonds.
- `--pim-emit-json` - also emit `core_*.json` instruction files alongside
`core_*.pim`. Default: off.
- `--pim-export-spatial-dataflow=<none|spatial1|spatial2|spatial3|spatial4|all>` -
control Spatial dataflow CSV reports for the graph, trivially merged graph,
scheduled, and realized snapshots under `reports/`. Default: `none`.
- `--pim-conv-lowering=<auto|legacy|depthwise|packed-im2col|streamed-patch|streamed-packed|output-channel-tiled|input-k-tiled|tiled-2d>` -
select the convolution lowering strategy. Default: `auto`.
- `--pim-conv-im2col-max-elements=<N>` - maximum globally materialized im2col
elements per convolution before streaming. Default: `1048576`.
- `--pim-conv-stream-chunk-positions=<N>` - maximum output positions per
streamed convolution chunk. Default: `1024`.
- `--pim-report-conv-lowering=<true|false>` - emit a bounded convolution
lowering report. Default: `true`.
- `--pim-detect-communication-deadlock` - statically simulate expanded
send/receive ordering and reject blocking deadlocks. Default: off.
- `--pim-verify-bufferization-copy-freedom` - run the expensive official Pim
tensor-copy freedom proof before bufferization. Default: off.
## Standard Pim hardware profile
Raptor's standard development and YOLO validation profile is:
| Parameter | Value |
| --- | ---: |
| Cores | 144 |
| Crossbars per core | 64 |
| Crossbar size | 128 × 128 |
Canonical compiler flags:
`--crossbar-count=64 --crossbar-size=128 --core-count=144`
`--crossbar-size`, `--crossbar-count`, and `--core-count` remain mandatory and
must be passed explicitly to the compiler.
Example:
## Compile a model
```bash
./build_release/Release/bin/onnx-mlir model.onnx -o /tmp/raptor/model \
@@ -217,131 +66,19 @@ Example:
--crossbar-count=64 --crossbar-size=128 --core-count=144
```
This writes Pim artifacts under `/tmp/raptor/pim/`.
This writes artifacts under `/tmp/raptor/pim/`. All three hardware flags are
mandatory positive values. The example uses the development/YOLO profile;
[Pimcomp comparison profiles](validation/networks/pimcomp_models/README.md#hardware-profiles)
differ, and validation defaults to Arch-A.
## Validation
## Documentation
Functional validation compiles ONNX models, compares native ONNX-MLIR and Pim
simulator outputs, and optionally reports latency, power, and energy. See
[`validation/README.md`](validation/README.md) for prerequisites, usage,
options, artifacts, and results.
Start with the [documentation index](docs/README.md). It links the
[compiler architecture](docs/compiler/architecture.md),
[compiler options](docs/compiler/options.md), and
[evaluation methodology](docs/evaluation/methodology.md).
## Build
Initialize submodules first:
```bash
git submodule update --init --recursive
```
The project follows ONNX-MLIR's build requirements. The CI workflow documents
the currently used versions and setup:
- CMake 4.3.0 in CI,
- LLVM/MLIR checked out under `onnx-mlir/llvm-project`,
- Protobuf `v34.0`,
- Rust stable for `pim-simulator`,
- Python packages `numpy`, `onnx`, `colorama` for validation.
### Protobuf
Install Protobuf if your system does not already provide a compatible version:
```bash
git clone --depth 1 --branch v34.0 https://github.com/protocolbuffers/protobuf
cmake -S protobuf -B protobuf/build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-Dprotobuf_BUILD_TESTS=OFF
cmake --build protobuf/build
sudo cmake --install protobuf/build
```
You can then remove the temporary checkout:
```bash
rm -rf protobuf
```
### MLIR
Follow the ONNX-MLIR instructions in
`onnx-mlir/docs/BuildOnLinuxOSX.md` to build LLVM/MLIR. The local Raptor build
expects `MLIR_DIR` to point at the MLIR CMake package, for example:
```bash
MLIR_DIR=$(pwd)/onnx-mlir/llvm-project/build_release/lib/cmake/mlir
```
If your LLVM build directory is named `build` instead of `build_release`, adjust
the path accordingly.
### Raptor
Configure a release build:
```bash
MLIR_DIR=$(pwd)/onnx-mlir/llvm-project/build_release/lib/cmake/mlir
cmake -S . -B build_release -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DONNX_MLIR_ACCELERATORS=PIM \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DMLIR_DIR=${MLIR_DIR}
```
Configure a debug build similarly:
```bash
MLIR_DIR=$(pwd)/onnx-mlir/llvm-project/build_debug/lib/cmake/mlir
cmake -S . -B build_debug -G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DONNX_MLIR_ACCELERATORS=PIM \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DMLIR_DIR=${MLIR_DIR}
```
For debug development, using `mold` can reduce link time and memory use:
```bash
cmake -S . -B build_debug -G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DONNX_MLIR_ACCELERATORS=PIM \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DMLIR_DIR=${MLIR_DIR} \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=mold" \
-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=mold" \
-DCMAKE_MODULE_LINKER_FLAGS="-fuse-ld=mold"
```
Build the compiler with CMake:
```bash
cmake --build ./build_release
cmake --build ./build_debug
```
Do not invoke `ninja` directly for this project; use `cmake --build` so CMake's
configuration and generated shims stay consistent.
If a build fails because Protobuf headers are missing fixed-width integer
definitions, patch the affected Protobuf-generated files by adding
`#include <cstdint>`.
## Tests
The Rust simulator has its own tests:
```bash
cd backend-simulators/pim/pim-simulator
cargo test
```
## Repository layout
- `src/PIM/` - PIM accelerator implementation.
- `test/PIM/` - PIM C++ unit tests.
- `validation/` - functional validation scripts, ONNX operation tests, network
slices, and pimsim config generation.
- `backend-simulators/pim/pim-simulator/` - in-tree Rust functional simulator.
- `backend-simulators/pim/pimsim-nn/` - non-functional simulator submodule.
- `pimcomp_utils/` - local comparison helpers for Pimcomp.
- `.github/actions/` and `.github/workflows/validate_operations.yml` - CI setup
for MLIR/Protobuf caching, building Raptor, and validation.
[Validation commands](validation/README.md), the
[operation inventory](validation/operations/INVENTORY.md), and the
[Graph Explorer](tools/raptor_graph_explorer/README.md) remain beside their
components.
docs/README.md
+39
View File
@@ -0,0 +1,39 @@
# Raptor documentation
Start with the [build and quick start](../README.md). Commands in these guides
run from the repository root unless stated otherwise.
## Compiler
- [Architecture, compilation pipeline, source layout, and simulator tests](compiler/architecture.md)
- [Pim compiler options and defaults](compiler/options.md)
- [Spatial physical layouts and compatibility matrix](compiler/spatial-layouts.md)
- [Scheduling ownership, costs, capacity, and complexity](compiler/scheduling.md)
- [Scheduling implementation contracts and model limitations](compiler/scheduling-implementation.md)
## Evaluation
- [Measurement methodology and canonical result datasets](evaluation/methodology.md)
- [Pimcomp operation support, standalone reproduction, and GA controls](evaluation/pimcomp-reproducibility.md)
- [Validation commands and defaults](../validation/README.md)
- [Pimcomp comparison and plotting](../validation/tools/pim/pimcomp/compare/README.md)
- [Raptor ablation and plotting](../validation/tools/pim/ablation/README.md)
- [Pimcomp correctness experiments](../validation/tools/pim/pimcomp/correctness/README.md)
## Component references
- [Operation suite](../validation/operations/README.md) and [all 177 cases](../validation/operations/INVENTORY.md)
- [Comparison models, provenance, checksums, and hardware profiles](../validation/networks/pimcomp_models/README.md)
- [Graph Explorer usage](../tools/raptor_graph_explorer/README.md) and [architecture/API](../tools/raptor_graph_explorer/ARCHITECTURE.md)
## Normative engineering contracts
These contracts stay in `.agents/invariants/`; general documentation links to
them rather than restating their requirements:
- [Graph compute batch fragments](../.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md)
- [Compiler performance](../.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md)
- [Pimsim performance oracle](../.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md)
- [PIM synchronization](../.agents/invariants/PIM_SYNCHRONIZATION_INVARIANT.md)
- [Pipeline scheduling](../.agents/invariants/PIPELINE_SCHEDULING_INVARIANT.md)
- [Spatial target generality](../.agents/invariants/SPATIAL_TARGET_GENERALITY_INVARIANT.md)
docs/compiler/architecture.md
+51
View File
@@ -0,0 +1,51 @@
# Compiler architecture and pipeline
See the [documentation index](../README.md) for related compiler and evaluation
references. Source paths below are relative to the repository root.
## Compilation pipeline
```text
ONNX-MLIR -> Spatial -> Pim (tensor) -> Pim (bufferized) -> Pim artifacts
```
| Stage | Responsibility and source |
|---|---|
| ONNX → Spatial | Operation lowering, target-neutral layout planning, and plan materialization in `src/PIM/Conversion/ONNXToSpatial`. |
| Spatial scheduling | Graph merging, scheduling, verification, and communication realization in `src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes`; see [scheduling ownership](scheduling.md). |
| Spatial → Pim | Cores, communication, packing, and global tensors in `src/PIM/Conversion/SpatialToPim`. |
| Bufferization | Writable destinations, one-shot analysis, memory normalization, and verification in `src/PIM/Dialect/Pim/Passes/Transforms/Bufferization`. |
| Local-memory planning | Whole-core lifetimes and address reuse in `src/PIM/Dialect/Pim/Passes/Transforms/LocalMemoryPlanning`. |
| Verification/codegen | Verify the explicit memory plan and emit artifacts in `src/PIM/Passes/PimCodegen` and `src/PIM/Compiler`. |
Spatial plans select physical layouts and insert explicit materializations at
boundaries. Dense NCHW is the default; compatible operations can use row-strip
layouts, while Flatten can reassemble fragmented inputs. Function results
remain dense NCHW. See the [layout compatibility reference](spatial-layouts.md)
for operation alternatives, legality checks, and planner behavior.
## Validation and tests
[Validation instructions](../../validation/README.md) cover ONNX-MLIR reference
comparison, functional simulation, performance measurement, and artifacts.
The Rust functional simulator reads Raptor's artifact directory;
`pimsim-nn` supplies latency, power, and energy estimates.
```bash
cd backend-simulators/pim/pim-simulator
cargo test
```
## Repository layout
- `src/PIM/`: accelerator, dialects, conversions, analyses, and codegen.
- `test/PIM/`: compiler unit tests.
- `validation/`: operation/network suites and simulator configuration.
- `backend-simulators/pim/pim-simulator/`: Rust functional simulator.
- `backend-simulators/pim/pimsim-nn/`: performance simulator submodule.
- `tools/raptor_graph_explorer/`: [Spatial CSV viewer](../../tools/raptor_graph_explorer/README.md).
- `pimcomp_utils/`: comparison helpers containing local paths.
- `.github/`: CI build and validation setup.
CMake exposes `src/PIM` and `test/PIM` through generated shims under
`onnx-mlir/src/Accelerators/PIM` and `onnx-mlir/test/accelerators/PIM`.
docs/compiler/options.md
+28
View File
@@ -0,0 +1,28 @@
# Pim compiler options
See the [quick start](../../README.md#compile-a-model) for a complete command
and the [documentation index](../README.md) for the compilation pipeline.
Pass these to `onnx-mlir`; `--help` also lists inherited ONNX-MLIR options.
Boolean switches default to off unless stated otherwise.
| Option | Purpose / default |
|---|---|
| `--maccel=PIM` | Select the PIM accelerator. |
| `--EmitSpatial`, `--EmitPim`, `--EmitPimBufferized`, `--EmitPimCodegen` | Stop at a pipeline stage; default: codegen. |
| `--core-count=N`, `--crossbar-size=N`, `--crossbar-count=N` | Required hardware resources; crossbars are square. |
| `--pipeline=N` | Throughput stages; default: 1 (latency scheduling). |
| `--pim-target-config=PATH` | Target cost/topology configuration; resources must match hardware flags. Default: built-in model. |
| `--pim-memory-report=summary\|none` | Memory report at `reports/memory_report.txt`; default: `summary`. |
| `--pim-only-codegen` | Run the codegen tail on bufferized Pim IR. |
| `--pim-emit-json` | Also emit per-core JSON instructions. |
| `--pim-export-spatial-dataflow=none\|spatial1\|spatial2\|spatial3\|spatial4\|all` | Export graph, trivially merged, scheduled, and realized CSV snapshots under `reports/`; default: `none`. |
| `--pim-conv-lowering=STRATEGY` | `auto` (default), `legacy`, `depthwise`, `packed-im2col`, `streamed-patch`, `streamed-packed`, `output-channel-tiled`, `input-k-tiled`, or `tiled-2d`. |
| `--pim-conv-im2col-max-elements=N` | Global im2col limit before streaming; default: 1048576. |
| `--pim-conv-stream-chunk-positions=N` | Output positions per streamed chunk; default: 1024. |
| `--pim-report-conv-lowering=true\|false` | Bounded lowering report; default: true. |
| `--pim-disable-spatial-planning` | Choose each plan's first dense alternative; ONNX rewrites and graph merging remain enabled. |
| `--pim-disable-synchronization` | Omit wait/sync for ablation; removes correctness guarantees for buffer reuse and restart. |
| `--pim-detect-communication-deadlock` | Check expanded blocking send/receive ordering. |
| `--pim-verify-bufferization-copy-freedom` | Run the expensive pre-bufferization copy-freedom proof. |
docs/compiler/scheduling-implementation.md
+98
View File
@@ -0,0 +1,98 @@
# Scheduling implementation notes
The [scheduling overview](scheduling.md) maps ownership and summarizes costs/complexity.
Normative graph, pipeline, synchronization, and target contracts are linked
there; the details below describe current implementation choices.
## Publication, order, and local materialization
[`materializeScheduledCompute`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledComputeMaterialization.cpp) records
graph IDs for publication identity, not execution order. Class lanes retain
graph encounter order independently of physical core IDs, preserving regular
source-lane mappings. Within each core, instances sort by verified scheduler
slots. Only consecutive compatible instances form homogeneous runs; class-lane
ordering must not reorder a core's work.
`insertLocalFragment` replaces a complete singleton insertion with a leading
unit-dimension reshape view only when:
- source and destination have static shapes;
- destination rank is source rank + 1, with leading dimension 1 and remaining
dimensions exactly equal to the source shape;
- element type and encoding match;
- offset/size/stride lists all match destination rank; and
- every offset is statically zero, every size is the full corresponding
destination dimension, and every stride is statically one.
This includes scalar sources. Other insertions retain `tensor.insert_slice`
semantics rather than being converted into views.
Receive forwarding belongs to
[`PimBufferizationPass.cpp`](../../src/PIM/Dialect/Pim/Passes/Transforms/Bufferization/PimBufferizationPass.cpp),
after copy lowering and contiguity normalization. It requires a private
allocation and single-use receive/copy chain in the same block, zero source and
receive offsets, and equal sizes. The receive can sink to the copy target past
memory-effect-free operations and `memref.load` (including lane lookup loads).
Writes, synchronization, and other effectful operations are barriers. Forwarding
does not establish synchronization policy.
## Transfer accounting and transpose costs
[`ComputeGraph.cpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ComputeGraph.cpp) uses the same cached publication map for
dependency extraction and projected transfer volume. Physical slots map back
to graph lanes; graph-range overlap is not a byte estimate. Blueprint edges
belong to this shared graph for both scheduling strategies.
Exact projections charge selected physical fragments individually, including
each fragment's byte, endpoint-memory, and packet rounding. Conservative batch
dependencies charge a producer chunk as one aggregate: scale element count by
chunk lanes, multiply by effective bitwidth, then round to bytes once before
memory-access/flit rounding. Dividing an already rounded whole-result byte
count would lose sub-byte payloads. These two granularities are deliberate.
Spatial can analyze sub-byte storage; PIM communication lowering requires
byte-sized elements. Blueprint assembly charges per fragment. A full fragment
receive directly into assembly storage omits duplicate endpoint-memory edge
cost; later copy elimination can leave this estimate conservative.
Execution transpose costs share `getTransposeRun` with lowering: round memory
reads/writes per run and add vector execution when source stride is not one.
The [Pimsim vector unit](../../backend-simulators/pim/pimsim-nn/src/core/component/VectorUnit.cpp)
charges vector payload bytes, not the full strided address footprint. The
compiler model follows that accounting; instruction setup/issue and contention
remain approximations, not extra inferred memory traffic.
## Priorities, placement, and recovery limits
[`PipelineScheduling.cpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp) caches a per-template
adjustment between physical execution cost and nominal volume priority, scaling
it by chunk lanes. Partitioning and packing use the nominal priorities;
transposes count as bulk moves there. PEFT, final schedule evaluation, and
reported per-core demand use physical estimates. These quantities must not be
substituted for one another when interpreting results.
Placement compares maximum per-core serial compute plus incident same-stage
transfer demand, then completion/span and compute load. Each participating core
is charged incident transfer demand as a placement proxy; those charges are not
summed twice into elapsed time. Within 1% demand difference, span breaks the tie.
Ordered packing plus dependency-tail priority biases cuts away from long
dependent groups; it is not a throughput prediction. Measured simulator periods
remain authoritative, including when a proxy improves.
Core budgets come from stage allocation and bounded capacity recovery. Failed
packing budgets are not lower bounds. [PEFT reservations](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeftScheduler.cpp)
prefer weight reuse when distinct weights exceed half the fleet's one-copy
resident capacity; otherwise they favor communication/load. The reuse-first
constructor supplies feasibility recovery without replacing a successful
reservation, and can concentrate work. No exact combinatorial solver proves
infeasibility after heuristic failure.
[`ResidentWeightPacking.hpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp) adds at most 17
constructions after initial failure: one reverse traversal, then up to 16
single-choice replays sharing 65,536 candidate checks. Scratch buffers are reused;
an extra `O(T)` failed-assignment snapshot is created only after both initial
orders fail. Weight indexing still has `O(R W)` worst-case equality work; partition
DP and latency insertion-gap searches remain superlinear as listed in the
scheduling overview. Cached PEFT center distances avoid a topology scan per task/candidate;
projection and priority analyses share/cache work rather than revisiting every
template body per chunk. Ready-transfer signature searches remain another
potential compile-time hotspot.
docs/compiler/scheduling.md
+101
View File
@@ -0,0 +1,101 @@
# Spatial scheduling
`ScheduleAndRealizeSpatial` constructs a shared `ComputeGraph`, then selects
latency PEFT or pipeline scheduling. Pipeline mode does not run PEFT.
See [implementation notes](scheduling-implementation.md) for fragment-view eligibility,
receive forwarding, transfer rounding, cost-model boundaries, and recovery limits.
## Ownership
| Responsibility | Owner |
|---|---|
| Instances, dependencies, execution/transfer costs, resident weights | `ComputeGraph`, `ComputeInstanceUtils` |
| Target resources and physical link costs | Compiler target adapter via `SchedulingTarget` |
| Latency priorities, reservations, placement, per-core order | `PeftScheduler` |
| Capacity-driven workload splitting | `PipelineWorkloadPreparation` |
| Groups, stage partitioning, core budgets | `buildPipelineGroups`, `partitionPipelineGroups`, `assignPipelineStages` |
| Topology clustering and stage-local packing | `PipelineCoreLayout`, `packPipelineStages`, `IndexedResidentWeights` |
| Timing evaluation and publication equivalence | `evaluateSchedule`, `populateScheduleEquivalenceClasses` |
| Selected execution order and homogeneous runs | Scheduled compute planning/materialization |
| Routing, forwarding, blocking communication order | Deferred transfer planning and communication scheduling |
| READY/FREE counts, registers, overwrite/restart boundaries | Deferred boundary planning/realization |
Both strategies produce `MergeScheduleResult`: core assignments, contiguous
per-core slots, estimated starts, last-task markers, equivalence classes, and
cost summaries. Pipeline results also assign each core a stage. Materialization
uses verified slots for execution order and merges only consecutive compatible
instances. Graph/class lane identity does not imply core identity or execution
order.
Dependencies and transfer costs share `getGraphBatchPublicationMap`.
Statically proven physical-slot projections support many-to-many edges;
unproven projections retain complete producer dependencies, including region
captures. Malformed publication metadata fails graph construction.
Normative contracts:
[graph fragments](../../.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md),
[pipeline stages](../../.agents/invariants/PIPELINE_SCHEDULING_INVARIANT.md),
[target boundary](../../.agents/invariants/SPATIAL_TARGET_GENERALITY_INVARIANT.md),
[synchronization](../../.agents/invariants/PIM_SYNCHRONIZATION_INVARIANT.md),
and [performance](../../.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md).
## Costs and interpretation
Execution and fixed transfer costs use integer nanoseconds, converted from
target cycles. Bytes, elements, flits, weight slots, and lanes remain resource
quantities. Memory work is rounded at its actual fragment/run granularity;
runtime-free views cost zero. Transposes charge memory traffic and strided
vector work; concatenation charges per-input blocks per outer row. Direct
receives into assembly storage avoid duplicate endpoint-memory charges.
Sub-byte storage is analyzable in Spatial but rejected by byte-addressed PIM
communication lowering.
The evaluator separates serial work, per-core work, dependency critical path,
summed transfer work, single-inference completion, and stage spans. Dependency
paths exclude core-order constraints; completion includes them. Stage spans
exclude cross-stage routing, restart/host synchronization, and contention.
Simulator latency and measured throughput remain the performance gates.
Pipeline partitioning uses ordered-packing/dependency-tail priorities and
nominal movement volumes. PEFT and final evaluation use physical cost estimates.
Placement ranks maximum per-core compute/incident-transfer demand, then stage
completion and compute load; within 1% demand, stage span breaks the tie.
These are placement heuristics, not a steady-state timing model.
## Capacity and recovery
Distinct-weight union / per-core capacity is a lower bound; an oversized task
is infeasible. Ordered pack counts estimate budgets, and failed heuristic
packing does not prove infeasibility. Exhausted budgets restart from lower
bounds and constructive packing establishes feasibility.
Latency reservations try load/communication-aware placement, then
largest-weight-first and shared reuse-first best fit on failure. Pipeline
packing compares arrival/load-aware candidates and uses constructive recovery.
The shared constructor retries in reverse order, then at most 16 single-choice
deviations within 65,536 core-candidate checks. Successful placements are
preserved. Exhausted search produces a bounded diagnostic, not a claim of
proved hardware infeasibility.
## Complexity and limits
Let `T` be tasks, `E` dependencies, `P` cores, `G` groups, `S` stages,
`R` weight references, `W` distinct weights, and `C` modeled work.
| Analysis | Time / storage |
|---|---|
| Schedule evaluation | `O((T + E) log T + P)` / `O(T + E + P)` |
| Weight indexing | Up to `O(R W)` equality comparisons |
| Weight lower bounds | `O(R + W)` / `O(W)` bits |
| Constructive best fit | `O(T log T + P R + P W)` / `O(P W)` bits plus `O(T + P)`; bounded retries only on failure |
| Partition DP | Up to `O(S G² ((T + R) log C + W))` plus dependency traversal / `O(S G²)` cache |
| Topology clustering / cached PEFT distances | `O(P²)` time |
| Concat cost extraction | `O(A D)` for `A` inputs of rank `D`; constant auxiliary space |
| Cached pipeline priorities | `O(T + D)` time / `O(T + G)` space for `D` template-body operations |
| Exact projections | `O(Q + L)` time/temporary space for `Q` slot visits and `L` producer lanes; publication maps cached |
| Conservative producer tracing | Linear in the input projection DAG, including captures |
Latency insertion-gap searches can remain quadratic. Issue overhead,
eliminated assembly copies, contention, and pipeline synchronization remain
cost approximations. Changes require matched execution measurements.
docs/compiler/spatial-layouts.md
+63
View File
@@ -0,0 +1,63 @@
# Spatial layout capabilities
This reference describes the alternatives advertised by
[`SpatialLayoutCapabilities.cpp`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Analyses/SpatialLayoutCapabilities.cpp),
through `SpatialLayoutCapabilityInterface`. Each alternative specifies all
operand layouts and one result layout; it is not a global promise that every
shape or target can use that layout.
## Physical layouts
Logical tensor coordinates remain NCHW. The physical enum is defined in
[`Spatial.td`](../../src/PIM/Dialect/Spatial/Spatial.td).
| Layout | Contract and current use |
|---|---|
| `DenseNCHW` | Ordinary dense storage; first/default alternative for every layout-aware plan. |
| `NHWCRowStrip` | Physical row strips of channel vectors for logically NCHW tensors; compatible operations can preserve this representation through a chain. |
| `Fragmented` | Existing fragmented input accepted by Flatten for dense reconstruction; no current capability advertises it as a result alternative. |
| `NCHWRowStrip` | Reserved NCHW-oriented row-strip enum; no current capability advertises it as an alternative. |
## Compatibility matrix
Every operation below also advertises all-dense operands → `DenseNCHW`.
Here `R` means `NHWCRowStrip`, `D` means `DenseNCHW`; Conv's other operands
retain the layouts supplied to its interface on the `R` path.
| Plan operation | Additional operand → result alternatives | Interface gate |
|---|---|---|
| Conv2D | Dense input → R; R input → R | `canLowerConvPlanToRowStrip`; `canConsumeAndProduceRowStrip`, respectively. |
| Flatten | Fragmented → D; R → D | Fragmented input already present; `canLowerFlattenFromRowStrip` for R. |
| Relu | R → R | Input already R. |
| SiLU | R → R | Input already R; intrinsic cost −3 instead of the generic R cost −2. |
| ResizeNearest | R → R | Input already R and `canLowerResizeNearestPlanToRowStrip`. |
| MaxPool2D | D → R; R → R | `canLowerMaxPoolPlanToRowStrip`. |
| GlobalAveragePool | D → R; R → R | `canLowerGlobalAveragePoolPlanToRowStrip`. |
| BiasAdd | (R data, D bias) → R | Ranked result, R data input, and `isSupportedBiasAddValue` for the bias shape. |
| Add | (R, R) → R | Both data operands already R. |
| Concat | All R inputs → R | Nonempty operand list, all already R. |
The named predicates live in the [plan-lowering implementation](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms)
and [bias helper](../../src/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp). They check operation geometry and
target resources; the matrix does not replace those legality checks.
## Selection and materialization
[`SpatialLayoutPlanningPass.cpp`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms/SpatialLayoutPlanningPass.cpp)
starts every plan at its dense alternative. It scores intrinsic cost, operand
layout mismatches, and downstream inability to accept the candidate layout.
Strictly cheaper alternatives replace the current selection; ties preserve it.
Traversal alternates forward/reverse order for at most `2 × plan_count + 1`
rounds. Nonconvergence fails the pass with a diagnostic.
Results returned from the function must remain dense. The pass records selected
layouts and inserts explicit `spat.materialize_layout` operations where selected
producer/consumer contracts disagree, including non-layout-aware consumers of
non-dense values. It verifies exact operand/result compatibility before
[`LowerSpatialPlans`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms/LowerSpatialPlansPass.cpp) lowers plans
and materializations.
`--pim-disable-spatial-planning` skips cost selection, choosing alternative 0
for every plan. The planning/materialization/verification pass still runs.
ONNX rewrites and later graph merging are unchanged; disabling layout planning
does not request fusion or removal of separate elementwise graph nodes.
docs/evaluation/methodology.md
+75
View File
@@ -0,0 +1,75 @@
# Evaluation methodology
The [documentation index](../README.md) links execution guides, model provenance,
reproduction details, and correctness experiments.
## Canonical result datasets and output roots
The authoritative default aggregate CSVs are:
- [Comparison results](../../validation/tools/pim/pimcomp/compare/results_comparison.csv)
- [Ablation results](../../validation/tools/pim/ablation/results_ablation.csv)
They are intentionally checked in beside their scripts. Default benchmark runs
update the corresponding canonical CSV; default plotting reads its current
contents. Plot generation does not run benchmarks or establish publication
provenance. Preserve the model/configuration/source/build revisions, invocation,
reference inputs, and original per-case reports when publishing measurements.
An explicit custom `--out-dir PATH` keeps aggregate CSVs under that experiment
root. Model artifacts still default to `validation/networks/pimcomp_models`;
changing aggregate placement does not change artifact or shared-reference paths.
Per-variant ablation summaries remain `<artifact-root>/ablation/<variant>/results_comparison.csv`.
Use `--input PATH` to plot a custom aggregate. Plot destinations can be selected
with `--output-dir PATH`. Cleanup of default model artifacts preserves the two
canonical datasets.
## Functional validity and performance availability
Functional PASS/FAIL and performance availability are independent. A numerically
correct program can fail performance simulation or complete no inference before
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
per-inference measurements become `NA` in aggregates, not zero. With zero
completed outputs, interval, throughput, and energy per inference are unavailable;
measured average chip power can still be reported. Functional status is retained.
Ablation percentages require positive finite values for both the variant and
its matching baseline. Tiny positive measurements retain significant digits
instead of being rounded to zero in the comparison CSV. An extreme valid
decrease can round to `-100.00%`; that value alone is not evidence of failure.
Reciprocal output interval and throughput cannot both have that decrease.
The ablation plotter warns and displays `N/A` for such invalid paired percentages,
while leaving the CSV and functional status unchanged.
## Report reuse and implementation revisions
The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
describes fingerprint validation and failed-report reuse. Fingerprints cover
model/configuration contents and experiment settings, but do not identify
compiler/tool implementations, built binaries, or manually changed reference
artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant
compiler after those change, and record build/source revisions for publication.
No revision cache is inferred from Git HEAD, which would miss dirty sources,
external tools, and binaries built from another checkout.
## Metric interpretation
Metrics come from Pimsim's
[`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp)
and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp).
- Latency mode measures time to complete one inference.
- Throughput-mode latency is measurement duration / completed iterations,
equal to `1000 / throughput` before rounding. It is not end-to-end pipelined
inference latency; multiplying it by depth does not establish that quantity.
- Average power is total modeled static + dynamic chip energy / elapsed time.
- Energy per inference is total energy / completed iterations (one in latency
mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size
division is applied.
Fast mode measures throughput over the last two convergence windows but power
and energy over the entire run up to early stop. Energy includes startup and
work in flight, so it need not equal average power / reported throughput.
Use `--no-fast` for full-duration measurements. Consult the
[correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows
as performance comparisons.
docs/evaluation/pimcomp-reproducibility.md
+111
View File
@@ -0,0 +1,111 @@
# Pimcomp reproduction reference
The [model README](../../validation/networks/pimcomp_models/README.md) owns provenance, checksums, hardware profiles, and
numeric-artifact limitations. The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
owns suite commands, reports, reuse rules, and metric interpretation.
## Frontend acceptance versus scheduled computation
The current [frontend mapping](../../third_party/PIMCOMP-NN/frontend/frontend.py)
accepts exactly these ONNX node types:
```text
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
```
`Constant` is consumed as metadata rather than emitted as a backend node.
Other types cause `operation: <type> not considered` and stop conversion at
the first occurrence. In particular, YOLO11n's Split and Resize are unsupported;
the emitted prefix cannot represent a complete inference.
The latency scheduler's explicit no-consider set, also used by hierarchy
mapping and GA post-operation accounting, is:
```text
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
Softmax, Squeeze, Transpose
```
See [ElementPipelineSchedule.cpp](../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
[HierarchyMapping.cpp](../../third_party/PIMCOMP-NN/backend/HierarchyMapping.cpp),
and [GeneticAlgorithm.cpp](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.cpp).
These accepted nodes receive no standalone latency instructions while remaining
in the backend graph. Acceptance therefore does not establish timed computation.
Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into
Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific
Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer.
Those transformations differ from both frontend rejection and leaving a node
unscheduled. In particular, the [BatchNorm correctness gap](../../validation/tools/pim/pimcomp/correctness/README.md#resnet-batchnorm-gap)
shows why a fusion label is not proof of numeric equivalence.
Pimsim accepts ISA instructions rather than ONNX operators. Its
[instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp)
rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must
not be approximated in the oracle. This is why the dedicated GoogLeNet artifact
removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other
operations, including MatMul/Transpose and detection decoding, for Raptor.
## Standalone frontend/backend reproduction
Build Pimcomp with CMake and an OpenMP-capable C++ compiler:
```bash
cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
```
The backend resolves `../config.json`, `../models/JSON/<name>.json`, and
`../output/` relative to its working directory, not its executable. The frontend
rewrites its input ONNX batch dimension in place. Use an isolated workspace and
a model copy, as the comparison driver does, to preserve source/config files.
From the Raptor root, after installing the [frontend dependencies](../../third_party/PIMCOMP-NN/README.md):
```bash
RAPTOR_ROOT=$PWD
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
PIMCOMP_RUN=$(mktemp -d)
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
--model_path "$PIMCOMP_RUN/vgg8.onnx" \
--save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
(
cd "$PIMCOMP_RUN/work"
OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
-m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
)
```
Outputs remain under `$PIMCOMP_RUN/output`. `-v=YES` emits verification data;
`-s=YES` emits `SimulationInfo.gz`; `-o=YES` saves the evaluation report.
Use `-p=batch` for throughput scheduling, with the matching throughput config
when running Pimsim. The paper's batch experiment used 128 samples; the current
comparison runner's functional batch default is 64. Element mode represents one
inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular
crossbars remain unsupported by Raptor.
For the bundled Model Zoo graphs, copy `models/JSON/resnet18.json`,
`resnet34.json`, or `googlenet.json` from Pimcomp to the workspace and change
`-m` accordingly. That reproduces bundled graph scheduling; use the suite
runner to regenerate graphs from the exact checked-in performance ONNX files.
## Genetic algorithm and reproducibility limits
[`GeneticAlgorithm.h`](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.h)
sets population 200 and at most 1000 iterations. `-r=GA` selects it; standalone
backend default replication is `balance`, whereas the suite explicitly uses GA.
GA initialization calls `srand(1)`. Fitness evaluation uses OpenMP with static
work distribution; `OMP_NUM_THREADS` bounds its parallelism (use 1 for serial
reproduction), otherwise the OpenMP runtime chooses the thread count.
The fixed GA seed does not make every backend stage deterministic:
`ElementPipelineSchedule.cpp` separately calls `srand(time(NULL))`. Keep model
hashes, source/build revisions, profile, mode, and thread count with experiment
results. Compare correctness
against the original ONNX reference, including the known batch and BatchNorm
failures; neither omitted operators nor prefilled reference intermediates repair
those programs.
requirements.txt
+1
View File
@@ -1,4 +1,5 @@
colorama>=0.4.6,<1
matplotlib>=3.8,<4
numpy>=1.26.4,<3
onnx>=1.17,<2
onnxsim>=0.6.5,<1
src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/README.md
+6 -174
View File
@@ -1,177 +1,9 @@
# Spatial scheduling
`ScheduleAndRealizeSpatial` selects one scheduling strategy after constructing
the shared `ComputeGraph`. Pipeline scheduling does not run, require, or repair
a latency PEFT placement.
The [scheduling overview](../../../../../../../../docs/compiler/scheduling.md)
documents ownership, costs, capacity recovery, complexity, and links to the
normative invariants. See [implementation details](../../../../../../../../docs/compiler/scheduling-implementation.md)
for fragment views, receive forwarding, transfer rounding, and placement models.
## Ownership and flow
| Decision | Owner |
| --- | --- |
| Compute instances, dependency extraction, execution/transfer costs, resident-weight identities | `ComputeGraph` and `ComputeInstanceUtils` |
| Target configuration and physical link costs | Compiler target adapter, exposed through `SchedulingTarget` |
| Latency priorities, reservations, physical assignment, insertion gaps, per-core order | `PeftScheduler` |
| Capacity-driven additive workload splitting | `PipelineWorkloadPreparation` |
| Operation groups, stage adjacency and input constraints | `buildPipelineGroups` |
| Stage partitioning | `partitionPipelineGroups` |
| Stage core budgets | `assignPipelineStages` |
| Physical topology clustering | `PipelineCoreLayout` |
| Stage-local placement and resident-weight packing | `packPipelineStages` and `IndexedResidentWeights` |
| Schedule timing evaluation | `evaluateSchedule` |
| Publication-compatible core equivalence classes | `populateScheduleEquivalenceClasses` |
| Emission of the selected core order and homogeneous runs | Scheduled compute planning/materialization |
| Routing, local forwarding and transfer families | Deferred transfer planning |
| Deadlock-free blocking communication order | Deferred communication scheduling |
| READY/FREE counts, registers, overwrite boundaries and restart synchronization | Deferred boundary planning |
| Explicit communication and synchronization operations | Deferred boundary realization |
Both strategies produce `MergeScheduleResult`: physical core assignment,
contiguous per-core execution slots, estimated starts, last-task markers,
equivalence classes and cost summaries. Pipeline schedules also provide the
stage of every physical core. Graph identity identifies publications; it does
not order execution. Materialization sorts only by the verified scheduler slots
and combines only consecutive compatible graph instances.
Class lanes keep graph encounter order independently of physical core IDs,
preserving regular source-lane mappings without changing any core's execution
order. Complete singleton fragment publications use reshape views only with
matching static shapes, zero offsets, complete sizes and unit strides, including
scalar fragments. Other insertions retain their slice semantics. Receive forwarding
runs after copy normalization and can cross lookup-table loads, while writes
and synchronization remain barriers.
Physical fragment slots are resolved through `getGraphBatchPublicationMap`.
Direct statically evaluable batch projections can have many-to-many edges.
Unproven projections retain the complete producer dependency. Blueprint
dependencies belong to the shared graph, rather than a separate pipeline-only
dependency graph.
Sources reached through intermediate projections retain conservative producer
dependencies, including region captures. Graph construction reports malformed
publication metadata through normal pass failure.
Projected communication costs count the selected physical fragments through
the same publication map; graph-range overlap is not a transfer-volume estimate.
Conservative costs scale payload bytes before rounding memory/packet costs, so
small fragments retain endpoint and packet overhead. Conservative producer-chunk
payloads scale elements/bits before byte rounding, preserving one aggregate
chunk estimate; projected publication costs retain their per-fragment granularity.
Spatial sub-byte types reach analysis, while the current PIM byte-addressed
communication lowering rejects non-byte-sized element storage.
## Cost dimensions
Execution times and fixed transfer costs use integer nanoseconds. Hardware
cycles are converted using the corresponding target period. Bytes, elements,
flits, resident-weight slots and lane counts remain resource quantities.
Blueprint reconstruction charges memory work separately for each fragment;
small projected slices are not assumed to require a single copy. Full receives
that can write directly into assembly storage do not charge endpoint memory work
again on their edges. Later copy elimination can make this assembly estimate
conservative. Blueprint reconstruction uses target memory read/write costs; element counts
and cycles are not added to nanoseconds. Runtime-free views cost zero.
Runtime transposes charge target memory read/write traffic rather than generic
nested-operation fallback cost. Their shared lowering/run analysis charges rounded
memory accesses per run and vector execution for strided runs. The matched oracle
charges vector payload bytes, not the full strided address footprint. Instruction
setup/issue time and contention remain approximation boundaries.
Concatenation charges each input block separately
for each outer row, including memory-width rounding. Axis, size and target-period
scaling have focused coverage.
The evaluator distinguishes total serial work, maximum per-core work,
dependency critical path, summed edge transfer latency, single-inference
completion and individual stage spans. The dependency path excludes core-order
constraints; completion includes them. Summed communication is a work metric,
not elapsed time. Stage spans exclude cross-stage routing, restart/host-buffer
synchronization and network contention. They are scheduling proxies, not an
authoritative prediction of steady-state throughput. Simulator latency and
measured throughput period remain the performance gates.
Partitioning retains its successful ordered-packing plus dependency-tail
priority. This deliberately biases cuts away from long dependent groups and
is not reported as predicted execution time. Pipeline partitioning and packing
use nominal volume priorities: transpose traffic is weighted as a bulk movement,
preserving the measured heuristic without pretending that individual runs are
free. The per-operation priority adjustment is cached and scaled by chunk lanes.
PEFT and final schedule evaluation always use physical execution estimates.
Pipeline candidate placement compares
maximum per-core serial compute/incident-transfer demand, then stage completion
and compute load. Incident transfer latency is charged to each participating
core as a placement proxy; it is not added twice to predicted elapsed time.
This preserves useful parallelism that an arrival-time-only ranking can lose.
The final reported compute/incident-transfer demand uses physical execution
estimates, separately from the priority scores used to select the placement.
When serial-demand estimates are within 1%, stage span breaks the near tie;
this is a heuristic resolution rule, not a pipeline timing model. Core budgets
come from stage allocation and capacity recovery. A proposed additional one-core
load refinement was removed after a matched throughput regression; improving
the placement proxy did not establish a better execution period.
## Capacity and recovery
The distinct-weight union divided by per-core capacity is a provable lower
bound; an oversized task is individually infeasible. Ordered pack counts are
heuristic core-budget estimates for unrestricted placement. Their failure does
not prove hardware infeasibility. When these estimates exhaust the fleet, core
allocation restarts from the lower bounds and actual packing establishes
feasibility.
Latency reservations prioritize physical communication/load, or reuse under
high weight pressure. If graph-order reservations fail, a largest-weight-first
retry retains load-aware placement before using the shared reuse-first best-fit
constructor. The constructor establishes feasibility; its concentrated assignment
does not replace a successful load-aware reservation. Pipeline packing compares arrival-aware and load-aware placements;
ordered packing and the same shared constructor supply constructive recovery
assignments. Failed search budgets guide bounded core reallocation, but are not
resource lower bounds. Diagnostics distinguish a proved capacity limit from an
unsuccessful heuristic search. No exact combinatorial solver is used.
The shared constructor retries once in reverse order after greedy failure.
Only if both orders fail, it replays at most16 single-choice deviations with a
shared budget of65536 core-candidate checks, reusing its scratch buffers.
Successful existing assignments are preserved; exhausted repair is not proof of
infeasibility. PEFT propagates reservation failures to the scheduling pass for
a bounded diagnostic and normal pass failure.
## Complexity
Let `T` be tasks, `E` dependencies, `P` processors, `G` operation groups, `S`
stages, `R` resident-weight references, `W` distinct weights and `C` total modeled
work.
- Schedule evaluation takes `O((T + E) log T + P)` time with deterministic ready
ordering, and `O(T + E + P)` space.
- Resident-weight indexing retains `O(R W)` worst-case equality comparisons.
Lower bounds take `O(R + W)` time and `O(W)` bits. Constructive best fit takes
`O(T log T + P R + P W)` time, with `O(P W)` bits and `O(T + P)` additional
storage. It runs on failure paths rather than replacing successful placement.
Recovery adds at most17 constructions with the same complexity (one reverse
traversal plus16 single-choice replays); repair candidate checks are also
capped at65536. Scratch storage is reused, with one extra `O(T)` failed-assignment
snapshot only after both original attempts fail.
- The partition DP remains superlinear: up to
`O(S G² ((T + R) log C + W))` time for ordered segment estimates, plus group
dependency traversal, and `O(S G²)` cache space. This bounded existing search
is retained rather than replaced with global placement optimization.
- Physical topology clustering and cached PEFT center distances take `O(P²)`
time. Center lookups are constant time instead of repeating a full physical
topology scan for each task/candidate processor.
- Concatenation cost extraction takes `O(A D)` time and constant auxiliary space
for `A` inputs of rank `D`; it does not expand the outer rows or copy blocks.
- Graph execution costs are scaled by chunk lane counts. Pipeline priority
analysis adds two cost traversals per operation template and caches the
adjustment, taking `O(T + D)` time and `O(T + G)` space for `D` total template
body operations. It does not revisit template bodies for every chunk.
- Exact projection analysis visits each selected physical slot once per
consumer chunk and input use, with a producer-lane bitset for deduplication.
If `L` is producer lanes and `Q` is projected slot visits, it takes
`O(Q + L)` time and `O(L + Q)` temporary space; publication maps are cached
across all chunks. Cost extraction shares this traversal with dependencies.
- Conservative producer tracing visits each value/projection once per input,
including captured region operands, with linear time and space in that input's
projection DAG. Direct physical-batch edges retain their existing fast path.
- Pipeline mode removes a complete PEFT run, including its reservation and
insertion-gap searches. Latency mode retains those algorithms and their
existing potentially quadratic gap traversal.
Remaining approximation boundaries include operation issue overhead, eliminated
assembly copies, communication contention and pipeline synchronization. Weight
indexing and ready-transfer signature searches remain possible compiler
hotspots. Changes to these require matched execution measurements.
Implementation sources remain in this directory; project-wide compiler
navigation starts at [docs/README.md](../../../../../../../../docs/README.md).
tools/raptor_graph_explorer/ARCHITECTURE.md
+131
View File
@@ -0,0 +1,131 @@
# Graph Explorer architecture and API
The [README](README.md) covers installation, input headers, commands, and tests.
This reference describes the contracts maintained by the implementation.
## Ingestion and identity
[`discovery.py`](raptor_graph_explorer/discovery.py) recursively pairs reports
by their full filename prefix, including ZIP inputs, and rejects unsafe archive
paths. That prefix is `report_id`; unrelated reports never share endpoints.
[`ingest.py`](raptor_graph_explorer/ingest.py) streams CSVs in batches of 2,000
rows through temporary SQLite staging tables. Node IDs, operation IDs, lanes,
cores, and channels remain separate fields. Known node prefixes are `gc`,
`gcb`, `sc`, `scb`; unknown ones retain `unknown:<prefix>`. Explicit operation
and lane columns are authoritative. Blank operation IDs, and blank batch lanes,
may use a parseable source ID with a warning. Explicit edge lanes override
endpoint lanes, with contradictory values diagnosed. Missing numbers stay
SQL NULL; malformed rows are diagnosed and omitted, never coerced to zero.
Extra columns survive in `attributes_json`.
Duplicate node IDs retain the first source row and produce diagnostics. Bulk
joins resolve edges within their report; unknown endpoints are omitted with
bounded examples. Staging tables are dropped even on error. The ingestion
connection closes before indexes, aggregates, motifs, and diagnostics are
built. Secondary indexes are created after raw insertion. CSVs without extra
columns reuse an empty JSON representation.
Raw edge IDs use report ID plus original CSV row number; aggregate IDs use
report ID and grouping keys. Input filesystem paths and browser layout do not
define identity. The manifest stores versions and counts, not duplicate raw
rows or machine-specific input paths. Database schema compatibility is checked
before serving; incompatible outputs must be rebuilt.
## Aggregation and exact lane mappings
Operation nodes group by `(report_id, op_id)`, retaining kind counts, lane/core
cardinalities and ranges, and SSA summaries. Operation edges group by the
source/target operation pair, retaining counts, weights, tensor types, channels,
and exact mapping classification. Core aggregation has a distinct `unassigned`
group for missing cores. Node-kind aggregation is a small diagnostic view.
[`mappings.py`](raptor_graph_explorer/mappings.py) uses all normalized lane
pairs, in the following precedence order:
| Class | Meaning |
|---|---|
| `scalar_to_scalar` | Both lanes absent for every pair. |
| `scalar_to_batch` / `batch_to_scalar` | One side entirely absent, the other entirely present; range, cardinality, and contiguity recorded. |
| `identity` | One-to-one relation with equal lanes. |
| `constant_offset` | One-to-one with one nonzero target-minus-source offset. |
| `permutation` | Remaining one-to-one relation with equal cardinalities. |
| `broadcast` / `fan_in` | One source to many targets / many sources to one target. |
| `stencil` | Multiple sources share exactly the same target-offset set, containing 2–64 offsets. |
| `irregular` | Complete lane data matching none of the preceding forms. |
| `unknown` | Empty or partially missing relation outside the scalar/batch cases. |
SQL computes degrees/cardinalities; stencil comparison streams one operation
pair at a time. No sampling determines classification. Mapping rows remain
paginated; matrices show exact sparse points up to the configured threshold,
then bounded density bins.
## Display projection and layout
[`projection.py`](raptor_graph_explorer/projection.py) rebuilds the entire
display from the current expanded-operation set. Expanding replaces an
operation aggregate with its raw nodes, including isolated nodes. An aggregate
edge survives only while both endpoint operations are collapsed. Otherwise its
raw edges replace it: each endpoint is raw if its operation is expanded and
aggregate if collapsed. The complete set determines endpoints, so expansion
order cannot leave stale endpoints or duplicate aggregate/raw representations.
Raw view expands every operation, preserving each valid source edge once.
Operation ranks use the complete operation graph, including isolated nodes.
Collapsed operations occupy one row; expanded lanes use compact rows in numeric
order, followed by deterministic lane-less rows. Sparse lane numbers leave no
empty rows, and nodes sharing a lane get centered deterministic horizontal
offsets. Operation/raw views use stable server coordinates restored by Reset
layout. Core/node-kind views use ELK. Views flow top-to-bottom.
The [viewer](raptor_graph_explorer/static/index.html) first requests an
operation aggregate before applying the report's view selection. Raw selections
use raw details rather than aggregate-only endpoints. Operation labels use the
first SSA name, falling back to source identity; label collision handling and
bounded node sizes keep dense views usable. Dependencies are pinned in that
file (Graphology, Sigma.js, ELK); first browser use requires CDN access.
## Structural motifs
[`motifs.py`](raptor_graph_explorer/motifs.py) loads only the operation aggregate
into NetworkX. It condenses SCCs into a DAG, attaches synthetic source/sink
nodes, and computes dominators and reverse post-dominators. Branching entries
are paired with their immediate post-dominator; regions retain nodes reachable
before that exit which can reach it, rejecting external entrances/exits.
One shortest representative path per branch avoids enumerating all paths.
`exact_diamond` has two direct one-node branches; `rhomboid` has disjoint
branch interiors; `split_rejoin` covers other valid single-entry/single-exit
regions. Identical memberships are deduplicated; nested motifs remain.
SCC members map back to operation IDs, with deterministic representative
operations for multi-operation SCC endpoints/paths and `contains_cycle` markers.
Screen geometry never determines motif membership.
## Read-only API
All routes below use GET; see [`api.py`](raptor_graph_explorer/api.py).
Responses include `Cache-Control: no-store` to avoid mixing viewer revisions.
| Route | Result / relevant query parameters |
|---|---|
| `/api/manifest` | Tool/schema versions and report counts. |
| `/api/reports` | Available reports. |
| `/api/reports/{report_id}/summary` | Counts, mapping classes, diagnostics. |
| `/api/reports/{report_id}/graph/{level}` | Complete `operation`, `core`, or `node_kind` aggregate. |
| `/api/reports/{report_id}/display-graph` | Deterministic projection; `view=operation` or `raw`, repeated `expanded_op_id`, `expand_all`. |
| `/api/reports/{report_id}/motifs` | Structural motifs. |
| `/api/aggregate-nodes/{aggregate_id}` | Aggregate details; operation SSA names capped at 100. |
| `/api/aggregate-edges/{aggregate_edge_id}` | Aggregate edge details. |
| `/api/aggregate-edges/{aggregate_edge_id}/mapping` | Exact raw lane rows; `limit`, `offset`. Operation edges only. |
| `/api/aggregate-edges/{aggregate_edge_id}/matrix` | Exact sparse points or density bins. Operation edges only. |
| `/api/reports/{report_id}/raw-nodes` | Raw node page; `limit`, `offset`. |
| `/api/reports/{report_id}/raw-edges` | Raw edge page; `limit`, `offset`. |
| `/api/reports/{report_id}/subgraph` | `aggregate_id`, `direction=in/out/both`, `depth`, optional `expand_lanes`. |
`create_app` defaults: page size 100 with a maximum of 500; expansion cap 5,000;
exact matrix threshold 2,000 points; density grid 64×64. Subgraph depth defaults
to 1 and caps at 5. Oversized raw/expanded projections return HTTP 400 without a
partial graph. The projection cap counts raw nodes plus incident raw edges;
collapsed aggregate nodes/edges are additional. Aggregate endpoints have no
pagination or expansion cap, so very large operation graphs remain a scaling
limit. Raw CSV graphs are never loaded wholesale into NetworkX.
tools/raptor_graph_explorer/README.md
+73 -184
View File
@@ -1,228 +1,117 @@
# Raptor Graph Explorer
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
Python 3.13+ tool for exploring Raptor Spatial dataflow CSVs in a browser.
It streams reports into SQLite, aggregates operations/cores/node kinds, and
provides bounded raw-lane inspection. Compiler output remains unchanged.
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
## Install and run
## Architecture
```text
CSV pairs / directories / ZIP archives
-> recursive discovery and safe ZIP extraction
-> streaming CSV validation and SQLite staging
-> normalized raw_nodes and raw_edges
-> operation, core, and node-kind aggregates
-> exact lane mappings
-> SCC condensation and structural split/rejoin motifs
-> graph.sqlite + manifest.json
-> read-only FastAPI API
-> Sigma.js WebGL viewer with stable operation/lane layout
```
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
## Installation
From the Raptor repository root:
From the repository root, create a Python 3.13+ environment if `.venv` does not
exist:
```bash
python3.13 -m venv .venv
source .venv/bin/activate
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
```
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
## Commands
Build a persistent output:
For a new or existing environment, verify the version before installing. Select
a compatible environment if this check fails; do not reuse an older Python:
```bash
raptor-graph-explorer build \
validation/networks/vgg16/depth_07/raptor/reports \
--output /tmp/vgg_graph
.venv/bin/python -c 'import sys; assert sys.version_info >= (3, 13), "Python 3.13+ required"'
.venv/bin/python -m pip install -e 'tools/raptor_graph_explorer[dev]'
.venv/bin/raptor-graph-explorer build /path/to/raptor/reports --output /tmp/raptor_graph
.venv/bin/raptor-graph-explorer serve /tmp/raptor_graph --port 8765
```
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
Open `http://127.0.0.1:8765`. The server is read-only, has no authentication,
and is intended for local use. Browser dependencies load from pinned CDNs and
require network access on first use; preprocessing and inspection work offline.
Serve an existing output:
Other commands:
```bash
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
# Build and serve a directory, CSV pair, or ZIP archive.
.venv/bin/raptor-graph-explorer run /path/to/reports.zip --output /tmp/raptor_graph
# Print counts, mappings, motifs, and diagnostics.
.venv/bin/raptor-graph-explorer inspect /tmp/raptor_graph
```
Build and immediately serve:
`build` and `run` accept multiple input paths and discover reports recursively.
`--overwrite` replaces an existing output. `build --strict` rejects
data-integrity errors; otherwise valid pairs and unambiguous rows are retained.
Outputs are `graph.sqlite` and `manifest.json`. Rebuild outputs if the tool
reports an unsupported schema version.
```bash
raptor-graph-explorer run \
/mnt/data/conv_relu_conv.zip \
--output /tmp/conv_relu_conv_graph \
--overwrite
```
See [architecture and API guarantees](ARCHITECTURE.md) for identity, expansion,
mapping/motif analysis, ingestion constraints, and the endpoint reference.
Print counts, mapping classes, SCCs, motifs, and diagnostics:
## Input contract
```bash
raptor-graph-explorer inspect /tmp/vgg_graph
```
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
## Supported CSV reports
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial3_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial3_scheduled_no_comm.edges.csv`.
Node columns:
Pair files by the complete prefix before `.nodes.csv` and `.edges.csv`.
Required headers:
```csv
Id,op_id,lane,core,ssa_name
```
Edge columns:
```csv
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
```
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
Explicit operation IDs and lanes take precedence over parsed node IDs. Missing
numeric values remain missing; malformed values produce diagnostics. Extra
columns are preserved. Edges resolve only within their report; unknown
endpoints are diagnosed and omitted. Unsafe ZIP paths are rejected.
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
Graph lanes, physical fragment slots, scheduled lanes, cores, and channels are
distinct identities. Spatial3 reports contain explicit communication only;
isolated nodes are valid and no dependencies are inferred from SSA text.
## Output
## Views and analysis
```text
OUTPUT_DIRECTORY/
├── graph.sqlite
└── manifest.json
```
- **Operation:** aggregates by operation and supports lane expansion, exact
mapping details, and structural motifs.
- **Core / Node kind:** summarize placement and operation kinds.
- **Raw instances:** displays source nodes and valid edges, including isolated
nodes; spatial4 reports select this view automatically.
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
The viewer first loads an operation aggregate. Search, filters, expansion,
selection details, and layout controls support focused inspection. Raw and
expanded graphs cap raw nodes plus incident raw edges at 5,000; exceeding it returns an
error rather than a partial graph. Use an aggregate view or increase
`create_app(..., expansion_cap=...)` when the machine can handle it.
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
Lane mappings use all normalized pairs, without sampling. Classes include
scalar/batch relations, identity, constant offset, permutation, broadcast,
fan-in, stencil, irregular, and unknown. Small matrices show exact sparse
points; large ones use a bounded 64×64 density view.
## Aggregate views
Motifs identify structural single-entry/single-exit split/rejoin regions on the
operation graph's SCC condensation. `exact_diamond`, `rhomboid`, and
`split_rejoin` describe progressively broader forms; a diamond-shaped drawing
alone is not evidence of a motif. Cycles remain marked.
### Operation
## API and limits
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
Read-only FastAPI endpoints expose manifests, report summaries, aggregate
graphs/details, mappings, matrices, motifs, and raw pages. See
[`api.py`](raptor_graph_explorer/api.py) for routes and `create_app` limits.
Raw pages default to 100 rows and cap at 500; subgraph depth caps at five.
Aggregate graph endpoints return the complete aggregate, so very large
operation counts can still be expensive.
### Core
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
### Node kind
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
### Raw instances
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial4 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
## Exact lane mappings
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
- `scalar_to_scalar`: every pair has two absent lanes.
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
- `batch_to_scalar`: the corresponding source-lane case.
- `identity`: a one-to-one relation with equal source and target lanes.
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
- `broadcast`: one distinct source lane maps to multiple target lanes.
- `fan_in`: multiple source lanes map to one distinct target lane.
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
- `irregular`: complete lane data matches none of the exact forms.
- `unknown`: lane data is partially missing outside the scalar/batch cases.
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
## Structural motifs
Motifs are not inferred from rendered geometry. For each operation graph the tool:
1. computes strongly connected components and condenses them into a DAG;
2. attaches synthetic source and sink nodes;
3. computes dominators and reverse-graph post-dominators;
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
5. retains nodes reachable before the exit that can also reach the exit;
6. rejects unexpected entrances to, or exits from, internal nodes;
7. records one shortest representative path per branch without enumerating all paths;
8. maps condensed components back to operation aggregate IDs.
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
## Viewer and API
The viewer initially fetches an aggregate operation graph, so a previously selected raw view cannot make a new report exceed the display safety cap before the first render. Browser responses disable caching so HTML and JavaScript from different tool revisions cannot be mixed. Sigma renders the graph with WebGL. Controls cover report/view selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Edges use a fixed width. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Nodes without an SSA name fall back to their source identity. Mapping panels remain available for operation aggregate edges.
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
Operation ranks are calculated from the complete operation graph, including isolated operations, and every view flows top-to-bottom. A collapsed operation reserves one displayed row. Expanded nodes use compact rows for distinct lanes in numeric order, followed by deterministic lane-less rows, so sparse lane numbers create no empty geometric space. Raw nodes sharing a lane receive centered deterministic horizontal offsets. Operation and raw views use stable server coordinates, and `Reset layout` restores them; core and node-kind views use ELK, and `Rerun layout` runs it again. Operation labels use the first SSA name; Sigma's collision grid thins normal labels, and a label is skipped when its measured screen rectangle intersects another visible node. Selected or hovered nodes keep the original white bubble with black text. Selecting a motif frames its member nodes on the first click. Aggregate-node sizes remain bounded.
Browser dependencies are pinned in one place, `static/index.html`:
- Graphology 0.25.4;
- Sigma.js 2.4.0;
- ELK.js 0.11.1.
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
Read-only endpoints include:
```text
/api/manifest
/api/reports
/api/reports/{report_id}/summary
/api/reports/{report_id}/graph/{level}
/api/reports/{report_id}/display-graph
/api/reports/{report_id}/motifs
/api/aggregate-nodes/{aggregate_id}
/api/aggregate-edges/{aggregate_edge_id}
/api/aggregate-edges/{aggregate_edge_id}/mapping
/api/aggregate-edges/{aggregate_edge_id}/matrix
/api/reports/{report_id}/raw-nodes
/api/reports/{report_id}/raw-edges
/api/reports/{report_id}/subgraph
```
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
## Performance behavior
- CSV readers stream rows and insert them in bounded batches.
- Temporary SQLite staging tables are function-scoped and dropped immediately; bulk joins resolve endpoints without per-edge node queries.
- CSVs without additional columns use a constant empty JSON representation.
- Ingestion closes its write connection before derived indexes, aggregation, motifs, and diagnostics reopen the database.
- Secondary indexes are created after raw insertion and focus on serving and aggregation queries.
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
- Only aggregate operation nodes/edges enter NetworkX.
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
- API raw pages, display projections, expansions, and matrices have hard limits.
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
## Tests and troubleshooting
```bash
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
.venv/bin/python -m pytest tools/raptor_graph_explorer/tests
RAPTOR_GRAPH_SLOW=1 .venv/bin/python -m pytest \
tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
```
## Current limitations
The opt-in test generates 100 operations, 1,000 lanes, and 100,000 raw edges.
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
## Troubleshooting
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
- **Output already exists:** pass `--overwrite` only when replacement is intended.
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
No Raptor build or compiler modification is required.
- **No complete pairs:** check matching prefixes and both CSV suffixes.
- **Strict build failed:** inspect diagnostics from a build without `--strict`.
- **Raw/expanded display returns 400:** use aggregation or adjust the cap.
- **Blank viewer with working APIs:** check browser CDN access and console.
- **No motifs:** valid when no structural split/rejoin region exists.
validation/README.md
+87 -219
View File
@@ -1,265 +1,133 @@
# Raptor validation
`validate.py` validates every ONNX model below a selected directory. For each
model it can:
1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile Pim artifacts with Raptor;
4. run the reference implementation and functional Pim simulator;
5. compare their outputs;
6. run Pimsim to report latency, throughput, power, and energy.
Run the script from the repository root with the repository Python environment.
`validate.py` discovers ONNX models recursively, compiles a native ONNX-MLIR
reference and Raptor artifacts, compares functional simulator outputs, and
optionally measures Pimsim latency, throughput, power, and energy.
Run from the repository root using `.venv`.
## Prerequisites
- A built Raptor compiler, normally
`build_release/Release/bin/onnx-mlir`.
- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
- The Rust toolchain used by the functional simulator.
- The Rust functional simulator under
`backend-simulators/pim/pim-simulator`, unless overridden.
- A built `pimsim-nn` under
`backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
skipped or its path is overridden.
- Built release Raptor and ONNX-MLIR runtime headers.
- `numpy`, `onnx`, and `colorama` installed in the repository environment.
- Rust toolchain and `backend-simulators/pim/pim-simulator`.
- Built `backend-simulators/pim/pimsim-nn/build`, unless performance
simulation is skipped.
## Basic usage
See the [build guide](../README.md#build).
Validate the complete operation suite:
## Usage
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations \
--verbose \
--raptor-extra-arg=--pim-detect-communication-deadlock \
--raptor-extra-arg=--pim-export-spatial-dataflow=none
--raptor-extra-arg=--pim-detect-communication-deadlock
```
Validate one operation category or case:
Run the complete operations suite outside the sandbox so parallel workers can
start. For a focused case, change `--operations-dir` to
`validation/operations/gemm/small` or a network directory such as
`validation/networks/yolo11n/depth_04`.
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small
```
Defaults run latency (one input) and throughput (pipeline 4, four distinct
inputs). Both reuse reference artifacts, and every throughput output is
compared with its own reference. Models run with four workers by default.
Results go to `validation_results.csv` in the selected model tree.
Validate a network or network slice:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/networks/yolo11n/depth_04
```
`--operations-dir` may point to any directory tree containing `.onnx` files.
The script discovers them recursively and writes `validation_results.csv` in
that directory while retaining separate latency and throughput terminal tables.
## Pim validation tools
- [Pimcomp model suite](networks/pimcomp_models/README.md)
- [Pimcomp model comparison tools](tools/pim/pimcomp/compare/README.md)
- [Pimcomp correctness study](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation study](tools/pim/ablation/README.md)
## Validation modes
The default mode runs latency and throughput in one validation job. Latency
uses one input, while throughput uses `--pipeline=4` with four distinct inputs.
Both modes reuse the generated input batch, native runner, and reference
outputs, and every throughput output is compared with its own reference.
Use `--compile-only` to build the reference runner and Pim artifacts without
executing either implementation:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small \
--compile-only
```
Use `--run-only` to reuse those artifacts and perform input generation,
reference execution, simulation, and comparison:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small \
--run-only
```
`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
with a diagnostic if its required compiled artifacts are missing.
## Parallel execution and output
Models run in parallel using all available CPUs by default. Set the worker
count with `-j` or `--jobs`:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations \
--jobs 8
```
Append `--compile-only` to build artifacts without execution, then
`--run-only` to reuse them for input generation, execution, and comparison.
These flags are mutually exclusive; run-only fails if compiled artifacts are
missing. `--clean` removes generated artifacts and stale locks, and requires
neither compiler nor include paths.
## Options
| Option | Description |
Use `--help` for the full command reference.
| Option | Purpose / default |
|---|---|
| `-h`, `--help` | Print command help and exit. |
| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
| `--non-functional-simulator-build-dir PATH` | Pimsim build directory. Defaults to the in-tree build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
| `--skip-non-functional-simulation` | Skip Pimsim latency, throughput, power, and energy measurement. |
| `--no-fast` | Disable fast throughput convergence for authoritative full-duration Pimsim experiments. |
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
| `--core-count INT` | Pim core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
| `--clean` | Remove generated validation artifacts and exit. |
| `--compile-only` | Compile reference and Pim artifacts without execution or comparison. |
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average Pim pass timings. |
| `--raptor-path PATH` | Compiler binary; required except for cleanup. |
| `--onnx-include-dir PATH` | Runtime headers; required except for cleanup. |
| `--operations-dir PATH` | Model tree; default: `validation/operations`. |
| `--simulator-dir PATH` | Override the in-tree functional simulator crate. |
| `--non-functional-simulator-build-dir PATH` | Override the in-tree Pimsim build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Performance profile; default: Arch-A. |
| `--core-count N`, `--crossbar-count N`, `--crossbar-size N` | Raptor resources; defaults: Arch-A (168, 96, 128). |
| `--skip-non-functional-simulation` | Skip performance measurement. |
| `--no-fast` | Use full-duration throughput measurement. |
| `--threshold FLOAT`, `--relative-threshold FLOAT` | Output tolerances; defaults: `1e-3`, `1e-5`. |
| `--seed N` | Input RNG seed; default: 0. |
| `--raptor-extra-arg=ARG` | Forward compiler arguments; repeat as needed. |
| `--command-timeout-seconds FLOAT` | Subprocess timeout; default: 1000000 seconds. |
| `-j N`, `--jobs N` | Parallel workers; default: 4. |
| `--compile-only`, `--run-only`, `--clean` | Compile, reuse, or remove artifacts. |
| `--verbose` | Include passing subprocess logs and average Pim pass timings. |
Arguments beginning with `--` that are passed through to Raptor should use the
equals form:
Forwarded options beginning with `--` require the equals form, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.
```bash
--raptor-extra-arg=--pim-detect-communication-deadlock
```
## Performance and result interpretation
## Hardware profiles and non-functional simulation
The selected [hardware profile](networks/pimcomp_models/README.md#hardware-profiles)
must match the core/crossbar flags. A mismatch disables performance simulation
and prints the incompatible values; functional validation still runs.
Canonical configs and meshes live under `validation/pimsim_configs/pimcomp/`.
The selected Pimcomp profile must match `--core-count`, `--crossbar-count`, and
`--crossbar-size`. A mismatch disables only non-functional simulation and
prints the incompatible values; functional validation still runs.
Throughput defaults to `pimsim-nn --fast` with a 1000 ms convergence deadline.
It compares consecutive two-round windows at 1% tolerance and falls back to
full-duration results if convergence fails. Use `--no-fast` for authoritative
experiments. See [metric definitions](../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, power, and energy interpretation.
The checked-in profiles are under
`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
`latency_config.json`, while throughput uses
`throughput_config_<time>ms.json` and the mesh beside it.
Functional compilation/output comparison determines PASS/FAIL. Performance
failures show `ERROR` without changing a functional PASS; explicit unsupported
operations show `UNSUPPORTED`. Pimsim's `vsoftmax` is unsupported.
Skipped measurements remain visibly skipped.
Throughput measurement defaults to `pimsim-nn --fast` with a 1000 ms
convergence deadline. Fast mode compares consecutive two-round windows with a
fixed 1% tolerance and falls back to the legacy full-duration result if it
does not converge. Use `--no-fast` for authoritative experiments.
Exit codes: `0` for all models passing or successful cleanup; `1` for validation
failure, an invalid model directory, or no ONNX files; `2` for invalid arguments.
Use `--skip-non-functional-simulation` when latency, throughput, power, and energy are not required.
The summary reports non-functional results as measured, failed, unsupported, or
skipped.
## Artifacts and suites
Overall PASS/FAIL is determined by compilation and functional output
comparison. A non-functional simulation failure remains visible as `ERROR` in
the corresponding latency, throughput, power, or energy columns but does not
change a functional PASS.
Pimsim does not currently implement the `vsoftmax` instruction. When its
explicit unsupported-op diagnostic is encountered, Softmax validations retain
their functional PASS and show `UNSUPPORTED` in the non-functional columns.
Other Pimsim failures remain `ERROR`.
## Generated artifacts
Generated files are grouped below an `artifacts/` directory beside each model
or operation case. This keeps checked-in ONNX files and generated trees
separate and lets `--clean` remove the complete workspace, including stale
validation lock files:
The regular validator uses these paths beside each model:
| Path | Contents |
|---|---|
| `artifacts/inputs.csv` | Generated inputs, one batch entry per line. |
| `artifacts/inputs/`, `artifacts/outputs/`, `artifacts/runner/` | Inputs, reference outputs, and the runner shared by latency and throughput validation. |
| `artifacts/raptor/pim/`, `artifacts/simulation/latency/` | Latency Pim artifacts and functional simulator outputs. |
| `artifacts/raptor/throughput/pim/`, `artifacts/simulation/throughput/` | Pipeline-4, batch-4 throughput Pim artifacts and functional simulator outputs. |
| `artifacts/common/inputs/` | Shared generated input CSV files. |
| `artifacts/common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
| `artifacts/common/runner/` | Shared reference runner source, build tree, and library. |
| `artifacts/<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and Pim artifacts. |
| `artifacts/<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
| `artifacts/<arch>/<mode>[/pipelineN]/pimcomp/` | Pimcomp graph, instruction, simulator, and comparison-report artifacts. |
| `artifacts/inputs.csv` | Input batch manifest. |
| `artifacts/{inputs,outputs,runner}/` | Inputs, native reference outputs, and runner. |
| `artifacts/raptor/pim/` | Latency compiler artifacts. |
| `artifacts/raptor/throughput/pim/` | Pipeline-4 throughput artifacts. |
| `artifacts/simulation/{latency,throughput}/` | Functional simulator outputs. |
Each comparison's `raptor/` directory may include `spatial0.mlir`,
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.
The [comparison runner](tools/pim/pimcomp/compare/README.md) has separate
architecture/pipeline directories and shared `artifacts/common/` references.
Remove these artifacts for every discovered model with:
- [Operation cases and generation](operations/README.md)
- [Pimcomp models, provenance, and hardware](networks/pimcomp_models/README.md)
- [Pimcomp correctness experiments](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation](tools/pim/ablation/README.md)
```bash
.venv/bin/python validation/validate.py \
--operations-dir validation/operations/gemm/small \
--clean
```
`--clean` does not require `--raptor-path` or `--onnx-include-dir`.
## Available suites
Checked-in validation networks under `validation/networks/` include `vgg16`,
`yolo11n`, and `yolo11nv2`.
The generated operation inventory is documented in
[`operations/README.md`](operations/README.md). Regenerate its models with:
```bash
.venv/bin/python validation/operations/gen_tests.py
```
Other network suites under `validation/networks/` include `vgg16`, `yolo11n`,
and `yolo11nv2`.
## Manual functional simulator tracing
After validation has produced an `artifacts/raptor/pim/` directory, rerun the functional
simulator with tracing from its crate directory:
After validation produces Pim artifacts, run from the simulator crate:
```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
--package pim-simulator --bin pim-simulator -- \
-f /path/to/workspace/artifacts/raptor/pim \
-o /path/to/workspace/artifacts/simulation/out.bin \
-f /path/to/artifacts/raptor/pim \
-o /path/to/artifacts/simulation/out.bin \
-d <addr0>,<size0>,<addr1>,<size1>,... \
--mode latency \
--batch-size 1 \
--input-dir /path/to/workspace/artifacts/simulation/inputs
--mode latency --batch-size 1 \
--input-dir /path/to/artifacts/simulation/inputs
```
Throughput mode additionally requires `--batch-size N` and at least `N`
`input_<index>.bin` files in `--input-dir`. Each input binary concatenates the model tensors in graph
input order. The comparison validator also writes one native reference and one
`simulation/*_iterations/output_*.bin` dump per batch entry, and checks every
entry rather than only the final output.
Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
validator normally derives the `-d` address and byte ranges from
`raptor/pim/config.json` and the model output shapes.
## Results and exit status
The final table reports latency and throughput functional pass/fail state plus
non-functional latency, throughput, power, and energy. The summary includes pass/fail totals, non-functional simulation
counts, total measured latency, and average Pim pass timings when `--verbose`
is enabled.
- Exit status `0`: all discovered models passed, or cleanup completed.
- Exit status `1`: validation failed, the model directory was invalid, or no
`.onnx` models were found.
- Exit status `2`: command-line arguments were invalid or required arguments
were missing.
Derive output address/byte ranges from `raptor/pim/config.json` and model output
shapes. Traces appear as `TraceCore0`, `TraceCore1`, etc. beside `out.bin`.
Throughput needs `--mode throughput --batch-size N` and at least `N`
`input_<index>.bin` files, each concatenating tensors in graph-input order.
The comparison validator records and checks one output per batch entry.
validation/networks/pimcomp_models/README.md
+65 -345
View File
@@ -1,102 +1,36 @@
# Pimcomp comparison models
This directory contains the four networks evaluated in
[Pimcomp: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
additional compiler comparison model.
See the runner-generated [results_comparison.csv](results_comparison.csv) for the current comparison
results. Rows are retained separately for each model, architecture, mode, and
pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
the Raptor and Pimcomp artifacts; rows without a generated report contain `NA`.
Use `--archs` to select the architecture rows to generate; existing rows for
other architectures remain unchanged.
Models for comparing Raptor with
[Pimcomp](https://arxiv.org/pdf/2411.09159): the paper's four networks plus YOLO11n.
Use the [comparison runner](../../tools/pim/pimcomp/compare/README.md) for commands,
options, artifact locations, and plots.
The [reproduction reference](../../../docs/evaluation/pimcomp-reproducibility.md) records exact supported/ignored
operations, standalone commands, configuration layout, and GA/OpenMP controls.
## Models and provenance
| Directory | Model | Input | Provenance |
|--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`. |
| `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as Pimcomp's frontend does. |
| `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). |
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [Pimcomp VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
| `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
| Directory | Model / input | Source |
|---|---|---|
| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. |
| `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. |
| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12). |
| `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. |
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Derived from `../yolo11n/depth_51/yolo11n_depth_51.onnx`. |
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit Pimsim-ready GoogLeNet model.
It removes the two LRN nodes and terminal Softmax from the original model,
so that the comparison covers only operations scheduled by Pimcomp and supported by Pimsim.
The paper's checkpoints were not published. The three Model Zoo networks
produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is
suitable for compiler/simulator comparison, not paper-accuracy reproduction.
The paper's MNIST description for ResNet-18 differs from the released graph's
ImageNet input shape.
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit Pimsim-ready YOLO11n model.
It removes the Softmax nodes from the original model,
so that the compiled artifact can be simulated in Pimsim.
Dedicated performance models change the original computation:
## Unsupported and ignored operations
- `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal
Softmax, which Pimcomp does not schedule.
- `yolo11n/yolo11n-pimsim-nn.onnx` removes two Softmax nodes unsupported by
Pimsim. Use the original YOLO model for functional validation.
Pimcomp's frontend accepts exactly these ONNX operations:
```text
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
```
`Constant` is consumed as frontend metadata rather than emitted as a Pimcomp
node. Every other ONNX operation is unsupported: the frontend prints
`operation: <type> not considered` and stops at the first occurrence. Thus the
complete unsupported set is the complement of the allowlist above for the
model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and
`Resize` nodes.
Pimcomp's low-latency scheduler, hierarchy mapper, and genetic algorithm use
this complete explicit no-consider set:
```text
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
Softmax, Squeeze, Transpose
```
Those nodes do not receive scheduled latency instructions when they remain in
the backend graph. Before that point the frontend may fuse BatchNormalization
and activation nodes into Conv/Gemm, convert a
Reshape-Transpose-Reshape channel-shuffle pattern to `OP_SHUFFLE`, remove a
specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its
consumer. These transformations do not make an otherwise standalone ignored
operation timed.
Pimsim consumes Pim ISA instructions. It supports every named opcode
in the shared serialized range except `vsoftmax` (opcode 21),
which is rejected explicitly in both JSON and binary input. It
silently ignores no opcode; unknown names and numbers are errors.
These boundaries explain the dedicated artifacts:
- GoogLeNet's two LRN nodes and terminal Softmax perform real computation but
are ignored by Pimcomp, so the common latency artifact removes them. Its
inference Dropout and shape-only Reshape can remain without adding compute.
- YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can
run in Pimsim. Every other node, including MatMul, Transpose, and the
final detection-decoding tail, remains present and timed by Raptor. No
Pimcomp latency is reported because its frontend stops at `Split` and also
lacks `Resize`; compiling that prefix would not represent YOLO11n.
The authoritative lists are in
[`frontend.py`](../../../third_party/PIMCOMP-NN/frontend/frontend.py),
[`ElementPipelineSchedule.cpp`](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
[`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and
[`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp).
The Pimcomp authors did not publish the ONNX checkpoints used by the paper.
Running Pimcomp's frontend on the three Model Zoo files above produces JSON
graphs exactly equal to Pimcomp's bundled `resnet18.json`, `resnet34.json`,
and `googlenet.json`.
There is no VGG-8 artifact in the ONNX Model Zoo or any Pimcomp revision.
The included VGG-8 therefore has deterministic random weights and is suitable
for compiler and simulator comparison, not paper-accuracy reproduction. The
paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the
published Pimcomp graphs and ResNet Model Zoo artifacts use ImageNet shapes.
Current SHA-256 checksums:
Model SHA-256 checksums:
```text
788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx
@@ -107,275 +41,61 @@ a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/goog
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
```
## Paper hardware profiles
## Hardware profiles
The files in
[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
encode Table V's explicit resource parameters.
Each profile subdirectory contains pre-generated latency and throughput
Pimsim configs plus its matching mesh; comparison and validation reference
these canonical artifacts directly.
Canonical configs and meshes are in
[`validation/pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp).
They encode Table V's resources:
| Config | Cores | Crossbars/core | Crossbar | Cell | Pimcomp layout |
|------------------------------|------------------:|---------------:|------------|------:|-----------------|
| `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
| `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
| `arch-c/latency_config.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` |
| Profile | Cores | Crossbars/core | Crossbar | Mesh |
|---|---:|---:|---|---|
| Arch-A | 168 | 96 | `128x128` | `12x14` |
| Arch-B | 138 | 128 | `128x128` | `6x23` |
| Arch-C | 64 (16 chips × 4) | 8 | `512x1024` | Flattened `8x8` |
`adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
The paper does not give a two-dimensional core topology for Arch-A/B, so the
factorizations above preserve core count but cannot reproduce unpublished NoC
placement details. Released Pimcomp has no chip-count field; Arch-C is
therefore flattened to 64 cores and does not model chip boundaries.
All profiles use 2-bit cells and 16 ADCs for 16-bit fixed-point weights.
Arch-A/B topology factorizations preserve core counts; the paper does not
publish placement details. Arch-C does not model chip boundaries, and Raptor's
square-crossbar option cannot match its rectangular arrays. Do not change
hardware resources to bypass this limitation.
The remaining latency and power values come from Pimcomp's released default
configuration. Consequently, instruction/resource comparisons are
reproducible, but absolute paper power and energy numbers are not.
Other timing/power parameters use Pimcomp's released defaults. Resource and
instruction comparisons are reproducible; absolute paper power/energy values
are not.
## Build and validate the ONNX files
## Build and run
From the Raptor repository root:
From the repository root, after the [Raptor build](../../../README.md#build):
```bash
.venv/bin/python -m pip install -r requirements.txt
cmake --build ./build_release
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
.venv/bin/python -c \
'from pathlib import Path; import onnx; [onnx.checker.check_model(onnx.load(p)) for p in Path("validation/networks/pimcomp_models").glob("*/*.onnx")]'
```
Do not build either project with `ninja` directly.
## Compile with Pimcomp
Pimcomp reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
select one paper profile, and restore it when the shell exits:
```bash
RAPTOR_ROOT=$PWD
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
PIMCOMP_CONFIGS="$RAPTOR_ROOT/validation/pimsim_configs/pimcomp"
CONFIG_BACKUP=$(mktemp)
cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json"
```
The Model Zoo files map exactly to Pimcomp's bundled model names, so compile
them directly:
```bash
cd "$PIMCOMP/build"
# High-throughput mode; the paper evaluates batches of 128 samples.
./PIMCOMP-NN -m=resnet18 -r=balance -p=batch -o=YES -v=YES -s=YES
./PIMCOMP-NN -m=resnet34 -r=balance -p=batch -o=YES -v=YES -s=YES
./PIMCOMP-NN -m=googlenet -r=balance -p=batch -o=YES -v=YES -s=YES
# Low-latency mode; the paper uses batch size 1.
./PIMCOMP-NN -m=resnet18 -r=balance -p=element -o=YES -v=YES -s=YES
./PIMCOMP-NN -m=resnet34 -r=balance -p=element -o=YES -v=YES -s=YES
./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES
```
VGG-8 first needs Pimcomp's JSON frontend. Use a temporary ONNX copy because
the released frontend rewrites the input batch dimension in place:
```bash
cd /path/to/Raptor
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx /tmp/vgg8-pimcomp.onnx
.venv/bin/python third_party/PIMCOMP-NN/frontend/frontend.py \
--model_path /tmp/vgg8-pimcomp.onnx \
--save_path third_party/PIMCOMP-NN/models/JSON/vgg8_paper_reconstructed.json
cd third_party/PIMCOMP-NN/build
./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=batch -o=YES -v=YES -s=YES
./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES
```
Repeat after selecting `arch-b/latency_config.json` and
`arch-c/latency_config.json`. All four models were
compiled successfully in both modes with all three configs. The released
random placement code occasionally segfaults; an unchanged retry succeeded in
the observed cases.
The paper's optimizer uses a genetic algorithm with population 200 and up to
1000 iterations. The checked-out Pimcomp submodule already has both paper
settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness
evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS`
to control its parallelism; otherwise OpenMP uses the available CPUs. The GA
uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.
## Compare Raptor and Pimcomp
The comparison driver uses one random input and one native ONNX-MLIR reference,
compiles both instruction streams, runs both through Pimsim, runs
functional validation through `pim-simulator`, and writes Markdown and JSON
reports.
To reproduce the default `arch-a`/`arch-b` architectures and both
latency/throughput modes, use the model-by-model runner. Use `--archs` to
specify a different architecture set. It verifies the paper GA settings, expects Raptor
and the existing `third_party/PIMCOMP-NN/build` tree to already be built, then
runs the comparisons in parallel and regenerates `results_comparison.csv` from the JSON
reports:
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```
Use `--archs arch-a --mode latency` for only the Arch-A latency experiment, or
`--archs arch-a arch-b arch-c` to run all three architectures.
The runner expects existing Pimcomp and Pimsim builds, verifies the paper's GA
settings (population 200, up to 1000 iterations), and uses isolated frontend
workspaces without changing the source models or shared Pimcomp configuration.
Default aggregate results are written beside the
[comparison runner](../../tools/pim/pimcomp/compare/results_comparison.csv);
explicit custom output roots retain their own aggregate CSVs.
Each model directory has a shared ignored `artifacts/common/` directory containing
`inputs/`, `outputs/`, and the native `runner/`.
Pimsim configs and network meshes remain canonical under
`validation/pimsim_configs/pimcomp/` and are referenced in place.
Architecture- and pipeline-specific `raptor/`, `simulation/`, and Pimcomp
artifacts remain under each `artifacts/<arch>/<mode>[/pipelineN]/` comparison
directory; ablation variants use
`artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`. Pimcomp outputs are prepared
once per model/architecture/mode and linked into the other pipeline directories;
`comparison_report.{md,json}`
live under its `pimcomp/`. The frontend regenerates
one isolated `models/JSON/` graph because Pimcomp requires that relative
layout; it is removed after a successful backend run and the shared submodule
model directory is never modified. The original ONNX model is passed to the
frontend unchanged.
Pimcomp's source tree and build directory remain unchanged at runtime. Use
`--models vgg8` to run one model, `--mode throughput` to select one mode,
`--pipeline 4` to select one throughput pipeline, `--only raptor` or
`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
to inspect every command, or `--out-dir PATH` to keep results outside
`validation/`. Use `--clean` to remove generated comparison artifacts and
summaries, including stale reference lock files. Selecting a subset replaces only those comparison rows and
recomputes the aggregate `results_comparison.csv`; missing shared inputs, outputs, or the
reference runner are generated even for an isolated run. Use `--jobs 4` to cap
parallel comparisons. The per-stage timeout is unlimited by default; pass a
positive `--timeout-seconds` value to impose one.
Throughput comparisons default to `pimsim-nn --fast` with a 1000 ms
convergence deadline. Add `--no-fast` for authoritative full-duration runs.
Pimcomp receives the original ONNX model, and its frontend applies native
BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
The runner continues after a failed model so all reports are produced.
## Correctness and precision limits
The known Pimcomp batch-scheduling correctness issue and a reproducible
reference-intermediate prefill experiment are documented in
[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
Pimcomp does not support YOLO11n's `Split` and `Resize`; no Pimcomp performance
number is reported for that model. Accepted frontend operations can also be
ignored by scheduling; see the authoritative
[frontend](../../../third_party/PIMCOMP-NN/frontend/frontend.py) and
[latency scheduler](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp).
Known [batch-scheduling and BatchNorm gaps](../../tools/pim/pimcomp/correctness/README.md)
must remain visible as failed functional validation.
Arch-A low-latency example:
Functional validation uses FP32 programs in the Rust simulator against native
ONNX-MLIR. Pimcomp's Rust export expands byte-sized storage/transfers to FP32.
For performance comparison, Pimcomp's original `SimulationInfo.gz` remains
unchanged; a separate Raptor `raptor/pimsim_nn/` view scales FP32 storage and
transfers to one byte per element with `setbw 8, 8`. These performance artifacts
are not numerically quantized programs and must not be used for correctness.
```bash
RAPTOR_ROOT=$PWD
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py" \
--model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/artifacts/arch-a/latency" \
--pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
--core-count 168 \
--crossbar-count 96 \
--crossbar-size 128 \
--mesh-rows 12 \
--mesh-cols 14 \
--pimsim-mode latency \
--pimcomp-pipeline element
```
Use the same command with
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released Pimcomp cannot
compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
as YOLO11n would produce a misleading latency, so no Pimcomp number is
reported for this model.
For Arch-A high throughput, use `--pimsim-mode throughput
--pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a
`6x23` mesh, and
`validation/pimsim_configs/pimcomp/arch-b/latency_config.json`.
If only semantic and instruction comparison is required, add
`--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both
semantic validations with maximum output differences below `5e-10`.
The comparison runner enables `--fail-on-error`, so a failed compiler,
simulation, or semantic validation makes the command fail while preserving the
generated report.
### Numeric precision and simulator artifacts
The functional and non-functional simulators intentionally consume different
artifacts:
- Raptor and Pimcomp are validated against the native ONNX-MLIR reference as
FP32 programs in the Rust simulator. Raptor's emitted program is already
FP32. The Pimcomp-to-Rust export expands its element-addressed storage and
byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector
`imm_len` fields as element counts.
- Pimcomp's original `SimulationInfo.gz` is copied unchanged for Pimsim.
Pimcomp hardcodes `setbw 8, 8` and one byte per element without performing
numerical quantization; this artifact is used only for latency estimation.
- Raptor's original FP32 artifact remains unchanged for functional validation.
A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its
byte-addressed storage and transfer sizes from four bytes to one byte per
element. Vector `imm_len` fields remain element counts. Like Pimcomp's
artifact, this view is not numerically valid and is used only for a fair
non-functional comparison.
The ISA defines vector lengths in elements, while `ld`, `st`, `lldi`, `lmv`,
`send`, `recv`, addresses, and non-vector offsets are byte-based. Do not use
either latency-only artifact for semantic validation.
Current Raptor status:
- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
It removes the two LRN nodes and terminal softmax that Pimcomp does not
schedule.
- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
`512x1024` arrays can therefore be compiled by Pimcomp but not compared
exactly with Raptor.
Do not change the hardware profile to bypass either limitation; that would no
longer be a paper-matched comparison.
## Monolith fallback
The local `monolith` SSH alias points to the high-memory host. Copy only this
suite and the comparison driver; `-L` materializes the ResNet-18 symlink because
the canonical `resnet18/depth_68/resnet18_depth_68.onnx` file may not exist remotely:
```bash
REMOTE_REPO=/home/gmagnani/Project/Raptor
rsync -azL validation/networks/pimcomp_models/ \
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
rsync -az validation/pimsim_configs/pimcomp/ \
"monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
rsync -az validation/tools/pim/ \
"monolith:$REMOTE_REPO/validation/tools/pim/"
rsync -az --exclude=.git --exclude=build --exclude=output \
third_party/PIMCOMP-NN/ \
"monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
```
Then use the same commands over SSH:
```bash
ssh monolith
cd /home/gmagnani/Project/Raptor
# One-time setup if the repository virtual environment is absent.
python3 -m venv .venv
.venv/bin/python -m pip install numpy onnx onnxruntime colorama
# Run every configured comparison in parallel.
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```
Copy reports back without transferring large compiler artifacts:
```bash
rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \
validation/networks/pimcomp_models/
```
Vector `imm_len` is an element count. Addresses and transfer sizes for
`ld`, `st`, `lldi`, `lmv`, `send`, and `recv` are byte-based.
validation/operations/INVENTORY.md
+286
View File
@@ -0,0 +1,286 @@
# Operation test inventory
All 177 checked-in ONNX models, grouped by validator category. Each row describes
the regression or operation feature exercised; follow its link for the model.
[Generation and validation commands](README.md) remain in the short README.
[gen_tests.py](gen_tests.py) defines generated shapes, attributes, constants, and
seeds. It regenerates 160 cases; 17 fixtures are checked-in only: the five Gemv
cases, `mul/after_conv_scalar_constant`, and eleven Conv cases (the six
`strategy_*` cases plus `input_224_7x7_stride2`, `kernel_2x2`,
`output_channel_grouping_minimal`, `relu_conv_store`, `with_constant`).
Do not infer the complete inventory from generator output alone. Strategy
fixtures target particular lowering paths; actual selection depends on the
hardware profile and compiler flags.
## Add (5) — `add/`
| Case | Description |
|---|---|
| [`after_gemm`](add/after_gemm/add_after_gemm.onnx) | Gemm followed by Add with a broadcast bias vector. |
| [`basic`](add/basic/add_basic.onnx) | Elementwise Add on two inputs with identical shapes. |
| [`broadcast_row`](add/broadcast_row/add_broadcast_row.onnx) | Elementwise Add with row-vector broadcasting. |
| [`channel_broadcast_1024`](add/channel_broadcast_1024/add_channel_broadcast_1024.onnx) | NCHW per-channel broadcasting over 1024 channels. |
| [`leading_dimension_broadcast`](add/leading_dimension_broadcast/add_leading_dimension_broadcast.onnx) | Trailing-dimension broadcasting across leading dimensions. |
## Concat (3) — `concat/`
| Case | Description |
|---|---|
| [`channel_axis`](concat/channel_axis/concat_channel_axis.onnx) | Concatenates two runtime NCHW tensors along the channel axis. |
| [`negative_axis`](concat/negative_axis/concat_negative_axis.onnx) | Concatenates tensors using a negative axis. |
| [`three_inputs_channel_axis`](concat/three_inputs_channel_axis/concat_three_inputs_channel_axis.onnx) | Concatenates three runtime NCHW tensors along the channel axis. |
## Conv (42) — `conv/`
| Case | Description |
|---|---|
| [`batch_2`](conv/batch_2/conv_batch_2.onnx) | Batched Conv with SAME_UPPER padding and bias. |
| [`batch_4_pointwise`](conv/batch_4_pointwise/conv_batch_4_pointwise.onnx) | Pointwise Conv with batch size four. |
| [`input_224_7x7_stride2`](conv/input_224_7x7_stride2/conv_input_224_7x7_stride2.onnx) | 224x224 RGB Conv with 64 output channels, a 7x7 kernel, stride two, and bias. |
| [`depthwise_1024_channels`](conv/depthwise_1024_channels/conv_depthwise_1024_channels.onnx) | Depthwise pointwise Conv with 1024 groups. |
| [`depthwise_grouped`](conv/depthwise_grouped/conv_depthwise_grouped.onnx) | Depthwise-style grouped Conv with one input channel per group. |
| [`dilated_3x3`](conv/dilated_3x3/conv_dilated_3x3.onnx) | Conv with a dilated 3x3 kernel. |
| [`dynamic`](conv/dynamic/conv_dynamic.onnx) | Conv with runtime input and weight tensors. |
| [`explicit_padding`](conv/explicit_padding/conv_explicit_padding.onnx) | 3x3 Conv with symmetric explicit padding. |
| [`grouped_many_groups`](conv/grouped_many_groups/conv_grouped_many_groups.onnx) | Pointwise Conv with many groups and high channel counts. |
| [`grouped_two_groups`](conv/grouped_two_groups/conv_grouped_two_groups.onnx) | Two-group pointwise Conv with bias. |
| [`huge_pointwise_1024`](conv/huge_pointwise_1024/conv_huge_pointwise_1024.onnx) | Pointwise Conv with 1024 input and output channels. |
| [`huge_pointwise_1024_dynamic`](conv/huge_pointwise_1024_dynamic/conv_huge_pointwise_1024_dynamic.onnx) | The 1024-channel pointwise Conv with runtime weights. |
| [`kernel_3x3`](conv/kernel_3x3/conv_kernel_3x3.onnx) | Basic 3x3 Conv without padding. |
| [`kernel_equals_input_spatial`](conv/kernel_equals_input_spatial/conv_kernel_equals_input_spatial.onnx) | Conv whose kernel covers the full spatial input. |
| [`large_input_channels_1x1`](conv/large_input_channels_1x1/conv_large_input_channels_1x1.onnx) | Pointwise Conv with 1024 input channels and modest output width. |
| [`large_output_channels_1x1`](conv/large_output_channels_1x1/conv_large_output_channels_1x1.onnx) | Pointwise Conv with modest input width and 1024 output channels. |
| [`large_spatial`](conv/large_spatial/conv_large_spatial.onnx) | 3x3 Conv on a larger spatial input. |
| [`multi_channel`](conv/multi_channel/conv_multi_channel.onnx) | 3x3 Conv with multiple input and output channels. |
| [`non_square_kernel_1x3`](conv/non_square_kernel_1x3/conv_non_square_kernel_1x3.onnx) | Conv with a non-square 1x3 kernel. |
| [`non_square_kernel_3x1`](conv/non_square_kernel_3x1/conv_non_square_kernel_3x1.onnx) | Conv with a non-square 3x1 kernel. |
| [`non_uniform_stride`](conv/non_uniform_stride/conv_non_uniform_stride.onnx) | Conv with different height and width strides. |
| [`output_channel_grouping_minimal`](conv/output_channel_grouping_minimal/conv_output_channel_grouping_minimal.onnx) | Minimal 64-to-256 pointwise Conv for output-channel grouping. |
| [`pointwise_1x1`](conv/pointwise_1x1/conv_pointwise_1x1.onnx) | Basic pointwise channel-mixing Conv. |
| [`pointwise_tiled_chain`](conv/pointwise_tiled_chain/conv_pointwise_tiled_chain.onnx) | Relu and chained pointwise Convs with a tiled intermediate. |
| [`real_asymmetric_padding`](conv/real_asymmetric_padding/conv_real_asymmetric_padding.onnx) | Conv with asymmetric explicit padding. |
| [`relu_conv_store`](conv/relu_conv_store/conv_relu_conv_store.onnx) | Conv, Relu, and a second Conv to validate an intermediate stored result. |
| [`same_lower_3x3`](conv/same_lower_3x3/conv_same_lower_3x3.onnx) | 3x3 Conv with SAME_LOWER padding. |
| [`same_padding_3x3`](conv/same_padding_3x3/conv_same_padding_3x3.onnx) | 3x3 Conv with SAME_UPPER padding. |
| [`kernel_2x2`](conv/kernel_2x2/conv_kernel_2x2.onnx) | Hand-authored Conv with a 2x2 kernel. |
| [`stride_2`](conv/stride_2/conv_stride_2.onnx) | 3x3 Conv with stride two. |
| [`strategy_depthwise_16`](conv/strategy_depthwise_16/conv_strategy_depthwise_16.onnx) | 16-channel depthwise 3x3 Conv for depthwise lowering. |
| [`strategy_input_k_tiled`](conv/strategy_input_k_tiled/conv_strategy_input_k_tiled.onnx) | 32-channel 3x3 Conv sized to exercise input-K tiling. |
| [`strategy_output_channel_tiled`](conv/strategy_output_channel_tiled/conv_strategy_output_channel_tiled.onnx) | 8-to-192 3x3 Conv sized to exercise output-channel tiling. |
| [`strategy_streamed_packed`](conv/strategy_streamed_packed/conv_strategy_streamed_packed.onnx) | 3-to-16 3x3 Conv on 128x128 input for streamed packed lowering. |
| [`strategy_streamed_patch`](conv/strategy_streamed_patch/conv_strategy_streamed_patch.onnx) | 3-to-16 3x3 Conv on 64x64 input for streamed patch lowering. |
| [`strategy_tiled_2d`](conv/strategy_tiled_2d/conv_strategy_tiled_2d.onnx) | 32-to-192 3x3 Conv sized to exercise 2D tiling. |
| [`with_bias_3x3`](conv/with_bias_3x3/conv_with_bias_3x3.onnx) | Multi-channel 3x3 Conv with bias. |
| [`with_constant`](conv/with_constant/conv_with_constant.onnx) | Hand-authored SAME_UPPER Conv with constant weight and bias. |
| [`without_kernel_shape_attr`](conv/without_kernel_shape_attr/conv_without_kernel_shape_attr.onnx) | Conv whose kernel shape is inferred from its weight tensor. |
| [`yolo11n_depthwise_head`](conv/yolo11n_depthwise_head/conv_yolo11n_depthwise_head.onnx) | YOLO11n pointwise-to-depthwise head boundary at `80x80`, preserving row fragments. |
| [`yolo11n_heavy`](conv/yolo11n_heavy/conv_yolo11n_heavy.onnx) | Two largest standard YOLO11n Conv-SiLU blocks by MAC count at `64x80x80`. |
| [`yolo11n_stem`](conv/yolo11n_stem/conv_yolo11n_stem.onnx) | First two YOLO11n `Conv-SiLU` blocks at `640x640`, including the distributed activation boundary. |
## Div (6) — `div/`
| Case | Description |
|---|---|
| [`after_gemm`](div/after_gemm/div_after_gemm.onnx) | Gemm followed by Div with a broadcast divisor vector. |
| [`basic`](div/basic/div_basic.onnx) | Elementwise Div by a same-shape constant tensor. |
| [`channel_broadcast_1024`](div/channel_broadcast_1024/div_channel_broadcast_1024.onnx) | Div with NCHW per-channel broadcasting over 1024 channels. |
| [`leading_dimension_broadcast`](div/leading_dimension_broadcast/div_leading_dimension_broadcast.onnx) | Div with trailing-dimension broadcasting. |
| [`runtime_scalar_rhs`](div/runtime_scalar_rhs/div_runtime_scalar_rhs.onnx) | Div of a runtime tensor by a scalar initializer. |
| [`scalar_constant`](div/scalar_constant/div_scalar_constant.onnx) | Div with scalar broadcasting on a 2D tensor. |
## Gather (5) — `gather/`
| Case | Description |
|---|---|
| [`3d_input_axis1`](gather/3d_input_axis1/gather_3d_input_axis1.onnx) | Gathers along axis one of a 3D input. |
| [`axis0_matrix_indices`](gather/axis0_matrix_indices/gather_axis0_matrix_indices.onnx) | Gathers rows using a 2D indices tensor. |
| [`axis1`](gather/axis1/gather_axis1.onnx) | Gathers selected columns from a 2D tensor. |
| [`negative_axis`](gather/negative_axis/gather_negative_axis.onnx) | Gathers using a negative axis. |
| [`negative_indices`](gather/negative_indices/gather_negative_indices.onnx) | Gathers with negative indices along axis zero. |
## Gemm (21) — `gemm/`
| Case | Description |
|---|---|
| [`alpha_beta`](gemm/alpha_beta/gemm_alpha_beta.onnx) | Applies non-default alpha and beta scaling with bias. |
| [`bias_rank2_broadcast`](gemm/bias_rank2_broadcast/gemm_bias_rank2_broadcast.onnx) | Broadcasts a rank-2 bias across output rows. |
| [`dynamic`](gemm/dynamic/gemm_dynamic.onnx) | Uses both matrix operands at runtime. |
| [`dynamic_alpha`](gemm/dynamic_alpha/gemm_dynamic_alpha.onnx) | Uses runtime operands with non-default alpha scaling. |
| [`dynamic_beta`](gemm/dynamic_beta/gemm_dynamic_beta.onnx) | Uses runtime operands and bias with non-default beta scaling. |
| [`dynamic_bias`](gemm/dynamic_bias/gemm_dynamic_bias.onnx) | Uses runtime matrix operands and runtime bias. |
| [`dynamic_bias_alpha_beta`](gemm/dynamic_bias_alpha_beta/gemm_dynamic_bias_alpha_beta.onnx) | Combines runtime operands and bias with alpha and beta scaling. |
| [`dynamic_transpose_b`](gemm/dynamic_transpose_b/gemm_dynamic_transpose_b.onnx) | Transposes a runtime right-hand matrix. |
| [`huge_1024`](gemm/huge_1024/gemm_huge_1024.onnx) | Uses 1024-wide inner and output dimensions. |
| [`large`](gemm/large/gemm_large.onnx) | Exercises larger rectangular matrices. |
| [`large_k_small_n`](gemm/large_k_small_n/gemm_large_k_small_n.onnx) | Uses a large reduction dimension and narrow output. |
| [`non_square`](gemm/non_square/gemm_non_square.onnx) | Uses different reduction and output widths. |
| [`scalar_bias`](gemm/scalar_bias/gemm_scalar_bias.onnx) | Broadcasts a scalar bias to the full output. |
| [`square_weights`](gemm/square_weights/gemm_square_weights.onnx) | Basic Gemm with square weights. |
| [`small`](gemm/small/gemm_small.onnx) | Tiny Gemm for fast focused validation. |
| [`small_k_large_n`](gemm/small_k_large_n/gemm_small_k_large_n.onnx) | Uses a modest reduction dimension and wide output. |
| [`transpose_a`](gemm/transpose_a/gemm_transpose_a.onnx) | Transposes the left-hand matrix. |
| [`transpose_a_and_b`](gemm/transpose_a_and_b/gemm_transpose_a_and_b.onnx) | Transposes both matrix operands. |
| [`transpose_b`](gemm/transpose_b/gemm_transpose_b.onnx) | Transposes the right-hand weight matrix. |
| [`transpose_b_with_bias`](gemm/transpose_b_with_bias/gemm_transpose_b_with_bias.onnx) | Combines a transposed weight matrix with bias. |
| [`with_bias`](gemm/with_bias/gemm_with_bias.onnx) | Basic matrix product with vector bias. |
## Gemv (5) — `gemv/`
| Case | Description |
|---|---|
| [`all_constant`](gemv/all_constant/gemv_all_constant.onnx) | Vector-matrix product with all inputs constant. |
| [`constant_weight`](gemv/constant_weight/gemv_constant_weight.onnx) | Basic single-row vector-matrix product with constant weights. |
| [`non_uniform_bias`](gemv/non_uniform_bias/gemv_non_uniform_bias.onnx) | Adds a non-uniform constant bias pattern. |
| [`uniform_bias`](gemv/uniform_bias/gemv_uniform_bias.onnx) | Adds a uniform constant bias pattern. |
| [`scalar_bias`](gemv/scalar_bias/gemv_scalar_bias.onnx) | Adds a scalar broadcast bias. |
## MatMul (12) — `matmul/`
| Case | Description |
|---|---|
| [`basic`](matmul/basic/matmul_basic.onnx) | Direct 2D MatMul with constant right-hand matrix. |
| [`batched_3d`](matmul/batched_3d/matmul_batched_3d.onnx) | Batched 3D MatMul with matching batch dimensions. |
| [`batched_3d_dynamic`](matmul/batched_3d_dynamic/matmul_batched_3d_dynamic.onnx) | Batched 3D MatMul with both operands at runtime. |
| [`batched_left_constant`](matmul/batched_left_constant/matmul_batched_left_constant.onnx) | Batched 3D MatMul with constant left-hand matrix. |
| [`batched_lhs_broadcast`](matmul/batched_lhs_broadcast/matmul_batched_lhs_broadcast.onnx) | Broadcasts a 2D left-hand matrix across a batched right-hand tensor. |
| [`batched_rhs_broadcast`](matmul/batched_rhs_broadcast/matmul_batched_rhs_broadcast.onnx) | Broadcasts a 2D right-hand matrix across a batched left-hand tensor. |
| [`dynamic`](matmul/dynamic/matmul_dynamic.onnx) | Direct 2D MatMul with both operands at runtime. |
| [`huge_1024`](matmul/huge_1024/matmul_huge_1024.onnx) | Uses 1024-wide inner and output dimensions. |
| [`left_constant`](matmul/left_constant/matmul_left_constant.onnx) | Direct 2D MatMul with constant left-hand matrix. |
| [`matrix_vector`](matmul/matrix_vector/matmul_matrix_vector.onnx) | Matrix-vector multiplication producing a 1D output. |
| [`vector_matrix`](matmul/vector_matrix/matmul_vector_matrix.onnx) | Vector-matrix multiplication producing a 1D output. |
| [`yolo_attention`](matmul/yolo_attention/matmul_yolo_attention.onnx) | YOLO11n rank-4 dynamic MatMul-scale-transpose-MatMul attention chain. |
## Mul (6) — `mul/`
| Case | Description |
|---|---|
| [`after_conv`](mul/after_conv/mul_after_conv.onnx) | Conv followed by per-channel scaling. |
| [`after_conv_scalar_constant`](mul/after_conv_scalar_constant/mul_after_conv_scalar_constant.onnx) | Conv followed by Mul with a scalar constant. |
| [`basic`](mul/basic/mul_basic.onnx) | Elementwise Mul on two inputs with identical shapes. |
| [`channel_broadcast_1024`](mul/channel_broadcast_1024/mul_channel_broadcast_1024.onnx) | Mul with NCHW per-channel broadcasting over 1024 channels. |
| [`leading_dimension_broadcast`](mul/leading_dimension_broadcast/mul_leading_dimension_broadcast.onnx) | Mul with trailing-dimension broadcasting. |
| [`scalar_constant`](mul/scalar_constant/mul_scalar_constant.onnx) | Mul with scalar broadcasting. |
## Pool (15) — `pool/`
| Case | Description |
|---|---|
| [`avg_basic`](pool/avg_basic/pool_avg_basic.onnx) | AveragePool with a 2x2 kernel and unit stride. |
| [`avg_ceil_mode`](pool/avg_ceil_mode/pool_avg_ceil_mode.onnx) | AveragePool with ceil mode enabled. |
| [`avg_explicit_padding`](pool/avg_explicit_padding/pool_avg_explicit_padding.onnx) | Explicitly padded AveragePool excluding pad from the divisor. |
| [`avg_include_pad`](pool/avg_include_pad/pool_avg_include_pad.onnx) | Explicitly padded AveragePool including pad in the divisor. |
| [`avg_large_channels`](pool/avg_large_channels/pool_avg_large_channels.onnx) | AveragePool with a large channel count and small spatial extent. |
| [`avg_non_uniform_stride`](pool/avg_non_uniform_stride/pool_avg_non_uniform_stride.onnx) | AveragePool with different height and width strides. |
| [`avg_real_asymmetric_padding`](pool/avg_real_asymmetric_padding/pool_avg_real_asymmetric_padding.onnx) | AveragePool with asymmetric explicit padding. |
| [`max_after_conv`](pool/max_after_conv/pool_max_after_conv.onnx) | Conv followed by MaxPool. |
| [`max_basic`](pool/max_basic/pool_max_basic.onnx) | MaxPool with a 2x2 kernel and unit stride. |
| [`max_ceil_mode`](pool/max_ceil_mode/pool_max_ceil_mode.onnx) | MaxPool with ceil mode enabled. |
| [`max_global_style_kernel_equals_input`](pool/max_global_style_kernel_equals_input/pool_max_global_style_kernel_equals_input.onnx) | MaxPool whose kernel covers the full spatial input. |
| [`max_non_square_kernel`](pool/max_non_square_kernel/pool_max_non_square_kernel.onnx) | MaxPool with a non-square kernel. |
| [`max_real_asymmetric_padding`](pool/max_real_asymmetric_padding/pool_max_real_asymmetric_padding.onnx) | MaxPool with asymmetric explicit padding. |
| [`max_same_upper`](pool/max_same_upper/pool_max_same_upper.onnx) | MaxPool with SAME_UPPER padding. |
| [`max_stride2_multichannel`](pool/max_stride2_multichannel/pool_max_stride2_multichannel.onnx) | Multi-channel MaxPool with stride two. |
## ReduceMean (17) — `reduce_mean/`
| Case | Description |
|---|---|
| [`4d_spatial`](reduce_mean/4d_spatial/reduce_mean_4d_spatial.onnx) | Reduces height and width of an NCHW tensor while preserving rank. |
| [`4d_spatial_keepdims_0`](reduce_mean/4d_spatial_keepdims_0/reduce_mean_4d_spatial_keepdims_0.onnx) | Reduces NCHW height and width while dropping those axes. |
| [`after_conv`](reduce_mean/after_conv/reduce_mean_after_conv.onnx) | Conv followed by a spatial ReduceMean. |
| [`all_axes_keepdims_0`](reduce_mean/all_axes_keepdims_0/reduce_mean_all_axes_keepdims_0.onnx) | Reduces all axes to a scalar. |
| [`all_axes_keepdims_1`](reduce_mean/all_axes_keepdims_1/reduce_mean_all_axes_keepdims_1.onnx) | Reduces all axes while preserving rank. |
| [`basic`](reduce_mean/basic/reduce_mean_basic.onnx) | Reduces a feature dimension while preserving rank. |
| [`channel_axis_nchw`](reduce_mean/channel_axis_nchw/reduce_mean_channel_axis_nchw.onnx) | Reduces the channel axis of an NCHW tensor. |
| [`keepdims_0`](reduce_mean/keepdims_0/reduce_mean_keepdims_0.onnx) | Reduces a feature dimension and drops that axis. |
| [`large_dimension_1024`](reduce_mean/large_dimension_1024/reduce_mean_large_dimension_1024.onnx) | Reduces a dimension of length 1024. |
| [`legacy_axes_1_2_keepdims_1`](reduce_mean/legacy_axes_1_2_keepdims_1/reduce_mean_legacy_axes_1_2_keepdims_1.onnx) | Opset-18 reduction over multiple positive axes. |
| [`legacy_axis1_keepdims_0`](reduce_mean/legacy_axis1_keepdims_0/reduce_mean_legacy_axis1_keepdims_0.onnx) | Opset-18 reduction over one axis while dropping it. |
| [`legacy_axis1_keepdims_1`](reduce_mean/legacy_axis1_keepdims_1/reduce_mean_legacy_axis1_keepdims_1.onnx) | Opset-18 reduction over one axis while preserving rank. |
| [`legacy_empty_axes_noop`](reduce_mean/legacy_empty_axes_noop/reduce_mean_legacy_empty_axes_noop.onnx) | Opset-18 empty-axes no-op followed by Relu. |
| [`legacy_nchw_spatial`](reduce_mean/legacy_nchw_spatial/reduce_mean_legacy_nchw_spatial.onnx) | Opset-18 spatial reduction on NCHW input. |
| [`legacy_negative_axis`](reduce_mean/legacy_negative_axis/reduce_mean_legacy_negative_axis.onnx) | Opset-18 reduction using a negative axis. |
| [`legacy_reduce_all_keepdims_1`](reduce_mean/legacy_reduce_all_keepdims_1/reduce_mean_legacy_reduce_all_keepdims_1.onnx) | Opset-18 all-axis reduction with the axes input omitted. |
| [`negative_axis`](reduce_mean/negative_axis/reduce_mean_negative_axis.onnx) | ReduceMean using a negative axis. |
## Relu (4) — `relu/`
| Case | Description |
|---|---|
| [`4d`](relu/4d/relu_4d.onnx) | Standalone Relu on an NCHW tensor. |
| [`after_conv`](relu/after_conv/relu_after_conv.onnx) | Conv followed by Relu. |
| [`after_gemm`](relu/after_gemm/relu_after_gemm.onnx) | Gemm followed by Relu. |
| [`basic`](relu/basic/relu_basic.onnx) | Standalone Relu on a 2D tensor. |
## Reshape (4) — `reshape/`
| Case | Description |
|---|---|
| [`4d_to_2d_flatten`](reshape/4d_to_2d_flatten/reshape_4d_to_2d_flatten.onnx) | Flattens a 4D tensor into a 2D view. |
| [`infer_dim_minus_one`](reshape/infer_dim_minus_one/reshape_infer_dim_minus_one.onnx) | Uses `-1` to infer one output dimension. |
| [`same_rank`](reshape/same_rank/reshape_same_rank.onnx) | Changes shape without changing rank. |
| [`zero_copies_input_dim`](reshape/zero_copies_input_dim/reshape_zero_copies_input_dim.onnx) | Uses `0` to copy an input dimension. |
## Resize (6) — `resize/`
| Case | Description |
|---|---|
| [`height_only`](resize/height_only/resize_height_only.onnx) | Nearest-neighbor resize of only the height dimension. |
| [`nearest_2x`](resize/nearest_2x/resize_nearest_2x.onnx) | Nearest-neighbor upsampling by a factor of two. |
| [`nearest_downsample`](resize/nearest_downsample/resize_nearest_downsample.onnx) | Nearest-neighbor downsampling. |
| [`non_uniform_scales`](resize/non_uniform_scales/resize_non_uniform_scales.onnx) | Nearest-neighbor resize with different spatial scales. |
| [`width_only`](resize/width_only/resize_width_only.onnx) | Nearest-neighbor resize of only the width dimension. |
| [`with_sizes`](resize/with_sizes/resize_with_sizes.onnx) | Resize using explicit output sizes instead of scales. |
## Sigmoid (3) — `sigmoid/`
| Case | Description |
|---|---|
| [`4d`](sigmoid/4d/sigmoid_4d.onnx) | Standalone Sigmoid on an NCHW tensor. |
| [`after_gemm`](sigmoid/after_gemm/sigmoid_after_gemm.onnx) | Gemm followed by Sigmoid. |
| [`basic`](sigmoid/basic/sigmoid_basic.onnx) | Standalone Sigmoid on a 2D tensor. |
## Slice (8) — `slice/`
| Case | Description |
|---|---|
| [`2d_basic`](slice/2d_basic/slice_2d_basic.onnx) | Slices a 2D tensor with explicit axes and unit steps. |
| [`after_conv`](slice/after_conv/slice_after_conv.onnx) | Conv followed by a spatial crop. |
| [`default_axes`](slice/default_axes/slice_default_axes.onnx) | Omits axes and steps to use positional defaults. |
| [`large_channel_1024`](slice/large_channel_1024/slice_large_channel_1024.onnx) | Slices a channel range from a 1024-channel tensor. |
| [`nchw_spatial_crop`](slice/nchw_spatial_crop/slice_nchw_spatial_crop.onnx) | Crops the spatial axes of an NCHW tensor. |
| [`negative_axis`](slice/negative_axis/slice_negative_axis.onnx) | Slices using a negative axis. |
| [`negative_indices`](slice/negative_indices/slice_negative_indices.onnx) | Slices using negative indices. |
| [`step2`](slice/step2/slice_step2.onnx) | Slices using a positive step greater than one. |
## Softmax (5) — `softmax/`
| Case | Description |
|---|---|
| [`3d_last_axis`](softmax/3d_last_axis/softmax_3d_last_axis.onnx) | Softmax over the last axis of a 3D tensor. |
| [`basic`](softmax/basic/softmax_basic.onnx) | Softmax over the last dimension of a 2D tensor. |
| [`channel_axis`](softmax/channel_axis/softmax_channel_axis.onnx) | Softmax over the channel axis of an NCHW tensor. |
| [`large_dimension_1024`](softmax/large_dimension_1024/softmax_large_dimension_1024.onnx) | Softmax over a last dimension of length 1024. |
| [`negative_axis`](softmax/negative_axis/softmax_negative_axis.onnx) | Softmax using a negative axis. |
## Split (4) — `split/`
| Case | Description |
|---|---|
| [`basic`](split/basic/split_basic.onnx) | Splits a 2D tensor into two explicit output sizes. |
| [`equal_three_way`](split/equal_three_way/split_equal_three_way.onnx) | Splits a 2D tensor evenly into three outputs. |
| [`negative_axis`](split/negative_axis/split_negative_axis.onnx) | Splits using a negative axis. |
| [`uneven_channel_axis_4d`](split/uneven_channel_axis_4d/split_uneven_channel_axis_4d.onnx) | Splits an NCHW channel axis into uneven outputs. |
## Sub (6) — `sub/`
| Case | Description |
|---|---|
| [`after_gemm`](sub/after_gemm/sub_after_gemm.onnx) | Gemm followed by Sub with a broadcast constant vector. |
| [`basic`](sub/basic/sub_basic.onnx) | Elementwise Sub on two runtime inputs with identical shapes. |
| [`broadcast_row`](sub/broadcast_row/sub_broadcast_row.onnx) | Sub with a broadcast row-vector right-hand constant. |
| [`channel_broadcast_1024`](sub/channel_broadcast_1024/sub_channel_broadcast_1024.onnx) | Sub with NCHW per-channel broadcasting over 1024 channels. |
| [`constant_lhs_broadcast`](sub/constant_lhs_broadcast/sub_constant_lhs_broadcast.onnx) | Sub with a broadcast constant left-hand operand. |
| [`leading_dimension_broadcast`](sub/leading_dimension_broadcast/sub_leading_dimension_broadcast.onnx) | Sub with trailing-dimension broadcasting. |
validation/operations/README.md
+19 -302
View File
@@ -1,319 +1,36 @@
# Operation validation suite
This directory contains the ONNX models used by `validation/validate.py` to
validate individual operations through compilation, Pim simulation, and
comparison with the ONNX-MLIR reference runtime.
## Naming
Every model uses the same path convention:
```text
<category>/<case>/<category>_<case>.onnx
```
The `<category>/<case>` pair is the operation ID printed by the validator.
Use lowercase `snake_case` for both components. Keep the generator function,
graph name, directory, and filename based on the same operation ID when adding
a test.
ONNX cases for compilation, functional Pim simulation, and comparison with the
ONNX-MLIR reference runtime. Run commands from the repository root.
## Generate and validate
Regenerate all generated models from the repository root:
```bash
.venv/bin/python validation/operations/gen_tests.py
```
Run the complete suite with deadlock detection:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations \
--raptor-extra-arg=--pim-detect-communication-deadlock \
--raptor-extra-arg=--pim-export-spatial-dataflow=none
--raptor-extra-arg=--pim-detect-communication-deadlock
```
Use `--compile-only` for compiler and deadlock checks, then `--run-only` to
reuse those artifacts for reference execution, simulation, and comparison.
The validator prints separate latency and throughput operation tables before
its summary and writes all of their rows to `validation_results.csv`.
The complete suite uses parallel workers and must run outside the sandbox.
Point `--operations-dir` at a category or case, such as `gemm/small` below this
directory, for focused validation. Use `--compile-only` followed by `--run-only`
to separate compilation from execution.
## Complete inventory
See the [validation guide](../README.md) for prerequisites, options, artifacts,
and result interpretation.
The suite contains 177 models. Tensor shapes, attributes, and constants are
defined in `gen_tests.py` and in the checked-in ONNX models.
## Cases and naming
### Add (5)
Models use `<category>/<case>/<category>_<case>.onnx`. The validator prints
`<category>/<case>` as the operation ID. Use lowercase `snake_case` and keep
the generator function, graph name, directory, and filename consistent.
| Case | Description |
|---|---|
| `after_gemm` | Gemm followed by Add with a broadcast bias vector. |
| `basic` | Elementwise Add on two inputs with identical shapes. |
| `broadcast_row` | Elementwise Add with row-vector broadcasting. |
| `channel_broadcast_1024` | NCHW per-channel broadcasting over 1024 channels. |
| `leading_dimension_broadcast` | Trailing-dimension broadcasting across leading dimensions. |
### Concat (3)
| Case | Description |
|---|---|
| `channel_axis` | Concatenates two runtime NCHW tensors along the channel axis. |
| `negative_axis` | Concatenates tensors using a negative axis. |
| `three_inputs_channel_axis` | Concatenates three runtime NCHW tensors along the channel axis. |
### Conv (42)
| Case | Description |
|---|---|
| `batch_2` | Batched Conv with SAME_UPPER padding and bias. |
| `batch_4_pointwise` | Pointwise Conv with batch size four. |
| `input_224_7x7_stride2` | 224x224 RGB Conv with 64 output channels, a 7x7 kernel, stride two, and bias. |
| `depthwise_1024_channels` | Depthwise pointwise Conv with 1024 groups. |
| `depthwise_grouped` | Depthwise-style grouped Conv with one input channel per group. |
| `dilated_3x3` | Conv with a dilated 3x3 kernel. |
| `dynamic` | Conv with runtime input and weight tensors. |
| `explicit_padding` | 3x3 Conv with symmetric explicit padding. |
| `grouped_many_groups` | Pointwise Conv with many groups and high channel counts. |
| `grouped_two_groups` | Two-group pointwise Conv with bias. |
| `huge_pointwise_1024` | Pointwise Conv with 1024 input and output channels. |
| `huge_pointwise_1024_dynamic` | The 1024-channel pointwise Conv with runtime weights. |
| `kernel_3x3` | Basic 3x3 Conv without padding. |
| `kernel_equals_input_spatial` | Conv whose kernel covers the full spatial input. |
| `large_input_channels_1x1` | Pointwise Conv with 1024 input channels and modest output width. |
| `large_output_channels_1x1` | Pointwise Conv with modest input width and 1024 output channels. |
| `large_spatial` | 3x3 Conv on a larger spatial input. |
| `multi_channel` | 3x3 Conv with multiple input and output channels. |
| `non_square_kernel_1x3` | Conv with a non-square 1x3 kernel. |
| `non_square_kernel_3x1` | Conv with a non-square 3x1 kernel. |
| `non_uniform_stride` | Conv with different height and width strides. |
| `output_channel_grouping_minimal` | Minimal 64-to-256 pointwise Conv for output-channel grouping. |
| `pointwise_1x1` | Basic pointwise channel-mixing Conv. |
| `pointwise_tiled_chain` | Relu and chained pointwise Convs with a tiled intermediate. |
| `real_asymmetric_padding` | Conv with asymmetric explicit padding. |
| `relu_conv_store` | Conv, Relu, and a second Conv to validate an intermediate stored result. |
| `same_lower_3x3` | 3x3 Conv with SAME_LOWER padding. |
| `same_padding_3x3` | 3x3 Conv with SAME_UPPER padding. |
| `kernel_2x2` | Hand-authored Conv with a 2x2 kernel. |
| `stride_2` | 3x3 Conv with stride two. |
| `strategy_depthwise_16` | 16-channel depthwise 3x3 Conv for depthwise lowering. |
| `strategy_input_k_tiled` | 32-channel 3x3 Conv sized to exercise input-K tiling. |
| `strategy_output_channel_tiled` | 8-to-192 3x3 Conv sized to exercise output-channel tiling. |
| `strategy_streamed_packed` | 3-to-16 3x3 Conv on 128x128 input for streamed packed lowering. |
| `strategy_streamed_patch` | 3-to-16 3x3 Conv on 64x64 input for streamed patch lowering. |
| `strategy_tiled_2d` | 32-to-192 3x3 Conv sized to exercise 2D tiling. |
| `with_bias_3x3` | Multi-channel 3x3 Conv with bias. |
| `with_constant` | Hand-authored SAME_UPPER Conv with constant weight and bias. |
| `without_kernel_shape_attr` | Conv whose kernel shape is inferred from its weight tensor. |
| `yolo11n_depthwise_head` | YOLO11n pointwise-to-depthwise head boundary at `80x80`, preserving row fragments. |
| `yolo11n_heavy` | Two largest standard YOLO11n Conv-SiLU blocks by MAC count at `64x80x80`. |
| `yolo11n_stem` | First two YOLO11n `Conv-SiLU` blocks at `640x640`, including the distributed activation boundary. |
### Div (6)
| Case | Description |
|---|---|
| `after_gemm` | Gemm followed by Div with a broadcast divisor vector. |
| `basic` | Elementwise Div by a same-shape constant tensor. |
| `channel_broadcast_1024` | Div with NCHW per-channel broadcasting over 1024 channels. |
| `leading_dimension_broadcast` | Div with trailing-dimension broadcasting. |
| `runtime_scalar_rhs` | Div of a runtime tensor by a scalar initializer. |
| `scalar_constant` | Div with scalar broadcasting on a 2D tensor. |
### Gather (5)
| Case | Description |
|---|---|
| `3d_input_axis1` | Gathers along axis one of a 3D input. |
| `axis0_matrix_indices` | Gathers rows using a 2D indices tensor. |
| `axis1` | Gathers selected columns from a 2D tensor. |
| `negative_axis` | Gathers using a negative axis. |
| `negative_indices` | Gathers with negative indices along axis zero. |
### Gemm (21)
| Case | Description |
|---|---|
| `alpha_beta` | Applies non-default alpha and beta scaling with bias. |
| `bias_rank2_broadcast` | Broadcasts a rank-2 bias across output rows. |
| `dynamic` | Uses both matrix operands at runtime. |
| `dynamic_alpha` | Uses runtime operands with non-default alpha scaling. |
| `dynamic_beta` | Uses runtime operands and bias with non-default beta scaling. |
| `dynamic_bias` | Uses runtime matrix operands and runtime bias. |
| `dynamic_bias_alpha_beta` | Combines runtime operands and bias with alpha and beta scaling. |
| `dynamic_transpose_b` | Transposes a runtime right-hand matrix. |
| `huge_1024` | Uses 1024-wide inner and output dimensions. |
| `large` | Exercises larger rectangular matrices. |
| `large_k_small_n` | Uses a large reduction dimension and narrow output. |
| `non_square` | Uses different reduction and output widths. |
| `scalar_bias` | Broadcasts a scalar bias to the full output. |
| `square_weights` | Basic Gemm with square weights. |
| `small` | Tiny Gemm for fast focused validation. |
| `small_k_large_n` | Uses a modest reduction dimension and wide output. |
| `transpose_a` | Transposes the left-hand matrix. |
| `transpose_a_and_b` | Transposes both matrix operands. |
| `transpose_b` | Transposes the right-hand weight matrix. |
| `transpose_b_with_bias` | Combines a transposed weight matrix with bias. |
| `with_bias` | Basic matrix product with vector bias. |
### Gemv (5)
| Case | Description |
|---|---|
| `all_constant` | Vector-matrix product with all inputs constant. |
| `constant_weight` | Basic single-row vector-matrix product with constant weights. |
| `non_uniform_bias` | Adds a non-uniform constant bias pattern. |
| `uniform_bias` | Adds a uniform constant bias pattern. |
| `scalar_bias` | Adds a scalar broadcast bias. |
### MatMul (12)
| Case | Description |
|---|---|
| `basic` | Direct 2D MatMul with constant right-hand matrix. |
| `batched_3d` | Batched 3D MatMul with matching batch dimensions. |
| `batched_3d_dynamic` | Batched 3D MatMul with both operands at runtime. |
| `batched_left_constant` | Batched 3D MatMul with constant left-hand matrix. |
| `batched_lhs_broadcast` | Broadcasts a 2D left-hand matrix across a batched right-hand tensor. |
| `batched_rhs_broadcast` | Broadcasts a 2D right-hand matrix across a batched left-hand tensor. |
| `dynamic` | Direct 2D MatMul with both operands at runtime. |
| `huge_1024` | Uses 1024-wide inner and output dimensions. |
| `left_constant` | Direct 2D MatMul with constant left-hand matrix. |
| `matrix_vector` | Matrix-vector multiplication producing a 1D output. |
| `vector_matrix` | Vector-matrix multiplication producing a 1D output. |
| `yolo_attention` | YOLO11n rank-4 dynamic MatMul-scale-transpose-MatMul attention chain. |
### Mul (6)
| Case | Description |
|---|---|
| `after_conv` | Conv followed by per-channel scaling. |
| `after_conv_scalar_constant` | Conv followed by Mul with a scalar constant. |
| `basic` | Elementwise Mul on two inputs with identical shapes. |
| `channel_broadcast_1024` | Mul with NCHW per-channel broadcasting over 1024 channels. |
| `leading_dimension_broadcast` | Mul with trailing-dimension broadcasting. |
| `scalar_constant` | Mul with scalar broadcasting. |
### Pool (15)
| Case | Description |
|---|---|
| `avg_basic` | AveragePool with a 2x2 kernel and unit stride. |
| `avg_ceil_mode` | AveragePool with ceil mode enabled. |
| `avg_explicit_padding` | Explicitly padded AveragePool excluding pad from the divisor. |
| `avg_include_pad` | Explicitly padded AveragePool including pad in the divisor. |
| `avg_large_channels` | AveragePool with a large channel count and small spatial extent. |
| `avg_non_uniform_stride` | AveragePool with different height and width strides. |
| `avg_real_asymmetric_padding` | AveragePool with asymmetric explicit padding. |
| `max_after_conv` | Conv followed by MaxPool. |
| `max_basic` | MaxPool with a 2x2 kernel and unit stride. |
| `max_ceil_mode` | MaxPool with ceil mode enabled. |
| `max_global_style_kernel_equals_input` | MaxPool whose kernel covers the full spatial input. |
| `max_non_square_kernel` | MaxPool with a non-square kernel. |
| `max_real_asymmetric_padding` | MaxPool with asymmetric explicit padding. |
| `max_same_upper` | MaxPool with SAME_UPPER padding. |
| `max_stride2_multichannel` | Multi-channel MaxPool with stride two. |
### ReduceMean (17)
| Case | Description |
|---|---|
| `4d_spatial` | Reduces height and width of an NCHW tensor while preserving rank. |
| `4d_spatial_keepdims_0` | Reduces NCHW height and width while dropping those axes. |
| `after_conv` | Conv followed by a spatial ReduceMean. |
| `all_axes_keepdims_0` | Reduces all axes to a scalar. |
| `all_axes_keepdims_1` | Reduces all axes while preserving rank. |
| `basic` | Reduces a feature dimension while preserving rank. |
| `channel_axis_nchw` | Reduces the channel axis of an NCHW tensor. |
| `keepdims_0` | Reduces a feature dimension and drops that axis. |
| `large_dimension_1024` | Reduces a dimension of length 1024. |
| `legacy_axes_1_2_keepdims_1` | Opset-18 reduction over multiple positive axes. |
| `legacy_axis1_keepdims_0` | Opset-18 reduction over one axis while dropping it. |
| `legacy_axis1_keepdims_1` | Opset-18 reduction over one axis while preserving rank. |
| `legacy_empty_axes_noop` | Opset-18 empty-axes no-op followed by Relu. |
| `legacy_nchw_spatial` | Opset-18 spatial reduction on NCHW input. |
| `legacy_negative_axis` | Opset-18 reduction using a negative axis. |
| `legacy_reduce_all_keepdims_1` | Opset-18 all-axis reduction with the axes input omitted. |
| `negative_axis` | ReduceMean using a negative axis. |
### Relu (4)
| Case | Description |
|---|---|
| `4d` | Standalone Relu on an NCHW tensor. |
| `after_conv` | Conv followed by Relu. |
| `after_gemm` | Gemm followed by Relu. |
| `basic` | Standalone Relu on a 2D tensor. |
### Reshape (4)
| Case | Description |
|---|---|
| `4d_to_2d_flatten` | Flattens a 4D tensor into a 2D view. |
| `infer_dim_minus_one` | Uses `-1` to infer one output dimension. |
| `same_rank` | Changes shape without changing rank. |
| `zero_copies_input_dim` | Uses `0` to copy an input dimension. |
### Resize (6)
| Case | Description |
|---|---|
| `height_only` | Nearest-neighbor resize of only the height dimension. |
| `nearest_2x` | Nearest-neighbor upsampling by a factor of two. |
| `nearest_downsample` | Nearest-neighbor downsampling. |
| `non_uniform_scales` | Nearest-neighbor resize with different spatial scales. |
| `width_only` | Nearest-neighbor resize of only the width dimension. |
| `with_sizes` | Resize using explicit output sizes instead of scales. |
### Sigmoid (3)
| Case | Description |
|---|---|
| `4d` | Standalone Sigmoid on an NCHW tensor. |
| `after_gemm` | Gemm followed by Sigmoid. |
| `basic` | Standalone Sigmoid on a 2D tensor. |
### Slice (8)
| Case | Description |
|---|---|
| `2d_basic` | Slices a 2D tensor with explicit axes and unit steps. |
| `after_conv` | Conv followed by a spatial crop. |
| `default_axes` | Omits axes and steps to use positional defaults. |
| `large_channel_1024` | Slices a channel range from a 1024-channel tensor. |
| `nchw_spatial_crop` | Crops the spatial axes of an NCHW tensor. |
| `negative_axis` | Slices using a negative axis. |
| `negative_indices` | Slices using negative indices. |
| `step2` | Slices using a positive step greater than one. |
### Softmax (5)
| Case | Description |
|---|---|
| `3d_last_axis` | Softmax over the last axis of a 3D tensor. |
| `basic` | Softmax over the last dimension of a 2D tensor. |
| `channel_axis` | Softmax over the channel axis of an NCHW tensor. |
| `large_dimension_1024` | Softmax over a last dimension of length 1024. |
| `negative_axis` | Softmax using a negative axis. |
### Split (4)
| Case | Description |
|---|---|
| `basic` | Splits a 2D tensor into two explicit output sizes. |
| `equal_three_way` | Splits a 2D tensor evenly into three outputs. |
| `negative_axis` | Splits using a negative axis. |
| `uneven_channel_axis_4d` | Splits an NCHW channel axis into uneven outputs. |
### Sub (6)
| Case | Description |
|---|---|
| `after_gemm` | Gemm followed by Sub with a broadcast constant vector. |
| `basic` | Elementwise Sub on two runtime inputs with identical shapes. |
| `broadcast_row` | Sub with a broadcast row-vector right-hand constant. |
| `channel_broadcast_1024` | Sub with NCHW per-channel broadcasting over 1024 channels. |
| `constant_lhs_broadcast` | Sub with a broadcast constant left-hand operand. |
| `leading_dimension_broadcast` | Sub with trailing-dimension broadcasting. |
The [complete inventory](INVENTORY.md) describes all 177 checked-in cases.
The suite covers arithmetic and broadcasting, convolution strategies, matrix
products, pooling, reductions, activations, and tensor indexing/layout changes,
including chained and YOLO-derived regression cases. The case directories and
[`gen_tests.py`](gen_tests.py) define the inventory, shapes, attributes, and
constants; some checked-in models are hand-authored.
validation/tests/test_pim_experiments.py
+393
View File
@@ -0,0 +1,393 @@
from __future__ import annotations
import argparse
import copy
import csv
import importlib.util
import json
import os
from pathlib import Path
import sys
import pytest
TOOLS = Path(__file__).resolve().parents[1] / "tools/pim"
sys.path.insert(0, str(TOOLS / "ablation"))
sys.path.insert(0, str(TOOLS / "pimcomp/compare"))
import run_ablation as ablation
import run_pimcomp_models as suite
import compare_raptor_pimcomp_model as comparator
def test_ablation_results_stay_in_independent_output_roots(tmp_path):
variants = ablation.selected_variants(["no-plan"])
outputs = []
for name, value in (("first", 80), ("second", 60)):
root = tmp_path / name
for variant in variants:
path = ablation.variant_output_dir(root, variant) / "results_comparison.csv"
path.parent.mkdir(parents=True)
row = {"arch": "arch-a", "model": "vgg8", "mode": "latency",
"raptor_pipeline": "1", "raptor_functional_validation": "PASS"}
row.update({field: 100 if variant.name == "baseline" else value
for field, _ in ablation.PERCENTAGE_FIELDS})
with path.open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
output, failures = ablation.aggregate_results(root, variants)
assert failures == []
assert output == root / "results_ablation.csv"
outputs.append(output)
for output, change in zip(outputs, ("-20.00%", "-40.00%")):
with output.open(newline="") as stream:
rows = list(csv.DictReader(stream))
assert rows[0]["latency"] == "+0.00%"
assert rows[1]["latency"] == change
@pytest.mark.parametrize("custom_input", [False, True])
def test_ablation_plots_follow_the_selected_experiment(tmp_path, monkeypatch, custom_input):
spec = importlib.util.spec_from_file_location("ablation_plots", TOOLS / "ablation/generate_plots.py")
plots = importlib.util.module_from_spec(spec)
monkeypatch.setitem(sys.modules, spec.name, plots)
spec.loader.exec_module(plots)
expected = tmp_path / "experiment/results_ablation.csv" if custom_input else (
ablation.DEFAULT_RESULTS_PATH
)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom_input else [])])
result = plots.Result("baseline", "arch-a", "vgg8", "latency", 1, "PASS", {})
inputs, outputs = [], []
monkeypatch.setattr(plots, "configure_style", lambda: None)
monkeypatch.setattr(plots, "load_results", lambda path: inputs.append(path) or [result])
monkeypatch.setattr(plots, "draw_metric", lambda rows, metric, label, output: outputs.append(output))
plots.main()
assert inputs == [expected]
assert outputs == [expected.parent / "plots"] * 3
@pytest.fixture
def recorded_failure(tmp_path):
model, functional, config = (tmp_path / name for name in ("model.onnx", "functional.onnx", "config.json"))
for path in (model, functional, config):
path.write_text(path.name)
args = argparse.Namespace(
pimcomp_config=config, pimsim_mode="throughput", pimsim_time_ms=1000,
pimcomp_pipeline="batch", pimcomp_replication="GA", pimcomp_model_name=None,
seed=0, batch_size=64, threshold=1e-3, rtol=1e-4, no_fast=False,
skip_pimsim_nn=False,
)
hardware = {"core_count": 168, "crossbar_size": 128}
context = comparator.pimcomp_reuse_context(args, model, functional, hardware)
report = {
"pimcomp_model_source": "original_onnx", "pimcomp_reuse_context": context,
"pimcomp_validation": {"passed": False, "max_diffs": {}, "status": "failed",
"error": "Unsupported model"},
"pimcomp_performance": {"skipped": True, "reason": "Compilation failed"},
"pimcomp_instruction_summary": comparator.empty_instruction_summary("No artifact"),
"paths": {}, "steps": [],
"failures": [{"stage": "Compile Pimcomp frontend", "error": "Unsupported model"}],
}
path = tmp_path / "comparison_report.json"
path.write_text(json.dumps(report))
return path, report, args, model, functional, hardware
def test_failed_pimcomp_report_needs_no_simulation_artifacts(recorded_failure):
path, report, *_ = recorded_failure
reused = comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"])
assert reused == report
assert not reused["pimcomp_validation"]["passed"]
failures = []
comparator.restore_side_records(reused, "pimcomp", failures, [])
assert failures == report["failures"]
@pytest.mark.parametrize("setting,value", [
("pimsim_mode", "latency"), ("pimcomp_pipeline", "element"),
("pimsim_time_ms", 2000), ("pimcomp_replication", "balance"),
("seed", 1), ("batch_size", 128), ("no_fast", True),
("skip_pimsim_nn", True), ("threshold", 0.1), ("rtol", 0.01),
])
def test_reuse_rejects_different_experiment_settings(recorded_failure, setting, value):
path, report, args, model, functional, hardware = recorded_failure
setattr(args, setting, value)
expected = comparator.pimcomp_reuse_context(args, model, functional, hardware)
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
comparator.load_reused_pimcomp_report(path, expected)
@pytest.mark.parametrize("changed", ["model", "functional", "config", "hardware", "model_path"])
def test_reuse_rejects_changed_inputs_and_architecture(recorded_failure, changed):
path, report, args, model, functional, hardware = recorded_failure
if changed == "hardware":
hardware["core_count"] = 138
elif changed == "model_path":
replacement = model.with_name("other.onnx")
replacement.write_bytes(model.read_bytes())
model = replacement
else:
{"model": model, "functional": functional, "config": args.pimcomp_config}[changed].write_text("changed")
expected = comparator.pimcomp_reuse_context(args, model, functional, hardware)
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
comparator.load_reused_pimcomp_report(path, expected)
@pytest.mark.parametrize("damage", ["json", "legacy", "validation", "performance", "summary", "steps"])
def test_reuse_rejects_incomplete_reports(recorded_failure, damage):
path, report, *_ = recorded_failure
expected = copy.deepcopy(report["pimcomp_reuse_context"])
if damage == "json":
path.write_text("{")
else:
field = {"legacy": "pimcomp_reuse_context", "validation": "pimcomp_validation",
"performance": "pimcomp_performance", "summary": "pimcomp_instruction_summary",
"steps": "steps"}[damage]
report.pop(field)
path.write_text(json.dumps(report))
with pytest.raises(ValueError, match="Rerun Pimcomp"):
comparator.load_reused_pimcomp_report(path, expected)
@pytest.mark.parametrize("only,fresh", [(None, True), (None, False), ("pimcomp", True), ("raptor", False)])
def test_suite_reruns_anchor_and_shares_only_new_reports(tmp_path, monkeypatch, only, fresh):
root = tmp_path / "suite"
for _, pipeline, _ in suite.COMPARISONS[1:]:
report = suite.result_dir(root, "vgg8", "arch-a", "throughput", pipeline) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
report.write_text("{}")
os.utime(report, ns=(1, 1))
batches = []
def run_jobs(jobs, max_workers, log_dir, offset):
assert max_workers == 1
batches.append([command for _, command, _ in jobs])
for _, command, _ in jobs:
output = Path(command[command.index("--out-dir") + 1]) / "pimcomp/comparison_report.json"
if fresh or len(batches) > 1:
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text('{"new": true}')
return [], offset + len(jobs)
summary = tmp_path / "results.csv"
summary.write_text("summary\n")
monkeypatch.setattr(suite, "validate_pimcomp_source", lambda: None)
monkeypatch.setattr(suite, "run", lambda *a, **k: 0)
monkeypatch.setattr(suite, "run_comparison_jobs", run_jobs)
monkeypatch.setattr(suite, "write_results_csv", lambda *a: summary)
monkeypatch.setattr(suite, "comparison_passed", lambda report: True)
monkeypatch.setattr(sys, "argv", [
"run_pimcomp_models.py", "--out-dir", str(root), "--models", "vgg8",
"--archs", "arch-a", "--mode", "throughput", "--jobs", "1",
*(["--only", only] if only else []),
])
assert suite.main() == 0
if only == "raptor":
assert len(batches[0]) == 3
assert all("--reuse-pimcomp-report" in command for command in batches[0])
else:
assert len(batches[0]) == 1
assert "--reuse-pimcomp-report" not in batches[0][0]
assert all(("--reuse-pimcomp-report" in command) == fresh for command in batches[1])
assert ("--reuse-raptor-report" in batches[0][0]) == (only == "pimcomp")
def test_missing_pimcomp_report_is_not_reused(tmp_path):
assert not suite.pimcomp_report_ready(tmp_path / "output", None)
def test_missing_pimcomp_report_is_rejected_at_load(recorded_failure):
path, report, *_ = recorded_failure
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
comparator.load_reused_pimcomp_report(path.with_name("missing.json"), report["pimcomp_reuse_context"])
def test_canonical_default_result_locations():
assert suite.DEFAULT_RESULTS_PATH == TOOLS / "pimcomp/compare/results_comparison.csv"
assert ablation.DEFAULT_RESULTS_PATH == TOOLS / "ablation/results_ablation.csv"
def test_comparison_default_and_custom_aggregate_paths(tmp_path, monkeypatch):
canonical = tmp_path / "scripts/results_comparison.csv"
artifact_root = tmp_path / "models"
monkeypatch.setattr(suite, "DEFAULT_RESULTS_PATH", canonical)
monkeypatch.setattr(suite, "SUITE", artifact_root)
assert suite.write_results_csv(None, "arch-a", []) == canonical
original = canonical.read_bytes()
for name in ("first", "second"):
custom = tmp_path / name
assert suite.write_results_csv(custom, "arch-a", []) == custom / "results_comparison.csv"
assert canonical.read_bytes() == original
variant = suite.write_results_csv(None, "arch-a", [], ablation_variant="baseline")
assert variant == artifact_root / "ablation/baseline/results_comparison.csv"
suite.clean_artifacts(None, [])
assert canonical.read_bytes() == original
@pytest.mark.parametrize("custom", [False, True])
def test_ablation_cli_selects_canonical_or_custom_output(tmp_path, monkeypatch, custom):
artifact_root = tmp_path / "models"
canonical = tmp_path / "scripts/results_ablation.csv"
monkeypatch.setattr(ablation, "DEFAULT_OUT_DIR", artifact_root)
monkeypatch.setattr(ablation, "DEFAULT_RESULTS_PATH", canonical)
monkeypatch.setattr(ablation.subprocess, "Popen", lambda *a, **k: argparse.Namespace(wait=lambda: 0))
calls = []
monkeypatch.setattr(ablation, "aggregate_results", lambda root, *a, **k: (calls.append((root, k["output"])) or None, []))
command = ["--variant", "baseline", "--jobs", "1"]
if custom:
command += ["--out-dir", str(tmp_path / "custom")]
assert ablation.main(command) == 0
assert calls == [(tmp_path / "custom", None) if custom else (artifact_root, canonical)]
@pytest.mark.parametrize("value", [None, "", "NA", "0", "0.00", "-1", "nan", "inf"])
def test_ablation_percentages_reject_unavailable_or_invalid_values(value):
assert ablation.percentage_difference(value, "100") == "NA"
assert ablation.percentage_difference("100", value) == "NA"
def test_tiny_positive_measurements_and_extreme_changes_are_preserved():
value = 1e-12
assert float(suite.format_value(value)) == value
assert ablation.percentage_difference(suite.format_value(value), "100") == "-100.00%"
assert ablation.percentage_difference("1000000", "1") == "+99999900.00%"
@pytest.mark.parametrize("status", [{"error": "timeout"}, {"skipped": True}, {"unsupported": True}])
def test_failed_performance_never_uses_stale_numeric_fields(status):
metrics = {"latency_ms": 10, "throughput": 100, "average_power_mw": 20,
"average_energy_pj": 30, **status}
assert all(value is None for value in suite.performance_values(metrics).values())
@pytest.mark.parametrize("value", [None, "NA", 0, -1, float("nan"), float("inf")])
def test_comparison_aggregation_rejects_invalid_measurements(value):
metrics = suite.performance_values({"latency_ms": value, "throughput": value,
"average_power_mw": value, "average_energy_pj": value})
assert all(number is None for number in metrics.values())
@pytest.mark.parametrize("primary", [None, "NA", 0, -1, float("nan"), float("inf"), -float("inf")])
def test_latency_uses_valid_fallback_after_invalid_primary(primary):
metrics = suite.performance_values({"latency_ms": primary, "average_latency_ms": "0.0063"})
assert metrics["latency"] == 0.0063
def test_latency_preserves_candidate_priority():
metrics = suite.performance_values({"latency_ms": "0.014", "average_latency_ms": 0.015})
assert metrics["latency"] == 0.014
@pytest.mark.parametrize("primary,fallback", [
(None, "NA"), ("NA", None), (0, -1), (-1, 0),
(float("nan"), float("inf")), (float("inf"), -float("inf")),
])
def test_latency_is_missing_when_all_candidates_are_invalid(primary, fallback):
metrics = suite.performance_values({"latency_ms": primary, "average_latency_ms": fallback})
assert metrics["latency"] is None
@pytest.mark.parametrize("invalid", [None, "NA", 0, -1, float("nan"), float("inf"), -float("inf")])
@pytest.mark.parametrize("first_valid", range(5))
def test_latency_priority_across_structured_and_parsed_metrics(monkeypatch, invalid, first_valid):
candidates = [invalid if index < first_valid else index + 1 for index in range(4)]
parsed = dict(zip(("latency_ms", "average_latency_ms"), candidates[2:]))
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
performance = dict(zip(("latency_ms", "average_latency_ms"), candidates[:2]))
assert suite.performance_values(performance)["latency"] == (first_valid + 1 if first_valid < 4 else None)
def test_structured_average_latency_precedes_parsed_latency(monkeypatch):
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: {"latency_ms": 1})
assert suite.performance_values({"average_latency_ms": 2})["latency"] == 2
@pytest.mark.parametrize("field,metric", [
("throughput", "throughput"), ("average_power_mw", "power"), ("average_energy_pj", "energy"),
])
@pytest.mark.parametrize("structured,parsed,expected", [
(10, 20, 10), (None, 20, 20), ("NA", 20, 20), (0, 20, 20), (-1, 20, 20),
(float("nan"), 20, 20), (float("inf"), 20, 20), (-float("inf"), 20, 20),
(0, float("nan"), None),
])
def test_measurement_priority_across_sources(monkeypatch, field, metric, structured, parsed, expected):
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: {field: parsed})
assert suite.performance_values({field: structured})[metric] == expected
@pytest.mark.parametrize("status", [
{"error": "timeout"}, {"skipped": True}, {"unsupported": True},
{"output_count": 0},
])
def test_parsed_measurements_respect_simulation_status(monkeypatch, status):
parsed = {"latency_ms": 1, "throughput": 2, "average_power_mw": 3, "average_energy_pj": 4}
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
assert suite.performance_values(status) == {
"latency": None, "throughput": None, "power": 3 if "output_count" in status else None,
"energy": None,
}
def test_parsed_zero_outputs_preserve_only_measured_power(monkeypatch):
parsed = {"output_count": 0, "latency_ms": 1, "throughput": 2,
"average_power_mw": 3, "average_energy_pj": 4}
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
assert suite.performance_values({}) == {"latency": None, "throughput": None, "power": 3, "energy": None}
@pytest.mark.parametrize("value,expected", [
(1e-12, "1e-12"), (0.00499999, "0.00499999"), (0.005, "0.005"),
(0.00500001, "0.00500001"), (0.0063, "0.0063"), (0.014, "0.014"),
(0.015, "0.015"), (1.23456789, "1.23457"), (123.456789, "123.457"),
(123456789, "1.23457e+08"), (1e20, "1e+20"),
])
def test_measurements_use_six_significant_digits(value, expected):
assert suite.format_value(value) == expected
@pytest.mark.parametrize("value", [None, 0, -1, float("nan"), float("inf"), -float("inf")])
def test_format_value_preserves_missing_measurements(value):
assert suite.format_value(value) == "NA"
def test_zero_outputs_preserve_only_measured_average_power():
metrics = suite.performance_values({"output_count": 0, "average_latency_ms": 10,
"throughput": 100, "average_energy_pj": 30,
"average_power_mw": 20})
assert metrics == {"latency": None, "throughput": None, "energy": None, "power": 20}
def test_functional_pass_survives_unavailable_performance(tmp_path):
report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True},
"raptor_performance": {"error": "timeout", "throughput": 0}}))
output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),))
with output.open(newline="") as stream:
row = next(csv.DictReader(stream))
assert row["raptor_functional_validation"] == "PASS"
assert all(row[field] == "NA" for field, _ in ablation.PERCENTAGE_FIELDS)
def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path):
variants = ablation.selected_variants(["no-plan"])
for variant in variants:
path = ablation.variant_output_dir(tmp_path, variant) / "results_comparison.csv"
path.parent.mkdir(parents=True)
row = {"arch": "arch-a", "model": "googlenet", "mode": "throughput",
"raptor_pipeline": "4", "raptor_functional_validation": "PASS"}
row.update({name: "100" if variant.name == "baseline" else "0.00"
for name, _ in ablation.PERCENTAGE_FIELDS})
with path.open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(row))
writer.writeheader()
writer.writerow(row)
destination = tmp_path / "canonical/results_ablation.csv"
output, failures = ablation.aggregate_results(tmp_path, variants, output=destination)
assert not failures and output == destination
with output.open(newline="") as stream:
rows = list(csv.DictReader(stream))
assert rows[1]["functional_validation"] == "PASS"
assert all(rows[1][metric] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS)
validation/tests/test_pim_plots.py
+157
View File
@@ -0,0 +1,157 @@
from __future__ import annotations
import csv
import importlib.util
import math
from pathlib import Path
import subprocess
import sys
import pytest
TOOLS = Path(__file__).resolve().parents[1] / "tools/pim"
@pytest.fixture
def plotters(monkeypatch):
modules = []
for name, directory in (("comparison_plots", "pimcomp/compare"), ("ablation_plots", "ablation")):
monkeypatch.syspath_prepend(str(TOOLS / directory))
spec = importlib.util.spec_from_file_location(name, TOOLS / directory / "generate_plots.py")
module = importlib.util.module_from_spec(spec)
monkeypatch.setitem(sys.modules, name, module)
spec.loader.exec_module(module)
modules.append(module)
return modules
@pytest.fixture
def comparison_csv():
with (TOOLS / "pimcomp/compare/results_comparison.csv").open(newline="") as stream:
reader = csv.DictReader(stream)
return reader.fieldnames, list(reader)
def write_csv(path, fields, rows):
with path.open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=fields)
writer.writeheader()
writer.writerows(rows)
return path
@pytest.mark.parametrize("custom", [False, True])
def test_comparison_plot_cli_input_paths(plotters, tmp_path, monkeypatch, custom):
comparison, _ = plotters
expected = tmp_path / "custom.csv" if custom else TOOLS / "pimcomp/compare/results_comparison.csv"
seen = []
class InputCaptured(Exception):
pass
def capture(path):
seen.append(path)
raise InputCaptured
monkeypatch.setattr(comparison, "load_results", capture)
monkeypatch.setattr(comparison, "configure_style", lambda: None)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom else [])])
with pytest.raises(InputCaptured):
comparison.main()
assert seen == [expected]
def test_exact_comparison_configuration_matrix(plotters, comparison_csv, tmp_path):
comparison, _ = plotters
fields, rows = comparison_csv
results = comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
keys = {(row.arch, row.model, row.mode, row.pipeline) for row in results}
assert keys == {(arch, model, mode, stage) for arch in comparison.ARCHITECTURES
for model in comparison.MODELS for mode, stage in comparison.CONFIGURATIONS}
@pytest.mark.parametrize("damage,message", [("missing", "Missing"), ("duplicate", "Duplicate"),
("arch", "Unexpected"), ("model", "Unexpected"),
("pipeline", "Unexpected"), ("mode", "Unexpected")])
def test_comparison_rejects_bad_matrix_keys(plotters, comparison_csv, tmp_path, damage, message):
comparison, _ = plotters
fields, rows = comparison_csv
if damage == "missing":
rows.pop()
elif damage == "duplicate":
rows[-1] = dict(rows[-2]) # Still 40 rows and 20 per architecture.
else:
field = "raptor_pipeline" if damage == "pipeline" else damage
rows[-1][field] = {"arch": "arch-c", "model": "other", "pipeline": "3", "mode": "other"}[damage]
with pytest.raises(ValueError, match=message):
comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
def test_matrix_validation_survives_optimized_python(comparison_csv, tmp_path):
fields, rows = comparison_csv
path = write_csv(tmp_path / "duplicate.csv", fields, [*rows[:-1], rows[-2]])
program = """import sys
from pathlib import Path
sys.path.insert(0, sys.argv[1])
import generate_plots
try:
generate_plots.load_results(Path(sys.argv[2]))
except ValueError as exc:
if 'Duplicate configuration' not in str(exc): raise
else:
raise RuntimeError('Invalid matrix accepted with Python -O')
"""
subprocess.run([sys.executable, "-O", "-c", program, str(TOOLS / "pimcomp/compare"), str(path)], check=True)
@pytest.mark.parametrize("field,value", [("pimcomp_latency_ms", "999"),
("pimcomp_throughput_samples_s", "999"), ("pimcomp_power_mw", "999"),
("pimcomp_energy_pj", "999"), ("pimcomp_functional_validation", "FAIL")])
def test_repeated_pimcomp_batch_results_must_agree(plotters, comparison_csv, tmp_path, field, value):
comparison, _ = plotters
fields, rows = comparison_csv
row = next(row for row in rows if row["arch"] == "arch-a" and row["model"] == "vgg8"
and row["raptor_pipeline"] == "4")
row[field] = value
with pytest.raises(ValueError, match="inconsistent shared Pimcomp batch results"):
comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
def test_legacy_invalid_ablation_pairs_are_unavailable_without_changing_pass(plotters):
_, ablation = plotters
with pytest.warns(UserWarning, match="2 invalid reciprocal"):
rows = ablation.load_results(TOOLS / "ablation/results_ablation.csv")
invalid = [row for row in rows if row.model == "googlenet" and row.variant == "no-plan"
and row.mode == "throughput"]
assert len(invalid) == 2
assert all(row.validation == "PASS" and math.isnan(row.metrics["latency"])
and math.isnan(row.metrics["throughput"]) for row in invalid)
def test_ablation_missing_values_and_legitimate_extremes(plotters, tmp_path):
_, ablation = plotters
row = {"variant": "no-plan", "arch": "arch-a", "model": "vgg8", "mode": "throughput",
"raptor_pipeline": "4", "functional_validation": "FAIL", "latency": "-100.00%",
"throughput": "+100000000.00%", "power": "NA", "energy": "-100.00%"}
result = ablation.load_results(write_csv(tmp_path / "extreme.csv", list(row), [row]))[0]
assert result.validation == "FAIL"
assert result.metrics["latency"] == result.metrics["energy"] == -100
assert result.metrics["throughput"] == 100000000
assert math.isnan(result.metrics["power"])
def test_both_plotters_export_png_and_svg(plotters, tmp_path):
comparison, ablation = plotters
comparison.configure_style()
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
selected = [row for row in rows if row.model == "vgg8" and row.mode == "latency"]
comparison.draw_absolute(selected, "latency", "Latency (ms)", "comparison-smoke", tmp_path)
row = ablation.Result("baseline", "arch-a", "vgg8", "latency", 1, "PASS",
{"latency": math.nan, "power": 0, "energy": 0, "throughput": math.nan})
ablation.draw_metric([row], "latency", "Latency change (%)", tmp_path)
for name in ("comparison-smoke", "latency_pipeline1_latency"):
assert (tmp_path / f"png/{name}.png").read_bytes().startswith(b"\x89PNG\r\n\x1a\n")
svg = (tmp_path / f"svg/{name}.svg").read_text()
assert "<svg" in svg
assert "N/A" in (tmp_path / "svg/latency_pipeline1_latency.svg").read_text()
validation/tools/pim/ablation/.gitignore
+1
View File
@@ -0,0 +1 @@
plots/
validation/tools/pim/ablation/README.md
+55 -60
View File
@@ -1,78 +1,73 @@
# Raptor compiler ablation
`run_ablation.py` performs the complete synchronization/Spatial-planning
ablation study on the Pimcomp model suite. By default it runs `vgg8`,
`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
explicitly.
Latency is pipeline 1; the wrapper invokes the suite runner separately for
latency and throughput pipeline 4.
Run the synchronization/Spatial-planning ablation from the repository root:
```bash
.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
--jobs 4
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
```
## Variants
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
| Variant | Raptor options |
|---|---|
| `baseline` | None. |
| `baseline` | None |
| `no-sync` | `--pim-disable-synchronization` |
| `no-plan` | `--pim-disable-spatial-planning` |
| `no-sync-no-plan` | Both options. |
| `no-sync-no-plan` | Both options |
Every variant runs Raptor only. Pimcomp is not compiled, validated, or
simulated; its validation column is `SKIPPED`, not `FAIL`. Reference inputs and
outputs are generated once under the shared common-artifact root and reused by
every variant. Ctrl+C terminates the active variant and all of its worker jobs.
## Options and outputs
Removing synchronization deliberately removes host-buffer lifetime and pipeline
restart guarantees. The `no-sync` variants can therefore fail numerical
validation, especially across repeated throughput inputs. Those failures remain
`FAIL` and cause a nonzero exit; a completed ablation is not proof that every
variant is correct. Baseline and `no-plan` must still pass numerical validation.
The combined CSV retains `functional_validation` for every row. Performance
percentages from failing variants describe incorrect programs, not valid
speedups. Throughput deadline averages likewise do not establish a converged
steady-state period.
- `--variant NAME`: repeat to select variants; baseline is always included.
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
- `--dry-run`: print commands without modifying files.
The percentage baseline is `baseline`, so its available values are `+0.00%`.
Every other
variant reports the signed percentage difference of its Raptor metrics from
that reference for the same model, architecture, mode, and pipeline. Positive
values mean the metric is higher; negative values mean it is lower.
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
replaces the default latency/pipeline-4 case set. `--only` is rejected because
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
## Options
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
reference inputs, outputs, and runners are shared under `artifacts/common/`.
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
Normal comparison results are preserved.
| Option | Description and default |
|---|---|
| `-h`, `--help` | Show help and exit. |
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
By default, the combined [results_ablation.csv](results_ablation.csv) is written
beside this script and printed after the run. An explicit `--out-dir PATH`
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
Default model artifacts and per-variant summaries remain in the locations above.
The aggregate records case identifiers,
`functional_validation`, and signed percentage changes in latency, throughput,
power, and energy against the matching baseline case. Positive means higher;
baseline values are zero.
All options of
[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
the suite runner's comparison mode and is rejected by this wrapper, which
always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
forwarded, the wrapper uses its default latency and throughput/pipeline-4
case set. Supplying either option overrides that default and is passed through
as one suite-runner invocation per variant.
Disabling synchronization removes host-buffer lifetime and pipeline restart
guarantees, so those variants may fail numerical validation and exit nonzero.
Baseline and `no-plan` must pass. Percentages from failing variants are not
valid speedups; deadline averages do not prove steady-state convergence.
The feature-full `baseline` variant is rerun with the same selected cases and
forwarded options as the disabled variants. All four variants are stored below
each model's `artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared
reference artifacts remain under each model's `artifacts/common` directory.
Normal comparison artifacts and summaries are preserved. Compact per-variant
comparison summaries remain under
`<out-dir>/ablation/<variant>/results_comparison.csv` after aggregation.
The combined table is written to `<out-dir>/results_ablation.csv`; it contains the
variant, case identifiers, actual `functional_validation` status, and signed
percentage `latency`, `throughput`, `power`, and `energy` columns. These are
Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor
only.
## Plots
```bash
.venv/bin/python validation/tools/pim/ablation/generate_plots.py
# For a custom experiment root:
.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
--input /path/to/experiment/results_ablation.csv
```
Requires Matplotlib from the root `requirements.txt`. By default, reads the
canonical CSV beside this script. Plots go to
`plots/png/` and `plots/svg/` beside the selected CSV; override with
`--output-dir PATH`.
Single-variant comparison CSVs are not accepted.
The checked-in CSV is an intentional result dataset; plotting uses its current
contents without running benchmarks. Invalid reciprocal percentage pairs render
as `N/A`; individual legitimate extreme changes are preserved. See
[performance availability](../../../../docs/evaluation/methodology.md#functional-validity-and-performance-availability).
Plots separate modes and pipelines. Throughput-mode latency means output
interval. Failed validation is labeled `INVALID`, other non-PASS statuses keep
their names, and missing measurements show `N/A`. See the
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
validation/tools/pim/ablation/generate_plots.py
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env python3
"""Plot percentage changes from an experiment's combined results_ablation.csv."""
from __future__ import annotations
import argparse
import csv
import math
import sys
import warnings
from dataclasses import dataclass
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from plot_style import COLORS, configure_style, save_figure, value_label
import matplotlib.pyplot as plt
from matplotlib.patches import Patch
from matplotlib.ticker import PercentFormatter
from run_ablation import ABLATION_DEFAULT_MODELS, DEFAULT_RESULTS_PATH, RESULT_FIELDS, VARIANTS
VARIANT_COLORS = {variant.name: color for variant, color in zip(VARIANTS, COLORS)}
STATUSES = ("PASS", "FAIL", "SKIPPED", "NA")
@dataclass(frozen=True)
class Result:
variant: str
arch: str
model: str
mode: str
pipeline: int
validation: str
metrics: dict[str, float]
def percentage(value: str) -> float:
if value == "NA":
return math.nan
if not value.endswith("%"):
raise ValueError(f"Expected a percentage or NA, got {value!r}")
parsed = float(value[:-1])
if not math.isfinite(parsed):
raise ValueError(f"Non-finite percentage: {value!r}")
return parsed
def load_results(path: Path) -> list[Result]:
rows = []
seen = set()
invalid_pairs = []
with path.open(newline="", encoding="utf-8-sig") as stream:
reader = csv.DictReader(stream)
if not {"variant", *RESULT_FIELDS} <= set(reader.fieldnames or []):
raise ValueError("Expected the combined ablation CSV with variant and percentage columns; "
"a per-variant results_comparison.csv cannot be used here.")
for row in reader:
variant, mode, status = row["variant"], row["mode"], row["functional_validation"]
if variant not in VARIANT_COLORS or mode not in ("latency", "throughput") or status not in STATUSES:
raise ValueError(f"Unknown ablation variant, mode, or validation status: {variant}, {mode}, {status}")
pipeline = int(row["raptor_pipeline"])
key = (variant, row["arch"], row["model"], mode, pipeline)
if key in seen:
raise ValueError(f"Duplicate ablation configuration: {key}")
seen.add(key)
metrics = {metric: percentage(row[metric])
for metric in ("latency", "throughput", "power", "energy")}
# Each may round to -100% alone; reciprocal metrics cannot both do so.
if mode == "throughput" and metrics["latency"] == metrics["throughput"] == -100:
metrics["latency"] = metrics["throughput"] = math.nan
invalid_pairs.append(key)
rows.append(Result(variant, row["arch"], row["model"], mode, pipeline, status, metrics))
if not rows:
raise ValueError("The ablation CSV contains no results")
if invalid_pairs:
warnings.warn(f"{len(invalid_pairs)} invalid reciprocal latency/throughput percentage pairs; "
f"displaying N/A. Examples: {invalid_pairs[:3]}", stacklevel=2)
return rows
def draw_metric(results: list[Result], metric: str, ylabel: str, output_dir: Path) -> None:
architectures = sorted({row.arch for row in results})
present_models = {row.model for row in results}
models = [model for model in ABLATION_DEFAULT_MODELS if model in present_models]
models += sorted(present_models - set(models))
variants = [name for name in VARIANT_COLORS if any(row.variant == name for row in results)]
lookup = {(row.arch, row.model, row.variant): row for row in results}
values = [row.metrics[metric] for row in results if math.isfinite(row.metrics[metric])]
lower, upper = min(0, min(values, default=0)), max(0, max(values, default=0))
padding = max(upper - lower, 1) * 0.35
fig, axes = plt.subplots(1, len(architectures), sharey=True, squeeze=False,
figsize=(7 * len(architectures), 5.2), layout="constrained")
handles = [Patch(facecolor=VARIANT_COLORS[name], label=name) for name in variants]
if len(values) < len(architectures) * len(models) * len(variants):
handles.append(Patch(facecolor="white", edgecolor="#B2BAC5", label="N/A: missing measurement"))
fig.legend(handles=handles, loc="outside upper center", ncol=4, frameon=False, fontsize=9.5)
width = 0.8 / len(variants)
mode, pipeline = results[0].mode, results[0].pipeline
for ax, arch in zip(axes[0], architectures):
ax.set_axisbelow(True)
ax.set_ylim(lower - padding if lower < 0 else 0, upper + padding)
ax.set_xlim(-0.5, len(models) - 0.5)
ax.set_xticks(range(len(models)), [model.upper() for model in models], fontsize=9.5)
ax.set_xlabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
f"{mode.capitalize()}, {pipeline} stage" + ("s" if pipeline != 1 else ""),
fontsize=11, weight="bold", labelpad=8)
ax.set_ylabel(ylabel, fontsize=10.5)
ax.yaxis.set_major_formatter(PercentFormatter(xmax=100, decimals=0))
ax.tick_params(axis="both", length=0, pad=6, labelleft=True)
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
ax.axhline(0, color="#536478", linewidth=1.2, linestyle="--")
for index, model in enumerate(models):
if index % 2 == 0:
ax.axvspan(index - 0.5, index + 0.5, facecolor="#F7F9FC", zorder=0)
for offset, variant in enumerate(variants):
x = index + (offset - (len(variants) - 1) / 2) * width
row = lookup.get((arch, model, variant))
value = row.metrics[metric] if row else math.nan
if not math.isfinite(value):
ax.text(x, padding * 0.05, "N/A", rotation=90, ha="center", va="bottom",
fontsize=8, color="#85909F")
continue
ax.bar(x, value, width=width * 0.9, color=VARIANT_COLORS[variant],
edgecolor="none", linewidth=0, zorder=2)
direction = -1 if value < 0 else 1
value_label(ax, x, value + direction * padding * 0.05, f"{value:+,.2f}%",
row.validation, direction)
save_figure(fig, f"{mode}_pipeline{pipeline}_{metric}", output_dir)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--input", type=Path, default=DEFAULT_RESULTS_PATH)
parser.add_argument("--output-dir", type=Path)
args = parser.parse_args()
if args.output_dir is None:
args.output_dir = args.input.parent / "plots"
configure_style()
results = load_results(args.input)
print(f"Results: {len(results)} configurations, "
f"{sum(row.validation != 'PASS' for row in results)} without PASS validation")
for mode, pipeline in sorted({(row.mode, row.pipeline) for row in results}):
rows = [row for row in results if row.mode == mode and row.pipeline == pipeline]
for metric, label in (
("latency", "Single-inference latency" if mode == "latency" else "Output interval"),
("throughput", "Throughput"), ("power", "Average power"), ("energy", "Energy per inference"),
):
if metric == "throughput" and mode == "latency":
continue
draw_metric(rows, metric, f"{label} change from baseline (%)", args.output_dir)
if __name__ == "__main__":
main()
validation/tools/pim/ablation/results_ablation.csv
+65
View File
@@ -0,0 +1,65 @@
variant,arch,model,mode,raptor_pipeline,functional_validation,latency,throughput,power,energy
baseline,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
no-plan,arch-a,googlenet,latency,1,PASS,+11513.29%,NA,-69.42%,+3452.69%
no-sync-no-plan,arch-a,googlenet,latency,1,PASS,+11512.99%,NA,-69.42%,+3452.59%
baseline,arch-a,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-a,googlenet,throughput,4,FAIL,+26.17%,-20.74%,+9.97%,+18.50%
no-plan,arch-a,googlenet,throughput,4,PASS,-100.00%,-100.00%,-62.73%,-100.00%
no-sync-no-plan,arch-a,googlenet,throughput,4,FAIL,+4094.63%,-97.62%,-59.86%,+1311.01%
baseline,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
no-sync-no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
baseline,arch-a,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-a,resnet18,throughput,4,FAIL,+110.65%,-52.55%,+43.25%,+177.48%
no-plan,arch-a,resnet18,throughput,4,PASS,+1598.93%,-94.12%,-51.04%,+664.87%
no-sync-no-plan,arch-a,resnet18,throughput,4,FAIL,+1174.21%,-92.16%,-45.06%,+543.74%
baseline,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
no-sync-no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
baseline,arch-a,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-a,resnet34,throughput,4,FAIL,+33.65%,-25.21%,+31.76%,+15.56%
no-plan,arch-a,resnet34,throughput,4,PASS,+1726.82%,-94.52%,-54.50%,+455.98%
no-sync-no-plan,arch-a,resnet34,throughput,4,FAIL,+813.41%,-89.04%,-49.77%,+206.84%
baseline,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
no-sync-no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
baseline,arch-a,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-a,vgg8,throughput,4,PASS,+2.33%,-1.72%,+18.49%,-0.98%
no-plan,arch-a,vgg8,throughput,4,PASS,+1251.16%,-92.58%,-56.68%,+577.04%
no-sync-no-plan,arch-a,vgg8,throughput,4,PASS,+1260.47%,-92.63%,-53.36%,+467.83%
baseline,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
no-plan,arch-b,googlenet,latency,1,PASS,+9849.64%,NA,-70.51%,+2833.89%
no-sync-no-plan,arch-b,googlenet,latency,1,PASS,+9849.55%,NA,-70.51%,+2833.89%
baseline,arch-b,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-b,googlenet,throughput,4,FAIL,+11.45%,-10.29%,+21.82%,+19.27%
no-plan,arch-b,googlenet,throughput,4,PASS,-100.00%,-100.00%,-63.39%,-100.00%
no-sync-no-plan,arch-b,googlenet,throughput,4,FAIL,+3348.28%,-97.10%,-61.58%,+1026.09%
baseline,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
no-plan,arch-b,resnet18,latency,1,PASS,+2935.63%,NA,-63.12%,+1019.81%
no-sync-no-plan,arch-b,resnet18,latency,1,PASS,+2936.09%,NA,-63.12%,+1019.97%
baseline,arch-b,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-b,resnet18,throughput,4,FAIL,+8.03%,-7.44%,+47.59%,+40.61%
no-plan,arch-b,resnet18,throughput,4,PASS,+1511.85%,-93.80%,-52.93%,+560.80%
no-sync-no-plan,arch-b,resnet18,throughput,4,FAIL,+867.12%,-89.67%,-45.41%,+359.83%
baseline,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-plan,arch-b,resnet34,latency,1,PASS,+2551.16%,NA,-62.51%,+893.70%
no-sync-no-plan,arch-b,resnet34,latency,1,PASS,+2550.91%,NA,-62.51%,+893.62%
baseline,arch-b,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-b,resnet34,throughput,4,FAIL,+28.25%,-22.13%,+49.76%,+57.30%
no-plan,arch-b,resnet34,throughput,4,PASS,+2427.17%,-96.05%,-49.58%,+938.62%
no-sync-no-plan,arch-b,resnet34,throughput,4,FAIL,+531.79%,-84.19%,-42.32%,+197.08%
baseline,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-sync,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
no-sync-no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
baseline,arch-b,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
no-sync,arch-b,vgg8,throughput,4,PASS,+23.91%,-19.82%,+25.67%,+23.93%
no-plan,arch-b,vgg8,throughput,4,PASS,+1158.70%,-92.07%,-58.95%,+408.33%
no-sync-no-plan,arch-b,vgg8,throughput,4,PASS,+1172.83%,-92.17%,-55.83%,+327.00%
1 variant arch model mode raptor_pipeline functional_validation latency throughput power energy
2 baseline arch-a googlenet latency 1 PASS +0.00% NA +0.00% +0.00%
3 no-sync arch-a googlenet latency 1 PASS +0.00% NA +0.00% -0.00%
4 no-plan arch-a googlenet latency 1 PASS +11513.29% NA -69.42% +3452.69%
5 no-sync-no-plan arch-a googlenet latency 1 PASS +11512.99% NA -69.42% +3452.59%
6 baseline arch-a googlenet throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
7 no-sync arch-a googlenet throughput 4 FAIL +26.17% -20.74% +9.97% +18.50%
8 no-plan arch-a googlenet throughput 4 PASS -100.00% -100.00% -62.73% -100.00%
9 no-sync-no-plan arch-a googlenet throughput 4 FAIL +4094.63% -97.62% -59.86% +1311.01%
10 baseline arch-a resnet18 latency 1 PASS +0.00% NA +0.00% +0.00%
11 no-sync arch-a resnet18 latency 1 PASS +0.00% NA +0.00% +0.00%
12 no-plan arch-a resnet18 latency 1 PASS +3191.30% NA -60.97% +1184.24%
13 no-sync-no-plan arch-a resnet18 latency 1 PASS +3191.30% NA -60.97% +1184.24%
14 baseline arch-a resnet18 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
15 no-sync arch-a resnet18 throughput 4 FAIL +110.65% -52.55% +43.25% +177.48%
16 no-plan arch-a resnet18 throughput 4 PASS +1598.93% -94.12% -51.04% +664.87%
17 no-sync-no-plan arch-a resnet18 throughput 4 FAIL +1174.21% -92.16% -45.06% +543.74%
18 baseline arch-a resnet34 latency 1 PASS +0.00% NA +0.00% +0.00%
19 no-sync arch-a resnet34 latency 1 PASS +0.00% NA +0.00% +0.00%
20 no-plan arch-a resnet34 latency 1 PASS +2361.37% NA -56.53% +969.99%
21 no-sync-no-plan arch-a resnet34 latency 1 PASS +2361.37% NA -56.53% +969.99%
22 baseline arch-a resnet34 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
23 no-sync arch-a resnet34 throughput 4 FAIL +33.65% -25.21% +31.76% +15.56%
24 no-plan arch-a resnet34 throughput 4 PASS +1726.82% -94.52% -54.50% +455.98%
25 no-sync-no-plan arch-a resnet34 throughput 4 FAIL +813.41% -89.04% -49.77% +206.84%
26 baseline arch-a vgg8 latency 1 PASS +0.00% NA +0.00% +0.00%
27 no-sync arch-a vgg8 latency 1 PASS +0.00% NA +0.00% +0.00%
28 no-plan arch-a vgg8 latency 1 PASS +1800.66% NA -44.65% +953.39%
29 no-sync-no-plan arch-a vgg8 latency 1 PASS +1800.66% NA -44.65% +953.39%
30 baseline arch-a vgg8 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
31 no-sync arch-a vgg8 throughput 4 PASS +2.33% -1.72% +18.49% -0.98%
32 no-plan arch-a vgg8 throughput 4 PASS +1251.16% -92.58% -56.68% +577.04%
33 no-sync-no-plan arch-a vgg8 throughput 4 PASS +1260.47% -92.63% -53.36% +467.83%
34 baseline arch-b googlenet latency 1 PASS +0.00% NA +0.00% +0.00%
35 no-sync arch-b googlenet latency 1 PASS +0.00% NA +0.00% -0.00%
36 no-plan arch-b googlenet latency 1 PASS +9849.64% NA -70.51% +2833.89%
37 no-sync-no-plan arch-b googlenet latency 1 PASS +9849.55% NA -70.51% +2833.89%
38 baseline arch-b googlenet throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
39 no-sync arch-b googlenet throughput 4 FAIL +11.45% -10.29% +21.82% +19.27%
40 no-plan arch-b googlenet throughput 4 PASS -100.00% -100.00% -63.39% -100.00%
41 no-sync-no-plan arch-b googlenet throughput 4 FAIL +3348.28% -97.10% -61.58% +1026.09%
42 baseline arch-b resnet18 latency 1 PASS +0.00% NA +0.00% +0.00%
43 no-sync arch-b resnet18 latency 1 PASS +0.00% NA +0.00% -0.00%
44 no-plan arch-b resnet18 latency 1 PASS +2935.63% NA -63.12% +1019.81%
45 no-sync-no-plan arch-b resnet18 latency 1 PASS +2936.09% NA -63.12% +1019.97%
46 baseline arch-b resnet18 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
47 no-sync arch-b resnet18 throughput 4 FAIL +8.03% -7.44% +47.59% +40.61%
48 no-plan arch-b resnet18 throughput 4 PASS +1511.85% -93.80% -52.93% +560.80%
49 no-sync-no-plan arch-b resnet18 throughput 4 FAIL +867.12% -89.67% -45.41% +359.83%
50 baseline arch-b resnet34 latency 1 PASS +0.00% NA +0.00% +0.00%
51 no-sync arch-b resnet34 latency 1 PASS +0.00% NA +0.00% +0.00%
52 no-plan arch-b resnet34 latency 1 PASS +2551.16% NA -62.51% +893.70%
53 no-sync-no-plan arch-b resnet34 latency 1 PASS +2550.91% NA -62.51% +893.62%
54 baseline arch-b resnet34 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
55 no-sync arch-b resnet34 throughput 4 FAIL +28.25% -22.13% +49.76% +57.30%
56 no-plan arch-b resnet34 throughput 4 PASS +2427.17% -96.05% -49.58% +938.62%
57 no-sync-no-plan arch-b resnet34 throughput 4 FAIL +531.79% -84.19% -42.32% +197.08%
58 baseline arch-b vgg8 latency 1 PASS +0.00% NA +0.00% +0.00%
59 no-sync arch-b vgg8 latency 1 PASS +0.00% NA +0.00% +0.00%
60 no-plan arch-b vgg8 latency 1 PASS +1784.97% NA -49.50% +849.73%
61 no-sync-no-plan arch-b vgg8 latency 1 PASS +1784.97% NA -49.50% +849.73%
62 baseline arch-b vgg8 throughput 4 PASS +0.00% +0.00% +0.00% +0.00%
63 no-sync arch-b vgg8 throughput 4 PASS +23.91% -19.82% +25.67% +23.93%
64 no-plan arch-b vgg8 throughput 4 PASS +1158.70% -92.07% -58.95% +408.33%
65 no-sync-no-plan arch-b vgg8 throughput 4 PASS +1172.83% -92.17% -55.83% +327.00%
validation/tools/pim/ablation/run_ablation.py
+15 -9
View File
@@ -45,6 +45,7 @@ VARIANTS = (
VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS}
COMPARISON_RESULTS_FILENAME = "results_comparison.csv"
ABLATION_RESULTS_FILENAME = "results_ablation.csv"
DEFAULT_RESULTS_PATH = Path(__file__).with_name(ABLATION_RESULTS_FILENAME)
CASE_FIELDS = ("arch", "model", "mode", "raptor_pipeline")
PERCENTAGE_FIELDS = (
("raptor_latency_ms", "latency"),
@@ -68,8 +69,7 @@ def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[
parser.add_argument(
"--out-dir",
type=Path,
default=DEFAULT_OUT_DIR,
help="Artifact root (default: validation/networks/pimcomp_models).",
help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_ablation.csv beside this script.",
)
add_models_argument(parser, ABLATION_DEFAULT_MODELS)
parser.add_argument(
@@ -160,9 +160,10 @@ def percentage_difference(value: str | None, reference: str | None) -> str:
baseline = float(reference) if reference is not None else math.nan
except ValueError:
return "NA"
if not math.isfinite(current) or not math.isfinite(baseline) or baseline == 0:
if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0:
return "NA"
return f"{(current - baseline) / baseline * 100:+.2f}%"
change = (current / baseline - 1) * 100
return f"{change:+.2f}%" if math.isfinite(change) else "NA"
def aggregate_results(
@@ -170,6 +171,7 @@ def aggregate_results(
variants: list[Variant],
selected_cases: frozenset[tuple[str, str]] | None = None,
selected_models: frozenset[str] | None = None,
output: Path | None = None,
) -> tuple[Path | None, list[str]]:
fields: list[str] | None = None
rows_by_variant: dict[str, list[dict[str, str]]] = {}
@@ -233,7 +235,8 @@ def aggregate_results(
int(row["raptor_pipeline"]),
)
)
output = out_dir / ABLATION_RESULTS_FILENAME
output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME
output.parent.mkdir(parents=True, exist_ok=True)
with output.open("w", newline="", encoding="utf-8") as stream:
writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n")
writer.writeheader()
@@ -247,7 +250,7 @@ def main(argv: list[str] | None = None) -> int:
print(f"Missing Pimcomp runner: {RUNNER}", file=sys.stderr)
return 1
variants = selected_variants(args.variants)
out_dir = args.out_dir.resolve()
out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR
case_sets = runner_argument_sets(forwarded)
if args.dry_run:
for variant in variants:
@@ -299,14 +302,17 @@ def main(argv: list[str] | None = None) -> int:
variants,
selected_cases,
frozenset(args.models),
output=DEFAULT_RESULTS_PATH if args.out_dir is None else None,
)
failed.extend(aggregation_failures)
if output is not None:
print(f"Ablation results: {output}")
print(f"[{output.name}]")
print(output.read_text(encoding="utf-8"), end="", flush=True)
if failed:
print("Failed: " + "; ".join(failed), file=sys.stderr)
return 1
return 0
if output is not None:
print(f"Ablation results: {output}", flush=True)
return 1 if failed else 0
if __name__ == "__main__":
validation/tools/pim/pimcomp/compare/.gitignore
+1
View File
@@ -0,0 +1 @@
plots/
validation/tools/pim/pimcomp/compare/README.md
+91 -45
View File
@@ -1,62 +1,108 @@
# Pimcomp model comparison
These scripts compare Raptor with Pimcomp on the models in
`validation/networks/pimcomp_models`.
`run_pimcomp_models.py` runs the supported model suite across the configured
architectures and simulation modes. `compare_raptor_pimcomp_model.py` is the
lower-level one-model comparison used by the suite runner.
Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
Run commands from the repository root with the repository virtual environment,
built release Raptor, Pimcomp, and `pimsim-nn`.
## Suite runner
Run all five models, Arch-A and Arch-B, latency, and throughput:
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```
### Options
Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
2/4/8, four workers, and a functional throughput batch of 64.
Use `--help` for the complete option reference.
| Option | Description and default |
| Option | Purpose |
|---|---|
| `-h`, `--help` | Show help and exit. |
| `--out-dir PATH` | Suite root; artifacts are placed below `<out-dir>/<model>/artifacts/`. Default: `validation/networks/pimcomp_models`. |
| `--common-dir PATH` | Shared root for per-model reference artifacts. Default: `<out-dir>/<model>/artifacts/common`. |
| `--models MODEL [...]` | Models to run: `vgg8`, `resnet18`, `resnet34`, `googlenet`, `yolo11n`. Default: all five; pass a subset to select specific models. |
| `--archs ARCH [...]` | Pimcomp hardware profiles to run. Default: `arch-a arch-b`. |
| `--mode {latency,throughput} [...]` | Simulation modes. Default: both. |
| `--only {raptor,pimcomp}` | Re-run only one compiler side and reuse the other side's existing report. Default: run both sides. |
| `--raptor-only` | Run Raptor without compiling, validating, or simulating Pimcomp. Default: off. |
| `--pipeline {1,2,4,8}` | Select one Raptor pipeline. Default: latency pipeline 1 and throughput pipelines 2, 4, and 8. |
| `--pimsim-time-ms INT` | Throughput convergence deadline. Default: `1000`. |
| `--batch-size INT` | Functional throughput batch size. Default: `128`. |
| `--timeout-seconds FLOAT` | Per-stage timeout; `0` means unlimited. Default: `0`. |
| `-j INT`, `--jobs INT` | Parallel comparison workers. Default: `4`. |
| `--clean` | Remove generated comparison artifacts and summaries, then exit. Default: off. |
| `--ablation-variant NAME` | Put generated artifacts below `<mode>[/pipelineN]/ablation/NAME` and write the summary below `ablation/NAME`. Default: none. |
| `--dry-run` | Print commands without modifying files. Default: off. |
| `--no-fast` | Disable fast throughput convergence. Default: off. |
| `--raptor-extra-arg=ARG` | Extra Raptor compiler argument; repeat for multiple arguments. Default: none. |
| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
| `--common-dir PATH` | Override the shared reference-artifact root. |
| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
| `--raptor-only` | Run Raptor without invoking Pimcomp. |
| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
| `--dry-run` | Print commands without modifying files. |
| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |
Arguments beginning with `--` must use the equals form when passed through:
Use the equals form for forwarded flags, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py \
--models vgg8 \
--raptor-extra-arg=--pim-disable-synchronization
```
With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
beside this script. An explicit `--out-dir PATH` writes
`PATH/results_comparison.csv`; selecting a subset updates only those rows.
Default model artifacts remain under `validation/networks/pimcomp_models`.
Artifacts use
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
written report, including failures without simulation artifacts. Missing or
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
compiler, simulation, or functional comparison fails.
The runner writes `results_comparison.csv` under the selected result root. It reuses
shared model inputs, outputs, and reference runners, and reuses Pimcomp
artifacts between pipelines in the same model/architecture/mode group. With
`--raptor-only`, only Raptor results are generated and Pimcomp is not invoked.
All generated files are kept below each model's `artifacts/` directory;
`--clean` also removes any `.lock` files left by an interrupted reference
generation.
Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
with a regeneration diagnostic. `--only raptor` requires compatible reports;
`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
results. Ordinary runs also rerun the group's first case, even if an old report
exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
Tool/build revisions and manually edited reference artifacts are not hashed;
rerun the relevant compiler after those change.
## One-model comparator
`compare_raptor_pimcomp_model.py` compares one ONNX model. Its required
arguments are `--model PATH` and `--out-dir PATH`; use `--help` for the full
lower-level interface. The suite runner supplies the model, hardware profile,
simulation mode, and reuse paths automatically.
`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
Use `--help` for hardware and simulation options. The suite runner supplies
these automatically.
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
`validate.py` uses `1e-3`/`1e-5`.
## Comparison charts
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
```
Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
`results_comparison.csv` beside the script and writes `plots/png/` and
`plots/svg/` there. Use `--input PATH` for a runner-generated CSV and
`--output-dir PATH` to override the destination. The plotter requires the full
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
configurations are rejected before plots are written.
| Plot family | Meaning |
|---|---|
| `validation_status` | Recorded functional status for each compiler and experiment. |
| `latency_*` | Single-inference latency, latency mode only. |
| `throughput_*` | Inferences/s, throughput mode only. |
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
| `power_*`, `energy_*` | Average power and energy per inference. |
| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |
Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
winning factor minus one, with parity at zero.
Failed validation is labeled `INVALID`; other non-PASS statuses keep their
names, and missing measurements show `N/A`. Relative bars require both
compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
must agree before being collapsed. `FAIL` alone does not identify the cause.
Plots consume the CSV's current contents; they do not run benchmarks or establish
publication provenance. The checked-in dataset is intentional. See the
[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).
## Metric interpretation
See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, units, numerical validity, and result provenance.
validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py
+62 -6
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import argparse
import fcntl
import gzip
import hashlib
import importlib.util
import json
import mmap
@@ -1162,6 +1163,60 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None:
return Path(value) if value else None
def pimcomp_reuse_context(
args: argparse.Namespace,
model: Path,
functional_model: Path,
hardware: dict[str, int],
) -> dict[str, Any]:
files = {"model": model, "functional_model": functional_model,
"config": args.pimcomp_config}
digests = {}
for path in set(files.values()):
with path.open("rb") as stream:
digests[path] = hashlib.file_digest(stream, "sha256").hexdigest()
return {
"version": 1,
"files": {name: {"path": str(path), "sha256": digests[path]}
for name, path in files.items()},
"hardware": hardware,
"settings": {name: getattr(args, name) for name in (
"pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline",
"pimcomp_replication", "pimcomp_model_name", "seed", "batch_size",
"threshold", "rtol", "no_fast", "skip_pimsim_nn",
)},
}
def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]:
try:
report = json.loads(path.read_text(encoding="utf-8"))
if (not isinstance(report, dict) or context is None
or report.get("pimcomp_reuse_context") != context
or report.get("pimcomp_model_source") != "original_onnx"):
raise ValueError("model, configuration, or experiment settings differ; legacy reports lack fingerprints")
for name, kind in (("pimcomp_validation", dict), ("pimcomp_performance", dict),
("pimcomp_instruction_summary", dict), ("paths", dict),
("failures", list), ("steps", list)):
if not isinstance(report.get(name), kind):
raise ValueError(f"missing or invalid {name}")
validation = CompareResult(**report["pimcomp_validation"])
if (type(validation.passed) is not bool or not isinstance(validation.max_diffs, dict)
or validation.status not in ("done", "failed", "skipped")):
raise ValueError("invalid functional validation result")
if not report["pimcomp_performance"]:
raise ValueError("missing performance result or failure status")
summary = report["pimcomp_instruction_summary"]
if not {"active_cores", "total_instructions", "op_counts"} <= summary.keys():
raise ValueError("incomplete instruction summary")
restore_side_records(report, "pimcomp", [], [])
for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"):
path_from_report(report, name)
return report
except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc:
raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc
def restore_side_records(
report: dict[str, Any],
side: str,
@@ -1579,6 +1634,11 @@ def main():
if loaded_hardware is not None:
hardware = loaded_hardware
reuse_context = None if args.raptor_only else try_stage(
failures, "Identify Pimcomp experiment", pimcomp_reuse_context,
args, model_path, functional_model_path, hardware,
)
model_io = try_stage(
failures,
"Load model inputs",
@@ -1597,12 +1657,7 @@ def main():
run_pimcomp = not args.raptor_only and not reuse_pimcomp
if reuse_pimcomp:
reused_pimcomp_report_path = reused_pimcomp_report_path.resolve()
if not reused_pimcomp_report_path.exists():
raise ValueError(f"Missing Pimcomp report: {reused_pimcomp_report_path}")
with open(reused_pimcomp_report_path, "r", encoding="utf-8") as f:
reused_pimcomp = json.load(f)
if reused_pimcomp.get("pimcomp_model_source") != "original_onnx":
raise ValueError("Reused Pimcomp artifacts were not generated from the original ONNX model")
reused_pimcomp = load_reused_pimcomp_report(reused_pimcomp_report_path, reuse_context)
pimcomp_validation = CompareResult(**reused_pimcomp["pimcomp_validation"])
pimcomp_perf = reused_pimcomp["pimcomp_performance"]
pimcomp_instr = reused_pimcomp["pimcomp_instruction_summary"]
@@ -2011,6 +2066,7 @@ def main():
"pimcomp_pipeline": args.pimcomp_pipeline,
"pimcomp_replication": args.pimcomp_replication,
"pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx",
"pimcomp_reuse_context": reuse_context,
"pimcomp_config": str(args.pimcomp_config),
"raptor_extra_args": args.raptor_extra_arg,
"reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
validation/tools/pim/pimcomp/compare/generate_plots.py
Executable
+514
View File
@@ -0,0 +1,514 @@
#!/usr/bin/env python3
"""Generate publication-ready Raptor vs Pimcomp comparison charts.
Usage:
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
Dependencies: matplotlib (no pandas or seaborn required).
By default, reads results_comparison.csv beside this script and writes plots/
beside it, with outputs grouped in png/ and svg/ under the output directory.
YOLO models without Pimcomp measurements get separate Raptor-only charts.
Comparisons use vertical bars for ratios, diverging advantages, and absolute
values, with compiler colors and patterns for the Raptor pipeline stages.
Throughput-mode latency is the output interval, a bottleneck-stage proxy,
not end-to-end inference latency. Separate throughput plots use inferences/s.
Energy is reported per inference; power is averaged over the simulation.
Numerical results whose compiler did not pass functional validation have a red
status following their value label; missing Pimcomp
measurements appear as 'N/A'. The fixed compiler palette is shared by
all metrics. Relative plots use Pimcomp / Raptor for latency, power and energy,
and Raptor / Pimcomp for throughput, so >1 favors Raptor. Diverging plots put the winner's
factor above (Raptor) or below (Pimcomp) parity, with 1x at the zero line.
"""
from __future__ import annotations
import argparse
import csv
import math
import sys
from dataclasses import dataclass
from itertools import groupby
from pathlib import Path
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.patches import Patch, Rectangle
from matplotlib.lines import Line2D
from matplotlib.ticker import FuncFormatter, MultipleLocator
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from plot_style import COLORS, HATCH_COLOR, TEXT_COLOR, configure_style, fit_value_labels, save_figure, value_label
COMPILER_COLORS = {"raptor": COLORS[0], "pimcomp": COLORS[2]}
STAGE_PATTERNS = {1: "", 2: "..", 4: "//", 8: "\\\\"}
ARCHITECTURES = ("arch-a", "arch-b")
MODELS = ("vgg8", "resnet18", "resnet34", "googlenet", "yolo11n")
CONFIGURATIONS = (("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8))
LINE_COLOR = "#A8B2C0"
VALIDATION_STYLES = {
"PASS": ("#168C8B", "white", ""),
"FAIL": ("#D18D32", HATCH_COLOR, ""),
"SKIPPED": ("#E7DFF0", TEXT_COLOR, ""),
"NA": ("#E8EDF1", TEXT_COLOR, ""),
}
@dataclass(frozen=True)
class Result:
arch: str
model: str
mode: str
pipeline: int
pimcomp_schedule: str
raptor_validation: str
pimcomp_validation: str
latency_speedup: float
power: float
energy: float
raptor_latency: float
pimcomp_latency: float
raptor_throughput: float
pimcomp_throughput: float
raptor_power: float
pimcomp_power: float
raptor_energy: float
pimcomp_energy: float
@property
def comparison_passed(self) -> bool:
return self.raptor_validation == self.pimcomp_validation == "PASS"
@property
def throughput_speedup(self) -> float:
return quotient(self.raptor_throughput, self.pimcomp_throughput)
def positive_float(text: str | None) -> float:
try:
value = float(text)
except (TypeError, ValueError):
return math.nan
return value if math.isfinite(value) and value > 0 else math.nan
def quotient(numerator: float, denominator: float) -> float:
if not (math.isfinite(numerator) and math.isfinite(denominator)):
return math.nan
return numerator / denominator
def validation_status(text: str) -> str:
status = text.strip().upper()
if status not in VALIDATION_STYLES:
raise ValueError(f"Unknown functional-validation status: {text!r}")
return status
def load_results(path: Path) -> list[Result]:
with path.open(newline="", encoding="utf-8-sig") as handle:
source = list(csv.DictReader(handle))
rows = []
expected = {(arch, model, mode, stage)
for arch in ARCHITECTURES for model in MODELS
for mode, stage in CONFIGURATIONS}
seen = {}
for line, row in enumerate(source, 2):
try:
arch, model, mode = (row[field].strip().lower() for field in ("arch", "model", "mode"))
stage = int(row["raptor_pipeline"])
except (KeyError, AttributeError, TypeError, ValueError) as exc:
raise ValueError(f"Invalid configuration key at CSV row {line}: {exc}") from exc
key = arch, model, mode, stage
if key not in expected:
raise ValueError(f"Unexpected configuration at CSV row {line}: {key}")
if key in seen:
raise ValueError(f"Duplicate configuration {key} at CSV rows {seen[key]} and {line}")
seen[key] = line
schedule = row["pimcomp_pipeline"].strip().lower()
if schedule != ("element" if mode == "latency" else "batch"):
raise ValueError(f"{key}: incompatible Pimcomp schedule {schedule!r}")
rows.append(Result(
arch=arch,
model=model,
mode=mode,
pipeline=stage,
pimcomp_schedule=schedule,
raptor_validation=validation_status(row["raptor_functional_validation"]),
pimcomp_validation=validation_status(row["pimcomp_functional_validation"]),
latency_speedup=quotient(positive_float(row["pimcomp_latency_ms"]),
positive_float(row["raptor_latency_ms"])),
power=quotient(positive_float(row["pimcomp_power_mw"]),
positive_float(row["raptor_power_mw"])),
energy=quotient(positive_float(row["pimcomp_energy_pj"]),
positive_float(row["raptor_energy_pj"])),
raptor_latency=positive_float(row["raptor_latency_ms"]),
pimcomp_latency=positive_float(row["pimcomp_latency_ms"]),
raptor_throughput=positive_float(row["raptor_throughput_samples_s"]),
pimcomp_throughput=positive_float(row["pimcomp_throughput_samples_s"]),
raptor_power=positive_float(row["raptor_power_mw"]),
pimcomp_power=positive_float(row["pimcomp_power_mw"]),
raptor_energy=positive_float(row["raptor_energy_pj"]),
pimcomp_energy=positive_float(row["pimcomp_energy_pj"]),
))
missing = expected - seen.keys()
if missing:
raise ValueError(f"Missing {len(missing)} expected configurations. Examples: {sorted(missing)[:5]}")
rows.sort(key=lambda r: (ARCHITECTURES.index(r.arch), MODELS.index(r.model), r.pipeline))
absolute_columns(rows)
return rows
def advantage_compiler(ratio: float) -> str | None:
if ratio > 1:
return "raptor"
if ratio < 1:
return "pimcomp"
return None
def figure_legend(fig, results: list[Result], field: str,
absolute=False, raptor_only=False):
compilers = ("raptor",) if raptor_only else ("raptor", "pimcomp")
if absolute:
measurements = [(row, compiler, getattr(row, f"{compiler}_{field}"))
for row, compiler in absolute_columns(results, raptor_only)]
else:
measurements = [(row, None, getattr(row, field)) for row in results]
available = [(row, compiler, value) for row, compiler, value in measurements
if math.isfinite(value)]
stages = {row.pipeline for row, compiler, _ in available if compiler != "pimcomp"}
handles = []
visible_compilers = {compiler if absolute else advantage_compiler(value)
for _, compiler, value in available} - {None}
if len(visible_compilers) > 1:
for compiler in compilers:
handles.append(Patch(facecolor=COMPILER_COLORS[compiler],
label=compiler.capitalize() + ("" if absolute else " advantage")))
handles += [Patch(facecolor="white", edgecolor=TEXT_COLOR, hatch=STAGE_PATTERNS[depth],
label=f"{depth} stage" + (" (latency)" if depth == 1 else "s"))
for depth in sorted(stages) if stages != {1}]
styles = []
if len(available) < len(measurements):
styles.append(Patch(facecolor="white", edgecolor="#B2BAC5",
label="N/A: missing measurement"))
handles += styles
if handles:
fig.legend(handles=handles, loc="outside upper center", ncol=(len(handles) + 1) // 2,
frameon=False, fontsize=9.5, handlelength=1.7, columnspacing=1.5,
handletextpad=0.6, labelspacing=0.4)
def absolute_columns(results: list[Result], raptor_only: bool = False) -> list[tuple[Result, str]]:
columns = []
for (arch, model), group in groupby(results, key=lambda row: (row.arch, row.model)):
rows = sorted(group, key=lambda row: row.pipeline)
for row in rows:
columns.append((row, "raptor"))
if row.mode == "latency" and not raptor_only:
columns.append((row, "pimcomp"))
throughput = [row for row in rows if row.mode == "throughput"]
if throughput and not raptor_only:
signatures = {
(row.pimcomp_schedule, row.pimcomp_validation,
*(None if math.isnan(value) else value for value in (
row.pimcomp_latency, row.pimcomp_throughput, row.pimcomp_power, row.pimcomp_energy)))
for row in throughput
}
if len(signatures) != 1:
raise ValueError(f"{arch}/{model}: inconsistent shared Pimcomp batch results across pipelines")
columns.append((throughput[0], "pimcomp"))
return columns
def label_groups(ax, results: list[Result], positions: list[float] | None = None) -> None:
if positions is None:
positions = list(range(len(results)))
centers, labels = [], []
start = 0
for index, (model, rows) in enumerate(groupby(results, key=lambda row: row.model)):
end = start + len(list(rows))
centers.append((positions[start] + positions[end - 1]) / 2)
labels.append(model.upper())
left = ax.get_xlim()[0] if start == 0 else (positions[start - 1] + positions[start]) / 2
right = ax.get_xlim()[1] if end == len(results) else (positions[end - 1] + positions[end]) / 2
if index % 2 == 0:
ax.axvspan(left, right, facecolor="#F7F9FC", zorder=0)
if index > 0:
ax.axvline(left, color="#DDE3EB", linewidth=0.9)
start = end
ax.set_xticks(centers, labels, fontsize=9.5)
def diverging_ratio(ratio: float) -> float:
return ratio - 1 if ratio >= 1 else 1 - 1 / ratio
def draw_validation(results: list[Result], output_dir: Path) -> None:
columns = absolute_columns(results)
configurations = list(dict.fromkeys((row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule,
compiler) for row, compiler in columns))
positions = []
for index, (mode, _, _) in enumerate(configurations):
gap = 0.4 if index and mode != configurations[index - 1][0] else 0
positions.append(positions[-1] + 1 + gap if index else 0)
configuration_x = dict(zip(configurations, positions))
headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor"
else "Pimcomp\nbatch" if mode == "throughput" else compiler.capitalize()
for mode, stage, compiler in configurations]
fig, axes = plt.subplots(1, 2, figsize=(14, 3.8), layout="constrained")
for ax, arch in zip(axes, ARCHITECTURES):
ax.set_xlim(positions[0] - 0.55, positions[-1] + 0.55)
ax.set_ylim(len(MODELS) - 0.5, -0.5)
centers, modes = [], []
for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]):
group_positions = [x for _, x in group]
centers.append((group_positions[0] + group_positions[-1]) / 2)
modes.append(mode.capitalize())
ax.set_xticks(centers, modes, fontsize=10)
ax.set_xticks(positions, headers, minor=True)
ax.tick_params(axis="x", which="minor", top=True, labeltop=True,
bottom=False, labelbottom=False, length=0, pad=6, labelsize=9)
ax.set_yticks(range(len(MODELS)), [model.upper() for model in MODELS], fontsize=9)
ax.tick_params(axis="both", length=0, pad=6)
architecture = "ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B"
ax.set_xlabel(architecture, fontsize=11, weight="bold", labelpad=8)
for row, compiler in columns:
if row.arch != arch:
continue
key = (row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule, compiler)
x, y = configuration_x[key], MODELS.index(row.model)
status = getattr(row, f"{compiler}_validation")
color, text_color, _ = VALIDATION_STYLES[status]
ax.add_patch(Rectangle((x - 0.5, y - 0.42), 1, 0.84,
facecolor=color, edgecolor="#657081", linewidth=0.5))
ax.text(x, y, "N/A" if status == "NA" else status,
color=text_color, ha="center", va="center", fontsize=9)
save_figure(fig, "validation_status", output_dir)
def draw_metric(results: list[Result], field: str, ylabel: str,
basename: str, output_dir: Path, diverging: bool = False) -> None:
data = [diverging_ratio(getattr(row, field)) if diverging else getattr(row, field)
for row in results]
available = [v for v in data if math.isfinite(v)]
upper = max(0, max(available, default=1.0))
lower = min(0, min(available, default=0)) if diverging else 0
padding = max(upper - lower, 1.0) * 0.3
y_max = math.ceil((upper + padding) * 2) / 2
y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0
y_interval = 1.0 if y_max - y_min > 4.5 else 0.5
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
layout="constrained")
figure_legend(fig, results, field)
for ax, arch in zip(axes, ARCHITECTURES):
selected = [row for row in results if row.arch == arch]
ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
fontsize=11, weight="bold", labelpad=8)
ax.set_axisbelow(True)
ax.set_ylim(y_min, y_max)
ax.set_xlim(-0.5, len(selected) - 0.5)
label_groups(ax, selected)
ax.tick_params(axis="y", labelleft=True, pad=5, length=0)
ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6)
ax.yaxis.set_major_locator(MultipleLocator(y_interval))
if diverging:
ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}×"))
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
ax.axhline(0 if diverging else 1, color="#536478", linewidth=1.2,
linestyle="--", zorder=3)
ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6)
for position, row in enumerate(selected):
ratio = getattr(row, field)
if not math.isfinite(ratio):
ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
fontsize=9.5, color="#85909F", fontstyle="italic")
continue
compiler = advantage_compiler(ratio)
statuses = [status for status in (row.raptor_validation, row.pimcomp_validation) if status != "PASS"]
status = "FAIL" if "FAIL" in statuses else next(iter(statuses), "PASS")
pattern = STAGE_PATTERNS[row.pipeline]
measurement = diverging_ratio(ratio) if diverging else ratio
ax.bar(position, measurement, width=0.72,
color=COMPILER_COLORS.get(compiler, "#657081"),
hatch=pattern,
edgecolor=TEXT_COLOR if pattern else "none",
linewidth=0.7 if pattern else 0.0, zorder=2)
direction = -1 if measurement < 0 else 1
factor = max(ratio, 1 / ratio) if diverging else ratio
value_label(ax, position, measurement + direction * (y_max - y_min) * 0.012,
f"{factor:.2f}×", status, direction, fontsize=9.1)
fit_value_labels(fig)
save_figure(fig, basename, output_dir)
def draw_absolute(results: list[Result], field: str, ylabel: str,
basename: str, output_dir: Path, scale: float = 1.0,
raptor_only: bool = False) -> None:
columns = absolute_columns(results, raptor_only)
available = [getattr(row, f"{compiler}_{field}") * scale
for row, compiler in columns
if math.isfinite(getattr(row, f"{compiler}_{field}"))]
y_max = max(available, default=1.0) * 1.35
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
layout="constrained")
figure_legend(fig, results, field, absolute=True, raptor_only=raptor_only)
for ax, arch in zip(axes, ARCHITECTURES):
selected = [(row, compiler) for row, compiler in columns if row.arch == arch]
positions = []
for index, (row, _) in enumerate(selected):
if index == 0:
positions.append(0.0)
continue
previous = selected[index - 1][0]
gap = 0.8 if row.model != previous.model else 0.4 if row.mode != previous.mode else 0.0
positions.append(positions[-1] + 1.0 + gap)
ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
fontsize=11, weight="bold", labelpad=8)
ax.set_axisbelow(True)
ax.set_ylim(0, y_max)
if positions:
ax.set_xlim(positions[0] - 0.9, positions[-1] + 0.9)
else:
ax.set_xlim(-0.9, 0.9)
label_groups(ax, [row for row, _ in selected], positions)
if {row.mode for row, _ in selected} == {"latency", "throughput"}:
centers, labels = [], []
for (_, mode), group in groupby(enumerate(selected), key=lambda item: (item[1][0].model, item[1][0].mode)):
group_positions = [positions[position] for position, _ in group]
centers.append((group_positions[0] + group_positions[-1]) / 2)
labels.append(mode.capitalize())
ax.set_xticks(centers, labels, minor=True)
ax.tick_params(axis="x", which="minor", length=0, pad=4, labelsize=8)
ax.set_ylabel(ylabel, fontsize=10.5)
if field == "throughput":
ax.ticklabel_format(axis="y", style="plain", useOffset=False)
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
ax.tick_params(axis="y", length=0, pad=6)
ax.tick_params(axis="x", which="major", length=0,
pad=22 if {row.mode for row, _ in selected} == {"latency", "throughput"} else 6)
for x, (row, compiler) in zip(positions, selected):
measurement = getattr(row, f"{compiler}_{field}") * scale
if not math.isfinite(measurement):
ax.text(x, y_max * 0.02, "N/A", rotation=90, ha="center",
va="bottom", fontsize=8, color="#85909F")
continue
status = getattr(row, f"{compiler}_validation")
pattern = STAGE_PATTERNS[row.pipeline] if compiler == "raptor" else ""
ax.bar(x, measurement, width=1.0,
color=COMPILER_COLORS[compiler],
hatch=pattern,
edgecolor=TEXT_COLOR if pattern else "none",
linewidth=0.7 if pattern else 0.0, zorder=2)
label = (f"{measurement:,.2f}".rstrip("0").rstrip(".")
if field == "throughput" else f"{measurement:.3g}")
value_label(ax, x, measurement + y_max * 0.012, label, status)
fit_value_labels(fig)
save_figure(fig, basename, output_dir)
def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1]
stages = sorted({row.pipeline for row in rows})
panels = [(model, color) for model, color in zip(MODELS, COLORS)
if any(row.model == model for row in rows)]
lookup = {(row.arch, row.model, row.pipeline): row for row in rows}
rates = {key: quotient(1000, row.raptor_latency) if row.mode == "latency" else row.raptor_throughput
for key, row in lookup.items()}
fig, axes = plt.subplots(len(ARCHITECTURES), len(panels), sharex=True, sharey="col",
figsize=(3 * len(panels), 7.2), squeeze=False, layout="constrained")
for column, (model, color) in enumerate(panels):
for index, arch in enumerate(ARCHITECTURES):
ax = axes[index, column]
keys = [(arch, model, stage) for stage in stages]
values = [rates.get(key, math.nan) for key in keys]
valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys]
ax.plot(stages, values, color=color, linewidth=1.8)
for stage, value, passed in zip(stages, values, valid):
if math.isfinite(value):
marker = ("s" if stage == 1 else "o") if passed else "x"
ax.plot(stage, value, color=color, marker=marker,
markersize=6, linestyle="none")
ax.set_xticks(stages)
ax.set_xlabel(model.upper(), fontsize=10.5, weight="bold", labelpad=8)
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
ax.tick_params(axis="both", length=0, pad=6, labelbottom=True, labelleft=True)
ax.ticklabel_format(axis="y", style="plain", useOffset=False)
if column == 0:
ax.set_ylabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
"Throughput (inferences/s)", fontsize=10.5)
for ax in axes.flat:
ax.set_ylim(bottom=0)
handles = [Line2D([], [], color="#657081", marker=marker, linestyle="none", label=label)
for marker, label in (("s", "Derived as 1000ms / latency_ms"),
("o", "Throughput measurement"))]
if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()):
handles.append(Line2D([], [], color="#424B58", marker="x", linestyle="none",
label="Raptor validation not PASS"))
fig.legend(handles=handles, loc="outside upper center", ncol=len(handles), frameon=False, fontsize=9.5)
fig.supxlabel("Raptor pipeline stages", fontsize=11)
save_figure(fig, "raptor_pipeline_throughput", output_dir)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
script_dir = Path(__file__).resolve().parent
parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv")
parser.add_argument("--output-dir", type=Path, default=script_dir / "plots")
args = parser.parse_args()
configure_style()
results = load_results(args.input)
valid = sum(row.comparison_passed for row in results)
print(f"Results: {len(results)} configurations, {valid} valid comparisons, "
f"{len(results)-valid} comparisons without two PASS statuses")
draw_validation(results, args.output_dir)
draw_pipeline_effect(results, args.output_dir)
yolo_only_models = {
model for model in MODELS if model.startswith("yolo")
and not any(math.isfinite(getattr(row, f"pimcomp_{field}"))
for row in results if row.model == model
for field in ("latency", "power", "energy"))
}
comparisons = [row for row in results if row.model not in yolo_only_models]
yolo_only = [row for row in results if row.model in yolo_only_models]
for metric, name, label, mode in [
("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None),
("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"),
("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"),
("power", "power", "Average power advantage (x)", None),
("energy", "energy", "Energy per inference advantage (x)", None),
]:
selected = [row for row in comparisons if mode is None or row.mode == mode]
draw_metric(selected, metric, label,
f"{name}_comparison", args.output_dir)
draw_metric(selected, metric, label,
f"{name}_diverging", args.output_dir, diverging=True)
for metric, name, label, scale, mode in [
("latency", "longest_stage_latency", "Longest stage latency (ms)", 1.0, None),
("latency", "latency", "Single-inference latency (ms)", 1.0, "latency"),
("power", "power", "Average power (mW)", 1.0, None),
("energy", "energy", "Energy per inference (mJ)", 1e-9, None),
("throughput", "throughput", "Throughput (inferences/s)", 1.0, "throughput"),
]:
for suffix, group, raptor_only in (
("absolute", comparisons, False), ("yolo", yolo_only, True)
):
selected = [row for row in group if mode is None or row.mode == mode]
if not selected:
continue
draw_absolute(selected, metric, label,
f"{name}_{suffix}", args.output_dir, scale,
raptor_only=raptor_only)
if __name__ == "__main__":
main()
validation/tools/pim/pimcomp/compare/results_comparison.csv
+41
View File
@@ -0,0 +1,41 @@
arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,423.80,482.60,431626597.00,541691691.00,raptor,1.14
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.86,1.04,1160.00,965.00,432.99,482.60,416224904.20,541691691.00,raptor,1.20
arch-a,vgg8,throughput,8,batch,PASS,PASS,1.63,1.04,615.00,965.00,308.12,482.60,616026711.90,541691691.00,pimcomp,1.57
arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394498463.12,13868866808.12,raptor,4.15
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.62,13.46,51.00,74.30,376.10,481.60,8024732847.00,6563451004.00,pimcomp,1.46
arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.92,13.46,50.20,74.30,368.24,481.60,9023016592.00,6563451004.00,pimcomp,1.48
arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
arch-a,resnet34,throughput,2,batch,PASS,FAIL,22.26,23.58,44.90,42.40,473.84,515.68,16417303180.00,12283627410.00,raptor,1.06
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.37,23.58,36.50,42.40,415.90,515.68,17019112320.00,12283627410.00,pimcomp,1.16
arch-a,resnet34,throughput,8,batch,PASS,FAIL,41.04,23.58,24.40,42.40,346.15,515.68,17351854700.00,12283627410.00,pimcomp,1.74
arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.88,231.19,5513234559.92,14547744283.24,raptor,6.44
arch-a,googlenet,throughput,2,batch,PASS,FAIL,10.16,18.26,98.50,54.80,544.21,435.11,6029999329.00,8164787803.00,raptor,1.80
arch-a,googlenet,throughput,4,batch,PASS,FAIL,11.96,18.26,83.60,54.80,473.16,435.11,6741923526.00,8164787803.00,raptor,1.53
arch-a,googlenet,throughput,8,batch,PASS,FAIL,20.66,18.26,48.40,54.80,348.76,435.11,8421537146.00,8164787803.00,pimcomp,1.13
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,120.93,NA,8.27,NA,313.20,NA,41830757130.00,NA,NA,NA
arch-a,yolo11n,throughput,4,batch,PASS,FAIL,92.06,NA,10.90,NA,351.03,NA,39451980450.00,NA,NA,NA
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,104.07,NA,9.61,NA,330.76,NA,44002712680.00,NA,NA,NA
arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
arch-b,vgg8,throughput,2,batch,PASS,PASS,1.18,1.07,846.00,932.00,327.05,409.47,429183159.10,439288835.80,pimcomp,1.10
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.92,1.07,1090.00,932.00,383.82,409.47,464794094.50,439288835.80,raptor,1.17
arch-b,vgg8,throughput,8,batch,PASS,PASS,1.62,1.07,616.00,932.00,276.49,409.47,524048148.70,439288835.80,pimcomp,1.51
arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
arch-b,resnet18,throughput,2,batch,PASS,FAIL,13.08,13.43,76.40,74.40,440.79,484.80,6399942695.00,6569655551.00,raptor,1.03
arch-b,resnet18,throughput,4,batch,PASS,FAIL,20.68,13.43,48.40,74.40,327.57,484.80,7777606323.00,6569655551.00,pimcomp,1.54
arch-b,resnet18,throughput,8,batch,PASS,FAIL,21.17,13.43,47.20,74.40,317.64,484.80,8897039159.00,6569655551.00,pimcomp,1.58
arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
arch-b,resnet34,throughput,4,batch,PASS,FAIL,39.61,22.85,25.20,43.80,315.01,493.33,14629161660.00,11392215660.00,pimcomp,1.74
arch-b,resnet34,throughput,8,batch,PASS,FAIL,49.34,22.85,20.30,43.80,285.76,493.33,16591020470.00,11392215660.00,pimcomp,2.16
arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422799605.92,9182062336.24,raptor,3.38
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.83,21.39,102.00,46.80,520.79,417.87,5669213312.00,9014726090.00,raptor,2.18
arch-b,googlenet,throughput,4,batch,PASS,FAIL,14.47,21.39,69.10,46.80,401.01,417.87,6835460477.00,9014726090.00,raptor,1.48
arch-b,googlenet,throughput,8,batch,PASS,FAIL,37.28,21.39,26.80,46.80,245.38,417.87,11453848400.00,9014726090.00,pimcomp,1.75
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,118.15,NA,8.46,NA,287.59,NA,37062981240.00,NA,NA,NA
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,94.18,NA,10.60,NA,316.45,NA,37424491250.00,NA,NA,NA
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,142.86,NA,7.00,NA,290.68,NA,41525990280.00,NA,NA,NA
1 arch model mode raptor_pipeline pimcomp_pipeline raptor_functional_validation pimcomp_functional_validation raptor_latency_ms pimcomp_latency_ms raptor_throughput_samples_s pimcomp_throughput_samples_s raptor_power_mw pimcomp_power_mw raptor_energy_pj pimcomp_energy_pj better_compiler speedup
2 arch-a vgg8 latency 1 element PASS PASS 1.52 7.99 NA NA 321.05 200.11 487455139.04 1597904071.12 raptor 5.26
3 arch-a vgg8 throughput 2 batch PASS PASS 0.91 1.04 1100.00 965.00 423.80 482.60 431626597.00 541691691.00 raptor 1.14
4 arch-a vgg8 throughput 4 batch PASS PASS 0.86 1.04 1160.00 965.00 432.99 482.60 416224904.20 541691691.00 raptor 1.20
5 arch-a vgg8 throughput 8 batch PASS PASS 1.63 1.04 615.00 965.00 308.12 482.60 616026711.90 541691691.00 pimcomp 1.57
6 arch-a resnet18 latency 1 element PASS FAIL 14.03 58.18 NA NA 455.67 238.39 6394498463.12 13868866808.12 raptor 4.15
7 arch-a resnet18 throughput 2 batch PASS FAIL 11.73 13.46 85.20 74.30 503.11 481.60 6628073894.00 6563451004.00 raptor 1.15
8 arch-a resnet18 throughput 4 batch PASS FAIL 19.62 13.46 51.00 74.30 376.10 481.60 8024732847.00 6563451004.00 pimcomp 1.46
9 arch-a resnet18 throughput 8 batch PASS FAIL 19.92 13.46 50.20 74.30 368.24 481.60 9023016592.00 6563451004.00 pimcomp 1.48
10 arch-a resnet34 latency 1 element PASS FAIL 29.72 66.71 NA NA 411.81 277.49 12238321706.68 18511425563.68 raptor 2.24
11 arch-a resnet34 throughput 2 batch PASS FAIL 22.26 23.58 44.90 42.40 473.84 515.68 16417303180.00 12283627410.00 raptor 1.06
12 arch-a resnet34 throughput 4 batch PASS FAIL 27.37 23.58 36.50 42.40 415.90 515.68 17019112320.00 12283627410.00 pimcomp 1.16
13 arch-a resnet34 throughput 8 batch PASS FAIL 41.04 23.58 24.40 42.40 346.15 515.68 17351854700.00 12283627410.00 pimcomp 1.74
14 arch-a googlenet latency 1 element PASS PASS 9.78 62.92 NA NA 563.88 231.19 5513234559.92 14547744283.24 raptor 6.44
15 arch-a googlenet throughput 2 batch PASS FAIL 10.16 18.26 98.50 54.80 544.21 435.11 6029999329.00 8164787803.00 raptor 1.80
16 arch-a googlenet throughput 4 batch PASS FAIL 11.96 18.26 83.60 54.80 473.16 435.11 6741923526.00 8164787803.00 raptor 1.53
17 arch-a googlenet throughput 8 batch PASS FAIL 20.66 18.26 48.40 54.80 348.76 435.11 8421537146.00 8164787803.00 pimcomp 1.13
18 arch-a yolo11n latency 1 element PASS FAIL 147.49 NA NA NA 290.17 NA 42797722129.00 NA NA NA
19 arch-a yolo11n throughput 2 batch PASS FAIL 120.93 NA 8.27 NA 313.20 NA 41830757130.00 NA NA NA
20 arch-a yolo11n throughput 4 batch PASS FAIL 92.06 NA 10.90 NA 351.03 NA 39451980450.00 NA NA NA
21 arch-a yolo11n throughput 8 batch PASS FAIL 104.07 NA 9.61 NA 330.76 NA 44002712680.00 NA NA NA
22 arch-b vgg8 latency 1 element PASS PASS 1.53 7.15 NA NA 292.56 173.77 448581679.04 1242814988.12 raptor 4.66
23 arch-b vgg8 throughput 2 batch PASS PASS 1.18 1.07 846.00 932.00 327.05 409.47 429183159.10 439288835.80 pimcomp 1.10
24 arch-b vgg8 throughput 4 batch PASS PASS 0.92 1.07 1090.00 932.00 383.82 409.47 464794094.50 439288835.80 raptor 1.17
25 arch-b vgg8 throughput 8 batch PASS PASS 1.62 1.07 616.00 932.00 276.49 409.47 524048148.70 439288835.80 pimcomp 1.51
26 arch-b resnet18 latency 1 element PASS FAIL 15.38 64.38 NA NA 400.67 201.66 6160441550.12 12983432281.12 raptor 4.19
27 arch-b resnet18 throughput 2 batch PASS FAIL 13.08 13.43 76.40 74.40 440.79 484.80 6399942695.00 6569655551.00 raptor 1.03
28 arch-b resnet18 throughput 4 batch PASS FAIL 20.68 13.43 48.40 74.40 327.57 484.80 7777606323.00 6569655551.00 pimcomp 1.54
29 arch-b resnet18 throughput 8 batch PASS FAIL 21.17 13.43 47.20 74.40 317.64 484.80 8897039159.00 6569655551.00 pimcomp 1.58
30 arch-b resnet34 latency 1 element PASS FAIL 27.95 107.20 NA NA 397.21 206.48 11103517952.68 22134174654.68 raptor 3.83
31 arch-b resnet34 throughput 2 batch PASS FAIL 22.28 22.85 44.90 43.80 438.24 493.33 14218683390.00 11392215660.00 raptor 1.03
32 arch-b resnet34 throughput 4 batch PASS FAIL 39.61 22.85 25.20 43.80 315.01 493.33 14629161660.00 11392215660.00 pimcomp 1.74
33 arch-b resnet34 throughput 8 batch PASS FAIL 49.34 22.85 20.30 43.80 285.76 493.33 16591020470.00 11392215660.00 pimcomp 2.16
34 arch-b googlenet latency 1 element PASS PASS 11.22 37.94 NA NA 483.27 242.04 5422799605.92 9182062336.24 raptor 3.38
35 arch-b googlenet throughput 2 batch PASS FAIL 9.83 21.39 102.00 46.80 520.79 417.87 5669213312.00 9014726090.00 raptor 2.18
36 arch-b googlenet throughput 4 batch PASS FAIL 14.47 21.39 69.10 46.80 401.01 417.87 6835460477.00 9014726090.00 raptor 1.48
37 arch-b googlenet throughput 8 batch PASS FAIL 37.28 21.39 26.80 46.80 245.38 417.87 11453848400.00 9014726090.00 pimcomp 1.75
38 arch-b yolo11n latency 1 element PASS FAIL 168.14 NA NA NA 245.31 NA 41248047514.00 NA NA NA
39 arch-b yolo11n throughput 2 batch PASS FAIL 118.15 NA 8.46 NA 287.59 NA 37062981240.00 NA NA NA
40 arch-b yolo11n throughput 4 batch PASS FAIL 94.18 NA 10.60 NA 316.45 NA 37424491250.00 NA NA NA
41 arch-b yolo11n throughput 8 batch PASS FAIL 142.86 NA 7.00 NA 290.68 NA 41525990280.00 NA NA NA
validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
+53 -36
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import argparse
import csv
import json
import math
import os
import shlex
import shutil
@@ -43,6 +44,7 @@ COMPARISONS = (
("throughput", 8, "batch"),
)
RESULTS_FILENAME = "results_comparison.csv"
DEFAULT_RESULTS_PATH = Path(__file__).with_name(RESULTS_FILENAME)
@dataclass(frozen=True)
@@ -128,9 +130,9 @@ def write_results_csv(
comparisons: tuple[tuple[str, int, str], ...] = COMPARISONS,
ablation_variant: str | None = None,
) -> Path:
output = (root or SUITE) / RESULTS_FILENAME
output = DEFAULT_RESULTS_PATH if root is None else root / RESULTS_FILENAME
if ablation_variant is not None:
output = output.parent / "ablation" / ablation_variant / RESULTS_FILENAME
output = (root or SUITE) / "ablation" / ablation_variant / RESULTS_FILENAME
fields = (
"arch",
"model",
@@ -247,18 +249,26 @@ def write_results_csv(
def performance_values(performance: dict) -> dict[str, float | None]:
fields = {"latency": ("latency_ms", "average_latency_ms"),
"throughput": ("throughput",), "power": ("average_power_mw",),
"energy": ("average_energy_pj",)}
values = dict.fromkeys(fields)
if performance.get("error") or performance.get("skipped") or performance.get("unsupported"):
return values
parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
return {
"latency": (
performance.get("latency_ms")
or performance.get("average_latency_ms")
or parsed.get("latency_ms")
or parsed.get("average_latency_ms")
),
"throughput": performance.get("throughput") or parsed.get("throughput"),
"power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
"energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
}
for metric, names in fields.items():
for value in (source.get(name) for source in (performance, parsed) for name in names):
try:
number = float(value)
except (TypeError, ValueError):
continue
if math.isfinite(number) and number > 0:
values[metric] = number
break
if performance.get("output_count", parsed.get("output_count")) == 0:
for metric in ("latency", "throughput", "energy"):
values[metric] = None
return values
def comparison_passed(report: dict) -> bool:
@@ -283,7 +293,9 @@ def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
def format_value(value: float | None) -> str:
return "NA" if value is None else f"{value:.2f}"
if value is None or not math.isfinite(value) or value <= 0:
return "NA"
return f"{value:.6g}"
def print_stage(title: str, color: str) -> None:
@@ -352,15 +364,12 @@ def comparison_command(
raptor_extra_args: list[str] | tuple[str, ...] = (),
raptor_only: bool = False,
reuse_raptor_report: Path | None = None,
reuse_pimcomp_dir: Path | None = None,
reuse_pimcomp_report: Path | None = None,
) -> list[str]:
time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
reuse_args = []
if reuse_raptor_report is not None:
reuse_args.extend(["--reuse-raptor-report", str(reuse_raptor_report)])
if reuse_pimcomp_dir is not None:
reuse_args.extend(["--reuse-pimcomp-dir", str(reuse_pimcomp_dir)])
if reuse_pimcomp_report is not None:
reuse_args.extend(["--reuse-pimcomp-report", str(reuse_pimcomp_report)])
return [
@@ -430,6 +439,12 @@ def comparison_command_for(
reuse_shared_pimcomp: bool,
) -> list[str]:
report = spec.output_dir / "pimcomp/comparison_report.json"
reuse_pimcomp_report = None
if not args.raptor_only:
if args.only == "raptor":
reuse_pimcomp_report = report
elif reuse_shared_pimcomp:
reuse_pimcomp_report = spec.shared_pimcomp_dir.parent / "comparison_report.json"
return comparison_command(
spec.model,
spec.functional_model,
@@ -450,20 +465,21 @@ def comparison_command_for(
if args.only == "pimcomp"
else None
),
reuse_pimcomp_dir=(
spec.shared_pimcomp_dir
if reuse_shared_pimcomp and args.only != "raptor" and not args.raptor_only
else None
),
reuse_pimcomp_report=report if args.only == "raptor" and not args.raptor_only else None,
reuse_pimcomp_report=reuse_pimcomp_report,
)
def pimcomp_artifact_ready(output_dir: Path) -> bool:
return all(
(output_dir / name).is_file()
for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
) and (output_dir.parent / "comparison_report.json").is_file()
def pimcomp_report_stamp(output_dir: Path) -> tuple[int, int, int] | None:
try:
stat = (output_dir.parent / "comparison_report.json").stat()
except FileNotFoundError:
return None
return stat.st_ino, stat.st_mtime_ns, stat.st_ctime_ns
def pimcomp_report_ready(output_dir: Path, previous: tuple[int, int, int] | None) -> bool:
stamp = pimcomp_report_stamp(output_dir)
return stamp is not None and stamp != previous
def run_comparison_jobs(
@@ -533,7 +549,7 @@ def main() -> int:
parser.add_argument(
"--out-dir",
type=Path,
help="Suite root; generated artifacts go below each model's artifacts/ directory (default: validation/networks/pimcomp_models).",
help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_comparison.csv beside this script.",
)
parser.add_argument(
"--common-dir",
@@ -687,11 +703,6 @@ def main() -> int:
required = comparison_dir / "pimcomp/comparison_report.json"
if not required.exists():
missing_reuse.append(str(required))
elif args.only == "raptor":
if json.loads(required.read_text(encoding="utf-8")).get(
"pimcomp_model_source"
) != "original_onnx":
missing_reuse.append(f"{required} (not generated from the original ONNX model)")
if missing_reuse:
parser.error(
f"--only {args.only} needs existing comparison artifacts: "
@@ -713,7 +724,7 @@ def main() -> int:
f"Comparison jobs: "
f"{sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())}"
)
print(f"Results root: {out_dir or SUITE}")
print(f"Artifacts root: {out_dir or SUITE}")
print("=" * 72)
print_stage("Prepare shared artifacts", STAGE_COLORS["Build runner"])
@@ -799,6 +810,10 @@ def main() -> int:
print_directly = min(args.jobs, len(comparison_specs)) == 1
try:
with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-pimcomp-")) as log_dir:
previous_reports = {
spec.shared_pimcomp_dir: pimcomp_report_stamp(spec.shared_pimcomp_dir)
for spec in anchor_specs
}
anchor_failed, log_offset = run_comparison_jobs(
anchor_jobs,
args.jobs,
@@ -812,7 +827,9 @@ def main() -> int:
comparison_command_for(
spec,
args,
reuse_shared_pimcomp=pimcomp_artifact_ready(spec.shared_pimcomp_dir),
reuse_shared_pimcomp=pimcomp_report_ready(
spec.shared_pimcomp_dir, previous_reports[spec.shared_pimcomp_dir]
),
),
None,
)
validation/tools/pim/pimcomp/correctness/README.md
+39 -97
View File
@@ -1,24 +1,18 @@
# Pimcomp correctness experiments
Run from the repository root using the repository virtual environment, built
release Raptor, Pimcomp frontend/backend, and `pimsim-nn`.
## Synchronization ordering
The synchronization experiment checks whether repeated PIMCOMP and Raptor
programs preserve cross-core global-memory generations under legal execution
schedules. It builds a small two-convolution model for Arch-A, Arch-B, and
Arch-C, finds cross-core `ST`/`LD` dependencies, and compares greedy,
bounded-stall, randomized, and adversarial runs using the Rust simulator's
provenance trace. Byte-identical batch inputs isolate intermediate-memory
ordering from the separate host-input lifetime issue.
Checks whether repeated PIMCOMP and Raptor programs preserve cross-core
global-memory generations under legal execution schedules. A two-convolution
model runs on Arch-A/B/C with greedy, bounded-stall, randomized, and adversarial
schedules; Rust provenance traces identify cross-core `ST`/`LD` dependencies.
Byte-identical batch inputs isolate intermediate-memory ordering from host-input
lifetime. Diagnostic scheduling does not change the performance oracle.
The experiment records static dependency evidence, dynamic provenance,
functional output comparisons, architecture contract classifications, and
per-artifact PIMCOMP/Raptor conclusions. Diagnostic scheduling changes only
the functional simulator's execution order; it does not modify PIMCOMP or the
`pimsim-nn` performance oracle.
Prerequisites are the repository virtual environment, the release Raptor
compiler, built PIMCOMP frontend/backend, and the existing `pimsim-nn` build.
Run from the repository root with a new or empty output directory:
Use a new or empty output directory:
```bash
.venv/bin/python \
@@ -27,102 +21,50 @@ Run from the repository root with a new or empty output directory:
--batch-size 4 --seed 0 --self-check
```
The directory contains:
Outputs: `adversarial_memory_sync_report.{json,md}` and per-architecture
evidence. Contract classifications use
[`architecture_evidence.json`](synchronization/architecture_evidence.json).
- [`run_experiment.py`](synchronization/run_experiment.py): orchestration,
classification, reporting, and self-checks.
- [`global_memory.py`](synchronization/global_memory.py): artifact compilation,
dependency analysis, simulator execution, and provenance helpers.
- [`architecture_contract.py`](synchronization/architecture_contract.py) and
[`architecture_evidence.json`](synchronization/architecture_evidence.json):
conservative architecture-contract evidence and labels.
## Batch prefill reproduction
The main outputs are `adversarial_memory_sync_report.json` and
`adversarial_memory_sync_report.md`, with per-architecture evidence below the
same output root.
## Pimcomp batch prefill reproduction
Pimcomp's batch scheduler currently emits an incomplete standalone program for
models containing post operations. The generated `VerificationInfo.json` uses a
negative `source_address` to identify the preceding node, but the batch
verifier resolves that address by copying the provider tensor directly from
ONNX Runtime's `intermediate_result` table. It does not read the simulated
global-memory buffer.
The corresponding post-operation scheduling call is commented out in
Pimcomp's batch scheduler omits post-operation producers for some Pool, ReLU,
Concat, and Reshape tensors. Its verifier resolves negative `source_address`
entries using ONNX Runtime intermediates rather than simulated global memory.
The corresponding scheduling call is disabled in
[`BatchPipelineSchedule.cpp`](../../../../../third_party/PIMCOMP-NN/backend/BatchPipelineSchedule.cpp#L1535).
Consequently, Pool, ReLU, Concat, and Reshape provider tensors can be consumed
by later `LD` instructions without any generated `ST` producer. The comparison
exporter initializes those host-side intermediate buffers to zero, which makes
the missing computation visible in Rust functional validation.
For the checked-in GoogLeNet throughput/pipeline2 artifact, 39 provider tensors
are referenced by batch loads. Nineteen have generated stores; twenty are
never written. Preloading the provider tensors with the same ONNX Runtime
intermediates used by Pimcomp's verifier makes the exported program pass. This
reproduces the verifier's input contract; it does not repair Pimcomp's batch
schedule.
Run the default reproduction from the repository root:
Rust validation starts those buffers at zero, exposing the missing computation.
```bash
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py
```
The launcher accepts an alternate comparison directory, model, work directory,
and shared reference-artifact directory:
The default uses GoogLeNet Arch-A throughput/pipeline2. Optional positional
arguments are comparison directory, model, work directory, and shared reference
directory; the default work directory is `correctness/prefill/` below the
comparison artifacts.
```bash
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py \
validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2 \
validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx \
validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2/correctness/prefill \
validation/networks/pimcomp_models/googlenet/artifacts/common
```
The experiment compares the original memory image with one prefilled using
[`batch_memory.py`](prefill/batch_memory.py). In the recorded case, 20 of 39
loaded provider tensors have no store producer; prefill reduces maximum
reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's
verifier contract; it does not repair the standalone batch program.
Without the optional work-directory argument, the experiment uses the same
`correctness/prefill/` directory below the comparison artifacts.
## ResNet BatchNorm gap
It runs the exported artifact once with its original memory image and once
with [`batch_memory.py`](prefill/batch_memory.py), then compares both
outputs with the recorded native reference. The expected GoogLeNet result is a
baseline maximum difference near `6.70705` and a prefilled maximum difference
near `4.05e-6`.
Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
removes BatchNormalization without folding its affine transform into weights
and bias. Its [verifier](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
substitutes identity BatchNorm parameters, so agreement with that verifier does
not establish equivalence to the original ONNX model.
The issue is in Pimcomp batch scheduling/validation semantics, not in the Rust
simulator's vector-length interpretation. Vector lengths remain element counts
as specified by the reference ISA.
## ResNet BatchNorm correctness gap
Pimcomp has a separate correctness limitation in its ResNet element pipeline.
`BatchNormalization` is an ONNX operation. The frontend's
[`fuse_operators()` pass](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
marks it as fused and removes the node from the
scheduled graph, but the released path does not fold the BatchNorm affine
transform into the convolution weights and bias. The corresponding verifier
workaround in [`verification.py`](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
replaces BatchNorm parameters with identity values (scale and variance equal
to one, bias and mean equal to zero) before running ONNX Runtime.
Therefore Pimcomp's native verifier and exported element program agree with
each other, but they do not implement the original ResNet ONNX model. On the
current ResNet-18 Arch-A latency artifact, using the same input:
Recorded ResNet-18 Arch-A latency errors on the same input:
| Comparison | Maximum absolute difference |
|---|---:|
| Pimcomp native verifier vs Rust export | `6.7e-8` |
| Rust export vs original ONNX reference | `4.8294563` |
The Pimcomp output ranges from approximately `-0.187` to `0.220`, while the
original ONNX output ranges from `-3.572` to `4.834`; 462 of 1000 final
elements differ by more than one. This is not a Rust simulator or Python
exporter regression. It is a Pimcomp model-semantics mismatch caused by
dropping BatchNorm numerics. The same issue affects the ResNet-34 latency
artifact. VGG and GoogLeNet do not show this particular mismatch because they
do not contain the same ResNet BatchNorm path.
The latency comparison intentionally uses the original ONNX reference, so
these Pimcomp rows must remain `FAIL` until Pimcomp folds BatchNorm correctly
or the comparison explicitly uses a BatchNorm-neutralized reference.
The same gap affects ResNet-34 latency. Comparison rows use the original ONNX
reference and must remain `FAIL` until BatchNorm numerics are implemented
correctly. This is independent of simulator vector lengths, which remain
element counts.
validation/tools/pim/plot_style.py
+137
View File
@@ -0,0 +1,137 @@
"""Shared appearance and PNG/SVG export for PIM result plots."""
from pathlib import Path
from math import ceil
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.patches import Patch
from matplotlib.path import Path as PlotPath
from matplotlib.patheffects import AbstractPathEffect
from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath
TEXT_COLOR = "#243449"
HATCH_COLOR = "#C62828"
COLORS = ("#2B65A1", "#168C8B", "#D18D32", "#8062A5", "#B65D60")
class CenteredHatch(AbstractPathEffect):
"""Anchor dot and diagonal hatch patterns to each rectangle's center."""
def draw_path(self, renderer, gc, tpath, affine, rgbFace=None):
hatch = gc.get_hatch() or ""
base_gc = renderer.new_gc()
base_gc.copy_properties(gc)
base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\"))
renderer.draw_path(base_gc, tpath, affine, rgbFace)
base_gc.restore()
bounds = affine.transform_path(tpath).get_extents()
center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2
vertices = []
diagonal = 2 ** -0.5
for symbols, dx, dy in (("/", diagonal, diagonal), ("\\", diagonal, -diagonal)):
count = sum(hatch.count(symbol) for symbol in symbols)
if count == 0:
continue
spacing = renderer.points_to_pixels(12 / count)
half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2
half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2
extent = ceil(half_span / spacing)
for step in range(-extent, extent + 1):
x = center_x - dy * step * spacing
y = center_y + dx * step * spacing
vertices.extend(((x - dx * half_length, y - dy * half_length),
(x + dx * half_length, y + dy * half_length)))
line_gc = renderer.new_gc()
line_gc.copy_properties(gc)
line_gc.set_hatch(None)
line_gc.set_foreground(gc.get_hatch_color())
line_gc.set_linewidth(gc.get_hatch_linewidth())
line_gc.set_clip_path(TransformedPath(tpath, affine))
if vertices:
renderer.draw_path(line_gc, PlotPath(vertices, [PlotPath.MOVETO, PlotPath.LINETO] * (len(vertices) // 2)),
IdentityTransform())
if "." in hatch:
spacing = renderer.points_to_pixels(12 / hatch.count("."))
x_extent = ceil(bounds.width / (2 * spacing))
y_extent = ceil(bounds.height / (2 * spacing))
centers = [(center_x + x * spacing, center_y + y * spacing)
for x in range(-x_extent, x_extent + 1)
for y in range(-y_extent, y_extent + 1)]
line_gc.set_linewidth(0)
renderer.draw_markers(line_gc, PlotPath.unit_circle(),
Affine2D().scale(renderer.points_to_pixels(0.9)),
PlotPath(centers), IdentityTransform(), gc.get_hatch_color())
line_gc.restore()
def configure_style() -> None:
plt.rcParams.update({
"font.family": "DejaVu Sans",
"axes.spines.top": False,
"axes.spines.right": False,
"axes.labelcolor": TEXT_COLOR,
"text.color": TEXT_COLOR,
"xtick.color": TEXT_COLOR,
"ytick.color": TEXT_COLOR,
"axes.edgecolor": "#CFD6DF",
"savefig.facecolor": "white",
"figure.facecolor": "white",
})
def value_label(ax, x, y, label: str, status: str = "PASS", direction: int = 1,
fontsize: float = 8) -> None:
alignment = "bottom" if direction > 0 else "top"
rotation = 90 if direction > 0 else -90
text = ax.text(x, y, label, rotation=rotation, ha="center", va=alignment,
fontsize=fontsize, color=TEXT_COLOR)
if status != "PASS":
height = text.get_window_extent(ax.figure.canvas.get_renderer()).height
offset = direction * (height * 72 / ax.figure.dpi + 3)
ax.annotate("INVALID" if status == "FAIL" else status, (x, y),
xytext=(0, offset), textcoords="offset points",
rotation=rotation, ha="center", va=alignment, fontsize=fontsize,
color=HATCH_COLOR)
def fit_value_labels(fig) -> None:
"""Reserve vertical space for rotated labels using their rendered size."""
for ax in fig.axes:
for text in ax.texts:
text.set_in_layout(False)
fig.canvas.draw()
renderer = fig.canvas.get_renderer()
limits = []
for ax in fig.axes:
bounds = ax.get_window_extent(renderer)
labels = []
for text in ax.texts:
x, y = text.xy if hasattr(text, "xy") else text.get_position()
anchor = ax.transData.transform((x, y))[1]
extent = text.get_window_extent(renderer)
labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height,
max(0, extent.y1 - anchor + 3) / bounds.height))
lower, upper = ax.get_ylim()
for _ in range(12):
lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)])
limits.append((lower, upper))
for ax in fig.axes:
ax.set_ylim(min(lower for lower, _ in limits), max(upper for _, upper in limits))
def save_figure(fig, basename: str, output_dir: Path) -> None:
for patch in fig.findobj(Patch):
hatch = patch.get_hatch() or ""
if any(symbol in hatch for symbol in "./\\"):
patch.set_path_effects([CenteredHatch()])
for extension in ("png", "svg"):
target = output_dir / extension / f"{basename}.{extension}"
target.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(target, dpi=220, bbox_inches="tight", pad_inches=0.04)
print(target)
plt.close(fig)