nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add ablation study
Validate Operations / validate-operations (push) Has been cancelled
Details

Browse Source 
normalize names and artifact paths
This commit is contained in:
NiccoloN
2026-08-20 17:58:02 +02:00
parent add20e56eb
commit b009e1ff08
67 changed files with 1573 additions and 993 deletions
Download Patch File Download Diff File Expand all files Collapse all files
validation/networks/pimcomp_models/README.md
+66 -62
View File
@@ -1,38 +1,38 @@
# PIMCOMP comparison models
# Pimcomp comparison models
This directory contains the four networks evaluated in
[PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
[Pimcomp: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
additional compiler comparison model.
See the runner-generated [results.csv](results.csv) for the current comparison
See the runner-generated [results_comparison.csv](results_comparison.csv) for the current comparison
results. Rows are retained separately for each model, architecture, mode, and
pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
Running the runner with `--arch arch-b` or `--arch arch-c` appends those
architecture rows without replacing the existing `arch-a` entries.
the Raptor and Pimcomp artifacts; rows without a generated report contain `NA`.
Use `--archs` to select the architecture rows to generate; existing rows for
other architectures remain unchanged.
## Models and provenance
| Directory | Model | Input | Provenance |
|--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`. |
| `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as PIMCOMP's frontend does. |
| `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as Pimcomp's frontend does. |
| `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). |
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [Pimcomp VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
| `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit Pimsim-ready GoogLeNet model.
It removes the two LRN nodes and terminal Softmax from the original model,
so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
so that the comparison covers only operations scheduled by Pimcomp and supported by Pimsim.
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit Pimsim-ready YOLO11n model.
It removes the Softmax nodes from the original model,
so that the compiled artifact can be simulated in pimsim-nn.
so that the compiled artifact can be simulated in Pimsim.
## Unsupported and ignored operations
PIMCOMP's frontend accepts exactly these ONNX operations:
Pimcomp's frontend accepts exactly these ONNX operations:
```text
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
@@ -40,14 +40,14 @@ Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
```
`Constant` is consumed as frontend metadata rather than emitted as a PIMCOMP
`Constant` is consumed as frontend metadata rather than emitted as a Pimcomp
node. Every other ONNX operation is unsupported: the frontend prints
`operation: <type> not considered` and stops at the first occurrence. Thus the
complete unsupported set is the complement of the allowlist above for the
model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and
`Resize` nodes.
PIMCOMP's low-latency scheduler, hierarchy mapper, and genetic algorithm use
Pimcomp's low-latency scheduler, hierarchy mapper, and genetic algorithm use
this complete explicit no-consider set:
```text
@@ -63,7 +63,7 @@ specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its
consumer. These transformations do not make an otherwise standalone ignored
operation timed.
`pimsim-nn` consumes PIM ISA instructions. It supports every named opcode
Pimsim consumes Pim ISA instructions. It supports every named opcode
in the shared serialized range except `vsoftmax` (opcode 21),
which is rejected explicitly in both JSON and binary input. It
silently ignores no opcode; unknown names and numbers are errors.
@@ -71,12 +71,12 @@ silently ignores no opcode; unknown names and numbers are errors.
These boundaries explain the dedicated artifacts:
- GoogLeNet's two LRN nodes and terminal Softmax perform real computation but
are ignored by PIMCOMP, so the common latency artifact removes them. Its
are ignored by Pimcomp, so the common latency artifact removes them. Its
inference Dropout and shape-only Reshape can remain without adding compute.
- YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can
run in `pimsim-nn`. Every other node, including MatMul, Transpose, and the
run in Pimsim. Every other node, including MatMul, Transpose, and the
final detection-decoding tail, remains present and timed by Raptor. No
PIMCOMP latency is reported because its frontend stops at `Split` and also
Pimcomp latency is reported because its frontend stops at `Split` and also
lacks `Resize`; compiling that prefix would not represent YOLO11n.
The authoritative lists are in
@@ -85,16 +85,16 @@ The authoritative lists are in
[`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and
[`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp).
The PIMCOMP authors did not publish the ONNX checkpoints used by the paper.
Running PIMCOMP's frontend on the three Model Zoo files above produces JSON
graphs exactly equal to PIMCOMP-NN's bundled `resnet18.json`, `resnet34.json`,
The Pimcomp authors did not publish the ONNX checkpoints used by the paper.
Running Pimcomp's frontend on the three Model Zoo files above produces JSON
graphs exactly equal to Pimcomp's bundled `resnet18.json`, `resnet34.json`,
and `googlenet.json`.
There is no VGG-8 artifact in the ONNX Model Zoo or any PIMCOMP-NN revision.
There is no VGG-8 artifact in the ONNX Model Zoo or any Pimcomp revision.
The included VGG-8 therefore has deterministic random weights and is suitable
for compiler and simulator comparison, not paper-accuracy reproduction. The
paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the
published PIMCOMP graphs and ResNet Model Zoo artifacts use ImageNet shapes.
published Pimcomp graphs and ResNet Model Zoo artifacts use ImageNet shapes.
Current SHA-256 checksums:
@@ -113,10 +113,10 @@ The files in
[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
encode Table V's explicit resource parameters.
Each profile subdirectory contains pre-generated latency and throughput
`pimsim-nn` configs plus its matching mesh; comparison and validation reference
Pimsim configs plus its matching mesh; comparison and validation reference
these canonical artifacts directly.
| Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout |
| Config | Cores | Crossbars/core | Crossbar | Cell | Pimcomp layout |
|------------------------------|------------------:|---------------:|------------|------:|-----------------|
| `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
| `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
@@ -125,10 +125,10 @@ these canonical artifacts directly.
`adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
The paper does not give a two-dimensional core topology for Arch-A/B, so the
factorizations above preserve core count but cannot reproduce unpublished NoC
placement details. Released PIMCOMP-NN has no chip-count field; Arch-C is
placement details. Released Pimcomp has no chip-count field; Arch-C is
therefore flattened to 64 cores and does not model chip boundaries.
The remaining latency and power values come from PIMCOMP-NN's released default
The remaining latency and power values come from Pimcomp's released default
configuration. Consequently, instruction/resource comparisons are
reproducible, but absolute paper power and energy numbers are not.
@@ -147,9 +147,9 @@ cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
Do not build either project with `ninja` directly.
## Compile with PIMCOMP
## Compile with Pimcomp
PIMCOMP-NN reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
Pimcomp reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
select one paper profile, and restore it when the shell exits:
```bash
@@ -162,7 +162,7 @@ trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json"
```
The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile
The Model Zoo files map exactly to Pimcomp's bundled model names, so compile
them directly:
```bash
@@ -179,7 +179,7 @@ cd "$PIMCOMP/build"
./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES
```
VGG-8 first needs PIMCOMP's JSON frontend. Use a temporary ONNX copy because
VGG-8 first needs Pimcomp's JSON frontend. Use a temporary ONNX copy because
the released frontend rewrites the input batch dimension in place:
```bash
@@ -201,62 +201,66 @@ random placement code occasionally segfaults; an unchanged retry succeeded in
the observed cases.
The paper's optimizer uses a genetic algorithm with population 200 and up to
1000 iterations. The checked-out PIMCOMP submodule already has both paper
1000 iterations. The checked-out Pimcomp submodule already has both paper
settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness
evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS`
to control its parallelism; otherwise OpenMP uses the available CPUs. The GA
uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.
## Compare Raptor and PIMCOMP
## Compare Raptor and Pimcomp
The comparison driver uses one random input and one native ONNX-MLIR reference,
compiles both instruction streams, runs both through `pimsim-nn`, runs
compiles both instruction streams, runs both through Pimsim, runs
functional validation through `pim-simulator`, and writes Markdown and JSON
reports.
To reproduce all configured architectures and both latency/throughput modes,
use the model-by-model runner. It verifies the paper GA settings, expects Raptor
To reproduce the default `arch-a`/`arch-b` architectures and both
latency/throughput modes, use the model-by-model runner. Use `--archs` to
specify a different architecture set. It verifies the paper GA settings, expects Raptor
and the existing `third_party/PIMCOMP-NN/build` tree to already be built, then
runs the comparisons in parallel and regenerates `results.csv` from the JSON
runs the comparisons in parallel and regenerates `results_comparison.csv` from the JSON
reports:
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```
Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
Use `--archs arch-a --mode latency` for only the Arch-A latency experiment, or
`--archs arch-a arch-b arch-c` to run all three architectures.
Each model directory has a shared ignored `common/` directory containing
Each model directory has a shared ignored `artifacts/common/` directory containing
`inputs/`, `outputs/`, and the native `runner/`.
Pimsim-nn configs and network meshes remain canonical under
Pimsim configs and network meshes remain canonical under
`validation/pimsim_configs/pimcomp/` and are referenced in place.
Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
artifacts remain under each comparison directory; PIMCOMP outputs are prepared
Architecture- and pipeline-specific `raptor/`, `simulation/`, and Pimcomp
artifacts remain under each `artifacts/<arch>/<mode>[/pipelineN]/` comparison
directory; ablation variants use
`artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`. Pimcomp outputs are prepared
once per model/architecture/mode and linked into the other pipeline directories;
`comparison_report.{md,json}`
live under its `pimcomp/`. The frontend regenerates
one isolated `models/JSON/` graph because PIMCOMP requires that relative
one isolated `models/JSON/` graph because Pimcomp requires that relative
layout; it is removed after a successful backend run and the shared submodule
model directory is never modified. The original ONNX model is passed to the
frontend unchanged.
PIMCOMP's source tree and build directory remain unchanged at runtime. Use
Pimcomp's source tree and build directory remain unchanged at runtime. Use
`--models vgg8` to run one model, `--mode throughput` to select one mode,
`--pipeline 4` to select one throughput pipeline, `--only raptor` or
`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
to inspect every command, or `--out-dir PATH` to keep results outside
`validation/`. Use `--clean` to remove generated comparison artifacts and
summaries. Selecting a subset replaces only those comparison rows and
recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
summaries, including stale reference lock files. Selecting a subset replaces only those comparison rows and
recomputes the aggregate `results_comparison.csv`; missing shared inputs, outputs, or the
reference runner are generated even for an isolated run. Use `--jobs 4` to cap
parallel comparisons. The per-stage timeout is unlimited by default; pass a
positive `--timeout-seconds` value to impose one.
Throughput comparisons default to `pimsim-nn --fast` with a 1000 ms
convergence deadline. Add `--no-fast` for authoritative full-duration runs.
PIMCOMP receives the original ONNX model, and its frontend applies native
Pimcomp receives the original ONNX model, and its frontend applies native
BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
The runner continues after a failed model so all reports are produced.
The known PIMCOMP batch-scheduling correctness issue and a reproducible
The known Pimcomp batch-scheduling correctness issue and a reproducible
reference-intermediate prefill experiment are documented in
[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
@@ -265,9 +269,9 @@ Arch-A low-latency example:
```bash
RAPTOR_ROOT=$PWD
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py" \
--model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/artifacts/arch-a/latency" \
--pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
--core-count 168 \
--crossbar-count 96 \
@@ -279,10 +283,10 @@ RAPTOR_ROOT=$PWD
```
Use the same command with
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released Pimcomp cannot
compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
as YOLO11n would produce a misleading latency, so no PIMCOMP number is
as YOLO11n would produce a misleading latency, so no Pimcomp number is
reported for this model.
For Arch-A high throughput, use `--pimsim-mode throughput
@@ -303,18 +307,18 @@ generated report.
The functional and non-functional simulators intentionally consume different
artifacts:
- Raptor and PIMCOMP are validated against the native ONNX-MLIR reference as
- Raptor and Pimcomp are validated against the native ONNX-MLIR reference as
FP32 programs in the Rust simulator. Raptor's emitted program is already
FP32. The PIMCOMP-to-Rust export expands its element-addressed storage and
FP32. The Pimcomp-to-Rust export expands its element-addressed storage and
byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector
`imm_len` fields as element counts.
- PIMCOMP's original `SimulationInfo.gz` is copied unchanged for `pimsim-nn`.
PIMCOMP hardcodes `setbw 8, 8` and one byte per element without performing
- Pimcomp's original `SimulationInfo.gz` is copied unchanged for Pimsim.
Pimcomp hardcodes `setbw 8, 8` and one byte per element without performing
numerical quantization; this artifact is used only for latency estimation.
- Raptor's original FP32 artifact remains unchanged for functional validation.
A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its
byte-addressed storage and transfer sizes from four bytes to one byte per
element. Vector `imm_len` fields remain element counts. Like PIMCOMP's
element. Vector `imm_len` fields remain element counts. Like Pimcomp's
artifact, this view is not numerically valid and is used only for a fair
non-functional comparison.
@@ -326,10 +330,10 @@ Current Raptor status:
- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
It removes the two LRN nodes and terminal softmax that PIMCOMP does not
It removes the two LRN nodes and terminal softmax that Pimcomp does not
schedule.
- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
`512x1024` arrays can therefore be compiled by PIMCOMP but not compared
`512x1024` arrays can therefore be compiled by Pimcomp but not compared
exactly with Raptor.
Do not change the hardware profile to bypass either limitation; that would no
@@ -365,7 +369,7 @@ python3 -m venv .venv
.venv/bin/python -m pip install numpy onnx onnxruntime colorama
# Run every configured comparison in parallel.
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```
Copy reports back without transferring large compiler artifacts: