add plot generation scripts adjust validation tools add up-to-date results
This commit is contained in:
NiccoloN
+87
-219
@@ -1,265 +1,133 @@
|
||||
# Raptor validation
|
||||
|
||||
`validate.py` validates every ONNX model below a selected directory. For each
|
||||
model it can:
|
||||
|
||||
1. compile an ONNX-MLIR reference library and runner;
|
||||
2. generate deterministic random inputs;
|
||||
3. compile Pim artifacts with Raptor;
|
||||
4. run the reference implementation and functional Pim simulator;
|
||||
5. compare their outputs;
|
||||
6. run Pimsim to report latency, throughput, power, and energy.
|
||||
|
||||
Run the script from the repository root with the repository Python environment.
|
||||
`validate.py` discovers ONNX models recursively, compiles a native ONNX-MLIR
|
||||
reference and Raptor artifacts, compares functional simulator outputs, and
|
||||
optionally measures Pimsim latency, throughput, power, and energy.
|
||||
Run from the repository root using `.venv`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A built Raptor compiler, normally
|
||||
`build_release/Release/bin/onnx-mlir`.
|
||||
- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
|
||||
- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
|
||||
- The Rust toolchain used by the functional simulator.
|
||||
- The Rust functional simulator under
|
||||
`backend-simulators/pim/pim-simulator`, unless overridden.
|
||||
- A built `pimsim-nn` under
|
||||
`backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
|
||||
skipped or its path is overridden.
|
||||
- Built release Raptor and ONNX-MLIR runtime headers.
|
||||
- `numpy`, `onnx`, and `colorama` installed in the repository environment.
|
||||
- Rust toolchain and `backend-simulators/pim/pim-simulator`.
|
||||
- Built `backend-simulators/pim/pimsim-nn/build`, unless performance
|
||||
simulation is skipped.
|
||||
|
||||
## Basic usage
|
||||
See the [build guide](../README.md#build).
|
||||
|
||||
Validate the complete operation suite:
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations \
|
||||
--verbose \
|
||||
--raptor-extra-arg=--pim-detect-communication-deadlock \
|
||||
--raptor-extra-arg=--pim-export-spatial-dataflow=none
|
||||
--raptor-extra-arg=--pim-detect-communication-deadlock
|
||||
```
|
||||
|
||||
Validate one operation category or case:
|
||||
Run the complete operations suite outside the sandbox so parallel workers can
|
||||
start. For a focused case, change `--operations-dir` to
|
||||
`validation/operations/gemm/small` or a network directory such as
|
||||
`validation/networks/yolo11n/depth_04`.
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations/gemm/small
|
||||
```
|
||||
Defaults run latency (one input) and throughput (pipeline 4, four distinct
|
||||
inputs). Both reuse reference artifacts, and every throughput output is
|
||||
compared with its own reference. Models run with four workers by default.
|
||||
Results go to `validation_results.csv` in the selected model tree.
|
||||
|
||||
Validate a network or network slice:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/networks/yolo11n/depth_04
|
||||
```
|
||||
|
||||
`--operations-dir` may point to any directory tree containing `.onnx` files.
|
||||
The script discovers them recursively and writes `validation_results.csv` in
|
||||
that directory while retaining separate latency and throughput terminal tables.
|
||||
|
||||
## Pim validation tools
|
||||
|
||||
- [Pimcomp model suite](networks/pimcomp_models/README.md)
|
||||
- [Pimcomp model comparison tools](tools/pim/pimcomp/compare/README.md)
|
||||
- [Pimcomp correctness study](tools/pim/pimcomp/correctness/README.md)
|
||||
- [Raptor compiler ablation study](tools/pim/ablation/README.md)
|
||||
|
||||
## Validation modes
|
||||
|
||||
The default mode runs latency and throughput in one validation job. Latency
|
||||
uses one input, while throughput uses `--pipeline=4` with four distinct inputs.
|
||||
Both modes reuse the generated input batch, native runner, and reference
|
||||
outputs, and every throughput output is compared with its own reference.
|
||||
|
||||
Use `--compile-only` to build the reference runner and Pim artifacts without
|
||||
executing either implementation:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations/gemm/small \
|
||||
--compile-only
|
||||
```
|
||||
|
||||
Use `--run-only` to reuse those artifacts and perform input generation,
|
||||
reference execution, simulation, and comparison:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations/gemm/small \
|
||||
--run-only
|
||||
```
|
||||
|
||||
`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
|
||||
with a diagnostic if its required compiled artifacts are missing.
|
||||
|
||||
## Parallel execution and output
|
||||
|
||||
Models run in parallel using all available CPUs by default. Set the worker
|
||||
count with `-j` or `--jobs`:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations \
|
||||
--jobs 8
|
||||
```
|
||||
Append `--compile-only` to build artifacts without execution, then
|
||||
`--run-only` to reuse them for input generation, execution, and comparison.
|
||||
These flags are mutually exclusive; run-only fails if compiled artifacts are
|
||||
missing. `--clean` removes generated artifacts and stale locks, and requires
|
||||
neither compiler nor include paths.
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
Use `--help` for the full command reference.
|
||||
|
||||
| Option | Purpose / default |
|
||||
|---|---|
|
||||
| `-h`, `--help` | Print command help and exit. |
|
||||
| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
|
||||
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
|
||||
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
|
||||
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
|
||||
| `--non-functional-simulator-build-dir PATH` | Pimsim build directory. Defaults to the in-tree build. |
|
||||
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
|
||||
| `--skip-non-functional-simulation` | Skip Pimsim latency, throughput, power, and energy measurement. |
|
||||
| `--no-fast` | Disable fast throughput convergence for authoritative full-duration Pimsim experiments. |
|
||||
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
|
||||
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
|
||||
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
|
||||
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
|
||||
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
|
||||
| `--core-count INT` | Pim core count passed to Raptor. Defaults to the Arch-A value, `168`. |
|
||||
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
|
||||
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
|
||||
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
|
||||
| `--clean` | Remove generated validation artifacts and exit. |
|
||||
| `--compile-only` | Compile reference and Pim artifacts without execution or comparison. |
|
||||
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
|
||||
| `--verbose` | Print passing per-stage and subprocess logs, plus average Pim pass timings. |
|
||||
| `--raptor-path PATH` | Compiler binary; required except for cleanup. |
|
||||
| `--onnx-include-dir PATH` | Runtime headers; required except for cleanup. |
|
||||
| `--operations-dir PATH` | Model tree; default: `validation/operations`. |
|
||||
| `--simulator-dir PATH` | Override the in-tree functional simulator crate. |
|
||||
| `--non-functional-simulator-build-dir PATH` | Override the in-tree Pimsim build. |
|
||||
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Performance profile; default: Arch-A. |
|
||||
| `--core-count N`, `--crossbar-count N`, `--crossbar-size N` | Raptor resources; defaults: Arch-A (168, 96, 128). |
|
||||
| `--skip-non-functional-simulation` | Skip performance measurement. |
|
||||
| `--no-fast` | Use full-duration throughput measurement. |
|
||||
| `--threshold FLOAT`, `--relative-threshold FLOAT` | Output tolerances; defaults: `1e-3`, `1e-5`. |
|
||||
| `--seed N` | Input RNG seed; default: 0. |
|
||||
| `--raptor-extra-arg=ARG` | Forward compiler arguments; repeat as needed. |
|
||||
| `--command-timeout-seconds FLOAT` | Subprocess timeout; default: 1000000 seconds. |
|
||||
| `-j N`, `--jobs N` | Parallel workers; default: 4. |
|
||||
| `--compile-only`, `--run-only`, `--clean` | Compile, reuse, or remove artifacts. |
|
||||
| `--verbose` | Include passing subprocess logs and average Pim pass timings. |
|
||||
|
||||
Arguments beginning with `--` that are passed through to Raptor should use the
|
||||
equals form:
|
||||
Forwarded options beginning with `--` require the equals form, for example
|
||||
`--raptor-extra-arg=--pim-detect-communication-deadlock`.
|
||||
|
||||
```bash
|
||||
--raptor-extra-arg=--pim-detect-communication-deadlock
|
||||
```
|
||||
## Performance and result interpretation
|
||||
|
||||
## Hardware profiles and non-functional simulation
|
||||
The selected [hardware profile](networks/pimcomp_models/README.md#hardware-profiles)
|
||||
must match the core/crossbar flags. A mismatch disables performance simulation
|
||||
and prints the incompatible values; functional validation still runs.
|
||||
Canonical configs and meshes live under `validation/pimsim_configs/pimcomp/`.
|
||||
|
||||
The selected Pimcomp profile must match `--core-count`, `--crossbar-count`, and
|
||||
`--crossbar-size`. A mismatch disables only non-functional simulation and
|
||||
prints the incompatible values; functional validation still runs.
|
||||
Throughput defaults to `pimsim-nn --fast` with a 1000 ms convergence deadline.
|
||||
It compares consecutive two-round windows at 1% tolerance and falls back to
|
||||
full-duration results if convergence fails. Use `--no-fast` for authoritative
|
||||
experiments. See [metric definitions](../docs/evaluation/methodology.md#metric-interpretation)
|
||||
for measurement windows, power, and energy interpretation.
|
||||
|
||||
The checked-in profiles are under
|
||||
`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
|
||||
`latency_config.json`, while throughput uses
|
||||
`throughput_config_<time>ms.json` and the mesh beside it.
|
||||
Functional compilation/output comparison determines PASS/FAIL. Performance
|
||||
failures show `ERROR` without changing a functional PASS; explicit unsupported
|
||||
operations show `UNSUPPORTED`. Pimsim's `vsoftmax` is unsupported.
|
||||
Skipped measurements remain visibly skipped.
|
||||
|
||||
Throughput measurement defaults to `pimsim-nn --fast` with a 1000 ms
|
||||
convergence deadline. Fast mode compares consecutive two-round windows with a
|
||||
fixed 1% tolerance and falls back to the legacy full-duration result if it
|
||||
does not converge. Use `--no-fast` for authoritative experiments.
|
||||
Exit codes: `0` for all models passing or successful cleanup; `1` for validation
|
||||
failure, an invalid model directory, or no ONNX files; `2` for invalid arguments.
|
||||
|
||||
Use `--skip-non-functional-simulation` when latency, throughput, power, and energy are not required.
|
||||
The summary reports non-functional results as measured, failed, unsupported, or
|
||||
skipped.
|
||||
## Artifacts and suites
|
||||
|
||||
Overall PASS/FAIL is determined by compilation and functional output
|
||||
comparison. A non-functional simulation failure remains visible as `ERROR` in
|
||||
the corresponding latency, throughput, power, or energy columns but does not
|
||||
change a functional PASS.
|
||||
|
||||
Pimsim does not currently implement the `vsoftmax` instruction. When its
|
||||
explicit unsupported-op diagnostic is encountered, Softmax validations retain
|
||||
their functional PASS and show `UNSUPPORTED` in the non-functional columns.
|
||||
Other Pimsim failures remain `ERROR`.
|
||||
|
||||
## Generated artifacts
|
||||
|
||||
Generated files are grouped below an `artifacts/` directory beside each model
|
||||
or operation case. This keeps checked-in ONNX files and generated trees
|
||||
separate and lets `--clean` remove the complete workspace, including stale
|
||||
validation lock files:
|
||||
The regular validator uses these paths beside each model:
|
||||
|
||||
| Path | Contents |
|
||||
|---|---|
|
||||
| `artifacts/inputs.csv` | Generated inputs, one batch entry per line. |
|
||||
| `artifacts/inputs/`, `artifacts/outputs/`, `artifacts/runner/` | Inputs, reference outputs, and the runner shared by latency and throughput validation. |
|
||||
| `artifacts/raptor/pim/`, `artifacts/simulation/latency/` | Latency Pim artifacts and functional simulator outputs. |
|
||||
| `artifacts/raptor/throughput/pim/`, `artifacts/simulation/throughput/` | Pipeline-4, batch-4 throughput Pim artifacts and functional simulator outputs. |
|
||||
| `artifacts/common/inputs/` | Shared generated input CSV files. |
|
||||
| `artifacts/common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
|
||||
| `artifacts/common/runner/` | Shared reference runner source, build tree, and library. |
|
||||
| `artifacts/<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and Pim artifacts. |
|
||||
| `artifacts/<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
|
||||
| `artifacts/<arch>/<mode>[/pipelineN]/pimcomp/` | Pimcomp graph, instruction, simulator, and comparison-report artifacts. |
|
||||
| `artifacts/inputs.csv` | Input batch manifest. |
|
||||
| `artifacts/{inputs,outputs,runner}/` | Inputs, native reference outputs, and runner. |
|
||||
| `artifacts/raptor/pim/` | Latency compiler artifacts. |
|
||||
| `artifacts/raptor/throughput/pim/` | Pipeline-4 throughput artifacts. |
|
||||
| `artifacts/simulation/{latency,throughput}/` | Functional simulator outputs. |
|
||||
|
||||
Each comparison's `raptor/` directory may include `spatial0.mlir`,
|
||||
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
|
||||
`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
|
||||
`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.
|
||||
The [comparison runner](tools/pim/pimcomp/compare/README.md) has separate
|
||||
architecture/pipeline directories and shared `artifacts/common/` references.
|
||||
|
||||
Remove these artifacts for every discovered model with:
|
||||
- [Operation cases and generation](operations/README.md)
|
||||
- [Pimcomp models, provenance, and hardware](networks/pimcomp_models/README.md)
|
||||
- [Pimcomp correctness experiments](tools/pim/pimcomp/correctness/README.md)
|
||||
- [Raptor compiler ablation](tools/pim/ablation/README.md)
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--operations-dir validation/operations/gemm/small \
|
||||
--clean
|
||||
```
|
||||
|
||||
`--clean` does not require `--raptor-path` or `--onnx-include-dir`.
|
||||
|
||||
## Available suites
|
||||
|
||||
Checked-in validation networks under `validation/networks/` include `vgg16`,
|
||||
`yolo11n`, and `yolo11nv2`.
|
||||
|
||||
The generated operation inventory is documented in
|
||||
[`operations/README.md`](operations/README.md). Regenerate its models with:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/operations/gen_tests.py
|
||||
```
|
||||
Other network suites under `validation/networks/` include `vgg16`, `yolo11n`,
|
||||
and `yolo11nv2`.
|
||||
|
||||
## Manual functional simulator tracing
|
||||
|
||||
After validation has produced an `artifacts/raptor/pim/` directory, rerun the functional
|
||||
simulator with tracing from its crate directory:
|
||||
After validation produces Pim artifacts, run from the simulator crate:
|
||||
|
||||
```bash
|
||||
cd backend-simulators/pim/pim-simulator
|
||||
cargo run --no-default-features --features tracing --release \
|
||||
--package pim-simulator --bin pim-simulator -- \
|
||||
-f /path/to/workspace/artifacts/raptor/pim \
|
||||
-o /path/to/workspace/artifacts/simulation/out.bin \
|
||||
-f /path/to/artifacts/raptor/pim \
|
||||
-o /path/to/artifacts/simulation/out.bin \
|
||||
-d <addr0>,<size0>,<addr1>,<size1>,... \
|
||||
--mode latency \
|
||||
--batch-size 1 \
|
||||
--input-dir /path/to/workspace/artifacts/simulation/inputs
|
||||
--mode latency --batch-size 1 \
|
||||
--input-dir /path/to/artifacts/simulation/inputs
|
||||
```
|
||||
|
||||
Throughput mode additionally requires `--batch-size N` and at least `N`
|
||||
`input_<index>.bin` files in `--input-dir`. Each input binary concatenates the model tensors in graph
|
||||
input order. The comparison validator also writes one native reference and one
|
||||
`simulation/*_iterations/output_*.bin` dump per batch entry, and checks every
|
||||
entry rather than only the final output.
|
||||
|
||||
Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
|
||||
validator normally derives the `-d` address and byte ranges from
|
||||
`raptor/pim/config.json` and the model output shapes.
|
||||
|
||||
## Results and exit status
|
||||
|
||||
The final table reports latency and throughput functional pass/fail state plus
|
||||
non-functional latency, throughput, power, and energy. The summary includes pass/fail totals, non-functional simulation
|
||||
counts, total measured latency, and average Pim pass timings when `--verbose`
|
||||
is enabled.
|
||||
|
||||
- Exit status `0`: all discovered models passed, or cleanup completed.
|
||||
- Exit status `1`: validation failed, the model directory was invalid, or no
|
||||
`.onnx` models were found.
|
||||
- Exit status `2`: command-line arguments were invalid or required arguments
|
||||
were missing.
|
||||
Derive output address/byte ranges from `raptor/pim/config.json` and model output
|
||||
shapes. Traces appear as `TraceCore0`, `TraceCore1`, etc. beside `out.bin`.
|
||||
Throughput needs `--mode throughput --batch-size N` and at least `N`
|
||||
`input_<index>.bin` files, each concatenating tensors in graph-input order.
|
||||
The comparison validator records and checks one output per batch entry.
|
||||
|
||||
@@ -1,102 +1,36 @@
|
||||
# Pimcomp comparison models
|
||||
|
||||
This directory contains the four networks evaluated in
|
||||
[Pimcomp: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
|
||||
VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
|
||||
additional compiler comparison model.
|
||||
|
||||
See the runner-generated [results_comparison.csv](results_comparison.csv) for the current comparison
|
||||
results. Rows are retained separately for each model, architecture, mode, and
|
||||
pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
|
||||
the Raptor and Pimcomp artifacts; rows without a generated report contain `NA`.
|
||||
Use `--archs` to select the architecture rows to generate; existing rows for
|
||||
other architectures remain unchanged.
|
||||
Models for comparing Raptor with
|
||||
[Pimcomp](https://arxiv.org/pdf/2411.09159): the paper's four networks plus YOLO11n.
|
||||
Use the [comparison runner](../../tools/pim/pimcomp/compare/README.md) for commands,
|
||||
options, artifact locations, and plots.
|
||||
The [reproduction reference](../../../docs/evaluation/pimcomp-reproducibility.md) records exact supported/ignored
|
||||
operations, standalone commands, configuration layout, and GA/OpenMP controls.
|
||||
|
||||
## Models and provenance
|
||||
|
||||
| Directory | Model | Input | Provenance |
|
||||
|--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`. |
|
||||
| `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as Pimcomp's frontend does. |
|
||||
| `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). |
|
||||
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [Pimcomp VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
|
||||
| `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
|
||||
| Directory | Model / input | Source |
|
||||
|---|---|---|
|
||||
| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. |
|
||||
| `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. |
|
||||
| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12). |
|
||||
| `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. |
|
||||
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Derived from `../yolo11n/depth_51/yolo11n_depth_51.onnx`. |
|
||||
|
||||
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit Pimsim-ready GoogLeNet model.
|
||||
It removes the two LRN nodes and terminal Softmax from the original model,
|
||||
so that the comparison covers only operations scheduled by Pimcomp and supported by Pimsim.
|
||||
The paper's checkpoints were not published. The three Model Zoo networks
|
||||
produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is
|
||||
suitable for compiler/simulator comparison, not paper-accuracy reproduction.
|
||||
The paper's MNIST description for ResNet-18 differs from the released graph's
|
||||
ImageNet input shape.
|
||||
|
||||
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit Pimsim-ready YOLO11n model.
|
||||
It removes the Softmax nodes from the original model,
|
||||
so that the compiled artifact can be simulated in Pimsim.
|
||||
Dedicated performance models change the original computation:
|
||||
|
||||
## Unsupported and ignored operations
|
||||
- `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal
|
||||
Softmax, which Pimcomp does not schedule.
|
||||
- `yolo11n/yolo11n-pimsim-nn.onnx` removes two Softmax nodes unsupported by
|
||||
Pimsim. Use the original YOLO model for functional validation.
|
||||
|
||||
Pimcomp's frontend accepts exactly these ONNX operations:
|
||||
|
||||
```text
|
||||
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
|
||||
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
|
||||
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
|
||||
```
|
||||
|
||||
`Constant` is consumed as frontend metadata rather than emitted as a Pimcomp
|
||||
node. Every other ONNX operation is unsupported: the frontend prints
|
||||
`operation: <type> not considered` and stops at the first occurrence. Thus the
|
||||
complete unsupported set is the complement of the allowlist above for the
|
||||
model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and
|
||||
`Resize` nodes.
|
||||
|
||||
Pimcomp's low-latency scheduler, hierarchy mapper, and genetic algorithm use
|
||||
this complete explicit no-consider set:
|
||||
|
||||
```text
|
||||
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
|
||||
Softmax, Squeeze, Transpose
|
||||
```
|
||||
|
||||
Those nodes do not receive scheduled latency instructions when they remain in
|
||||
the backend graph. Before that point the frontend may fuse BatchNormalization
|
||||
and activation nodes into Conv/Gemm, convert a
|
||||
Reshape-Transpose-Reshape channel-shuffle pattern to `OP_SHUFFLE`, remove a
|
||||
specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its
|
||||
consumer. These transformations do not make an otherwise standalone ignored
|
||||
operation timed.
|
||||
|
||||
Pimsim consumes Pim ISA instructions. It supports every named opcode
|
||||
in the shared serialized range except `vsoftmax` (opcode 21),
|
||||
which is rejected explicitly in both JSON and binary input. It
|
||||
silently ignores no opcode; unknown names and numbers are errors.
|
||||
|
||||
These boundaries explain the dedicated artifacts:
|
||||
|
||||
- GoogLeNet's two LRN nodes and terminal Softmax perform real computation but
|
||||
are ignored by Pimcomp, so the common latency artifact removes them. Its
|
||||
inference Dropout and shape-only Reshape can remain without adding compute.
|
||||
- YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can
|
||||
run in Pimsim. Every other node, including MatMul, Transpose, and the
|
||||
final detection-decoding tail, remains present and timed by Raptor. No
|
||||
Pimcomp latency is reported because its frontend stops at `Split` and also
|
||||
lacks `Resize`; compiling that prefix would not represent YOLO11n.
|
||||
|
||||
The authoritative lists are in
|
||||
[`frontend.py`](../../../third_party/PIMCOMP-NN/frontend/frontend.py),
|
||||
[`ElementPipelineSchedule.cpp`](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
|
||||
[`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and
|
||||
[`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp).
|
||||
|
||||
The Pimcomp authors did not publish the ONNX checkpoints used by the paper.
|
||||
Running Pimcomp's frontend on the three Model Zoo files above produces JSON
|
||||
graphs exactly equal to Pimcomp's bundled `resnet18.json`, `resnet34.json`,
|
||||
and `googlenet.json`.
|
||||
|
||||
There is no VGG-8 artifact in the ONNX Model Zoo or any Pimcomp revision.
|
||||
The included VGG-8 therefore has deterministic random weights and is suitable
|
||||
for compiler and simulator comparison, not paper-accuracy reproduction. The
|
||||
paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the
|
||||
published Pimcomp graphs and ResNet Model Zoo artifacts use ImageNet shapes.
|
||||
|
||||
Current SHA-256 checksums:
|
||||
Model SHA-256 checksums:
|
||||
|
||||
```text
|
||||
788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx
|
||||
@@ -107,275 +41,61 @@ a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/goog
|
||||
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
|
||||
```
|
||||
|
||||
## Paper hardware profiles
|
||||
## Hardware profiles
|
||||
|
||||
The files in
|
||||
[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
|
||||
encode Table V's explicit resource parameters.
|
||||
Each profile subdirectory contains pre-generated latency and throughput
|
||||
Pimsim configs plus its matching mesh; comparison and validation reference
|
||||
these canonical artifacts directly.
|
||||
Canonical configs and meshes are in
|
||||
[`validation/pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp).
|
||||
They encode Table V's resources:
|
||||
|
||||
| Config | Cores | Crossbars/core | Crossbar | Cell | Pimcomp layout |
|
||||
|------------------------------|------------------:|---------------:|------------|------:|-----------------|
|
||||
| `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
|
||||
| `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
|
||||
| `arch-c/latency_config.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` |
|
||||
| Profile | Cores | Crossbars/core | Crossbar | Mesh |
|
||||
|---|---:|---:|---|---|
|
||||
| Arch-A | 168 | 96 | `128x128` | `12x14` |
|
||||
| Arch-B | 138 | 128 | `128x128` | `6x23` |
|
||||
| Arch-C | 64 (16 chips × 4) | 8 | `512x1024` | Flattened `8x8` |
|
||||
|
||||
`adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
|
||||
The paper does not give a two-dimensional core topology for Arch-A/B, so the
|
||||
factorizations above preserve core count but cannot reproduce unpublished NoC
|
||||
placement details. Released Pimcomp has no chip-count field; Arch-C is
|
||||
therefore flattened to 64 cores and does not model chip boundaries.
|
||||
All profiles use 2-bit cells and 16 ADCs for 16-bit fixed-point weights.
|
||||
Arch-A/B topology factorizations preserve core counts; the paper does not
|
||||
publish placement details. Arch-C does not model chip boundaries, and Raptor's
|
||||
square-crossbar option cannot match its rectangular arrays. Do not change
|
||||
hardware resources to bypass this limitation.
|
||||
|
||||
The remaining latency and power values come from Pimcomp's released default
|
||||
configuration. Consequently, instruction/resource comparisons are
|
||||
reproducible, but absolute paper power and energy numbers are not.
|
||||
Other timing/power parameters use Pimcomp's released defaults. Resource and
|
||||
instruction comparisons are reproducible; absolute paper power/energy values
|
||||
are not.
|
||||
|
||||
## Build and validate the ONNX files
|
||||
## Build and run
|
||||
|
||||
From the Raptor repository root:
|
||||
From the repository root, after the [Raptor build](../../../README.md#build):
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pip install -r requirements.txt
|
||||
cmake --build ./build_release
|
||||
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
|
||||
|
||||
.venv/bin/python -c \
|
||||
'from pathlib import Path; import onnx; [onnx.checker.check_model(onnx.load(p)) for p in Path("validation/networks/pimcomp_models").glob("*/*.onnx")]'
|
||||
```
|
||||
|
||||
Do not build either project with `ninja` directly.
|
||||
|
||||
## Compile with Pimcomp
|
||||
|
||||
Pimcomp reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
|
||||
select one paper profile, and restore it when the shell exits:
|
||||
|
||||
```bash
|
||||
RAPTOR_ROOT=$PWD
|
||||
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
|
||||
PIMCOMP_CONFIGS="$RAPTOR_ROOT/validation/pimsim_configs/pimcomp"
|
||||
CONFIG_BACKUP=$(mktemp)
|
||||
cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
|
||||
trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
|
||||
cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json"
|
||||
```
|
||||
|
||||
The Model Zoo files map exactly to Pimcomp's bundled model names, so compile
|
||||
them directly:
|
||||
|
||||
```bash
|
||||
cd "$PIMCOMP/build"
|
||||
|
||||
# High-throughput mode; the paper evaluates batches of 128 samples.
|
||||
./PIMCOMP-NN -m=resnet18 -r=balance -p=batch -o=YES -v=YES -s=YES
|
||||
./PIMCOMP-NN -m=resnet34 -r=balance -p=batch -o=YES -v=YES -s=YES
|
||||
./PIMCOMP-NN -m=googlenet -r=balance -p=batch -o=YES -v=YES -s=YES
|
||||
|
||||
# Low-latency mode; the paper uses batch size 1.
|
||||
./PIMCOMP-NN -m=resnet18 -r=balance -p=element -o=YES -v=YES -s=YES
|
||||
./PIMCOMP-NN -m=resnet34 -r=balance -p=element -o=YES -v=YES -s=YES
|
||||
./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES
|
||||
```
|
||||
|
||||
VGG-8 first needs Pimcomp's JSON frontend. Use a temporary ONNX copy because
|
||||
the released frontend rewrites the input batch dimension in place:
|
||||
|
||||
```bash
|
||||
cd /path/to/Raptor
|
||||
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx /tmp/vgg8-pimcomp.onnx
|
||||
.venv/bin/python third_party/PIMCOMP-NN/frontend/frontend.py \
|
||||
--model_path /tmp/vgg8-pimcomp.onnx \
|
||||
--save_path third_party/PIMCOMP-NN/models/JSON/vgg8_paper_reconstructed.json
|
||||
|
||||
cd third_party/PIMCOMP-NN/build
|
||||
./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=batch -o=YES -v=YES -s=YES
|
||||
./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES
|
||||
```
|
||||
|
||||
Repeat after selecting `arch-b/latency_config.json` and
|
||||
`arch-c/latency_config.json`. All four models were
|
||||
compiled successfully in both modes with all three configs. The released
|
||||
random placement code occasionally segfaults; an unchanged retry succeeded in
|
||||
the observed cases.
|
||||
|
||||
The paper's optimizer uses a genetic algorithm with population 200 and up to
|
||||
1000 iterations. The checked-out Pimcomp submodule already has both paper
|
||||
settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness
|
||||
evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS`
|
||||
to control its parallelism; otherwise OpenMP uses the available CPUs. The GA
|
||||
uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.
|
||||
|
||||
## Compare Raptor and Pimcomp
|
||||
|
||||
The comparison driver uses one random input and one native ONNX-MLIR reference,
|
||||
compiles both instruction streams, runs both through Pimsim, runs
|
||||
functional validation through `pim-simulator`, and writes Markdown and JSON
|
||||
reports.
|
||||
|
||||
To reproduce the default `arch-a`/`arch-b` architectures and both
|
||||
latency/throughput modes, use the model-by-model runner. Use `--archs` to
|
||||
specify a different architecture set. It verifies the paper GA settings, expects Raptor
|
||||
and the existing `third_party/PIMCOMP-NN/build` tree to already be built, then
|
||||
runs the comparisons in parallel and regenerates `results_comparison.csv` from the JSON
|
||||
reports:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
|
||||
```
|
||||
|
||||
Use `--archs arch-a --mode latency` for only the Arch-A latency experiment, or
|
||||
`--archs arch-a arch-b arch-c` to run all three architectures.
|
||||
The runner expects existing Pimcomp and Pimsim builds, verifies the paper's GA
|
||||
settings (population 200, up to 1000 iterations), and uses isolated frontend
|
||||
workspaces without changing the source models or shared Pimcomp configuration.
|
||||
Default aggregate results are written beside the
|
||||
[comparison runner](../../tools/pim/pimcomp/compare/results_comparison.csv);
|
||||
explicit custom output roots retain their own aggregate CSVs.
|
||||
|
||||
Each model directory has a shared ignored `artifacts/common/` directory containing
|
||||
`inputs/`, `outputs/`, and the native `runner/`.
|
||||
Pimsim configs and network meshes remain canonical under
|
||||
`validation/pimsim_configs/pimcomp/` and are referenced in place.
|
||||
Architecture- and pipeline-specific `raptor/`, `simulation/`, and Pimcomp
|
||||
artifacts remain under each `artifacts/<arch>/<mode>[/pipelineN]/` comparison
|
||||
directory; ablation variants use
|
||||
`artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`. Pimcomp outputs are prepared
|
||||
once per model/architecture/mode and linked into the other pipeline directories;
|
||||
`comparison_report.{md,json}`
|
||||
live under its `pimcomp/`. The frontend regenerates
|
||||
one isolated `models/JSON/` graph because Pimcomp requires that relative
|
||||
layout; it is removed after a successful backend run and the shared submodule
|
||||
model directory is never modified. The original ONNX model is passed to the
|
||||
frontend unchanged.
|
||||
Pimcomp's source tree and build directory remain unchanged at runtime. Use
|
||||
`--models vgg8` to run one model, `--mode throughput` to select one mode,
|
||||
`--pipeline 4` to select one throughput pipeline, `--only raptor` or
|
||||
`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
|
||||
to inspect every command, or `--out-dir PATH` to keep results outside
|
||||
`validation/`. Use `--clean` to remove generated comparison artifacts and
|
||||
summaries, including stale reference lock files. Selecting a subset replaces only those comparison rows and
|
||||
recomputes the aggregate `results_comparison.csv`; missing shared inputs, outputs, or the
|
||||
reference runner are generated even for an isolated run. Use `--jobs 4` to cap
|
||||
parallel comparisons. The per-stage timeout is unlimited by default; pass a
|
||||
positive `--timeout-seconds` value to impose one.
|
||||
Throughput comparisons default to `pimsim-nn --fast` with a 1000 ms
|
||||
convergence deadline. Add `--no-fast` for authoritative full-duration runs.
|
||||
Pimcomp receives the original ONNX model, and its frontend applies native
|
||||
BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
|
||||
The runner continues after a failed model so all reports are produced.
|
||||
## Correctness and precision limits
|
||||
|
||||
The known Pimcomp batch-scheduling correctness issue and a reproducible
|
||||
reference-intermediate prefill experiment are documented in
|
||||
[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
|
||||
Pimcomp does not support YOLO11n's `Split` and `Resize`; no Pimcomp performance
|
||||
number is reported for that model. Accepted frontend operations can also be
|
||||
ignored by scheduling; see the authoritative
|
||||
[frontend](../../../third_party/PIMCOMP-NN/frontend/frontend.py) and
|
||||
[latency scheduler](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp).
|
||||
Known [batch-scheduling and BatchNorm gaps](../../tools/pim/pimcomp/correctness/README.md)
|
||||
must remain visible as failed functional validation.
|
||||
|
||||
Arch-A low-latency example:
|
||||
Functional validation uses FP32 programs in the Rust simulator against native
|
||||
ONNX-MLIR. Pimcomp's Rust export expands byte-sized storage/transfers to FP32.
|
||||
For performance comparison, Pimcomp's original `SimulationInfo.gz` remains
|
||||
unchanged; a separate Raptor `raptor/pimsim_nn/` view scales FP32 storage and
|
||||
transfers to one byte per element with `setbw 8, 8`. These performance artifacts
|
||||
are not numerically quantized programs and must not be used for correctness.
|
||||
|
||||
```bash
|
||||
RAPTOR_ROOT=$PWD
|
||||
|
||||
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py" \
|
||||
--model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
|
||||
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/artifacts/arch-a/latency" \
|
||||
--pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
|
||||
--core-count 168 \
|
||||
--crossbar-count 96 \
|
||||
--crossbar-size 128 \
|
||||
--mesh-rows 12 \
|
||||
--mesh-cols 14 \
|
||||
--pimsim-mode latency \
|
||||
--pimcomp-pipeline element
|
||||
```
|
||||
|
||||
Use the same command with
|
||||
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released Pimcomp cannot
|
||||
compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
|
||||
for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
|
||||
as YOLO11n would produce a misleading latency, so no Pimcomp number is
|
||||
reported for this model.
|
||||
|
||||
For Arch-A high throughput, use `--pimsim-mode throughput
|
||||
--pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a
|
||||
`6x23` mesh, and
|
||||
`validation/pimsim_configs/pimcomp/arch-b/latency_config.json`.
|
||||
|
||||
If only semantic and instruction comparison is required, add
|
||||
`--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both
|
||||
semantic validations with maximum output differences below `5e-10`.
|
||||
|
||||
The comparison runner enables `--fail-on-error`, so a failed compiler,
|
||||
simulation, or semantic validation makes the command fail while preserving the
|
||||
generated report.
|
||||
|
||||
### Numeric precision and simulator artifacts
|
||||
|
||||
The functional and non-functional simulators intentionally consume different
|
||||
artifacts:
|
||||
|
||||
- Raptor and Pimcomp are validated against the native ONNX-MLIR reference as
|
||||
FP32 programs in the Rust simulator. Raptor's emitted program is already
|
||||
FP32. The Pimcomp-to-Rust export expands its element-addressed storage and
|
||||
byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector
|
||||
`imm_len` fields as element counts.
|
||||
- Pimcomp's original `SimulationInfo.gz` is copied unchanged for Pimsim.
|
||||
Pimcomp hardcodes `setbw 8, 8` and one byte per element without performing
|
||||
numerical quantization; this artifact is used only for latency estimation.
|
||||
- Raptor's original FP32 artifact remains unchanged for functional validation.
|
||||
A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its
|
||||
byte-addressed storage and transfer sizes from four bytes to one byte per
|
||||
element. Vector `imm_len` fields remain element counts. Like Pimcomp's
|
||||
artifact, this view is not numerically valid and is used only for a fair
|
||||
non-functional comparison.
|
||||
|
||||
The ISA defines vector lengths in elements, while `ld`, `st`, `lldi`, `lmv`,
|
||||
`send`, `recv`, addresses, and non-vector offsets are byte-based. Do not use
|
||||
either latency-only artifact for semantic validation.
|
||||
|
||||
Current Raptor status:
|
||||
|
||||
- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
|
||||
- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
|
||||
It removes the two LRN nodes and terminal softmax that Pimcomp does not
|
||||
schedule.
|
||||
- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
|
||||
`512x1024` arrays can therefore be compiled by Pimcomp but not compared
|
||||
exactly with Raptor.
|
||||
|
||||
Do not change the hardware profile to bypass either limitation; that would no
|
||||
longer be a paper-matched comparison.
|
||||
|
||||
## Monolith fallback
|
||||
|
||||
The local `monolith` SSH alias points to the high-memory host. Copy only this
|
||||
suite and the comparison driver; `-L` materializes the ResNet-18 symlink because
|
||||
the canonical `resnet18/depth_68/resnet18_depth_68.onnx` file may not exist remotely:
|
||||
|
||||
```bash
|
||||
REMOTE_REPO=/home/gmagnani/Project/Raptor
|
||||
rsync -azL validation/networks/pimcomp_models/ \
|
||||
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
|
||||
rsync -az validation/pimsim_configs/pimcomp/ \
|
||||
"monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
|
||||
rsync -az validation/tools/pim/ \
|
||||
"monolith:$REMOTE_REPO/validation/tools/pim/"
|
||||
rsync -az --exclude=.git --exclude=build --exclude=output \
|
||||
third_party/PIMCOMP-NN/ \
|
||||
"monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
|
||||
```
|
||||
|
||||
Then use the same commands over SSH:
|
||||
|
||||
```bash
|
||||
ssh monolith
|
||||
cd /home/gmagnani/Project/Raptor
|
||||
|
||||
# One-time setup if the repository virtual environment is absent.
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install numpy onnx onnxruntime colorama
|
||||
|
||||
# Run every configured comparison in parallel.
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
|
||||
```
|
||||
|
||||
Copy reports back without transferring large compiler artifacts:
|
||||
|
||||
```bash
|
||||
rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \
|
||||
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \
|
||||
validation/networks/pimcomp_models/
|
||||
```
|
||||
Vector `imm_len` is an element count. Addresses and transfer sizes for
|
||||
`ld`, `st`, `lldi`, `lmv`, `send`, and `recv` are byte-based.
|
||||
|
||||
@@ -0,0 +1,286 @@
|
||||
# Operation test inventory
|
||||
|
||||
All 177 checked-in ONNX models, grouped by validator category. Each row describes
|
||||
the regression or operation feature exercised; follow its link for the model.
|
||||
[Generation and validation commands](README.md) remain in the short README.
|
||||
|
||||
[gen_tests.py](gen_tests.py) defines generated shapes, attributes, constants, and
|
||||
seeds. It regenerates 160 cases; 17 fixtures are checked-in only: the five Gemv
|
||||
cases, `mul/after_conv_scalar_constant`, and eleven Conv cases (the six
|
||||
`strategy_*` cases plus `input_224_7x7_stride2`, `kernel_2x2`,
|
||||
`output_channel_grouping_minimal`, `relu_conv_store`, `with_constant`).
|
||||
Do not infer the complete inventory from generator output alone. Strategy
|
||||
fixtures target particular lowering paths; actual selection depends on the
|
||||
hardware profile and compiler flags.
|
||||
|
||||
## Add (5) — `add/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`after_gemm`](add/after_gemm/add_after_gemm.onnx) | Gemm followed by Add with a broadcast bias vector. |
|
||||
| [`basic`](add/basic/add_basic.onnx) | Elementwise Add on two inputs with identical shapes. |
|
||||
| [`broadcast_row`](add/broadcast_row/add_broadcast_row.onnx) | Elementwise Add with row-vector broadcasting. |
|
||||
| [`channel_broadcast_1024`](add/channel_broadcast_1024/add_channel_broadcast_1024.onnx) | NCHW per-channel broadcasting over 1024 channels. |
|
||||
| [`leading_dimension_broadcast`](add/leading_dimension_broadcast/add_leading_dimension_broadcast.onnx) | Trailing-dimension broadcasting across leading dimensions. |
|
||||
|
||||
## Concat (3) — `concat/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`channel_axis`](concat/channel_axis/concat_channel_axis.onnx) | Concatenates two runtime NCHW tensors along the channel axis. |
|
||||
| [`negative_axis`](concat/negative_axis/concat_negative_axis.onnx) | Concatenates tensors using a negative axis. |
|
||||
| [`three_inputs_channel_axis`](concat/three_inputs_channel_axis/concat_three_inputs_channel_axis.onnx) | Concatenates three runtime NCHW tensors along the channel axis. |
|
||||
|
||||
## Conv (42) — `conv/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`batch_2`](conv/batch_2/conv_batch_2.onnx) | Batched Conv with SAME_UPPER padding and bias. |
|
||||
| [`batch_4_pointwise`](conv/batch_4_pointwise/conv_batch_4_pointwise.onnx) | Pointwise Conv with batch size four. |
|
||||
| [`input_224_7x7_stride2`](conv/input_224_7x7_stride2/conv_input_224_7x7_stride2.onnx) | 224x224 RGB Conv with 64 output channels, a 7x7 kernel, stride two, and bias. |
|
||||
| [`depthwise_1024_channels`](conv/depthwise_1024_channels/conv_depthwise_1024_channels.onnx) | Depthwise pointwise Conv with 1024 groups. |
|
||||
| [`depthwise_grouped`](conv/depthwise_grouped/conv_depthwise_grouped.onnx) | Depthwise-style grouped Conv with one input channel per group. |
|
||||
| [`dilated_3x3`](conv/dilated_3x3/conv_dilated_3x3.onnx) | Conv with a dilated 3x3 kernel. |
|
||||
| [`dynamic`](conv/dynamic/conv_dynamic.onnx) | Conv with runtime input and weight tensors. |
|
||||
| [`explicit_padding`](conv/explicit_padding/conv_explicit_padding.onnx) | 3x3 Conv with symmetric explicit padding. |
|
||||
| [`grouped_many_groups`](conv/grouped_many_groups/conv_grouped_many_groups.onnx) | Pointwise Conv with many groups and high channel counts. |
|
||||
| [`grouped_two_groups`](conv/grouped_two_groups/conv_grouped_two_groups.onnx) | Two-group pointwise Conv with bias. |
|
||||
| [`huge_pointwise_1024`](conv/huge_pointwise_1024/conv_huge_pointwise_1024.onnx) | Pointwise Conv with 1024 input and output channels. |
|
||||
| [`huge_pointwise_1024_dynamic`](conv/huge_pointwise_1024_dynamic/conv_huge_pointwise_1024_dynamic.onnx) | The 1024-channel pointwise Conv with runtime weights. |
|
||||
| [`kernel_3x3`](conv/kernel_3x3/conv_kernel_3x3.onnx) | Basic 3x3 Conv without padding. |
|
||||
| [`kernel_equals_input_spatial`](conv/kernel_equals_input_spatial/conv_kernel_equals_input_spatial.onnx) | Conv whose kernel covers the full spatial input. |
|
||||
| [`large_input_channels_1x1`](conv/large_input_channels_1x1/conv_large_input_channels_1x1.onnx) | Pointwise Conv with 1024 input channels and modest output width. |
|
||||
| [`large_output_channels_1x1`](conv/large_output_channels_1x1/conv_large_output_channels_1x1.onnx) | Pointwise Conv with modest input width and 1024 output channels. |
|
||||
| [`large_spatial`](conv/large_spatial/conv_large_spatial.onnx) | 3x3 Conv on a larger spatial input. |
|
||||
| [`multi_channel`](conv/multi_channel/conv_multi_channel.onnx) | 3x3 Conv with multiple input and output channels. |
|
||||
| [`non_square_kernel_1x3`](conv/non_square_kernel_1x3/conv_non_square_kernel_1x3.onnx) | Conv with a non-square 1x3 kernel. |
|
||||
| [`non_square_kernel_3x1`](conv/non_square_kernel_3x1/conv_non_square_kernel_3x1.onnx) | Conv with a non-square 3x1 kernel. |
|
||||
| [`non_uniform_stride`](conv/non_uniform_stride/conv_non_uniform_stride.onnx) | Conv with different height and width strides. |
|
||||
| [`output_channel_grouping_minimal`](conv/output_channel_grouping_minimal/conv_output_channel_grouping_minimal.onnx) | Minimal 64-to-256 pointwise Conv for output-channel grouping. |
|
||||
| [`pointwise_1x1`](conv/pointwise_1x1/conv_pointwise_1x1.onnx) | Basic pointwise channel-mixing Conv. |
|
||||
| [`pointwise_tiled_chain`](conv/pointwise_tiled_chain/conv_pointwise_tiled_chain.onnx) | Relu and chained pointwise Convs with a tiled intermediate. |
|
||||
| [`real_asymmetric_padding`](conv/real_asymmetric_padding/conv_real_asymmetric_padding.onnx) | Conv with asymmetric explicit padding. |
|
||||
| [`relu_conv_store`](conv/relu_conv_store/conv_relu_conv_store.onnx) | Conv, Relu, and a second Conv to validate an intermediate stored result. |
|
||||
| [`same_lower_3x3`](conv/same_lower_3x3/conv_same_lower_3x3.onnx) | 3x3 Conv with SAME_LOWER padding. |
|
||||
| [`same_padding_3x3`](conv/same_padding_3x3/conv_same_padding_3x3.onnx) | 3x3 Conv with SAME_UPPER padding. |
|
||||
| [`kernel_2x2`](conv/kernel_2x2/conv_kernel_2x2.onnx) | Hand-authored Conv with a 2x2 kernel. |
|
||||
| [`stride_2`](conv/stride_2/conv_stride_2.onnx) | 3x3 Conv with stride two. |
|
||||
| [`strategy_depthwise_16`](conv/strategy_depthwise_16/conv_strategy_depthwise_16.onnx) | 16-channel depthwise 3x3 Conv for depthwise lowering. |
|
||||
| [`strategy_input_k_tiled`](conv/strategy_input_k_tiled/conv_strategy_input_k_tiled.onnx) | 32-channel 3x3 Conv sized to exercise input-K tiling. |
|
||||
| [`strategy_output_channel_tiled`](conv/strategy_output_channel_tiled/conv_strategy_output_channel_tiled.onnx) | 8-to-192 3x3 Conv sized to exercise output-channel tiling. |
|
||||
| [`strategy_streamed_packed`](conv/strategy_streamed_packed/conv_strategy_streamed_packed.onnx) | 3-to-16 3x3 Conv on 128x128 input for streamed packed lowering. |
|
||||
| [`strategy_streamed_patch`](conv/strategy_streamed_patch/conv_strategy_streamed_patch.onnx) | 3-to-16 3x3 Conv on 64x64 input for streamed patch lowering. |
|
||||
| [`strategy_tiled_2d`](conv/strategy_tiled_2d/conv_strategy_tiled_2d.onnx) | 32-to-192 3x3 Conv sized to exercise 2D tiling. |
|
||||
| [`with_bias_3x3`](conv/with_bias_3x3/conv_with_bias_3x3.onnx) | Multi-channel 3x3 Conv with bias. |
|
||||
| [`with_constant`](conv/with_constant/conv_with_constant.onnx) | Hand-authored SAME_UPPER Conv with constant weight and bias. |
|
||||
| [`without_kernel_shape_attr`](conv/without_kernel_shape_attr/conv_without_kernel_shape_attr.onnx) | Conv whose kernel shape is inferred from its weight tensor. |
|
||||
| [`yolo11n_depthwise_head`](conv/yolo11n_depthwise_head/conv_yolo11n_depthwise_head.onnx) | YOLO11n pointwise-to-depthwise head boundary at `80x80`, preserving row fragments. |
|
||||
| [`yolo11n_heavy`](conv/yolo11n_heavy/conv_yolo11n_heavy.onnx) | Two largest standard YOLO11n Conv-SiLU blocks by MAC count at `64x80x80`. |
|
||||
| [`yolo11n_stem`](conv/yolo11n_stem/conv_yolo11n_stem.onnx) | First two YOLO11n `Conv-SiLU` blocks at `640x640`, including the distributed activation boundary. |
|
||||
|
||||
## Div (6) — `div/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`after_gemm`](div/after_gemm/div_after_gemm.onnx) | Gemm followed by Div with a broadcast divisor vector. |
|
||||
| [`basic`](div/basic/div_basic.onnx) | Elementwise Div by a same-shape constant tensor. |
|
||||
| [`channel_broadcast_1024`](div/channel_broadcast_1024/div_channel_broadcast_1024.onnx) | Div with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| [`leading_dimension_broadcast`](div/leading_dimension_broadcast/div_leading_dimension_broadcast.onnx) | Div with trailing-dimension broadcasting. |
|
||||
| [`runtime_scalar_rhs`](div/runtime_scalar_rhs/div_runtime_scalar_rhs.onnx) | Div of a runtime tensor by a scalar initializer. |
|
||||
| [`scalar_constant`](div/scalar_constant/div_scalar_constant.onnx) | Div with scalar broadcasting on a 2D tensor. |
|
||||
|
||||
## Gather (5) — `gather/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`3d_input_axis1`](gather/3d_input_axis1/gather_3d_input_axis1.onnx) | Gathers along axis one of a 3D input. |
|
||||
| [`axis0_matrix_indices`](gather/axis0_matrix_indices/gather_axis0_matrix_indices.onnx) | Gathers rows using a 2D indices tensor. |
|
||||
| [`axis1`](gather/axis1/gather_axis1.onnx) | Gathers selected columns from a 2D tensor. |
|
||||
| [`negative_axis`](gather/negative_axis/gather_negative_axis.onnx) | Gathers using a negative axis. |
|
||||
| [`negative_indices`](gather/negative_indices/gather_negative_indices.onnx) | Gathers with negative indices along axis zero. |
|
||||
|
||||
## Gemm (21) — `gemm/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`alpha_beta`](gemm/alpha_beta/gemm_alpha_beta.onnx) | Applies non-default alpha and beta scaling with bias. |
|
||||
| [`bias_rank2_broadcast`](gemm/bias_rank2_broadcast/gemm_bias_rank2_broadcast.onnx) | Broadcasts a rank-2 bias across output rows. |
|
||||
| [`dynamic`](gemm/dynamic/gemm_dynamic.onnx) | Uses both matrix operands at runtime. |
|
||||
| [`dynamic_alpha`](gemm/dynamic_alpha/gemm_dynamic_alpha.onnx) | Uses runtime operands with non-default alpha scaling. |
|
||||
| [`dynamic_beta`](gemm/dynamic_beta/gemm_dynamic_beta.onnx) | Uses runtime operands and bias with non-default beta scaling. |
|
||||
| [`dynamic_bias`](gemm/dynamic_bias/gemm_dynamic_bias.onnx) | Uses runtime matrix operands and runtime bias. |
|
||||
| [`dynamic_bias_alpha_beta`](gemm/dynamic_bias_alpha_beta/gemm_dynamic_bias_alpha_beta.onnx) | Combines runtime operands and bias with alpha and beta scaling. |
|
||||
| [`dynamic_transpose_b`](gemm/dynamic_transpose_b/gemm_dynamic_transpose_b.onnx) | Transposes a runtime right-hand matrix. |
|
||||
| [`huge_1024`](gemm/huge_1024/gemm_huge_1024.onnx) | Uses 1024-wide inner and output dimensions. |
|
||||
| [`large`](gemm/large/gemm_large.onnx) | Exercises larger rectangular matrices. |
|
||||
| [`large_k_small_n`](gemm/large_k_small_n/gemm_large_k_small_n.onnx) | Uses a large reduction dimension and narrow output. |
|
||||
| [`non_square`](gemm/non_square/gemm_non_square.onnx) | Uses different reduction and output widths. |
|
||||
| [`scalar_bias`](gemm/scalar_bias/gemm_scalar_bias.onnx) | Broadcasts a scalar bias to the full output. |
|
||||
| [`square_weights`](gemm/square_weights/gemm_square_weights.onnx) | Basic Gemm with square weights. |
|
||||
| [`small`](gemm/small/gemm_small.onnx) | Tiny Gemm for fast focused validation. |
|
||||
| [`small_k_large_n`](gemm/small_k_large_n/gemm_small_k_large_n.onnx) | Uses a modest reduction dimension and wide output. |
|
||||
| [`transpose_a`](gemm/transpose_a/gemm_transpose_a.onnx) | Transposes the left-hand matrix. |
|
||||
| [`transpose_a_and_b`](gemm/transpose_a_and_b/gemm_transpose_a_and_b.onnx) | Transposes both matrix operands. |
|
||||
| [`transpose_b`](gemm/transpose_b/gemm_transpose_b.onnx) | Transposes the right-hand weight matrix. |
|
||||
| [`transpose_b_with_bias`](gemm/transpose_b_with_bias/gemm_transpose_b_with_bias.onnx) | Combines a transposed weight matrix with bias. |
|
||||
| [`with_bias`](gemm/with_bias/gemm_with_bias.onnx) | Basic matrix product with vector bias. |
|
||||
|
||||
## Gemv (5) — `gemv/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`all_constant`](gemv/all_constant/gemv_all_constant.onnx) | Vector-matrix product with all inputs constant. |
|
||||
| [`constant_weight`](gemv/constant_weight/gemv_constant_weight.onnx) | Basic single-row vector-matrix product with constant weights. |
|
||||
| [`non_uniform_bias`](gemv/non_uniform_bias/gemv_non_uniform_bias.onnx) | Adds a non-uniform constant bias pattern. |
|
||||
| [`uniform_bias`](gemv/uniform_bias/gemv_uniform_bias.onnx) | Adds a uniform constant bias pattern. |
|
||||
| [`scalar_bias`](gemv/scalar_bias/gemv_scalar_bias.onnx) | Adds a scalar broadcast bias. |
|
||||
|
||||
## MatMul (12) — `matmul/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`basic`](matmul/basic/matmul_basic.onnx) | Direct 2D MatMul with constant right-hand matrix. |
|
||||
| [`batched_3d`](matmul/batched_3d/matmul_batched_3d.onnx) | Batched 3D MatMul with matching batch dimensions. |
|
||||
| [`batched_3d_dynamic`](matmul/batched_3d_dynamic/matmul_batched_3d_dynamic.onnx) | Batched 3D MatMul with both operands at runtime. |
|
||||
| [`batched_left_constant`](matmul/batched_left_constant/matmul_batched_left_constant.onnx) | Batched 3D MatMul with constant left-hand matrix. |
|
||||
| [`batched_lhs_broadcast`](matmul/batched_lhs_broadcast/matmul_batched_lhs_broadcast.onnx) | Broadcasts a 2D left-hand matrix across a batched right-hand tensor. |
|
||||
| [`batched_rhs_broadcast`](matmul/batched_rhs_broadcast/matmul_batched_rhs_broadcast.onnx) | Broadcasts a 2D right-hand matrix across a batched left-hand tensor. |
|
||||
| [`dynamic`](matmul/dynamic/matmul_dynamic.onnx) | Direct 2D MatMul with both operands at runtime. |
|
||||
| [`huge_1024`](matmul/huge_1024/matmul_huge_1024.onnx) | Uses 1024-wide inner and output dimensions. |
|
||||
| [`left_constant`](matmul/left_constant/matmul_left_constant.onnx) | Direct 2D MatMul with constant left-hand matrix. |
|
||||
| [`matrix_vector`](matmul/matrix_vector/matmul_matrix_vector.onnx) | Matrix-vector multiplication producing a 1D output. |
|
||||
| [`vector_matrix`](matmul/vector_matrix/matmul_vector_matrix.onnx) | Vector-matrix multiplication producing a 1D output. |
|
||||
| [`yolo_attention`](matmul/yolo_attention/matmul_yolo_attention.onnx) | YOLO11n rank-4 dynamic MatMul-scale-transpose-MatMul attention chain. |
|
||||
|
||||
## Mul (6) — `mul/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`after_conv`](mul/after_conv/mul_after_conv.onnx) | Conv followed by per-channel scaling. |
|
||||
| [`after_conv_scalar_constant`](mul/after_conv_scalar_constant/mul_after_conv_scalar_constant.onnx) | Conv followed by Mul with a scalar constant. |
|
||||
| [`basic`](mul/basic/mul_basic.onnx) | Elementwise Mul on two inputs with identical shapes. |
|
||||
| [`channel_broadcast_1024`](mul/channel_broadcast_1024/mul_channel_broadcast_1024.onnx) | Mul with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| [`leading_dimension_broadcast`](mul/leading_dimension_broadcast/mul_leading_dimension_broadcast.onnx) | Mul with trailing-dimension broadcasting. |
|
||||
| [`scalar_constant`](mul/scalar_constant/mul_scalar_constant.onnx) | Mul with scalar broadcasting. |
|
||||
|
||||
## Pool (15) — `pool/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`avg_basic`](pool/avg_basic/pool_avg_basic.onnx) | AveragePool with a 2x2 kernel and unit stride. |
|
||||
| [`avg_ceil_mode`](pool/avg_ceil_mode/pool_avg_ceil_mode.onnx) | AveragePool with ceil mode enabled. |
|
||||
| [`avg_explicit_padding`](pool/avg_explicit_padding/pool_avg_explicit_padding.onnx) | Explicitly padded AveragePool excluding pad from the divisor. |
|
||||
| [`avg_include_pad`](pool/avg_include_pad/pool_avg_include_pad.onnx) | Explicitly padded AveragePool including pad in the divisor. |
|
||||
| [`avg_large_channels`](pool/avg_large_channels/pool_avg_large_channels.onnx) | AveragePool with a large channel count and small spatial extent. |
|
||||
| [`avg_non_uniform_stride`](pool/avg_non_uniform_stride/pool_avg_non_uniform_stride.onnx) | AveragePool with different height and width strides. |
|
||||
| [`avg_real_asymmetric_padding`](pool/avg_real_asymmetric_padding/pool_avg_real_asymmetric_padding.onnx) | AveragePool with asymmetric explicit padding. |
|
||||
| [`max_after_conv`](pool/max_after_conv/pool_max_after_conv.onnx) | Conv followed by MaxPool. |
|
||||
| [`max_basic`](pool/max_basic/pool_max_basic.onnx) | MaxPool with a 2x2 kernel and unit stride. |
|
||||
| [`max_ceil_mode`](pool/max_ceil_mode/pool_max_ceil_mode.onnx) | MaxPool with ceil mode enabled. |
|
||||
| [`max_global_style_kernel_equals_input`](pool/max_global_style_kernel_equals_input/pool_max_global_style_kernel_equals_input.onnx) | MaxPool whose kernel covers the full spatial input. |
|
||||
| [`max_non_square_kernel`](pool/max_non_square_kernel/pool_max_non_square_kernel.onnx) | MaxPool with a non-square kernel. |
|
||||
| [`max_real_asymmetric_padding`](pool/max_real_asymmetric_padding/pool_max_real_asymmetric_padding.onnx) | MaxPool with asymmetric explicit padding. |
|
||||
| [`max_same_upper`](pool/max_same_upper/pool_max_same_upper.onnx) | MaxPool with SAME_UPPER padding. |
|
||||
| [`max_stride2_multichannel`](pool/max_stride2_multichannel/pool_max_stride2_multichannel.onnx) | Multi-channel MaxPool with stride two. |
|
||||
|
||||
## ReduceMean (17) — `reduce_mean/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`4d_spatial`](reduce_mean/4d_spatial/reduce_mean_4d_spatial.onnx) | Reduces height and width of an NCHW tensor while preserving rank. |
|
||||
| [`4d_spatial_keepdims_0`](reduce_mean/4d_spatial_keepdims_0/reduce_mean_4d_spatial_keepdims_0.onnx) | Reduces NCHW height and width while dropping those axes. |
|
||||
| [`after_conv`](reduce_mean/after_conv/reduce_mean_after_conv.onnx) | Conv followed by a spatial ReduceMean. |
|
||||
| [`all_axes_keepdims_0`](reduce_mean/all_axes_keepdims_0/reduce_mean_all_axes_keepdims_0.onnx) | Reduces all axes to a scalar. |
|
||||
| [`all_axes_keepdims_1`](reduce_mean/all_axes_keepdims_1/reduce_mean_all_axes_keepdims_1.onnx) | Reduces all axes while preserving rank. |
|
||||
| [`basic`](reduce_mean/basic/reduce_mean_basic.onnx) | Reduces a feature dimension while preserving rank. |
|
||||
| [`channel_axis_nchw`](reduce_mean/channel_axis_nchw/reduce_mean_channel_axis_nchw.onnx) | Reduces the channel axis of an NCHW tensor. |
|
||||
| [`keepdims_0`](reduce_mean/keepdims_0/reduce_mean_keepdims_0.onnx) | Reduces a feature dimension and drops that axis. |
|
||||
| [`large_dimension_1024`](reduce_mean/large_dimension_1024/reduce_mean_large_dimension_1024.onnx) | Reduces a dimension of length 1024. |
|
||||
| [`legacy_axes_1_2_keepdims_1`](reduce_mean/legacy_axes_1_2_keepdims_1/reduce_mean_legacy_axes_1_2_keepdims_1.onnx) | Opset-18 reduction over multiple positive axes. |
|
||||
| [`legacy_axis1_keepdims_0`](reduce_mean/legacy_axis1_keepdims_0/reduce_mean_legacy_axis1_keepdims_0.onnx) | Opset-18 reduction over one axis while dropping it. |
|
||||
| [`legacy_axis1_keepdims_1`](reduce_mean/legacy_axis1_keepdims_1/reduce_mean_legacy_axis1_keepdims_1.onnx) | Opset-18 reduction over one axis while preserving rank. |
|
||||
| [`legacy_empty_axes_noop`](reduce_mean/legacy_empty_axes_noop/reduce_mean_legacy_empty_axes_noop.onnx) | Opset-18 empty-axes no-op followed by Relu. |
|
||||
| [`legacy_nchw_spatial`](reduce_mean/legacy_nchw_spatial/reduce_mean_legacy_nchw_spatial.onnx) | Opset-18 spatial reduction on NCHW input. |
|
||||
| [`legacy_negative_axis`](reduce_mean/legacy_negative_axis/reduce_mean_legacy_negative_axis.onnx) | Opset-18 reduction using a negative axis. |
|
||||
| [`legacy_reduce_all_keepdims_1`](reduce_mean/legacy_reduce_all_keepdims_1/reduce_mean_legacy_reduce_all_keepdims_1.onnx) | Opset-18 all-axis reduction with the axes input omitted. |
|
||||
| [`negative_axis`](reduce_mean/negative_axis/reduce_mean_negative_axis.onnx) | ReduceMean using a negative axis. |
|
||||
|
||||
## Relu (4) — `relu/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`4d`](relu/4d/relu_4d.onnx) | Standalone Relu on an NCHW tensor. |
|
||||
| [`after_conv`](relu/after_conv/relu_after_conv.onnx) | Conv followed by Relu. |
|
||||
| [`after_gemm`](relu/after_gemm/relu_after_gemm.onnx) | Gemm followed by Relu. |
|
||||
| [`basic`](relu/basic/relu_basic.onnx) | Standalone Relu on a 2D tensor. |
|
||||
|
||||
## Reshape (4) — `reshape/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`4d_to_2d_flatten`](reshape/4d_to_2d_flatten/reshape_4d_to_2d_flatten.onnx) | Flattens a 4D tensor into a 2D view. |
|
||||
| [`infer_dim_minus_one`](reshape/infer_dim_minus_one/reshape_infer_dim_minus_one.onnx) | Uses `-1` to infer one output dimension. |
|
||||
| [`same_rank`](reshape/same_rank/reshape_same_rank.onnx) | Changes shape without changing rank. |
|
||||
| [`zero_copies_input_dim`](reshape/zero_copies_input_dim/reshape_zero_copies_input_dim.onnx) | Uses `0` to copy an input dimension. |
|
||||
|
||||
## Resize (6) — `resize/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`height_only`](resize/height_only/resize_height_only.onnx) | Nearest-neighbor resize of only the height dimension. |
|
||||
| [`nearest_2x`](resize/nearest_2x/resize_nearest_2x.onnx) | Nearest-neighbor upsampling by a factor of two. |
|
||||
| [`nearest_downsample`](resize/nearest_downsample/resize_nearest_downsample.onnx) | Nearest-neighbor downsampling. |
|
||||
| [`non_uniform_scales`](resize/non_uniform_scales/resize_non_uniform_scales.onnx) | Nearest-neighbor resize with different spatial scales. |
|
||||
| [`width_only`](resize/width_only/resize_width_only.onnx) | Nearest-neighbor resize of only the width dimension. |
|
||||
| [`with_sizes`](resize/with_sizes/resize_with_sizes.onnx) | Resize using explicit output sizes instead of scales. |
|
||||
|
||||
## Sigmoid (3) — `sigmoid/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`4d`](sigmoid/4d/sigmoid_4d.onnx) | Standalone Sigmoid on an NCHW tensor. |
|
||||
| [`after_gemm`](sigmoid/after_gemm/sigmoid_after_gemm.onnx) | Gemm followed by Sigmoid. |
|
||||
| [`basic`](sigmoid/basic/sigmoid_basic.onnx) | Standalone Sigmoid on a 2D tensor. |
|
||||
|
||||
## Slice (8) — `slice/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`2d_basic`](slice/2d_basic/slice_2d_basic.onnx) | Slices a 2D tensor with explicit axes and unit steps. |
|
||||
| [`after_conv`](slice/after_conv/slice_after_conv.onnx) | Conv followed by a spatial crop. |
|
||||
| [`default_axes`](slice/default_axes/slice_default_axes.onnx) | Omits axes and steps to use positional defaults. |
|
||||
| [`large_channel_1024`](slice/large_channel_1024/slice_large_channel_1024.onnx) | Slices a channel range from a 1024-channel tensor. |
|
||||
| [`nchw_spatial_crop`](slice/nchw_spatial_crop/slice_nchw_spatial_crop.onnx) | Crops the spatial axes of an NCHW tensor. |
|
||||
| [`negative_axis`](slice/negative_axis/slice_negative_axis.onnx) | Slices using a negative axis. |
|
||||
| [`negative_indices`](slice/negative_indices/slice_negative_indices.onnx) | Slices using negative indices. |
|
||||
| [`step2`](slice/step2/slice_step2.onnx) | Slices using a positive step greater than one. |
|
||||
|
||||
## Softmax (5) — `softmax/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`3d_last_axis`](softmax/3d_last_axis/softmax_3d_last_axis.onnx) | Softmax over the last axis of a 3D tensor. |
|
||||
| [`basic`](softmax/basic/softmax_basic.onnx) | Softmax over the last dimension of a 2D tensor. |
|
||||
| [`channel_axis`](softmax/channel_axis/softmax_channel_axis.onnx) | Softmax over the channel axis of an NCHW tensor. |
|
||||
| [`large_dimension_1024`](softmax/large_dimension_1024/softmax_large_dimension_1024.onnx) | Softmax over a last dimension of length 1024. |
|
||||
| [`negative_axis`](softmax/negative_axis/softmax_negative_axis.onnx) | Softmax using a negative axis. |
|
||||
|
||||
## Split (4) — `split/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`basic`](split/basic/split_basic.onnx) | Splits a 2D tensor into two explicit output sizes. |
|
||||
| [`equal_three_way`](split/equal_three_way/split_equal_three_way.onnx) | Splits a 2D tensor evenly into three outputs. |
|
||||
| [`negative_axis`](split/negative_axis/split_negative_axis.onnx) | Splits using a negative axis. |
|
||||
| [`uneven_channel_axis_4d`](split/uneven_channel_axis_4d/split_uneven_channel_axis_4d.onnx) | Splits an NCHW channel axis into uneven outputs. |
|
||||
|
||||
## Sub (6) — `sub/`
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| [`after_gemm`](sub/after_gemm/sub_after_gemm.onnx) | Gemm followed by Sub with a broadcast constant vector. |
|
||||
| [`basic`](sub/basic/sub_basic.onnx) | Elementwise Sub on two runtime inputs with identical shapes. |
|
||||
| [`broadcast_row`](sub/broadcast_row/sub_broadcast_row.onnx) | Sub with a broadcast row-vector right-hand constant. |
|
||||
| [`channel_broadcast_1024`](sub/channel_broadcast_1024/sub_channel_broadcast_1024.onnx) | Sub with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| [`constant_lhs_broadcast`](sub/constant_lhs_broadcast/sub_constant_lhs_broadcast.onnx) | Sub with a broadcast constant left-hand operand. |
|
||||
| [`leading_dimension_broadcast`](sub/leading_dimension_broadcast/sub_leading_dimension_broadcast.onnx) | Sub with trailing-dimension broadcasting. |
|
||||
+19
-302
@@ -1,319 +1,36 @@
|
||||
# Operation validation suite
|
||||
|
||||
This directory contains the ONNX models used by `validation/validate.py` to
|
||||
validate individual operations through compilation, Pim simulation, and
|
||||
comparison with the ONNX-MLIR reference runtime.
|
||||
|
||||
## Naming
|
||||
|
||||
Every model uses the same path convention:
|
||||
|
||||
```text
|
||||
<category>/<case>/<category>_<case>.onnx
|
||||
```
|
||||
|
||||
The `<category>/<case>` pair is the operation ID printed by the validator.
|
||||
Use lowercase `snake_case` for both components. Keep the generator function,
|
||||
graph name, directory, and filename based on the same operation ID when adding
|
||||
a test.
|
||||
ONNX cases for compilation, functional Pim simulation, and comparison with the
|
||||
ONNX-MLIR reference runtime. Run commands from the repository root.
|
||||
|
||||
## Generate and validate
|
||||
|
||||
Regenerate all generated models from the repository root:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/operations/gen_tests.py
|
||||
```
|
||||
|
||||
Run the complete suite with deadlock detection:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/validate.py \
|
||||
--raptor-path build_release/Release/bin/onnx-mlir \
|
||||
--onnx-include-dir onnx-mlir/include \
|
||||
--operations-dir validation/operations \
|
||||
--raptor-extra-arg=--pim-detect-communication-deadlock \
|
||||
--raptor-extra-arg=--pim-export-spatial-dataflow=none
|
||||
--raptor-extra-arg=--pim-detect-communication-deadlock
|
||||
```
|
||||
|
||||
Use `--compile-only` for compiler and deadlock checks, then `--run-only` to
|
||||
reuse those artifacts for reference execution, simulation, and comparison.
|
||||
The validator prints separate latency and throughput operation tables before
|
||||
its summary and writes all of their rows to `validation_results.csv`.
|
||||
The complete suite uses parallel workers and must run outside the sandbox.
|
||||
Point `--operations-dir` at a category or case, such as `gemm/small` below this
|
||||
directory, for focused validation. Use `--compile-only` followed by `--run-only`
|
||||
to separate compilation from execution.
|
||||
|
||||
## Complete inventory
|
||||
See the [validation guide](../README.md) for prerequisites, options, artifacts,
|
||||
and result interpretation.
|
||||
|
||||
The suite contains 177 models. Tensor shapes, attributes, and constants are
|
||||
defined in `gen_tests.py` and in the checked-in ONNX models.
|
||||
## Cases and naming
|
||||
|
||||
### Add (5)
|
||||
Models use `<category>/<case>/<category>_<case>.onnx`. The validator prints
|
||||
`<category>/<case>` as the operation ID. Use lowercase `snake_case` and keep
|
||||
the generator function, graph name, directory, and filename consistent.
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `after_gemm` | Gemm followed by Add with a broadcast bias vector. |
|
||||
| `basic` | Elementwise Add on two inputs with identical shapes. |
|
||||
| `broadcast_row` | Elementwise Add with row-vector broadcasting. |
|
||||
| `channel_broadcast_1024` | NCHW per-channel broadcasting over 1024 channels. |
|
||||
| `leading_dimension_broadcast` | Trailing-dimension broadcasting across leading dimensions. |
|
||||
|
||||
### Concat (3)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `channel_axis` | Concatenates two runtime NCHW tensors along the channel axis. |
|
||||
| `negative_axis` | Concatenates tensors using a negative axis. |
|
||||
| `three_inputs_channel_axis` | Concatenates three runtime NCHW tensors along the channel axis. |
|
||||
|
||||
### Conv (42)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `batch_2` | Batched Conv with SAME_UPPER padding and bias. |
|
||||
| `batch_4_pointwise` | Pointwise Conv with batch size four. |
|
||||
| `input_224_7x7_stride2` | 224x224 RGB Conv with 64 output channels, a 7x7 kernel, stride two, and bias. |
|
||||
| `depthwise_1024_channels` | Depthwise pointwise Conv with 1024 groups. |
|
||||
| `depthwise_grouped` | Depthwise-style grouped Conv with one input channel per group. |
|
||||
| `dilated_3x3` | Conv with a dilated 3x3 kernel. |
|
||||
| `dynamic` | Conv with runtime input and weight tensors. |
|
||||
| `explicit_padding` | 3x3 Conv with symmetric explicit padding. |
|
||||
| `grouped_many_groups` | Pointwise Conv with many groups and high channel counts. |
|
||||
| `grouped_two_groups` | Two-group pointwise Conv with bias. |
|
||||
| `huge_pointwise_1024` | Pointwise Conv with 1024 input and output channels. |
|
||||
| `huge_pointwise_1024_dynamic` | The 1024-channel pointwise Conv with runtime weights. |
|
||||
| `kernel_3x3` | Basic 3x3 Conv without padding. |
|
||||
| `kernel_equals_input_spatial` | Conv whose kernel covers the full spatial input. |
|
||||
| `large_input_channels_1x1` | Pointwise Conv with 1024 input channels and modest output width. |
|
||||
| `large_output_channels_1x1` | Pointwise Conv with modest input width and 1024 output channels. |
|
||||
| `large_spatial` | 3x3 Conv on a larger spatial input. |
|
||||
| `multi_channel` | 3x3 Conv with multiple input and output channels. |
|
||||
| `non_square_kernel_1x3` | Conv with a non-square 1x3 kernel. |
|
||||
| `non_square_kernel_3x1` | Conv with a non-square 3x1 kernel. |
|
||||
| `non_uniform_stride` | Conv with different height and width strides. |
|
||||
| `output_channel_grouping_minimal` | Minimal 64-to-256 pointwise Conv for output-channel grouping. |
|
||||
| `pointwise_1x1` | Basic pointwise channel-mixing Conv. |
|
||||
| `pointwise_tiled_chain` | Relu and chained pointwise Convs with a tiled intermediate. |
|
||||
| `real_asymmetric_padding` | Conv with asymmetric explicit padding. |
|
||||
| `relu_conv_store` | Conv, Relu, and a second Conv to validate an intermediate stored result. |
|
||||
| `same_lower_3x3` | 3x3 Conv with SAME_LOWER padding. |
|
||||
| `same_padding_3x3` | 3x3 Conv with SAME_UPPER padding. |
|
||||
| `kernel_2x2` | Hand-authored Conv with a 2x2 kernel. |
|
||||
| `stride_2` | 3x3 Conv with stride two. |
|
||||
| `strategy_depthwise_16` | 16-channel depthwise 3x3 Conv for depthwise lowering. |
|
||||
| `strategy_input_k_tiled` | 32-channel 3x3 Conv sized to exercise input-K tiling. |
|
||||
| `strategy_output_channel_tiled` | 8-to-192 3x3 Conv sized to exercise output-channel tiling. |
|
||||
| `strategy_streamed_packed` | 3-to-16 3x3 Conv on 128x128 input for streamed packed lowering. |
|
||||
| `strategy_streamed_patch` | 3-to-16 3x3 Conv on 64x64 input for streamed patch lowering. |
|
||||
| `strategy_tiled_2d` | 32-to-192 3x3 Conv sized to exercise 2D tiling. |
|
||||
| `with_bias_3x3` | Multi-channel 3x3 Conv with bias. |
|
||||
| `with_constant` | Hand-authored SAME_UPPER Conv with constant weight and bias. |
|
||||
| `without_kernel_shape_attr` | Conv whose kernel shape is inferred from its weight tensor. |
|
||||
| `yolo11n_depthwise_head` | YOLO11n pointwise-to-depthwise head boundary at `80x80`, preserving row fragments. |
|
||||
| `yolo11n_heavy` | Two largest standard YOLO11n Conv-SiLU blocks by MAC count at `64x80x80`. |
|
||||
| `yolo11n_stem` | First two YOLO11n `Conv-SiLU` blocks at `640x640`, including the distributed activation boundary. |
|
||||
|
||||
### Div (6)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `after_gemm` | Gemm followed by Div with a broadcast divisor vector. |
|
||||
| `basic` | Elementwise Div by a same-shape constant tensor. |
|
||||
| `channel_broadcast_1024` | Div with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| `leading_dimension_broadcast` | Div with trailing-dimension broadcasting. |
|
||||
| `runtime_scalar_rhs` | Div of a runtime tensor by a scalar initializer. |
|
||||
| `scalar_constant` | Div with scalar broadcasting on a 2D tensor. |
|
||||
|
||||
### Gather (5)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `3d_input_axis1` | Gathers along axis one of a 3D input. |
|
||||
| `axis0_matrix_indices` | Gathers rows using a 2D indices tensor. |
|
||||
| `axis1` | Gathers selected columns from a 2D tensor. |
|
||||
| `negative_axis` | Gathers using a negative axis. |
|
||||
| `negative_indices` | Gathers with negative indices along axis zero. |
|
||||
|
||||
### Gemm (21)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `alpha_beta` | Applies non-default alpha and beta scaling with bias. |
|
||||
| `bias_rank2_broadcast` | Broadcasts a rank-2 bias across output rows. |
|
||||
| `dynamic` | Uses both matrix operands at runtime. |
|
||||
| `dynamic_alpha` | Uses runtime operands with non-default alpha scaling. |
|
||||
| `dynamic_beta` | Uses runtime operands and bias with non-default beta scaling. |
|
||||
| `dynamic_bias` | Uses runtime matrix operands and runtime bias. |
|
||||
| `dynamic_bias_alpha_beta` | Combines runtime operands and bias with alpha and beta scaling. |
|
||||
| `dynamic_transpose_b` | Transposes a runtime right-hand matrix. |
|
||||
| `huge_1024` | Uses 1024-wide inner and output dimensions. |
|
||||
| `large` | Exercises larger rectangular matrices. |
|
||||
| `large_k_small_n` | Uses a large reduction dimension and narrow output. |
|
||||
| `non_square` | Uses different reduction and output widths. |
|
||||
| `scalar_bias` | Broadcasts a scalar bias to the full output. |
|
||||
| `square_weights` | Basic Gemm with square weights. |
|
||||
| `small` | Tiny Gemm for fast focused validation. |
|
||||
| `small_k_large_n` | Uses a modest reduction dimension and wide output. |
|
||||
| `transpose_a` | Transposes the left-hand matrix. |
|
||||
| `transpose_a_and_b` | Transposes both matrix operands. |
|
||||
| `transpose_b` | Transposes the right-hand weight matrix. |
|
||||
| `transpose_b_with_bias` | Combines a transposed weight matrix with bias. |
|
||||
| `with_bias` | Basic matrix product with vector bias. |
|
||||
|
||||
### Gemv (5)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `all_constant` | Vector-matrix product with all inputs constant. |
|
||||
| `constant_weight` | Basic single-row vector-matrix product with constant weights. |
|
||||
| `non_uniform_bias` | Adds a non-uniform constant bias pattern. |
|
||||
| `uniform_bias` | Adds a uniform constant bias pattern. |
|
||||
| `scalar_bias` | Adds a scalar broadcast bias. |
|
||||
|
||||
### MatMul (12)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `basic` | Direct 2D MatMul with constant right-hand matrix. |
|
||||
| `batched_3d` | Batched 3D MatMul with matching batch dimensions. |
|
||||
| `batched_3d_dynamic` | Batched 3D MatMul with both operands at runtime. |
|
||||
| `batched_left_constant` | Batched 3D MatMul with constant left-hand matrix. |
|
||||
| `batched_lhs_broadcast` | Broadcasts a 2D left-hand matrix across a batched right-hand tensor. |
|
||||
| `batched_rhs_broadcast` | Broadcasts a 2D right-hand matrix across a batched left-hand tensor. |
|
||||
| `dynamic` | Direct 2D MatMul with both operands at runtime. |
|
||||
| `huge_1024` | Uses 1024-wide inner and output dimensions. |
|
||||
| `left_constant` | Direct 2D MatMul with constant left-hand matrix. |
|
||||
| `matrix_vector` | Matrix-vector multiplication producing a 1D output. |
|
||||
| `vector_matrix` | Vector-matrix multiplication producing a 1D output. |
|
||||
| `yolo_attention` | YOLO11n rank-4 dynamic MatMul-scale-transpose-MatMul attention chain. |
|
||||
|
||||
### Mul (6)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `after_conv` | Conv followed by per-channel scaling. |
|
||||
| `after_conv_scalar_constant` | Conv followed by Mul with a scalar constant. |
|
||||
| `basic` | Elementwise Mul on two inputs with identical shapes. |
|
||||
| `channel_broadcast_1024` | Mul with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| `leading_dimension_broadcast` | Mul with trailing-dimension broadcasting. |
|
||||
| `scalar_constant` | Mul with scalar broadcasting. |
|
||||
|
||||
### Pool (15)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `avg_basic` | AveragePool with a 2x2 kernel and unit stride. |
|
||||
| `avg_ceil_mode` | AveragePool with ceil mode enabled. |
|
||||
| `avg_explicit_padding` | Explicitly padded AveragePool excluding pad from the divisor. |
|
||||
| `avg_include_pad` | Explicitly padded AveragePool including pad in the divisor. |
|
||||
| `avg_large_channels` | AveragePool with a large channel count and small spatial extent. |
|
||||
| `avg_non_uniform_stride` | AveragePool with different height and width strides. |
|
||||
| `avg_real_asymmetric_padding` | AveragePool with asymmetric explicit padding. |
|
||||
| `max_after_conv` | Conv followed by MaxPool. |
|
||||
| `max_basic` | MaxPool with a 2x2 kernel and unit stride. |
|
||||
| `max_ceil_mode` | MaxPool with ceil mode enabled. |
|
||||
| `max_global_style_kernel_equals_input` | MaxPool whose kernel covers the full spatial input. |
|
||||
| `max_non_square_kernel` | MaxPool with a non-square kernel. |
|
||||
| `max_real_asymmetric_padding` | MaxPool with asymmetric explicit padding. |
|
||||
| `max_same_upper` | MaxPool with SAME_UPPER padding. |
|
||||
| `max_stride2_multichannel` | Multi-channel MaxPool with stride two. |
|
||||
|
||||
### ReduceMean (17)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `4d_spatial` | Reduces height and width of an NCHW tensor while preserving rank. |
|
||||
| `4d_spatial_keepdims_0` | Reduces NCHW height and width while dropping those axes. |
|
||||
| `after_conv` | Conv followed by a spatial ReduceMean. |
|
||||
| `all_axes_keepdims_0` | Reduces all axes to a scalar. |
|
||||
| `all_axes_keepdims_1` | Reduces all axes while preserving rank. |
|
||||
| `basic` | Reduces a feature dimension while preserving rank. |
|
||||
| `channel_axis_nchw` | Reduces the channel axis of an NCHW tensor. |
|
||||
| `keepdims_0` | Reduces a feature dimension and drops that axis. |
|
||||
| `large_dimension_1024` | Reduces a dimension of length 1024. |
|
||||
| `legacy_axes_1_2_keepdims_1` | Opset-18 reduction over multiple positive axes. |
|
||||
| `legacy_axis1_keepdims_0` | Opset-18 reduction over one axis while dropping it. |
|
||||
| `legacy_axis1_keepdims_1` | Opset-18 reduction over one axis while preserving rank. |
|
||||
| `legacy_empty_axes_noop` | Opset-18 empty-axes no-op followed by Relu. |
|
||||
| `legacy_nchw_spatial` | Opset-18 spatial reduction on NCHW input. |
|
||||
| `legacy_negative_axis` | Opset-18 reduction using a negative axis. |
|
||||
| `legacy_reduce_all_keepdims_1` | Opset-18 all-axis reduction with the axes input omitted. |
|
||||
| `negative_axis` | ReduceMean using a negative axis. |
|
||||
|
||||
### Relu (4)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `4d` | Standalone Relu on an NCHW tensor. |
|
||||
| `after_conv` | Conv followed by Relu. |
|
||||
| `after_gemm` | Gemm followed by Relu. |
|
||||
| `basic` | Standalone Relu on a 2D tensor. |
|
||||
|
||||
### Reshape (4)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `4d_to_2d_flatten` | Flattens a 4D tensor into a 2D view. |
|
||||
| `infer_dim_minus_one` | Uses `-1` to infer one output dimension. |
|
||||
| `same_rank` | Changes shape without changing rank. |
|
||||
| `zero_copies_input_dim` | Uses `0` to copy an input dimension. |
|
||||
|
||||
### Resize (6)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `height_only` | Nearest-neighbor resize of only the height dimension. |
|
||||
| `nearest_2x` | Nearest-neighbor upsampling by a factor of two. |
|
||||
| `nearest_downsample` | Nearest-neighbor downsampling. |
|
||||
| `non_uniform_scales` | Nearest-neighbor resize with different spatial scales. |
|
||||
| `width_only` | Nearest-neighbor resize of only the width dimension. |
|
||||
| `with_sizes` | Resize using explicit output sizes instead of scales. |
|
||||
|
||||
### Sigmoid (3)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `4d` | Standalone Sigmoid on an NCHW tensor. |
|
||||
| `after_gemm` | Gemm followed by Sigmoid. |
|
||||
| `basic` | Standalone Sigmoid on a 2D tensor. |
|
||||
|
||||
### Slice (8)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `2d_basic` | Slices a 2D tensor with explicit axes and unit steps. |
|
||||
| `after_conv` | Conv followed by a spatial crop. |
|
||||
| `default_axes` | Omits axes and steps to use positional defaults. |
|
||||
| `large_channel_1024` | Slices a channel range from a 1024-channel tensor. |
|
||||
| `nchw_spatial_crop` | Crops the spatial axes of an NCHW tensor. |
|
||||
| `negative_axis` | Slices using a negative axis. |
|
||||
| `negative_indices` | Slices using negative indices. |
|
||||
| `step2` | Slices using a positive step greater than one. |
|
||||
|
||||
### Softmax (5)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `3d_last_axis` | Softmax over the last axis of a 3D tensor. |
|
||||
| `basic` | Softmax over the last dimension of a 2D tensor. |
|
||||
| `channel_axis` | Softmax over the channel axis of an NCHW tensor. |
|
||||
| `large_dimension_1024` | Softmax over a last dimension of length 1024. |
|
||||
| `negative_axis` | Softmax using a negative axis. |
|
||||
|
||||
### Split (4)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `basic` | Splits a 2D tensor into two explicit output sizes. |
|
||||
| `equal_three_way` | Splits a 2D tensor evenly into three outputs. |
|
||||
| `negative_axis` | Splits using a negative axis. |
|
||||
| `uneven_channel_axis_4d` | Splits an NCHW channel axis into uneven outputs. |
|
||||
|
||||
### Sub (6)
|
||||
|
||||
| Case | Description |
|
||||
|---|---|
|
||||
| `after_gemm` | Gemm followed by Sub with a broadcast constant vector. |
|
||||
| `basic` | Elementwise Sub on two runtime inputs with identical shapes. |
|
||||
| `broadcast_row` | Sub with a broadcast row-vector right-hand constant. |
|
||||
| `channel_broadcast_1024` | Sub with NCHW per-channel broadcasting over 1024 channels. |
|
||||
| `constant_lhs_broadcast` | Sub with a broadcast constant left-hand operand. |
|
||||
| `leading_dimension_broadcast` | Sub with trailing-dimension broadcasting. |
|
||||
The [complete inventory](INVENTORY.md) describes all 177 checked-in cases.
|
||||
The suite covers arithmetic and broadcasting, convolution strategies, matrix
|
||||
products, pooling, reductions, activations, and tensor indexing/layout changes,
|
||||
including chained and YOLO-derived regression cases. The case directories and
|
||||
[`gen_tests.py`](gen_tests.py) define the inventory, shapes, attributes, and
|
||||
constants; some checked-in models are hand-authored.
|
||||
|
||||
@@ -0,0 +1,393 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import csv
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
TOOLS = Path(__file__).resolve().parents[1] / "tools/pim"
|
||||
sys.path.insert(0, str(TOOLS / "ablation"))
|
||||
sys.path.insert(0, str(TOOLS / "pimcomp/compare"))
|
||||
|
||||
import run_ablation as ablation
|
||||
import run_pimcomp_models as suite
|
||||
import compare_raptor_pimcomp_model as comparator
|
||||
|
||||
|
||||
def test_ablation_results_stay_in_independent_output_roots(tmp_path):
|
||||
variants = ablation.selected_variants(["no-plan"])
|
||||
outputs = []
|
||||
for name, value in (("first", 80), ("second", 60)):
|
||||
root = tmp_path / name
|
||||
for variant in variants:
|
||||
path = ablation.variant_output_dir(root, variant) / "results_comparison.csv"
|
||||
path.parent.mkdir(parents=True)
|
||||
row = {"arch": "arch-a", "model": "vgg8", "mode": "latency",
|
||||
"raptor_pipeline": "1", "raptor_functional_validation": "PASS"}
|
||||
row.update({field: 100 if variant.name == "baseline" else value
|
||||
for field, _ in ablation.PERCENTAGE_FIELDS})
|
||||
with path.open("w", newline="") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=list(row))
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
output, failures = ablation.aggregate_results(root, variants)
|
||||
assert failures == []
|
||||
assert output == root / "results_ablation.csv"
|
||||
outputs.append(output)
|
||||
for output, change in zip(outputs, ("-20.00%", "-40.00%")):
|
||||
with output.open(newline="") as stream:
|
||||
rows = list(csv.DictReader(stream))
|
||||
assert rows[0]["latency"] == "+0.00%"
|
||||
assert rows[1]["latency"] == change
|
||||
|
||||
|
||||
@pytest.mark.parametrize("custom_input", [False, True])
|
||||
def test_ablation_plots_follow_the_selected_experiment(tmp_path, monkeypatch, custom_input):
|
||||
spec = importlib.util.spec_from_file_location("ablation_plots", TOOLS / "ablation/generate_plots.py")
|
||||
plots = importlib.util.module_from_spec(spec)
|
||||
monkeypatch.setitem(sys.modules, spec.name, plots)
|
||||
spec.loader.exec_module(plots)
|
||||
expected = tmp_path / "experiment/results_ablation.csv" if custom_input else (
|
||||
ablation.DEFAULT_RESULTS_PATH
|
||||
)
|
||||
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom_input else [])])
|
||||
result = plots.Result("baseline", "arch-a", "vgg8", "latency", 1, "PASS", {})
|
||||
inputs, outputs = [], []
|
||||
monkeypatch.setattr(plots, "configure_style", lambda: None)
|
||||
monkeypatch.setattr(plots, "load_results", lambda path: inputs.append(path) or [result])
|
||||
monkeypatch.setattr(plots, "draw_metric", lambda rows, metric, label, output: outputs.append(output))
|
||||
plots.main()
|
||||
assert inputs == [expected]
|
||||
assert outputs == [expected.parent / "plots"] * 3
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorded_failure(tmp_path):
|
||||
model, functional, config = (tmp_path / name for name in ("model.onnx", "functional.onnx", "config.json"))
|
||||
for path in (model, functional, config):
|
||||
path.write_text(path.name)
|
||||
args = argparse.Namespace(
|
||||
pimcomp_config=config, pimsim_mode="throughput", pimsim_time_ms=1000,
|
||||
pimcomp_pipeline="batch", pimcomp_replication="GA", pimcomp_model_name=None,
|
||||
seed=0, batch_size=64, threshold=1e-3, rtol=1e-4, no_fast=False,
|
||||
skip_pimsim_nn=False,
|
||||
)
|
||||
hardware = {"core_count": 168, "crossbar_size": 128}
|
||||
context = comparator.pimcomp_reuse_context(args, model, functional, hardware)
|
||||
report = {
|
||||
"pimcomp_model_source": "original_onnx", "pimcomp_reuse_context": context,
|
||||
"pimcomp_validation": {"passed": False, "max_diffs": {}, "status": "failed",
|
||||
"error": "Unsupported model"},
|
||||
"pimcomp_performance": {"skipped": True, "reason": "Compilation failed"},
|
||||
"pimcomp_instruction_summary": comparator.empty_instruction_summary("No artifact"),
|
||||
"paths": {}, "steps": [],
|
||||
"failures": [{"stage": "Compile Pimcomp frontend", "error": "Unsupported model"}],
|
||||
}
|
||||
path = tmp_path / "comparison_report.json"
|
||||
path.write_text(json.dumps(report))
|
||||
return path, report, args, model, functional, hardware
|
||||
|
||||
|
||||
def test_failed_pimcomp_report_needs_no_simulation_artifacts(recorded_failure):
|
||||
path, report, *_ = recorded_failure
|
||||
reused = comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"])
|
||||
assert reused == report
|
||||
assert not reused["pimcomp_validation"]["passed"]
|
||||
failures = []
|
||||
comparator.restore_side_records(reused, "pimcomp", failures, [])
|
||||
assert failures == report["failures"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("setting,value", [
|
||||
("pimsim_mode", "latency"), ("pimcomp_pipeline", "element"),
|
||||
("pimsim_time_ms", 2000), ("pimcomp_replication", "balance"),
|
||||
("seed", 1), ("batch_size", 128), ("no_fast", True),
|
||||
("skip_pimsim_nn", True), ("threshold", 0.1), ("rtol", 0.01),
|
||||
])
|
||||
def test_reuse_rejects_different_experiment_settings(recorded_failure, setting, value):
|
||||
path, report, args, model, functional, hardware = recorded_failure
|
||||
setattr(args, setting, value)
|
||||
expected = comparator.pimcomp_reuse_context(args, model, functional, hardware)
|
||||
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
|
||||
comparator.load_reused_pimcomp_report(path, expected)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("changed", ["model", "functional", "config", "hardware", "model_path"])
|
||||
def test_reuse_rejects_changed_inputs_and_architecture(recorded_failure, changed):
|
||||
path, report, args, model, functional, hardware = recorded_failure
|
||||
if changed == "hardware":
|
||||
hardware["core_count"] = 138
|
||||
elif changed == "model_path":
|
||||
replacement = model.with_name("other.onnx")
|
||||
replacement.write_bytes(model.read_bytes())
|
||||
model = replacement
|
||||
else:
|
||||
{"model": model, "functional": functional, "config": args.pimcomp_config}[changed].write_text("changed")
|
||||
expected = comparator.pimcomp_reuse_context(args, model, functional, hardware)
|
||||
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
|
||||
comparator.load_reused_pimcomp_report(path, expected)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage", ["json", "legacy", "validation", "performance", "summary", "steps"])
|
||||
def test_reuse_rejects_incomplete_reports(recorded_failure, damage):
|
||||
path, report, *_ = recorded_failure
|
||||
expected = copy.deepcopy(report["pimcomp_reuse_context"])
|
||||
if damage == "json":
|
||||
path.write_text("{")
|
||||
else:
|
||||
field = {"legacy": "pimcomp_reuse_context", "validation": "pimcomp_validation",
|
||||
"performance": "pimcomp_performance", "summary": "pimcomp_instruction_summary",
|
||||
"steps": "steps"}[damage]
|
||||
report.pop(field)
|
||||
path.write_text(json.dumps(report))
|
||||
with pytest.raises(ValueError, match="Rerun Pimcomp"):
|
||||
comparator.load_reused_pimcomp_report(path, expected)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("only,fresh", [(None, True), (None, False), ("pimcomp", True), ("raptor", False)])
|
||||
def test_suite_reruns_anchor_and_shares_only_new_reports(tmp_path, monkeypatch, only, fresh):
|
||||
root = tmp_path / "suite"
|
||||
for _, pipeline, _ in suite.COMPARISONS[1:]:
|
||||
report = suite.result_dir(root, "vgg8", "arch-a", "throughput", pipeline) / "pimcomp/comparison_report.json"
|
||||
report.parent.mkdir(parents=True)
|
||||
report.write_text("{}")
|
||||
os.utime(report, ns=(1, 1))
|
||||
batches = []
|
||||
|
||||
def run_jobs(jobs, max_workers, log_dir, offset):
|
||||
assert max_workers == 1
|
||||
batches.append([command for _, command, _ in jobs])
|
||||
for _, command, _ in jobs:
|
||||
output = Path(command[command.index("--out-dir") + 1]) / "pimcomp/comparison_report.json"
|
||||
if fresh or len(batches) > 1:
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text('{"new": true}')
|
||||
return [], offset + len(jobs)
|
||||
|
||||
summary = tmp_path / "results.csv"
|
||||
summary.write_text("summary\n")
|
||||
monkeypatch.setattr(suite, "validate_pimcomp_source", lambda: None)
|
||||
monkeypatch.setattr(suite, "run", lambda *a, **k: 0)
|
||||
monkeypatch.setattr(suite, "run_comparison_jobs", run_jobs)
|
||||
monkeypatch.setattr(suite, "write_results_csv", lambda *a: summary)
|
||||
monkeypatch.setattr(suite, "comparison_passed", lambda report: True)
|
||||
monkeypatch.setattr(sys, "argv", [
|
||||
"run_pimcomp_models.py", "--out-dir", str(root), "--models", "vgg8",
|
||||
"--archs", "arch-a", "--mode", "throughput", "--jobs", "1",
|
||||
*(["--only", only] if only else []),
|
||||
])
|
||||
assert suite.main() == 0
|
||||
if only == "raptor":
|
||||
assert len(batches[0]) == 3
|
||||
assert all("--reuse-pimcomp-report" in command for command in batches[0])
|
||||
else:
|
||||
assert len(batches[0]) == 1
|
||||
assert "--reuse-pimcomp-report" not in batches[0][0]
|
||||
assert all(("--reuse-pimcomp-report" in command) == fresh for command in batches[1])
|
||||
assert ("--reuse-raptor-report" in batches[0][0]) == (only == "pimcomp")
|
||||
|
||||
|
||||
def test_missing_pimcomp_report_is_not_reused(tmp_path):
|
||||
assert not suite.pimcomp_report_ready(tmp_path / "output", None)
|
||||
|
||||
|
||||
def test_missing_pimcomp_report_is_rejected_at_load(recorded_failure):
|
||||
path, report, *_ = recorded_failure
|
||||
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
|
||||
comparator.load_reused_pimcomp_report(path.with_name("missing.json"), report["pimcomp_reuse_context"])
|
||||
|
||||
|
||||
def test_canonical_default_result_locations():
|
||||
assert suite.DEFAULT_RESULTS_PATH == TOOLS / "pimcomp/compare/results_comparison.csv"
|
||||
assert ablation.DEFAULT_RESULTS_PATH == TOOLS / "ablation/results_ablation.csv"
|
||||
|
||||
|
||||
def test_comparison_default_and_custom_aggregate_paths(tmp_path, monkeypatch):
|
||||
canonical = tmp_path / "scripts/results_comparison.csv"
|
||||
artifact_root = tmp_path / "models"
|
||||
monkeypatch.setattr(suite, "DEFAULT_RESULTS_PATH", canonical)
|
||||
monkeypatch.setattr(suite, "SUITE", artifact_root)
|
||||
assert suite.write_results_csv(None, "arch-a", []) == canonical
|
||||
original = canonical.read_bytes()
|
||||
for name in ("first", "second"):
|
||||
custom = tmp_path / name
|
||||
assert suite.write_results_csv(custom, "arch-a", []) == custom / "results_comparison.csv"
|
||||
assert canonical.read_bytes() == original
|
||||
variant = suite.write_results_csv(None, "arch-a", [], ablation_variant="baseline")
|
||||
assert variant == artifact_root / "ablation/baseline/results_comparison.csv"
|
||||
suite.clean_artifacts(None, [])
|
||||
assert canonical.read_bytes() == original
|
||||
|
||||
|
||||
@pytest.mark.parametrize("custom", [False, True])
|
||||
def test_ablation_cli_selects_canonical_or_custom_output(tmp_path, monkeypatch, custom):
|
||||
artifact_root = tmp_path / "models"
|
||||
canonical = tmp_path / "scripts/results_ablation.csv"
|
||||
monkeypatch.setattr(ablation, "DEFAULT_OUT_DIR", artifact_root)
|
||||
monkeypatch.setattr(ablation, "DEFAULT_RESULTS_PATH", canonical)
|
||||
monkeypatch.setattr(ablation.subprocess, "Popen", lambda *a, **k: argparse.Namespace(wait=lambda: 0))
|
||||
calls = []
|
||||
monkeypatch.setattr(ablation, "aggregate_results", lambda root, *a, **k: (calls.append((root, k["output"])) or None, []))
|
||||
command = ["--variant", "baseline", "--jobs", "1"]
|
||||
if custom:
|
||||
command += ["--out-dir", str(tmp_path / "custom")]
|
||||
assert ablation.main(command) == 0
|
||||
assert calls == [(tmp_path / "custom", None) if custom else (artifact_root, canonical)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [None, "", "NA", "0", "0.00", "-1", "nan", "inf"])
|
||||
def test_ablation_percentages_reject_unavailable_or_invalid_values(value):
|
||||
assert ablation.percentage_difference(value, "100") == "NA"
|
||||
assert ablation.percentage_difference("100", value) == "NA"
|
||||
|
||||
|
||||
def test_tiny_positive_measurements_and_extreme_changes_are_preserved():
|
||||
value = 1e-12
|
||||
assert float(suite.format_value(value)) == value
|
||||
assert ablation.percentage_difference(suite.format_value(value), "100") == "-100.00%"
|
||||
assert ablation.percentage_difference("1000000", "1") == "+99999900.00%"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [{"error": "timeout"}, {"skipped": True}, {"unsupported": True}])
|
||||
def test_failed_performance_never_uses_stale_numeric_fields(status):
|
||||
metrics = {"latency_ms": 10, "throughput": 100, "average_power_mw": 20,
|
||||
"average_energy_pj": 30, **status}
|
||||
assert all(value is None for value in suite.performance_values(metrics).values())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [None, "NA", 0, -1, float("nan"), float("inf")])
|
||||
def test_comparison_aggregation_rejects_invalid_measurements(value):
|
||||
metrics = suite.performance_values({"latency_ms": value, "throughput": value,
|
||||
"average_power_mw": value, "average_energy_pj": value})
|
||||
assert all(number is None for number in metrics.values())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("primary", [None, "NA", 0, -1, float("nan"), float("inf"), -float("inf")])
|
||||
def test_latency_uses_valid_fallback_after_invalid_primary(primary):
|
||||
metrics = suite.performance_values({"latency_ms": primary, "average_latency_ms": "0.0063"})
|
||||
assert metrics["latency"] == 0.0063
|
||||
|
||||
|
||||
def test_latency_preserves_candidate_priority():
|
||||
metrics = suite.performance_values({"latency_ms": "0.014", "average_latency_ms": 0.015})
|
||||
assert metrics["latency"] == 0.014
|
||||
|
||||
|
||||
@pytest.mark.parametrize("primary,fallback", [
|
||||
(None, "NA"), ("NA", None), (0, -1), (-1, 0),
|
||||
(float("nan"), float("inf")), (float("inf"), -float("inf")),
|
||||
])
|
||||
def test_latency_is_missing_when_all_candidates_are_invalid(primary, fallback):
|
||||
metrics = suite.performance_values({"latency_ms": primary, "average_latency_ms": fallback})
|
||||
assert metrics["latency"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("invalid", [None, "NA", 0, -1, float("nan"), float("inf"), -float("inf")])
|
||||
@pytest.mark.parametrize("first_valid", range(5))
|
||||
def test_latency_priority_across_structured_and_parsed_metrics(monkeypatch, invalid, first_valid):
|
||||
candidates = [invalid if index < first_valid else index + 1 for index in range(4)]
|
||||
parsed = dict(zip(("latency_ms", "average_latency_ms"), candidates[2:]))
|
||||
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
|
||||
performance = dict(zip(("latency_ms", "average_latency_ms"), candidates[:2]))
|
||||
assert suite.performance_values(performance)["latency"] == (first_valid + 1 if first_valid < 4 else None)
|
||||
|
||||
|
||||
def test_structured_average_latency_precedes_parsed_latency(monkeypatch):
|
||||
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: {"latency_ms": 1})
|
||||
assert suite.performance_values({"average_latency_ms": 2})["latency"] == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,metric", [
|
||||
("throughput", "throughput"), ("average_power_mw", "power"), ("average_energy_pj", "energy"),
|
||||
])
|
||||
@pytest.mark.parametrize("structured,parsed,expected", [
|
||||
(10, 20, 10), (None, 20, 20), ("NA", 20, 20), (0, 20, 20), (-1, 20, 20),
|
||||
(float("nan"), 20, 20), (float("inf"), 20, 20), (-float("inf"), 20, 20),
|
||||
(0, float("nan"), None),
|
||||
])
|
||||
def test_measurement_priority_across_sources(monkeypatch, field, metric, structured, parsed, expected):
|
||||
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: {field: parsed})
|
||||
assert suite.performance_values({field: structured})[metric] == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [
|
||||
{"error": "timeout"}, {"skipped": True}, {"unsupported": True},
|
||||
{"output_count": 0},
|
||||
])
|
||||
def test_parsed_measurements_respect_simulation_status(monkeypatch, status):
|
||||
parsed = {"latency_ms": 1, "throughput": 2, "average_power_mw": 3, "average_energy_pj": 4}
|
||||
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
|
||||
assert suite.performance_values(status) == {
|
||||
"latency": None, "throughput": None, "power": 3 if "output_count" in status else None,
|
||||
"energy": None,
|
||||
}
|
||||
|
||||
|
||||
def test_parsed_zero_outputs_preserve_only_measured_power(monkeypatch):
|
||||
parsed = {"output_count": 0, "latency_ms": 1, "throughput": 2,
|
||||
"average_power_mw": 3, "average_energy_pj": 4}
|
||||
monkeypatch.setattr(suite, "parse_pimsim_nn_metrics", lambda raw: parsed)
|
||||
assert suite.performance_values({}) == {"latency": None, "throughput": None, "power": 3, "energy": None}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value,expected", [
|
||||
(1e-12, "1e-12"), (0.00499999, "0.00499999"), (0.005, "0.005"),
|
||||
(0.00500001, "0.00500001"), (0.0063, "0.0063"), (0.014, "0.014"),
|
||||
(0.015, "0.015"), (1.23456789, "1.23457"), (123.456789, "123.457"),
|
||||
(123456789, "1.23457e+08"), (1e20, "1e+20"),
|
||||
])
|
||||
def test_measurements_use_six_significant_digits(value, expected):
|
||||
assert suite.format_value(value) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [None, 0, -1, float("nan"), float("inf"), -float("inf")])
|
||||
def test_format_value_preserves_missing_measurements(value):
|
||||
assert suite.format_value(value) == "NA"
|
||||
|
||||
|
||||
def test_zero_outputs_preserve_only_measured_average_power():
|
||||
metrics = suite.performance_values({"output_count": 0, "average_latency_ms": 10,
|
||||
"throughput": 100, "average_energy_pj": 30,
|
||||
"average_power_mw": 20})
|
||||
assert metrics == {"latency": None, "throughput": None, "energy": None, "power": 20}
|
||||
|
||||
|
||||
def test_functional_pass_survives_unavailable_performance(tmp_path):
|
||||
report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json"
|
||||
report.parent.mkdir(parents=True)
|
||||
report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True},
|
||||
"raptor_performance": {"error": "timeout", "throughput": 0}}))
|
||||
output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),))
|
||||
with output.open(newline="") as stream:
|
||||
row = next(csv.DictReader(stream))
|
||||
assert row["raptor_functional_validation"] == "PASS"
|
||||
assert all(row[field] == "NA" for field, _ in ablation.PERCENTAGE_FIELDS)
|
||||
|
||||
|
||||
def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path):
|
||||
variants = ablation.selected_variants(["no-plan"])
|
||||
for variant in variants:
|
||||
path = ablation.variant_output_dir(tmp_path, variant) / "results_comparison.csv"
|
||||
path.parent.mkdir(parents=True)
|
||||
row = {"arch": "arch-a", "model": "googlenet", "mode": "throughput",
|
||||
"raptor_pipeline": "4", "raptor_functional_validation": "PASS"}
|
||||
row.update({name: "100" if variant.name == "baseline" else "0.00"
|
||||
for name, _ in ablation.PERCENTAGE_FIELDS})
|
||||
with path.open("w", newline="") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=list(row))
|
||||
writer.writeheader()
|
||||
writer.writerow(row)
|
||||
destination = tmp_path / "canonical/results_ablation.csv"
|
||||
output, failures = ablation.aggregate_results(tmp_path, variants, output=destination)
|
||||
assert not failures and output == destination
|
||||
with output.open(newline="") as stream:
|
||||
rows = list(csv.DictReader(stream))
|
||||
assert rows[1]["functional_validation"] == "PASS"
|
||||
assert all(rows[1][metric] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS)
|
||||
@@ -0,0 +1,157 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import importlib.util
|
||||
import math
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
TOOLS = Path(__file__).resolve().parents[1] / "tools/pim"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def plotters(monkeypatch):
|
||||
modules = []
|
||||
for name, directory in (("comparison_plots", "pimcomp/compare"), ("ablation_plots", "ablation")):
|
||||
monkeypatch.syspath_prepend(str(TOOLS / directory))
|
||||
spec = importlib.util.spec_from_file_location(name, TOOLS / directory / "generate_plots.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
monkeypatch.setitem(sys.modules, name, module)
|
||||
spec.loader.exec_module(module)
|
||||
modules.append(module)
|
||||
return modules
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def comparison_csv():
|
||||
with (TOOLS / "pimcomp/compare/results_comparison.csv").open(newline="") as stream:
|
||||
reader = csv.DictReader(stream)
|
||||
return reader.fieldnames, list(reader)
|
||||
|
||||
|
||||
def write_csv(path, fields, rows):
|
||||
with path.open("w", newline="") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=fields)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
return path
|
||||
|
||||
|
||||
@pytest.mark.parametrize("custom", [False, True])
|
||||
def test_comparison_plot_cli_input_paths(plotters, tmp_path, monkeypatch, custom):
|
||||
comparison, _ = plotters
|
||||
expected = tmp_path / "custom.csv" if custom else TOOLS / "pimcomp/compare/results_comparison.csv"
|
||||
seen = []
|
||||
|
||||
class InputCaptured(Exception):
|
||||
pass
|
||||
|
||||
def capture(path):
|
||||
seen.append(path)
|
||||
raise InputCaptured
|
||||
|
||||
monkeypatch.setattr(comparison, "load_results", capture)
|
||||
monkeypatch.setattr(comparison, "configure_style", lambda: None)
|
||||
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom else [])])
|
||||
with pytest.raises(InputCaptured):
|
||||
comparison.main()
|
||||
assert seen == [expected]
|
||||
|
||||
|
||||
def test_exact_comparison_configuration_matrix(plotters, comparison_csv, tmp_path):
|
||||
comparison, _ = plotters
|
||||
fields, rows = comparison_csv
|
||||
results = comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
|
||||
keys = {(row.arch, row.model, row.mode, row.pipeline) for row in results}
|
||||
assert keys == {(arch, model, mode, stage) for arch in comparison.ARCHITECTURES
|
||||
for model in comparison.MODELS for mode, stage in comparison.CONFIGURATIONS}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("damage,message", [("missing", "Missing"), ("duplicate", "Duplicate"),
|
||||
("arch", "Unexpected"), ("model", "Unexpected"),
|
||||
("pipeline", "Unexpected"), ("mode", "Unexpected")])
|
||||
def test_comparison_rejects_bad_matrix_keys(plotters, comparison_csv, tmp_path, damage, message):
|
||||
comparison, _ = plotters
|
||||
fields, rows = comparison_csv
|
||||
if damage == "missing":
|
||||
rows.pop()
|
||||
elif damage == "duplicate":
|
||||
rows[-1] = dict(rows[-2]) # Still 40 rows and 20 per architecture.
|
||||
else:
|
||||
field = "raptor_pipeline" if damage == "pipeline" else damage
|
||||
rows[-1][field] = {"arch": "arch-c", "model": "other", "pipeline": "3", "mode": "other"}[damage]
|
||||
with pytest.raises(ValueError, match=message):
|
||||
comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
|
||||
|
||||
|
||||
def test_matrix_validation_survives_optimized_python(comparison_csv, tmp_path):
|
||||
fields, rows = comparison_csv
|
||||
path = write_csv(tmp_path / "duplicate.csv", fields, [*rows[:-1], rows[-2]])
|
||||
program = """import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, sys.argv[1])
|
||||
import generate_plots
|
||||
try:
|
||||
generate_plots.load_results(Path(sys.argv[2]))
|
||||
except ValueError as exc:
|
||||
if 'Duplicate configuration' not in str(exc): raise
|
||||
else:
|
||||
raise RuntimeError('Invalid matrix accepted with Python -O')
|
||||
"""
|
||||
subprocess.run([sys.executable, "-O", "-c", program, str(TOOLS / "pimcomp/compare"), str(path)], check=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field,value", [("pimcomp_latency_ms", "999"),
|
||||
("pimcomp_throughput_samples_s", "999"), ("pimcomp_power_mw", "999"),
|
||||
("pimcomp_energy_pj", "999"), ("pimcomp_functional_validation", "FAIL")])
|
||||
def test_repeated_pimcomp_batch_results_must_agree(plotters, comparison_csv, tmp_path, field, value):
|
||||
comparison, _ = plotters
|
||||
fields, rows = comparison_csv
|
||||
row = next(row for row in rows if row["arch"] == "arch-a" and row["model"] == "vgg8"
|
||||
and row["raptor_pipeline"] == "4")
|
||||
row[field] = value
|
||||
with pytest.raises(ValueError, match="inconsistent shared Pimcomp batch results"):
|
||||
comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
|
||||
|
||||
|
||||
def test_legacy_invalid_ablation_pairs_are_unavailable_without_changing_pass(plotters):
|
||||
_, ablation = plotters
|
||||
with pytest.warns(UserWarning, match="2 invalid reciprocal"):
|
||||
rows = ablation.load_results(TOOLS / "ablation/results_ablation.csv")
|
||||
invalid = [row for row in rows if row.model == "googlenet" and row.variant == "no-plan"
|
||||
and row.mode == "throughput"]
|
||||
assert len(invalid) == 2
|
||||
assert all(row.validation == "PASS" and math.isnan(row.metrics["latency"])
|
||||
and math.isnan(row.metrics["throughput"]) for row in invalid)
|
||||
|
||||
|
||||
def test_ablation_missing_values_and_legitimate_extremes(plotters, tmp_path):
|
||||
_, ablation = plotters
|
||||
row = {"variant": "no-plan", "arch": "arch-a", "model": "vgg8", "mode": "throughput",
|
||||
"raptor_pipeline": "4", "functional_validation": "FAIL", "latency": "-100.00%",
|
||||
"throughput": "+100000000.00%", "power": "NA", "energy": "-100.00%"}
|
||||
result = ablation.load_results(write_csv(tmp_path / "extreme.csv", list(row), [row]))[0]
|
||||
assert result.validation == "FAIL"
|
||||
assert result.metrics["latency"] == result.metrics["energy"] == -100
|
||||
assert result.metrics["throughput"] == 100000000
|
||||
assert math.isnan(result.metrics["power"])
|
||||
|
||||
|
||||
def test_both_plotters_export_png_and_svg(plotters, tmp_path):
|
||||
comparison, ablation = plotters
|
||||
comparison.configure_style()
|
||||
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
|
||||
selected = [row for row in rows if row.model == "vgg8" and row.mode == "latency"]
|
||||
comparison.draw_absolute(selected, "latency", "Latency (ms)", "comparison-smoke", tmp_path)
|
||||
row = ablation.Result("baseline", "arch-a", "vgg8", "latency", 1, "PASS",
|
||||
{"latency": math.nan, "power": 0, "energy": 0, "throughput": math.nan})
|
||||
ablation.draw_metric([row], "latency", "Latency change (%)", tmp_path)
|
||||
for name in ("comparison-smoke", "latency_pipeline1_latency"):
|
||||
assert (tmp_path / f"png/{name}.png").read_bytes().startswith(b"\x89PNG\r\n\x1a\n")
|
||||
svg = (tmp_path / f"svg/{name}.svg").read_text()
|
||||
assert "<svg" in svg
|
||||
assert "N/A" in (tmp_path / "svg/latency_pipeline1_latency.svg").read_text()
|
||||
@@ -0,0 +1 @@
|
||||
plots/
|
||||
@@ -1,78 +1,73 @@
|
||||
# Raptor compiler ablation
|
||||
|
||||
`run_ablation.py` performs the complete synchronization/Spatial-planning
|
||||
ablation study on the Pimcomp model suite. By default it runs `vgg8`,
|
||||
`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
|
||||
and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
|
||||
explicitly.
|
||||
Latency is pipeline 1; the wrapper invokes the suite runner separately for
|
||||
latency and throughput pipeline 4.
|
||||
Run the synchronization/Spatial-planning ablation from the repository root:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
|
||||
--jobs 4
|
||||
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
|
||||
```
|
||||
|
||||
## Variants
|
||||
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
|
||||
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
|
||||
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
|
||||
|
||||
| Variant | Raptor options |
|
||||
|---|---|
|
||||
| `baseline` | None. |
|
||||
| `baseline` | None |
|
||||
| `no-sync` | `--pim-disable-synchronization` |
|
||||
| `no-plan` | `--pim-disable-spatial-planning` |
|
||||
| `no-sync-no-plan` | Both options. |
|
||||
| `no-sync-no-plan` | Both options |
|
||||
|
||||
Every variant runs Raptor only. Pimcomp is not compiled, validated, or
|
||||
simulated; its validation column is `SKIPPED`, not `FAIL`. Reference inputs and
|
||||
outputs are generated once under the shared common-artifact root and reused by
|
||||
every variant. Ctrl+C terminates the active variant and all of its worker jobs.
|
||||
## Options and outputs
|
||||
|
||||
Removing synchronization deliberately removes host-buffer lifetime and pipeline
|
||||
restart guarantees. The `no-sync` variants can therefore fail numerical
|
||||
validation, especially across repeated throughput inputs. Those failures remain
|
||||
`FAIL` and cause a nonzero exit; a completed ablation is not proof that every
|
||||
variant is correct. Baseline and `no-plan` must still pass numerical validation.
|
||||
The combined CSV retains `functional_validation` for every row. Performance
|
||||
percentages from failing variants describe incorrect programs, not valid
|
||||
speedups. Throughput deadline averages likewise do not establish a converged
|
||||
steady-state period.
|
||||
- `--variant NAME`: repeat to select variants; baseline is always included.
|
||||
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
|
||||
- `--dry-run`: print commands without modifying files.
|
||||
|
||||
The percentage baseline is `baseline`, so its available values are `+0.00%`.
|
||||
Every other
|
||||
variant reports the signed percentage difference of its Raptor metrics from
|
||||
that reference for the same model, architecture, mode, and pipeline. Positive
|
||||
values mean the metric is higher; negative values mean it is lower.
|
||||
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
|
||||
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
|
||||
replaces the default latency/pipeline-4 case set. `--only` is rejected because
|
||||
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
|
||||
|
||||
## Options
|
||||
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
|
||||
reference inputs, outputs, and runners are shared under `artifacts/common/`.
|
||||
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
|
||||
Normal comparison results are preserved.
|
||||
|
||||
| Option | Description and default |
|
||||
|---|---|
|
||||
| `-h`, `--help` | Show help and exit. |
|
||||
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
|
||||
| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
|
||||
| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
|
||||
| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
|
||||
By default, the combined [results_ablation.csv](results_ablation.csv) is written
|
||||
beside this script and printed after the run. An explicit `--out-dir PATH`
|
||||
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
|
||||
Default model artifacts and per-variant summaries remain in the locations above.
|
||||
The aggregate records case identifiers,
|
||||
`functional_validation`, and signed percentage changes in latency, throughput,
|
||||
power, and energy against the matching baseline case. Positive means higher;
|
||||
baseline values are zero.
|
||||
|
||||
All options of
|
||||
[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
|
||||
`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
|
||||
`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
|
||||
`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
|
||||
the suite runner's comparison mode and is rejected by this wrapper, which
|
||||
always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
|
||||
forwarded, the wrapper uses its default latency and throughput/pipeline-4
|
||||
case set. Supplying either option overrides that default and is passed through
|
||||
as one suite-runner invocation per variant.
|
||||
Disabling synchronization removes host-buffer lifetime and pipeline restart
|
||||
guarantees, so those variants may fail numerical validation and exit nonzero.
|
||||
Baseline and `no-plan` must pass. Percentages from failing variants are not
|
||||
valid speedups; deadline averages do not prove steady-state convergence.
|
||||
|
||||
The feature-full `baseline` variant is rerun with the same selected cases and
|
||||
forwarded options as the disabled variants. All four variants are stored below
|
||||
each model's `artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared
|
||||
reference artifacts remain under each model's `artifacts/common` directory.
|
||||
Normal comparison artifacts and summaries are preserved. Compact per-variant
|
||||
comparison summaries remain under
|
||||
`<out-dir>/ablation/<variant>/results_comparison.csv` after aggregation.
|
||||
The combined table is written to `<out-dir>/results_ablation.csv`; it contains the
|
||||
variant, case identifiers, actual `functional_validation` status, and signed
|
||||
percentage `latency`, `throughput`, `power`, and `energy` columns. These are
|
||||
Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor
|
||||
only.
|
||||
## Plots
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/ablation/generate_plots.py
|
||||
# For a custom experiment root:
|
||||
.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
|
||||
--input /path/to/experiment/results_ablation.csv
|
||||
```
|
||||
|
||||
Requires Matplotlib from the root `requirements.txt`. By default, reads the
|
||||
canonical CSV beside this script. Plots go to
|
||||
`plots/png/` and `plots/svg/` beside the selected CSV; override with
|
||||
`--output-dir PATH`.
|
||||
Single-variant comparison CSVs are not accepted.
|
||||
|
||||
The checked-in CSV is an intentional result dataset; plotting uses its current
|
||||
contents without running benchmarks. Invalid reciprocal percentage pairs render
|
||||
as `N/A`; individual legitimate extreme changes are preserved. See
|
||||
[performance availability](../../../../docs/evaluation/methodology.md#functional-validity-and-performance-availability).
|
||||
|
||||
Plots separate modes and pipelines. Throughput-mode latency means output
|
||||
interval. Failed validation is labeled `INVALID`, other non-PASS statuses keep
|
||||
their names, and missing measurements show `N/A`. See the
|
||||
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Plot percentage changes from an experiment's combined results_ablation.csv."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import math
|
||||
import sys
|
||||
import warnings
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from plot_style import COLORS, configure_style, save_figure, value_label
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Patch
|
||||
from matplotlib.ticker import PercentFormatter
|
||||
|
||||
from run_ablation import ABLATION_DEFAULT_MODELS, DEFAULT_RESULTS_PATH, RESULT_FIELDS, VARIANTS
|
||||
|
||||
|
||||
VARIANT_COLORS = {variant.name: color for variant, color in zip(VARIANTS, COLORS)}
|
||||
STATUSES = ("PASS", "FAIL", "SKIPPED", "NA")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Result:
|
||||
variant: str
|
||||
arch: str
|
||||
model: str
|
||||
mode: str
|
||||
pipeline: int
|
||||
validation: str
|
||||
metrics: dict[str, float]
|
||||
|
||||
|
||||
def percentage(value: str) -> float:
|
||||
if value == "NA":
|
||||
return math.nan
|
||||
if not value.endswith("%"):
|
||||
raise ValueError(f"Expected a percentage or NA, got {value!r}")
|
||||
parsed = float(value[:-1])
|
||||
if not math.isfinite(parsed):
|
||||
raise ValueError(f"Non-finite percentage: {value!r}")
|
||||
return parsed
|
||||
|
||||
|
||||
def load_results(path: Path) -> list[Result]:
|
||||
rows = []
|
||||
seen = set()
|
||||
invalid_pairs = []
|
||||
with path.open(newline="", encoding="utf-8-sig") as stream:
|
||||
reader = csv.DictReader(stream)
|
||||
if not {"variant", *RESULT_FIELDS} <= set(reader.fieldnames or []):
|
||||
raise ValueError("Expected the combined ablation CSV with variant and percentage columns; "
|
||||
"a per-variant results_comparison.csv cannot be used here.")
|
||||
for row in reader:
|
||||
variant, mode, status = row["variant"], row["mode"], row["functional_validation"]
|
||||
if variant not in VARIANT_COLORS or mode not in ("latency", "throughput") or status not in STATUSES:
|
||||
raise ValueError(f"Unknown ablation variant, mode, or validation status: {variant}, {mode}, {status}")
|
||||
pipeline = int(row["raptor_pipeline"])
|
||||
key = (variant, row["arch"], row["model"], mode, pipeline)
|
||||
if key in seen:
|
||||
raise ValueError(f"Duplicate ablation configuration: {key}")
|
||||
seen.add(key)
|
||||
metrics = {metric: percentage(row[metric])
|
||||
for metric in ("latency", "throughput", "power", "energy")}
|
||||
# Each may round to -100% alone; reciprocal metrics cannot both do so.
|
||||
if mode == "throughput" and metrics["latency"] == metrics["throughput"] == -100:
|
||||
metrics["latency"] = metrics["throughput"] = math.nan
|
||||
invalid_pairs.append(key)
|
||||
rows.append(Result(variant, row["arch"], row["model"], mode, pipeline, status, metrics))
|
||||
if not rows:
|
||||
raise ValueError("The ablation CSV contains no results")
|
||||
if invalid_pairs:
|
||||
warnings.warn(f"{len(invalid_pairs)} invalid reciprocal latency/throughput percentage pairs; "
|
||||
f"displaying N/A. Examples: {invalid_pairs[:3]}", stacklevel=2)
|
||||
return rows
|
||||
|
||||
|
||||
def draw_metric(results: list[Result], metric: str, ylabel: str, output_dir: Path) -> None:
|
||||
architectures = sorted({row.arch for row in results})
|
||||
present_models = {row.model for row in results}
|
||||
models = [model for model in ABLATION_DEFAULT_MODELS if model in present_models]
|
||||
models += sorted(present_models - set(models))
|
||||
variants = [name for name in VARIANT_COLORS if any(row.variant == name for row in results)]
|
||||
lookup = {(row.arch, row.model, row.variant): row for row in results}
|
||||
values = [row.metrics[metric] for row in results if math.isfinite(row.metrics[metric])]
|
||||
lower, upper = min(0, min(values, default=0)), max(0, max(values, default=0))
|
||||
padding = max(upper - lower, 1) * 0.35
|
||||
fig, axes = plt.subplots(1, len(architectures), sharey=True, squeeze=False,
|
||||
figsize=(7 * len(architectures), 5.2), layout="constrained")
|
||||
handles = [Patch(facecolor=VARIANT_COLORS[name], label=name) for name in variants]
|
||||
if len(values) < len(architectures) * len(models) * len(variants):
|
||||
handles.append(Patch(facecolor="white", edgecolor="#B2BAC5", label="N/A: missing measurement"))
|
||||
fig.legend(handles=handles, loc="outside upper center", ncol=4, frameon=False, fontsize=9.5)
|
||||
width = 0.8 / len(variants)
|
||||
mode, pipeline = results[0].mode, results[0].pipeline
|
||||
for ax, arch in zip(axes[0], architectures):
|
||||
ax.set_axisbelow(True)
|
||||
ax.set_ylim(lower - padding if lower < 0 else 0, upper + padding)
|
||||
ax.set_xlim(-0.5, len(models) - 0.5)
|
||||
ax.set_xticks(range(len(models)), [model.upper() for model in models], fontsize=9.5)
|
||||
ax.set_xlabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
|
||||
f"{mode.capitalize()}, {pipeline} stage" + ("s" if pipeline != 1 else ""),
|
||||
fontsize=11, weight="bold", labelpad=8)
|
||||
ax.set_ylabel(ylabel, fontsize=10.5)
|
||||
ax.yaxis.set_major_formatter(PercentFormatter(xmax=100, decimals=0))
|
||||
ax.tick_params(axis="both", length=0, pad=6, labelleft=True)
|
||||
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
|
||||
ax.axhline(0, color="#536478", linewidth=1.2, linestyle="--")
|
||||
for index, model in enumerate(models):
|
||||
if index % 2 == 0:
|
||||
ax.axvspan(index - 0.5, index + 0.5, facecolor="#F7F9FC", zorder=0)
|
||||
for offset, variant in enumerate(variants):
|
||||
x = index + (offset - (len(variants) - 1) / 2) * width
|
||||
row = lookup.get((arch, model, variant))
|
||||
value = row.metrics[metric] if row else math.nan
|
||||
if not math.isfinite(value):
|
||||
ax.text(x, padding * 0.05, "N/A", rotation=90, ha="center", va="bottom",
|
||||
fontsize=8, color="#85909F")
|
||||
continue
|
||||
ax.bar(x, value, width=width * 0.9, color=VARIANT_COLORS[variant],
|
||||
edgecolor="none", linewidth=0, zorder=2)
|
||||
direction = -1 if value < 0 else 1
|
||||
value_label(ax, x, value + direction * padding * 0.05, f"{value:+,.2f}%",
|
||||
row.validation, direction)
|
||||
save_figure(fig, f"{mode}_pipeline{pipeline}_{metric}", output_dir)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--input", type=Path, default=DEFAULT_RESULTS_PATH)
|
||||
parser.add_argument("--output-dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.output_dir is None:
|
||||
args.output_dir = args.input.parent / "plots"
|
||||
configure_style()
|
||||
results = load_results(args.input)
|
||||
print(f"Results: {len(results)} configurations, "
|
||||
f"{sum(row.validation != 'PASS' for row in results)} without PASS validation")
|
||||
for mode, pipeline in sorted({(row.mode, row.pipeline) for row in results}):
|
||||
rows = [row for row in results if row.mode == mode and row.pipeline == pipeline]
|
||||
for metric, label in (
|
||||
("latency", "Single-inference latency" if mode == "latency" else "Output interval"),
|
||||
("throughput", "Throughput"), ("power", "Average power"), ("energy", "Energy per inference"),
|
||||
):
|
||||
if metric == "throughput" and mode == "latency":
|
||||
continue
|
||||
draw_metric(rows, metric, f"{label} change from baseline (%)", args.output_dir)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,65 @@
|
||||
variant,arch,model,mode,raptor_pipeline,functional_validation,latency,throughput,power,energy
|
||||
baseline,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-a,googlenet,latency,1,PASS,+11513.29%,NA,-69.42%,+3452.69%
|
||||
no-sync-no-plan,arch-a,googlenet,latency,1,PASS,+11512.99%,NA,-69.42%,+3452.59%
|
||||
baseline,arch-a,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,googlenet,throughput,4,FAIL,+26.17%,-20.74%,+9.97%,+18.50%
|
||||
no-plan,arch-a,googlenet,throughput,4,PASS,-100.00%,-100.00%,-62.73%,-100.00%
|
||||
no-sync-no-plan,arch-a,googlenet,throughput,4,FAIL,+4094.63%,-97.62%,-59.86%,+1311.01%
|
||||
baseline,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
|
||||
no-sync-no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
|
||||
baseline,arch-a,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet18,throughput,4,FAIL,+110.65%,-52.55%,+43.25%,+177.48%
|
||||
no-plan,arch-a,resnet18,throughput,4,PASS,+1598.93%,-94.12%,-51.04%,+664.87%
|
||||
no-sync-no-plan,arch-a,resnet18,throughput,4,FAIL,+1174.21%,-92.16%,-45.06%,+543.74%
|
||||
baseline,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
|
||||
no-sync-no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
|
||||
baseline,arch-a,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet34,throughput,4,FAIL,+33.65%,-25.21%,+31.76%,+15.56%
|
||||
no-plan,arch-a,resnet34,throughput,4,PASS,+1726.82%,-94.52%,-54.50%,+455.98%
|
||||
no-sync-no-plan,arch-a,resnet34,throughput,4,FAIL,+813.41%,-89.04%,-49.77%,+206.84%
|
||||
baseline,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
|
||||
no-sync-no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
|
||||
baseline,arch-a,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,vgg8,throughput,4,PASS,+2.33%,-1.72%,+18.49%,-0.98%
|
||||
no-plan,arch-a,vgg8,throughput,4,PASS,+1251.16%,-92.58%,-56.68%,+577.04%
|
||||
no-sync-no-plan,arch-a,vgg8,throughput,4,PASS,+1260.47%,-92.63%,-53.36%,+467.83%
|
||||
baseline,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-b,googlenet,latency,1,PASS,+9849.64%,NA,-70.51%,+2833.89%
|
||||
no-sync-no-plan,arch-b,googlenet,latency,1,PASS,+9849.55%,NA,-70.51%,+2833.89%
|
||||
baseline,arch-b,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,googlenet,throughput,4,FAIL,+11.45%,-10.29%,+21.82%,+19.27%
|
||||
no-plan,arch-b,googlenet,throughput,4,PASS,-100.00%,-100.00%,-63.39%,-100.00%
|
||||
no-sync-no-plan,arch-b,googlenet,throughput,4,FAIL,+3348.28%,-97.10%,-61.58%,+1026.09%
|
||||
baseline,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-b,resnet18,latency,1,PASS,+2935.63%,NA,-63.12%,+1019.81%
|
||||
no-sync-no-plan,arch-b,resnet18,latency,1,PASS,+2936.09%,NA,-63.12%,+1019.97%
|
||||
baseline,arch-b,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet18,throughput,4,FAIL,+8.03%,-7.44%,+47.59%,+40.61%
|
||||
no-plan,arch-b,resnet18,throughput,4,PASS,+1511.85%,-93.80%,-52.93%,+560.80%
|
||||
no-sync-no-plan,arch-b,resnet18,throughput,4,FAIL,+867.12%,-89.67%,-45.41%,+359.83%
|
||||
baseline,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-b,resnet34,latency,1,PASS,+2551.16%,NA,-62.51%,+893.70%
|
||||
no-sync-no-plan,arch-b,resnet34,latency,1,PASS,+2550.91%,NA,-62.51%,+893.62%
|
||||
baseline,arch-b,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet34,throughput,4,FAIL,+28.25%,-22.13%,+49.76%,+57.30%
|
||||
no-plan,arch-b,resnet34,throughput,4,PASS,+2427.17%,-96.05%,-49.58%,+938.62%
|
||||
no-sync-no-plan,arch-b,resnet34,throughput,4,FAIL,+531.79%,-84.19%,-42.32%,+197.08%
|
||||
baseline,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
|
||||
no-sync-no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
|
||||
baseline,arch-b,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,vgg8,throughput,4,PASS,+23.91%,-19.82%,+25.67%,+23.93%
|
||||
no-plan,arch-b,vgg8,throughput,4,PASS,+1158.70%,-92.07%,-58.95%,+408.33%
|
||||
no-sync-no-plan,arch-b,vgg8,throughput,4,PASS,+1172.83%,-92.17%,-55.83%,+327.00%
|
||||
|
@@ -45,6 +45,7 @@ VARIANTS = (
|
||||
VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS}
|
||||
COMPARISON_RESULTS_FILENAME = "results_comparison.csv"
|
||||
ABLATION_RESULTS_FILENAME = "results_ablation.csv"
|
||||
DEFAULT_RESULTS_PATH = Path(__file__).with_name(ABLATION_RESULTS_FILENAME)
|
||||
CASE_FIELDS = ("arch", "model", "mode", "raptor_pipeline")
|
||||
PERCENTAGE_FIELDS = (
|
||||
("raptor_latency_ms", "latency"),
|
||||
@@ -68,8 +69,7 @@ def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[
|
||||
parser.add_argument(
|
||||
"--out-dir",
|
||||
type=Path,
|
||||
default=DEFAULT_OUT_DIR,
|
||||
help="Artifact root (default: validation/networks/pimcomp_models).",
|
||||
help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_ablation.csv beside this script.",
|
||||
)
|
||||
add_models_argument(parser, ABLATION_DEFAULT_MODELS)
|
||||
parser.add_argument(
|
||||
@@ -160,9 +160,10 @@ def percentage_difference(value: str | None, reference: str | None) -> str:
|
||||
baseline = float(reference) if reference is not None else math.nan
|
||||
except ValueError:
|
||||
return "NA"
|
||||
if not math.isfinite(current) or not math.isfinite(baseline) or baseline == 0:
|
||||
if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0:
|
||||
return "NA"
|
||||
return f"{(current - baseline) / baseline * 100:+.2f}%"
|
||||
change = (current / baseline - 1) * 100
|
||||
return f"{change:+.2f}%" if math.isfinite(change) else "NA"
|
||||
|
||||
|
||||
def aggregate_results(
|
||||
@@ -170,6 +171,7 @@ def aggregate_results(
|
||||
variants: list[Variant],
|
||||
selected_cases: frozenset[tuple[str, str]] | None = None,
|
||||
selected_models: frozenset[str] | None = None,
|
||||
output: Path | None = None,
|
||||
) -> tuple[Path | None, list[str]]:
|
||||
fields: list[str] | None = None
|
||||
rows_by_variant: dict[str, list[dict[str, str]]] = {}
|
||||
@@ -233,7 +235,8 @@ def aggregate_results(
|
||||
int(row["raptor_pipeline"]),
|
||||
)
|
||||
)
|
||||
output = out_dir / ABLATION_RESULTS_FILENAME
|
||||
output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with output.open("w", newline="", encoding="utf-8") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n")
|
||||
writer.writeheader()
|
||||
@@ -247,7 +250,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
print(f"Missing Pimcomp runner: {RUNNER}", file=sys.stderr)
|
||||
return 1
|
||||
variants = selected_variants(args.variants)
|
||||
out_dir = args.out_dir.resolve()
|
||||
out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR
|
||||
case_sets = runner_argument_sets(forwarded)
|
||||
if args.dry_run:
|
||||
for variant in variants:
|
||||
@@ -299,14 +302,17 @@ def main(argv: list[str] | None = None) -> int:
|
||||
variants,
|
||||
selected_cases,
|
||||
frozenset(args.models),
|
||||
output=DEFAULT_RESULTS_PATH if args.out_dir is None else None,
|
||||
)
|
||||
failed.extend(aggregation_failures)
|
||||
if output is not None:
|
||||
print(f"Ablation results: {output}")
|
||||
print(f"[{output.name}]")
|
||||
print(output.read_text(encoding="utf-8"), end="", flush=True)
|
||||
if failed:
|
||||
print("Failed: " + "; ".join(failed), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
if output is not None:
|
||||
print(f"Ablation results: {output}", flush=True)
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
plots/
|
||||
@@ -1,62 +1,108 @@
|
||||
# Pimcomp model comparison
|
||||
|
||||
These scripts compare Raptor with Pimcomp on the models in
|
||||
`validation/networks/pimcomp_models`.
|
||||
|
||||
`run_pimcomp_models.py` runs the supported model suite across the configured
|
||||
architectures and simulation modes. `compare_raptor_pimcomp_model.py` is the
|
||||
lower-level one-model comparison used by the suite runner.
|
||||
Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
|
||||
Run commands from the repository root with the repository virtual environment,
|
||||
built release Raptor, Pimcomp, and `pimsim-nn`.
|
||||
|
||||
## Suite runner
|
||||
|
||||
Run all five models, Arch-A and Arch-B, latency, and throughput:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
|
||||
```
|
||||
|
||||
### Options
|
||||
Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
|
||||
2/4/8, four workers, and a functional throughput batch of 64.
|
||||
Use `--help` for the complete option reference.
|
||||
|
||||
| Option | Description and default |
|
||||
| Option | Purpose |
|
||||
|---|---|
|
||||
| `-h`, `--help` | Show help and exit. |
|
||||
| `--out-dir PATH` | Suite root; artifacts are placed below `<out-dir>/<model>/artifacts/`. Default: `validation/networks/pimcomp_models`. |
|
||||
| `--common-dir PATH` | Shared root for per-model reference artifacts. Default: `<out-dir>/<model>/artifacts/common`. |
|
||||
| `--models MODEL [...]` | Models to run: `vgg8`, `resnet18`, `resnet34`, `googlenet`, `yolo11n`. Default: all five; pass a subset to select specific models. |
|
||||
| `--archs ARCH [...]` | Pimcomp hardware profiles to run. Default: `arch-a arch-b`. |
|
||||
| `--mode {latency,throughput} [...]` | Simulation modes. Default: both. |
|
||||
| `--only {raptor,pimcomp}` | Re-run only one compiler side and reuse the other side's existing report. Default: run both sides. |
|
||||
| `--raptor-only` | Run Raptor without compiling, validating, or simulating Pimcomp. Default: off. |
|
||||
| `--pipeline {1,2,4,8}` | Select one Raptor pipeline. Default: latency pipeline 1 and throughput pipelines 2, 4, and 8. |
|
||||
| `--pimsim-time-ms INT` | Throughput convergence deadline. Default: `1000`. |
|
||||
| `--batch-size INT` | Functional throughput batch size. Default: `128`. |
|
||||
| `--timeout-seconds FLOAT` | Per-stage timeout; `0` means unlimited. Default: `0`. |
|
||||
| `-j INT`, `--jobs INT` | Parallel comparison workers. Default: `4`. |
|
||||
| `--clean` | Remove generated comparison artifacts and summaries, then exit. Default: off. |
|
||||
| `--ablation-variant NAME` | Put generated artifacts below `<mode>[/pipelineN]/ablation/NAME` and write the summary below `ablation/NAME`. Default: none. |
|
||||
| `--dry-run` | Print commands without modifying files. Default: off. |
|
||||
| `--no-fast` | Disable fast throughput convergence. Default: off. |
|
||||
| `--raptor-extra-arg=ARG` | Extra Raptor compiler argument; repeat for multiple arguments. Default: none. |
|
||||
| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
|
||||
| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
|
||||
| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
|
||||
| `--common-dir PATH` | Override the shared reference-artifact root. |
|
||||
| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
|
||||
| `--raptor-only` | Run Raptor without invoking Pimcomp. |
|
||||
| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
|
||||
| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
|
||||
| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
|
||||
| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
|
||||
| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
|
||||
| `--dry-run` | Print commands without modifying files. |
|
||||
| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |
|
||||
|
||||
Arguments beginning with `--` must use the equals form when passed through:
|
||||
Use the equals form for forwarded flags, for example
|
||||
`--raptor-extra-arg=--pim-detect-communication-deadlock`.
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py \
|
||||
--models vgg8 \
|
||||
--raptor-extra-arg=--pim-disable-synchronization
|
||||
```
|
||||
With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
|
||||
beside this script. An explicit `--out-dir PATH` writes
|
||||
`PATH/results_comparison.csv`; selecting a subset updates only those rows.
|
||||
Default model artifacts remain under `validation/networks/pimcomp_models`.
|
||||
Artifacts use
|
||||
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
|
||||
under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
|
||||
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
|
||||
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
|
||||
written report, including failures without simulation artifacts. Missing or
|
||||
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
|
||||
compiler, simulation, or functional comparison fails.
|
||||
|
||||
The runner writes `results_comparison.csv` under the selected result root. It reuses
|
||||
shared model inputs, outputs, and reference runners, and reuses Pimcomp
|
||||
artifacts between pipelines in the same model/architecture/mode group. With
|
||||
`--raptor-only`, only Raptor results are generated and Pimcomp is not invoked.
|
||||
All generated files are kept below each model's `artifacts/` directory;
|
||||
`--clean` also removes any `.lock` files left by an interrupted reference
|
||||
generation.
|
||||
Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
|
||||
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
|
||||
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
|
||||
with a regeneration diagnostic. `--only raptor` requires compatible reports;
|
||||
`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
|
||||
results. Ordinary runs also rerun the group's first case, even if an old report
|
||||
exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
|
||||
Tool/build revisions and manually edited reference artifacts are not hashed;
|
||||
rerun the relevant compiler after those change.
|
||||
|
||||
## One-model comparator
|
||||
|
||||
`compare_raptor_pimcomp_model.py` compares one ONNX model. Its required
|
||||
arguments are `--model PATH` and `--out-dir PATH`; use `--help` for the full
|
||||
lower-level interface. The suite runner supplies the model, hardware profile,
|
||||
simulation mode, and reuse paths automatically.
|
||||
`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
|
||||
Use `--help` for hardware and simulation options. The suite runner supplies
|
||||
these automatically.
|
||||
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
|
||||
`validate.py` uses `1e-3`/`1e-5`.
|
||||
|
||||
## Comparison charts
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
|
||||
```
|
||||
|
||||
Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
|
||||
`results_comparison.csv` beside the script and writes `plots/png/` and
|
||||
`plots/svg/` there. Use `--input PATH` for a runner-generated CSV and
|
||||
`--output-dir PATH` to override the destination. The plotter requires the full
|
||||
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
|
||||
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
|
||||
configurations are rejected before plots are written.
|
||||
|
||||
| Plot family | Meaning |
|
||||
|---|---|
|
||||
| `validation_status` | Recorded functional status for each compiler and experiment. |
|
||||
| `latency_*` | Single-inference latency, latency mode only. |
|
||||
| `throughput_*` | Inferences/s, throughput mode only. |
|
||||
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
|
||||
| `power_*`, `energy_*` | Average power and energy per inference. |
|
||||
| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |
|
||||
|
||||
Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
|
||||
`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
|
||||
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
|
||||
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
|
||||
winning factor minus one, with parity at zero.
|
||||
|
||||
Failed validation is labeled `INVALID`; other non-PASS statuses keep their
|
||||
names, and missing measurements show `N/A`. Relative bars require both
|
||||
compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
|
||||
must agree before being collapsed. `FAIL` alone does not identify the cause.
|
||||
|
||||
Plots consume the CSV's current contents; they do not run benchmarks or establish
|
||||
publication provenance. The checked-in dataset is intentional. See the
|
||||
[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).
|
||||
|
||||
## Metric interpretation
|
||||
|
||||
See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
|
||||
for measurement windows, units, numerical validity, and result provenance.
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import fcntl
|
||||
import gzip
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import mmap
|
||||
@@ -1162,6 +1163,60 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None:
|
||||
return Path(value) if value else None
|
||||
|
||||
|
||||
def pimcomp_reuse_context(
|
||||
args: argparse.Namespace,
|
||||
model: Path,
|
||||
functional_model: Path,
|
||||
hardware: dict[str, int],
|
||||
) -> dict[str, Any]:
|
||||
files = {"model": model, "functional_model": functional_model,
|
||||
"config": args.pimcomp_config}
|
||||
digests = {}
|
||||
for path in set(files.values()):
|
||||
with path.open("rb") as stream:
|
||||
digests[path] = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
return {
|
||||
"version": 1,
|
||||
"files": {name: {"path": str(path), "sha256": digests[path]}
|
||||
for name, path in files.items()},
|
||||
"hardware": hardware,
|
||||
"settings": {name: getattr(args, name) for name in (
|
||||
"pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline",
|
||||
"pimcomp_replication", "pimcomp_model_name", "seed", "batch_size",
|
||||
"threshold", "rtol", "no_fast", "skip_pimsim_nn",
|
||||
)},
|
||||
}
|
||||
|
||||
|
||||
def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]:
|
||||
try:
|
||||
report = json.loads(path.read_text(encoding="utf-8"))
|
||||
if (not isinstance(report, dict) or context is None
|
||||
or report.get("pimcomp_reuse_context") != context
|
||||
or report.get("pimcomp_model_source") != "original_onnx"):
|
||||
raise ValueError("model, configuration, or experiment settings differ; legacy reports lack fingerprints")
|
||||
for name, kind in (("pimcomp_validation", dict), ("pimcomp_performance", dict),
|
||||
("pimcomp_instruction_summary", dict), ("paths", dict),
|
||||
("failures", list), ("steps", list)):
|
||||
if not isinstance(report.get(name), kind):
|
||||
raise ValueError(f"missing or invalid {name}")
|
||||
validation = CompareResult(**report["pimcomp_validation"])
|
||||
if (type(validation.passed) is not bool or not isinstance(validation.max_diffs, dict)
|
||||
or validation.status not in ("done", "failed", "skipped")):
|
||||
raise ValueError("invalid functional validation result")
|
||||
if not report["pimcomp_performance"]:
|
||||
raise ValueError("missing performance result or failure status")
|
||||
summary = report["pimcomp_instruction_summary"]
|
||||
if not {"active_cores", "total_instructions", "op_counts"} <= summary.keys():
|
||||
raise ValueError("incomplete instruction summary")
|
||||
restore_side_records(report, "pimcomp", [], [])
|
||||
for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"):
|
||||
path_from_report(report, name)
|
||||
return report
|
||||
except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc:
|
||||
raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc
|
||||
|
||||
|
||||
def restore_side_records(
|
||||
report: dict[str, Any],
|
||||
side: str,
|
||||
@@ -1579,6 +1634,11 @@ def main():
|
||||
if loaded_hardware is not None:
|
||||
hardware = loaded_hardware
|
||||
|
||||
reuse_context = None if args.raptor_only else try_stage(
|
||||
failures, "Identify Pimcomp experiment", pimcomp_reuse_context,
|
||||
args, model_path, functional_model_path, hardware,
|
||||
)
|
||||
|
||||
model_io = try_stage(
|
||||
failures,
|
||||
"Load model inputs",
|
||||
@@ -1597,12 +1657,7 @@ def main():
|
||||
run_pimcomp = not args.raptor_only and not reuse_pimcomp
|
||||
if reuse_pimcomp:
|
||||
reused_pimcomp_report_path = reused_pimcomp_report_path.resolve()
|
||||
if not reused_pimcomp_report_path.exists():
|
||||
raise ValueError(f"Missing Pimcomp report: {reused_pimcomp_report_path}")
|
||||
with open(reused_pimcomp_report_path, "r", encoding="utf-8") as f:
|
||||
reused_pimcomp = json.load(f)
|
||||
if reused_pimcomp.get("pimcomp_model_source") != "original_onnx":
|
||||
raise ValueError("Reused Pimcomp artifacts were not generated from the original ONNX model")
|
||||
reused_pimcomp = load_reused_pimcomp_report(reused_pimcomp_report_path, reuse_context)
|
||||
pimcomp_validation = CompareResult(**reused_pimcomp["pimcomp_validation"])
|
||||
pimcomp_perf = reused_pimcomp["pimcomp_performance"]
|
||||
pimcomp_instr = reused_pimcomp["pimcomp_instruction_summary"]
|
||||
@@ -2011,6 +2066,7 @@ def main():
|
||||
"pimcomp_pipeline": args.pimcomp_pipeline,
|
||||
"pimcomp_replication": args.pimcomp_replication,
|
||||
"pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx",
|
||||
"pimcomp_reuse_context": reuse_context,
|
||||
"pimcomp_config": str(args.pimcomp_config),
|
||||
"raptor_extra_args": args.raptor_extra_arg,
|
||||
"reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
|
||||
|
||||
+514
@@ -0,0 +1,514 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate publication-ready Raptor vs Pimcomp comparison charts.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
|
||||
|
||||
Dependencies: matplotlib (no pandas or seaborn required).
|
||||
By default, reads results_comparison.csv beside this script and writes plots/
|
||||
beside it, with outputs grouped in png/ and svg/ under the output directory.
|
||||
|
||||
YOLO models without Pimcomp measurements get separate Raptor-only charts.
|
||||
Comparisons use vertical bars for ratios, diverging advantages, and absolute
|
||||
values, with compiler colors and patterns for the Raptor pipeline stages.
|
||||
Throughput-mode latency is the output interval, a bottleneck-stage proxy,
|
||||
not end-to-end inference latency. Separate throughput plots use inferences/s.
|
||||
Energy is reported per inference; power is averaged over the simulation.
|
||||
Numerical results whose compiler did not pass functional validation have a red
|
||||
status following their value label; missing Pimcomp
|
||||
measurements appear as 'N/A'. The fixed compiler palette is shared by
|
||||
all metrics. Relative plots use Pimcomp / Raptor for latency, power and energy,
|
||||
and Raptor / Pimcomp for throughput, so >1 favors Raptor. Diverging plots put the winner's
|
||||
factor above (Raptor) or below (Pimcomp) parity, with 1x at the zero line.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from itertools import groupby
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Patch, Rectangle
|
||||
from matplotlib.lines import Line2D
|
||||
from matplotlib.ticker import FuncFormatter, MultipleLocator
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from plot_style import COLORS, HATCH_COLOR, TEXT_COLOR, configure_style, fit_value_labels, save_figure, value_label
|
||||
|
||||
|
||||
COMPILER_COLORS = {"raptor": COLORS[0], "pimcomp": COLORS[2]}
|
||||
STAGE_PATTERNS = {1: "", 2: "..", 4: "//", 8: "\\\\"}
|
||||
ARCHITECTURES = ("arch-a", "arch-b")
|
||||
MODELS = ("vgg8", "resnet18", "resnet34", "googlenet", "yolo11n")
|
||||
CONFIGURATIONS = (("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8))
|
||||
LINE_COLOR = "#A8B2C0"
|
||||
VALIDATION_STYLES = {
|
||||
"PASS": ("#168C8B", "white", ""),
|
||||
"FAIL": ("#D18D32", HATCH_COLOR, ""),
|
||||
"SKIPPED": ("#E7DFF0", TEXT_COLOR, ""),
|
||||
"NA": ("#E8EDF1", TEXT_COLOR, ""),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Result:
|
||||
arch: str
|
||||
model: str
|
||||
mode: str
|
||||
pipeline: int
|
||||
pimcomp_schedule: str
|
||||
raptor_validation: str
|
||||
pimcomp_validation: str
|
||||
latency_speedup: float
|
||||
power: float
|
||||
energy: float
|
||||
raptor_latency: float
|
||||
pimcomp_latency: float
|
||||
raptor_throughput: float
|
||||
pimcomp_throughput: float
|
||||
raptor_power: float
|
||||
pimcomp_power: float
|
||||
raptor_energy: float
|
||||
pimcomp_energy: float
|
||||
|
||||
@property
|
||||
def comparison_passed(self) -> bool:
|
||||
return self.raptor_validation == self.pimcomp_validation == "PASS"
|
||||
|
||||
@property
|
||||
def throughput_speedup(self) -> float:
|
||||
return quotient(self.raptor_throughput, self.pimcomp_throughput)
|
||||
|
||||
|
||||
def positive_float(text: str | None) -> float:
|
||||
try:
|
||||
value = float(text)
|
||||
except (TypeError, ValueError):
|
||||
return math.nan
|
||||
return value if math.isfinite(value) and value > 0 else math.nan
|
||||
|
||||
|
||||
def quotient(numerator: float, denominator: float) -> float:
|
||||
if not (math.isfinite(numerator) and math.isfinite(denominator)):
|
||||
return math.nan
|
||||
return numerator / denominator
|
||||
|
||||
|
||||
def validation_status(text: str) -> str:
|
||||
status = text.strip().upper()
|
||||
if status not in VALIDATION_STYLES:
|
||||
raise ValueError(f"Unknown functional-validation status: {text!r}")
|
||||
return status
|
||||
|
||||
|
||||
def load_results(path: Path) -> list[Result]:
|
||||
with path.open(newline="", encoding="utf-8-sig") as handle:
|
||||
source = list(csv.DictReader(handle))
|
||||
rows = []
|
||||
expected = {(arch, model, mode, stage)
|
||||
for arch in ARCHITECTURES for model in MODELS
|
||||
for mode, stage in CONFIGURATIONS}
|
||||
seen = {}
|
||||
for line, row in enumerate(source, 2):
|
||||
try:
|
||||
arch, model, mode = (row[field].strip().lower() for field in ("arch", "model", "mode"))
|
||||
stage = int(row["raptor_pipeline"])
|
||||
except (KeyError, AttributeError, TypeError, ValueError) as exc:
|
||||
raise ValueError(f"Invalid configuration key at CSV row {line}: {exc}") from exc
|
||||
key = arch, model, mode, stage
|
||||
if key not in expected:
|
||||
raise ValueError(f"Unexpected configuration at CSV row {line}: {key}")
|
||||
if key in seen:
|
||||
raise ValueError(f"Duplicate configuration {key} at CSV rows {seen[key]} and {line}")
|
||||
seen[key] = line
|
||||
schedule = row["pimcomp_pipeline"].strip().lower()
|
||||
if schedule != ("element" if mode == "latency" else "batch"):
|
||||
raise ValueError(f"{key}: incompatible Pimcomp schedule {schedule!r}")
|
||||
rows.append(Result(
|
||||
arch=arch,
|
||||
model=model,
|
||||
mode=mode,
|
||||
pipeline=stage,
|
||||
pimcomp_schedule=schedule,
|
||||
raptor_validation=validation_status(row["raptor_functional_validation"]),
|
||||
pimcomp_validation=validation_status(row["pimcomp_functional_validation"]),
|
||||
latency_speedup=quotient(positive_float(row["pimcomp_latency_ms"]),
|
||||
positive_float(row["raptor_latency_ms"])),
|
||||
power=quotient(positive_float(row["pimcomp_power_mw"]),
|
||||
positive_float(row["raptor_power_mw"])),
|
||||
energy=quotient(positive_float(row["pimcomp_energy_pj"]),
|
||||
positive_float(row["raptor_energy_pj"])),
|
||||
raptor_latency=positive_float(row["raptor_latency_ms"]),
|
||||
pimcomp_latency=positive_float(row["pimcomp_latency_ms"]),
|
||||
raptor_throughput=positive_float(row["raptor_throughput_samples_s"]),
|
||||
pimcomp_throughput=positive_float(row["pimcomp_throughput_samples_s"]),
|
||||
raptor_power=positive_float(row["raptor_power_mw"]),
|
||||
pimcomp_power=positive_float(row["pimcomp_power_mw"]),
|
||||
raptor_energy=positive_float(row["raptor_energy_pj"]),
|
||||
pimcomp_energy=positive_float(row["pimcomp_energy_pj"]),
|
||||
))
|
||||
missing = expected - seen.keys()
|
||||
if missing:
|
||||
raise ValueError(f"Missing {len(missing)} expected configurations. Examples: {sorted(missing)[:5]}")
|
||||
rows.sort(key=lambda r: (ARCHITECTURES.index(r.arch), MODELS.index(r.model), r.pipeline))
|
||||
absolute_columns(rows)
|
||||
return rows
|
||||
|
||||
|
||||
def advantage_compiler(ratio: float) -> str | None:
|
||||
if ratio > 1:
|
||||
return "raptor"
|
||||
if ratio < 1:
|
||||
return "pimcomp"
|
||||
return None
|
||||
|
||||
|
||||
def figure_legend(fig, results: list[Result], field: str,
|
||||
absolute=False, raptor_only=False):
|
||||
compilers = ("raptor",) if raptor_only else ("raptor", "pimcomp")
|
||||
if absolute:
|
||||
measurements = [(row, compiler, getattr(row, f"{compiler}_{field}"))
|
||||
for row, compiler in absolute_columns(results, raptor_only)]
|
||||
else:
|
||||
measurements = [(row, None, getattr(row, field)) for row in results]
|
||||
available = [(row, compiler, value) for row, compiler, value in measurements
|
||||
if math.isfinite(value)]
|
||||
stages = {row.pipeline for row, compiler, _ in available if compiler != "pimcomp"}
|
||||
handles = []
|
||||
visible_compilers = {compiler if absolute else advantage_compiler(value)
|
||||
for _, compiler, value in available} - {None}
|
||||
if len(visible_compilers) > 1:
|
||||
for compiler in compilers:
|
||||
handles.append(Patch(facecolor=COMPILER_COLORS[compiler],
|
||||
label=compiler.capitalize() + ("" if absolute else " advantage")))
|
||||
handles += [Patch(facecolor="white", edgecolor=TEXT_COLOR, hatch=STAGE_PATTERNS[depth],
|
||||
label=f"{depth} stage" + (" (latency)" if depth == 1 else "s"))
|
||||
for depth in sorted(stages) if stages != {1}]
|
||||
styles = []
|
||||
if len(available) < len(measurements):
|
||||
styles.append(Patch(facecolor="white", edgecolor="#B2BAC5",
|
||||
label="N/A: missing measurement"))
|
||||
handles += styles
|
||||
if handles:
|
||||
fig.legend(handles=handles, loc="outside upper center", ncol=(len(handles) + 1) // 2,
|
||||
frameon=False, fontsize=9.5, handlelength=1.7, columnspacing=1.5,
|
||||
handletextpad=0.6, labelspacing=0.4)
|
||||
|
||||
|
||||
def absolute_columns(results: list[Result], raptor_only: bool = False) -> list[tuple[Result, str]]:
|
||||
columns = []
|
||||
for (arch, model), group in groupby(results, key=lambda row: (row.arch, row.model)):
|
||||
rows = sorted(group, key=lambda row: row.pipeline)
|
||||
for row in rows:
|
||||
columns.append((row, "raptor"))
|
||||
if row.mode == "latency" and not raptor_only:
|
||||
columns.append((row, "pimcomp"))
|
||||
throughput = [row for row in rows if row.mode == "throughput"]
|
||||
if throughput and not raptor_only:
|
||||
signatures = {
|
||||
(row.pimcomp_schedule, row.pimcomp_validation,
|
||||
*(None if math.isnan(value) else value for value in (
|
||||
row.pimcomp_latency, row.pimcomp_throughput, row.pimcomp_power, row.pimcomp_energy)))
|
||||
for row in throughput
|
||||
}
|
||||
if len(signatures) != 1:
|
||||
raise ValueError(f"{arch}/{model}: inconsistent shared Pimcomp batch results across pipelines")
|
||||
columns.append((throughput[0], "pimcomp"))
|
||||
return columns
|
||||
|
||||
|
||||
def label_groups(ax, results: list[Result], positions: list[float] | None = None) -> None:
|
||||
if positions is None:
|
||||
positions = list(range(len(results)))
|
||||
centers, labels = [], []
|
||||
start = 0
|
||||
for index, (model, rows) in enumerate(groupby(results, key=lambda row: row.model)):
|
||||
end = start + len(list(rows))
|
||||
centers.append((positions[start] + positions[end - 1]) / 2)
|
||||
labels.append(model.upper())
|
||||
left = ax.get_xlim()[0] if start == 0 else (positions[start - 1] + positions[start]) / 2
|
||||
right = ax.get_xlim()[1] if end == len(results) else (positions[end - 1] + positions[end]) / 2
|
||||
if index % 2 == 0:
|
||||
ax.axvspan(left, right, facecolor="#F7F9FC", zorder=0)
|
||||
if index > 0:
|
||||
ax.axvline(left, color="#DDE3EB", linewidth=0.9)
|
||||
start = end
|
||||
ax.set_xticks(centers, labels, fontsize=9.5)
|
||||
|
||||
|
||||
def diverging_ratio(ratio: float) -> float:
|
||||
return ratio - 1 if ratio >= 1 else 1 - 1 / ratio
|
||||
|
||||
|
||||
def draw_validation(results: list[Result], output_dir: Path) -> None:
|
||||
columns = absolute_columns(results)
|
||||
configurations = list(dict.fromkeys((row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule,
|
||||
compiler) for row, compiler in columns))
|
||||
positions = []
|
||||
for index, (mode, _, _) in enumerate(configurations):
|
||||
gap = 0.4 if index and mode != configurations[index - 1][0] else 0
|
||||
positions.append(positions[-1] + 1 + gap if index else 0)
|
||||
configuration_x = dict(zip(configurations, positions))
|
||||
headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor"
|
||||
else "Pimcomp\nbatch" if mode == "throughput" else compiler.capitalize()
|
||||
for mode, stage, compiler in configurations]
|
||||
fig, axes = plt.subplots(1, 2, figsize=(14, 3.8), layout="constrained")
|
||||
for ax, arch in zip(axes, ARCHITECTURES):
|
||||
ax.set_xlim(positions[0] - 0.55, positions[-1] + 0.55)
|
||||
ax.set_ylim(len(MODELS) - 0.5, -0.5)
|
||||
centers, modes = [], []
|
||||
for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]):
|
||||
group_positions = [x for _, x in group]
|
||||
centers.append((group_positions[0] + group_positions[-1]) / 2)
|
||||
modes.append(mode.capitalize())
|
||||
ax.set_xticks(centers, modes, fontsize=10)
|
||||
ax.set_xticks(positions, headers, minor=True)
|
||||
ax.tick_params(axis="x", which="minor", top=True, labeltop=True,
|
||||
bottom=False, labelbottom=False, length=0, pad=6, labelsize=9)
|
||||
ax.set_yticks(range(len(MODELS)), [model.upper() for model in MODELS], fontsize=9)
|
||||
ax.tick_params(axis="both", length=0, pad=6)
|
||||
architecture = "ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B"
|
||||
ax.set_xlabel(architecture, fontsize=11, weight="bold", labelpad=8)
|
||||
for row, compiler in columns:
|
||||
if row.arch != arch:
|
||||
continue
|
||||
key = (row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule, compiler)
|
||||
x, y = configuration_x[key], MODELS.index(row.model)
|
||||
status = getattr(row, f"{compiler}_validation")
|
||||
color, text_color, _ = VALIDATION_STYLES[status]
|
||||
ax.add_patch(Rectangle((x - 0.5, y - 0.42), 1, 0.84,
|
||||
facecolor=color, edgecolor="#657081", linewidth=0.5))
|
||||
ax.text(x, y, "N/A" if status == "NA" else status,
|
||||
color=text_color, ha="center", va="center", fontsize=9)
|
||||
save_figure(fig, "validation_status", output_dir)
|
||||
|
||||
|
||||
def draw_metric(results: list[Result], field: str, ylabel: str,
|
||||
basename: str, output_dir: Path, diverging: bool = False) -> None:
|
||||
data = [diverging_ratio(getattr(row, field)) if diverging else getattr(row, field)
|
||||
for row in results]
|
||||
available = [v for v in data if math.isfinite(v)]
|
||||
upper = max(0, max(available, default=1.0))
|
||||
lower = min(0, min(available, default=0)) if diverging else 0
|
||||
padding = max(upper - lower, 1.0) * 0.3
|
||||
y_max = math.ceil((upper + padding) * 2) / 2
|
||||
y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0
|
||||
y_interval = 1.0 if y_max - y_min > 4.5 else 0.5
|
||||
|
||||
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
|
||||
layout="constrained")
|
||||
figure_legend(fig, results, field)
|
||||
|
||||
for ax, arch in zip(axes, ARCHITECTURES):
|
||||
selected = [row for row in results if row.arch == arch]
|
||||
ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
|
||||
fontsize=11, weight="bold", labelpad=8)
|
||||
ax.set_axisbelow(True)
|
||||
ax.set_ylim(y_min, y_max)
|
||||
ax.set_xlim(-0.5, len(selected) - 0.5)
|
||||
label_groups(ax, selected)
|
||||
ax.tick_params(axis="y", labelleft=True, pad=5, length=0)
|
||||
ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6)
|
||||
ax.yaxis.set_major_locator(MultipleLocator(y_interval))
|
||||
if diverging:
|
||||
ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}×"))
|
||||
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
|
||||
ax.axhline(0 if diverging else 1, color="#536478", linewidth=1.2,
|
||||
linestyle="--", zorder=3)
|
||||
ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6)
|
||||
|
||||
for position, row in enumerate(selected):
|
||||
ratio = getattr(row, field)
|
||||
if not math.isfinite(ratio):
|
||||
ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
|
||||
fontsize=9.5, color="#85909F", fontstyle="italic")
|
||||
continue
|
||||
compiler = advantage_compiler(ratio)
|
||||
statuses = [status for status in (row.raptor_validation, row.pimcomp_validation) if status != "PASS"]
|
||||
status = "FAIL" if "FAIL" in statuses else next(iter(statuses), "PASS")
|
||||
pattern = STAGE_PATTERNS[row.pipeline]
|
||||
measurement = diverging_ratio(ratio) if diverging else ratio
|
||||
ax.bar(position, measurement, width=0.72,
|
||||
color=COMPILER_COLORS.get(compiler, "#657081"),
|
||||
hatch=pattern,
|
||||
edgecolor=TEXT_COLOR if pattern else "none",
|
||||
linewidth=0.7 if pattern else 0.0, zorder=2)
|
||||
direction = -1 if measurement < 0 else 1
|
||||
factor = max(ratio, 1 / ratio) if diverging else ratio
|
||||
value_label(ax, position, measurement + direction * (y_max - y_min) * 0.012,
|
||||
f"{factor:.2f}×", status, direction, fontsize=9.1)
|
||||
|
||||
fit_value_labels(fig)
|
||||
save_figure(fig, basename, output_dir)
|
||||
|
||||
|
||||
def draw_absolute(results: list[Result], field: str, ylabel: str,
|
||||
basename: str, output_dir: Path, scale: float = 1.0,
|
||||
raptor_only: bool = False) -> None:
|
||||
columns = absolute_columns(results, raptor_only)
|
||||
available = [getattr(row, f"{compiler}_{field}") * scale
|
||||
for row, compiler in columns
|
||||
if math.isfinite(getattr(row, f"{compiler}_{field}"))]
|
||||
y_max = max(available, default=1.0) * 1.35
|
||||
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
|
||||
layout="constrained")
|
||||
figure_legend(fig, results, field, absolute=True, raptor_only=raptor_only)
|
||||
|
||||
for ax, arch in zip(axes, ARCHITECTURES):
|
||||
selected = [(row, compiler) for row, compiler in columns if row.arch == arch]
|
||||
positions = []
|
||||
for index, (row, _) in enumerate(selected):
|
||||
if index == 0:
|
||||
positions.append(0.0)
|
||||
continue
|
||||
previous = selected[index - 1][0]
|
||||
gap = 0.8 if row.model != previous.model else 0.4 if row.mode != previous.mode else 0.0
|
||||
positions.append(positions[-1] + 1.0 + gap)
|
||||
ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
|
||||
fontsize=11, weight="bold", labelpad=8)
|
||||
ax.set_axisbelow(True)
|
||||
ax.set_ylim(0, y_max)
|
||||
if positions:
|
||||
ax.set_xlim(positions[0] - 0.9, positions[-1] + 0.9)
|
||||
else:
|
||||
ax.set_xlim(-0.9, 0.9)
|
||||
label_groups(ax, [row for row, _ in selected], positions)
|
||||
if {row.mode for row, _ in selected} == {"latency", "throughput"}:
|
||||
centers, labels = [], []
|
||||
for (_, mode), group in groupby(enumerate(selected), key=lambda item: (item[1][0].model, item[1][0].mode)):
|
||||
group_positions = [positions[position] for position, _ in group]
|
||||
centers.append((group_positions[0] + group_positions[-1]) / 2)
|
||||
labels.append(mode.capitalize())
|
||||
ax.set_xticks(centers, labels, minor=True)
|
||||
ax.tick_params(axis="x", which="minor", length=0, pad=4, labelsize=8)
|
||||
ax.set_ylabel(ylabel, fontsize=10.5)
|
||||
if field == "throughput":
|
||||
ax.ticklabel_format(axis="y", style="plain", useOffset=False)
|
||||
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
|
||||
ax.tick_params(axis="y", length=0, pad=6)
|
||||
ax.tick_params(axis="x", which="major", length=0,
|
||||
pad=22 if {row.mode for row, _ in selected} == {"latency", "throughput"} else 6)
|
||||
|
||||
for x, (row, compiler) in zip(positions, selected):
|
||||
measurement = getattr(row, f"{compiler}_{field}") * scale
|
||||
if not math.isfinite(measurement):
|
||||
ax.text(x, y_max * 0.02, "N/A", rotation=90, ha="center",
|
||||
va="bottom", fontsize=8, color="#85909F")
|
||||
continue
|
||||
status = getattr(row, f"{compiler}_validation")
|
||||
pattern = STAGE_PATTERNS[row.pipeline] if compiler == "raptor" else ""
|
||||
ax.bar(x, measurement, width=1.0,
|
||||
color=COMPILER_COLORS[compiler],
|
||||
hatch=pattern,
|
||||
edgecolor=TEXT_COLOR if pattern else "none",
|
||||
linewidth=0.7 if pattern else 0.0, zorder=2)
|
||||
label = (f"{measurement:,.2f}".rstrip("0").rstrip(".")
|
||||
if field == "throughput" else f"{measurement:.3g}")
|
||||
value_label(ax, x, measurement + y_max * 0.012, label, status)
|
||||
|
||||
fit_value_labels(fig)
|
||||
save_figure(fig, basename, output_dir)
|
||||
|
||||
|
||||
def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
|
||||
rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1]
|
||||
stages = sorted({row.pipeline for row in rows})
|
||||
panels = [(model, color) for model, color in zip(MODELS, COLORS)
|
||||
if any(row.model == model for row in rows)]
|
||||
lookup = {(row.arch, row.model, row.pipeline): row for row in rows}
|
||||
rates = {key: quotient(1000, row.raptor_latency) if row.mode == "latency" else row.raptor_throughput
|
||||
for key, row in lookup.items()}
|
||||
fig, axes = plt.subplots(len(ARCHITECTURES), len(panels), sharex=True, sharey="col",
|
||||
figsize=(3 * len(panels), 7.2), squeeze=False, layout="constrained")
|
||||
for column, (model, color) in enumerate(panels):
|
||||
for index, arch in enumerate(ARCHITECTURES):
|
||||
ax = axes[index, column]
|
||||
keys = [(arch, model, stage) for stage in stages]
|
||||
values = [rates.get(key, math.nan) for key in keys]
|
||||
valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys]
|
||||
ax.plot(stages, values, color=color, linewidth=1.8)
|
||||
for stage, value, passed in zip(stages, values, valid):
|
||||
if math.isfinite(value):
|
||||
marker = ("s" if stage == 1 else "o") if passed else "x"
|
||||
ax.plot(stage, value, color=color, marker=marker,
|
||||
markersize=6, linestyle="none")
|
||||
ax.set_xticks(stages)
|
||||
ax.set_xlabel(model.upper(), fontsize=10.5, weight="bold", labelpad=8)
|
||||
ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
|
||||
ax.tick_params(axis="both", length=0, pad=6, labelbottom=True, labelleft=True)
|
||||
ax.ticklabel_format(axis="y", style="plain", useOffset=False)
|
||||
if column == 0:
|
||||
ax.set_ylabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
|
||||
"Throughput (inferences/s)", fontsize=10.5)
|
||||
for ax in axes.flat:
|
||||
ax.set_ylim(bottom=0)
|
||||
handles = [Line2D([], [], color="#657081", marker=marker, linestyle="none", label=label)
|
||||
for marker, label in (("s", "Derived as 1000ms / latency_ms"),
|
||||
("o", "Throughput measurement"))]
|
||||
if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()):
|
||||
handles.append(Line2D([], [], color="#424B58", marker="x", linestyle="none",
|
||||
label="Raptor validation not PASS"))
|
||||
fig.legend(handles=handles, loc="outside upper center", ncol=len(handles), frameon=False, fontsize=9.5)
|
||||
fig.supxlabel("Raptor pipeline stages", fontsize=11)
|
||||
save_figure(fig, "raptor_pipeline_throughput", output_dir)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
script_dir = Path(__file__).resolve().parent
|
||||
parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv")
|
||||
parser.add_argument("--output-dir", type=Path, default=script_dir / "plots")
|
||||
args = parser.parse_args()
|
||||
configure_style()
|
||||
results = load_results(args.input)
|
||||
valid = sum(row.comparison_passed for row in results)
|
||||
print(f"Results: {len(results)} configurations, {valid} valid comparisons, "
|
||||
f"{len(results)-valid} comparisons without two PASS statuses")
|
||||
draw_validation(results, args.output_dir)
|
||||
draw_pipeline_effect(results, args.output_dir)
|
||||
yolo_only_models = {
|
||||
model for model in MODELS if model.startswith("yolo")
|
||||
and not any(math.isfinite(getattr(row, f"pimcomp_{field}"))
|
||||
for row in results if row.model == model
|
||||
for field in ("latency", "power", "energy"))
|
||||
}
|
||||
comparisons = [row for row in results if row.model not in yolo_only_models]
|
||||
yolo_only = [row for row in results if row.model in yolo_only_models]
|
||||
for metric, name, label, mode in [
|
||||
("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None),
|
||||
("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"),
|
||||
("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"),
|
||||
("power", "power", "Average power advantage (x)", None),
|
||||
("energy", "energy", "Energy per inference advantage (x)", None),
|
||||
]:
|
||||
selected = [row for row in comparisons if mode is None or row.mode == mode]
|
||||
draw_metric(selected, metric, label,
|
||||
f"{name}_comparison", args.output_dir)
|
||||
draw_metric(selected, metric, label,
|
||||
f"{name}_diverging", args.output_dir, diverging=True)
|
||||
for metric, name, label, scale, mode in [
|
||||
("latency", "longest_stage_latency", "Longest stage latency (ms)", 1.0, None),
|
||||
("latency", "latency", "Single-inference latency (ms)", 1.0, "latency"),
|
||||
("power", "power", "Average power (mW)", 1.0, None),
|
||||
("energy", "energy", "Energy per inference (mJ)", 1e-9, None),
|
||||
("throughput", "throughput", "Throughput (inferences/s)", 1.0, "throughput"),
|
||||
]:
|
||||
for suffix, group, raptor_only in (
|
||||
("absolute", comparisons, False), ("yolo", yolo_only, True)
|
||||
):
|
||||
selected = [row for row in group if mode is None or row.mode == mode]
|
||||
if not selected:
|
||||
continue
|
||||
draw_absolute(selected, metric, label,
|
||||
f"{name}_{suffix}", args.output_dir, scale,
|
||||
raptor_only=raptor_only)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,41 @@
|
||||
arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
|
||||
arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26
|
||||
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,423.80,482.60,431626597.00,541691691.00,raptor,1.14
|
||||
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.86,1.04,1160.00,965.00,432.99,482.60,416224904.20,541691691.00,raptor,1.20
|
||||
arch-a,vgg8,throughput,8,batch,PASS,PASS,1.63,1.04,615.00,965.00,308.12,482.60,616026711.90,541691691.00,pimcomp,1.57
|
||||
arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394498463.12,13868866808.12,raptor,4.15
|
||||
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
|
||||
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.62,13.46,51.00,74.30,376.10,481.60,8024732847.00,6563451004.00,pimcomp,1.46
|
||||
arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.92,13.46,50.20,74.30,368.24,481.60,9023016592.00,6563451004.00,pimcomp,1.48
|
||||
arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
|
||||
arch-a,resnet34,throughput,2,batch,PASS,FAIL,22.26,23.58,44.90,42.40,473.84,515.68,16417303180.00,12283627410.00,raptor,1.06
|
||||
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.37,23.58,36.50,42.40,415.90,515.68,17019112320.00,12283627410.00,pimcomp,1.16
|
||||
arch-a,resnet34,throughput,8,batch,PASS,FAIL,41.04,23.58,24.40,42.40,346.15,515.68,17351854700.00,12283627410.00,pimcomp,1.74
|
||||
arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.88,231.19,5513234559.92,14547744283.24,raptor,6.44
|
||||
arch-a,googlenet,throughput,2,batch,PASS,FAIL,10.16,18.26,98.50,54.80,544.21,435.11,6029999329.00,8164787803.00,raptor,1.80
|
||||
arch-a,googlenet,throughput,4,batch,PASS,FAIL,11.96,18.26,83.60,54.80,473.16,435.11,6741923526.00,8164787803.00,raptor,1.53
|
||||
arch-a,googlenet,throughput,8,batch,PASS,FAIL,20.66,18.26,48.40,54.80,348.76,435.11,8421537146.00,8164787803.00,pimcomp,1.13
|
||||
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,120.93,NA,8.27,NA,313.20,NA,41830757130.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,4,batch,PASS,FAIL,92.06,NA,10.90,NA,351.03,NA,39451980450.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,104.07,NA,9.61,NA,330.76,NA,44002712680.00,NA,NA,NA
|
||||
arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
|
||||
arch-b,vgg8,throughput,2,batch,PASS,PASS,1.18,1.07,846.00,932.00,327.05,409.47,429183159.10,439288835.80,pimcomp,1.10
|
||||
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.92,1.07,1090.00,932.00,383.82,409.47,464794094.50,439288835.80,raptor,1.17
|
||||
arch-b,vgg8,throughput,8,batch,PASS,PASS,1.62,1.07,616.00,932.00,276.49,409.47,524048148.70,439288835.80,pimcomp,1.51
|
||||
arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
|
||||
arch-b,resnet18,throughput,2,batch,PASS,FAIL,13.08,13.43,76.40,74.40,440.79,484.80,6399942695.00,6569655551.00,raptor,1.03
|
||||
arch-b,resnet18,throughput,4,batch,PASS,FAIL,20.68,13.43,48.40,74.40,327.57,484.80,7777606323.00,6569655551.00,pimcomp,1.54
|
||||
arch-b,resnet18,throughput,8,batch,PASS,FAIL,21.17,13.43,47.20,74.40,317.64,484.80,8897039159.00,6569655551.00,pimcomp,1.58
|
||||
arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
|
||||
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
|
||||
arch-b,resnet34,throughput,4,batch,PASS,FAIL,39.61,22.85,25.20,43.80,315.01,493.33,14629161660.00,11392215660.00,pimcomp,1.74
|
||||
arch-b,resnet34,throughput,8,batch,PASS,FAIL,49.34,22.85,20.30,43.80,285.76,493.33,16591020470.00,11392215660.00,pimcomp,2.16
|
||||
arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422799605.92,9182062336.24,raptor,3.38
|
||||
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.83,21.39,102.00,46.80,520.79,417.87,5669213312.00,9014726090.00,raptor,2.18
|
||||
arch-b,googlenet,throughput,4,batch,PASS,FAIL,14.47,21.39,69.10,46.80,401.01,417.87,6835460477.00,9014726090.00,raptor,1.48
|
||||
arch-b,googlenet,throughput,8,batch,PASS,FAIL,37.28,21.39,26.80,46.80,245.38,417.87,11453848400.00,9014726090.00,pimcomp,1.75
|
||||
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,118.15,NA,8.46,NA,287.59,NA,37062981240.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,94.18,NA,10.60,NA,316.45,NA,37424491250.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,142.86,NA,7.00,NA,290.68,NA,41525990280.00,NA,NA,NA
|
||||
|
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
@@ -43,6 +44,7 @@ COMPARISONS = (
|
||||
("throughput", 8, "batch"),
|
||||
)
|
||||
RESULTS_FILENAME = "results_comparison.csv"
|
||||
DEFAULT_RESULTS_PATH = Path(__file__).with_name(RESULTS_FILENAME)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -128,9 +130,9 @@ def write_results_csv(
|
||||
comparisons: tuple[tuple[str, int, str], ...] = COMPARISONS,
|
||||
ablation_variant: str | None = None,
|
||||
) -> Path:
|
||||
output = (root or SUITE) / RESULTS_FILENAME
|
||||
output = DEFAULT_RESULTS_PATH if root is None else root / RESULTS_FILENAME
|
||||
if ablation_variant is not None:
|
||||
output = output.parent / "ablation" / ablation_variant / RESULTS_FILENAME
|
||||
output = (root or SUITE) / "ablation" / ablation_variant / RESULTS_FILENAME
|
||||
fields = (
|
||||
"arch",
|
||||
"model",
|
||||
@@ -247,18 +249,26 @@ def write_results_csv(
|
||||
|
||||
|
||||
def performance_values(performance: dict) -> dict[str, float | None]:
|
||||
fields = {"latency": ("latency_ms", "average_latency_ms"),
|
||||
"throughput": ("throughput",), "power": ("average_power_mw",),
|
||||
"energy": ("average_energy_pj",)}
|
||||
values = dict.fromkeys(fields)
|
||||
if performance.get("error") or performance.get("skipped") or performance.get("unsupported"):
|
||||
return values
|
||||
parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
|
||||
return {
|
||||
"latency": (
|
||||
performance.get("latency_ms")
|
||||
or performance.get("average_latency_ms")
|
||||
or parsed.get("latency_ms")
|
||||
or parsed.get("average_latency_ms")
|
||||
),
|
||||
"throughput": performance.get("throughput") or parsed.get("throughput"),
|
||||
"power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
|
||||
"energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
|
||||
}
|
||||
for metric, names in fields.items():
|
||||
for value in (source.get(name) for source in (performance, parsed) for name in names):
|
||||
try:
|
||||
number = float(value)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if math.isfinite(number) and number > 0:
|
||||
values[metric] = number
|
||||
break
|
||||
if performance.get("output_count", parsed.get("output_count")) == 0:
|
||||
for metric in ("latency", "throughput", "energy"):
|
||||
values[metric] = None
|
||||
return values
|
||||
|
||||
|
||||
def comparison_passed(report: dict) -> bool:
|
||||
@@ -283,7 +293,9 @@ def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
|
||||
|
||||
|
||||
def format_value(value: float | None) -> str:
|
||||
return "NA" if value is None else f"{value:.2f}"
|
||||
if value is None or not math.isfinite(value) or value <= 0:
|
||||
return "NA"
|
||||
return f"{value:.6g}"
|
||||
|
||||
|
||||
def print_stage(title: str, color: str) -> None:
|
||||
@@ -352,15 +364,12 @@ def comparison_command(
|
||||
raptor_extra_args: list[str] | tuple[str, ...] = (),
|
||||
raptor_only: bool = False,
|
||||
reuse_raptor_report: Path | None = None,
|
||||
reuse_pimcomp_dir: Path | None = None,
|
||||
reuse_pimcomp_report: Path | None = None,
|
||||
) -> list[str]:
|
||||
time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
|
||||
reuse_args = []
|
||||
if reuse_raptor_report is not None:
|
||||
reuse_args.extend(["--reuse-raptor-report", str(reuse_raptor_report)])
|
||||
if reuse_pimcomp_dir is not None:
|
||||
reuse_args.extend(["--reuse-pimcomp-dir", str(reuse_pimcomp_dir)])
|
||||
if reuse_pimcomp_report is not None:
|
||||
reuse_args.extend(["--reuse-pimcomp-report", str(reuse_pimcomp_report)])
|
||||
return [
|
||||
@@ -430,6 +439,12 @@ def comparison_command_for(
|
||||
reuse_shared_pimcomp: bool,
|
||||
) -> list[str]:
|
||||
report = spec.output_dir / "pimcomp/comparison_report.json"
|
||||
reuse_pimcomp_report = None
|
||||
if not args.raptor_only:
|
||||
if args.only == "raptor":
|
||||
reuse_pimcomp_report = report
|
||||
elif reuse_shared_pimcomp:
|
||||
reuse_pimcomp_report = spec.shared_pimcomp_dir.parent / "comparison_report.json"
|
||||
return comparison_command(
|
||||
spec.model,
|
||||
spec.functional_model,
|
||||
@@ -450,20 +465,21 @@ def comparison_command_for(
|
||||
if args.only == "pimcomp"
|
||||
else None
|
||||
),
|
||||
reuse_pimcomp_dir=(
|
||||
spec.shared_pimcomp_dir
|
||||
if reuse_shared_pimcomp and args.only != "raptor" and not args.raptor_only
|
||||
else None
|
||||
),
|
||||
reuse_pimcomp_report=report if args.only == "raptor" and not args.raptor_only else None,
|
||||
reuse_pimcomp_report=reuse_pimcomp_report,
|
||||
)
|
||||
|
||||
|
||||
def pimcomp_artifact_ready(output_dir: Path) -> bool:
|
||||
return all(
|
||||
(output_dir / name).is_file()
|
||||
for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
|
||||
) and (output_dir.parent / "comparison_report.json").is_file()
|
||||
def pimcomp_report_stamp(output_dir: Path) -> tuple[int, int, int] | None:
|
||||
try:
|
||||
stat = (output_dir.parent / "comparison_report.json").stat()
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
return stat.st_ino, stat.st_mtime_ns, stat.st_ctime_ns
|
||||
|
||||
|
||||
def pimcomp_report_ready(output_dir: Path, previous: tuple[int, int, int] | None) -> bool:
|
||||
stamp = pimcomp_report_stamp(output_dir)
|
||||
return stamp is not None and stamp != previous
|
||||
|
||||
|
||||
def run_comparison_jobs(
|
||||
@@ -533,7 +549,7 @@ def main() -> int:
|
||||
parser.add_argument(
|
||||
"--out-dir",
|
||||
type=Path,
|
||||
help="Suite root; generated artifacts go below each model's artifacts/ directory (default: validation/networks/pimcomp_models).",
|
||||
help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_comparison.csv beside this script.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--common-dir",
|
||||
@@ -687,11 +703,6 @@ def main() -> int:
|
||||
required = comparison_dir / "pimcomp/comparison_report.json"
|
||||
if not required.exists():
|
||||
missing_reuse.append(str(required))
|
||||
elif args.only == "raptor":
|
||||
if json.loads(required.read_text(encoding="utf-8")).get(
|
||||
"pimcomp_model_source"
|
||||
) != "original_onnx":
|
||||
missing_reuse.append(f"{required} (not generated from the original ONNX model)")
|
||||
if missing_reuse:
|
||||
parser.error(
|
||||
f"--only {args.only} needs existing comparison artifacts: "
|
||||
@@ -713,7 +724,7 @@ def main() -> int:
|
||||
f"Comparison jobs: "
|
||||
f"{sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())}"
|
||||
)
|
||||
print(f"Results root: {out_dir or SUITE}")
|
||||
print(f"Artifacts root: {out_dir or SUITE}")
|
||||
print("=" * 72)
|
||||
|
||||
print_stage("Prepare shared artifacts", STAGE_COLORS["Build runner"])
|
||||
@@ -799,6 +810,10 @@ def main() -> int:
|
||||
print_directly = min(args.jobs, len(comparison_specs)) == 1
|
||||
try:
|
||||
with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-pimcomp-")) as log_dir:
|
||||
previous_reports = {
|
||||
spec.shared_pimcomp_dir: pimcomp_report_stamp(spec.shared_pimcomp_dir)
|
||||
for spec in anchor_specs
|
||||
}
|
||||
anchor_failed, log_offset = run_comparison_jobs(
|
||||
anchor_jobs,
|
||||
args.jobs,
|
||||
@@ -812,7 +827,9 @@ def main() -> int:
|
||||
comparison_command_for(
|
||||
spec,
|
||||
args,
|
||||
reuse_shared_pimcomp=pimcomp_artifact_ready(spec.shared_pimcomp_dir),
|
||||
reuse_shared_pimcomp=pimcomp_report_ready(
|
||||
spec.shared_pimcomp_dir, previous_reports[spec.shared_pimcomp_dir]
|
||||
),
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
@@ -1,24 +1,18 @@
|
||||
# Pimcomp correctness experiments
|
||||
|
||||
Run from the repository root using the repository virtual environment, built
|
||||
release Raptor, Pimcomp frontend/backend, and `pimsim-nn`.
|
||||
|
||||
## Synchronization ordering
|
||||
|
||||
The synchronization experiment checks whether repeated PIMCOMP and Raptor
|
||||
programs preserve cross-core global-memory generations under legal execution
|
||||
schedules. It builds a small two-convolution model for Arch-A, Arch-B, and
|
||||
Arch-C, finds cross-core `ST`/`LD` dependencies, and compares greedy,
|
||||
bounded-stall, randomized, and adversarial runs using the Rust simulator's
|
||||
provenance trace. Byte-identical batch inputs isolate intermediate-memory
|
||||
ordering from the separate host-input lifetime issue.
|
||||
Checks whether repeated PIMCOMP and Raptor programs preserve cross-core
|
||||
global-memory generations under legal execution schedules. A two-convolution
|
||||
model runs on Arch-A/B/C with greedy, bounded-stall, randomized, and adversarial
|
||||
schedules; Rust provenance traces identify cross-core `ST`/`LD` dependencies.
|
||||
Byte-identical batch inputs isolate intermediate-memory ordering from host-input
|
||||
lifetime. Diagnostic scheduling does not change the performance oracle.
|
||||
|
||||
The experiment records static dependency evidence, dynamic provenance,
|
||||
functional output comparisons, architecture contract classifications, and
|
||||
per-artifact PIMCOMP/Raptor conclusions. Diagnostic scheduling changes only
|
||||
the functional simulator's execution order; it does not modify PIMCOMP or the
|
||||
`pimsim-nn` performance oracle.
|
||||
|
||||
Prerequisites are the repository virtual environment, the release Raptor
|
||||
compiler, built PIMCOMP frontend/backend, and the existing `pimsim-nn` build.
|
||||
Run from the repository root with a new or empty output directory:
|
||||
Use a new or empty output directory:
|
||||
|
||||
```bash
|
||||
.venv/bin/python \
|
||||
@@ -27,102 +21,50 @@ Run from the repository root with a new or empty output directory:
|
||||
--batch-size 4 --seed 0 --self-check
|
||||
```
|
||||
|
||||
The directory contains:
|
||||
Outputs: `adversarial_memory_sync_report.{json,md}` and per-architecture
|
||||
evidence. Contract classifications use
|
||||
[`architecture_evidence.json`](synchronization/architecture_evidence.json).
|
||||
|
||||
- [`run_experiment.py`](synchronization/run_experiment.py): orchestration,
|
||||
classification, reporting, and self-checks.
|
||||
- [`global_memory.py`](synchronization/global_memory.py): artifact compilation,
|
||||
dependency analysis, simulator execution, and provenance helpers.
|
||||
- [`architecture_contract.py`](synchronization/architecture_contract.py) and
|
||||
[`architecture_evidence.json`](synchronization/architecture_evidence.json):
|
||||
conservative architecture-contract evidence and labels.
|
||||
## Batch prefill reproduction
|
||||
|
||||
The main outputs are `adversarial_memory_sync_report.json` and
|
||||
`adversarial_memory_sync_report.md`, with per-architecture evidence below the
|
||||
same output root.
|
||||
|
||||
## Pimcomp batch prefill reproduction
|
||||
|
||||
Pimcomp's batch scheduler currently emits an incomplete standalone program for
|
||||
models containing post operations. The generated `VerificationInfo.json` uses a
|
||||
negative `source_address` to identify the preceding node, but the batch
|
||||
verifier resolves that address by copying the provider tensor directly from
|
||||
ONNX Runtime's `intermediate_result` table. It does not read the simulated
|
||||
global-memory buffer.
|
||||
|
||||
The corresponding post-operation scheduling call is commented out in
|
||||
Pimcomp's batch scheduler omits post-operation producers for some Pool, ReLU,
|
||||
Concat, and Reshape tensors. Its verifier resolves negative `source_address`
|
||||
entries using ONNX Runtime intermediates rather than simulated global memory.
|
||||
The corresponding scheduling call is disabled in
|
||||
[`BatchPipelineSchedule.cpp`](../../../../../third_party/PIMCOMP-NN/backend/BatchPipelineSchedule.cpp#L1535).
|
||||
Consequently, Pool, ReLU, Concat, and Reshape provider tensors can be consumed
|
||||
by later `LD` instructions without any generated `ST` producer. The comparison
|
||||
exporter initializes those host-side intermediate buffers to zero, which makes
|
||||
the missing computation visible in Rust functional validation.
|
||||
|
||||
For the checked-in GoogLeNet throughput/pipeline2 artifact, 39 provider tensors
|
||||
are referenced by batch loads. Nineteen have generated stores; twenty are
|
||||
never written. Preloading the provider tensors with the same ONNX Runtime
|
||||
intermediates used by Pimcomp's verifier makes the exported program pass. This
|
||||
reproduces the verifier's input contract; it does not repair Pimcomp's batch
|
||||
schedule.
|
||||
|
||||
Run the default reproduction from the repository root:
|
||||
Rust validation starts those buffers at zero, exposing the missing computation.
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py
|
||||
```
|
||||
|
||||
The launcher accepts an alternate comparison directory, model, work directory,
|
||||
and shared reference-artifact directory:
|
||||
The default uses GoogLeNet Arch-A throughput/pipeline2. Optional positional
|
||||
arguments are comparison directory, model, work directory, and shared reference
|
||||
directory; the default work directory is `correctness/prefill/` below the
|
||||
comparison artifacts.
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py \
|
||||
validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2 \
|
||||
validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx \
|
||||
validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2/correctness/prefill \
|
||||
validation/networks/pimcomp_models/googlenet/artifacts/common
|
||||
```
|
||||
The experiment compares the original memory image with one prefilled using
|
||||
[`batch_memory.py`](prefill/batch_memory.py). In the recorded case, 20 of 39
|
||||
loaded provider tensors have no store producer; prefill reduces maximum
|
||||
reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's
|
||||
verifier contract; it does not repair the standalone batch program.
|
||||
|
||||
Without the optional work-directory argument, the experiment uses the same
|
||||
`correctness/prefill/` directory below the comparison artifacts.
|
||||
## ResNet BatchNorm gap
|
||||
|
||||
It runs the exported artifact once with its original memory image and once
|
||||
with [`batch_memory.py`](prefill/batch_memory.py), then compares both
|
||||
outputs with the recorded native reference. The expected GoogLeNet result is a
|
||||
baseline maximum difference near `6.70705` and a prefilled maximum difference
|
||||
near `4.05e-6`.
|
||||
Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
|
||||
removes BatchNormalization without folding its affine transform into weights
|
||||
and bias. Its [verifier](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
|
||||
substitutes identity BatchNorm parameters, so agreement with that verifier does
|
||||
not establish equivalence to the original ONNX model.
|
||||
|
||||
The issue is in Pimcomp batch scheduling/validation semantics, not in the Rust
|
||||
simulator's vector-length interpretation. Vector lengths remain element counts
|
||||
as specified by the reference ISA.
|
||||
|
||||
## ResNet BatchNorm correctness gap
|
||||
|
||||
Pimcomp has a separate correctness limitation in its ResNet element pipeline.
|
||||
`BatchNormalization` is an ONNX operation. The frontend's
|
||||
[`fuse_operators()` pass](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
|
||||
marks it as fused and removes the node from the
|
||||
scheduled graph, but the released path does not fold the BatchNorm affine
|
||||
transform into the convolution weights and bias. The corresponding verifier
|
||||
workaround in [`verification.py`](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
|
||||
replaces BatchNorm parameters with identity values (scale and variance equal
|
||||
to one, bias and mean equal to zero) before running ONNX Runtime.
|
||||
|
||||
Therefore Pimcomp's native verifier and exported element program agree with
|
||||
each other, but they do not implement the original ResNet ONNX model. On the
|
||||
current ResNet-18 Arch-A latency artifact, using the same input:
|
||||
Recorded ResNet-18 Arch-A latency errors on the same input:
|
||||
|
||||
| Comparison | Maximum absolute difference |
|
||||
|---|---:|
|
||||
| Pimcomp native verifier vs Rust export | `6.7e-8` |
|
||||
| Rust export vs original ONNX reference | `4.8294563` |
|
||||
|
||||
The Pimcomp output ranges from approximately `-0.187` to `0.220`, while the
|
||||
original ONNX output ranges from `-3.572` to `4.834`; 462 of 1000 final
|
||||
elements differ by more than one. This is not a Rust simulator or Python
|
||||
exporter regression. It is a Pimcomp model-semantics mismatch caused by
|
||||
dropping BatchNorm numerics. The same issue affects the ResNet-34 latency
|
||||
artifact. VGG and GoogLeNet do not show this particular mismatch because they
|
||||
do not contain the same ResNet BatchNorm path.
|
||||
|
||||
The latency comparison intentionally uses the original ONNX reference, so
|
||||
these Pimcomp rows must remain `FAIL` until Pimcomp folds BatchNorm correctly
|
||||
or the comparison explicitly uses a BatchNorm-neutralized reference.
|
||||
The same gap affects ResNet-34 latency. Comparison rows use the original ONNX
|
||||
reference and must remain `FAIL` until BatchNorm numerics are implemented
|
||||
correctly. This is independent of simulator vector lengths, which remain
|
||||
element counts.
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Shared appearance and PNG/SVG export for PIM result plots."""
|
||||
|
||||
from pathlib import Path
|
||||
from math import ceil
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Patch
|
||||
from matplotlib.path import Path as PlotPath
|
||||
from matplotlib.patheffects import AbstractPathEffect
|
||||
from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath
|
||||
|
||||
|
||||
TEXT_COLOR = "#243449"
|
||||
HATCH_COLOR = "#C62828"
|
||||
COLORS = ("#2B65A1", "#168C8B", "#D18D32", "#8062A5", "#B65D60")
|
||||
|
||||
|
||||
class CenteredHatch(AbstractPathEffect):
|
||||
"""Anchor dot and diagonal hatch patterns to each rectangle's center."""
|
||||
|
||||
def draw_path(self, renderer, gc, tpath, affine, rgbFace=None):
|
||||
hatch = gc.get_hatch() or ""
|
||||
base_gc = renderer.new_gc()
|
||||
base_gc.copy_properties(gc)
|
||||
base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\"))
|
||||
renderer.draw_path(base_gc, tpath, affine, rgbFace)
|
||||
base_gc.restore()
|
||||
|
||||
bounds = affine.transform_path(tpath).get_extents()
|
||||
center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2
|
||||
vertices = []
|
||||
diagonal = 2 ** -0.5
|
||||
for symbols, dx, dy in (("/", diagonal, diagonal), ("\\", diagonal, -diagonal)):
|
||||
count = sum(hatch.count(symbol) for symbol in symbols)
|
||||
if count == 0:
|
||||
continue
|
||||
spacing = renderer.points_to_pixels(12 / count)
|
||||
half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2
|
||||
half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2
|
||||
extent = ceil(half_span / spacing)
|
||||
for step in range(-extent, extent + 1):
|
||||
x = center_x - dy * step * spacing
|
||||
y = center_y + dx * step * spacing
|
||||
vertices.extend(((x - dx * half_length, y - dy * half_length),
|
||||
(x + dx * half_length, y + dy * half_length)))
|
||||
line_gc = renderer.new_gc()
|
||||
line_gc.copy_properties(gc)
|
||||
line_gc.set_hatch(None)
|
||||
line_gc.set_foreground(gc.get_hatch_color())
|
||||
line_gc.set_linewidth(gc.get_hatch_linewidth())
|
||||
line_gc.set_clip_path(TransformedPath(tpath, affine))
|
||||
if vertices:
|
||||
renderer.draw_path(line_gc, PlotPath(vertices, [PlotPath.MOVETO, PlotPath.LINETO] * (len(vertices) // 2)),
|
||||
IdentityTransform())
|
||||
if "." in hatch:
|
||||
spacing = renderer.points_to_pixels(12 / hatch.count("."))
|
||||
x_extent = ceil(bounds.width / (2 * spacing))
|
||||
y_extent = ceil(bounds.height / (2 * spacing))
|
||||
centers = [(center_x + x * spacing, center_y + y * spacing)
|
||||
for x in range(-x_extent, x_extent + 1)
|
||||
for y in range(-y_extent, y_extent + 1)]
|
||||
line_gc.set_linewidth(0)
|
||||
renderer.draw_markers(line_gc, PlotPath.unit_circle(),
|
||||
Affine2D().scale(renderer.points_to_pixels(0.9)),
|
||||
PlotPath(centers), IdentityTransform(), gc.get_hatch_color())
|
||||
line_gc.restore()
|
||||
|
||||
|
||||
def configure_style() -> None:
|
||||
plt.rcParams.update({
|
||||
"font.family": "DejaVu Sans",
|
||||
"axes.spines.top": False,
|
||||
"axes.spines.right": False,
|
||||
"axes.labelcolor": TEXT_COLOR,
|
||||
"text.color": TEXT_COLOR,
|
||||
"xtick.color": TEXT_COLOR,
|
||||
"ytick.color": TEXT_COLOR,
|
||||
"axes.edgecolor": "#CFD6DF",
|
||||
"savefig.facecolor": "white",
|
||||
"figure.facecolor": "white",
|
||||
})
|
||||
|
||||
|
||||
def value_label(ax, x, y, label: str, status: str = "PASS", direction: int = 1,
|
||||
fontsize: float = 8) -> None:
|
||||
alignment = "bottom" if direction > 0 else "top"
|
||||
rotation = 90 if direction > 0 else -90
|
||||
text = ax.text(x, y, label, rotation=rotation, ha="center", va=alignment,
|
||||
fontsize=fontsize, color=TEXT_COLOR)
|
||||
if status != "PASS":
|
||||
height = text.get_window_extent(ax.figure.canvas.get_renderer()).height
|
||||
offset = direction * (height * 72 / ax.figure.dpi + 3)
|
||||
ax.annotate("INVALID" if status == "FAIL" else status, (x, y),
|
||||
xytext=(0, offset), textcoords="offset points",
|
||||
rotation=rotation, ha="center", va=alignment, fontsize=fontsize,
|
||||
color=HATCH_COLOR)
|
||||
|
||||
|
||||
def fit_value_labels(fig) -> None:
|
||||
"""Reserve vertical space for rotated labels using their rendered size."""
|
||||
for ax in fig.axes:
|
||||
for text in ax.texts:
|
||||
text.set_in_layout(False)
|
||||
fig.canvas.draw()
|
||||
renderer = fig.canvas.get_renderer()
|
||||
limits = []
|
||||
for ax in fig.axes:
|
||||
bounds = ax.get_window_extent(renderer)
|
||||
labels = []
|
||||
for text in ax.texts:
|
||||
x, y = text.xy if hasattr(text, "xy") else text.get_position()
|
||||
anchor = ax.transData.transform((x, y))[1]
|
||||
extent = text.get_window_extent(renderer)
|
||||
labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height,
|
||||
max(0, extent.y1 - anchor + 3) / bounds.height))
|
||||
lower, upper = ax.get_ylim()
|
||||
for _ in range(12):
|
||||
lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
|
||||
upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)])
|
||||
limits.append((lower, upper))
|
||||
for ax in fig.axes:
|
||||
ax.set_ylim(min(lower for lower, _ in limits), max(upper for _, upper in limits))
|
||||
|
||||
|
||||
def save_figure(fig, basename: str, output_dir: Path) -> None:
|
||||
for patch in fig.findobj(Patch):
|
||||
hatch = patch.get_hatch() or ""
|
||||
if any(symbol in hatch for symbol in "./\\"):
|
||||
patch.set_path_effects([CenteredHatch()])
|
||||
for extension in ("png", "svg"):
|
||||
target = output_dir / extension / f"{basename}.{extension}"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(target, dpi=220, bbox_inches="tight", pad_inches=0.04)
|
||||
print(target)
|
||||
plt.close(fig)
|
||||
Reference in New Issue
Block a user