nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

rework documentation
Validate Operations / validate-operations (push) Failing after 3h10m55s
Details

Browse Source 
add plot generation scripts
adjust validation tools
add up-to-date results
This commit is contained in:
NiccoloN
2026-10-19 16:34:01 +02:00
parent 9053cc5e8b
commit 034c197260
33 changed files with 3077 additions and 1798 deletions
Download Patch File Download Diff File Expand all files Collapse all files
validation/README.md
+87 -219
View File
@@ -1,265 +1,133 @@
# Raptor validation
`validate.py` validates every ONNX model below a selected directory. For each
model it can:
1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile Pim artifacts with Raptor;
4. run the reference implementation and functional Pim simulator;
5. compare their outputs;
6. run Pimsim to report latency, throughput, power, and energy.
Run the script from the repository root with the repository Python environment.
`validate.py` discovers ONNX models recursively, compiles a native ONNX-MLIR
reference and Raptor artifacts, compares functional simulator outputs, and
optionally measures Pimsim latency, throughput, power, and energy.
Run from the repository root using `.venv`.
## Prerequisites
- A built Raptor compiler, normally
`build_release/Release/bin/onnx-mlir`.
- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
- The Rust toolchain used by the functional simulator.
- The Rust functional simulator under
`backend-simulators/pim/pim-simulator`, unless overridden.
- A built `pimsim-nn` under
`backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
skipped or its path is overridden.
- Built release Raptor and ONNX-MLIR runtime headers.
- `numpy`, `onnx`, and `colorama` installed in the repository environment.
- Rust toolchain and `backend-simulators/pim/pim-simulator`.
- Built `backend-simulators/pim/pimsim-nn/build`, unless performance
simulation is skipped.
## Basic usage
See the [build guide](../README.md#build).
Validate the complete operation suite:
## Usage
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations \
--verbose \
--raptor-extra-arg=--pim-detect-communication-deadlock \
--raptor-extra-arg=--pim-export-spatial-dataflow=none
--raptor-extra-arg=--pim-detect-communication-deadlock
```
Validate one operation category or case:
Run the complete operations suite outside the sandbox so parallel workers can
start. For a focused case, change `--operations-dir` to
`validation/operations/gemm/small` or a network directory such as
`validation/networks/yolo11n/depth_04`.
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small
```
Defaults run latency (one input) and throughput (pipeline 4, four distinct
inputs). Both reuse reference artifacts, and every throughput output is
compared with its own reference. Models run with four workers by default.
Results go to `validation_results.csv` in the selected model tree.
Validate a network or network slice:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/networks/yolo11n/depth_04
```
`--operations-dir` may point to any directory tree containing `.onnx` files.
The script discovers them recursively and writes `validation_results.csv` in
that directory while retaining separate latency and throughput terminal tables.
## Pim validation tools
- [Pimcomp model suite](networks/pimcomp_models/README.md)
- [Pimcomp model comparison tools](tools/pim/pimcomp/compare/README.md)
- [Pimcomp correctness study](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation study](tools/pim/ablation/README.md)
## Validation modes
The default mode runs latency and throughput in one validation job. Latency
uses one input, while throughput uses `--pipeline=4` with four distinct inputs.
Both modes reuse the generated input batch, native runner, and reference
outputs, and every throughput output is compared with its own reference.
Use `--compile-only` to build the reference runner and Pim artifacts without
executing either implementation:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small \
--compile-only
```
Use `--run-only` to reuse those artifacts and perform input generation,
reference execution, simulation, and comparison:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations/gemm/small \
--run-only
```
`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
with a diagnostic if its required compiled artifacts are missing.
## Parallel execution and output
Models run in parallel using all available CPUs by default. Set the worker
count with `-j` or `--jobs`:
```bash
.venv/bin/python validation/validate.py \
--raptor-path build_release/Release/bin/onnx-mlir \
--onnx-include-dir onnx-mlir/include \
--operations-dir validation/operations \
--jobs 8
```
Append `--compile-only` to build artifacts without execution, then
`--run-only` to reuse them for input generation, execution, and comparison.
These flags are mutually exclusive; run-only fails if compiled artifacts are
missing. `--clean` removes generated artifacts and stale locks, and requires
neither compiler nor include paths.
## Options
| Option | Description |
Use `--help` for the full command reference.
| Option | Purpose / default |
|---|---|
| `-h`, `--help` | Print command help and exit. |
| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
| `--non-functional-simulator-build-dir PATH` | Pimsim build directory. Defaults to the in-tree build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
| `--skip-non-functional-simulation` | Skip Pimsim latency, throughput, power, and energy measurement. |
| `--no-fast` | Disable fast throughput convergence for authoritative full-duration Pimsim experiments. |
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
| `--core-count INT` | Pim core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
| `--clean` | Remove generated validation artifacts and exit. |
| `--compile-only` | Compile reference and Pim artifacts without execution or comparison. |
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average Pim pass timings. |
| `--raptor-path PATH` | Compiler binary; required except for cleanup. |
| `--onnx-include-dir PATH` | Runtime headers; required except for cleanup. |
| `--operations-dir PATH` | Model tree; default: `validation/operations`. |
| `--simulator-dir PATH` | Override the in-tree functional simulator crate. |
| `--non-functional-simulator-build-dir PATH` | Override the in-tree Pimsim build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Performance profile; default: Arch-A. |
| `--core-count N`, `--crossbar-count N`, `--crossbar-size N` | Raptor resources; defaults: Arch-A (168, 96, 128). |
| `--skip-non-functional-simulation` | Skip performance measurement. |
| `--no-fast` | Use full-duration throughput measurement. |
| `--threshold FLOAT`, `--relative-threshold FLOAT` | Output tolerances; defaults: `1e-3`, `1e-5`. |
| `--seed N` | Input RNG seed; default: 0. |
| `--raptor-extra-arg=ARG` | Forward compiler arguments; repeat as needed. |
| `--command-timeout-seconds FLOAT` | Subprocess timeout; default: 1000000 seconds. |
| `-j N`, `--jobs N` | Parallel workers; default: 4. |
| `--compile-only`, `--run-only`, `--clean` | Compile, reuse, or remove artifacts. |
| `--verbose` | Include passing subprocess logs and average Pim pass timings. |
Arguments beginning with `--` that are passed through to Raptor should use the
equals form:
Forwarded options beginning with `--` require the equals form, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.
```bash
--raptor-extra-arg=--pim-detect-communication-deadlock
```
## Performance and result interpretation
## Hardware profiles and non-functional simulation
The selected [hardware profile](networks/pimcomp_models/README.md#hardware-profiles)
must match the core/crossbar flags. A mismatch disables performance simulation
and prints the incompatible values; functional validation still runs.
Canonical configs and meshes live under `validation/pimsim_configs/pimcomp/`.
The selected Pimcomp profile must match `--core-count`, `--crossbar-count`, and
`--crossbar-size`. A mismatch disables only non-functional simulation and
prints the incompatible values; functional validation still runs.
Throughput defaults to `pimsim-nn --fast` with a 1000 ms convergence deadline.
It compares consecutive two-round windows at 1% tolerance and falls back to
full-duration results if convergence fails. Use `--no-fast` for authoritative
experiments. See [metric definitions](../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, power, and energy interpretation.
The checked-in profiles are under
`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
`latency_config.json`, while throughput uses
`throughput_config_<time>ms.json` and the mesh beside it.
Functional compilation/output comparison determines PASS/FAIL. Performance
failures show `ERROR` without changing a functional PASS; explicit unsupported
operations show `UNSUPPORTED`. Pimsim's `vsoftmax` is unsupported.
Skipped measurements remain visibly skipped.
Throughput measurement defaults to `pimsim-nn --fast` with a 1000 ms
convergence deadline. Fast mode compares consecutive two-round windows with a
fixed 1% tolerance and falls back to the legacy full-duration result if it
does not converge. Use `--no-fast` for authoritative experiments.
Exit codes: `0` for all models passing or successful cleanup; `1` for validation
failure, an invalid model directory, or no ONNX files; `2` for invalid arguments.
Use `--skip-non-functional-simulation` when latency, throughput, power, and energy are not required.
The summary reports non-functional results as measured, failed, unsupported, or
skipped.
## Artifacts and suites
Overall PASS/FAIL is determined by compilation and functional output
comparison. A non-functional simulation failure remains visible as `ERROR` in
the corresponding latency, throughput, power, or energy columns but does not
change a functional PASS.
Pimsim does not currently implement the `vsoftmax` instruction. When its
explicit unsupported-op diagnostic is encountered, Softmax validations retain
their functional PASS and show `UNSUPPORTED` in the non-functional columns.
Other Pimsim failures remain `ERROR`.
## Generated artifacts
Generated files are grouped below an `artifacts/` directory beside each model
or operation case. This keeps checked-in ONNX files and generated trees
separate and lets `--clean` remove the complete workspace, including stale
validation lock files:
The regular validator uses these paths beside each model:
| Path | Contents |
|---|---|
| `artifacts/inputs.csv` | Generated inputs, one batch entry per line. |
| `artifacts/inputs/`, `artifacts/outputs/`, `artifacts/runner/` | Inputs, reference outputs, and the runner shared by latency and throughput validation. |
| `artifacts/raptor/pim/`, `artifacts/simulation/latency/` | Latency Pim artifacts and functional simulator outputs. |
| `artifacts/raptor/throughput/pim/`, `artifacts/simulation/throughput/` | Pipeline-4, batch-4 throughput Pim artifacts and functional simulator outputs. |
| `artifacts/common/inputs/` | Shared generated input CSV files. |
| `artifacts/common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
| `artifacts/common/runner/` | Shared reference runner source, build tree, and library. |
| `artifacts/<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and Pim artifacts. |
| `artifacts/<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
| `artifacts/<arch>/<mode>[/pipelineN]/pimcomp/` | Pimcomp graph, instruction, simulator, and comparison-report artifacts. |
| `artifacts/inputs.csv` | Input batch manifest. |
| `artifacts/{inputs,outputs,runner}/` | Inputs, native reference outputs, and runner. |
| `artifacts/raptor/pim/` | Latency compiler artifacts. |
| `artifacts/raptor/throughput/pim/` | Pipeline-4 throughput artifacts. |
| `artifacts/simulation/{latency,throughput}/` | Functional simulator outputs. |
Each comparison's `raptor/` directory may include `spatial0.mlir`,
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.
The [comparison runner](tools/pim/pimcomp/compare/README.md) has separate
architecture/pipeline directories and shared `artifacts/common/` references.
Remove these artifacts for every discovered model with:
- [Operation cases and generation](operations/README.md)
- [Pimcomp models, provenance, and hardware](networks/pimcomp_models/README.md)
- [Pimcomp correctness experiments](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation](tools/pim/ablation/README.md)
```bash
.venv/bin/python validation/validate.py \
--operations-dir validation/operations/gemm/small \
--clean
```
`--clean` does not require `--raptor-path` or `--onnx-include-dir`.
## Available suites
Checked-in validation networks under `validation/networks/` include `vgg16`,
`yolo11n`, and `yolo11nv2`.
The generated operation inventory is documented in
[`operations/README.md`](operations/README.md). Regenerate its models with:
```bash
.venv/bin/python validation/operations/gen_tests.py
```
Other network suites under `validation/networks/` include `vgg16`, `yolo11n`,
and `yolo11nv2`.
## Manual functional simulator tracing
After validation has produced an `artifacts/raptor/pim/` directory, rerun the functional
simulator with tracing from its crate directory:
After validation produces Pim artifacts, run from the simulator crate:
```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
--package pim-simulator --bin pim-simulator -- \
-f /path/to/workspace/artifacts/raptor/pim \
-o /path/to/workspace/artifacts/simulation/out.bin \
-f /path/to/artifacts/raptor/pim \
-o /path/to/artifacts/simulation/out.bin \
-d <addr0>,<size0>,<addr1>,<size1>,... \
--mode latency \
--batch-size 1 \
--input-dir /path/to/workspace/artifacts/simulation/inputs
--mode latency --batch-size 1 \
--input-dir /path/to/artifacts/simulation/inputs
```
Throughput mode additionally requires `--batch-size N` and at least `N`
`input_<index>.bin` files in `--input-dir`. Each input binary concatenates the model tensors in graph
input order. The comparison validator also writes one native reference and one
`simulation/*_iterations/output_*.bin` dump per batch entry, and checks every
entry rather than only the final output.
Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
validator normally derives the `-d` address and byte ranges from
`raptor/pim/config.json` and the model output shapes.
## Results and exit status
The final table reports latency and throughput functional pass/fail state plus
non-functional latency, throughput, power, and energy. The summary includes pass/fail totals, non-functional simulation
counts, total measured latency, and average Pim pass timings when `--verbose`
is enabled.
- Exit status `0`: all discovered models passed, or cleanup completed.
- Exit status `1`: validation failed, the model directory was invalid, or no
`.onnx` models were found.
- Exit status `2`: command-line arguments were invalid or required arguments
were missing.
Derive output address/byte ranges from `raptor/pim/config.json` and model output
shapes. Traces appear as `TraceCore0`, `TraceCore1`, etc. beside `out.bin`.
Throughput needs `--mode throughput --batch-size N` and at least `N`
`input_<index>.bin` files, each concatenating tensors in graph-input order.
The comparison validator records and checks one output per batch entry.