nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add ablation study
Validate Operations / validate-operations (push) Has been cancelled
Details

Browse Source 
normalize names and artifact paths
This commit is contained in:
NiccoloN
2026-08-20 17:58:02 +02:00
parent add20e56eb
commit b009e1ff08
67 changed files with 1573 additions and 993 deletions
Download Patch File Download Diff File Expand all files Collapse all files
validation/README.md
+41 -54
View File
@@ -1,14 +1,14 @@
# Raptor Validation
# Raptor validation
`validate.py` validates every ONNX model below a selected directory. For each
model it can:
1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile PIM artifacts with Raptor;
4. run the reference implementation and functional PIM simulator;
3. compile Pim artifacts with Raptor;
4. run the reference implementation and functional Pim simulator;
5. compare their outputs;
6. run `pimsim-nn` to report latency, throughput, power, and energy.
6. run Pimsim to report latency, throughput, power, and energy.
Run the script from the repository root with the repository Python environment.
@@ -61,29 +61,12 @@ Validate a network or network slice:
The script discovers them recursively and writes `validation_results.csv` in
that directory while retaining separate latency and throughput terminal tables.
## Raptor vs PIMCOMP comparison
## Pim validation tools
The PIMCOMP paper-model suite has a one-command multi-architecture comparison:
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
```
The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
expects Raptor and the existing `third_party/PIMCOMP-NN/build` tree to already
be built, and compares the configured paper models in parallel. Set `--jobs` to
control comparison workers;
the runner leaves `OMP_NUM_THREADS` at its environment default. Use
`--models vgg8` for one model or `--dry-run` to print the commands.
Generated artifacts are stored beside each model and ignored by Git. The
comparison reuses model-level `common/inputs/`, `common/outputs/`,
`common/runner/` across architectures, modes, and pipelines. Pimsim configs and
network meshes are referenced directly from
`validation/pimsim_configs/pimcomp/`. Architecture- and pipeline-specific `raptor/`, `simulation/`, and
PIMCOMP artifacts stay under each comparison directory. See
[`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for
profiles, model provenance, limitations, and remote execution.
- [Pimcomp model suite](networks/pimcomp_models/README.md)
- [Pimcomp model comparison tools](tools/pim/pimcomp/compare/README.md)
- [Pimcomp correctness study](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation study](tools/pim/ablation/README.md)
## Validation modes
@@ -92,7 +75,7 @@ uses one input, while throughput uses `--pipeline=4` with four distinct inputs.
Both modes reuse the generated input batch, native runner, and reference
outputs, and every throughput output is compared with its own reference.
Use `--compile-only` to build the reference runner and PIM artifacts without
Use `--compile-only` to build the reference runner and Pim artifacts without
executing either implementation:
```bash
@@ -139,23 +122,23 @@ count with `-j` or `--jobs`:
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
| `--non-functional-simulator-build-dir PATH` | `pimsim-nn` build directory. Defaults to the in-tree build. |
| `--non-functional-simulator-build-dir PATH` | Pimsim build directory. Defaults to the in-tree build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency, throughput, power, and energy measurement. |
| `--no-fast` | Disable fast throughput convergence for authoritative full-duration `pimsim-nn` experiments. |
| `--skip-non-functional-simulation` | Skip Pimsim latency, throughput, power, and energy measurement. |
| `--no-fast` | Disable fast throughput convergence for authoritative full-duration Pimsim experiments. |
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
| `--core-count INT` | PIM core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--core-count INT` | Pim core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
| `--clean` | Remove generated validation artifacts and exit. |
| `--compile-only` | Compile reference and PIM artifacts without execution or comparison. |
| `--compile-only` | Compile reference and Pim artifacts without execution or comparison. |
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average PIM pass timings. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average Pim pass timings. |
Arguments beginning with `--` that are passed through to Raptor should use the
equals form:
@@ -166,7 +149,7 @@ equals form:
## Hardware profiles and non-functional simulation
The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
The selected Pimcomp profile must match `--core-count`, `--crossbar-count`, and
`--crossbar-size`. A mismatch disables only non-functional simulation and
prints the incompatible values; functional validation still runs.
@@ -189,27 +172,30 @@ comparison. A non-functional simulation failure remains visible as `ERROR` in
the corresponding latency, throughput, power, or energy columns but does not
change a functional PASS.
`pimsim-nn` does not currently implement the `vsoftmax` instruction. When its
Pimsim does not currently implement the `vsoftmax` instruction. When its
explicit unsupported-op diagnostic is encountered, Softmax validations retain
their functional PASS and show `UNSUPPORTED` in the non-functional columns.
Other `pimsim-nn` failures remain `ERROR`.
Other Pimsim failures remain `ERROR`.
## Generated artifacts
Artifacts are written beside each model:
Generated files are grouped below an `artifacts/` directory beside each model
or operation case. This keeps checked-in ONNX files and generated trees
separate and lets `--clean` remove the complete workspace, including stale
validation lock files:
| Path | Contents |
|---|---|
| `inputs.csv` | Generated inputs, one batch entry per line. |
| `inputs/`, `outputs/`, `runner/` | Inputs, reference outputs, and the runner shared by latency and throughput validation. |
| `raptor/pim/`, `simulation/latency/` | Latency PIM artifacts and functional simulator outputs. |
| `raptor/throughput/pim/`, `simulation/throughput/` | Pipeline-4, batch-4 throughput PIM artifacts and functional simulator outputs. |
| `common/inputs/` | Shared generated input CSV files. |
| `common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
| `common/runner/` | Shared reference runner source, build tree, and library. |
| `<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and PIM artifacts. |
| `<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
| `<arch>/<mode>[/pipelineN]/pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
| `artifacts/inputs.csv` | Generated inputs, one batch entry per line. |
| `artifacts/inputs/`, `artifacts/outputs/`, `artifacts/runner/` | Inputs, reference outputs, and the runner shared by latency and throughput validation. |
| `artifacts/raptor/pim/`, `artifacts/simulation/latency/` | Latency Pim artifacts and functional simulator outputs. |
| `artifacts/raptor/throughput/pim/`, `artifacts/simulation/throughput/` | Pipeline-4, batch-4 throughput Pim artifacts and functional simulator outputs. |
| `artifacts/common/inputs/` | Shared generated input CSV files. |
| `artifacts/common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
| `artifacts/common/runner/` | Shared reference runner source, build tree, and library. |
| `artifacts/<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and Pim artifacts. |
| `artifacts/<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
| `artifacts/<arch>/<mode>[/pipelineN]/pimcomp/` | Pimcomp graph, instruction, simulator, and comparison-report artifacts. |
Each comparison's `raptor/` directory may include `spatial0.mlir`,
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
@@ -240,22 +226,23 @@ The generated operation inventory is documented in
## Manual functional simulator tracing
After validation has produced a `raptor/pim/` directory, rerun the functional
After validation has produced an `artifacts/raptor/pim/` directory, rerun the functional
simulator with tracing from its crate directory:
```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
--package pim-simulator --bin pim-simulator -- \
-f /path/to/workspace/raptor/pim \
-o /path/to/workspace/simulation/out.bin \
-f /path/to/workspace/artifacts/raptor/pim \
-o /path/to/workspace/artifacts/simulation/out.bin \
-d <addr0>,<size0>,<addr1>,<size1>,... \
--mode latency \
--input /path/to/workspace/simulation/inputs/input_0.bin
--batch-size 1 \
--input-dir /path/to/workspace/artifacts/simulation/inputs
```
Throughput mode additionally requires `--batch-size N` and exactly `N`
`--input` arguments. Each input binary concatenates the model tensors in graph
Throughput mode additionally requires `--batch-size N` and at least `N`
`input_<index>.bin` files in `--input-dir`. Each input binary concatenates the model tensors in graph
input order. The comparison validator also writes one native reference and one
`simulation/*_iterations/output_*.bin` dump per batch entry, and checks every
entry rather than only the final output.
@@ -268,7 +255,7 @@ validator normally derives the `-d` address and byte ranges from
The final table reports latency and throughput functional pass/fail state plus
non-functional latency, throughput, power, and energy. The summary includes pass/fail totals, non-functional simulation
counts, total measured latency, and average PIM pass timings when `--verbose`
counts, total measured latency, and average Pim pass timings when `--verbose`
is enabled.
- Exit status `0`: all discovered models passed, or cleanup completed.