Raptor/validation/README.md

Raptor Validation

validate.py validates every ONNX model below a selected directory. For each model it can:

1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile PIM artifacts with Raptor;
4. run the reference implementation and functional PIM simulator;
5. compare their outputs;
6. run pimsim-nn to report latency and power.

Run the script from the repository root with the repository Python environment.

Prerequisites

• A built Raptor compiler, normally build_release/Release/bin/onnx-mlir.
• ONNX-MLIR runtime headers, normally onnx-mlir/include.
• The numpy, onnx, and colorama packages installed in .venv.
• The Rust toolchain used by the functional simulator.
• The Rust functional simulator under backend-simulators/pim/pim-simulator, unless overridden.
• A built pimsim-nn under backend-simulators/pim/pimsim-nn/build, unless non-functional simulation is skipped or its path is overridden.

Basic usage

Validate the complete operation suite:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --verbose \
  --raptor-extra-arg=--pim-detect-communication-deadlock \
  --raptor-extra-arg=--pim-export-spatial-dataflow=none

Validate one operation category or case:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small

Validate a network or network slice:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/networks/yolo11n/depth_04

--operations-dir may point to any directory tree containing .onnx files. The script discovers them recursively.

Validation modes

The default mode performs the complete workflow.

Use --compile-only to build the reference runner and PIM artifacts without executing either implementation:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --compile-only

Use --run-only to reuse those artifacts and perform input generation, reference execution, simulation, and comparison:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --run-only

--compile-only and --run-only are mutually exclusive. Run-only mode fails with a diagnostic if its required compiled artifacts are missing.

Parallel execution and output

Models run in parallel using all available CPUs by default. Set the worker count with -j or --jobs:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --jobs 8

Options

Option	Description
-h, --help	Print command help and exit.
--raptor-path PATH	Raptor compiler binary. Required unless --clean is used.
--onnx-include-dir PATH	ONNX-MLIR runtime include directory. Required unless --clean is used.
--operations-dir PATH	Directory tree containing models. Defaults to validation/operations.
--simulator-dir PATH	Functional pim-simulator crate directory. Defaults to the in-tree simulator.
--non-functional-simulator-build-dir PATH	pimsim-nn build directory. Defaults to the in-tree build.
--pimcomp-config {arch-a,arch-b,arch-c}	Non-functional hardware/timing profile. Defaults to arch-a.
--skip-non-functional-simulation	Skip pimsim-nn latency and power measurement.
--threshold FLOAT	Absolute output-comparison tolerance. Defaults to 1e-3.
--relative-threshold FLOAT	Relative output-comparison tolerance. Defaults to 1e-5.
--seed INT	Seed for generated inputs. Defaults to 0.
--crossbar-size INT	Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, 128.
--crossbar-count INT	Crossbars per core passed to Raptor. Defaults to the Arch-A value, 96.
--core-count INT	PIM core count passed to Raptor. Defaults to the Arch-A value, 168.
--raptor-extra-arg=ARG	Additional Raptor compiler argument. Repeat for multiple arguments.
--command-timeout-seconds FLOAT	Timeout for each compiler, runner, and simulator subprocess. Defaults to 1000000.0.
-j INT, --jobs INT	Parallel validation workers. Defaults to all available CPUs and must be at least one.
--clean	Remove generated validation artifacts and exit.
--compile-only	Compile reference and PIM artifacts without execution or comparison.
--run-only	Reuse compiled artifacts and perform execution, simulation, and comparison.
--verbose	Print passing per-stage and subprocess logs, plus average PIM pass timings.

Arguments beginning with -- that are passed through to Raptor should use the equals form:

--raptor-extra-arg=--pim-detect-communication-deadlock

Hardware profiles and non-functional simulation

The selected PIMCOMP profile must match --core-count, --crossbar-count, and --crossbar-size. A mismatch disables only non-functional simulation and prints the incompatible values; functional validation still runs.

The checked-in profiles are under validation/pimsim_configs/pimcomp/<profile>/latency_config.json.

Use --skip-non-functional-simulation when latency and power are not required. The summary reports non-functional results as measured, failed, unsupported, or skipped.

Overall PASS/FAIL is determined by compilation and functional output comparison. A non-functional simulation failure remains visible as ERROR in the latency and power columns but does not change a functional PASS.

pimsim-nn does not currently implement the vsoftmax instruction. When its explicit unsupported-op diagnostic is encountered, Softmax validations retain their functional PASS and show UNSUPPORTED in both non-functional columns. Other pimsim-nn failures remain ERROR.

Generated artifacts

Artifacts are written beside each model:

Path	Contents
inputs/	Generated input CSV files.
outputs/	ONNX-MLIR reference output CSV files.
raptor/	Exported MLIR, dialect snapshots, reports, and final pim/ artifacts.
runner/	Generated reference runner source, build tree, and shared library.
simulation/out.bin	Functional simulator output used for comparison.

The raptor/ directory may include spatial0.mlir, spatial1_graph.mlir, spatial2_trivial_merged.mlir, spatial3_scheduled_no_comm.mlir, spatial4_scheduled.mlir, pim0.mlir, pim1_buff.mlir, pim2_folded.mlir, and pim3_memory_planned.mlir.

Remove these artifacts for every discovered model with:

.venv/bin/python validation/validate.py \
  --operations-dir validation/operations/gemm/small \
  --clean

--clean does not require --raptor-path or --onnx-include-dir.

Available suites

Checked-in validation networks under validation/networks/ include vgg16, yolo11n, and yolo11nv2.

The generated operation inventory is documented in operations/README.md. Regenerate its models with:

.venv/bin/python validation/operations/gen_tests.py

Manual functional simulator tracing

After validation has produced a raptor/pim/ directory, rerun the functional simulator with tracing from its crate directory:

cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/workspace/raptor/pim \
  -o /path/to/workspace/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,...

Tracing writes TraceCore0, TraceCore1, and so on beside out.bin. The validator normally derives the -d address and byte ranges from raptor/pim/config.json and the model output shapes.

Results and exit status

The final table reports functional pass/fail state and non-functional latency and power. The summary includes pass/fail totals, non-functional simulation counts, total measured latency, and average PIM pass timings when --verbose is enabled.

• Exit status 0: all discovered models passed, or cleanup completed.
• Exit status 1: validation failed, the model directory was invalid, or no .onnx models were found.
• Exit status 2: command-line arguments were invalid or required arguments were missing.