A built Raptor compiler, normally build_release/Release/bin/onnx-mlir.
ONNX-MLIR runtime headers, normally onnx-mlir/include.
The numpy, onnx, and colorama packages installed in .venv.
The Rust toolchain used by the functional simulator.
The Rust functional simulator under backend-simulators/pim/pim-simulator, unless overridden.
A built pimsim-nn under backend-simulators/pim/pimsim-nn/build, unless non-functional simulation is skipped or its path is overridden.

Basic usage

Validate the complete operation suite:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --verbose \
  --raptor-extra-arg=--pim-detect-communication-deadlock \
  --raptor-extra-arg=--pim-export-spatial-dataflow=none

Validate one operation category or case:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small

Validate a network or network slice:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/networks/yolo11n/depth_04

--operations-dir may point to any directory tree containing .onnx files. The script discovers them recursively.

Validation modes

The default mode performs the complete workflow.

Use --compile-only to build the reference runner and PIM artifacts without executing either implementation:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --compile-only

Use --run-only to reuse those artifacts and perform input generation, reference execution, simulation, and comparison:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --run-only

--compile-only and --run-only are mutually exclusive. Run-only mode fails with a diagnostic if its required compiled artifacts are missing.

Parallel execution and output

Models run in parallel using all available CPUs by default. Set the worker count with -j or --jobs:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --jobs 8

Each model's output is buffered as one readable block. Independent workspace jobs print their blocks when they finish, so their order may differ from the final table. The final result table is sorted by model path.

Models in the same directory run sequentially because they share generated workspace paths.

Options

Option	Description
`-h`, `--help`	Print command help and exit.
`--raptor-path PATH`	Raptor compiler binary. Required unless `--clean` is used.
`--onnx-include-dir PATH`	ONNX-MLIR runtime include directory. Required unless `--clean` is used.
`--operations-dir PATH`	Directory tree containing models. Defaults to `validation/operations`.
`--simulator-dir PATH`	Functional `pim-simulator` crate directory. Defaults to the in-tree simulator.
`--non-functional-simulator-build-dir PATH`	`pimsim-nn` build directory. Defaults to the in-tree build.
`--pimcomp-config {arch-a,arch-b,arch-c}`	Non-functional hardware/timing profile. Defaults to `arch-a`.
`--skip-non-functional-simulation`	Skip `pimsim-nn` latency and power measurement.
`--threshold FLOAT`	Absolute output-comparison tolerance. Defaults to `1e-3`.
`--relative-threshold FLOAT`	Relative output-comparison tolerance. Defaults to `1e-5`.
`--seed INT`	Seed for generated inputs. Defaults to `0`.
`--crossbar-size INT`	Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`.
`--crossbar-count INT`	Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`.
`--core-count INT`	PIM core count passed to Raptor. Defaults to the Arch-A value, `168`.
`--raptor-extra-arg=ARG`	Additional Raptor compiler argument. Repeat for multiple arguments.
`--command-timeout-seconds FLOAT`	Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`.
`-j INT`, `--jobs INT`	Parallel validation workers. Defaults to all available CPUs and must be at least one.
`--clean`	Remove generated validation artifacts and exit.
`--compile-only`	Compile reference and PIM artifacts without execution or comparison.
`--run-only`	Reuse compiled artifacts and perform execution, simulation, and comparison.
`--verbose`	Print passing per-stage and subprocess logs, plus average PIM pass timings.

Arguments beginning with -- that are passed through to Raptor should use the equals form:

--raptor-extra-arg=--pim-detect-communication-deadlock

Hardware profiles and non-functional simulation

The selected PIMCOMP profile must match --core-count, --crossbar-count, and --crossbar-size. A mismatch disables only non-functional simulation and prints the incompatible values; functional validation still runs.

The checked-in profiles are under validation/pimsim_configs/pimcomp/<profile>/latency_config.json.

Use --skip-non-functional-simulation when latency and power are not required. The summary reports non-functional results as measured, failed, unsupported, or skipped.

Overall PASS/FAIL is determined by compilation and functional output comparison. A non-functional simulation failure remains visible as ERROR in the latency and power columns but does not change a functional PASS.

pimsim-nn does not currently implement the vsoftmax instruction. When its explicit unsupported-op diagnostic is encountered, Softmax validations retain their functional PASS and show UNSUPPORTED in both non-functional columns. Other pimsim-nn failures remain ERROR.

Generated artifacts

Artifacts are written beside each model:

Path	Contents
`inputs/`	Generated input CSV files.
`outputs/`	ONNX-MLIR reference output CSV files.
`raptor/`	Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts.
`runner/`	Generated reference runner source, build tree, and shared library.
`simulation/out.bin`	Functional simulator output used for comparison.

The raptor/ directory may include spatial0.mlir, spatial1_graph.mlir, spatial2_trivial_merged.mlir, spatial3_scheduled_no_comm.mlir, spatial4_scheduled.mlir, pim0.mlir, pim1_buff.mlir, pim2_folded.mlir, and pim3_memory_planned.mlir.

Remove these artifacts for every discovered model with:

.venv/bin/python validation/validate.py \
  --operations-dir validation/operations/gemm/small \
  --clean

--clean does not require --raptor-path or --onnx-include-dir.

Available suites

Checked-in validation networks under validation/networks/ include vgg16, yolo11n, and yolo11nv2.

The generated operation inventory is documented in operations/README.md. Regenerate its models with:

.venv/bin/python validation/operations/gen_tests.py

Manual functional simulator tracing

After validation has produced a raptor/pim/ directory, rerun the functional simulator with tracing from its crate directory:

cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/workspace/raptor/pim \
  -o /path/to/workspace/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,...

Tracing writes TraceCore0, TraceCore1, and so on beside out.bin. The validator normally derives the -d address and byte ranges from raptor/pim/config.json and the model output shapes.

Results and exit status

The final table reports functional pass/fail state and non-functional latency and power. The summary includes pass/fail totals, non-functional simulation counts, total measured latency, and average PIM pass timings when --verbose is enabled.

Exit status 0: all discovered models passed, or cleanup completed.
Exit status 1: validation failed, the model directory was invalid, or no .onnx models were found.
Exit status 2: command-line arguments were invalid or required arguments were missing.