A built Raptor compiler, normally build_release/Release/bin/onnx-mlir.
ONNX-MLIR runtime headers, normally onnx-mlir/include.
The numpy, onnx, and colorama packages installed in .venv.
The Rust toolchain used by the functional simulator.
The Rust functional simulator under backend-simulators/pim/pim-simulator, unless overridden.
A built pimsim-nn under backend-simulators/pim/pimsim-nn/build, unless non-functional simulation is skipped or its path is overridden.

Basic usage

Validate the complete operation suite:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --verbose \
  --raptor-extra-arg=--pim-detect-communication-deadlock \
  --raptor-extra-arg=--pim-export-spatial-dataflow=none

Validate one operation category or case:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small

Validate a network or network slice:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/networks/yolo11n/depth_04

--operations-dir may point to any directory tree containing .onnx files. The script discovers them recursively and writes validation_results.csv in that directory while retaining separate latency and throughput terminal tables.

Pim validation tools

Validation modes

The default mode runs latency and throughput in one validation job. Latency uses one input, while throughput uses --pipeline=4 with four distinct inputs. Both modes reuse the generated input batch, native runner, and reference outputs, and every throughput output is compared with its own reference.

Use --compile-only to build the reference runner and Pim artifacts without executing either implementation:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --compile-only

Use --run-only to reuse those artifacts and perform input generation, reference execution, simulation, and comparison:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --run-only

--compile-only and --run-only are mutually exclusive. Run-only mode fails with a diagnostic if its required compiled artifacts are missing.

Parallel execution and output

Models run in parallel using all available CPUs by default. Set the worker count with -j or --jobs:

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --jobs 8

Options

Option	Description
`-h`, `--help`	Print command help and exit.
`--raptor-path PATH`	Raptor compiler binary. Required unless `--clean` is used.
`--onnx-include-dir PATH`	ONNX-MLIR runtime include directory. Required unless `--clean` is used.
`--operations-dir PATH`	Directory tree containing models. Defaults to `validation/operations`.
`--simulator-dir PATH`	Functional `pim-simulator` crate directory. Defaults to the in-tree simulator.
`--non-functional-simulator-build-dir PATH`	Pimsim build directory. Defaults to the in-tree build.
`--pimcomp-config {arch-a,arch-b,arch-c}`	Non-functional hardware/timing profile. Defaults to `arch-a`.
`--skip-non-functional-simulation`	Skip Pimsim latency, throughput, power, and energy measurement.
`--no-fast`	Disable fast throughput convergence for authoritative full-duration Pimsim experiments.
`--threshold FLOAT`	Absolute output-comparison tolerance. Defaults to `1e-3`.
`--relative-threshold FLOAT`	Relative output-comparison tolerance. Defaults to `1e-5`.
`--seed INT`	Seed for generated inputs. Defaults to `0`.
`--crossbar-size INT`	Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`.
`--crossbar-count INT`	Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`.
`--core-count INT`	Pim core count passed to Raptor. Defaults to the Arch-A value, `168`.
`--raptor-extra-arg=ARG`	Additional Raptor compiler argument. Repeat for multiple arguments.
`--command-timeout-seconds FLOAT`	Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`.
`-j INT`, `--jobs INT`	Parallel validation workers. Defaults to all available CPUs and must be at least one.
`--clean`	Remove generated validation artifacts and exit.
`--compile-only`	Compile reference and Pim artifacts without execution or comparison.
`--run-only`	Reuse compiled artifacts and perform execution, simulation, and comparison.
`--verbose`	Print passing per-stage and subprocess logs, plus average Pim pass timings.

Arguments beginning with -- that are passed through to Raptor should use the equals form:

--raptor-extra-arg=--pim-detect-communication-deadlock

Hardware profiles and non-functional simulation

The selected Pimcomp profile must match --core-count, --crossbar-count, and --crossbar-size. A mismatch disables only non-functional simulation and prints the incompatible values; functional validation still runs.

The checked-in profiles are under validation/pimsim_configs/pimcomp/<profile>/; latency uses latency_config.json, while throughput uses throughput_config_<time>ms.json and the mesh beside it.

Throughput measurement defaults to pimsim-nn --fast with a 1000 ms convergence deadline. Fast mode compares consecutive two-round windows with a fixed 1% tolerance and falls back to the legacy full-duration result if it does not converge. Use --no-fast for authoritative experiments.

Use --skip-non-functional-simulation when latency, throughput, power, and energy are not required. The summary reports non-functional results as measured, failed, unsupported, or skipped.

Overall PASS/FAIL is determined by compilation and functional output comparison. A non-functional simulation failure remains visible as ERROR in the corresponding latency, throughput, power, or energy columns but does not change a functional PASS.

Pimsim does not currently implement the vsoftmax instruction. When its explicit unsupported-op diagnostic is encountered, Softmax validations retain their functional PASS and show UNSUPPORTED in the non-functional columns. Other Pimsim failures remain ERROR.

Generated artifacts

Generated files are grouped below an artifacts/ directory beside each model or operation case. This keeps checked-in ONNX files and generated trees separate and lets --clean remove the complete workspace, including stale validation lock files:

Path	Contents
`artifacts/inputs.csv`	Generated inputs, one batch entry per line.
`artifacts/inputs/`, `artifacts/outputs/`, `artifacts/runner/`	Inputs, reference outputs, and the runner shared by latency and throughput validation.
`artifacts/raptor/pim/`, `artifacts/simulation/latency/`	Latency Pim artifacts and functional simulator outputs.
`artifacts/raptor/throughput/pim/`, `artifacts/simulation/throughput/`	Pipeline-4, batch-4 throughput Pim artifacts and functional simulator outputs.
`artifacts/common/inputs/`	Shared generated input CSV files.
`artifacts/common/outputs/`	Shared ONNX-MLIR reference output CSV files.
`artifacts/common/runner/`	Shared reference runner source, build tree, and library.
`artifacts/<arch>/<mode>[/pipelineN]/raptor/`	Architecture- and pipeline-specific Raptor MLIR and Pim artifacts.
`artifacts/<arch>/<mode>[/pipelineN]/simulation/`	Functional simulator outputs for that comparison.
`artifacts/<arch>/<mode>[/pipelineN]/pimcomp/`	Pimcomp graph, instruction, simulator, and comparison-report artifacts.

Each comparison's raptor/ directory may include spatial0.mlir, spatial1_graph.mlir, spatial2_trivial_merged.mlir, spatial3_scheduled_no_comm.mlir, spatial4_scheduled.mlir, pim0.mlir, pim1_buff.mlir, pim2_folded.mlir, and pim3_memory_planned.mlir.

Remove these artifacts for every discovered model with:

.venv/bin/python validation/validate.py \
  --operations-dir validation/operations/gemm/small \
  --clean

--clean does not require --raptor-path or --onnx-include-dir.

Available suites

Checked-in validation networks under validation/networks/ include vgg16, yolo11n, and yolo11nv2.

The generated operation inventory is documented in operations/README.md. Regenerate its models with:

.venv/bin/python validation/operations/gen_tests.py

Manual functional simulator tracing

After validation has produced an artifacts/raptor/pim/ directory, rerun the functional simulator with tracing from its crate directory:

cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/workspace/artifacts/raptor/pim \
  -o /path/to/workspace/artifacts/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,... \
  --mode latency \
  --batch-size 1 \
  --input-dir /path/to/workspace/artifacts/simulation/inputs

Throughput mode additionally requires --batch-size N and at least N input_<index>.bin files in --input-dir. Each input binary concatenates the model tensors in graph input order. The comparison validator also writes one native reference and one simulation/*_iterations/output_*.bin dump per batch entry, and checks every entry rather than only the final output.

Tracing writes TraceCore0, TraceCore1, and so on beside out.bin. The validator normally derives the -d address and byte ranges from raptor/pim/config.json and the model output shapes.

Results and exit status

The final table reports latency and throughput functional pass/fail state plus non-functional latency, throughput, power, and energy. The summary includes pass/fail totals, non-functional simulation counts, total measured latency, and average Pim pass timings when --verbose is enabled.

Exit status 0: all discovered models passed, or cleanup completed.
Exit status 1: validation failed, the model directory was invalid, or no .onnx models were found.
Exit status 2: command-line arguments were invalid or required arguments were missing.