Raptor/validation

Raptor validation

validate.py discovers ONNX models recursively, compiles a native ONNX-MLIR reference and Raptor artifacts, compares functional simulator outputs, and optionally measures Pimsim latency, throughput, power, and energy. Run from the repository root using .venv.

Prerequisites

• Built release Raptor and ONNX-MLIR runtime headers.
• numpy, onnx, and colorama installed in the repository environment.
• Rust toolchain and backend-simulators/pim/pim-simulator.
• Built backend-simulators/pim/pimsim-nn/build, unless performance simulation is skipped.

See the build guide.

Usage

.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --raptor-extra-arg=--pim-detect-communication-deadlock

Run the complete operations suite outside the sandbox so parallel workers can start. For a focused case, change --operations-dir to validation/operations/gemm/small or a network directory such as validation/networks/yolo11n/depth_04.

Defaults run latency (one input) and throughput (pipeline 4, four distinct inputs). Both reuse reference artifacts, and every throughput output is compared with its own reference. Models run with four workers by default. Results go to validation_results.csv in the selected model tree.

Append --compile-only to build artifacts without execution, then --run-only to reuse them for input generation, execution, and comparison. These flags are mutually exclusive; run-only fails if compiled artifacts are missing. --clean removes generated artifacts and stale locks, and requires neither compiler nor include paths.

Options

Use --help for the full command reference.

Option	Purpose / default
--raptor-path PATH	Compiler binary; required except for cleanup.
--onnx-include-dir PATH	Runtime headers; required except for cleanup.
--operations-dir PATH	Model tree; default: validation/operations.
--simulator-dir PATH	Override the in-tree functional simulator crate.
--non-functional-simulator-build-dir PATH	Override the in-tree Pimsim build.
--pimcomp-config {arch-a,arch-b,arch-c}	Performance profile; default: Arch-A.
--core-count N, --crossbar-count N, --crossbar-size N	Raptor resources; defaults: Arch-A (168, 96, 128).
--skip-non-functional-simulation	Skip performance measurement.
--no-fast	Use full-duration throughput measurement.
--threshold FLOAT, --relative-threshold FLOAT	Output tolerances; defaults: 1e-3, 1e-5.
--seed N	Input RNG seed; default: 0.
--raptor-extra-arg=ARG	Forward compiler arguments; repeat as needed.
--command-timeout-seconds FLOAT	Subprocess timeout; default: 1000000 seconds.
-j N, --jobs N	Parallel workers; default: 4.
--compile-only, --run-only, --clean	Compile, reuse, or remove artifacts.
--verbose	Include passing subprocess logs and average Pim pass timings.

Forwarded options beginning with -- require the equals form, for example --raptor-extra-arg=--pim-detect-communication-deadlock.

Performance and result interpretation

The selected hardware profile must match the core/crossbar flags. A mismatch disables performance simulation and prints the incompatible values; functional validation still runs. Canonical configs and meshes live under validation/pimsim_configs/pimcomp/.

Throughput defaults to pimsim-nn --fast with a 1000 ms convergence deadline. It compares consecutive two-round windows at 1% tolerance and falls back to full-duration results if convergence fails. Use --no-fast for authoritative experiments. See metric definitions for measurement windows, power, and energy interpretation.

Functional compilation/output comparison determines PASS/FAIL. Performance failures show ERROR without changing a functional PASS; explicit unsupported operations show UNSUPPORTED. Pimsim's vsoftmax is unsupported. Skipped measurements remain visibly skipped.

Exit codes: 0 for all models passing or successful cleanup; 1 for validation failure, an invalid model directory, or no ONNX files; 2 for invalid arguments.

Artifacts and suites

The regular validator uses these paths beside each model:

Path	Contents
artifacts/inputs.csv	Input batch manifest.
artifacts/{inputs,outputs,runner}/	Inputs, native reference outputs, and runner.
artifacts/raptor/pim/	Latency compiler artifacts.
artifacts/raptor/throughput/pim/	Pipeline-4 throughput artifacts.
artifacts/simulation/{latency,throughput}/	Functional simulator outputs.

The comparison runner has separate architecture/pipeline directories and shared artifacts/common/ references.

• Operation cases and generation
• Pimcomp models, provenance, and hardware
• Pimcomp correctness experiments
• Raptor compiler ablation

Other network suites under validation/networks/ include vgg16, yolo11n, and yolo11nv2.

Manual functional simulator tracing

After validation produces Pim artifacts, run from the simulator crate:

cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/artifacts/raptor/pim \
  -o /path/to/artifacts/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,... \
  --mode latency --batch-size 1 \
  --input-dir /path/to/artifacts/simulation/inputs

Derive output address/byte ranges from raptor/pim/config.json and model output shapes. Traces appear as TraceCore0, TraceCore1, etc. beside out.bin. Throughput needs --mode throughput --batch-size N and at least N input_<index>.bin files, each concatenating tensors in graph-input order. The comparison validator records and checks one output per batch entry.