Raptor/validation/README.md

# Raptor Validation

`validate.py` validates every ONNX model below a selected directory. For each
model it can:

1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile PIM artifacts with Raptor;
4. run the reference implementation and functional PIM simulator;
5. compare their outputs;
6. run `pimsim-nn` to report latency and power.

Run the script from the repository root with the repository Python environment.

## Prerequisites

- A built Raptor compiler, normally
  `build_release/Release/bin/onnx-mlir`.
- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
- The Rust toolchain used by the functional simulator.
- The Rust functional simulator under
  `backend-simulators/pim/pim-simulator`, unless overridden.
- A built `pimsim-nn` under
  `backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
  skipped or its path is overridden.

## Basic usage

Validate the complete operation suite:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --verbose \
  --raptor-extra-arg=--pim-detect-communication-deadlock \
  --raptor-extra-arg=--pim-export-spatial-dataflow=none
```

Validate one operation category or case:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small
```

Validate a network or network slice:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/networks/yolo11n/depth_04
```

`--operations-dir` may point to any directory tree containing `.onnx` files.
The script discovers them recursively.

## Validation modes

The default mode performs the complete workflow.

Use `--compile-only` to build the reference runner and PIM artifacts without
executing either implementation:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --compile-only
```

Use `--run-only` to reuse those artifacts and perform input generation,
reference execution, simulation, and comparison:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --run-only
```

`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
with a diagnostic if its required compiled artifacts are missing.

## Parallel execution and output

Models run in parallel using all available CPUs by default. Set the worker
count with `-j` or `--jobs`:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --jobs 8
```

Each model's output is buffered as one readable block. Independent workspace
jobs print their blocks when they finish, so their order may differ from the
final table. The final result table is sorted by model path.

Models in the same directory run sequentially because they share generated
workspace paths.

## Options

| Option | Description |
|---|---|
| `-h`, `--help` | Print command help and exit. |
| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
| `--non-functional-simulator-build-dir PATH` | `pimsim-nn` build directory. Defaults to the in-tree build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency and power measurement. |
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
| `--core-count INT` | PIM core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
| `--clean` | Remove generated validation artifacts and exit. |
| `--compile-only` | Compile reference and PIM artifacts without execution or comparison. |
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average PIM pass timings. |

Arguments beginning with `--` that are passed through to Raptor should use the
equals form:

```bash
--raptor-extra-arg=--pim-detect-communication-deadlock
```

## Hardware profiles and non-functional simulation

The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
`--crossbar-size`. A mismatch disables only non-functional simulation and
prints the incompatible values; functional validation still runs.

The checked-in profiles are under
`validation/pimsim_configs/pimcomp/<profile>/latency_config.json`.

Use `--skip-non-functional-simulation` when latency and power are not required.
The summary reports non-functional results as measured, failed, unsupported, or
skipped.

Overall PASS/FAIL is determined by compilation and functional output
comparison. A non-functional simulation failure remains visible as `ERROR` in
the latency and power columns but does not change a functional PASS.

`pimsim-nn` does not currently implement the `vsoftmax` instruction. When its
explicit unsupported-op diagnostic is encountered, Softmax validations retain
their functional PASS and show `UNSUPPORTED` in both non-functional columns.
Other `pimsim-nn` failures remain `ERROR`.

## Generated artifacts

Artifacts are written beside each model:

| Path | Contents |
|---|---|
| `inputs/` | Generated input CSV files. |
| `outputs/` | ONNX-MLIR reference output CSV files. |
| `raptor/` | Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts. |
| `runner/` | Generated reference runner source, build tree, and shared library. |
| `simulation/out.bin` | Functional simulator output used for comparison. |

The `raptor/` directory may include `spatial0.mlir`,
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.

Remove these artifacts for every discovered model with:

```bash
.venv/bin/python validation/validate.py \
  --operations-dir validation/operations/gemm/small \
  --clean
```

`--clean` does not require `--raptor-path` or `--onnx-include-dir`.

## Available suites

Checked-in validation networks under `validation/networks/` include `vgg16`,
`yolo11n`, and `yolo11nv2`.

The generated operation inventory is documented in
[`operations/README.md`](operations/README.md). Regenerate its models with:

```bash
.venv/bin/python validation/operations/gen_tests.py
```

## Manual functional simulator tracing

After validation has produced a `raptor/pim/` directory, rerun the functional
simulator with tracing from its crate directory:

```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/workspace/raptor/pim \
  -o /path/to/workspace/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,...
```

Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
validator normally derives the `-d` address and byte ranges from
`raptor/pim/config.json` and the model output shapes.

## Results and exit status

The final table reports functional pass/fail state and non-functional latency
and power. The summary includes pass/fail totals, non-functional simulation
counts, total measured latency, and average PIM pass timings when `--verbose`
is enabled.

- Exit status `0`: all discovered models passed, or cleanup completed.
- Exit status `1`: validation failed, the model directory was invalid, or no
  `.onnx` models were found.
- Exit status `2`: command-line arguments were invalid or required arguments
  were missing.