# Raptor Validation

`validate.py` validates every ONNX model below a selected directory. For each
model it can:

1. compile an ONNX-MLIR reference library and runner;
2. generate deterministic random inputs;
3. compile PIM artifacts with Raptor;
4. run the reference implementation and functional PIM simulator;
5. compare their outputs;
6. run `pimsim-nn` to report latency, power, and energy.

Run the script from the repository root with the repository Python environment.

## Prerequisites

- A built Raptor compiler, normally
  `build_release/Release/bin/onnx-mlir`.
- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
- The Rust toolchain used by the functional simulator.
- The Rust functional simulator under
  `backend-simulators/pim/pim-simulator`, unless overridden.
- A built `pimsim-nn` under
  `backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
  skipped or its path is overridden.

## Basic usage

Validate the complete operation suite:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --verbose \
  --raptor-extra-arg=--pim-detect-communication-deadlock \
  --raptor-extra-arg=--pim-export-spatial-dataflow=none
```

Validate one operation category or case:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small
```

Validate a network or network slice:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/networks/yolo11n/depth_04
```

`--operations-dir` may point to any directory tree containing `.onnx` files.
The script discovers them recursively and writes `validation_results.csv` in
that directory while retaining the terminal table.

## Raptor vs PIMCOMP comparison

The PIMCOMP paper-model suite has a one-command multi-architecture comparison:

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
```

The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, and compares
the four paper models in parallel. Set `--jobs` to control comparison workers;
the runner leaves `OMP_NUM_THREADS` at its environment default. Use
`--models vgg8` for one model or `--dry-run` to print the commands.

Generated artifacts are stored beside each model and ignored by Git. The
comparison reuses model-level `common/inputs/`, `common/outputs/`,
`common/runner/` across architectures, modes, and pipelines. Pimsim configs and
network meshes are referenced directly from
`validation/pimsim_configs/pimcomp/`. Architecture- and pipeline-specific `raptor/`, `simulation/`, and
PIMCOMP artifacts stay under each comparison directory. See
[`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for
profiles, model provenance, limitations, and remote execution.

## Validation modes

The default mode performs the complete workflow.

Use `--compile-only` to build the reference runner and PIM artifacts without
executing either implementation:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --compile-only
```

Use `--run-only` to reuse those artifacts and perform input generation,
reference execution, simulation, and comparison:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations/gemm/small \
  --run-only
```

`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
with a diagnostic if its required compiled artifacts are missing.

## Parallel execution and output

Models run in parallel using all available CPUs by default. Set the worker
count with `-j` or `--jobs`:

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --jobs 8
```

## Options

| Option | Description |
|---|---|
| `-h`, `--help` | Print command help and exit. |
| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
| `--non-functional-simulator-build-dir PATH` | `pimsim-nn` build directory. Defaults to the in-tree build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency, power, and energy measurement. |
| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
| `--core-count INT` | PIM core count passed to Raptor. Defaults to the Arch-A value, `168`. |
| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
| `--clean` | Remove generated validation artifacts and exit. |
| `--compile-only` | Compile reference and PIM artifacts without execution or comparison. |
| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
| `--verbose` | Print passing per-stage and subprocess logs, plus average PIM pass timings. |

Arguments beginning with `--` that are passed through to Raptor should use the
equals form:

```bash
--raptor-extra-arg=--pim-detect-communication-deadlock
```

## Hardware profiles and non-functional simulation

The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
`--crossbar-size`. A mismatch disables only non-functional simulation and
prints the incompatible values; functional validation still runs.

The checked-in profiles are under
`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
`latency_config.json`, while throughput uses
`throughput_config_<time>ms.json` and the mesh beside it.

Use `--skip-non-functional-simulation` when latency, power, and energy are not required.
The summary reports non-functional results as measured, failed, unsupported, or
skipped.

Overall PASS/FAIL is determined by compilation and functional output
comparison. A non-functional simulation failure remains visible as `ERROR` in
the latency, power, and energy columns but does not change a functional PASS.

`pimsim-nn` does not currently implement the `vsoftmax` instruction. When its
explicit unsupported-op diagnostic is encountered, Softmax validations retain
their functional PASS and show `UNSUPPORTED` in the non-functional columns.
Other `pimsim-nn` failures remain `ERROR`.

## Generated artifacts

Artifacts are written beside each model:

| Path | Contents |
|---|---|
| `common/inputs/` | Shared generated input CSV files. |
| `common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
| `common/runner/` | Shared reference runner source, build tree, and library. |
| `<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and PIM artifacts. |
| `<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
| `<arch>/<mode>[/pipelineN]/pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |

Each comparison's `raptor/` directory may include `spatial0.mlir`,
`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.

Remove these artifacts for every discovered model with:

```bash
.venv/bin/python validation/validate.py \
  --operations-dir validation/operations/gemm/small \
  --clean
```

`--clean` does not require `--raptor-path` or `--onnx-include-dir`.

## Available suites

Checked-in validation networks under `validation/networks/` include `vgg16`,
`yolo11n`, and `yolo11nv2`.

The generated operation inventory is documented in
[`operations/README.md`](operations/README.md). Regenerate its models with:

```bash
.venv/bin/python validation/operations/gen_tests.py
```

## Manual functional simulator tracing

After validation has produced a `raptor/pim/` directory, rerun the functional
simulator with tracing from its crate directory:

```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/workspace/raptor/pim \
  -o /path/to/workspace/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,...
```

Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
validator normally derives the `-d` address and byte ranges from
`raptor/pim/config.json` and the model output shapes.

## Results and exit status

The final table reports functional pass/fail state and non-functional latency
and power. The summary includes pass/fail totals, non-functional simulation
counts, total measured latency, and average PIM pass timings when `--verbose`
is enabled.

- Exit status `0`: all discovered models passed, or cleanup completed.
- Exit status `1`: validation failed, the model directory was invalid, or no
  `.onnx` models were found.
- Exit status `2`: command-line arguments were invalid or required arguments
  were missing.