Raptor/validation/README.md

# Raptor validation

`validate.py` discovers ONNX models recursively, compiles a native ONNX-MLIR
reference and Raptor artifacts, compares functional simulator outputs, and
optionally measures Pimsim latency, throughput, power, and energy.
Run from the repository root using `.venv`.

## Prerequisites

- Built release Raptor and ONNX-MLIR runtime headers.
- `numpy`, `onnx`, and `colorama` installed in the repository environment.
- Rust toolchain and `backend-simulators/pim/pim-simulator`.
- Built `backend-simulators/pim/pimsim-nn/build`, unless performance
  simulation is skipped.

See the [build guide](../README.md#build).

## Usage

```bash
.venv/bin/python validation/validate.py \
  --raptor-path build_release/Release/bin/onnx-mlir \
  --onnx-include-dir onnx-mlir/include \
  --operations-dir validation/operations \
  --raptor-extra-arg=--pim-detect-communication-deadlock
```

Run the complete operations suite outside the sandbox so parallel workers can
start. For a focused case, change `--operations-dir` to
`validation/operations/gemm/small` or a network directory such as
`validation/networks/yolo11n/depth_04`.

Defaults run latency (one input) and throughput (pipeline 4, four distinct
inputs). Both reuse reference artifacts, and every throughput output is
compared with its own reference. Models run with four workers by default.
Results go to `validation_results.csv` in the selected model tree.

Append `--compile-only` to build artifacts without execution, then
`--run-only` to reuse them for input generation, execution, and comparison.
These flags are mutually exclusive; run-only fails if compiled artifacts are
missing. `--clean` removes generated artifacts and stale locks, and requires
neither compiler nor include paths.

## Options

Use `--help` for the full command reference.

| Option | Purpose / default |
|---|---|
| `--raptor-path PATH` | Compiler binary; required except for cleanup. |
| `--onnx-include-dir PATH` | Runtime headers; required except for cleanup. |
| `--operations-dir PATH` | Model tree; default: `validation/operations`. |
| `--simulator-dir PATH` | Override the in-tree functional simulator crate. |
| `--non-functional-simulator-build-dir PATH` | Override the in-tree Pimsim build. |
| `--pimcomp-config {arch-a,arch-b,arch-c}` | Performance profile; default: Arch-A. |
| `--core-count N`, `--crossbar-count N`, `--crossbar-size N` | Raptor resources; defaults: Arch-A (168, 96, 128). |
| `--skip-non-functional-simulation` | Skip performance measurement. |
| `--no-fast` | Use full-duration throughput measurement. |
| `--threshold FLOAT`, `--relative-threshold FLOAT` | Output tolerances; defaults: `1e-3`, `1e-5`. |
| `--seed N` | Input RNG seed; default: 0. |
| `--raptor-extra-arg=ARG` | Forward compiler arguments; repeat as needed. |
| `--command-timeout-seconds FLOAT` | Subprocess timeout; default: 1000000 seconds. |
| `-j N`, `--jobs N` | Parallel workers; default: 4. |
| `--compile-only`, `--run-only`, `--clean` | Compile, reuse, or remove artifacts. |
| `--verbose` | Include passing subprocess logs and average Pim pass timings. |

Forwarded options beginning with `--` require the equals form, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.

## Performance and result interpretation

The selected [hardware profile](networks/pimcomp_models/README.md#hardware-profiles)
must match the core/crossbar flags. A mismatch disables performance simulation
and prints the incompatible values; functional validation still runs.
Canonical configs and meshes live under `validation/pimsim_configs/pimcomp/`.

Throughput defaults to `pimsim-nn --fast` with a 1000 ms convergence deadline.
It compares consecutive two-round windows at 1% tolerance and falls back to
full-duration results if convergence fails. Use `--no-fast` for authoritative
experiments. See [metric definitions](../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, power, and energy interpretation.

Functional compilation/output comparison determines PASS/FAIL. Performance
failures show `ERROR` without changing a functional PASS; explicit unsupported
operations show `UNSUPPORTED`. Pimsim-nn omits `vsoftmax` timing and energy;
the functional simulator still executes Softmax.
Skipped measurements remain visibly skipped.

Floating-point validation requires both checks for each output:

- Every element satisfies `abs(actual - reference) <= threshold + relative_threshold * abs(reference)`.
- For a nonzero reference, `norm(actual - reference) / norm(reference) <= relative_threshold`, using the L2 norm.

The second check prevents a large absolute tolerance from accepting a lost small output.
It does not replace or weaken the elementwise check. Set `--relative-threshold=0` for absolute-only floating-point comparison.
Zero references use the absolute check. Integer and Boolean outputs require exact agreement.
Nonfinite values and mismatched output shapes or counts fail validation.
Reports show maximum absolute differences and relative L2 errors.

Exit codes: `0` for all models passing or successful cleanup; `1` for validation
failure, an invalid model directory, or no ONNX files; `2` for invalid arguments.

## Artifacts and suites

The regular validator uses these paths beside each model:

| Path | Contents |
|---|---|
| `artifacts/inputs.csv` | Input batch manifest. |
| `artifacts/{inputs,outputs,runner}/` | Inputs, native reference outputs, and runner. |
| `artifacts/raptor/pim/` | Latency compiler artifacts. |
| `artifacts/raptor/throughput/pim/` | Pipeline-4 throughput artifacts. |
| `artifacts/simulation/{latency,throughput}/` | Functional simulator outputs. |

The [comparison runner](tools/pim/pimcomp/compare/README.md) has separate
architecture/pipeline directories and shared `artifacts/common/` references.

- [Operation cases and generation](operations/README.md)
- [Pimcomp models, provenance, and hardware](networks/pimcomp_models/README.md)
- [Pimcomp correctness experiments](tools/pim/pimcomp/correctness/README.md)
- [Raptor compiler ablation](tools/pim/ablation/README.md)

Other network suites under `validation/networks/` include `vgg16`, `yolo11n`,
and `yolo11nv2`.

## Manual functional simulator tracing

After validation produces Pim artifacts, run from the simulator crate:

```bash
cd backend-simulators/pim/pim-simulator
cargo run --no-default-features --features tracing --release \
  --package pim-simulator --bin pim-simulator -- \
  -f /path/to/artifacts/raptor/pim \
  -o /path/to/artifacts/simulation/out.bin \
  -d <addr0>,<size0>,<addr1>,<size1>,... \
  --mode latency --batch-size 1 \
  --input-dir /path/to/artifacts/simulation/inputs
```

Derive output address/byte ranges from `raptor/pim/config.json` and model output
shapes. Traces appear as `TraceCore0`, `TraceCore1`, etc. beside `out.bin`.
Throughput needs `--mode throughput --batch-size N` and at least `N`
`input_<index>.bin` files, each concatenating tensors in graph-input order.
The comparison validator records and checks one output per batch entry.