# Raptor compiler ablation

Run the synchronization/Spatial-planning ablation from the repository root:

```bash
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
```

Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.

| Variant | Raptor options |
|---|---|
| `baseline` | None |
| `no-sync` | `--pim-disable-synchronization` |
| `no-plan` | `--pim-disable-spatial-planning` |
| `no-sync-no-plan` | Both options |

## Options and outputs

- `--variant NAME`: repeat to select variants; baseline is always included.
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
- `--dry-run`: print commands without modifying files.

Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
replaces the default latency/pipeline-4 case set. `--only` is rejected because
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.

Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
reference inputs, outputs, and runners are shared under `artifacts/common/`.
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
Normal comparison results are preserved.

By default, the combined [results_ablation.csv](results_ablation.csv) is written
beside this script and printed after the run. An explicit `--out-dir PATH`
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
Default model artifacts and per-variant summaries remain in the locations above.
The aggregate records case identifiers,
`functional_validation`, and signed percentage changes in latency, throughput,
power, and energy against the matching baseline case. Positive means higher;
baseline values are zero.

Disabling synchronization removes host-buffer lifetime and pipeline restart
guarantees, so those variants may fail numerical validation and exit nonzero.
Baseline and `no-plan` must pass. Percentages from failing variants are not
valid speedups; deadline averages do not prove steady-state convergence.

## Plots

```bash
.venv/bin/python validation/tools/pim/ablation/generate_plots.py
# For a custom experiment root:
.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
  --input /path/to/experiment/results_ablation.csv
```

Requires Matplotlib from the root `requirements.txt`. By default, reads the
canonical CSV beside this script. Plots go to
`plots/png/` and `plots/svg/` beside the selected CSV; override with
`--output-dir PATH`.
Single-variant comparison CSVs are not accepted.

The checked-in CSV is an intentional result dataset; plotting uses its current
contents without running benchmarks. Invalid reciprocal percentage pairs render
as `N/A`; individual legitimate extreme changes are preserved. See
[performance availability](../../../../docs/evaluation/methodology.md#functional-validity-and-performance-availability).

Plots separate modes and pipelines. Throughput-mode latency means output
interval. Failed validation is labeled `INVALID`, other non-PASS statuses keep
their names, and missing measurements show `N/A`. See the
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).