Files
Raptor/validation/tools/pim/ablation/README.md
T
NiccoloN 49d6fb42cf
Validate Operations / validate-operations (push) Has been cancelled
update plots
2026-10-26 12:18:36 +01:00

56 lines
2.6 KiB
Markdown

# Raptor compiler ablation
Run the synchronization/Spatial-planning ablation from the repository root:
```bash
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
```
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
| Variant | Raptor options |
|---|---|
| `baseline` | None |
| `no-sync` | `--pim-disable-synchronization` |
| `no-plan` | `--pim-disable-spatial-planning` |
| `no-sync-no-plan` | Both options |
## Options and outputs
- `--variant NAME`: repeat to select variants; baseline is always included.
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
- `--dry-run`: print commands without modifying files.
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
replaces the default latency/pipeline-4 case set. `--only` is rejected because
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
reference inputs, outputs, and runners are shared under `artifacts/common/`.
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
Normal comparison results are preserved.
By default, the combined [results_ablation.csv](results_ablation.csv) is written
beside this script and printed after the run. An explicit `--out-dir PATH`
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
Default model artifacts and per-variant summaries remain in the locations above.
The aggregate records case identifiers,
`functional_validation`, and signed percentage changes in latency, throughput,
power, and energy against the matching baseline case. Positive means higher;
baseline values are zero.
Disabling synchronization removes host-buffer lifetime and pipeline restart
guarantees, so those variants may fail numerical validation and exit nonzero.
Baseline and `no-plan` must pass. Percentages from failing variants are not
valid speedups; deadline averages do not prove steady-state convergence.
## Result data
The checked-in `results_ablation.csv` is an intentional result dataset for tabular
reporting. Ablation experiments and aggregation remain available through the
runner above. Percentages from failing variants are not valid speedups. See the
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).