56 lines
2.6 KiB
Markdown
56 lines
2.6 KiB
Markdown
# Raptor compiler ablation
|
|
|
|
Run the synchronization/Spatial-planning ablation from the repository root:
|
|
|
|
```bash
|
|
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
|
|
```
|
|
|
|
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
|
|
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
|
|
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
|
|
|
|
| Variant | Raptor options |
|
|
|---|---|
|
|
| `baseline` | None |
|
|
| `no-sync` | `--pim-disable-synchronization` |
|
|
| `no-plan` | `--pim-disable-spatial-planning` |
|
|
| `no-sync-no-plan` | Both options |
|
|
|
|
## Options and outputs
|
|
|
|
- `--variant NAME`: repeat to select variants; baseline is always included.
|
|
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
|
|
- `--dry-run`: print commands without modifying files.
|
|
|
|
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
|
|
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
|
|
replaces the default latency/pipeline-4 case set. `--only` is rejected because
|
|
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
|
|
|
|
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
|
|
reference inputs, outputs, and runners are shared under `artifacts/common/`.
|
|
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
|
|
Normal comparison results are preserved.
|
|
|
|
By default, the combined [results_ablation.csv](results_ablation.csv) is written
|
|
beside this script and printed after the run. An explicit `--out-dir PATH`
|
|
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
|
|
Default model artifacts and per-variant summaries remain in the locations above.
|
|
The aggregate records case identifiers,
|
|
`functional_validation`, and signed percentage changes in latency, throughput,
|
|
power, and energy against the matching baseline case. Positive means higher;
|
|
baseline values are zero.
|
|
|
|
Disabling synchronization removes host-buffer lifetime and pipeline restart
|
|
guarantees, so those variants may fail numerical validation and exit nonzero.
|
|
Baseline and `no-plan` must pass. Percentages from failing variants are not
|
|
valid speedups; deadline averages do not prove steady-state convergence.
|
|
|
|
## Result data
|
|
|
|
The checked-in `results_ablation.csv` is an intentional result dataset for tabular
|
|
reporting. Ablation experiments and aggregation remain available through the
|
|
runner above. Percentages from failing variants are not valid speedups. See the
|
|
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
|