NiccoloN
79 lines
4.2 KiB
Markdown
79 lines
4.2 KiB
Markdown
# Raptor compiler ablation
|
|
|
|
`run_ablation.py` performs the complete synchronization/Spatial-planning
|
|
ablation study on the Pimcomp model suite. By default it runs `vgg8`,
|
|
`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
|
|
and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
|
|
explicitly.
|
|
Latency is pipeline 1; the wrapper invokes the suite runner separately for
|
|
latency and throughput pipeline 4.
|
|
|
|
```bash
|
|
.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
|
|
--jobs 4
|
|
```
|
|
|
|
## Variants
|
|
|
|
| Variant | Raptor options |
|
|
|---|---|
|
|
| `baseline` | None. |
|
|
| `no-sync` | `--pim-disable-synchronization` |
|
|
| `no-plan` | `--pim-disable-spatial-planning` |
|
|
| `no-sync-no-plan` | Both options. |
|
|
|
|
Every variant runs Raptor only. Pimcomp is not compiled, validated, or
|
|
simulated; its validation column is `SKIPPED`, not `FAIL`. Reference inputs and
|
|
outputs are generated once under the shared common-artifact root and reused by
|
|
every variant. Ctrl+C terminates the active variant and all of its worker jobs.
|
|
|
|
Removing synchronization deliberately removes host-buffer lifetime and pipeline
|
|
restart guarantees. The `no-sync` variants can therefore fail numerical
|
|
validation, especially across repeated throughput inputs. Those failures remain
|
|
`FAIL` and cause a nonzero exit; a completed ablation is not proof that every
|
|
variant is correct. Baseline and `no-plan` must still pass numerical validation.
|
|
The combined CSV retains `functional_validation` for every row. Performance
|
|
percentages from failing variants describe incorrect programs, not valid
|
|
speedups. Throughput deadline averages likewise do not establish a converged
|
|
steady-state period.
|
|
|
|
The percentage baseline is `baseline`, so its available values are `+0.00%`.
|
|
Every other
|
|
variant reports the signed percentage difference of its Raptor metrics from
|
|
that reference for the same model, architecture, mode, and pipeline. Positive
|
|
values mean the metric is higher; negative values mean it is lower.
|
|
|
|
## Options
|
|
|
|
| Option | Description and default |
|
|
|---|---|
|
|
| `-h`, `--help` | Show help and exit. |
|
|
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
|
|
| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
|
|
| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
|
|
| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
|
|
|
|
All options of
|
|
[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
|
|
`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
|
|
`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
|
|
`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
|
|
the suite runner's comparison mode and is rejected by this wrapper, which
|
|
always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
|
|
forwarded, the wrapper uses its default latency and throughput/pipeline-4
|
|
case set. Supplying either option overrides that default and is passed through
|
|
as one suite-runner invocation per variant.
|
|
|
|
The feature-full `baseline` variant is rerun with the same selected cases and
|
|
forwarded options as the disabled variants. All four variants are stored below
|
|
each model's `artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared
|
|
reference artifacts remain under each model's `artifacts/common` directory.
|
|
Normal comparison artifacts and summaries are preserved. Compact per-variant
|
|
comparison summaries remain under
|
|
`<out-dir>/ablation/<variant>/results_comparison.csv` after aggregation.
|
|
The combined table is written to `<out-dir>/results_ablation.csv`; it contains the
|
|
variant, case identifiers, actual `functional_validation` status, and signed
|
|
percentage `latency`, `throughput`, `power`, and `energy` columns. These are
|
|
Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor
|
|
only.
|