NiccoloN
b009e1ff08
Validate Operations / validate-operations (push) Has been cancelled
normalize names and artifact paths
67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
# Raptor compiler ablation
|
|
|
|
`run_ablation.py` performs the complete synchronization/Spatial-planning
|
|
ablation study on the Pimcomp model suite. By default it runs `vgg8`,
|
|
`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
|
|
and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
|
|
explicitly.
|
|
Latency is pipeline 1; the wrapper invokes the suite runner separately for
|
|
latency and throughput pipeline 4.
|
|
|
|
```bash
|
|
.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
|
|
--jobs 4
|
|
```
|
|
|
|
## Variants
|
|
|
|
| Variant | Raptor options |
|
|
|---|---|
|
|
| `baseline` | None. |
|
|
| `no-synchronization` | `--pim-disable-synchronization` |
|
|
| `no-spatial-planning` | `--pim-disable-spatial-planning` |
|
|
| `no-synchronization-no-spatial-planning` | Both options. |
|
|
|
|
Every variant runs Raptor only. Pimcomp is not compiled, validated, or
|
|
simulated. Reference inputs and outputs are generated once under the shared
|
|
common-artifact root and reused by every variant. Ctrl+C terminates the active
|
|
variant and all of its worker jobs.
|
|
|
|
The percentage baseline is `no-synchronization-no-spatial-planning`: both
|
|
ablation features are disabled, so its values are `+0.00%`. Every other
|
|
variant reports the signed percentage difference of its Raptor metrics from
|
|
that reference for the same model, architecture, mode, and pipeline. Positive
|
|
values mean the metric is higher; negative values mean it is lower.
|
|
|
|
## Options
|
|
|
|
| Option | Description and default |
|
|
|---|---|
|
|
| `-h`, `--help` | Show help and exit. |
|
|
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts`; disabled variants use `<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
|
|
| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
|
|
| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
|
|
| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
|
|
|
|
All options of
|
|
[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
|
|
`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
|
|
`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
|
|
`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
|
|
the suite runner's comparison mode and is rejected by this wrapper, which
|
|
always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
|
|
forwarded, the wrapper uses its default latency and throughput/pipeline-4
|
|
case set. Supplying either option overrides that default and is passed through
|
|
as one suite-runner invocation per variant.
|
|
|
|
The feature-full `baseline` variant uses the normal `run_pimcomp_models.py`
|
|
artifact paths and is rerun with the same selected cases and forwarded options
|
|
as the disabled variants. The three disabled variants are stored below each model's
|
|
`artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared reference
|
|
artifacts remain under each model's `artifacts/common` directory. Transient
|
|
per-variant comparison summaries are removed after aggregation. The combined
|
|
table is written to `<out-dir>/results_ablation.csv`; it contains only the variant, case
|
|
identifiers, and signed `latency_percent`, `throughput_percent`,
|
|
`power_percent`, and `energy_percent` columns. These are Raptor metrics;
|
|
Pimcomp metrics are omitted because the ablation invokes Raptor only.
|