add ablation study

normalize names and artifact paths
2026-08-20 17:58:02 +02:00
parent add20e56eb
commit b009e1ff08
67 changed files with 1573 additions and 993 deletions
@@ -0,0 +1,66 @@
+# Raptor compiler ablation
+
+`run_ablation.py` performs the complete synchronization/Spatial-planning
+ablation study on the Pimcomp model suite. By default it runs `vgg8`,
+`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
+and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
+explicitly.
+Latency is pipeline 1; the wrapper invokes the suite runner separately for
+latency and throughput pipeline 4.
+
+```bash
+.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
+  --jobs 4
+```
+
+## Variants
+
+| Variant | Raptor options |
+|---|---|
+| `baseline` | None. |
+| `no-synchronization` | `--pim-disable-synchronization` |
+| `no-spatial-planning` | `--pim-disable-spatial-planning` |
+| `no-synchronization-no-spatial-planning` | Both options. |
+
+Every variant runs Raptor only. Pimcomp is not compiled, validated, or
+simulated. Reference inputs and outputs are generated once under the shared
+common-artifact root and reused by every variant. Ctrl+C terminates the active
+variant and all of its worker jobs.
+
+The percentage baseline is `no-synchronization-no-spatial-planning`: both
+ablation features are disabled, so its values are `+0.00%`. Every other
+variant reports the signed percentage difference of its Raptor metrics from
+that reference for the same model, architecture, mode, and pipeline. Positive
+values mean the metric is higher; negative values mean it is lower.
+
+## Options
+
+| Option | Description and default |
+|---|---|
+| `-h`, `--help` | Show help and exit. |
+| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts`; disabled variants use `<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
+| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
+| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
+| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
+
+All options of
+[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
+`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
+`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
+`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
+the suite runner's comparison mode and is rejected by this wrapper, which
+always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
+forwarded, the wrapper uses its default latency and throughput/pipeline-4
+case set. Supplying either option overrides that default and is passed through
+as one suite-runner invocation per variant.
+
+The feature-full `baseline` variant uses the normal `run_pimcomp_models.py`
+artifact paths and is rerun with the same selected cases and forwarded options
+as the disabled variants. The three disabled variants are stored below each model's
+`artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared reference
+artifacts remain under each model's `artifacts/common` directory. Transient
+per-variant comparison summaries are removed after aggregation. The combined
+table is written to `<out-dir>/results_ablation.csv`; it contains only the variant, case
+identifiers, and signed `latency_percent`, `throughput_percent`,
+`power_percent`, and `energy_percent` columns. These are Raptor metrics;
+Pimcomp metrics are omitted because the ablation invokes Raptor only.