NiccoloN
5f2db946f0
Validate Operations / validate-operations (push) Has been cancelled
fix failing no-plan throughput ablation runs update results
89 lines
4.7 KiB
Markdown
89 lines
4.7 KiB
Markdown
# Raptor compiler ablation
|
||
|
||
Run the synchronization/Spatial-planning ablation from the repository root:
|
||
|
||
```bash
|
||
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
|
||
```
|
||
|
||
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
|
||
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
|
||
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
|
||
|
||
| Variant | Raptor options |
|
||
|---|---|
|
||
| `baseline` | None; planning and synchronization enabled |
|
||
| `no-sync` | `--pim-disable-synchronization`; throughput only |
|
||
| `no-plan` | `--pim-disable-spatial-planning`; synchronization remains enabled |
|
||
|
||
Latency runs compare `baseline` with `no-plan`. Throughput runs compare
|
||
`baseline` with each disabled feature separately. No run disables both features.
|
||
The `baseline` artifact name identifies the fully enabled program, not the
|
||
denominator used for percentage gains.
|
||
|
||
## Options and outputs
|
||
|
||
- `--variant NAME`: repeat to select variants; baseline is always included.
|
||
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
|
||
- `--dry-run`: print commands without modifying files.
|
||
|
||
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
|
||
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
|
||
replaces the default latency/pipeline-4 case set. Mode-only selection runs all
|
||
suite pipelines for that mode. Pipeline-only selection uses its matching mode.
|
||
`--only` is rejected because
|
||
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
|
||
|
||
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
|
||
reference inputs, outputs, and runners are shared under `artifacts/common/`.
|
||
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
|
||
Normal comparison results are preserved.
|
||
|
||
By default, the combined [results_ablation.csv](results_ablation.csv) is written
|
||
beside this script and printed after the run. An explicit `--out-dir PATH`
|
||
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
|
||
Default model artifacts and per-variant summaries remain in the locations above.
|
||
The aggregate contains one row per comparison:
|
||
|
||
| Mode | `comparison` | `baseline_variant` | Main gain |
|
||
|---|---|---|---|
|
||
| Latency | `plan` | `no-plan` | `latency_gain_percent` |
|
||
| Throughput | `plan` | `no-plan` | `throughput_gain_percent` |
|
||
| Throughput | `sync` | `no-sync` | `throughput_gain_percent` |
|
||
|
||
Throughput gain is `100 × (enabled / disabled − 1)`.
|
||
Latency, power, and energy gains are reductions: `100 × (1 − enabled / disabled)`.
|
||
The corresponding columns use the suffix `_gain_percent`. Positive means improvement;
|
||
negative means a cost. Synchronization gain can be negative. No sign is forced.
|
||
Both programs' functional statuses are recorded as `functional_validation`
|
||
and `baseline_functional_validation`. The latter belongs to the disabled feature.
|
||
|
||
Disabling synchronization removes host-buffer lifetime and pipeline restart
|
||
guarantees, so those variants may fail numerical validation. A completed
|
||
`no-sync` measurement retains `FAIL` in the CSV and is reported as diagnostic
|
||
in the runner summary. Compile, execution, missing-output, and measurement
|
||
failures still make the run fail. Baseline and `no-plan` must pass. Percentages involving failing programs remain
|
||
diagnostic comparisons, not valid speedups.
|
||
|
||
When fast-mode convergence fails, positive fallback throughput remains available.
|
||
Its output interval uses the full simulation window. The per-case JSON report
|
||
records `throughput_measurement: full_duration_fallback` and
|
||
`fast_throughput_converged: false`. This result does not prove steady-state convergence.
|
||
Zero completed outputs still produce `NA` for per-inference metrics.
|
||
|
||
The `no-sync` compiler option keeps the normal selected schedule and omits
|
||
SYNC/WAIT instructions during code generation. Selection costs describe the
|
||
`synchronized-reference` program. Each backend attempt also reports a separate
|
||
`synchronization_ablation_cost`, with zero SYNC/WAIT work and no synchronization
|
||
dependencies. This diagnostic estimate is not used for selection. Its complete-state
|
||
recurrence can remain unproved because cores can advance independently.
|
||
Numerical validation still checks the emitted program and can fail.
|
||
|
||
## Result data
|
||
|
||
The checked-in `results_ablation.csv` is a historical dataset until the next run
|
||
regenerates it with the comparison schema above. Existing measurements are preserved.
|
||
Ablation experiments and aggregation remain available through the
|
||
runner above. Percentages from failing variants are not valid speedups. See the
|
||
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
|