034c197260
Validate Operations / validate-operations (push) Failing after 3h10m55s
add plot generation scripts adjust validation tools add up-to-date results
74 lines
3.4 KiB
Markdown
74 lines
3.4 KiB
Markdown
# Raptor compiler ablation
|
|
|
|
Run the synchronization/Spatial-planning ablation from the repository root:
|
|
|
|
```bash
|
|
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
|
|
```
|
|
|
|
Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
|
|
(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
|
|
explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
|
|
|
|
| Variant | Raptor options |
|
|
|---|---|
|
|
| `baseline` | None |
|
|
| `no-sync` | `--pim-disable-synchronization` |
|
|
| `no-plan` | `--pim-disable-spatial-planning` |
|
|
| `no-sync-no-plan` | Both options |
|
|
|
|
## Options and outputs
|
|
|
|
- `--variant NAME`: repeat to select variants; baseline is always included.
|
|
- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
|
|
- `--dry-run`: print commands without modifying files.
|
|
|
|
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
|
|
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
|
|
replaces the default latency/pipeline-4 case set. `--only` is rejected because
|
|
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
|
|
|
|
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
|
|
reference inputs, outputs, and runners are shared under `artifacts/common/`.
|
|
Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
|
|
Normal comparison results are preserved.
|
|
|
|
By default, the combined [results_ablation.csv](results_ablation.csv) is written
|
|
beside this script and printed after the run. An explicit `--out-dir PATH`
|
|
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
|
|
Default model artifacts and per-variant summaries remain in the locations above.
|
|
The aggregate records case identifiers,
|
|
`functional_validation`, and signed percentage changes in latency, throughput,
|
|
power, and energy against the matching baseline case. Positive means higher;
|
|
baseline values are zero.
|
|
|
|
Disabling synchronization removes host-buffer lifetime and pipeline restart
|
|
guarantees, so those variants may fail numerical validation and exit nonzero.
|
|
Baseline and `no-plan` must pass. Percentages from failing variants are not
|
|
valid speedups; deadline averages do not prove steady-state convergence.
|
|
|
|
## Plots
|
|
|
|
```bash
|
|
.venv/bin/python validation/tools/pim/ablation/generate_plots.py
|
|
# For a custom experiment root:
|
|
.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
|
|
--input /path/to/experiment/results_ablation.csv
|
|
```
|
|
|
|
Requires Matplotlib from the root `requirements.txt`. By default, reads the
|
|
canonical CSV beside this script. Plots go to
|
|
`plots/png/` and `plots/svg/` beside the selected CSV; override with
|
|
`--output-dir PATH`.
|
|
Single-variant comparison CSVs are not accepted.
|
|
|
|
The checked-in CSV is an intentional result dataset; plotting uses its current
|
|
contents without running benchmarks. Invalid reciprocal percentage pairs render
|
|
as `N/A`; individual legitimate extreme changes are preserved. See
|
|
[performance availability](../../../../docs/evaluation/methodology.md#functional-validity-and-performance-availability).
|
|
|
|
Plots separate modes and pipelines. Throughput-mode latency means output
|
|
interval. Failed validation is labeled `INVALID`, other non-PASS statuses keep
|
|
their names, and missing measurements show `N/A`. See the
|
|
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
|