NiccoloN
034c197260
add plot generation scripts adjust validation tools add up-to-date results
3.4 KiB
Raptor compiler ablation
Run the synchronization/Spatial-planning ablation from the repository root:
.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
Defaults: vgg8 resnet18 resnet34 googlenet, arch-a arch-b, latency
(pipeline 1), and throughput (pipeline 4). Select yolo11n or arch-c
explicitly when needed. Only Raptor runs; Pimcomp validation is SKIPPED.
| Variant | Raptor options |
|---|---|
baseline |
None |
no-sync |
--pim-disable-synchronization |
no-plan |
--pim-disable-spatial-planning |
no-sync-no-plan |
Both options |
Options and outputs
--variant NAME: repeat to select variants; baseline is always included.--out-dir PATH: artifact root; defaults tovalidation/networks/pimcomp_models.--dry-run: print commands without modifying files.
Other suite-runner options, including --models,
--archs, and --no-fast, are forwarded. Supplying --mode or --pipeline
replaces the default latency/pipeline-4 case set. --only is rejected because
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
Artifacts use <out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/;
reference inputs, outputs, and runners are shared under artifacts/common/.
Per-variant summaries use <out-dir>/ablation/<variant>/results_comparison.csv.
Normal comparison results are preserved.
By default, the combined results_ablation.csv is written
beside this script and printed after the run. An explicit --out-dir PATH
writes PATH/results_ablation.csv, keeping independent experiments separate.
Default model artifacts and per-variant summaries remain in the locations above.
The aggregate records case identifiers,
functional_validation, and signed percentage changes in latency, throughput,
power, and energy against the matching baseline case. Positive means higher;
baseline values are zero.
Disabling synchronization removes host-buffer lifetime and pipeline restart
guarantees, so those variants may fail numerical validation and exit nonzero.
Baseline and no-plan must pass. Percentages from failing variants are not
valid speedups; deadline averages do not prove steady-state convergence.
Plots
.venv/bin/python validation/tools/pim/ablation/generate_plots.py
# For a custom experiment root:
.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
--input /path/to/experiment/results_ablation.csv
Requires Matplotlib from the root requirements.txt. By default, reads the
canonical CSV beside this script. Plots go to
plots/png/ and plots/svg/ beside the selected CSV; override with
--output-dir PATH.
Single-variant comparison CSVs are not accepted.
The checked-in CSV is an intentional result dataset; plotting uses its current
contents without running benchmarks. Invalid reciprocal percentage pairs render
as N/A; individual legitimate extreme changes are preserved. See
performance availability.
Plots separate modes and pipelines. Throughput-mode latency means output
interval. Failed validation is labeled INVALID, other non-PASS statuses keep
their names, and missing measurements show N/A. See the
comparison metric definitions.