Raptor/validation/tools/pim/ablation/README.md

Raptor compiler ablation

run_ablation.py performs the complete synchronization/Spatial-planning ablation study on the Pimcomp model suite. By default it runs vgg8, resnet18, resnet34, and googlenet across arch-a and arch-b, latency, and throughput pipeline 4. arch-c and yolo11n are run only when selected explicitly. Latency is pipeline 1; the wrapper invokes the suite runner separately for latency and throughput pipeline 4.

.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
  --jobs 4

Variants

Variant	Raptor options
baseline	None.
no-sync	--pim-disable-synchronization
no-plan	--pim-disable-spatial-planning
no-sync-no-plan	Both options.

Every variant runs Raptor only. Pimcomp is not compiled, validated, or simulated; its validation column is SKIPPED, not FAIL. Reference inputs and outputs are generated once under the shared common-artifact root and reused by every variant. Ctrl+C terminates the active variant and all of its worker jobs.

Removing synchronization deliberately removes host-buffer lifetime and pipeline restart guarantees. The no-sync variants can therefore fail numerical validation, especially across repeated throughput inputs. Those failures remain FAIL and cause a nonzero exit; a completed ablation is not proof that every variant is correct. Baseline and no-plan must still pass numerical validation. The combined CSV retains functional_validation for every row. Performance percentages from failing variants describe incorrect programs, not valid speedups. Throughput deadline averages likewise do not establish a converged steady-state period.

The percentage baseline is baseline, so its available values are +0.00%. Every other variant reports the signed percentage difference of its Raptor metrics from that reference for the same model, architecture, mode, and pipeline. Positive values mean the metric is higher; negative values mean it is lower.

Options

Option	Description and default
-h, --help	Show help and exit.
--out-dir PATH	Suite root. Default: validation/networks/pimcomp_models. Model artifacts are stored below <out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>. Variant summaries remain under <out-dir>/ablation/<variant>/.
--models MODEL [...]	Models to run. Default: vgg8 resnet18 resnet34 googlenet; include yolo11n explicitly when needed.
--variant NAME	Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed.
--dry-run	Print the suite-runner commands that would run without modifying files. Default: off.

All options of run_pimcomp_models.py, including --archs, --mode, --pipeline, --no-fast, and --raptor-extra-arg=ARG, are forwarded to each variant. --out-dir, --variant, and --dry-run belong to this wrapper; --only is reserved for the suite runner's comparison mode and is rejected by this wrapper, which always invokes --raptor-only. If neither --mode nor --pipeline is forwarded, the wrapper uses its default latency and throughput/pipeline-4 case set. Supplying either option overrides that default and is passed through as one suite-runner invocation per variant.

The feature-full baseline variant is rerun with the same selected cases and forwarded options as the disabled variants. All four variants are stored below each model's artifacts/<arch>/<mode>[/pipelineN]/ablation/ directory. Shared reference artifacts remain under each model's artifacts/common directory. Normal comparison artifacts and summaries are preserved. Compact per-variant comparison summaries remain under <out-dir>/ablation/<variant>/results_comparison.csv after aggregation. The combined table is written to <out-dir>/results_ablation.csv; it contains the variant, case identifiers, actual functional_validation status, and signed percentage latency, throughput, power, and energy columns. These are Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor only.