Raptor/validation/tools/pim/pimcomp/compare

Pimcomp model comparison

Compare Raptor and Pimcomp on the model suite. Run commands from the repository root with the repository virtual environment, built release Raptor, Pimcomp, and pimsim-nn.

Suite runner

.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py

Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines 2/4/8, four workers, and a functional throughput batch of 64. Use --help for the complete option reference.

Option	Purpose
--models MODEL [...], --archs ARCH [...]	Select models and hardware profiles.
--mode {latency,throughput} [...], --pipeline {1,2,4,8}	Select experiments.
--out-dir PATH	Suite root; default: validation/networks/pimcomp_models.
--common-dir PATH	Override the shared reference-artifact root.
--only {raptor,pimcomp}	Rerun one compiler and reuse the other's existing report.
--raptor-only	Run Raptor without invoking Pimcomp.
--batch-size N, --jobs N	Set functional throughput batch size and worker count.
--pimsim-time-ms N	Throughput convergence deadline; default: 1000 ms.
--no-fast	Use full-duration throughput simulation for authoritative experiments.
--timeout-seconds N	Per-stage timeout; default: 0 (unlimited).
--raptor-extra-arg=ARG	Forward a compiler option; repeat as needed.
--dry-run	Print commands without modifying files.
--clean	Remove generated comparison artifacts, summaries, and stale locks.

Use the equals form for forwarded flags, for example --raptor-extra-arg=--pim-detect-communication-deadlock.

With no --out-dir, the runner writes results_comparison.csv beside this script. An explicit --out-dir PATH writes PATH/results_comparison.csv; selecting a subset updates only those rows. Default model artifacts remain under validation/networks/pimcomp_models. Artifacts use <out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/, with shared references under artifacts/common/ (common-functional/ for YOLO's canonical reference) and comparison_report.{md,json} under each comparison's pimcomp/. Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly written report, including failures without simulation artifacts. Missing or unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a compiler, simulation, or functional comparison fails.

Reuse checks model/configuration paths and SHA-256 contents, effective hardware, mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags. Malformed, incomplete, mismatched, or legacy reports without fingerprints fail with a regeneration diagnostic. --only raptor requires compatible reports; --only pimcomp reruns Pimcomp for each group and preserves existing Raptor results. Ordinary runs also rerun the group's first case, even if an old report exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity. Tool/build revisions and manually edited reference artifacts are not hashed; rerun the relevant compiler after those change.

One-model comparator

compare_raptor_pimcomp_model.py requires --model PATH and --out-dir PATH. Use --help for hardware and simulation options. The suite runner supplies these automatically. The comparator defaults to absolute/relative tolerances 1e-3/1e-4; validate.py uses 1e-3/1e-5.

Comparison charts

.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py

Requires Matplotlib from the root requirements.txt. Defaults to the supplied results_comparison.csv beside the script and writes plots/png/ and plots/svg/ there. Use --input PATH for a runner-generated CSV and --output-dir PATH to override the destination. The plotter requires the full 40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp configurations are rejected before plots are written.

Plot family	Meaning
validation_status	Recorded functional status for each compiler and experiment.
latency_*	Single-inference latency, latency mode only.
throughput_*	Inferences/s, throughput mode only.
longest_stage_latency_*	Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy.
power_*, energy_*	Average power and energy per inference.
raptor_pipeline_throughput	Raptor throughput versus stages; stage 1 is derived as 1000 / latency_ms, stages 2/4/8 are measured.

Suffixes _absolute and _yolo show measurements (_yolo is Raptor-only); _comparison shows ratios and _diverging shows equal-length bars for equal advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use winning factor minus one, with parity at zero.

Failed validation is labeled INVALID; other non-PASS statuses keep their names, and missing measurements show N/A. Relative bars require both compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses must agree before being collapsed. FAIL alone does not identify the cause.

Plots consume the CSV's current contents; they do not run benchmarks or establish publication provenance. The checked-in dataset is intentional. See the result methodology.

Metric interpretation

See the evaluation methodology for measurement windows, units, numerical validity, and result provenance.