Raptor/validation/tools/pim/pimcomp/compare

Pimcomp model comparison

Compare Raptor and Pimcomp on the model suite. Run commands from the repository root with the repository virtual environment, built release Raptor, Pimcomp, and pimsim-nn.

Suite runner

.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py

Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines 2/4/8, four workers, and a functional throughput batch of 64. Use --help for the complete option reference.

Option	Purpose
--models MODEL [...], --archs ARCH [...]	Select models and hardware profiles.
--mode {latency,throughput} [...], --pipeline {1,2,4,8}	Select experiments.
--out-dir PATH	Suite root; default: validation/networks/pimcomp_models.
--common-dir PATH	Override the shared reference-artifact root.
--only {raptor,pimcomp}	Rerun one compiler and reuse the other's existing report.
--raptor-only	Run Raptor without invoking Pimcomp.
--batch-size N, --jobs N	Set functional throughput batch size and worker count.
--pimsim-time-ms N	Throughput convergence deadline; default: 1000 ms.
--no-fast	Use full-duration throughput simulation for authoritative experiments.
--timeout-seconds N	Per-stage timeout; default: 0 (unlimited).
--raptor-extra-arg=ARG	Forward a compiler option; repeat as needed.
--dry-run	Print commands without modifying files.
--clean	Remove generated comparison artifacts, summaries, and stale locks.

Use the equals form for forwarded flags, for example --raptor-extra-arg=--pim-detect-communication-deadlock.

With no --out-dir, the runner writes results_comparison.csv beside this script. An explicit --out-dir PATH writes PATH/results_comparison.csv; selecting a subset updates only those rows. Default model artifacts remain under validation/networks/pimcomp_models. Artifacts use <out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/, with shared references under artifacts/common/ and comparison_report.{md,json} under each comparison's pimcomp/. YOLO and GoogLeNet use artifacts/common-full-model/ to avoid reusing historical Softmax-free references. Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly written report, including failures without simulation artifacts. Missing or unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a compiler, simulation, or functional comparison fails.

If fast-mode convergence fails, the reports and CSV retain valid full-duration fallback throughput and output interval values. This is not a converged steady-state result. Per-case JSON records throughput_measurement: full_duration_fallback and fast_throughput_converged: false; the text report labels the fallback. Zero completed outputs still give NA for per-inference metrics.

Reuse checks model/configuration paths and SHA-256 contents, effective hardware, mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags. Malformed, incomplete, mismatched, or legacy reports without fingerprints fail with a regeneration diagnostic. --only raptor requires compatible reports; --only pimcomp reruns Pimcomp for each group and preserves existing Raptor results. Ordinary runs also rerun the group's first case, even if an old report exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity. Tool/build revisions and manually edited reference artifacts are not hashed; rerun the relevant compiler after those change.

Functional validation uses the same ONNX model and Raptor program as the performance comparison. YOLO uses the full model, including both internal Softmax operations. The functional simulator executes Softmax; Pimsim-nn and the queue estimator omit its timing cost. Reported timing and energy exclude Softmax work. Other instructions and transfers retain their costs. GoogLeNet removes only its two LRN nodes and retains terminal Softmax. VGG8 and both ResNet graphs have no removed operators. The native reference uses the same model as each compiled program.

One-model comparator

compare_raptor_pimcomp_model.py requires --model PATH and --out-dir PATH. Use --help for hardware and simulation options. The suite runner supplies these automatically. The comparator defaults to absolute/relative tolerances 1e-3/1e-4; validate.py uses 1e-3/1e-5. Both use the same output checks, including a relative L2 limit for each nonzero reference output. --rtol=0 disables that additional limit in the comparator. Reports from an older validation policy require regeneration before reuse. An elementwise failure with relative L2 agreement remains FAIL. Its report notes possible numerical sensitivity. That note does not establish correctness. Every batch iteration must pass. Reported errors are maxima across iterations for each output.

Comparison charts

.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py

Requires Matplotlib from the root requirements.txt. Defaults to the supplied results_comparison.csv beside the script and writes plots/default/png/, plots/default/svg/, and plots/default/pdf/ there. Use --input PATH for a runner-generated CSV and --output-dir PATH to override the destination. The plotter requires the full 40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp configurations are rejected before plots are written.

Charts use the pastel palette and the Raptor name by default. Use --no-colors for grayscale output; --colors explicitly enables colors. Use --anonymize to replace displayed Raptor labels with Anonimized; --no-anonymize explicitly keeps the Raptor name. The flags are independent. Use --no-marker-borders for solid markers without outlines in the compact dot plot and line chart, including their legends. These markers use the darker border colors as solid fills. --marker-borders keeps the default outlines. Line colors and bar patterns remain unchanged. Input keys and output basenames remain stable. Use these output directories to keep the grayscale and colored anonymized versions:

.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --no-colors --anonymize \
  --output-dir validation/tools/pim/pimcomp/compare/plots/gray_anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --anonymize \
  --output-dir validation/tools/pim/pimcomp/compare/plots/anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --anonymize --no-marker-borders \
  --output-dir validation/tools/pim/pimcomp/compare/plots/no_border_anonimized

Plot family	Meaning
validation_status	Recorded functional status for each compiler and experiment.
validation_status_compact	Compact functional status matrix.
performance_energy_compact	Four horizontal dot panels: latency-mode speedup, throughput-mode speedup, and separate energy panels for both modes.
speedup_compact_*	Latency panel on the left and best-pipeline throughput panel on the right, with touching Arch-A/B bars per network.
energy_compact_*, power_compact_*	Energy reduction and power reduction for the same cases selected by the compact speedup plots.
latency_*	Single-inference latency, latency mode only.
throughput_*	Inferences/s, throughput mode only.
longest_stage_latency_*	Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy.
power_*, energy_*	Average power and energy per inference.
raptor_pipeline_throughput	Raptor throughput versus stages; stage 1 is derived as 1000 / latency_ms, stages 2/4/8 are measured.

Suffixes _absolute and _yolo show measurements (_yolo is Raptor-only); _comparison shows ratios and _diverging shows equal-length bars for equal advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use winning factor minus one, with parity at zero. Direct ratio axes start at 0.5, or 0.25 when a result is below 0.5. Panels in each figure share the same minimum.

The shared plot_style.py defines the manuscript palette and display names. Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns, outlines, marker shapes, and status text preserve distinctions in grayscale. Compact bar plots use plain blue for Arch-A and hatched sage green for Arch-B. The left panel contains latency-mode measurements. The right panel contains throughput-mode measurements. Each network has touching Arch-A/B bars. All compact figures use the display names Latency-mode and Throughput-mode. Each architecture/network pair uses the highest recorded throughput across pipelines. Energy and power use those same cases, without selecting a different pipeline for each metric. Validation labels remain visible for the selected cases. Units appear in axis titles and bar values; axis tick labels contain numbers only.

The performance_energy_compact figure includes the eight Arch-A/B comparisons for VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It excludes YOLO11n. All panels share a linear ratio axis from 0.5, or 0.25 when needed, through the largest ratio, with a small right margin. A dashed line marks parity at 1. The shared axis title and ticks appear at the top, below the legend. The Ratio (x) title is centered over the axis. Panel titles appear to the left, rotate 90° counterclockwise, and use centered text on at most two lines. Titles start with Latency-mode or Throughput-mode, followed by the metric. The PIMCOMP-better and Raptor-better indicators flank the shared axis title. The figure has no mean markers. Labels give ratios to two decimal places; marker positions use unrounded ratios. Each panel has separate Arch-A and Arch-B lanes. The legend identifies models through marker shapes and pastel colors. Result labels contain ratios and use vertical text above or below their architecture line, without connectors. Placement prefers above for Arch-A and below for Arch-B. Lane spacing and label placement avoid other labels, markers, and reference lines. Larger dark-red daggers mark invalid PIMCOMP comparisons, including every batch comparison, even when the CSV records PASS. Marker borders retain the model colors; ratio values use normal text colors. The legend dagger matches the result daggers. Both use the same font and size. Labels without daggers reserve the same space, so numeric values align within each lane. These simulated ratios remain visible. The two energy panels show latency-mode energy and throughput-selected energy. The throughput energy uses the selected throughput pipeline, without a separate energy optimization.

Throughput-stage bars touch within each network in both absolute and advantage plots. Bar-label widths are measured before export; dense figures expand only as needed to contain values and invalidity annotations. Legends use the available width and balance entries across rows when wrapping is necessary.

Comparison panels share y-axis titles and tick labels. Each network in the pipeline plot retains its own scale, with y-axis ticks on both architecture rows. Only the bottom row shows the shared x-axis ticks. Network names appear above the top row; architecture names appear on the right. That plot trims unused vertical range and is intended to span both manuscript columns. The line chart shows throughput only; validation status does not change its markers or colors.

Failed validation is labeled P. INVALID; other non-PASS statuses keep their names, and missing measurements show N/A. Relative bars require both compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses must agree before being collapsed. FAIL alone does not identify the cause.

Plots consume the CSV's current contents; they do not run benchmarks or establish publication provenance. The checked-in dataset is intentional. See the result methodology.

Metric interpretation

See the evaluation methodology for measurement windows, units, numerical validity, and result provenance.

Report reuse checks model and configuration hashes, compiler and simulator fingerprints, validation policy, actual tolerances, seed, batch size, and simulation options. Artifact signatures check instructions, weights, and memory images. Missing or changed artifacts require regeneration. Redundant Raptor JSON instruction dumps do not affect signatures when native binaries exist.

better_compiler is available only when both compilers pass functional validation. speedup retains the diagnostic timing ratio when both timings exist. A timing ratio for a failed implementation does not identify a better correct implementation. Valid-only plot aggregates require both validations to pass.