# Pimcomp model comparison

Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
Run commands from the repository root with the repository virtual environment,
built release Raptor, Pimcomp, and `pimsim-nn`.

## Suite runner

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```

Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
2/4/8, four workers, and a functional throughput batch of 64.
Use `--help` for the complete option reference.

| Option | Purpose |
|---|---|
| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
| `--common-dir PATH` | Override the shared reference-artifact root. |
| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
| `--raptor-only` | Run Raptor without invoking Pimcomp. |
| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
| `--dry-run` | Print commands without modifying files. |
| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |

Use the equals form for forwarded flags, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.

With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
beside this script. An explicit `--out-dir PATH` writes
`PATH/results_comparison.csv`; selecting a subset updates only those rows.
Default model artifacts remain under `validation/networks/pimcomp_models`.
Artifacts use
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
under `artifacts/common/`
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
YOLO and GoogLeNet use `artifacts/common-full-model/` to avoid reusing historical Softmax-free references.
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
written report, including failures without simulation artifacts. Missing or
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
compiler, simulation, or functional comparison fails.

If fast-mode convergence fails, the reports and CSV retain valid full-duration
fallback throughput and output interval values. This is not a converged steady-state result.
Per-case JSON records `throughput_measurement: full_duration_fallback` and
`fast_throughput_converged: false`; the text report labels the fallback.
Zero completed outputs still give `NA` for per-inference metrics.

Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
with a regeneration diagnostic. `--only raptor` requires compatible reports;
`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
results. Ordinary runs also rerun the group's first case, even if an old report
exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
Tool/build revisions and manually edited reference artifacts are not hashed;
rerun the relevant compiler after those change.

Functional validation uses the same ONNX model and Raptor program as the performance comparison.
YOLO uses the full model, including both internal Softmax operations.
The functional simulator executes Softmax; Pimsim-nn and the queue estimator omit its timing cost.
Reported timing and energy exclude Softmax work. Other instructions and transfers retain their costs.
GoogLeNet removes only its two LRN nodes and retains terminal Softmax.
VGG8 and both ResNet graphs have no removed operators.
The native reference uses the same model as each compiled program.

## One-model comparator

`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
Use `--help` for hardware and simulation options. The suite runner supplies
these automatically.
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
`validate.py` uses `1e-3`/`1e-5`.
Both use the [same output checks](../../../../README.md#performance-and-result-interpretation),
including a relative L2 limit for each nonzero reference output.
`--rtol=0` disables that additional limit in the comparator.
Reports from an older validation policy require regeneration before reuse.
An elementwise failure with relative L2 agreement remains `FAIL`.
Its report notes possible numerical sensitivity. That note does not establish correctness.
Every batch iteration must pass. Reported errors are maxima across iterations for each output.

## Comparison charts

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
```

Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
`results_comparison.csv` beside the script and writes `plots/default/png/`,
`plots/default/svg/`, and `plots/default/pdf/` there. Use `--input PATH` for a runner-generated CSV and
`--output-dir PATH` to override the destination. The plotter requires the full
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
configurations are rejected before plots are written.

Charts use the pastel palette and the Raptor name by default.
Use `--no-colors` for grayscale output; `--colors` explicitly enables colors.
Use `--anonymize` to replace displayed `Raptor` labels with `Anonimized`;
`--no-anonymize` explicitly keeps the Raptor name. The flags are independent.
Use `--no-marker-borders` for solid markers without outlines in the compact dot
plot and line chart, including their legends. These markers use the darker
border colors as solid fills. `--marker-borders` keeps the default outlines.
Line colors and bar patterns remain unchanged.
Input keys and output basenames remain stable. Use these output directories
to keep the grayscale and colored anonymized versions:

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --no-colors --anonymize \
  --output-dir validation/tools/pim/pimcomp/compare/plots/gray_anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --anonymize \
  --output-dir validation/tools/pim/pimcomp/compare/plots/anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
  --anonymize --no-marker-borders \
  --output-dir validation/tools/pim/pimcomp/compare/plots/no_border_anonimized
```

| Plot family | Meaning |
|---|---|
| `validation_status` | Recorded functional status for each compiler and experiment. |
| `validation_status_compact` | Compact functional status matrix. |
| `performance_energy_compact` | Four horizontal dot panels: latency-mode speedup, throughput-mode speedup, and separate energy panels for both modes. |
| `speedup_compact_*` | Latency panel on the left and best-pipeline throughput panel on the right, with touching Arch-A/B bars per network. |
| `energy_compact_*`, `power_compact_*` | Energy reduction and power reduction for the same cases selected by the compact speedup plots. |
| `latency_*` | Single-inference latency, latency mode only. |
| `throughput_*` | Inferences/s, throughput mode only. |
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
| `power_*`, `energy_*` | Average power and energy per inference. |
| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |

Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
winning factor minus one, with parity at zero.
Direct ratio axes start at 0.5, or 0.25 when a result is below 0.5.
Panels in each figure share the same minimum.

The shared `plot_style.py` defines the manuscript palette and display names.
Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns,
outlines, marker shapes, and status text preserve distinctions in grayscale.
Compact bar plots use plain blue for Arch-A and hatched sage green for Arch-B.
The left panel contains latency-mode measurements. The right panel contains
throughput-mode measurements. Each network has touching Arch-A/B bars.
All compact figures use the display names `Latency-mode` and `Throughput-mode`.
Each architecture/network pair uses the highest recorded throughput across pipelines.
Energy and power use those same cases, without selecting a different pipeline for each metric.
Validation labels remain visible for the selected cases. Units appear in axis
titles and bar values; axis tick labels contain numbers only.

The `performance_energy_compact` figure includes the eight Arch-A/B comparisons
for VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It excludes YOLO11n.
All panels share a linear ratio axis from 0.5, or 0.25 when needed, through the largest ratio,
with a small right margin. A dashed line marks parity at 1.
The shared axis title and ticks appear at the top, below the legend.
The `Ratio (x)` title is centered over the axis. Panel titles appear to the left,
rotate 90° counterclockwise, and use centered text on at most two lines.
Titles start with `Latency-mode` or `Throughput-mode`, followed by the metric.
The PIMCOMP-better and Raptor-better indicators flank the shared axis title.
The figure has no mean markers.
Labels give ratios to two decimal places; marker positions use unrounded ratios.
Each panel has separate Arch-A and Arch-B lanes. The legend identifies models
through marker shapes and pastel colors. Result labels contain ratios and use
vertical text above or below their architecture line, without connectors.
Placement prefers above for Arch-A and below for Arch-B.
Lane spacing and label placement avoid other labels, markers, and reference lines.
Larger dark-red daggers mark invalid PIMCOMP comparisons, including every batch comparison,
even when the CSV records PASS. Marker borders retain the model colors;
ratio values use normal text colors. The legend dagger matches the result daggers.
Both use the same font and size. Labels without daggers reserve the same space,
so numeric values align within each lane.
These simulated ratios remain visible.
The two energy panels show latency-mode energy and throughput-selected energy.
The throughput energy uses the selected throughput pipeline, without a separate
energy optimization.

Throughput-stage bars touch within each network in both absolute and advantage
plots. Bar-label widths are measured before export; dense figures expand only
as needed to contain values and invalidity annotations. Legends use the available
width and balance entries across rows when wrapping is necessary.

Comparison panels share y-axis titles and tick labels. Each network in the
pipeline plot retains its own scale, with y-axis ticks on both architecture rows.
Only the bottom row shows the shared x-axis ticks. Network names appear above
the top row; architecture names appear on the right. That plot
trims unused vertical range and is intended to span both manuscript columns.
The line chart shows throughput only; validation status does not change its markers or colors.

Failed validation is labeled `P. INVALID`; other non-PASS statuses keep their
names, and missing measurements show `N/A`. Relative bars require both
compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
must agree before being collapsed. `FAIL` alone does not identify the cause.

Plots consume the CSV's current contents; they do not run benchmarks or establish
publication provenance. The checked-in dataset is intentional. See the
[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).

## Metric interpretation

See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, units, numerical validity, and result provenance.

Report reuse checks model and configuration hashes, compiler and simulator
fingerprints, validation policy, actual tolerances, seed, batch size, and
simulation options. Artifact signatures check instructions, weights, and memory
images. Missing or changed artifacts require regeneration. Redundant Raptor
JSON instruction dumps do not affect signatures when native binaries exist.

`better_compiler` is available only when both compilers pass functional
validation. `speedup` retains the diagnostic timing ratio when both timings
exist. A timing ratio for a failed implementation does not identify a better
correct implementation. Valid-only plot aggregates require both validations
to pass.