# Pimcomp model comparison

Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
Run commands from the repository root with the repository virtual environment,
built release Raptor, Pimcomp, and `pimsim-nn`.

## Suite runner

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```

Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
2/4/8, four workers, and a functional throughput batch of 64.
Use `--help` for the complete option reference.

| Option | Purpose |
|---|---|
| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
| `--common-dir PATH` | Override the shared reference-artifact root. |
| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
| `--raptor-only` | Run Raptor without invoking Pimcomp. |
| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
| `--dry-run` | Print commands without modifying files. |
| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |

Use the equals form for forwarded flags, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.

With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
beside this script. An explicit `--out-dir PATH` writes
`PATH/results_comparison.csv`; selecting a subset updates only those rows.
Default model artifacts remain under `validation/networks/pimcomp_models`.
Artifacts use
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
written report, including failures without simulation artifacts. Missing or
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
compiler, simulation, or functional comparison fails.

Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
with a regeneration diagnostic. `--only raptor` requires compatible reports;
`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
results. Ordinary runs also rerun the group's first case, even if an old report
exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
Tool/build revisions and manually edited reference artifacts are not hashed;
rerun the relevant compiler after those change.

## One-model comparator

`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
Use `--help` for hardware and simulation options. The suite runner supplies
these automatically.
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
`validate.py` uses `1e-3`/`1e-5`.

## Comparison charts

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
```

Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
`results_comparison.csv` beside the script and writes `plots/png/` and
`plots/svg/` and `plots/pdf/` there. Use `--input PATH` for a runner-generated CSV and
`--output-dir PATH` to override the destination. The plotter requires the full
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
configurations are rejected before plots are written.

| Plot family | Meaning |
|---|---|
| `validation_status` | Recorded functional status for each compiler and experiment. |
| `validation_status_compact` | Compact functional status matrix. |
| `speedup_compact_*` | Touching single-inference latency and best-pipeline throughput bars for each network. |
| `energy_compact_*`, `power_compact_*` | Energy and power advantages for the same cases selected by the compact speedup plots. |
| `latency_*` | Single-inference latency, latency mode only. |
| `throughput_*` | Inferences/s, throughput mode only. |
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
| `power_*`, `energy_*` | Average power and energy per inference. |
| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |

Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
winning factor minus one, with parity at zero.

The shared `plot_style.py` defines the manuscript palette and display names.
Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns,
outlines, marker shapes, and status text preserve distinctions in grayscale.
Compact plots use blue for the single-inference case and hatched sage
green for the throughput case.
Each architecture/network pair uses the highest recorded throughput across pipelines.
Energy and power use those same cases, without selecting a different pipeline for each metric.
Validation labels remain visible for the selected cases. Units appear in axis
titles and bar values; axis tick labels contain numbers only.

Throughput-stage bars touch within each network in both absolute and advantage
plots. Bar-label widths are measured before export; dense figures expand only
as needed to contain values and invalidity annotations. Legends use the available
width and balance entries across rows when wrapping is necessary.

Comparison panels share y-axis titles and tick labels. Each network in the
pipeline plot retains its own scale, with y-axis ticks on both architecture rows.
Only the bottom row shows the shared x-axis ticks. Network names appear above
the top row; architecture names appear on the right. That plot
trims unused vertical range and is intended to span both manuscript columns.

Failed validation is labeled `P. INVALID`; other non-PASS statuses keep their
names, and missing measurements show `N/A`. Relative bars require both
compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
must agree before being collapsed. `FAIL` alone does not identify the cause.

Plots consume the CSV's current contents; they do not run benchmarks or establish
publication provenance. The checked-in dataset is intentional. See the
[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).

## Metric interpretation

See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, units, numerical validity, and result provenance.