# Pimcomp model comparison

Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
Run commands from the repository root with the repository virtual environment,
built release Raptor, Pimcomp, and `pimsim-nn`.

## Suite runner

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
```

Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
2/4/8, four workers, and a functional throughput batch of 64.
Use `--help` for the complete option reference.

| Option | Purpose |
|---|---|
| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
| `--common-dir PATH` | Override the shared reference-artifact root. |
| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
| `--raptor-only` | Run Raptor without invoking Pimcomp. |
| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
| `--dry-run` | Print commands without modifying files. |
| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |

Use the equals form for forwarded flags, for example
`--raptor-extra-arg=--pim-detect-communication-deadlock`.

With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
beside this script. An explicit `--out-dir PATH` writes
`PATH/results_comparison.csv`; selecting a subset updates only those rows.
Default model artifacts remain under `validation/networks/pimcomp_models`.
Artifacts use
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
written report, including failures without simulation artifacts. Missing or
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
compiler, simulation, or functional comparison fails.

Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
with a regeneration diagnostic. `--only raptor` requires compatible reports;
`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
results. Ordinary runs also rerun the group's first case, even if an old report
exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
Tool/build revisions and manually edited reference artifacts are not hashed;
rerun the relevant compiler after those change.

## One-model comparator

`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
Use `--help` for hardware and simulation options. The suite runner supplies
these automatically.
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
`validate.py` uses `1e-3`/`1e-5`.

## Comparison charts

```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
```

Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
`results_comparison.csv` beside the script and writes `plots/png/` and
`plots/svg/` there. Use `--input PATH` for a runner-generated CSV and
`--output-dir PATH` to override the destination. The plotter requires the full
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
configurations are rejected before plots are written.

| Plot family | Meaning |
|---|---|
| `validation_status` | Recorded functional status for each compiler and experiment. |
| `latency_*` | Single-inference latency, latency mode only. |
| `throughput_*` | Inferences/s, throughput mode only. |
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
| `power_*`, `energy_*` | Average power and energy per inference. |
| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |

Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
winning factor minus one, with parity at zero.

Failed validation is labeled `INVALID`; other non-PASS statuses keep their
names, and missing measurements show `N/A`. Relative bars require both
compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
must agree before being collapsed. `FAIL` alone does not identify the cause.

Plots consume the CSV's current contents; they do not run benchmarks or establish
publication provenance. The checked-in dataset is intentional. See the
[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).

## Metric interpretation

See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, units, numerical validity, and result provenance.