rework documentation

add plot generation scripts adjust validation tools add up-to-date results
2026-10-19 16:34:01 +02:00
parent 9053cc5e8b
commit 034c197260
33 changed files with 3077 additions and 1798 deletions
@@ -0,0 +1 @@
+plots/
@@ -1,78 +1,73 @@
 # Raptor compiler ablation

-`run_ablation.py` performs the complete synchronization/Spatial-planning
-ablation study on the Pimcomp model suite. By default it runs `vgg8`,
-`resnet18`, `resnet34`, and `googlenet` across `arch-a` and `arch-b`, latency,
-and throughput pipeline 4. `arch-c` and `yolo11n` are run only when selected
-explicitly.
-Latency is pipeline 1; the wrapper invokes the suite runner separately for
-latency and throughput pipeline 4.
+Run the synchronization/Spatial-planning ablation from the repository root:

 ```bash
-.venv/bin/python validation/tools/pim/ablation/run_ablation.py \
-  --jobs 4
+.venv/bin/python validation/tools/pim/ablation/run_ablation.py --jobs 4
 ```

-## Variants
+Defaults: `vgg8 resnet18 resnet34 googlenet`, `arch-a arch-b`, latency
+(pipeline 1), and throughput (pipeline 4). Select `yolo11n` or `arch-c`
+explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.

 | Variant | Raptor options |
 |---|---|
-| `baseline` | None. |
+| `baseline` | None |
 | `no-sync` | `--pim-disable-synchronization` |
 | `no-plan` | `--pim-disable-spatial-planning` |
-| `no-sync-no-plan` | Both options. |
+| `no-sync-no-plan` | Both options |

-Every variant runs Raptor only. Pimcomp is not compiled, validated, or
-simulated; its validation column is `SKIPPED`, not `FAIL`. Reference inputs and
-outputs are generated once under the shared common-artifact root and reused by
-every variant. Ctrl+C terminates the active variant and all of its worker jobs.
+## Options and outputs

-Removing synchronization deliberately removes host-buffer lifetime and pipeline
-restart guarantees. The `no-sync` variants can therefore fail numerical
-validation, especially across repeated throughput inputs. Those failures remain
-`FAIL` and cause a nonzero exit; a completed ablation is not proof that every
-variant is correct. Baseline and `no-plan` must still pass numerical validation.
-The combined CSV retains `functional_validation` for every row. Performance
-percentages from failing variants describe incorrect programs, not valid
-speedups. Throughput deadline averages likewise do not establish a converged
-steady-state period.
+- `--variant NAME`: repeat to select variants; baseline is always included.
+- `--out-dir PATH`: artifact root; defaults to `validation/networks/pimcomp_models`.
+- `--dry-run`: print commands without modifying files.

-The percentage baseline is `baseline`, so its available values are `+0.00%`.
-Every other
-variant reports the signed percentage difference of its Raptor metrics from
-that reference for the same model, architecture, mode, and pipeline. Positive
-values mean the metric is higher; negative values mean it is lower.
+Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
+`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
+replaces the default latency/pipeline-4 case set. `--only` is rejected because
+the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.

-## Options
+Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
+reference inputs, outputs, and runners are shared under `artifacts/common/`.
+Per-variant summaries use `<out-dir>/ablation/<variant>/results_comparison.csv`.
+Normal comparison results are preserved.

-| Option | Description and default |
-|---|---|
-| `-h`, `--help` | Show help and exit. |
-| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
-| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
-| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
-| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
+By default, the combined [results_ablation.csv](results_ablation.csv) is written
+beside this script and printed after the run. An explicit `--out-dir PATH`
+writes `PATH/results_ablation.csv`, keeping independent experiments separate.
+Default model artifacts and per-variant summaries remain in the locations above.
+The aggregate records case identifiers,
+`functional_validation`, and signed percentage changes in latency, throughput,
+power, and energy against the matching baseline case. Positive means higher;
+baseline values are zero.

-All options of
-[`run_pimcomp_models.py`](../pimcomp/compare/README.md), including
-`--archs`, `--mode`, `--pipeline`, `--no-fast`, and
-`--raptor-extra-arg=ARG`, are forwarded to each variant. `--out-dir`,
-`--variant`, and `--dry-run` belong to this wrapper; `--only` is reserved for
-the suite runner's comparison mode and is rejected by this wrapper, which
-always invokes `--raptor-only`. If neither `--mode` nor `--pipeline` is
-forwarded, the wrapper uses its default latency and throughput/pipeline-4
-case set. Supplying either option overrides that default and is passed through
-as one suite-runner invocation per variant.
+Disabling synchronization removes host-buffer lifetime and pipeline restart
+guarantees, so those variants may fail numerical validation and exit nonzero.
+Baseline and `no-plan` must pass. Percentages from failing variants are not
+valid speedups; deadline averages do not prove steady-state convergence.

-The feature-full `baseline` variant is rerun with the same selected cases and
-forwarded options as the disabled variants. All four variants are stored below
-each model's `artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared
-reference artifacts remain under each model's `artifacts/common` directory.
-Normal comparison artifacts and summaries are preserved. Compact per-variant
-comparison summaries remain under
-`<out-dir>/ablation/<variant>/results_comparison.csv` after aggregation.
-The combined table is written to `<out-dir>/results_ablation.csv`; it contains the
-variant, case identifiers, actual `functional_validation` status, and signed
-percentage `latency`, `throughput`, `power`, and `energy` columns. These are
-Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor
-only.
+## Plots
+
+```bash
+.venv/bin/python validation/tools/pim/ablation/generate_plots.py
+# For a custom experiment root:
+.venv/bin/python validation/tools/pim/ablation/generate_plots.py \
+  --input /path/to/experiment/results_ablation.csv
+```
+
+Requires Matplotlib from the root `requirements.txt`. By default, reads the
+canonical CSV beside this script. Plots go to
+`plots/png/` and `plots/svg/` beside the selected CSV; override with
+`--output-dir PATH`.
+Single-variant comparison CSVs are not accepted.
+
+The checked-in CSV is an intentional result dataset; plotting uses its current
+contents without running benchmarks. Invalid reciprocal percentage pairs render
+as `N/A`; individual legitimate extreme changes are preserved. See
+[performance availability](../../../../docs/evaluation/methodology.md#functional-validity-and-performance-availability).
+
+Plots separate modes and pipelines. Throughput-mode latency means output
+interval. Failed validation is labeled `INVALID`, other non-PASS statuses keep
+their names, and missing measurements show `N/A`. See the
+[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
@@ -0,0 +1,156 @@
+#!/usr/bin/env python3
+"""Plot percentage changes from an experiment's combined results_ablation.csv."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import math
+import sys
+import warnings
+from dataclasses import dataclass
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from plot_style import COLORS, configure_style, save_figure, value_label
+
+import matplotlib.pyplot as plt
+from matplotlib.patches import Patch
+from matplotlib.ticker import PercentFormatter
+
+from run_ablation import ABLATION_DEFAULT_MODELS, DEFAULT_RESULTS_PATH, RESULT_FIELDS, VARIANTS
+
+
+VARIANT_COLORS = {variant.name: color for variant, color in zip(VARIANTS, COLORS)}
+STATUSES = ("PASS", "FAIL", "SKIPPED", "NA")
+
+
+@dataclass(frozen=True)
+class Result:
+    variant: str
+    arch: str
+    model: str
+    mode: str
+    pipeline: int
+    validation: str
+    metrics: dict[str, float]
+
+
+def percentage(value: str) -> float:
+    if value == "NA":
+        return math.nan
+    if not value.endswith("%"):
+        raise ValueError(f"Expected a percentage or NA, got {value!r}")
+    parsed = float(value[:-1])
+    if not math.isfinite(parsed):
+        raise ValueError(f"Non-finite percentage: {value!r}")
+    return parsed
+
+
+def load_results(path: Path) -> list[Result]:
+    rows = []
+    seen = set()
+    invalid_pairs = []
+    with path.open(newline="", encoding="utf-8-sig") as stream:
+        reader = csv.DictReader(stream)
+        if not {"variant", *RESULT_FIELDS} <= set(reader.fieldnames or []):
+            raise ValueError("Expected the combined ablation CSV with variant and percentage columns; "
+                             "a per-variant results_comparison.csv cannot be used here.")
+        for row in reader:
+            variant, mode, status = row["variant"], row["mode"], row["functional_validation"]
+            if variant not in VARIANT_COLORS or mode not in ("latency", "throughput") or status not in STATUSES:
+                raise ValueError(f"Unknown ablation variant, mode, or validation status: {variant}, {mode}, {status}")
+            pipeline = int(row["raptor_pipeline"])
+            key = (variant, row["arch"], row["model"], mode, pipeline)
+            if key in seen:
+                raise ValueError(f"Duplicate ablation configuration: {key}")
+            seen.add(key)
+            metrics = {metric: percentage(row[metric])
+                       for metric in ("latency", "throughput", "power", "energy")}
+            # Each may round to -100% alone; reciprocal metrics cannot both do so.
+            if mode == "throughput" and metrics["latency"] == metrics["throughput"] == -100:
+                metrics["latency"] = metrics["throughput"] = math.nan
+                invalid_pairs.append(key)
+            rows.append(Result(variant, row["arch"], row["model"], mode, pipeline, status, metrics))
+    if not rows:
+        raise ValueError("The ablation CSV contains no results")
+    if invalid_pairs:
+        warnings.warn(f"{len(invalid_pairs)} invalid reciprocal latency/throughput percentage pairs; "
+                      f"displaying N/A. Examples: {invalid_pairs[:3]}", stacklevel=2)
+    return rows
+
+
+def draw_metric(results: list[Result], metric: str, ylabel: str, output_dir: Path) -> None:
+    architectures = sorted({row.arch for row in results})
+    present_models = {row.model for row in results}
+    models = [model for model in ABLATION_DEFAULT_MODELS if model in present_models]
+    models += sorted(present_models - set(models))
+    variants = [name for name in VARIANT_COLORS if any(row.variant == name for row in results)]
+    lookup = {(row.arch, row.model, row.variant): row for row in results}
+    values = [row.metrics[metric] for row in results if math.isfinite(row.metrics[metric])]
+    lower, upper = min(0, min(values, default=0)), max(0, max(values, default=0))
+    padding = max(upper - lower, 1) * 0.35
+    fig, axes = plt.subplots(1, len(architectures), sharey=True, squeeze=False,
+                             figsize=(7 * len(architectures), 5.2), layout="constrained")
+    handles = [Patch(facecolor=VARIANT_COLORS[name], label=name) for name in variants]
+    if len(values) < len(architectures) * len(models) * len(variants):
+        handles.append(Patch(facecolor="white", edgecolor="#B2BAC5", label="N/A: missing measurement"))
+    fig.legend(handles=handles, loc="outside upper center", ncol=4, frameon=False, fontsize=9.5)
+    width = 0.8 / len(variants)
+    mode, pipeline = results[0].mode, results[0].pipeline
+    for ax, arch in zip(axes[0], architectures):
+        ax.set_axisbelow(True)
+        ax.set_ylim(lower - padding if lower < 0 else 0, upper + padding)
+        ax.set_xlim(-0.5, len(models) - 0.5)
+        ax.set_xticks(range(len(models)), [model.upper() for model in models], fontsize=9.5)
+        ax.set_xlabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
+                      f"{mode.capitalize()}, {pipeline} stage" + ("s" if pipeline != 1 else ""),
+                      fontsize=11, weight="bold", labelpad=8)
+        ax.set_ylabel(ylabel, fontsize=10.5)
+        ax.yaxis.set_major_formatter(PercentFormatter(xmax=100, decimals=0))
+        ax.tick_params(axis="both", length=0, pad=6, labelleft=True)
+        ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
+        ax.axhline(0, color="#536478", linewidth=1.2, linestyle="--")
+        for index, model in enumerate(models):
+            if index % 2 == 0:
+                ax.axvspan(index - 0.5, index + 0.5, facecolor="#F7F9FC", zorder=0)
+            for offset, variant in enumerate(variants):
+                x = index + (offset - (len(variants) - 1) / 2) * width
+                row = lookup.get((arch, model, variant))
+                value = row.metrics[metric] if row else math.nan
+                if not math.isfinite(value):
+                    ax.text(x, padding * 0.05, "N/A", rotation=90, ha="center", va="bottom",
+                            fontsize=8, color="#85909F")
+                    continue
+                ax.bar(x, value, width=width * 0.9, color=VARIANT_COLORS[variant],
+                       edgecolor="none", linewidth=0, zorder=2)
+                direction = -1 if value < 0 else 1
+                value_label(ax, x, value + direction * padding * 0.05, f"{value:+,.2f}%",
+                            row.validation, direction)
+    save_figure(fig, f"{mode}_pipeline{pipeline}_{metric}", output_dir)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--input", type=Path, default=DEFAULT_RESULTS_PATH)
+    parser.add_argument("--output-dir", type=Path)
+    args = parser.parse_args()
+    if args.output_dir is None:
+        args.output_dir = args.input.parent / "plots"
+    configure_style()
+    results = load_results(args.input)
+    print(f"Results: {len(results)} configurations, "
+          f"{sum(row.validation != 'PASS' for row in results)} without PASS validation")
+    for mode, pipeline in sorted({(row.mode, row.pipeline) for row in results}):
+        rows = [row for row in results if row.mode == mode and row.pipeline == pipeline]
+        for metric, label in (
+            ("latency", "Single-inference latency" if mode == "latency" else "Output interval"),
+            ("throughput", "Throughput"), ("power", "Average power"), ("energy", "Energy per inference"),
+        ):
+            if metric == "throughput" and mode == "latency":
+                continue
+            draw_metric(rows, metric, f"{label} change from baseline (%)", args.output_dir)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,65 @@
+variant,arch,model,mode,raptor_pipeline,functional_validation,latency,throughput,power,energy
+baseline,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
+no-plan,arch-a,googlenet,latency,1,PASS,+11513.29%,NA,-69.42%,+3452.69%
+no-sync-no-plan,arch-a,googlenet,latency,1,PASS,+11512.99%,NA,-69.42%,+3452.59%
+baseline,arch-a,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-a,googlenet,throughput,4,FAIL,+26.17%,-20.74%,+9.97%,+18.50%
+no-plan,arch-a,googlenet,throughput,4,PASS,-100.00%,-100.00%,-62.73%,-100.00%
+no-sync-no-plan,arch-a,googlenet,throughput,4,FAIL,+4094.63%,-97.62%,-59.86%,+1311.01%
+baseline,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
+no-sync-no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
+baseline,arch-a,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-a,resnet18,throughput,4,FAIL,+110.65%,-52.55%,+43.25%,+177.48%
+no-plan,arch-a,resnet18,throughput,4,PASS,+1598.93%,-94.12%,-51.04%,+664.87%
+no-sync-no-plan,arch-a,resnet18,throughput,4,FAIL,+1174.21%,-92.16%,-45.06%,+543.74%
+baseline,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
+no-sync-no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
+baseline,arch-a,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-a,resnet34,throughput,4,FAIL,+33.65%,-25.21%,+31.76%,+15.56%
+no-plan,arch-a,resnet34,throughput,4,PASS,+1726.82%,-94.52%,-54.50%,+455.98%
+no-sync-no-plan,arch-a,resnet34,throughput,4,FAIL,+813.41%,-89.04%,-49.77%,+206.84%
+baseline,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
+no-sync-no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
+baseline,arch-a,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-a,vgg8,throughput,4,PASS,+2.33%,-1.72%,+18.49%,-0.98%
+no-plan,arch-a,vgg8,throughput,4,PASS,+1251.16%,-92.58%,-56.68%,+577.04%
+no-sync-no-plan,arch-a,vgg8,throughput,4,PASS,+1260.47%,-92.63%,-53.36%,+467.83%
+baseline,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
+no-plan,arch-b,googlenet,latency,1,PASS,+9849.64%,NA,-70.51%,+2833.89%
+no-sync-no-plan,arch-b,googlenet,latency,1,PASS,+9849.55%,NA,-70.51%,+2833.89%
+baseline,arch-b,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-b,googlenet,throughput,4,FAIL,+11.45%,-10.29%,+21.82%,+19.27%
+no-plan,arch-b,googlenet,throughput,4,PASS,-100.00%,-100.00%,-63.39%,-100.00%
+no-sync-no-plan,arch-b,googlenet,throughput,4,FAIL,+3348.28%,-97.10%,-61.58%,+1026.09%
+baseline,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
+no-plan,arch-b,resnet18,latency,1,PASS,+2935.63%,NA,-63.12%,+1019.81%
+no-sync-no-plan,arch-b,resnet18,latency,1,PASS,+2936.09%,NA,-63.12%,+1019.97%
+baseline,arch-b,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-b,resnet18,throughput,4,FAIL,+8.03%,-7.44%,+47.59%,+40.61%
+no-plan,arch-b,resnet18,throughput,4,PASS,+1511.85%,-93.80%,-52.93%,+560.80%
+no-sync-no-plan,arch-b,resnet18,throughput,4,FAIL,+867.12%,-89.67%,-45.41%,+359.83%
+baseline,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-plan,arch-b,resnet34,latency,1,PASS,+2551.16%,NA,-62.51%,+893.70%
+no-sync-no-plan,arch-b,resnet34,latency,1,PASS,+2550.91%,NA,-62.51%,+893.62%
+baseline,arch-b,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-b,resnet34,throughput,4,FAIL,+28.25%,-22.13%,+49.76%,+57.30%
+no-plan,arch-b,resnet34,throughput,4,PASS,+2427.17%,-96.05%,-49.58%,+938.62%
+no-sync-no-plan,arch-b,resnet34,throughput,4,FAIL,+531.79%,-84.19%,-42.32%,+197.08%
+baseline,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-sync,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
+no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
+no-sync-no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
+baseline,arch-b,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
+no-sync,arch-b,vgg8,throughput,4,PASS,+23.91%,-19.82%,+25.67%,+23.93%
+no-plan,arch-b,vgg8,throughput,4,PASS,+1158.70%,-92.07%,-58.95%,+408.33%
+no-sync-no-plan,arch-b,vgg8,throughput,4,PASS,+1172.83%,-92.17%,-55.83%,+327.00%
@@ -45,6 +45,7 @@ VARIANTS = (
 VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS}
 COMPARISON_RESULTS_FILENAME = "results_comparison.csv"
 ABLATION_RESULTS_FILENAME = "results_ablation.csv"
+DEFAULT_RESULTS_PATH = Path(__file__).with_name(ABLATION_RESULTS_FILENAME)
 CASE_FIELDS = ("arch", "model", "mode", "raptor_pipeline")
 PERCENTAGE_FIELDS = (
    ("raptor_latency_ms", "latency"),
@@ -68,8 +69,7 @@ def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[
    parser.add_argument(
        "--out-dir",
        type=Path,
-        default=DEFAULT_OUT_DIR,
-        help="Artifact root (default: validation/networks/pimcomp_models).",
+        help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_ablation.csv beside this script.",
    )
    add_models_argument(parser, ABLATION_DEFAULT_MODELS)
    parser.add_argument(
@@ -160,9 +160,10 @@ def percentage_difference(value: str | None, reference: str | None) -> str:
        baseline = float(reference) if reference is not None else math.nan
    except ValueError:
        return "NA"
-    if not math.isfinite(current) or not math.isfinite(baseline) or baseline == 0:
+    if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0:
        return "NA"
-    return f"{(current - baseline) / baseline * 100:+.2f}%"
+    change = (current / baseline - 1) * 100
+    return f"{change:+.2f}%" if math.isfinite(change) else "NA"


 def aggregate_results(
@@ -170,6 +171,7 @@ def aggregate_results(
    variants: list[Variant],
    selected_cases: frozenset[tuple[str, str]] | None = None,
    selected_models: frozenset[str] | None = None,
+    output: Path | None = None,
 ) -> tuple[Path | None, list[str]]:
    fields: list[str] | None = None
    rows_by_variant: dict[str, list[dict[str, str]]] = {}
@@ -233,7 +235,8 @@ def aggregate_results(
            int(row["raptor_pipeline"]),
        )
    )
-    output = out_dir / ABLATION_RESULTS_FILENAME
+    output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME
+    output.parent.mkdir(parents=True, exist_ok=True)
    with output.open("w", newline="", encoding="utf-8") as stream:
        writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n")
        writer.writeheader()
@@ -247,7 +250,7 @@ def main(argv: list[str] | None = None) -> int:
        print(f"Missing Pimcomp runner: {RUNNER}", file=sys.stderr)
        return 1
    variants = selected_variants(args.variants)
-    out_dir = args.out_dir.resolve()
+    out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR
    case_sets = runner_argument_sets(forwarded)
    if args.dry_run:
        for variant in variants:
@@ -299,14 +302,17 @@ def main(argv: list[str] | None = None) -> int:
        variants,
        selected_cases,
        frozenset(args.models),
+        output=DEFAULT_RESULTS_PATH if args.out_dir is None else None,
    )
    failed.extend(aggregation_failures)
    if output is not None:
-        print(f"Ablation results: {output}")
+        print(f"[{output.name}]")
+        print(output.read_text(encoding="utf-8"), end="", flush=True)
    if failed:
        print("Failed: " + "; ".join(failed), file=sys.stderr)
-        return 1
-    return 0
+    if output is not None:
+        print(f"Ablation results: {output}", flush=True)
+    return 1 if failed else 0


 if __name__ == "__main__":
@@ -0,0 +1 @@
+plots/
@@ -1,62 +1,108 @@
 # Pimcomp model comparison

-These scripts compare Raptor with Pimcomp on the models in
-`validation/networks/pimcomp_models`.
-
-`run_pimcomp_models.py` runs the supported model suite across the configured
-architectures and simulation modes. `compare_raptor_pimcomp_model.py` is the
-lower-level one-model comparison used by the suite runner.
+Compare Raptor and Pimcomp on the [model suite](../../../../networks/pimcomp_models/README.md).
+Run commands from the repository root with the repository virtual environment,
+built release Raptor, Pimcomp, and `pimsim-nn`.

 ## Suite runner

-Run all five models, Arch-A and Arch-B, latency, and throughput:
-
 ```bash
 .venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
 ```

-### Options
+Defaults: all five models, Arch-A/B, latency pipeline 1, throughput pipelines
+2/4/8, four workers, and a functional throughput batch of 64.
+Use `--help` for the complete option reference.

-| Option | Description and default |
+| Option | Purpose |
 |---|---|
-| `-h`, `--help` | Show help and exit. |
-| `--out-dir PATH` | Suite root; artifacts are placed below `<out-dir>/<model>/artifacts/`. Default: `validation/networks/pimcomp_models`. |
-| `--common-dir PATH` | Shared root for per-model reference artifacts. Default: `<out-dir>/<model>/artifacts/common`. |
-| `--models MODEL [...]` | Models to run: `vgg8`, `resnet18`, `resnet34`, `googlenet`, `yolo11n`. Default: all five; pass a subset to select specific models. |
-| `--archs ARCH [...]` | Pimcomp hardware profiles to run. Default: `arch-a arch-b`. |
-| `--mode {latency,throughput} [...]` | Simulation modes. Default: both. |
-| `--only {raptor,pimcomp}` | Re-run only one compiler side and reuse the other side's existing report. Default: run both sides. |
-| `--raptor-only` | Run Raptor without compiling, validating, or simulating Pimcomp. Default: off. |
-| `--pipeline {1,2,4,8}` | Select one Raptor pipeline. Default: latency pipeline 1 and throughput pipelines 2, 4, and 8. |
-| `--pimsim-time-ms INT` | Throughput convergence deadline. Default: `1000`. |
-| `--batch-size INT` | Functional throughput batch size. Default: `128`. |
-| `--timeout-seconds FLOAT` | Per-stage timeout; `0` means unlimited. Default: `0`. |
-| `-j INT`, `--jobs INT` | Parallel comparison workers. Default: `4`. |
-| `--clean` | Remove generated comparison artifacts and summaries, then exit. Default: off. |
-| `--ablation-variant NAME` | Put generated artifacts below `<mode>[/pipelineN]/ablation/NAME` and write the summary below `ablation/NAME`. Default: none. |
-| `--dry-run` | Print commands without modifying files. Default: off. |
-| `--no-fast` | Disable fast throughput convergence. Default: off. |
-| `--raptor-extra-arg=ARG` | Extra Raptor compiler argument; repeat for multiple arguments. Default: none. |
+| `--models MODEL [...]`, `--archs ARCH [...]` | Select models and hardware profiles. |
+| `--mode {latency,throughput} [...]`, `--pipeline {1,2,4,8}` | Select experiments. |
+| `--out-dir PATH` | Suite root; default: `validation/networks/pimcomp_models`. |
+| `--common-dir PATH` | Override the shared reference-artifact root. |
+| `--only {raptor,pimcomp}` | Rerun one compiler and reuse the other's existing report. |
+| `--raptor-only` | Run Raptor without invoking Pimcomp. |
+| `--batch-size N`, `--jobs N` | Set functional throughput batch size and worker count. |
+| `--pimsim-time-ms N` | Throughput convergence deadline; default: 1000 ms. |
+| `--no-fast` | Use full-duration throughput simulation for authoritative experiments. |
+| `--timeout-seconds N` | Per-stage timeout; default: 0 (unlimited). |
+| `--raptor-extra-arg=ARG` | Forward a compiler option; repeat as needed. |
+| `--dry-run` | Print commands without modifying files. |
+| `--clean` | Remove generated comparison artifacts, summaries, and stale locks. |

-Arguments beginning with `--` must use the equals form when passed through:
+Use the equals form for forwarded flags, for example
+`--raptor-extra-arg=--pim-detect-communication-deadlock`.

-```bash
-.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py \
-  --models vgg8 \
-  --raptor-extra-arg=--pim-disable-synchronization
-```
+With no `--out-dir`, the runner writes [results_comparison.csv](results_comparison.csv)
+beside this script. An explicit `--out-dir PATH` writes
+`PATH/results_comparison.csv`; selecting a subset updates only those rows.
+Default model artifacts remain under `validation/networks/pimcomp_models`.
+Artifacts use
+`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
+under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
+and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
+Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
+written report, including failures without simulation artifacts. Missing or
+unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
+compiler, simulation, or functional comparison fails.

-The runner writes `results_comparison.csv` under the selected result root. It reuses
-shared model inputs, outputs, and reference runners, and reuses Pimcomp
-artifacts between pipelines in the same model/architecture/mode group. With
-`--raptor-only`, only Raptor results are generated and Pimcomp is not invoked.
-All generated files are kept below each model's `artifacts/` directory;
-`--clean` also removes any `.lock` files left by an interrupted reference
-generation.
+Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
+mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
+Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
+with a regeneration diagnostic. `--only raptor` requires compatible reports;
+`--only pimcomp` reruns Pimcomp for each group and preserves existing Raptor
+results. Ordinary runs also rerun the group's first case, even if an old report
+exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
+Tool/build revisions and manually edited reference artifacts are not hashed;
+rerun the relevant compiler after those change.

 ## One-model comparator

-`compare_raptor_pimcomp_model.py` compares one ONNX model. Its required
-arguments are `--model PATH` and `--out-dir PATH`; use `--help` for the full
-lower-level interface. The suite runner supplies the model, hardware profile,
-simulation mode, and reuse paths automatically.
+`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
+Use `--help` for hardware and simulation options. The suite runner supplies
+these automatically.
+The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
+`validate.py` uses `1e-3`/`1e-5`.
+
+## Comparison charts
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
+```
+
+Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
+`results_comparison.csv` beside the script and writes `plots/png/` and
+`plots/svg/` there. Use `--input PATH` for a runner-generated CSV and
+`--output-dir PATH` to override the destination. The plotter requires the full
+40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
+pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
+configurations are rejected before plots are written.
+
+| Plot family | Meaning |
+|---|---|
+| `validation_status` | Recorded functional status for each compiler and experiment. |
+| `latency_*` | Single-inference latency, latency mode only. |
+| `throughput_*` | Inferences/s, throughput mode only. |
+| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
+| `power_*`, `energy_*` | Average power and energy per inference. |
+| `raptor_pipeline_throughput` | Raptor throughput versus stages; stage 1 is derived as `1000 / latency_ms`, stages 2/4/8 are measured. |
+
+Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
+`_comparison` shows ratios and `_diverging` shows equal-length bars for equal
+advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
+latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
+winning factor minus one, with parity at zero.
+
+Failed validation is labeled `INVALID`; other non-PASS statuses keep their
+names, and missing measurements show `N/A`. Relative bars require both
+compilers to pass to be valid. Repeated Pimcomp batch measurements and statuses
+must agree before being collapsed. `FAIL` alone does not identify the cause.
+
+Plots consume the CSV's current contents; they do not run benchmarks or establish
+publication provenance. The checked-in dataset is intentional. See the
+[result methodology](../../../../../docs/evaluation/methodology.md#canonical-result-datasets-and-output-roots).
+
+## Metric interpretation
+
+See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
+for measurement windows, units, numerical validity, and result provenance.
@@ -4,6 +4,7 @@ from __future__ import annotations
 import argparse
 import fcntl
 import gzip
+import hashlib
 import importlib.util
 import json
 import mmap
@@ -1162,6 +1163,60 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None:
    return Path(value) if value else None


+def pimcomp_reuse_context(
+    args: argparse.Namespace,
+    model: Path,
+    functional_model: Path,
+    hardware: dict[str, int],
+) -> dict[str, Any]:
+    files = {"model": model, "functional_model": functional_model,
+             "config": args.pimcomp_config}
+    digests = {}
+    for path in set(files.values()):
+        with path.open("rb") as stream:
+            digests[path] = hashlib.file_digest(stream, "sha256").hexdigest()
+    return {
+        "version": 1,
+        "files": {name: {"path": str(path), "sha256": digests[path]}
+                  for name, path in files.items()},
+        "hardware": hardware,
+        "settings": {name: getattr(args, name) for name in (
+            "pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline",
+            "pimcomp_replication", "pimcomp_model_name", "seed", "batch_size",
+            "threshold", "rtol", "no_fast", "skip_pimsim_nn",
+        )},
+    }
+
+
+def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]:
+    try:
+        report = json.loads(path.read_text(encoding="utf-8"))
+        if (not isinstance(report, dict) or context is None
+            or report.get("pimcomp_reuse_context") != context
+            or report.get("pimcomp_model_source") != "original_onnx"):
+            raise ValueError("model, configuration, or experiment settings differ; legacy reports lack fingerprints")
+        for name, kind in (("pimcomp_validation", dict), ("pimcomp_performance", dict),
+                           ("pimcomp_instruction_summary", dict), ("paths", dict),
+                           ("failures", list), ("steps", list)):
+            if not isinstance(report.get(name), kind):
+                raise ValueError(f"missing or invalid {name}")
+        validation = CompareResult(**report["pimcomp_validation"])
+        if (type(validation.passed) is not bool or not isinstance(validation.max_diffs, dict)
+            or validation.status not in ("done", "failed", "skipped")):
+            raise ValueError("invalid functional validation result")
+        if not report["pimcomp_performance"]:
+            raise ValueError("missing performance result or failure status")
+        summary = report["pimcomp_instruction_summary"]
+        if not {"active_cores", "total_instructions", "op_counts"} <= summary.keys():
+            raise ValueError("incomplete instruction summary")
+        restore_side_records(report, "pimcomp", [], [])
+        for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"):
+            path_from_report(report, name)
+        return report
+    except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc:
+        raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc
+
+
 def restore_side_records(
    report: dict[str, Any],
    side: str,
@@ -1579,6 +1634,11 @@ def main():
    if loaded_hardware is not None:
        hardware = loaded_hardware

+    reuse_context = None if args.raptor_only else try_stage(
+        failures, "Identify Pimcomp experiment", pimcomp_reuse_context,
+        args, model_path, functional_model_path, hardware,
+    )
+
    model_io = try_stage(
        failures,
        "Load model inputs",
@@ -1597,12 +1657,7 @@ def main():
    run_pimcomp = not args.raptor_only and not reuse_pimcomp
    if reuse_pimcomp:
        reused_pimcomp_report_path = reused_pimcomp_report_path.resolve()
-        if not reused_pimcomp_report_path.exists():
-            raise ValueError(f"Missing Pimcomp report: {reused_pimcomp_report_path}")
-        with open(reused_pimcomp_report_path, "r", encoding="utf-8") as f:
-            reused_pimcomp = json.load(f)
-        if reused_pimcomp.get("pimcomp_model_source") != "original_onnx":
-            raise ValueError("Reused Pimcomp artifacts were not generated from the original ONNX model")
+        reused_pimcomp = load_reused_pimcomp_report(reused_pimcomp_report_path, reuse_context)
        pimcomp_validation = CompareResult(**reused_pimcomp["pimcomp_validation"])
        pimcomp_perf = reused_pimcomp["pimcomp_performance"]
        pimcomp_instr = reused_pimcomp["pimcomp_instruction_summary"]
@@ -2011,6 +2066,7 @@ def main():
        "pimcomp_pipeline": args.pimcomp_pipeline,
        "pimcomp_replication": args.pimcomp_replication,
        "pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx",
+        "pimcomp_reuse_context": reuse_context,
        "pimcomp_config": str(args.pimcomp_config),
        "raptor_extra_args": args.raptor_extra_arg,
        "reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
@@ -0,0 +1,514 @@
+#!/usr/bin/env python3
+"""Generate publication-ready Raptor vs Pimcomp comparison charts.
+
+Usage:
+    .venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py
+
+Dependencies: matplotlib (no pandas or seaborn required).
+By default, reads results_comparison.csv beside this script and writes plots/
+beside it, with outputs grouped in png/ and svg/ under the output directory.
+
+YOLO models without Pimcomp measurements get separate Raptor-only charts.
+Comparisons use vertical bars for ratios, diverging advantages, and absolute
+values, with compiler colors and patterns for the Raptor pipeline stages.
+Throughput-mode latency is the output interval, a bottleneck-stage proxy,
+not end-to-end inference latency. Separate throughput plots use inferences/s.
+Energy is reported per inference; power is averaged over the simulation.
+Numerical results whose compiler did not pass functional validation have a red
+status following their value label; missing Pimcomp
+measurements appear as 'N/A'. The fixed compiler palette is shared by
+all metrics. Relative plots use Pimcomp / Raptor for latency, power and energy,
+and Raptor / Pimcomp for throughput, so >1 favors Raptor. Diverging plots put the winner's
+factor above (Raptor) or below (Pimcomp) parity, with 1x at the zero line.
+"""
+from __future__ import annotations
+
+import argparse
+import csv
+import math
+import sys
+from dataclasses import dataclass
+from itertools import groupby
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib.patches import Patch, Rectangle
+from matplotlib.lines import Line2D
+from matplotlib.ticker import FuncFormatter, MultipleLocator
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+from plot_style import COLORS, HATCH_COLOR, TEXT_COLOR, configure_style, fit_value_labels, save_figure, value_label
+
+
+COMPILER_COLORS = {"raptor": COLORS[0], "pimcomp": COLORS[2]}
+STAGE_PATTERNS = {1: "", 2: "..", 4: "//", 8: "\\\\"}
+ARCHITECTURES = ("arch-a", "arch-b")
+MODELS = ("vgg8", "resnet18", "resnet34", "googlenet", "yolo11n")
+CONFIGURATIONS = (("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8))
+LINE_COLOR = "#A8B2C0"
+VALIDATION_STYLES = {
+    "PASS": ("#168C8B", "white", ""),
+    "FAIL": ("#D18D32", HATCH_COLOR, ""),
+    "SKIPPED": ("#E7DFF0", TEXT_COLOR, ""),
+    "NA": ("#E8EDF1", TEXT_COLOR, ""),
+}
+
+
+@dataclass(frozen=True)
+class Result:
+    arch: str
+    model: str
+    mode: str
+    pipeline: int
+    pimcomp_schedule: str
+    raptor_validation: str
+    pimcomp_validation: str
+    latency_speedup: float
+    power: float
+    energy: float
+    raptor_latency: float
+    pimcomp_latency: float
+    raptor_throughput: float
+    pimcomp_throughput: float
+    raptor_power: float
+    pimcomp_power: float
+    raptor_energy: float
+    pimcomp_energy: float
+
+    @property
+    def comparison_passed(self) -> bool:
+        return self.raptor_validation == self.pimcomp_validation == "PASS"
+
+    @property
+    def throughput_speedup(self) -> float:
+        return quotient(self.raptor_throughput, self.pimcomp_throughput)
+
+
+def positive_float(text: str | None) -> float:
+    try:
+        value = float(text)
+    except (TypeError, ValueError):
+        return math.nan
+    return value if math.isfinite(value) and value > 0 else math.nan
+
+
+def quotient(numerator: float, denominator: float) -> float:
+    if not (math.isfinite(numerator) and math.isfinite(denominator)):
+        return math.nan
+    return numerator / denominator
+
+
+def validation_status(text: str) -> str:
+    status = text.strip().upper()
+    if status not in VALIDATION_STYLES:
+        raise ValueError(f"Unknown functional-validation status: {text!r}")
+    return status
+
+
+def load_results(path: Path) -> list[Result]:
+    with path.open(newline="", encoding="utf-8-sig") as handle:
+        source = list(csv.DictReader(handle))
+    rows = []
+    expected = {(arch, model, mode, stage)
+                for arch in ARCHITECTURES for model in MODELS
+                for mode, stage in CONFIGURATIONS}
+    seen = {}
+    for line, row in enumerate(source, 2):
+        try:
+            arch, model, mode = (row[field].strip().lower() for field in ("arch", "model", "mode"))
+            stage = int(row["raptor_pipeline"])
+        except (KeyError, AttributeError, TypeError, ValueError) as exc:
+            raise ValueError(f"Invalid configuration key at CSV row {line}: {exc}") from exc
+        key = arch, model, mode, stage
+        if key not in expected:
+            raise ValueError(f"Unexpected configuration at CSV row {line}: {key}")
+        if key in seen:
+            raise ValueError(f"Duplicate configuration {key} at CSV rows {seen[key]} and {line}")
+        seen[key] = line
+        schedule = row["pimcomp_pipeline"].strip().lower()
+        if schedule != ("element" if mode == "latency" else "batch"):
+            raise ValueError(f"{key}: incompatible Pimcomp schedule {schedule!r}")
+        rows.append(Result(
+            arch=arch,
+            model=model,
+            mode=mode,
+            pipeline=stage,
+            pimcomp_schedule=schedule,
+            raptor_validation=validation_status(row["raptor_functional_validation"]),
+            pimcomp_validation=validation_status(row["pimcomp_functional_validation"]),
+            latency_speedup=quotient(positive_float(row["pimcomp_latency_ms"]),
+                                     positive_float(row["raptor_latency_ms"])),
+            power=quotient(positive_float(row["pimcomp_power_mw"]),
+                           positive_float(row["raptor_power_mw"])),
+            energy=quotient(positive_float(row["pimcomp_energy_pj"]),
+                            positive_float(row["raptor_energy_pj"])),
+            raptor_latency=positive_float(row["raptor_latency_ms"]),
+            pimcomp_latency=positive_float(row["pimcomp_latency_ms"]),
+            raptor_throughput=positive_float(row["raptor_throughput_samples_s"]),
+            pimcomp_throughput=positive_float(row["pimcomp_throughput_samples_s"]),
+            raptor_power=positive_float(row["raptor_power_mw"]),
+            pimcomp_power=positive_float(row["pimcomp_power_mw"]),
+            raptor_energy=positive_float(row["raptor_energy_pj"]),
+            pimcomp_energy=positive_float(row["pimcomp_energy_pj"]),
+        ))
+    missing = expected - seen.keys()
+    if missing:
+        raise ValueError(f"Missing {len(missing)} expected configurations. Examples: {sorted(missing)[:5]}")
+    rows.sort(key=lambda r: (ARCHITECTURES.index(r.arch), MODELS.index(r.model), r.pipeline))
+    absolute_columns(rows)
+    return rows
+
+
+def advantage_compiler(ratio: float) -> str | None:
+    if ratio > 1:
+        return "raptor"
+    if ratio < 1:
+        return "pimcomp"
+    return None
+
+
+def figure_legend(fig, results: list[Result], field: str,
+                  absolute=False, raptor_only=False):
+    compilers = ("raptor",) if raptor_only else ("raptor", "pimcomp")
+    if absolute:
+        measurements = [(row, compiler, getattr(row, f"{compiler}_{field}"))
+                        for row, compiler in absolute_columns(results, raptor_only)]
+    else:
+        measurements = [(row, None, getattr(row, field)) for row in results]
+    available = [(row, compiler, value) for row, compiler, value in measurements
+                 if math.isfinite(value)]
+    stages = {row.pipeline for row, compiler, _ in available if compiler != "pimcomp"}
+    handles = []
+    visible_compilers = {compiler if absolute else advantage_compiler(value)
+                         for _, compiler, value in available} - {None}
+    if len(visible_compilers) > 1:
+        for compiler in compilers:
+            handles.append(Patch(facecolor=COMPILER_COLORS[compiler],
+                                 label=compiler.capitalize() + ("" if absolute else " advantage")))
+    handles += [Patch(facecolor="white", edgecolor=TEXT_COLOR, hatch=STAGE_PATTERNS[depth],
+                      label=f"{depth} stage" + (" (latency)" if depth == 1 else "s"))
+                for depth in sorted(stages) if stages != {1}]
+    styles = []
+    if len(available) < len(measurements):
+        styles.append(Patch(facecolor="white", edgecolor="#B2BAC5",
+                            label="N/A: missing measurement"))
+    handles += styles
+    if handles:
+        fig.legend(handles=handles, loc="outside upper center", ncol=(len(handles) + 1) // 2,
+                   frameon=False, fontsize=9.5, handlelength=1.7, columnspacing=1.5,
+                   handletextpad=0.6, labelspacing=0.4)
+
+
+def absolute_columns(results: list[Result], raptor_only: bool = False) -> list[tuple[Result, str]]:
+    columns = []
+    for (arch, model), group in groupby(results, key=lambda row: (row.arch, row.model)):
+        rows = sorted(group, key=lambda row: row.pipeline)
+        for row in rows:
+            columns.append((row, "raptor"))
+            if row.mode == "latency" and not raptor_only:
+                columns.append((row, "pimcomp"))
+        throughput = [row for row in rows if row.mode == "throughput"]
+        if throughput and not raptor_only:
+            signatures = {
+                (row.pimcomp_schedule, row.pimcomp_validation,
+                 *(None if math.isnan(value) else value for value in (
+                     row.pimcomp_latency, row.pimcomp_throughput, row.pimcomp_power, row.pimcomp_energy)))
+                for row in throughput
+            }
+            if len(signatures) != 1:
+                raise ValueError(f"{arch}/{model}: inconsistent shared Pimcomp batch results across pipelines")
+            columns.append((throughput[0], "pimcomp"))
+    return columns
+
+
+def label_groups(ax, results: list[Result], positions: list[float] | None = None) -> None:
+    if positions is None:
+        positions = list(range(len(results)))
+    centers, labels = [], []
+    start = 0
+    for index, (model, rows) in enumerate(groupby(results, key=lambda row: row.model)):
+        end = start + len(list(rows))
+        centers.append((positions[start] + positions[end - 1]) / 2)
+        labels.append(model.upper())
+        left = ax.get_xlim()[0] if start == 0 else (positions[start - 1] + positions[start]) / 2
+        right = ax.get_xlim()[1] if end == len(results) else (positions[end - 1] + positions[end]) / 2
+        if index % 2 == 0:
+            ax.axvspan(left, right, facecolor="#F7F9FC", zorder=0)
+        if index > 0:
+            ax.axvline(left, color="#DDE3EB", linewidth=0.9)
+        start = end
+    ax.set_xticks(centers, labels, fontsize=9.5)
+
+
+def diverging_ratio(ratio: float) -> float:
+    return ratio - 1 if ratio >= 1 else 1 - 1 / ratio
+
+
+def draw_validation(results: list[Result], output_dir: Path) -> None:
+    columns = absolute_columns(results)
+    configurations = list(dict.fromkeys((row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule,
+                                        compiler) for row, compiler in columns))
+    positions = []
+    for index, (mode, _, _) in enumerate(configurations):
+        gap = 0.4 if index and mode != configurations[index - 1][0] else 0
+        positions.append(positions[-1] + 1 + gap if index else 0)
+    configuration_x = dict(zip(configurations, positions))
+    headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor"
+               else "Pimcomp\nbatch" if mode == "throughput" else compiler.capitalize()
+               for mode, stage, compiler in configurations]
+    fig, axes = plt.subplots(1, 2, figsize=(14, 3.8), layout="constrained")
+    for ax, arch in zip(axes, ARCHITECTURES):
+        ax.set_xlim(positions[0] - 0.55, positions[-1] + 0.55)
+        ax.set_ylim(len(MODELS) - 0.5, -0.5)
+        centers, modes = [], []
+        for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]):
+            group_positions = [x for _, x in group]
+            centers.append((group_positions[0] + group_positions[-1]) / 2)
+            modes.append(mode.capitalize())
+        ax.set_xticks(centers, modes, fontsize=10)
+        ax.set_xticks(positions, headers, minor=True)
+        ax.tick_params(axis="x", which="minor", top=True, labeltop=True,
+                       bottom=False, labelbottom=False, length=0, pad=6, labelsize=9)
+        ax.set_yticks(range(len(MODELS)), [model.upper() for model in MODELS], fontsize=9)
+        ax.tick_params(axis="both", length=0, pad=6)
+        architecture = "ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B"
+        ax.set_xlabel(architecture, fontsize=11, weight="bold", labelpad=8)
+        for row, compiler in columns:
+            if row.arch != arch:
+                continue
+            key = (row.mode, row.pipeline if compiler == "raptor" else row.pimcomp_schedule, compiler)
+            x, y = configuration_x[key], MODELS.index(row.model)
+            status = getattr(row, f"{compiler}_validation")
+            color, text_color, _ = VALIDATION_STYLES[status]
+            ax.add_patch(Rectangle((x - 0.5, y - 0.42), 1, 0.84,
+                                   facecolor=color, edgecolor="#657081", linewidth=0.5))
+            ax.text(x, y, "N/A" if status == "NA" else status,
+                    color=text_color, ha="center", va="center", fontsize=9)
+    save_figure(fig, "validation_status", output_dir)
+
+
+def draw_metric(results: list[Result], field: str, ylabel: str,
+                basename: str, output_dir: Path, diverging: bool = False) -> None:
+    data = [diverging_ratio(getattr(row, field)) if diverging else getattr(row, field)
+            for row in results]
+    available = [v for v in data if math.isfinite(v)]
+    upper = max(0, max(available, default=1.0))
+    lower = min(0, min(available, default=0)) if diverging else 0
+    padding = max(upper - lower, 1.0) * 0.3
+    y_max = math.ceil((upper + padding) * 2) / 2
+    y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0
+    y_interval = 1.0 if y_max - y_min > 4.5 else 0.5
+
+    fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
+                             layout="constrained")
+    figure_legend(fig, results, field)
+
+    for ax, arch in zip(axes, ARCHITECTURES):
+        selected = [row for row in results if row.arch == arch]
+        ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
+                      fontsize=11, weight="bold", labelpad=8)
+        ax.set_axisbelow(True)
+        ax.set_ylim(y_min, y_max)
+        ax.set_xlim(-0.5, len(selected) - 0.5)
+        label_groups(ax, selected)
+        ax.tick_params(axis="y", labelleft=True, pad=5, length=0)
+        ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6)
+        ax.yaxis.set_major_locator(MultipleLocator(y_interval))
+        if diverging:
+            ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}×"))
+        ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
+        ax.axhline(0 if diverging else 1, color="#536478", linewidth=1.2,
+                   linestyle="--", zorder=3)
+        ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6)
+
+        for position, row in enumerate(selected):
+            ratio = getattr(row, field)
+            if not math.isfinite(ratio):
+                ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
+                        fontsize=9.5, color="#85909F", fontstyle="italic")
+                continue
+            compiler = advantage_compiler(ratio)
+            statuses = [status for status in (row.raptor_validation, row.pimcomp_validation) if status != "PASS"]
+            status = "FAIL" if "FAIL" in statuses else next(iter(statuses), "PASS")
+            pattern = STAGE_PATTERNS[row.pipeline]
+            measurement = diverging_ratio(ratio) if diverging else ratio
+            ax.bar(position, measurement, width=0.72,
+                    color=COMPILER_COLORS.get(compiler, "#657081"),
+                    hatch=pattern,
+                    edgecolor=TEXT_COLOR if pattern else "none",
+                    linewidth=0.7 if pattern else 0.0, zorder=2)
+            direction = -1 if measurement < 0 else 1
+            factor = max(ratio, 1 / ratio) if diverging else ratio
+            value_label(ax, position, measurement + direction * (y_max - y_min) * 0.012,
+                        f"{factor:.2f}×", status, direction, fontsize=9.1)
+
+    fit_value_labels(fig)
+    save_figure(fig, basename, output_dir)
+
+
+def draw_absolute(results: list[Result], field: str, ylabel: str,
+                  basename: str, output_dir: Path, scale: float = 1.0,
+                  raptor_only: bool = False) -> None:
+    columns = absolute_columns(results, raptor_only)
+    available = [getattr(row, f"{compiler}_{field}") * scale
+                 for row, compiler in columns
+                 if math.isfinite(getattr(row, f"{compiler}_{field}"))]
+    y_max = max(available, default=1.0) * 1.35
+    fig, axes = plt.subplots(1, 2, sharey=True, figsize=(14, 5.2),
+                             layout="constrained")
+    figure_legend(fig, results, field, absolute=True, raptor_only=raptor_only)
+
+    for ax, arch in zip(axes, ARCHITECTURES):
+        selected = [(row, compiler) for row, compiler in columns if row.arch == arch]
+        positions = []
+        for index, (row, _) in enumerate(selected):
+            if index == 0:
+                positions.append(0.0)
+                continue
+            previous = selected[index - 1][0]
+            gap = 0.8 if row.model != previous.model else 0.4 if row.mode != previous.mode else 0.0
+            positions.append(positions[-1] + 1.0 + gap)
+        ax.set_xlabel("ARCHITECTURE A" if arch == "arch-a" else "ARCHITECTURE B",
+                      fontsize=11, weight="bold", labelpad=8)
+        ax.set_axisbelow(True)
+        ax.set_ylim(0, y_max)
+        if positions:
+            ax.set_xlim(positions[0] - 0.9, positions[-1] + 0.9)
+        else:
+            ax.set_xlim(-0.9, 0.9)
+        label_groups(ax, [row for row, _ in selected], positions)
+        if {row.mode for row, _ in selected} == {"latency", "throughput"}:
+            centers, labels = [], []
+            for (_, mode), group in groupby(enumerate(selected), key=lambda item: (item[1][0].model, item[1][0].mode)):
+                group_positions = [positions[position] for position, _ in group]
+                centers.append((group_positions[0] + group_positions[-1]) / 2)
+                labels.append(mode.capitalize())
+            ax.set_xticks(centers, labels, minor=True)
+            ax.tick_params(axis="x", which="minor", length=0, pad=4, labelsize=8)
+        ax.set_ylabel(ylabel, fontsize=10.5)
+        if field == "throughput":
+            ax.ticklabel_format(axis="y", style="plain", useOffset=False)
+        ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
+        ax.tick_params(axis="y", length=0, pad=6)
+        ax.tick_params(axis="x", which="major", length=0,
+                       pad=22 if {row.mode for row, _ in selected} == {"latency", "throughput"} else 6)
+
+        for x, (row, compiler) in zip(positions, selected):
+            measurement = getattr(row, f"{compiler}_{field}") * scale
+            if not math.isfinite(measurement):
+                ax.text(x, y_max * 0.02, "N/A", rotation=90, ha="center",
+                        va="bottom", fontsize=8, color="#85909F")
+                continue
+            status = getattr(row, f"{compiler}_validation")
+            pattern = STAGE_PATTERNS[row.pipeline] if compiler == "raptor" else ""
+            ax.bar(x, measurement, width=1.0,
+                   color=COMPILER_COLORS[compiler],
+                   hatch=pattern,
+                   edgecolor=TEXT_COLOR if pattern else "none",
+                   linewidth=0.7 if pattern else 0.0, zorder=2)
+            label = (f"{measurement:,.2f}".rstrip("0").rstrip(".")
+                     if field == "throughput" else f"{measurement:.3g}")
+            value_label(ax, x, measurement + y_max * 0.012, label, status)
+
+    fit_value_labels(fig)
+    save_figure(fig, basename, output_dir)
+
+
+def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
+    rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1]
+    stages = sorted({row.pipeline for row in rows})
+    panels = [(model, color) for model, color in zip(MODELS, COLORS)
+              if any(row.model == model for row in rows)]
+    lookup = {(row.arch, row.model, row.pipeline): row for row in rows}
+    rates = {key: quotient(1000, row.raptor_latency) if row.mode == "latency" else row.raptor_throughput
+             for key, row in lookup.items()}
+    fig, axes = plt.subplots(len(ARCHITECTURES), len(panels), sharex=True, sharey="col",
+                             figsize=(3 * len(panels), 7.2), squeeze=False, layout="constrained")
+    for column, (model, color) in enumerate(panels):
+        for index, arch in enumerate(ARCHITECTURES):
+            ax = axes[index, column]
+            keys = [(arch, model, stage) for stage in stages]
+            values = [rates.get(key, math.nan) for key in keys]
+            valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys]
+            ax.plot(stages, values, color=color, linewidth=1.8)
+            for stage, value, passed in zip(stages, values, valid):
+                if math.isfinite(value):
+                    marker = ("s" if stage == 1 else "o") if passed else "x"
+                    ax.plot(stage, value, color=color, marker=marker,
+                            markersize=6, linestyle="none")
+            ax.set_xticks(stages)
+            ax.set_xlabel(model.upper(), fontsize=10.5, weight="bold", labelpad=8)
+            ax.grid(axis="y", color="#E6EAF0", linewidth=0.8)
+            ax.tick_params(axis="both", length=0, pad=6, labelbottom=True, labelleft=True)
+            ax.ticklabel_format(axis="y", style="plain", useOffset=False)
+            if column == 0:
+                ax.set_ylabel(f"{arch.replace('arch-', 'ARCHITECTURE ').upper()}\n"
+                              "Throughput (inferences/s)", fontsize=10.5)
+    for ax in axes.flat:
+        ax.set_ylim(bottom=0)
+    handles = [Line2D([], [], color="#657081", marker=marker, linestyle="none", label=label)
+               for marker, label in (("s", "Derived as 1000ms / latency_ms"),
+                                     ("o", "Throughput measurement"))]
+    if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()):
+        handles.append(Line2D([], [], color="#424B58", marker="x", linestyle="none",
+                              label="Raptor validation not PASS"))
+    fig.legend(handles=handles, loc="outside upper center", ncol=len(handles), frameon=False, fontsize=9.5)
+    fig.supxlabel("Raptor pipeline stages", fontsize=11)
+    save_figure(fig, "raptor_pipeline_throughput", output_dir)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    script_dir = Path(__file__).resolve().parent
+    parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv")
+    parser.add_argument("--output-dir", type=Path, default=script_dir / "plots")
+    args = parser.parse_args()
+    configure_style()
+    results = load_results(args.input)
+    valid = sum(row.comparison_passed for row in results)
+    print(f"Results: {len(results)} configurations, {valid} valid comparisons, "
+          f"{len(results)-valid} comparisons without two PASS statuses")
+    draw_validation(results, args.output_dir)
+    draw_pipeline_effect(results, args.output_dir)
+    yolo_only_models = {
+        model for model in MODELS if model.startswith("yolo")
+        and not any(math.isfinite(getattr(row, f"pimcomp_{field}"))
+                    for row in results if row.model == model
+                    for field in ("latency", "power", "energy"))
+    }
+    comparisons = [row for row in results if row.model not in yolo_only_models]
+    yolo_only = [row for row in results if row.model in yolo_only_models]
+    for metric, name, label, mode in [
+        ("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None),
+        ("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"),
+        ("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"),
+        ("power", "power", "Average power advantage (x)", None),
+        ("energy", "energy", "Energy per inference advantage (x)", None),
+    ]:
+        selected = [row for row in comparisons if mode is None or row.mode == mode]
+        draw_metric(selected, metric, label,
+                    f"{name}_comparison", args.output_dir)
+        draw_metric(selected, metric, label,
+                    f"{name}_diverging", args.output_dir, diverging=True)
+    for metric, name, label, scale, mode in [
+        ("latency", "longest_stage_latency", "Longest stage latency (ms)", 1.0, None),
+        ("latency", "latency", "Single-inference latency (ms)", 1.0, "latency"),
+        ("power", "power", "Average power (mW)", 1.0, None),
+        ("energy", "energy", "Energy per inference (mJ)", 1e-9, None),
+        ("throughput", "throughput", "Throughput (inferences/s)", 1.0, "throughput"),
+    ]:
+        for suffix, group, raptor_only in (
+            ("absolute", comparisons, False), ("yolo", yolo_only, True)
+        ):
+            selected = [row for row in group if mode is None or row.mode == mode]
+            if not selected:
+                continue
+            draw_absolute(selected, metric, label,
+                          f"{name}_{suffix}", args.output_dir, scale,
+                          raptor_only=raptor_only)
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,41 @@
+arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
+arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26
+arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,423.80,482.60,431626597.00,541691691.00,raptor,1.14
+arch-a,vgg8,throughput,4,batch,PASS,PASS,0.86,1.04,1160.00,965.00,432.99,482.60,416224904.20,541691691.00,raptor,1.20
+arch-a,vgg8,throughput,8,batch,PASS,PASS,1.63,1.04,615.00,965.00,308.12,482.60,616026711.90,541691691.00,pimcomp,1.57
+arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394498463.12,13868866808.12,raptor,4.15
+arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
+arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.62,13.46,51.00,74.30,376.10,481.60,8024732847.00,6563451004.00,pimcomp,1.46
+arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.92,13.46,50.20,74.30,368.24,481.60,9023016592.00,6563451004.00,pimcomp,1.48
+arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
+arch-a,resnet34,throughput,2,batch,PASS,FAIL,22.26,23.58,44.90,42.40,473.84,515.68,16417303180.00,12283627410.00,raptor,1.06
+arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.37,23.58,36.50,42.40,415.90,515.68,17019112320.00,12283627410.00,pimcomp,1.16
+arch-a,resnet34,throughput,8,batch,PASS,FAIL,41.04,23.58,24.40,42.40,346.15,515.68,17351854700.00,12283627410.00,pimcomp,1.74
+arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.88,231.19,5513234559.92,14547744283.24,raptor,6.44
+arch-a,googlenet,throughput,2,batch,PASS,FAIL,10.16,18.26,98.50,54.80,544.21,435.11,6029999329.00,8164787803.00,raptor,1.80
+arch-a,googlenet,throughput,4,batch,PASS,FAIL,11.96,18.26,83.60,54.80,473.16,435.11,6741923526.00,8164787803.00,raptor,1.53
+arch-a,googlenet,throughput,8,batch,PASS,FAIL,20.66,18.26,48.40,54.80,348.76,435.11,8421537146.00,8164787803.00,pimcomp,1.13
+arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
+arch-a,yolo11n,throughput,2,batch,PASS,FAIL,120.93,NA,8.27,NA,313.20,NA,41830757130.00,NA,NA,NA
+arch-a,yolo11n,throughput,4,batch,PASS,FAIL,92.06,NA,10.90,NA,351.03,NA,39451980450.00,NA,NA,NA
+arch-a,yolo11n,throughput,8,batch,PASS,FAIL,104.07,NA,9.61,NA,330.76,NA,44002712680.00,NA,NA,NA
+arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
+arch-b,vgg8,throughput,2,batch,PASS,PASS,1.18,1.07,846.00,932.00,327.05,409.47,429183159.10,439288835.80,pimcomp,1.10
+arch-b,vgg8,throughput,4,batch,PASS,PASS,0.92,1.07,1090.00,932.00,383.82,409.47,464794094.50,439288835.80,raptor,1.17
+arch-b,vgg8,throughput,8,batch,PASS,PASS,1.62,1.07,616.00,932.00,276.49,409.47,524048148.70,439288835.80,pimcomp,1.51
+arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
+arch-b,resnet18,throughput,2,batch,PASS,FAIL,13.08,13.43,76.40,74.40,440.79,484.80,6399942695.00,6569655551.00,raptor,1.03
+arch-b,resnet18,throughput,4,batch,PASS,FAIL,20.68,13.43,48.40,74.40,327.57,484.80,7777606323.00,6569655551.00,pimcomp,1.54
+arch-b,resnet18,throughput,8,batch,PASS,FAIL,21.17,13.43,47.20,74.40,317.64,484.80,8897039159.00,6569655551.00,pimcomp,1.58
+arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
+arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
+arch-b,resnet34,throughput,4,batch,PASS,FAIL,39.61,22.85,25.20,43.80,315.01,493.33,14629161660.00,11392215660.00,pimcomp,1.74
+arch-b,resnet34,throughput,8,batch,PASS,FAIL,49.34,22.85,20.30,43.80,285.76,493.33,16591020470.00,11392215660.00,pimcomp,2.16
+arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422799605.92,9182062336.24,raptor,3.38
+arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.83,21.39,102.00,46.80,520.79,417.87,5669213312.00,9014726090.00,raptor,2.18
+arch-b,googlenet,throughput,4,batch,PASS,FAIL,14.47,21.39,69.10,46.80,401.01,417.87,6835460477.00,9014726090.00,raptor,1.48
+arch-b,googlenet,throughput,8,batch,PASS,FAIL,37.28,21.39,26.80,46.80,245.38,417.87,11453848400.00,9014726090.00,pimcomp,1.75
+arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
+arch-b,yolo11n,throughput,2,batch,PASS,FAIL,118.15,NA,8.46,NA,287.59,NA,37062981240.00,NA,NA,NA
+arch-b,yolo11n,throughput,4,batch,PASS,FAIL,94.18,NA,10.60,NA,316.45,NA,37424491250.00,NA,NA,NA
+arch-b,yolo11n,throughput,8,batch,PASS,FAIL,142.86,NA,7.00,NA,290.68,NA,41525990280.00,NA,NA,NA
@@ -4,6 +4,7 @@ from __future__ import annotations
 import argparse
 import csv
 import json
+import math
 import os
 import shlex
 import shutil
@@ -43,6 +44,7 @@ COMPARISONS = (
    ("throughput", 8, "batch"),
 )
 RESULTS_FILENAME = "results_comparison.csv"
+DEFAULT_RESULTS_PATH = Path(__file__).with_name(RESULTS_FILENAME)


@dataclass(frozen=True)
@@ -128,9 +130,9 @@ def write_results_csv(
    comparisons: tuple[tuple[str, int, str], ...] = COMPARISONS,
    ablation_variant: str | None = None,
 ) -> Path:
-    output = (root or SUITE) / RESULTS_FILENAME
+    output = DEFAULT_RESULTS_PATH if root is None else root / RESULTS_FILENAME
    if ablation_variant is not None:
-        output = output.parent / "ablation" / ablation_variant / RESULTS_FILENAME
+        output = (root or SUITE) / "ablation" / ablation_variant / RESULTS_FILENAME
    fields = (
        "arch",
        "model",
@@ -247,18 +249,26 @@ def write_results_csv(


 def performance_values(performance: dict) -> dict[str, float | None]:
+    fields = {"latency": ("latency_ms", "average_latency_ms"),
+              "throughput": ("throughput",), "power": ("average_power_mw",),
+              "energy": ("average_energy_pj",)}
+    values = dict.fromkeys(fields)
+    if performance.get("error") or performance.get("skipped") or performance.get("unsupported"):
+        return values
    parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
-    return {
-        "latency": (
-            performance.get("latency_ms")
-            or performance.get("average_latency_ms")
-            or parsed.get("latency_ms")
-            or parsed.get("average_latency_ms")
-        ),
-        "throughput": performance.get("throughput") or parsed.get("throughput"),
-        "power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
-        "energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
-    }
+    for metric, names in fields.items():
+        for value in (source.get(name) for source in (performance, parsed) for name in names):
+            try:
+                number = float(value)
+            except (TypeError, ValueError):
+                continue
+            if math.isfinite(number) and number > 0:
+                values[metric] = number
+                break
+    if performance.get("output_count", parsed.get("output_count")) == 0:
+        for metric in ("latency", "throughput", "energy"):
+            values[metric] = None
+    return values


 def comparison_passed(report: dict) -> bool:
@@ -283,7 +293,9 @@ def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:


 def format_value(value: float | None) -> str:
-    return "NA" if value is None else f"{value:.2f}"
+    if value is None or not math.isfinite(value) or value <= 0:
+        return "NA"
+    return f"{value:.6g}"


 def print_stage(title: str, color: str) -> None:
@@ -352,15 +364,12 @@ def comparison_command(
    raptor_extra_args: list[str] | tuple[str, ...] = (),
    raptor_only: bool = False,
    reuse_raptor_report: Path | None = None,
-    reuse_pimcomp_dir: Path | None = None,
    reuse_pimcomp_report: Path | None = None,
 ) -> list[str]:
    time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
    reuse_args = []
    if reuse_raptor_report is not None:
        reuse_args.extend(["--reuse-raptor-report", str(reuse_raptor_report)])
-    if reuse_pimcomp_dir is not None:
-        reuse_args.extend(["--reuse-pimcomp-dir", str(reuse_pimcomp_dir)])
    if reuse_pimcomp_report is not None:
        reuse_args.extend(["--reuse-pimcomp-report", str(reuse_pimcomp_report)])
    return [
@@ -430,6 +439,12 @@ def comparison_command_for(
    reuse_shared_pimcomp: bool,
 ) -> list[str]:
    report = spec.output_dir / "pimcomp/comparison_report.json"
+    reuse_pimcomp_report = None
+    if not args.raptor_only:
+        if args.only == "raptor":
+            reuse_pimcomp_report = report
+        elif reuse_shared_pimcomp:
+            reuse_pimcomp_report = spec.shared_pimcomp_dir.parent / "comparison_report.json"
    return comparison_command(
        spec.model,
        spec.functional_model,
@@ -450,20 +465,21 @@ def comparison_command_for(
            if args.only == "pimcomp"
            else None
        ),
-        reuse_pimcomp_dir=(
-            spec.shared_pimcomp_dir
-            if reuse_shared_pimcomp and args.only != "raptor" and not args.raptor_only
-            else None
-        ),
-        reuse_pimcomp_report=report if args.only == "raptor" and not args.raptor_only else None,
+        reuse_pimcomp_report=reuse_pimcomp_report,
    )


-def pimcomp_artifact_ready(output_dir: Path) -> bool:
-    return all(
-        (output_dir / name).is_file()
-        for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
-    ) and (output_dir.parent / "comparison_report.json").is_file()
+def pimcomp_report_stamp(output_dir: Path) -> tuple[int, int, int] | None:
+    try:
+        stat = (output_dir.parent / "comparison_report.json").stat()
+    except FileNotFoundError:
+        return None
+    return stat.st_ino, stat.st_mtime_ns, stat.st_ctime_ns
+
+
+def pimcomp_report_ready(output_dir: Path, previous: tuple[int, int, int] | None) -> bool:
+    stamp = pimcomp_report_stamp(output_dir)
+    return stamp is not None and stamp != previous


 def run_comparison_jobs(
@@ -533,7 +549,7 @@ def main() -> int:
    parser.add_argument(
        "--out-dir",
        type=Path,
-        help="Suite root; generated artifacts go below each model's artifacts/ directory (default: validation/networks/pimcomp_models).",
+        help="Custom experiment root for artifacts and aggregate CSV. Default: model-suite artifacts and results_comparison.csv beside this script.",
    )
    parser.add_argument(
        "--common-dir",
@@ -687,11 +703,6 @@ def main() -> int:
                    required = comparison_dir / "pimcomp/comparison_report.json"
                    if not required.exists():
                        missing_reuse.append(str(required))
-                    elif args.only == "raptor":
-                        if json.loads(required.read_text(encoding="utf-8")).get(
-                            "pimcomp_model_source"
-                        ) != "original_onnx":
-                            missing_reuse.append(f"{required} (not generated from the original ONNX model)")
        if missing_reuse:
            parser.error(
                f"--only {args.only} needs existing comparison artifacts: "
@@ -713,7 +724,7 @@ def main() -> int:
        f"Comparison jobs: "
        f"{sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())}"
    )
-    print(f"Results root: {out_dir or SUITE}")
+    print(f"Artifacts root: {out_dir or SUITE}")
    print("=" * 72)

    print_stage("Prepare shared artifacts", STAGE_COLORS["Build runner"])
@@ -799,6 +810,10 @@ def main() -> int:
        print_directly = min(args.jobs, len(comparison_specs)) == 1
        try:
            with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-pimcomp-")) as log_dir:
+                previous_reports = {
+                    spec.shared_pimcomp_dir: pimcomp_report_stamp(spec.shared_pimcomp_dir)
+                    for spec in anchor_specs
+                }
                anchor_failed, log_offset = run_comparison_jobs(
                    anchor_jobs,
                    args.jobs,
@@ -812,7 +827,9 @@ def main() -> int:
                        comparison_command_for(
                            spec,
                            args,
-                            reuse_shared_pimcomp=pimcomp_artifact_ready(spec.shared_pimcomp_dir),
+                            reuse_shared_pimcomp=pimcomp_report_ready(
+                                spec.shared_pimcomp_dir, previous_reports[spec.shared_pimcomp_dir]
+                            ),
                        ),
                        None,
                    )
@@ -1,24 +1,18 @@
 # Pimcomp correctness experiments

+Run from the repository root using the repository virtual environment, built
+release Raptor, Pimcomp frontend/backend, and `pimsim-nn`.
+
 ## Synchronization ordering

-The synchronization experiment checks whether repeated PIMCOMP and Raptor
-programs preserve cross-core global-memory generations under legal execution
-schedules. It builds a small two-convolution model for Arch-A, Arch-B, and
-Arch-C, finds cross-core `ST`/`LD` dependencies, and compares greedy,
-bounded-stall, randomized, and adversarial runs using the Rust simulator's
-provenance trace. Byte-identical batch inputs isolate intermediate-memory
-ordering from the separate host-input lifetime issue.
+Checks whether repeated PIMCOMP and Raptor programs preserve cross-core
+global-memory generations under legal execution schedules. A two-convolution
+model runs on Arch-A/B/C with greedy, bounded-stall, randomized, and adversarial
+schedules; Rust provenance traces identify cross-core `ST`/`LD` dependencies.
+Byte-identical batch inputs isolate intermediate-memory ordering from host-input
+lifetime. Diagnostic scheduling does not change the performance oracle.

-The experiment records static dependency evidence, dynamic provenance,
-functional output comparisons, architecture contract classifications, and
-per-artifact PIMCOMP/Raptor conclusions. Diagnostic scheduling changes only
-the functional simulator's execution order; it does not modify PIMCOMP or the
-`pimsim-nn` performance oracle.
-
-Prerequisites are the repository virtual environment, the release Raptor
-compiler, built PIMCOMP frontend/backend, and the existing `pimsim-nn` build.
-Run from the repository root with a new or empty output directory:
+Use a new or empty output directory:

 ```bash
 .venv/bin/python \
@@ -27,102 +21,50 @@ Run from the repository root with a new or empty output directory:
  --batch-size 4 --seed 0 --self-check
 ```

-The directory contains:
+Outputs: `adversarial_memory_sync_report.{json,md}` and per-architecture
+evidence. Contract classifications use
+[`architecture_evidence.json`](synchronization/architecture_evidence.json).

- [`run_experiment.py`](synchronization/run_experiment.py): orchestration,
-  classification, reporting, and self-checks.
- [`global_memory.py`](synchronization/global_memory.py): artifact compilation,
-  dependency analysis, simulator execution, and provenance helpers.
- [`architecture_contract.py`](synchronization/architecture_contract.py) and
-  [`architecture_evidence.json`](synchronization/architecture_evidence.json):
-  conservative architecture-contract evidence and labels.
+## Batch prefill reproduction

-The main outputs are `adversarial_memory_sync_report.json` and
-`adversarial_memory_sync_report.md`, with per-architecture evidence below the
-same output root.
-
-## Pimcomp batch prefill reproduction
-
-Pimcomp's batch scheduler currently emits an incomplete standalone program for
-models containing post operations. The generated `VerificationInfo.json` uses a
-negative `source_address` to identify the preceding node, but the batch
-verifier resolves that address by copying the provider tensor directly from
-ONNX Runtime's `intermediate_result` table. It does not read the simulated
-global-memory buffer.
-
-The corresponding post-operation scheduling call is commented out in
+Pimcomp's batch scheduler omits post-operation producers for some Pool, ReLU,
+Concat, and Reshape tensors. Its verifier resolves negative `source_address`
+entries using ONNX Runtime intermediates rather than simulated global memory.
+The corresponding scheduling call is disabled in
 [`BatchPipelineSchedule.cpp`](../../../../../third_party/PIMCOMP-NN/backend/BatchPipelineSchedule.cpp#L1535).
-Consequently, Pool, ReLU, Concat, and Reshape provider tensors can be consumed
-by later `LD` instructions without any generated `ST` producer. The comparison
-exporter initializes those host-side intermediate buffers to zero, which makes
-the missing computation visible in Rust functional validation.
-
-For the checked-in GoogLeNet throughput/pipeline2 artifact, 39 provider tensors
-are referenced by batch loads. Nineteen have generated stores; twenty are
-never written. Preloading the provider tensors with the same ONNX Runtime
-intermediates used by Pimcomp's verifier makes the exported program pass. This
-reproduces the verifier's input contract; it does not repair Pimcomp's batch
-schedule.
-
-Run the default reproduction from the repository root:
+Rust validation starts those buffers at zero, exposing the missing computation.

 ```bash
 .venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py
 ```

-The launcher accepts an alternate comparison directory, model, work directory,
-and shared reference-artifact directory:
+The default uses GoogLeNet Arch-A throughput/pipeline2. Optional positional
+arguments are comparison directory, model, work directory, and shared reference
+directory; the default work directory is `correctness/prefill/` below the
+comparison artifacts.

-```bash
-.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py \
-  validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2 \
-  validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx \
-  validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2/correctness/prefill \
-  validation/networks/pimcomp_models/googlenet/artifacts/common
-```
+The experiment compares the original memory image with one prefilled using
+[`batch_memory.py`](prefill/batch_memory.py). In the recorded case, 20 of 39
+loaded provider tensors have no store producer; prefill reduces maximum
+reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's
+verifier contract; it does not repair the standalone batch program.

-Without the optional work-directory argument, the experiment uses the same
-`correctness/prefill/` directory below the comparison artifacts.
+## ResNet BatchNorm gap

-It runs the exported artifact once with its original memory image and once
-with [`batch_memory.py`](prefill/batch_memory.py), then compares both
-outputs with the recorded native reference. The expected GoogLeNet result is a
-baseline maximum difference near `6.70705` and a prefilled maximum difference
-near `4.05e-6`.
+Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
+removes BatchNormalization without folding its affine transform into weights
+and bias. Its [verifier](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
+substitutes identity BatchNorm parameters, so agreement with that verifier does
+not establish equivalence to the original ONNX model.

-The issue is in Pimcomp batch scheduling/validation semantics, not in the Rust
-simulator's vector-length interpretation. Vector lengths remain element counts
-as specified by the reference ISA.
-
-## ResNet BatchNorm correctness gap
-
-Pimcomp has a separate correctness limitation in its ResNet element pipeline.
-`BatchNormalization` is an ONNX operation. The frontend's
-[`fuse_operators()` pass](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
-marks it as fused and removes the node from the
-scheduled graph, but the released path does not fold the BatchNorm affine
-transform into the convolution weights and bias. The corresponding verifier
-workaround in [`verification.py`](../../../../../third_party/PIMCOMP-NN/verification/verification.py#L99)
-replaces BatchNorm parameters with identity values (scale and variance equal
-to one, bias and mean equal to zero) before running ONNX Runtime.
-
-Therefore Pimcomp's native verifier and exported element program agree with
-each other, but they do not implement the original ResNet ONNX model. On the
-current ResNet-18 Arch-A latency artifact, using the same input:
+Recorded ResNet-18 Arch-A latency errors on the same input:

 | Comparison | Maximum absolute difference |
 |---|---:|
 | Pimcomp native verifier vs Rust export | `6.7e-8` |
 | Rust export vs original ONNX reference | `4.8294563` |

-The Pimcomp output ranges from approximately `-0.187` to `0.220`, while the
-original ONNX output ranges from `-3.572` to `4.834`; 462 of 1000 final
-elements differ by more than one. This is not a Rust simulator or Python
-exporter regression. It is a Pimcomp model-semantics mismatch caused by
-dropping BatchNorm numerics. The same issue affects the ResNet-34 latency
-artifact. VGG and GoogLeNet do not show this particular mismatch because they
-do not contain the same ResNet BatchNorm path.
-
-The latency comparison intentionally uses the original ONNX reference, so
-these Pimcomp rows must remain `FAIL` until Pimcomp folds BatchNorm correctly
-or the comparison explicitly uses a BatchNorm-neutralized reference.
+The same gap affects ResNet-34 latency. Comparison rows use the original ONNX
+reference and must remain `FAIL` until BatchNorm numerics are implemented
+correctly. This is independent of simulator vector lengths, which remain
+element counts.
@@ -0,0 +1,137 @@
+"""Shared appearance and PNG/SVG export for PIM result plots."""
+
+from pathlib import Path
+from math import ceil
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+from matplotlib.patches import Patch
+from matplotlib.path import Path as PlotPath
+from matplotlib.patheffects import AbstractPathEffect
+from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath
+
+
+TEXT_COLOR = "#243449"
+HATCH_COLOR = "#C62828"
+COLORS = ("#2B65A1", "#168C8B", "#D18D32", "#8062A5", "#B65D60")
+
+
+class CenteredHatch(AbstractPathEffect):
+    """Anchor dot and diagonal hatch patterns to each rectangle's center."""
+
+    def draw_path(self, renderer, gc, tpath, affine, rgbFace=None):
+        hatch = gc.get_hatch() or ""
+        base_gc = renderer.new_gc()
+        base_gc.copy_properties(gc)
+        base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\"))
+        renderer.draw_path(base_gc, tpath, affine, rgbFace)
+        base_gc.restore()
+
+        bounds = affine.transform_path(tpath).get_extents()
+        center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2
+        vertices = []
+        diagonal = 2 ** -0.5
+        for symbols, dx, dy in (("/", diagonal, diagonal), ("\\", diagonal, -diagonal)):
+            count = sum(hatch.count(symbol) for symbol in symbols)
+            if count == 0:
+                continue
+            spacing = renderer.points_to_pixels(12 / count)
+            half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2
+            half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2
+            extent = ceil(half_span / spacing)
+            for step in range(-extent, extent + 1):
+                x = center_x - dy * step * spacing
+                y = center_y + dx * step * spacing
+                vertices.extend(((x - dx * half_length, y - dy * half_length),
+                                 (x + dx * half_length, y + dy * half_length)))
+        line_gc = renderer.new_gc()
+        line_gc.copy_properties(gc)
+        line_gc.set_hatch(None)
+        line_gc.set_foreground(gc.get_hatch_color())
+        line_gc.set_linewidth(gc.get_hatch_linewidth())
+        line_gc.set_clip_path(TransformedPath(tpath, affine))
+        if vertices:
+            renderer.draw_path(line_gc, PlotPath(vertices, [PlotPath.MOVETO, PlotPath.LINETO] * (len(vertices) // 2)),
+                               IdentityTransform())
+        if "." in hatch:
+            spacing = renderer.points_to_pixels(12 / hatch.count("."))
+            x_extent = ceil(bounds.width / (2 * spacing))
+            y_extent = ceil(bounds.height / (2 * spacing))
+            centers = [(center_x + x * spacing, center_y + y * spacing)
+                       for x in range(-x_extent, x_extent + 1)
+                       for y in range(-y_extent, y_extent + 1)]
+            line_gc.set_linewidth(0)
+            renderer.draw_markers(line_gc, PlotPath.unit_circle(),
+                                  Affine2D().scale(renderer.points_to_pixels(0.9)),
+                                  PlotPath(centers), IdentityTransform(), gc.get_hatch_color())
+        line_gc.restore()
+
+
+def configure_style() -> None:
+    plt.rcParams.update({
+        "font.family": "DejaVu Sans",
+        "axes.spines.top": False,
+        "axes.spines.right": False,
+        "axes.labelcolor": TEXT_COLOR,
+        "text.color": TEXT_COLOR,
+        "xtick.color": TEXT_COLOR,
+        "ytick.color": TEXT_COLOR,
+        "axes.edgecolor": "#CFD6DF",
+        "savefig.facecolor": "white",
+        "figure.facecolor": "white",
+    })
+
+
+def value_label(ax, x, y, label: str, status: str = "PASS", direction: int = 1,
+                fontsize: float = 8) -> None:
+    alignment = "bottom" if direction > 0 else "top"
+    rotation = 90 if direction > 0 else -90
+    text = ax.text(x, y, label, rotation=rotation, ha="center", va=alignment,
+                   fontsize=fontsize, color=TEXT_COLOR)
+    if status != "PASS":
+        height = text.get_window_extent(ax.figure.canvas.get_renderer()).height
+        offset = direction * (height * 72 / ax.figure.dpi + 3)
+        ax.annotate("INVALID" if status == "FAIL" else status, (x, y),
+                    xytext=(0, offset), textcoords="offset points",
+                    rotation=rotation, ha="center", va=alignment, fontsize=fontsize,
+                    color=HATCH_COLOR)
+
+
+def fit_value_labels(fig) -> None:
+    """Reserve vertical space for rotated labels using their rendered size."""
+    for ax in fig.axes:
+        for text in ax.texts:
+            text.set_in_layout(False)
+    fig.canvas.draw()
+    renderer = fig.canvas.get_renderer()
+    limits = []
+    for ax in fig.axes:
+        bounds = ax.get_window_extent(renderer)
+        labels = []
+        for text in ax.texts:
+            x, y = text.xy if hasattr(text, "xy") else text.get_position()
+            anchor = ax.transData.transform((x, y))[1]
+            extent = text.get_window_extent(renderer)
+            labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height,
+                           max(0, extent.y1 - anchor + 3) / bounds.height))
+        lower, upper = ax.get_ylim()
+        for _ in range(12):
+            lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
+            upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)])
+        limits.append((lower, upper))
+    for ax in fig.axes:
+        ax.set_ylim(min(lower for lower, _ in limits), max(upper for _, upper in limits))
+
+
+def save_figure(fig, basename: str, output_dir: Path) -> None:
+    for patch in fig.findobj(Patch):
+        hatch = patch.get_hatch() or ""
+        if any(symbol in hatch for symbol in "./\\"):
+            patch.set_path_effects([CenteredHatch()])
+    for extension in ("png", "svg"):
+        target = output_dir / extension / f"{basename}.{extension}"
+        target.parent.mkdir(parents=True, exist_ok=True)
+        fig.savefig(target, dpi=220, bbox_inches="tight", pad_inches=0.04)
+        print(target)
+    plt.close(fig)