even better more structured throughput scheduling

plot adjustments
validation adjustments

validation/tools/pim/ablation/README.md

@@ -12,10 +12,14 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
 
 | Variant | Raptor options |
 |---|---|
-| `baseline` | None |
-| `no-sync` | `--pim-disable-synchronization` |
-| `no-plan` | `--pim-disable-spatial-planning` |
-| `no-sync-no-plan` | Both options |
+| `baseline` | None; planning and synchronization enabled |
+| `no-sync` | `--pim-disable-synchronization`; throughput only |
+| `no-plan` | `--pim-disable-spatial-planning`; synchronization remains enabled |
+
+Latency runs compare `baseline` with `no-plan`. Throughput runs compare
+`baseline` with each disabled feature separately. No run disables both features.
+The `baseline` artifact name identifies the fully enabled program, not the
+denominator used for percentage gains.
 
 ## Options and outputs
 
@@ -25,7 +29,9 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
 
 Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
 `--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
-replaces the default latency/pipeline-4 case set. `--only` is rejected because
+replaces the default latency/pipeline-4 case set. Mode-only selection runs all
+suite pipelines for that mode. Pipeline-only selection uses its matching mode.
+`--only` is rejected because
 the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
 
 Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
@@ -37,19 +43,44 @@ By default, the combined [results_ablation.csv](results_ablation.csv) is written
 beside this script and printed after the run. An explicit `--out-dir PATH`
 writes `PATH/results_ablation.csv`, keeping independent experiments separate.
 Default model artifacts and per-variant summaries remain in the locations above.
-The aggregate records case identifiers,
-`functional_validation`, and signed percentage changes in latency, throughput,
-power, and energy against the matching baseline case. Positive means higher;
-baseline values are zero.
+The aggregate contains one row per comparison:
+
+| Mode | `comparison` | `baseline_variant` | Main gain |
+|---|---|---|---|
+| Latency | `plan` | `no-plan` | `latency_gain_percent` |
+| Throughput | `plan` | `no-plan` | `throughput_gain_percent` |
+| Throughput | `sync` | `no-sync` | `throughput_gain_percent` |
+
+Throughput gain is `100 × (enabled / disabled − 1)`.
+Latency, power, and energy gains are reductions: `100 × (1 − enabled / disabled)`.
+The corresponding columns use the suffix `_gain_percent`. Positive means improvement;
+negative means a cost. Synchronization gain can be negative. No sign is forced.
+Both programs' functional statuses are recorded as `functional_validation`
+and `baseline_functional_validation`. The latter belongs to the disabled feature.
 
 Disabling synchronization removes host-buffer lifetime and pipeline restart
 guarantees, so those variants may fail numerical validation and exit nonzero.
-Baseline and `no-plan` must pass. Percentages from failing variants are not
-valid speedups; deadline averages do not prove steady-state convergence.
+Baseline and `no-plan` must pass. Percentages involving failing programs remain
+diagnostic comparisons, not valid speedups.
+
+When fast-mode convergence fails, positive fallback throughput remains available.
+Its output interval uses the full simulation window. The per-case JSON report
+records `throughput_measurement: full_duration_fallback` and
+`fast_throughput_converged: false`. This result does not prove steady-state convergence.
+Zero completed outputs still produce `NA` for per-inference metrics.
+
+The `no-sync` compiler option keeps the normal selected schedule and omits
+SYNC/WAIT instructions during code generation. Selection costs describe the
+`synchronized-reference` program. Each backend attempt also reports a separate
+`synchronization_ablation_cost`, with zero SYNC/WAIT work and no synchronization
+dependencies. This diagnostic estimate is not used for selection. Its complete-state
+recurrence can remain unproved because cores can advance independently.
+Numerical validation still checks the emitted program and can fail.
 
 ## Result data
 
-The checked-in `results_ablation.csv` is an intentional result dataset for tabular
-reporting. Ablation experiments and aggregation remain available through the
+The checked-in `results_ablation.csv` is a historical dataset until the next run
+regenerates it with the comparison schema above. Existing measurements are preserved.
+Ablation experiments and aggregation remain available through the
 runner above. Percentages from failing variants are not valid speedups. See the
 [comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).

validation/tools/pim/ablation/run_ablation.py

@@ -37,10 +37,6 @@ VARIANTS = (
     Variant("baseline", ()),
     Variant("no-sync", ("--pim-disable-synchronization",)),
     Variant("no-plan", ("--pim-disable-spatial-planning",)),
-    Variant(
-        "no-sync-no-plan",
-        ("--pim-disable-synchronization", "--pim-disable-spatial-planning"),
-    ),
 )
 VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS}
 COMPARISON_RESULTS_FILENAME = "results_comparison.csv"
@@ -53,17 +49,19 @@ PERCENTAGE_FIELDS = (
     ("raptor_power_mw", "power"),
     ("raptor_energy_pj", "energy"),
 )
-RESULT_FIELDS = (*CASE_FIELDS, "functional_validation", *(target for _, target in PERCENTAGE_FIELDS))
+RESULT_FIELDS = (
+    "comparison", *CASE_FIELDS, "baseline_variant", "baseline_functional_validation",
+    "functional_validation", *(f"{target}_gain_percent" for _, target in PERCENTAGE_FIELDS),
+)
 DEFAULT_CASE_ARGUMENTS = (
     ("latency", ("--mode", "latency")),
     ("throughput/pipeline4", ("--mode", "throughput", "--pipeline", "4")),
 )
-DEFAULT_CASE_KEYS = frozenset({("latency", "1"), ("throughput", "4")})
 
 
 def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[str]]:
     parser = argparse.ArgumentParser(
-        description="Run the complete compiler ablation matrix on the Pimcomp model suite.",
+        description="Measure planning and synchronization gains on the Pimcomp model suite.",
         allow_abbrev=False,
     )
     parser.add_argument(
@@ -97,12 +95,30 @@ def has_option(arguments: list[str], option: str) -> bool:
 
 
 def runner_argument_sets(forwarded: list[str]) -> tuple[tuple[str, list[str]], ...]:
-    if has_option(forwarded, "--mode") or has_option(forwarded, "--pipeline"):
-        return (("requested", forwarded),)
-    return tuple(
-        (label, [*forwarded, *case_arguments])
-        for label, case_arguments in DEFAULT_CASE_ARGUMENTS
-    )
+    if not has_option(forwarded, "--mode") and not has_option(forwarded, "--pipeline"):
+        return tuple(
+            (label, [*forwarded, *case_arguments])
+            for label, case_arguments in DEFAULT_CASE_ARGUMENTS
+        )
+    parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
+    parser.add_argument("--mode", nargs="+", choices=("latency", "throughput"))
+    parser.add_argument("--pipeline", type=int, choices=(1, 2, 4, 8))
+    args, remaining = parser.parse_known_args(forwarded)
+    cases = [(mode, pipeline) for mode, pipeline in
+             (("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8))
+             if (args.mode is None or mode in args.mode)
+             and (args.pipeline is None or pipeline == args.pipeline)]
+    if not cases:
+        parser.error("--pipeline does not belong to the selected --mode")
+    return tuple((f"{mode}/pipeline{pipeline}",
+                  [*remaining, "--mode", mode, "--pipeline", str(pipeline)])
+                 for mode, pipeline in cases)
+
+
+def case_key(arguments: list[str]) -> tuple[str, str]:
+    mode = arguments[arguments.index("--mode") + 1]
+    pipeline = arguments[arguments.index("--pipeline") + 1] if "--pipeline" in arguments else "1"
+    return mode, pipeline
 
 
 def variant_output_dir(out_dir: Path, variant: Variant) -> Path:
@@ -154,7 +170,9 @@ def terminate_process_group(process: subprocess.Popen[bytes]) -> None:
         process.wait()
 
 
-def percentage_difference(value: str | None, reference: str | None) -> str:
+def percentage_difference(
+    value: str | None, reference: str | None, *, lower_is_better: bool = False
+) -> str:
     try:
         current = float(value) if value is not None else math.nan
         baseline = float(reference) if reference is not None else math.nan
@@ -162,7 +180,8 @@ def percentage_difference(value: str | None, reference: str | None) -> str:
         return "NA"
     if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0:
         return "NA"
-    change = (current / baseline - 1) * 100
+    ratio = current / baseline
+    change = (1 - ratio if lower_is_better else ratio - 1) * 100
     return f"{change:+.2f}%" if math.isfinite(change) else "NA"
 
 
@@ -202,25 +221,43 @@ def aggregate_results(
                 selected_rows.append(row)
     if fields is None:
         return None, failures
-    baseline_rows = {
+    enabled_rows = {
         tuple(row.get(field, "") for field in CASE_FIELDS): row
         for row in rows_by_variant.get("baseline", [])
     }
     if "baseline" not in rows_by_variant:
-        failures.append("baseline: missing percentage reference results")
+        failures.append("baseline: missing fully enabled results")
     result_rows = []
-    variant_order = {variant.name: index for index, variant in enumerate(VARIANTS)}
     for variant in variants:
+        if variant.name == "baseline":
+            continue
+        disabled_keys = {
+            tuple(row.get(field, "") for field in CASE_FIELDS)
+            for row in rows_by_variant.get(variant.name, [])
+        }
+        for key, row in enabled_rows.items():
+            if variant.name == "no-sync" and row.get("mode") != "throughput":
+                continue
+            if key not in disabled_keys:
+                failures.append(f"{variant.name}: missing baseline case " + "/".join(key))
         for row in rows_by_variant.get(variant.name, []):
-            reference = baseline_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS))
+            if variant.name == "no-sync" and row.get("mode") != "throughput":
+                continue
+            enabled = enabled_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS))
+            if enabled is None:
+                failures.append(f"{variant.name}: missing fully enabled case "
+                                + "/".join(row.get(field, "") for field in CASE_FIELDS))
             result_rows.append(
                 {
-                    "variant": variant.name,
+                    "comparison": "plan" if variant.name == "no-plan" else "sync",
                     **{field: row.get(field, "") for field in CASE_FIELDS},
-                    "functional_validation": row.get("raptor_functional_validation", "NA"),
+                    "baseline_variant": variant.name,
+                    "baseline_functional_validation": row.get("raptor_functional_validation", "NA"),
+                    "functional_validation": enabled.get("raptor_functional_validation", "NA") if enabled else "NA",
                     **{
-                        target: percentage_difference(
-                            row.get(source), reference.get(source) if reference else None
+                        f"{target}_gain_percent": percentage_difference(
+                            enabled.get(source) if enabled else None, row.get(source),
+                            lower_is_better=target != "throughput",
                         )
                         for source, target in PERCENTAGE_FIELDS
                     },
@@ -231,14 +268,14 @@ def aggregate_results(
             row["arch"],
             row["model"],
             row["mode"],
-            variant_order[row["variant"]],
             int(row["raptor_pipeline"]),
+            row["comparison"],
         )
     )
     output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME
     output.parent.mkdir(parents=True, exist_ok=True)
     with output.open("w", newline="", encoding="utf-8") as stream:
-        writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n")
+        writer = csv.DictWriter(stream, fieldnames=RESULT_FIELDS, lineterminator="\n")
         writer.writeheader()
         writer.writerows(result_rows)
     return output, failures
@@ -252,9 +289,13 @@ def main(argv: list[str] | None = None) -> int:
     variants = selected_variants(args.variants)
     out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR
     case_sets = runner_argument_sets(forwarded)
+    if not any(case_key(arguments)[0] == "throughput" for _, arguments in case_sets):
+        variants = [variant for variant in variants if variant.name != "no-sync"]
     if args.dry_run:
         for variant in variants:
             for _, case_forwarded in case_sets:
+                if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput":
+                    continue
                 print(
                     shlex.join(
                         runner_command(
@@ -275,6 +316,8 @@ def main(argv: list[str] | None = None) -> int:
     try:
         for variant in variants:
             for case_label, case_forwarded in case_sets:
+                if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput":
+                    continue
                 command = runner_command(
                     out_dir, variant, args.models, case_forwarded, out_dir, False
                 )
@@ -296,7 +339,7 @@ def main(argv: list[str] | None = None) -> int:
         return 130
 
     remove_lock_files(out_dir)
-    selected_cases = DEFAULT_CASE_KEYS if len(case_sets) > 1 else None
+    selected_cases = frozenset(case_key(arguments) for _, arguments in case_sets)
     output, aggregation_failures = aggregate_results(
         out_dir,
         variants,

validation/tools/pim/pimcomp/compare/README.md

@@ -39,13 +39,20 @@ beside this script. An explicit `--out-dir PATH` writes
 Default model artifacts remain under `validation/networks/pimcomp_models`.
 Artifacts use
 `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
-under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
+under `artifacts/common/`
 and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
+YOLO and GoogLeNet use `artifacts/common-full-model/` to avoid reusing historical Softmax-free references.
 Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
 written report, including failures without simulation artifacts. Missing or
 unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
 compiler, simulation, or functional comparison fails.
 
+If fast-mode convergence fails, the reports and CSV retain valid full-duration
+fallback throughput and output interval values. This is not a converged steady-state result.
+Per-case JSON records `throughput_measurement: full_duration_fallback` and
+`fast_throughput_converged: false`; the text report labels the fallback.
+Zero completed outputs still give `NA` for per-inference metrics.
+
 Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
 mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
 Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
@@ -56,6 +63,14 @@ exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
 Tool/build revisions and manually edited reference artifacts are not hashed;
 rerun the relevant compiler after those change.
 
+Functional validation uses the same ONNX model and Raptor program as the performance comparison.
+YOLO uses the full model, including both internal Softmax operations.
+The functional simulator executes Softmax; Pimsim-nn and the queue estimator omit its timing cost.
+Reported timing and energy exclude Softmax work. Other instructions and transfers retain their costs.
+GoogLeNet removes only its two LRN nodes and retains terminal Softmax.
+VGG8 and both ResNet graphs have no removed operators.
+The native reference uses the same model as each compiled program.
+
 ## One-model comparator
 
 `compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
@@ -63,6 +78,13 @@ Use `--help` for hardware and simulation options. The suite runner supplies
 these automatically.
 The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
 `validate.py` uses `1e-3`/`1e-5`.
+Both use the [same output checks](../../../../README.md#performance-and-result-interpretation),
+including a relative L2 limit for each nonzero reference output.
+`--rtol=0` disables that additional limit in the comparator.
+Reports from an older validation policy require regeneration before reuse.
+An elementwise failure with relative L2 agreement remains `FAIL`.
+Its report notes possible numerical sensitivity. That note does not establish correctness.
+Every batch iteration must pass. Reported errors are maxima across iterations for each output.
 
 ## Comparison charts
 
@@ -71,19 +93,43 @@ The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
 ```
 
 Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
-`results_comparison.csv` beside the script and writes `plots/png/` and
-`plots/svg/` and `plots/pdf/` there. Use `--input PATH` for a runner-generated CSV and
+`results_comparison.csv` beside the script and writes `plots/default/png/`,
+`plots/default/svg/`, and `plots/default/pdf/` there. Use `--input PATH` for a runner-generated CSV and
 `--output-dir PATH` to override the destination. The plotter requires the full
 40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
 pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
 configurations are rejected before plots are written.
 
+Charts use the pastel palette and the Raptor name by default.
+Use `--no-colors` for grayscale output; `--colors` explicitly enables colors.
+Use `--anonymize` to replace displayed `Raptor` labels with `Anonimized`;
+`--no-anonymize` explicitly keeps the Raptor name. The flags are independent.
+Use `--no-marker-borders` for solid markers without outlines in the compact dot
+plot and line chart, including their legends. These markers use the darker
+border colors as solid fills. `--marker-borders` keeps the default outlines.
+Line colors and bar patterns remain unchanged.
+Input keys and output basenames remain stable. Use these output directories
+to keep the grayscale and colored anonymized versions:
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
+  --no-colors --anonymize \
+  --output-dir validation/tools/pim/pimcomp/compare/plots/gray_anonimized
+.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
+  --anonymize \
+  --output-dir validation/tools/pim/pimcomp/compare/plots/anonimized
+.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
+  --anonymize --no-marker-borders \
+  --output-dir validation/tools/pim/pimcomp/compare/plots/no_border_anonimized
+```
+
 | Plot family | Meaning |
 |---|---|
 | `validation_status` | Recorded functional status for each compiler and experiment. |
 | `validation_status_compact` | Compact functional status matrix. |
-| `speedup_compact_*` | Touching single-inference latency and best-pipeline throughput bars for each network. |
-| `energy_compact_*`, `power_compact_*` | Energy and power advantages for the same cases selected by the compact speedup plots. |
+| `performance_energy_compact` | Four horizontal dot panels: latency-mode speedup, throughput-mode speedup, and separate energy panels for both modes. |
+| `speedup_compact_*` | Latency panel on the left and best-pipeline throughput panel on the right, with touching Arch-A/B bars per network. |
+| `energy_compact_*`, `power_compact_*` | Energy reduction and power reduction for the same cases selected by the compact speedup plots. |
 | `latency_*` | Single-inference latency, latency mode only. |
 | `throughput_*` | Inferences/s, throughput mode only. |
 | `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
@@ -95,17 +141,47 @@ Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
 advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
 latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
 winning factor minus one, with parity at zero.
+Direct ratio axes start at 0.5, or 0.25 when a result is below 0.5.
+Panels in each figure share the same minimum.
 
 The shared `plot_style.py` defines the manuscript palette and display names.
 Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns,
 outlines, marker shapes, and status text preserve distinctions in grayscale.
-Compact plots use blue for the single-inference case and hatched sage
-green for the throughput case.
+Compact bar plots use plain blue for Arch-A and hatched sage green for Arch-B.
+The left panel contains latency-mode measurements. The right panel contains
+throughput-mode measurements. Each network has touching Arch-A/B bars.
+All compact figures use the display names `Latency-mode` and `Throughput-mode`.
 Each architecture/network pair uses the highest recorded throughput across pipelines.
 Energy and power use those same cases, without selecting a different pipeline for each metric.
 Validation labels remain visible for the selected cases. Units appear in axis
 titles and bar values; axis tick labels contain numbers only.
 
+The `performance_energy_compact` figure includes the eight Arch-A/B comparisons
+for VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It excludes YOLO11n.
+All panels share a linear ratio axis from 0.5, or 0.25 when needed, through the largest ratio,
+with a small right margin. A dashed line marks parity at 1.
+The shared axis title and ticks appear at the top, below the legend.
+The `Ratio (x)` title is centered over the axis. Panel titles appear to the left,
+rotate 90° counterclockwise, and use centered text on at most two lines.
+Titles start with `Latency-mode` or `Throughput-mode`, followed by the metric.
+The PIMCOMP-better and Raptor-better indicators flank the shared axis title.
+The figure has no mean markers.
+Labels give ratios to two decimal places; marker positions use unrounded ratios.
+Each panel has separate Arch-A and Arch-B lanes. The legend identifies models
+through marker shapes and pastel colors. Result labels contain ratios and use
+vertical text above or below their architecture line, without connectors.
+Placement prefers above for Arch-A and below for Arch-B.
+Lane spacing and label placement avoid other labels, markers, and reference lines.
+Larger dark-red daggers mark invalid PIMCOMP comparisons, including every batch comparison,
+even when the CSV records PASS. Marker borders retain the model colors;
+ratio values use normal text colors. The legend dagger matches the result daggers.
+Both use the same font and size. Labels without daggers reserve the same space,
+so numeric values align within each lane.
+These simulated ratios remain visible.
+The two energy panels show latency-mode energy and throughput-selected energy.
+The throughput energy uses the selected throughput pipeline, without a separate
+energy optimization.
+
 Throughput-stage bars touch within each network in both absolute and advantage
 plots. Bar-label widths are measured before export; dense figures expand only
 as needed to contain values and invalidity annotations. Legends use the available
@@ -116,6 +192,7 @@ pipeline plot retains its own scale, with y-axis ticks on both architecture rows
 Only the bottom row shows the shared x-axis ticks. Network names appear above
 the top row; architecture names appear on the right. That plot
 trims unused vertical range and is intended to span both manuscript columns.
+The line chart shows throughput only; validation status does not change its markers or colors.
 
 Failed validation is labeled `P. INVALID`; other non-PASS statuses keep their
 names, and missing measurements show `N/A`. Relative bars require both
@@ -130,3 +207,15 @@ publication provenance. The checked-in dataset is intentional. See the
 
 See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
 for measurement windows, units, numerical validity, and result provenance.
+
+Report reuse checks model and configuration hashes, compiler and simulator
+fingerprints, validation policy, actual tolerances, seed, batch size, and
+simulation options. Artifact signatures check instructions, weights, and memory
+images. Missing or changed artifacts require regeneration. Redundant Raptor
+JSON instruction dumps do not affect signatures when native binaries exist.
+
+`better_compiler` is available only when both compilers pass functional
+validation. `speedup` retains the diagnostic timing ratio when both timings
+exist. A timing ratio for a failed implementation does not identify a better
+correct implementation. Valid-only plot aggregates require both validations
+to pass.

validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py

@@ -18,7 +18,7 @@ import time
 import types
 from collections import Counter
 from concurrent.futures import ThreadPoolExecutor
-from dataclasses import asdict, dataclass
+from dataclasses import asdict, dataclass, field
 from pathlib import Path
 from tempfile import TemporaryDirectory
 from typing import Any
@@ -45,6 +45,7 @@ from raptor_validation.onnx_utils import (
     write_inputs_to_memory_bin,
 )
 from raptor_validation.raptor import compile_with_raptor
+from raptor_validation.output_comparison import VALIDATION_POLICY, compare_output_arrays
 from raptor_validation.pimsim_nn import (
     export_raptor_pimsim_artifact,
     parse_pimsim_nn_metrics,
@@ -75,6 +76,7 @@ class CompareResult:
     max_diffs: dict[str, float]
     status: str = "done"
     error: str | None = None
+    relative_l2_errors: dict[str, float | None] = field(default_factory=dict)
 
 
 def load_pimcomp_exporter():
@@ -264,20 +266,29 @@ def compare_simulator_outputs(
     channel_last: bool = False,
 ) -> CompareResult:
     sim_arrays = parse_pim_simulator_outputs(output_bin, outputs_desc)
+    if len(sim_arrays) != len(outputs_desc):
+        raise ValueError("simulator output count differs from the model output count")
     max_diffs: dict[str, float] = {}
+    relative_errors: dict[str, float | None] = {}
     passed = True
-    for sim_array, (idx, name, _, shape) in zip(sim_arrays, outputs_desc):
+    error = None
+    for sim_array, (idx, name, elem_type, shape) in zip(sim_arrays, outputs_desc):
         if channel_last and len(shape) == 4:
             n, c, h, w = shape
             sim_array = sim_array.reshape(n, h, w, c).transpose(0, 3, 1, 2)
         csv_name = reference_dir / f"output{idx}_{sanitize_output_name(name)}.csv"
-        ref = np.loadtxt(csv_name, delimiter=",", dtype=np.float32).reshape(shape)
-        diff = np.abs(sim_array.astype(np.float64) - ref.astype(np.float64))
-        allowed = threshold + rtol * np.abs(ref.astype(np.float64))
-        max_diffs[name] = float(np.max(diff))
-        if not np.all(diff <= allowed):
-            passed = False
-    return CompareResult(passed=passed, max_diffs=max_diffs)
+        ref = np.loadtxt(csv_name, delimiter=",", dtype=_ONNX_TO_NP[elem_type]).reshape(shape)
+        comparison = compare_output_arrays(sim_array, ref, atol=threshold, rtol=rtol)
+        max_diffs[name] = comparison.max_abs_diff
+        relative_errors[name] = comparison.relative_l2_error
+        passed &= comparison.passed
+        if not comparison.passed and error is None:
+            if (comparison.relative_l2_error is not None
+                and comparison.relative_l2_error <= rtol and not comparison.pointwise_passed):
+                error = f"{name}: elementwise mismatch despite relative L2 agreement; possible numerical sensitivity, not proof of correctness"
+            else:
+                error = f"{name}: output tolerance exceeded"
+    return CompareResult(passed=passed, max_diffs=max_diffs, relative_l2_errors=relative_errors, error=error)
 
 
 def sanitize_output_name(name: str) -> str:
@@ -652,7 +663,9 @@ def run_functional_validation(
         stage="Run functional simulation",
     )
     max_diffs: dict[str, float] = {}
+    relative_errors: dict[str, float | None] = {}
     failed_iterations = []
+    failure_detail = None
     for index, reference_dir in enumerate(reference_dirs):
         result = compare_simulator_outputs(
             batch_output_dir / f"output_{index:06d}.bin",
@@ -664,12 +677,20 @@ def run_functional_validation(
         )
         if not result.passed:
             failed_iterations.append(index)
+            if failure_detail is None:
+                failure_detail = result.error
         for name, diff in result.max_diffs.items():
             max_diffs[name] = max(max_diffs.get(name, 0.0), diff)
+        for name, relative in result.relative_l2_errors.items():
+            if relative is not None:
+                relative_errors[name] = max(relative_errors.get(name) or 0.0, relative)
+            else:
+                relative_errors.setdefault(name, None)
     return CompareResult(
         passed=not failed_iterations,
         max_diffs=max_diffs,
-        error=(f"batch iterations failed: {failed_iterations}" if failed_iterations else None),
+        error=(f"batch iterations failed: {failed_iterations}; {failure_detail}" if failed_iterations else None),
+        relative_l2_errors=relative_errors,
     )
 
 
@@ -1126,6 +1147,8 @@ def perf_status(perf: dict[str, Any]) -> str:
         return "SKIPPED"
     if perf.get("error"):
         return "FAILED"
+    if perf.get("throughput_measurement") == "full_duration_fallback":
+        return "DONE (full-duration fallback; unconverged)"
     return "DONE"
 
 
@@ -1163,29 +1186,77 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None:
     return Path(value) if value else None
 
 
-def pimcomp_reuse_context(
-    args: argparse.Namespace,
-    model: Path,
-    functional_model: Path,
-    hardware: dict[str, int],
-) -> dict[str, Any]:
-    files = {"model": model, "functional_model": functional_model,
-             "config": args.pimcomp_config}
-    digests = {}
-    for path in set(files.values()):
-        with path.open("rb") as stream:
-            digests[path] = hashlib.file_digest(stream, "sha256").hexdigest()
-    return {
-        "version": 1,
-        "files": {name: {"path": str(path), "sha256": digests[path]}
-                  for name, path in files.items()},
-        "hardware": hardware,
-        "settings": {name: getattr(args, name) for name in (
-            "pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline",
-            "pimcomp_replication", "pimcomp_model_name", "seed", "batch_size",
-            "threshold", "rtol", "no_fast", "skip_pimsim_nn",
-        )},
-    }
+def fingerprint_files(files: dict[str, Path]) -> dict[str, Any]:
+    fingerprints = {}
+    for name, path in files.items():
+        digest = None
+        if path.is_file():
+            with path.open("rb") as stream:
+                digest = hashlib.file_digest(stream, "sha256").hexdigest()
+        fingerprints[name] = {"path": str(path), "sha256": digest}
+    return fingerprints
+
+
+def experiment_context(args, model, functional_model, hardware, side):
+    files = {"model": model, "functional_model": functional_model, "config": args.pimcomp_config,
+             "oracle": args.pimsim_nn_build_dir / "ChipTest",
+             "comparison": Path(__file__),
+             "validation": VALIDATION_DIR / "raptor_validation/output_comparison.py",
+             "functional_simulator": args.pim_simulator_dir / "target/release/pim-simulator"}
+    if side == "raptor":
+        files["compiler"] = args.raptor_path
+    else:
+        files["compiler"] = args.pimcomp_dir / "build/PIMCOMP-NN"
+        files["exporter"] = args.pimcomp_dir / "verification/export_to_pim_simulator.py"
+        files.update({f"frontend/{p.relative_to(args.pimcomp_dir / 'frontend')}": p
+                      for p in sorted((args.pimcomp_dir / "frontend").rglob("*.py"))})
+    settings = {name: getattr(args, name) for name in (
+        "pimsim_mode", "pimsim_time_ms", "seed", "batch_size", "threshold", "rtol",
+        "no_fast", "skip_pimsim_nn")}
+    settings.update({name: getattr(args, name) for name in (
+        ("raptor_extra_arg",) if side == "raptor" else
+        ("pimcomp_pipeline", "pimcomp_replication", "pimcomp_model_name"))})
+    return {"version": 2, "validation_policy": VALIDATION_POLICY,
+            "files": fingerprint_files(files), "hardware": hardware, "settings": settings}
+
+
+def pimcomp_reuse_context(args, model, functional_model, hardware):
+    return experiment_context(args, model, functional_model, hardware, "pimcomp")
+
+
+def artifact_signatures(report, side):
+    names = (("raptor_pim", "raptor_pimsim_nn", "reference_runner", "reference_library",
+              "reference_inputs", "batch_inputs") if side == "raptor" else
+             ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn"))
+    signatures = {}
+    content = {}
+    for name in names:
+        root = path_from_report(report, name)
+        if root is None:
+            continue
+        if not root.exists():
+            signatures[name] = None
+            continue
+        paths = [root] if root.is_file() else sorted(p for p in root.rglob("*") if p.is_file())
+        binary = root.is_dir() and any(root.glob("core_*.pim"))
+        digest = hashlib.sha256()
+        for path in paths:
+            # JSON instruction dumps are redundant when native binaries exist.
+            if binary and path.parent == root and path.match("core_*.json"):
+                continue
+            resolved = path.resolve()
+            if resolved not in content:
+                with path.open("rb") as stream:
+                    content[resolved] = hashlib.file_digest(stream, "sha256").hexdigest()
+            digest.update(str(path.relative_to(root) if root.is_dir() else path.name).encode())
+            digest.update(b"\0" + content[resolved].encode() + b"\0")
+        signatures[name] = digest.hexdigest()
+    return signatures
+
+
+def verify_reused_artifacts(report, side):
+    if report.get("artifact_signatures", {}).get(side) != artifact_signatures(report, side):
+        raise ValueError(f"Reused {side} artifacts differ; regenerate the report")
 
 
 def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]:
@@ -1212,6 +1283,7 @@ def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> di
         restore_side_records(report, "pimcomp", [], [])
         for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"):
             path_from_report(report, name)
+        verify_reused_artifacts(report, "pimcomp")
         return report
     except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc:
         raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc
@@ -1334,14 +1406,19 @@ def write_report(
     if pimcomp_validation.error:
         lines.append(f"- Pimcomp validation note: `{pimcomp_validation.error.splitlines()[0]}`")
 
-    lines.extend(["", "### Max output differences", ""])
+    lines.extend(["", "### Output errors", ""])
     diff_names = sorted(set(raptor_validation.max_diffs) | set(pimcomp_validation.max_diffs))
     if diff_names:
-        lines.extend(["| Output | Raptor max diff | Pimcomp max diff |", "|---|---:|---:|"])
+        lines.extend(["| Output | Raptor max diff | Pimcomp max diff | Raptor relative L2 | Pimcomp relative L2 |",
+                      "|---|---:|---:|---:|---:|"])
         for name in diff_names:
+            relative = [result.relative_l2_errors.get(name)
+                        for result in (raptor_validation, pimcomp_validation)]
+            relative_text = [f"{value:.6e}" if value is not None else "N/A" for value in relative]
             lines.append(
                 f"| `{name}` | {raptor_validation.max_diffs.get(name, float('nan')):.6e} | "
-                f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} |"
+                f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} | "
+                f"{relative_text[0]} | {relative_text[1]} |"
             )
     else:
         lines.append("No output differences are available because validation did not run or failed before comparison.")
@@ -1358,24 +1435,24 @@ def write_report(
     if pimsim_mode == "throughput":
         lines.extend(
             [
-                "| Compiler | Status | Avg latency | Throughput | Avg power | Avg energy | Output count |",
+                "| Compiler | Status | Output interval (ms) | Throughput | Average power (mW) | Energy per inference (pJ/inference) | Output count |",
                 "|---|---|---:|---:|---:|---:|---:|",
                 f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'average_latency_ms', 'ms')} | {perf_value(raptor_perf, 'throughput', 'samples/s')} | "
-                f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(raptor_perf, 'output_count')} |",
+                f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(raptor_perf, 'output_count')} |",
                 f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'average_latency_ms', 'ms')} | {perf_value(pimcomp_perf, 'throughput', 'samples/s')} | "
-                f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(pimcomp_perf, 'output_count')} |",
+                f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(pimcomp_perf, 'output_count')} |",
                 "",
             ]
         )
     else:
         lines.extend(
             [
-                "| Compiler | Status | Latency | Avg power | Avg energy |",
+                "| Compiler | Status | Latency (ms) | Average power (mW) | Energy per inference (pJ/inference) |",
                 "|---|---|---:|---:|---:|",
                 f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'latency_ms', 'ms')} | "
-                f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ')} |",
+                f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} |",
                 f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'latency_ms', 'ms')} | "
-                f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ')} |",
+                f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} |",
                 "",
             ]
         )
@@ -1471,8 +1548,10 @@ def main():
     parser.add_argument("--pim-simulator-dir", default=REPO / "backend-simulators/pim/pim-simulator", type=Path)
     parser.add_argument("--pimsim-nn-build-dir", default=REPO / "backend-simulators/pim/pimsim-nn/build", type=Path)
     parser.add_argument("--seed", type=int, default=0)
-    parser.add_argument("--threshold", type=float, default=1e-3)
-    parser.add_argument("--rtol", type=float, default=1e-4)
+    parser.add_argument("--threshold", type=float, default=1e-3,
+                        help="Absolute tolerance for per-element output comparison.")
+    parser.add_argument("--rtol", type=float, default=1e-4,
+                        help="Relative tolerance for per-element and output L2 comparison; 0 disables the L2 check.")
     parser.add_argument(
         "--timeout-seconds",
         type=float,
@@ -1671,6 +1750,14 @@ def main():
         reuse_report_path = args.reuse_raptor_report.resolve()
         with open(reuse_report_path, "r", encoding="utf-8") as f:
             reused = json.load(f)
+        if reused.get("validation_policy") != VALIDATION_POLICY:
+            raise ValueError("Reused Raptor validation policy differs; regenerate the Raptor report")
+        if reused.get("validation_tolerances") != {"atol": args.threshold, "rtol": args.rtol}:
+            raise ValueError("Reused Raptor validation tolerances differ; regenerate the Raptor report")
+        expected = experiment_context(args, model_path, functional_model_path, hardware, "raptor")
+        if reused.get("raptor_reuse_context") != expected:
+            raise ValueError("Reused Raptor model, compiler, simulation settings, or inputs differ; regenerate the report")
+        verify_reused_artifacts(reused, "raptor")
         reused_hardware = reused["hardware"]
         if reused_hardware != hardware:
             raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
@@ -1848,13 +1935,13 @@ def main():
             )
             raptor_validation = validation if validation is not None else failed_validation("Raptor validation failed")
         elif not reference_dirs:
-            raptor_validation = skipped_validation("Reference outputs are not available")
+            raptor_validation = failed_validation("Reference outputs are not available")
         elif not outputs_desc:
-            raptor_validation = skipped_validation("Output descriptors are not available")
+            raptor_validation = failed_validation("Output descriptors are not available")
         else:
-            raptor_validation = skipped_validation("Raptor input materialization failed")
+            raptor_validation = failed_validation("Raptor input materialization failed")
     elif not reuse_raptor:
-        raptor_validation = skipped_validation("Raptor Pim compilation did not produce a Pim directory")
+        raptor_validation = failed_validation("Raptor Pim compilation did not produce a Pim directory")
 
     pimcomp_model_path = model_path
 
@@ -2058,6 +2145,8 @@ def main():
     )
 
     json_report = {
+        "validation_policy": VALIDATION_POLICY,
+        "validation_tolerances": {"atol": args.threshold, "rtol": args.rtol},
         "model": str(model_path),
         "functional_model": str(functional_model_path),
         "hardware": hardware,
@@ -2066,7 +2155,9 @@ def main():
         "pimcomp_pipeline": args.pimcomp_pipeline,
         "pimcomp_replication": args.pimcomp_replication,
         "pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx",
-        "pimcomp_reuse_context": reuse_context,
+        "pimcomp_reuse_context": (reuse_context if reuse_pimcomp else
+            None if args.raptor_only else pimcomp_reuse_context(args, model_path, functional_model_path, hardware)),
+        "raptor_reuse_context": experiment_context(args, model_path, functional_model_path, hardware, "raptor"),
         "pimcomp_config": str(args.pimcomp_config),
         "raptor_extra_args": args.raptor_extra_arg,
         "reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
@@ -2087,6 +2178,8 @@ def main():
             "batch_inputs": optional_path(out_dir / "inputs.csv"),
             "batch_outputs": optional_path(out_dir / "simulation/out_iterations"),
             "reference_runner": optional_path(runner_path),
+            "reference_library": optional_path(common_dir / "runner" / f"{functional_model_path.stem}.so")
+                if (common_dir / "runner" / f"{functional_model_path.stem}.so").is_file() else None,
             "raptor_pim": optional_path(raptor_pim_dir),
             "raptor_functional_pim": optional_path(raptor_functional_pim_dir),
             "raptor_pimsim_nn": optional_path(raptor_pimsim_dir),
@@ -2097,6 +2190,8 @@ def main():
             "report_markdown": str(report_path),
         },
     }
+    json_report["artifact_signatures"] = {
+        side: artifact_signatures(json_report, side) for side in ("raptor", "pimcomp")}
     json_path = out_dir / "pimcomp/comparison_report.json"
     with open(json_path, "w", encoding="utf-8") as f:
         json.dump(json_report, f, indent=2)

validation/tools/pim/pimcomp/compare/generate_plots.py

@@ -11,6 +11,7 @@ beside it, with outputs grouped in png/, svg/, and pdf/ under the output directo
 YOLO models without PIMCOMP measurements get separate Raptor-only charts.
 Comparisons use vertical bars for ratios, diverging advantages, and absolute
 values, with compiler colors and patterns for the Raptor pipeline stages.
+A compact dot-plot summary combines latency, throughput, and both energy cases.
 Throughput-mode latency is the output interval, a bottleneck-stage proxy,
 not end-to-end inference latency. Separate throughput plots use inferences/s.
 Energy is reported per inference; power is averaged over the simulation.
@@ -36,11 +37,13 @@ matplotlib.use("Agg")
 import matplotlib.pyplot as plt
 from matplotlib.patches import Patch, Rectangle
 from matplotlib.lines import Line2D
+from matplotlib.text import Text
 from matplotlib.ticker import FuncFormatter, MultipleLocator
+from matplotlib.transforms import Bbox
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
-from plot_style import (ARCH_NAMES, BAND_COLOR, COMPILER_NAMES, COMPILER_STYLES,
-                        ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR, METRIC_STYLES,
+from plot_style import (ARCH_NAMES, ARCH_STYLES, BAND_COLOR, COMPACT_MODE_NAMES, COMPILER_NAMES, COMPILER_STYLES,
+                        DAGGER_COLOR, ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR,
                         MODE_NAMES, MODEL_NAMES, NETWORK_STYLES,
                         PALETTE, TEXT_COLOR, add_legend, configure_style, fit_value_labels,
                         fit_bar_widths, fit_x_tick_labels, save_figure, value_label)
@@ -276,7 +279,7 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal
         gap = 0.4 if index and mode != configurations[index - 1][0] else 0
         positions.append(positions[-1] + 1 + gap if index else 0)
     configuration_x = dict(zip(configurations, positions))
-    headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact
+    headers = [f"{COMPILER_NAMES['raptor']}\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact
                else "PIMCOMP\nbatch" if mode == "throughput" and compiler == "pimcomp" and not compact
                else COMPILER_NAMES[compiler]
                for mode, stage, compiler in configurations]
@@ -288,7 +291,8 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal
         for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]):
             group_positions = [x for _, x in group]
             centers.append((group_positions[0] + group_positions[-1]) / 2)
-            modes.append("Latency" if mode == "latency" else MODE_NAMES[mode])
+            modes.append(COMPACT_MODE_NAMES[mode] if compact else
+                         "Latency" if mode == "latency" else MODE_NAMES[mode])
         ax.set_xticks(centers, modes, fontsize=10)
         ax.set_xticks(positions, headers, minor=True)
         ax.tick_params(axis="x", which="minor", top=True, labeltop=True,
@@ -320,6 +324,140 @@ def select_compact_speedups(results: list[Result]) -> list[Result]:
     return selected
 
 
+def draw_compact_summary(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None:
+    selected = select_compact_speedups([row for row in results if row.model != "yolo11n"])
+    lookup = {(row.model, row.arch, row.mode): row for row in selected}
+    keys = [(model, arch) for model in MODELS if model != "yolo11n" for arch in ARCHITECTURES]
+    panels = ((f"{COMPACT_MODE_NAMES['latency']}\nspeedup", "latency", "latency_speedup"),
+              (f"{COMPACT_MODE_NAMES['throughput']}\nspeedup", "throughput", "throughput_speedup"),
+              (f"{COMPACT_MODE_NAMES['latency']}\nenergy reduction", "latency", "energy"),
+              (f"{COMPACT_MODE_NAMES['throughput']}\nenergy reduction", "throughput", "energy"))
+    measurements = []
+    for _, mode, field in panels:
+        points = []
+        for model, arch in keys:
+            row = lookup[(model, arch, mode)]
+            ratio = getattr(row, field)
+            if not math.isfinite(ratio) or ratio <= 0:
+                raise ValueError(f"Compact summary requires a positive {field} ratio for {arch}/{model}")
+            pimcomp_invalid = row.pimcomp_schedule == "batch" or row.pimcomp_validation == "FAIL"
+            status = ("P. INVALID" if pimcomp_invalid else
+                      f"{COMPILER_NAMES['raptor'][0]}. INVALID" if row.raptor_validation == "FAIL" else
+                      "NOT VALIDATED" if not row.comparison_passed else "")
+            points.append((row, ratio, status))
+        measurements.append(points)
+    ratios = [ratio for points in measurements for _, ratio, _ in points]
+    padding = max(max(ratios) - min(ratios), 1) * 0.04
+    limits = (0.25 if min(ratios) < 0.5 else 0.5, max(1, *ratios) + padding)
+    fig, axes = plt.subplots(4, 1, sharex=True, figsize=(6.8, 7.6), layout="constrained")
+    markers = dict(zip((model for model in MODELS if model != "yolo11n"), ("o", "s", "^", "D")))
+    lanes = {arch: index * 2 for index, arch in enumerate(ARCHITECTURES)}
+    daggers = {}
+    for ax, (title, _, _), points in zip(axes, panels, measurements):
+        ax.set_xlim(*limits)
+        ax.set_ylim(4, -2)
+        ax.set_ylabel(title, rotation=90, ha="center", va="bottom", multialignment="center",
+                      fontsize=8.5, labelpad=10)
+        ax.set_yticks(list(lanes.values()), [ARCH_NAMES[arch] for arch in lanes], fontsize=8)
+        ax.set_xticks([limits[0], *range(1, math.floor(limits[1]) + 1)])
+        ax.xaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{value:g}"))
+        ax.tick_params(axis="both", length=0, pad=4, labelsize=8.5)
+        ax.tick_params(axis="x", bottom=False, labelbottom=False,
+                       top=ax is axes[0], labeltop=ax is axes[0])
+        ax.grid(axis="x", linewidth=0.6)
+        ax.axvline(1, color=TEXT_COLOR, linewidth=0.9, linestyle="--", zorder=1)
+        for lane in lanes.values():
+            ax.axhline(lane, color=LINE_COLOR, linewidth=0.6)
+        ax.spines["left"].set_visible(False)
+        ax.spines["bottom"].set_visible(False)
+        ax.spines["top"].set_visible(ax is axes[0])
+        for row, ratio, status in points:
+            fill, border = NETWORK_STYLES[row.model]
+            lane = lanes[row.arch]
+            ax.plot(ratio, lane, marker=markers[row.model], markersize=6.5, linestyle="none",
+                    markerfacecolor=fill if marker_borders else border, markeredgecolor=border,
+                    markeredgewidth=1.1 if marker_borders else 0)
+            suffix = f" [{status}]" if status and status != "P. INVALID" else ""
+            label = f"{ratio:.2f}×{suffix}"
+            annotation = ax.annotate(label, (ratio, lane), xytext=(0, -5), textcoords="offset points",
+                        rotation=90, rotation_mode="anchor", ha="right", va="center",
+                        fontsize=7.5, color=TEXT_COLOR, math_fontfamily="cm")
+            annotation.set_in_layout(False)
+            dagger = ax.annotate(r"$\boldsymbol{\dagger}$", (0.5, 1), xycoords=annotation,
+                                 xytext=(0, 1), textcoords="offset points", rotation=90,
+                                 rotation_mode="anchor", ha="left", va="center", fontsize=9,
+                                 color=DAGGER_COLOR, math_fontfamily="cm",
+                                 alpha=1 if status == "P. INVALID" else 0)
+            dagger.set_in_layout(False)
+            daggers[annotation] = dagger
+    handles = [Line2D([], [], marker=marker, color=NETWORK_STYLES[model][1],
+                      markerfacecolor=NETWORK_STYLES[model][0 if marker_borders else 1], markersize=6.5,
+                      markeredgewidth=1 if marker_borders else 0,
+                      linestyle="none", label=MODEL_NAMES[model])
+               for model, marker in markers.items()]
+    handles.append(Text(0, 0, r"$\boldsymbol{\dagger}$", color=DAGGER_COLOR, fontsize=9,
+                        math_fontfamily="cm", label="PIMCOMP invalid"))
+    add_legend(fig, handles)
+    axes[0].xaxis.set_label_position("top")
+    axes[0].set_xlabel("Ratio (x)", fontsize=9, labelpad=7, va="bottom")
+    axes[0].xaxis.set_label_coords(0.5, 1.18)
+    axes[0].text(0, 1.18, "← PIMCOMP better", transform=axes[0].transAxes, ha="left", va="bottom", fontsize=8)
+    axes[0].text(1, 1.18, f"{COMPILER_NAMES['raptor']} better →", transform=axes[0].transAxes,
+                 ha="right", va="bottom", fontsize=8)
+    fig.canvas.draw()
+    renderer = fig.canvas.get_renderer()
+
+    def label_bounds(text):
+        return Bbox.union([text.get_window_extent(renderer), daggers[text].get_window_extent(renderer)])
+
+    def position_label(text, dx, dy):
+        text.set_ha("left" if dy > 0 else "right")
+        extra = (daggers[text].get_window_extent(renderer).height + renderer.points_to_pixels(1)
+                 if dy < 0 else 0)
+        text.set_position((dx, dy - extra * 72 / fig.dpi))
+
+    labels_by_axis = [[text for text in ax.texts if getattr(text, "xycoords", None) == "data"] for ax in axes]
+    label_height = max(label_bounds(text).height for labels in labels_by_axis for text in labels)
+    lane_gap = min(abs(ax.transData.transform((0, 0))[1] - ax.transData.transform((0, 2))[1])
+                   for ax in axes)
+    height_scale = (label_height + renderer.points_to_pixels(5)) / lane_gap
+    if height_scale > 1:
+        fig.set_size_inches(fig.get_figwidth(), fig.get_figheight() * height_scale)
+        fig.canvas.draw()
+        renderer = fig.canvas.get_renderer()
+    for ax, labels in zip(axes, labels_by_axis):
+        placed = []
+        for line in ax.lines:
+            placed.append(line.get_window_extent(renderer).padded(1))
+        bounds = ax.get_window_extent(renderer)
+        choices = []
+        for text in labels:
+            preferred_dy = 5 if text.xy[1] == lanes[ARCHITECTURES[0]] else -5
+            candidates = []
+            for dx in (0, -3, 3, -6, 6, -8, 8):
+                for dy in (-5, 5):
+                    position_label(text, dx, dy)
+                    box = label_bounds(text)
+                    overlap = any(box.overlaps(other) for other in placed)
+                    outside = not (bounds.x0 <= box.x0 <= box.x1 <= bounds.x1 and
+                                   bounds.y0 <= box.y0 <= box.y1 <= bounds.y1)
+                    if not outside and not overlap:
+                        candidates.append(((dy != preferred_dy, abs(dx) + abs(dy)), dx, dy, box))
+            choices.append((text, candidates))
+        layouts = [((0, 0), [])]
+        for text, candidates in sorted(choices, key=lambda choice: len(choice[1])):
+            layouts = sorted(
+                (((cost[0] + distance[0], cost[1] + distance[1]), assignments + [(text, dx, dy, box)])
+                 for cost, assignments in layouts for distance, dx, dy, box in candidates
+                 if not any(box.overlaps(other[3]) for other in assignments)),
+                key=lambda layout: layout[0])[:32]
+            if not layouts:
+                raise ValueError("Dot labels do not fit without overlap; increase lane spacing or figure width")
+        for text, dx, dy, _ in layouts[0][1]:
+            position_label(text, dx, dy)
+    save_figure(fig, "performance_energy_compact", output_dir)
+
+
 def draw_metric(results: list[Result], field: str, ylabel: str,
                 basename: str, output_dir: Path, diverging: bool = False,
                 compact: bool = False) -> None:
@@ -327,48 +465,52 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
             for row in results]
     available = [v for v in data if math.isfinite(v)]
     upper = max(0, max(available, default=1.0))
-    lower = min(0, min(available, default=0)) if diverging else 0
+    lower = (min(0, min(available, default=0)) if diverging else
+             0.25 if min(available, default=1) < 0.5 else 0.5)
     padding = max(upper - lower, 1.0) * 0.3
     y_max = math.ceil((upper + padding) * 2) / 2
-    y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0
+    y_min = math.floor((lower - padding) * 2) / 2 if diverging else lower
     y_interval = 1.0 if y_max - y_min > 4.5 else 0.5
 
     fig, axes = plt.subplots(1, 2, sharey=True, figsize=(8.5 if compact else 12, 4.4),
                              layout="constrained")
     if compact:
-        single_metric = "latency" if field == "speedup" else field
-        throughput_suffix = "" if field == "speedup" else f" {field}"
-        handles = [Patch(facecolor=METRIC_STYLES["latency"][0],
-                         edgecolor=METRIC_STYLES["latency"][1],
-                         label=f"Single-inference {single_metric}"),
-                   Patch(facecolor=METRIC_STYLES["throughput"][0],
-                         edgecolor=METRIC_STYLES["throughput"][1], hatch="//",
-                         label=f"Best-case throughput{throughput_suffix}")]
+        metric_suffix = "" if field == "speedup" else f" {field}"
+        panel_labels = {mode: label + metric_suffix for mode, label in COMPACT_MODE_NAMES.items()}
+        handles = [Patch(facecolor=fill, edgecolor=border, hatch=pattern, label=ARCH_NAMES[arch])
+                   for arch, (fill, border, pattern) in ARCH_STYLES.items()]
 
-    for ax, arch in zip(axes, ARCHITECTURES):
-        selected = [row for row in results if row.arch == arch]
+    panels = ("latency", "throughput") if compact else ARCHITECTURES
+    for panel_index, (ax, panel) in enumerate(zip(axes, panels)):
+        selected = [row for row in results if getattr(row, "mode" if compact else "arch") == panel]
+        if compact:
+            selected.sort(key=lambda row: (MODELS.index(row.model), ARCHITECTURES.index(row.arch)))
         positions = [index + index // 2 for index in range(len(selected))] if compact else grouped_positions(selected)
-        ax.set_xlabel(ARCH_NAMES[arch],
+        ax.set_xlabel(panel_labels[panel] if compact else ARCH_NAMES[panel],
                       fontsize=11, weight="bold", labelpad=8)
         ax.set_axisbelow(True)
         ax.set_ylim(y_min, y_max)
         ax.set_xlim(-0.9, positions[-1] + 0.9 if positions else 0.9)
         label_groups(ax, selected, positions)
-        ax.tick_params(axis="y", labelleft=arch == ARCHITECTURES[0], pad=5, length=0)
+        ax.tick_params(axis="y", labelleft=panel_index == 0, pad=5, length=0)
         ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6)
         ax.yaxis.set_major_locator(MultipleLocator(y_interval))
         if diverging:
             ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}"))
+        else:
+            ax.set_yticks([y_min], minor=True)
+            ax.yaxis.set_minor_formatter(FuncFormatter(lambda value, _: f"{value:g}"))
+            ax.tick_params(axis="y", which="minor", labelleft=panel_index == 0, pad=5, length=0)
         ax.grid(axis="y")
         ax.axhline(0 if diverging else 1, color=TEXT_COLOR, linewidth=1.0,
                    linestyle="--", zorder=3)
-        if arch == ARCHITECTURES[0]:
+        if panel_index == 0:
             ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6)
 
         for position, row in zip(positions, selected):
             ratio = getattr(row, field)
             if not math.isfinite(ratio):
-                ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
+                ax.text(position, (0 if diverging else y_min) + (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
                         fontsize=9.5, color=TEXT_COLOR, fontstyle="italic")
                 continue
             compiler = advantage_compiler(ratio)
@@ -377,8 +519,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
             pattern = STAGE_PATTERNS[row.pipeline]
             color, border = COMPILER_STYLES[compiler] if compiler else ("white", TEXT_COLOR)
             if compact:
-                pattern = "//" if row.mode == "throughput" else ""
-                color, border = METRIC_STYLES[row.mode]
+                color, border, pattern = ARCH_STYLES[row.arch]
             measurement = diverging_ratio(ratio) if diverging else ratio
             ax.bar(position, measurement, width=1.0,
                     color=color,
@@ -395,7 +536,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
     else:
         figure_legend(fig, results, field)
     fit_x_tick_labels(fig)
-    fit_value_labels(fig)
+    fit_value_labels(fig, lower_limit=None if diverging else y_min)
     save_figure(fig, basename, output_dir)
 
 
@@ -463,7 +604,7 @@ def draw_absolute(results: list[Result], field: str, ylabel: str,
     save_figure(fig, basename, output_dir)
 
 
-def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
+def draw_pipeline_effect(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None:
     rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1]
     stages = sorted({row.pipeline for row in rows})
     panels = [model for model in MODELS if any(row.model == model for row in rows)]
@@ -479,15 +620,12 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
             ax = axes[index, column]
             keys = [(arch, model, stage) for stage in stages]
             values = [rates.get(key, math.nan) for key in keys]
-            valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys]
             ax.plot(stages, values, color=border, linewidth=1.8)
-            for stage, value, passed in zip(stages, values, valid):
+            for stage, value in zip(stages, values):
                 if math.isfinite(value):
-                    marker = ("s" if stage == 1 else "o") if passed else "x"
-                    status = lookup[(arch, model, stage)].raptor_validation
-                    ax.plot(stage, value, color=ERROR_COLOR if status == "FAIL" else border,
-                            markerfacecolor=fill, marker=marker,
-                            markersize=6, linestyle="none")
+                    ax.plot(stage, value, color=border,
+                            markerfacecolor=fill if marker_borders else border, marker="s" if stage == 1 else "o",
+                            markeredgewidth=1 if marker_borders else 0, markersize=7, linestyle="none")
             ax.set_xticks(stages)
             if index == 0:
                 ax.set_title(MODEL_NAMES[model], fontsize=10.5, weight="normal", pad=8)
@@ -500,15 +638,13 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
                 ax.set_ylabel(ARCH_NAMES[arch], fontsize=10.5, weight="bold", rotation=270, labelpad=24)
     for ax in axes.flat:
         ax.margins(y=0.08)
-    handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white",
-                      marker=marker, linestyle="none", label=label)
+    handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white" if marker_borders else TEXT_COLOR,
+                      markeredgewidth=1 if marker_borders else 0,
+                      marker=marker, markersize=7, linestyle="none", label=label)
                for marker, label in (("s", "Derived as 1 / latency"),
                                      ("o", "Throughput measurement"))]
-    if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()):
-        handles.append(Line2D([], [], color=TEXT_COLOR, marker="x", linestyle="none",
-                              label="Raptor validation not PASS"))
     add_legend(fig, handles)
-    fig.supxlabel("Raptor pipeline stages", fontsize=11)
+    fig.supxlabel(f"{COMPILER_NAMES['raptor']} pipeline stages", fontsize=11)
     fit_x_tick_labels(fig)
     save_figure(fig, "raptor_pipeline_throughput", output_dir)
 
@@ -517,16 +653,22 @@ def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__)
     script_dir = Path(__file__).resolve().parent
     parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv")
-    parser.add_argument("--output-dir", type=Path, default=script_dir / "plots")
+    parser.add_argument("--output-dir", type=Path, default=script_dir / "plots/default")
+    parser.add_argument("--colors", action=argparse.BooleanOptionalAction, default=True,
+                        help="Use the pastel palette (default); --no-colors exports grayscale charts")
+    parser.add_argument("--anonymize", action=argparse.BooleanOptionalAction, default=False,
+                        help="Replace the displayed Raptor name with Anonimized")
+    parser.add_argument("--marker-borders", action=argparse.BooleanOptionalAction, default=True,
+                        help="Outline markers (default); --no-marker-borders uses solid markers in dot and line charts")
     args = parser.parse_args()
-    configure_style()
+    configure_style(colors=args.colors, anonymize=args.anonymize)
     results = load_results(args.input)
     valid = sum(row.comparison_passed for row in results)
     print(f"Results: {len(results)} configurations, {valid} valid comparisons, "
           f"{len(results)-valid} comparisons without two PASS statuses")
     draw_validation(results, args.output_dir)
     draw_validation(results, args.output_dir, compact=True)
-    draw_pipeline_effect(results, args.output_dir)
+    draw_pipeline_effect(results, args.output_dir, marker_borders=args.marker_borders)
     yolo_only_models = {
         model for model in MODELS if model.startswith("yolo")
         and not any(math.isfinite(getattr(row, f"pimcomp_{field}"))
@@ -536,9 +678,10 @@ def main() -> None:
     comparisons = [row for row in results if row.model not in yolo_only_models]
     yolo_only = [row for row in results if row.model in yolo_only_models]
     compact_speedups = select_compact_speedups(comparisons)
+    draw_compact_summary(comparisons, args.output_dir, marker_borders=args.marker_borders)
     for metric, label in (("speedup", "Speedup (x)"),
-                          ("energy", "Energy-per-inference advantage (x)"),
-                          ("power", "Average power advantage (x)")):
+                          ("energy", "Energy reduction (x)"),
+                          ("power", "Average power reduction (x)")):
         for suffix, diverging in (("comparison", False), ("diverging", True)):
             draw_metric(compact_speedups, metric, label,
                         f"{metric}_compact_{suffix}", args.output_dir, diverging=diverging, compact=True)
@@ -546,8 +689,8 @@ def main() -> None:
         ("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None),
         ("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"),
         ("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"),
-        ("power", "power", "Average power advantage (x)", None),
-        ("energy", "energy", "Energy-per-inference advantage (x)", None),
+        ("power", "power", "Average power reduction (x)", None),
+        ("energy", "energy", "Energy reduction (x)", None),
     ]:
         selected = [row for row in comparisons if mode is None or row.mode == mode]
         draw_metric(selected, metric, label,

validation/tools/pim/pimcomp/compare/results_comparison.csv

@@ -1,41 +1,41 @@
 arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
-arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26
-arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,429.31,482.60,407972136.20,541691691.00,raptor,1.14
-arch-a,vgg8,throughput,4,batch,PASS,PASS,0.78,1.04,1280.00,965.00,457.04,482.60,415761605.20,541691691.00,raptor,1.33
-arch-a,vgg8,throughput,8,batch,PASS,PASS,0.78,1.04,1290.00,965.00,464.07,482.60,396424007.70,541691691.00,raptor,1.34
-arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394478521.12,13868866808.12,raptor,4.15
-arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
-arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.21,13.46,52.10,74.30,380.10,481.60,7957033431.00,6563451004.00,pimcomp,1.43
-arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.93,13.46,50.20,74.30,361.80,481.60,7763795670.00,6563451004.00,pimcomp,1.48
-arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
-arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.27,23.58,43.00,42.40,471.72,515.68,13055037600.00,12283627410.00,raptor,1.01
-arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.47,23.58,36.40,42.40,422.22,515.68,14899055210.00,12283627410.00,pimcomp,1.16
-arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.84,23.58,25.10,42.40,352.28,515.68,17062658890.00,12283627410.00,pimcomp,1.69
-arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.87,231.19,5513262106.92,14547744283.24,raptor,6.44
-arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.42,18.26,119.00,54.80,610.75,435.11,5777456331.00,8164787803.00,raptor,2.17
-arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.73,18.26,93.20,54.80,528.03,435.11,6592892213.00,8164787803.00,raptor,1.70
-arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.12,18.26,98.80,54.80,523.81,435.11,6521737843.00,8164787803.00,raptor,1.80
-arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
-arch-a,yolo11n,throughput,2,batch,PASS,FAIL,102.34,NA,9.77,NA,337.88,NA,38308060150.00,NA,NA,NA
-arch-a,yolo11n,throughput,4,batch,PASS,FAIL,87.91,NA,11.40,NA,358.20,NA,38729846260.00,NA,NA,NA
-arch-a,yolo11n,throughput,8,batch,PASS,FAIL,83.68,NA,11.90,NA,365.64,NA,38623589740.00,NA,NA,NA
-arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
-arch-b,vgg8,throughput,2,batch,PASS,PASS,0.69,1.07,1460.00,932.00,456.04,409.47,403694238.30,439288835.80,raptor,1.57
-arch-b,vgg8,throughput,4,batch,PASS,PASS,0.67,1.07,1490.00,932.00,448.90,409.47,413526781.30,439288835.80,raptor,1.60
-arch-b,vgg8,throughput,8,batch,PASS,PASS,0.75,1.07,1330.00,932.00,425.74,409.47,394457860.80,439288835.80,raptor,1.43
-arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
-arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.12,13.43,82.50,74.40,461.41,484.80,6295321671.00,6569655551.00,raptor,1.11
-arch-b,resnet18,throughput,4,batch,PASS,FAIL,12.68,13.43,78.90,74.40,451.19,484.80,7096979122.00,6569655551.00,raptor,1.06
-arch-b,resnet18,throughput,8,batch,PASS,FAIL,20.59,13.43,48.60,74.40,321.63,484.80,7162570047.00,6569655551.00,pimcomp,1.53
-arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
-arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
-arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.09,22.85,34.40,43.80,364.96,493.33,16328513230.00,11392215660.00,pimcomp,1.27
-arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.85,21.00,43.80,289.82,493.33,16346419570.00,11392215660.00,pimcomp,2.09
-arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422781257.92,9182062336.24,raptor,3.38
-arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21,21.39,109.00,46.80,545.49,417.87,5593397699.00,9014726090.00,raptor,2.33
-arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56,21.39,117.00,46.80,555.26,417.87,5863121048.00,9014726090.00,raptor,2.50
-arch-b,googlenet,throughput,8,batch,PASS,FAIL,16.19,21.39,61.80,46.80,355.70,417.87,6726458739.00,9014726090.00,raptor,1.32
-arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
-arch-b,yolo11n,throughput,2,batch,PASS,FAIL,104.49,NA,9.57,NA,302.12,NA,35801517850.00,NA,NA,NA
-arch-b,yolo11n,throughput,4,batch,PASS,FAIL,108.25,NA,9.24,NA,297.48,NA,37629699620.00,NA,NA,NA
-arch-b,yolo11n,throughput,8,batch,PASS,FAIL,100.13,NA,9.99,NA,308.83,NA,37180785770.00,NA,NA,NA
+arch-a,vgg8,latency,1,element,PASS,PASS,1.51833,7.98507,NA,NA,321.046,200.111,4.87455e+08,1.5979e+09,raptor,5.26
+arch-a,vgg8,throughput,2,batch,PASS,PASS,0.906321,1.0364,1100,965,429.311,482.597,4.07972e+08,5.41692e+08,raptor,1.14
+arch-a,vgg8,throughput,4,batch,PASS,PASS,0.778242,1.0364,1280,965,457.039,482.597,4.15762e+08,5.41692e+08,raptor,1.33
+arch-a,vgg8,throughput,8,batch,PASS,PASS,0.778094,1.0364,1290,965,464.071,482.597,3.96424e+08,5.41692e+08,raptor,1.34
+arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,raptor,4.15
+arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.7314,13.4585,85.2,74.3,503.111,481.604,6.62807e+09,6.56345e+09,raptor,1.15
+arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.2072,13.4585,52.1,74.3,380.1,481.604,7.95703e+09,6.56345e+09,pimcomp,1.43
+arch-a,resnet18,throughput,8,batch,PASS,FAIL,14.1578,13.4585,70.6,74.3,453.416,481.604,6.77909e+09,6.56345e+09,pimcomp,1.05
+arch-a,resnet34,latency,1,element,PASS,FAIL,29.7186,66.7114,NA,NA,411.806,277.485,1.22383e+10,1.85114e+10,raptor,2.24
+arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.2705,23.5817,43,42.4,471.717,515.68,1.3055e+10,1.22836e+10,raptor,1.01
+arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.468,23.5817,36.4,42.4,422.223,515.68,1.48991e+10,1.22836e+10,pimcomp,1.16
+arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.8392,23.5817,25.1,42.4,352.28,515.68,1.70627e+10,1.22836e+10,pimcomp,1.69
+arch-a,googlenet,latency,1,element,PASS,PASS,9.77772,62.9248,NA,NA,563.864,231.193,5.5133e+09,1.45477e+10,raptor,6.44
+arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.41545,18.2643,119,54.8,610.746,435.107,5.77746e+09,8.16479e+09,raptor,2.17
+arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.7292,18.2643,93.2,54.8,528.026,435.107,6.59289e+09,8.16479e+09,raptor,1.70
+arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.1219,18.2643,98.8,54.8,523.808,435.107,6.52174e+09,8.16479e+09,raptor,1.80
+arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.174,NA,4.27977e+10,NA,NA,NA
+arch-a,yolo11n,throughput,2,batch,SKIPPED,FAIL,102.337,NA,9.77,NA,337.882,NA,3.83081e+10,NA,NA,NA
+arch-a,yolo11n,throughput,4,batch,SKIPPED,FAIL,87.9103,NA,11.4,NA,358.201,NA,3.87298e+10,NA,NA,NA
+arch-a,yolo11n,throughput,8,batch,SKIPPED,FAIL,82.9508,NA,12.1,NA,366.725,NA,3.85347e+10,NA,NA,NA
+arch-b,vgg8,latency,1,element,PASS,PASS,1.5333,7.15212,NA,NA,292.559,173.769,4.48582e+08,1.24281e+09,raptor,4.66
+arch-b,vgg8,throughput,2,batch,PASS,PASS,0.685556,1.07282,1460,932,456.044,409.47,4.03694e+08,4.39289e+08,raptor,1.57
+arch-b,vgg8,throughput,4,batch,PASS,PASS,0.673192,1.07282,1490,932,448.898,409.47,4.13527e+08,4.39289e+08,raptor,1.60
+arch-b,vgg8,throughput,8,batch,PASS,PASS,0.751106,1.07282,1330,932,425.742,409.47,3.94458e+08,4.39289e+08,raptor,1.43
+arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,raptor,4.19
+arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.1214,13.434,82.5,74.4,461.407,484.804,6.29532e+09,6.56966e+09,raptor,1.11
+arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3601,13.434,88,74.4,479.333,484.804,6.98745e+09,6.56966e+09,raptor,1.18
+arch-b,resnet18,throughput,8,batch,PASS,FAIL,13.1216,13.434,76.2,74.4,441.248,484.804,6.38271e+09,6.56966e+09,raptor,1.02
+arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,raptor,3.83
+arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.282,22.8454,44.9,43.8,438.238,493.332,1.42187e+10,1.13922e+10,raptor,1.03
+arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.0906,22.8454,34.4,43.8,364.956,493.332,1.63285e+10,1.13922e+10,pimcomp,1.27
+arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.8454,21,43.8,289.821,493.332,1.63464e+10,1.13922e+10,pimcomp,2.09
+arch-b,googlenet,latency,1,element,PASS,PASS,11.2212,37.9367,NA,NA,483.266,242.037,5.42282e+09,9.18206e+09,raptor,3.38
+arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21209,21.3902,109,46.8,545.488,417.866,5.5934e+09,9.01473e+09,raptor,2.33
+arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56194,21.3902,117,46.8,555.264,417.866,5.86312e+09,9.01473e+09,raptor,2.50
+arch-b,googlenet,throughput,8,batch,PASS,FAIL,10.8652,21.3902,92,46.8,483.477,417.866,6.12683e+09,9.01473e+09,raptor,1.97
+arch-b,yolo11n,latency,1,element,PASS,FAIL,168.144,NA,NA,NA,245.314,NA,4.1248e+10,NA,NA,NA
+arch-b,yolo11n,throughput,2,batch,SKIPPED,FAIL,104.494,NA,9.57,NA,302.121,NA,3.58015e+10,NA,NA,NA
+arch-b,yolo11n,throughput,4,batch,SKIPPED,FAIL,108.247,NA,9.24,NA,297.484,NA,3.76297e+10,NA,NA,NA
+arch-b,yolo11n,throughput,8,batch,SKIPPED,FAIL,100.129,NA,9.99,NA,308.833,NA,3.71808e+10,NA,NA,NA

validation/tools/pim/pimcomp/compare/run_pimcomp_models.py

@@ -23,13 +23,13 @@ REPO = Path(__file__).resolve().parents[5]
 sys.path.insert(0, str(REPO / "validation"))
 
 from raptor_validation.pimcomp_models import (
-    FUNCTIONAL_MODELS,
     MODELS,
     SUITE,
     add_models_argument,
 )
 from raptor_validation.artifacts import artifacts_dir, remove_lock_files
 from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics
+from raptor_validation.output_comparison import VALIDATION_POLICY
 from raptor_validation.validate_one import STAGE_COLORS
 
 PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
@@ -63,7 +63,7 @@ class ComparisonSpec:
 
 
 def model_dir(root: Path | None, name: str) -> Path:
-    return root / name if root is not None else MODELS[name].parent
+    return (root or SUITE) / name
 
 
 def result_dir(
@@ -83,9 +83,9 @@ def result_dir(
 
 
 def common_dir(root: Path | None, name: str, common_root: Path | None = None) -> Path:
-    suffix = "common" if FUNCTIONAL_MODELS[name] == MODELS[name] else "common-functional"
     base = common_root / name if common_root is not None else model_dir(root, name)
-    return artifacts_dir(base) / suffix
+    # Keep references that retain Softmax separate from historical outputs.
+    return artifacts_dir(base) / ("common-full-model" if name in ("yolo11n", "googlenet") else "common")
 
 
 def clean_artifacts(
@@ -199,6 +199,8 @@ def write_results_csv(
             }
             if report_path.exists():
                 report = json.loads(report_path.read_text(encoding="utf-8"))
+                if report.get("validation_policy") != VALIDATION_POLICY:
+                    raise ValueError(f"Report uses a different validation policy: {report_path}. Regenerate this comparison.")
                 raptor_values = performance_values(report.get("raptor_performance") or {})
                 pimcomp_values = performance_values(report.get("pimcomp_performance") or {})
                 raptor_metric = raptor_values["throughput"] if comparison_mode == "throughput" else raptor_values["latency"]
@@ -222,7 +224,11 @@ def write_results_csv(
                 if raptor_metric is not None and pimcomp_metric is not None:
                     min_metric = min(raptor_metric, pimcomp_metric)
                     row.update(
-                        better_compiler=comparison_winner(comparison_mode, raptor_metric, pimcomp_metric),
+                        better_compiler=(
+                            comparison_winner(comparison_mode, raptor_metric, pimcomp_metric)
+                            if row["raptor_functional_validation"] == row["pimcomp_functional_validation"] == "PASS"
+                            else "NA"
+                        ),
                         speedup=(
                             f"{max(raptor_metric, pimcomp_metric) / min_metric:.2f}"
                             if min_metric > 0
@@ -684,10 +690,9 @@ def main() -> int:
     )
 
     missing = [
-        str(path)
+        str(MODELS[name])
         for name in args.models
-        for path in (MODELS[name], FUNCTIONAL_MODELS[name])
-        if not path.exists()
+        if not MODELS[name].exists()
     ]
     if missing:
         parser.error(f"missing model(s): {', '.join(missing)}")
@@ -733,7 +738,7 @@ def main() -> int:
             try:
                 run(
                     prepare_common_command(
-                        FUNCTIONAL_MODELS[name],
+                        MODELS[name],
                         common_dir(out_dir, name, common_root),
                         args.timeout_seconds,
                         reference_batch_size,
@@ -770,7 +775,7 @@ def main() -> int:
                     ComparisonSpec(
                         label=label,
                         model=MODELS[name],
-                        functional_model=FUNCTIONAL_MODELS[name],
+                        functional_model=MODELS[name],
                         output_dir=model_result_dir,
                         common_dir=common_dir(out_dir, name, common_root),
                         config=configs[mode],

validation/tools/pim/pimcomp/correctness/README.md

@@ -49,6 +49,13 @@ loaded provider tensors have no store producer; prefill reduces maximum
 reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's
 verifier contract; it does not repair the standalone batch program.
 
+The reconstructed VGG8 batch program also loads intermediate Pool and Flatten
+buffers without corresponding store producers. Its output can remain zero.
+The small reference output can fit inside an absolute tolerance despite this
+missing computation. Comparison therefore requires a relative L2 check in
+addition to the elementwise check. A zero output against a nonzero reference
+has relative L2 error one and must fail the default relative tolerance.
+
 ## ResNet BatchNorm gap
 
 Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)

validation/tools/pim/plot_style.py

@@ -6,10 +6,14 @@ from math import ceil
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
+from matplotlib.colors import to_rgba
+from matplotlib.legend_handler import HandlerBase
+from matplotlib.lines import Line2D
 from matplotlib.patches import Patch
 from matplotlib.path import Path as PlotPath
 from matplotlib.patheffects import AbstractPathEffect
-from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath
+from matplotlib.text import Text
+from matplotlib.transforms import Affine2D, Bbox, IdentityTransform, TransformedPath
 
 
 PALETTE = {
@@ -25,11 +29,12 @@ PALETTE = {
 TEXT_COLOR = "#222222"
 MATRIX_TEXT_COLOR = "#1D2530"
 ERROR_COLOR = PALETTE["red"][1]
+DAGGER_COLOR = "#AA1C1C"
 GRID_COLOR = "#E6E9EC"
 LINE_COLOR = "#C9CED3"
 BAND_COLOR = "#F7F9FA"
 COMPILER_STYLES = {"raptor": PALETTE["sage"], "pimcomp": PALETTE["sand"]}
-METRIC_STYLES = {"latency": PALETTE["blue"], "throughput": PALETTE["sage"]}
+ARCH_STYLES = {"arch-a": (*PALETTE["blue"], ""), "arch-b": (*PALETTE["sage"], "//")}
 NETWORK_STYLES = {"vgg8": PALETTE["blue"], "resnet18": PALETTE["sage"],
                   "resnet34": PALETTE["sand"], "googlenet": PALETTE["purple"],
                   "yolo11n": PALETTE["red"]}
@@ -38,6 +43,8 @@ ARCH_NAMES = {"arch-a": "Arch-A", "arch-b": "Arch-B"}
 MODEL_NAMES = {"vgg8": "VGG-8", "resnet18": "ResNet-18", "resnet34": "ResNet-34",
                "googlenet": "GoogLeNet", "yolo11n": "YOLO11n"}
 MODE_NAMES = {"latency": "Single-inference", "throughput": "Throughput"}
+COMPACT_MODE_NAMES = {"latency": "Latency-mode", "throughput": "Throughput-mode"}
+_USE_COLORS = True
 
 
 class CenteredHatch(AbstractPathEffect):
@@ -47,11 +54,14 @@ class CenteredHatch(AbstractPathEffect):
         hatch = gc.get_hatch() or ""
         base_gc = renderer.new_gc()
         base_gc.copy_properties(gc)
-        base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\"))
+        base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\") or None)
         renderer.draw_path(base_gc, tpath, affine, rgbFace)
         base_gc.restore()
 
         bounds = affine.transform_path(tpath).get_extents()
+        clip_bounds = bounds if gc.get_clip_rectangle() is None else Bbox.intersection(bounds, gc.get_clip_rectangle())
+        if clip_bounds is None:
+            return
         center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2
         vertices = []
         diagonal = 2 ** -0.5
@@ -61,13 +71,16 @@ class CenteredHatch(AbstractPathEffect):
                 continue
             spacing = renderer.points_to_pixels(12 / count)
             half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2
-            half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2
             extent = ceil(half_span / spacing)
             for step in range(-extent, extent + 1):
                 x = center_x - dy * step * spacing
                 y = center_y + dx * step * spacing
-                vertices.extend(((x - dx * half_length, y - dy * half_length),
-                                 (x + dx * half_length, y + dy * half_length)))
+                y_limits = sorted(((clip_bounds.y0 - y) / dy, (clip_bounds.y1 - y) / dy))
+                start = max((clip_bounds.x0 - x) / dx, y_limits[0])
+                end = min((clip_bounds.x1 - x) / dx, y_limits[1])
+                if start < end:
+                    vertices.extend(((x + dx * start, y + dy * start),
+                                     (x + dx * end, y + dy * end)))
         line_gc = renderer.new_gc()
         line_gc.copy_properties(gc)
         line_gc.set_hatch(None)
@@ -81,17 +94,22 @@ class CenteredHatch(AbstractPathEffect):
             spacing = renderer.points_to_pixels(12 / hatch.count("."))
             x_extent = ceil(bounds.width / (2 * spacing))
             y_extent = ceil(bounds.height / (2 * spacing))
-            centers = [(center_x + x * spacing, center_y + y * spacing)
-                       for x in range(-x_extent, x_extent + 1)
-                       for y in range(-y_extent, y_extent + 1)]
+            # Trim geometry too: SVG importers can discard clip paths.
+            dots = [PlotPath.unit_circle().transformed(
+                        Affine2D().scale(renderer.points_to_pixels(0.9)).translate(
+                            center_x + x * spacing, center_y + y * spacing)).clip_to_bbox(clip_bounds)
+                    for x in range(-x_extent, x_extent + 1)
+                    for y in range(-y_extent, y_extent + 1)]
             line_gc.set_linewidth(0)
-            renderer.draw_markers(line_gc, PlotPath.unit_circle(),
-                                  Affine2D().scale(renderer.points_to_pixels(0.9)),
-                                  PlotPath(centers), IdentityTransform(), gc.get_hatch_color())
+            renderer.draw_path(line_gc, PlotPath.make_compound_path(*dots),
+                               IdentityTransform(), gc.get_hatch_color())
         line_gc.restore()
 
 
-def configure_style() -> None:
+def configure_style(*, colors: bool = True, anonymize: bool = False) -> None:
+    global _USE_COLORS
+    _USE_COLORS = colors
+    COMPILER_NAMES["raptor"] = "Anonimized" if anonymize else "Raptor"
     plt.rcParams.update({
         "font.family": "DejaVu Sans",
         "axes.spines.top": False,
@@ -109,6 +127,14 @@ def configure_style() -> None:
     })
 
 
+class TextLegendHandler(HandlerBase):
+    def create_artists(self, legend, original, xdescent, ydescent, width, height, fontsize, transform):
+        return [Text(width / 2 - xdescent, height / 2 - ydescent, original.get_text(),
+                     color=original.get_color(), fontproperties=original.get_fontproperties(),
+                     math_fontfamily=original.get_math_fontfamily(), ha="center", va="center",
+                     transform=transform)]
+
+
 def add_legend(fig, handles) -> None:
     """Use the fewest balanced legend rows that fit the figure width."""
     renderer = fig.canvas.get_renderer()
@@ -123,6 +149,7 @@ def add_legend(fig, handles) -> None:
         ordered = [handle for column in zip(*grid) for handle in column]
         legend = fig.legend(handles=[handle if handle is not None else Patch(visible=False) for handle in ordered],
                             labels=[handle.get_label() if handle is not None else "" for handle in ordered],
+                            handler_map={Text: TextLegendHandler()},
                             loc="outside upper center", ncols=columns, frameon=False, fontsize=9.5,
                             handlelength=1.7, columnspacing=1.5, handletextpad=0.6, labelspacing=0.4)
         for handle, patch, text in zip(ordered, legend.legend_handles, legend.get_texts()):
@@ -197,7 +224,7 @@ def fit_bar_widths(fig) -> None:
         fig.set_size_inches(fig.get_figwidth() * scale, fig.get_figheight())
 
 
-def fit_value_labels(fig) -> None:
+def fit_value_labels(fig, *, lower_limit: float | None = None) -> None:
     """Fit common vertical limits to the data and rendered value labels."""
     for ax in fig.axes:
         for text in ax.texts:
@@ -215,18 +242,39 @@ def fit_value_labels(fig) -> None:
             labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height,
                            max(0, extent.y1 - anchor + 3) / bounds.height))
         limits.append(ax.dataLim.intervaly if ax.has_data() else ax.get_ylim())
-    lower = min(0, *(lower for lower, _ in limits))
+    lower = min(0, *(lower for lower, _ in limits)) if lower_limit is None else lower_limit
     upper = max(0, *(upper for _, upper in limits))
     if lower == upper:
         upper = lower + 1
     for _ in range(12):
-        lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
+        if lower_limit is None:
+            lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
         upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)])
     for ax in fig.axes:
         ax.set_ylim(lower, upper)
 
 
+def grayscale_figure(fig) -> None:
+    def gray(color):
+        red, green, blue, alpha = to_rgba(color)
+        level = 0.2126 * red + 0.7152 * green + 0.0722 * blue
+        return level, level, level, alpha
+
+    for artist in fig.findobj():
+        if isinstance(artist, Patch):
+            artist.set_facecolor(gray(artist.get_facecolor()))
+            artist.set_edgecolor(gray(artist.get_edgecolor()))
+        elif isinstance(artist, Line2D):
+            artist.set_color(gray(artist.get_color()))
+            artist.set_markerfacecolor(gray(artist.get_markerfacecolor()))
+            artist.set_markeredgecolor(gray(artist.get_markeredgecolor()))
+        elif isinstance(artist, Text):
+            artist.set_color(gray(artist.get_color()))
+
+
 def save_figure(fig, basename: str, output_dir: Path) -> None:
+    if not _USE_COLORS:
+        grayscale_figure(fig)
     for patch in fig.findobj(Patch):
         hatch = patch.get_hatch() or ""
         if any(symbol in hatch for symbol in "./\\"):