even better more structured throughput scheduling
Validate Operations / validate-operations (push) Has been cancelled

plot adjustments
validation adjustments
This commit is contained in:
NiccoloN
2026-10-27 17:46:46 +01:00
parent 49d6fb42cf
commit a77edafcbf
46 changed files with 3265 additions and 1043 deletions
+13 -1
View File
@@ -82,9 +82,21 @@ for measurement windows, power, and energy interpretation.
Functional compilation/output comparison determines PASS/FAIL. Performance
failures show `ERROR` without changing a functional PASS; explicit unsupported
operations show `UNSUPPORTED`. Pimsim's `vsoftmax` is unsupported.
operations show `UNSUPPORTED`. Pimsim-nn omits `vsoftmax` timing and energy;
the functional simulator still executes Softmax.
Skipped measurements remain visibly skipped.
Floating-point validation requires both checks for each output:
- Every element satisfies `abs(actual - reference) <= threshold + relative_threshold * abs(reference)`.
- For a nonzero reference, `norm(actual - reference) / norm(reference) <= relative_threshold`, using the L2 norm.
The second check prevents a large absolute tolerance from accepting a lost small output.
It does not replace or weaken the elementwise check. Set `--relative-threshold=0` for absolute-only floating-point comparison.
Zero references use the absolute check. Integer and Boolean outputs require exact agreement.
Nonfinite values and mismatched output shapes or counts fail validation.
Reports show maximum absolute differences and relative L2 errors.
Exit codes: `0` for all models passing or successful cleanup; `1` for validation
failure, an invalid model directory, or no ONNX files; `2` for invalid arguments.
+19 -5
View File
@@ -13,9 +13,9 @@ operations, standalone commands, configuration layout, and GA/OpenMP controls.
|---|---|---|
| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. |
| `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. |
| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12). |
| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12), with only its two LRN nodes bypassed. |
| `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. |
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Derived from `../yolo11n/depth_51/yolo11n_depth_51.onnx`. |
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`; this directory retains the historical Softmax-free variant. |
The paper's checkpoints were not published. The three Model Zoo networks
produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is
@@ -23,12 +23,25 @@ suitable for compiler/simulator comparison, not paper-accuracy reproduction.
The paper's MNIST description for ResNet-18 differs from the released graph's
ImageNet input shape.
Dedicated performance models change the original computation:
The current GoogLeNet comparison uses `googlenet/googlenet-12-no-lrn.onnx`.
Only its two LRN nodes are bypassed because Raptor cannot lower LRN to PIM.
Its terminal Softmax, output `prob_1`, weights, and other operations are retained.
The native reference uses this same graph. VGG8 and both ResNet graphs have no
removed operators; ResNet-34 fixes the symbolic batch size to one.
Historical performance models change the original computation:
- `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal
Softmax, which Pimcomp does not schedule.
- `yolo11n/yolo11n-pimsim-nn.onnx` removes two Softmax nodes unsupported by
Pimsim. Use the original YOLO model for functional validation.
- `yolo11n/yolo11n-pimsim-nn.onnx` removes two internal Softmax nodes. It remains
available for reproducing historical measurements.
The current YOLO comparison uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`,
including both Softmax operations. Its native reference uses the same full model.
The functional simulator executes Softmax. Pimsim-nn skips `vsoftmax` before
fetch, and Raptor's PIM queue estimator omits the same instruction cost.
Timing and energy exclude Softmax work; they do not model hardware Softmax support.
Comparison artifacts remain under this suite's `yolo11n/artifacts/` directory.
Model SHA-256 checksums:
@@ -36,6 +49,7 @@ Model SHA-256 checksums:
788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx
c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx
c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx
c960c195953423388733cd0e60712cf425ca1f8917e264f5cf98e028752c61cf googlenet/googlenet-12-no-lrn.onnx
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx
396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
@@ -0,0 +1,44 @@
from dataclasses import dataclass
import math
import numpy as np
VALIDATION_POLICY = "pointwise-and-relative-l2-v1"
@dataclass(frozen=True)
class OutputComparison:
passed: bool
max_abs_diff: float
relative_l2_error: float | None
pointwise_passed: bool
def compare_output_arrays(actual, expected, *, atol, rtol):
"""Require pointwise tolerance and relative L2 tolerance for nonzero outputs."""
if any(not math.isfinite(value) or value < 0 for value in (atol, rtol)):
raise ValueError("output tolerances must be finite and non-negative")
actual, expected = np.asarray(actual), np.asarray(expected)
if actual.dtype.kind not in "biuf" or expected.dtype.kind not in "biuf":
raise ValueError("output comparison requires real numeric arrays")
if actual.shape != expected.shape:
raise ValueError(f"output shapes differ: {actual.shape} != {expected.shape}")
if not actual.size:
return OutputComparison(True, 0.0, None, True)
if not np.all(np.isfinite(actual)) or not np.all(np.isfinite(expected)):
return OutputComparison(False, math.inf, None, False)
if expected.dtype.kind in "biu":
difference = np.abs(actual.astype(object) - expected.astype(object))
passed = bool(np.array_equal(actual, expected))
return OutputComparison(passed, float(difference.max()), None, passed)
actual, expected = actual.astype(np.float64), expected.astype(np.float64)
with np.errstate(over="ignore", invalid="ignore", divide="ignore"):
difference = np.abs(actual - expected)
pointwise = bool(np.all(difference <= atol + rtol * np.abs(expected)))
scale = float(np.max(np.abs(expected)))
relative = (float(np.linalg.norm(difference / scale) /
np.linalg.norm(expected / scale)) if scale else None)
# A zero reference has no relative scale. rtol=0 explicitly requests absolute-only checks.
relative_passed = relative is None or rtol == 0 or relative <= rtol
return OutputComparison(pointwise and relative_passed, float(difference.max()), relative, pointwise)
@@ -10,11 +10,7 @@ MODELS = {
"vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
"resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
"resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
"googlenet": SUITE / "googlenet/googlenet-12-pimsim-nn.onnx",
"yolo11n": SUITE / "yolo11n/yolo11n-pimsim-nn.onnx",
}
FUNCTIONAL_MODELS = {
**MODELS,
"googlenet": SUITE / "googlenet/googlenet-12-no-lrn.onnx",
"yolo11n": REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx",
}
MODEL_NAMES = tuple(MODELS)
@@ -27,6 +27,12 @@ def read_raptor_instruction_count(path):
def parse_pimsim_nn_metrics(output):
metrics = {"raw_output": output}
if "using legacy full-duration throughput result" in output:
metrics["fast_throughput_converged"] = False
metrics["throughput_measurement"] = "full_duration_fallback"
elif "Fast throughput converged; stopping early" in output:
metrics["fast_throughput_converged"] = True
metrics["throughput_measurement"] = "converged_windows"
for name, pattern in _METRIC_PATTERNS.items():
match = re.search(pattern, output)
if match:
+19 -18
View File
@@ -24,6 +24,7 @@ from .onnx_utils import (
from .raptor import compile_with_raptor
from .pimsim_nn import export_raptor_pimsim_artifact, parse_pimsim_nn_metrics, read_raptor_instruction_count
from .subprocess_utils import run_command_with_reporter
from .output_comparison import compare_output_arrays
STAGE_TITLES = (
"Compile ONNX",
@@ -465,34 +466,35 @@ def parse_pim_simulator_outputs(output_bin_path, outputs_descriptor):
def validate_outputs(sim_arrays, runner_out_dir, outputs_descriptor, threshold, rtol, verbose):
if len(sim_arrays) != len(outputs_descriptor):
raise ValueError("simulator output count differs from the model output count")
all_passed = True
rows = []
for sim_array, (oi, name, _, shape) in zip(sim_arrays, outputs_descriptor):
for sim_array, (oi, name, elem_type, shape) in zip(sim_arrays, outputs_descriptor):
csv_name = f"output{oi}_{sanitize_output_name(name)}.csv"
runner_array = np.loadtxt(runner_out_dir / csv_name, delimiter=',', dtype=np.float32).reshape(shape)
sim_array64 = sim_array.astype(np.float64)
runner_array64 = runner_array.astype(np.float64)
abs_diff = np.abs(sim_array64 - runner_array64)
allowed_diff = threshold + rtol * np.abs(runner_array64)
max_diff = float(np.max(abs_diff))
passed = bool(np.all(abs_diff <= allowed_diff))
rows.append((name, f"{max_diff:.6e}", passed))
runner_array = np.loadtxt(runner_out_dir / csv_name, delimiter=',', dtype=_ONNX_TO_NP[elem_type]).reshape(shape)
comparison = compare_output_arrays(sim_array, runner_array, atol=threshold, rtol=rtol)
passed = comparison.passed
relative = comparison.relative_l2_error
rows.append((name, f"{comparison.max_abs_diff:.6e}",
f"{relative:.6e}" if relative is not None else "N/A", passed))
if not passed:
all_passed = False
name_width = max(len("Output"), *(len(name) for name, _, _ in rows))
diff_width = max(len("Max diff"), *(len(diff) for _, diff, _ in rows))
name_width = max(len("Output"), *(len(name) for name, _, _, _ in rows))
diff_width = max(len("Max diff"), *(len(diff) for _, diff, _, _ in rows))
relative_width = max(len("Relative L2"), *(len(relative) for _, _, relative, _ in rows))
result_width = len("Result")
separator = f" +-{'-' * name_width}-+-{'-' * diff_width}-+-{'-' * result_width}-+"
separator = f" +-{'-' * name_width}-+-{'-' * diff_width}-+-{'-' * relative_width}-+-{'-' * result_width}-+"
if verbose or not all_passed:
print(separator)
print(f" | {'Output'.ljust(name_width)} | {'Max diff'.ljust(diff_width)} | {'Result'} |")
print(f" | {'Output'.ljust(name_width)} | {'Max diff'.ljust(diff_width)} | {'Relative L2'.ljust(relative_width)} | {'Result'} |")
print(separator)
for name, diff_text, passed in rows:
for name, diff_text, relative, passed in rows:
status_text = ("PASS" if passed else "FAIL").ljust(result_width)
status = Fore.GREEN + status_text + Style.RESET_ALL if passed else Fore.RED + status_text + Style.RESET_ALL
print(f" | {name.ljust(name_width)} | {diff_text.ljust(diff_width)} | {status} |")
print(f" | {name.ljust(name_width)} | {diff_text.ljust(diff_width)} | {relative.ljust(relative_width)} | {status} |")
print(separator)
return all_passed
@@ -577,10 +579,9 @@ def validate_execution(
f"Latency: {state['metrics']['latency_ms']:.2f} ms"
if name == "latency" else
f"Throughput: {state['metrics']['throughput']:.2f} samples/s")
energy_unit = "pJ" if name == "latency" else "pJ/it"
print_info(
reporter, f"{metric}, Power: {state['metrics']['average_power_mw']:.2f} mW, "
f"Energy: {state['metrics']['average_energy_pj']:.2f} {energy_unit}")
reporter, f"{metric}, Average power: {state['metrics']['average_power_mw']:.2f} mW, "
f"Energy per inference: {state['metrics']['average_energy_pj']:.2f} pJ/inference")
except PimSimUnsupportedError as exc:
state["pimsim_status"] = PIMSIM_UNSUPPORTED
print_info(reporter, str(exc))
@@ -0,0 +1,75 @@
from pathlib import Path
import sys
import numpy as np
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from raptor_validation.output_comparison import compare_output_arrays
def test_absolute_floor_does_not_accept_lost_small_output():
result = compare_output_arrays(np.zeros(2), np.array([1e-4, -2e-4]), atol=1e-3, rtol=1e-4)
assert result.pointwise_passed
assert result.relative_l2_error == pytest.approx(1)
assert not result.passed
@pytest.mark.parametrize("scale", [1e-200, 1e-4, 1, 1e200])
def test_small_relative_roundoff_is_independent_of_scale(scale):
expected = np.array([1, -2, 3], dtype=np.float64) * scale
result = compare_output_arrays(expected * (1 + 1e-6), expected, atol=1e-3, rtol=1e-4)
assert result.passed
assert result.relative_l2_error == pytest.approx(1e-6)
def test_norm_agreement_cannot_hide_an_incorrect_small_element():
result = compare_output_arrays([1e6, .01], [1e6, 0.0], atol=1e-3, rtol=1e-4)
assert result.relative_l2_error < 1e-4
assert not result.pointwise_passed
assert not result.passed
@pytest.mark.parametrize("actual,expected", [([np.inf], [1.0]), ([1.0], [np.inf]),
([np.inf], [np.inf]), ([np.nan], [np.nan])])
def test_nonfinite_outputs_never_pass(actual, expected):
assert not compare_output_arrays(actual, expected, atol=1e-3, rtol=1e-4).passed
@pytest.mark.parametrize("actual,expected", [([1.0], [[1.0]]), ([1.0], [])])
def test_broadcasting_cannot_hide_output_shape_errors(actual, expected):
with pytest.raises(ValueError, match="shapes differ"):
compare_output_arrays(actual, expected, atol=1e-3, rtol=1e-4)
def test_zero_reference_retains_absolute_check():
result = compare_output_arrays([1e-6], [0.0], atol=1e-3, rtol=1e-4)
assert result.passed and result.relative_l2_error is None
assert not compare_output_arrays([.01], [0.0], atol=1e-3, rtol=1e-4).passed
def test_zero_relative_tolerance_requests_absolute_only_check():
result = compare_output_arrays([0.0], [1e-4], atol=1e-3, rtol=0)
assert result.passed
def test_integer_outputs_require_exact_agreement_without_float_rounding():
expected = np.array([2**53], np.int64)
result = compare_output_arrays(expected + 1, expected, atol=1e-3, rtol=1e-4)
assert not result.passed and result.max_abs_diff == 1
def test_empty_outputs_match():
assert compare_output_arrays([], [], atol=1e-3, rtol=1e-4).passed
@pytest.mark.parametrize("atol,rtol", [(-1, 0), (0, -1), (np.inf, 0), (0, np.nan)])
def test_invalid_tolerances_are_rejected(atol, rtol):
with pytest.raises(ValueError, match="tolerances"):
compare_output_arrays([0.0], [0.0], atol=atol, rtol=rtol)
def test_complex_outputs_are_not_silently_cast_to_real():
with pytest.raises(ValueError, match="real numeric"):
compare_output_arrays([1 + 2j], [1 + 0j], atol=1e-3, rtol=1e-4)
+395 -9
View File
@@ -8,6 +8,8 @@ import os
from pathlib import Path
import sys
import numpy as np
import onnx
import pytest
@@ -18,6 +20,93 @@ sys.path.insert(0, str(TOOLS / "pimcomp/compare"))
import run_ablation as ablation
import run_pimcomp_models as suite
import compare_raptor_pimcomp_model as comparator
from raptor_validation.validate_one import validate_outputs
@pytest.mark.parametrize("mode", ["latency", "throughput"])
def test_comparison_report_energy_is_per_inference(tmp_path, mode):
report = tmp_path / "comparison_report.md"
validation = comparator.CompareResult(True, {"output": 1e-7}, relative_l2_errors={"output": 1e-6})
metrics = {"average_power_mw": 2, "average_energy_pj": 30,
"latency_ms": 4, "average_latency_ms": 4,
"throughput": 250, "output_count": 6}
comparator.write_report(
report, model_path=tmp_path / "model.onnx", hardware={}, steps=[], failures=[],
raptor_validation=validation, pimcomp_validation=validation,
raptor_perf=metrics, pimcomp_perf=metrics, raptor_instr={}, pimcomp_instr={},
raptor_pass_timings={}, pimsim_mode=mode, pimcomp_pipeline="batch",
pimcomp_replication="weight")
text = report.read_text()
assert "Average power (mW) | Energy per inference (pJ/inference)" in text
assert "Avg latency" not in text
assert "Avg energy" not in text
assert metrics["average_energy_pj"] == 30
assert "Raptor relative L2 | Pimcomp relative L2" in text
assert "1.000000e-06 | 1.000000e-06" in text
@pytest.mark.parametrize("zero_output", [False, True])
def test_comparison_frontends_reject_missing_small_outputs(tmp_path, zero_output):
expected = np.array([[1e-4, -2e-4]], dtype=np.float32)
actual = np.zeros_like(expected) if zero_output else expected * np.float32(1 + 1e-6)
np.savetxt(tmp_path / "output0_output.csv", expected.reshape(-1), delimiter=",")
actual.tofile(tmp_path / "output.bin")
outputs = [(0, "output", 1, [1, 2])]
result = comparator.compare_simulator_outputs(
tmp_path / "output.bin", outputs, tmp_path, threshold=1e-3, rtol=1e-4)
assert result.passed == (not zero_output)
assert result.relative_l2_errors["output"] == pytest.approx(1 if zero_output else 1e-6, rel=.1)
assert validate_outputs([actual], tmp_path, outputs, 1e-3, 1e-4, False) == (not zero_output)
def test_comparison_preserves_double_reference_precision(tmp_path):
expected = np.array([1 + 2**-40], dtype=np.float64)
np.savetxt(tmp_path / "output0_output.csv", expected, delimiter=",")
expected.tofile(tmp_path / "output.bin")
outputs = [(0, "output", 11, [1])]
assert comparator.compare_simulator_outputs(
tmp_path / "output.bin", outputs, tmp_path, threshold=0, rtol=0).passed
assert validate_outputs([expected], tmp_path, outputs, 0, 0, False)
def test_relative_agreement_only_reports_possible_numerical_sensitivity(tmp_path):
expected = np.array([1e6, 0], dtype=np.float32)
actual = np.array([1e6, .01], dtype=np.float32)
np.savetxt(tmp_path / "output0_output.csv", expected, delimiter=",")
actual.tofile(tmp_path / "output.bin")
result = comparator.compare_simulator_outputs(
tmp_path / "output.bin", [(0, "output", 1, [2])], tmp_path, threshold=1e-3, rtol=1e-4)
assert not result.passed
assert result.relative_l2_errors["output"] < 1e-4
assert "possible numerical sensitivity, not proof of correctness" in result.error
def test_comparison_rejects_missing_output_arrays(tmp_path, monkeypatch):
outputs = [(0, "output", 1, [1])]
monkeypatch.setattr(comparator, "parse_pim_simulator_outputs", lambda *args: [])
with pytest.raises(ValueError, match="output count"):
comparator.compare_simulator_outputs(tmp_path / "missing.bin", outputs, tmp_path, threshold=1e-3, rtol=1e-4)
with pytest.raises(ValueError, match="output count"):
validate_outputs([], tmp_path, outputs, 1e-3, 1e-4, False)
@pytest.mark.parametrize("raptor_pass,pimcomp_pass,winner", [
(True, True, "pimcomp"), (True, False, "NA"),
(False, True, "NA"), (False, False, "NA"),
])
def test_timing_winner_requires_both_functional_passes(tmp_path, raptor_pass, pimcomp_pass, winner):
report = suite.result_dir(tmp_path, "vgg8", "arch-a", "latency", 1) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
data = {"validation_policy": suite.VALIDATION_POLICY}
for side, passed, latency in [("raptor", raptor_pass, 10), ("pimcomp", pimcomp_pass, 1)]:
data[side + "_validation"] = {"status": "done", "passed": passed}
data[side + "_performance"] = {"latency_ms": latency, "output_count": 1}
report.write_text(json.dumps(data))
output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("latency", 1, "element"),))
with output.open() as stream:
row = next(csv.DictReader(stream))
assert row["better_compiler"] == winner
assert row["speedup"] == "10.00" # Diagnostic timing remains available.
def test_ablation_results_stay_in_independent_output_roots(tmp_path):
@@ -30,7 +119,7 @@ def test_ablation_results_stay_in_independent_output_roots(tmp_path):
path.parent.mkdir(parents=True)
row = {"arch": "arch-a", "model": "vgg8", "mode": "latency",
"raptor_pipeline": "1", "raptor_functional_validation": "PASS"}
row.update({field: 100 if variant.name == "baseline" else value
row.update({field: value if variant.name == "baseline" else 100
for field, _ in ablation.PERCENTAGE_FIELDS})
with path.open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(row))
@@ -40,11 +129,124 @@ def test_ablation_results_stay_in_independent_output_roots(tmp_path):
assert failures == []
assert output == root / "results_ablation.csv"
outputs.append(output)
for output, change in zip(outputs, ("-20.00%", "-40.00%")):
for output, change in zip(outputs, ("+20.00%", "+40.00%")):
with output.open(newline="") as stream:
rows = list(csv.DictReader(stream))
assert rows[0]["latency"] == "+0.00%"
assert rows[1]["latency"] == change
assert len(rows) == 1
assert rows[0]["comparison"] == "plan"
assert rows[0]["baseline_variant"] == "no-plan"
assert rows[0]["latency_gain_percent"] == change
@pytest.mark.parametrize("options,expected", [
([], {("baseline", "latency", "1"), ("no-plan", "latency", "1"),
("baseline", "throughput", "4"), ("no-plan", "throughput", "4"),
("no-sync", "throughput", "4")}),
(["--mode", "latency"], {("baseline", "latency", "1"), ("no-plan", "latency", "1")}),
(["--pipeline=8"], {(variant, "throughput", "8") for variant in ("baseline", "no-plan", "no-sync")}),
(["--mode", "latency", "throughput", "--pipeline", "2"],
{(variant, "throughput", "2") for variant in ("baseline", "no-plan", "no-sync")}),
])
def test_ablation_runs_only_required_variants(options, expected, capsys):
import shlex
assert ablation.main(["--dry-run", *options]) == 0
commands = [shlex.split(line) for line in capsys.readouterr().out.splitlines()]
actual = set()
for command in commands:
variant = command[command.index("--ablation-variant") + 1]
actual.add((variant, *ablation.case_key(command)))
assert "--raptor-only" in command
assert ("--raptor-extra-arg=--pim-disable-synchronization" in command) == (variant == "no-sync")
assert ("--raptor-extra-arg=--pim-disable-spatial-planning" in command) == (variant == "no-plan")
assert actual == expected
assert len(commands) == len(expected)
def test_ablation_rejects_combined_disabled_variant():
with pytest.raises(SystemExit):
ablation.parse_args(["--variant", "no-sync-no-plan"])
def test_ablation_reports_gains_against_each_disabled_feature(tmp_path):
variants = ablation.selected_variants(None)
for variant in variants:
path = ablation.variant_output_dir(tmp_path, variant) / "results_comparison.csv"
path.parent.mkdir(parents=True)
rows = []
for mode, pipeline in (("latency", "1"), ("throughput", "4")):
if variant.name == "no-sync" and mode == "latency":
continue
rows.append({"arch": "arch-b", "model": "vgg8", "mode": mode,
"raptor_pipeline": pipeline,
"raptor_functional_validation": "FAIL" if variant.name == "no-sync" else "PASS",
"raptor_latency_ms": {"baseline": "8", "no-plan": "10", "no-sync": "6.4"}[variant.name],
"raptor_throughput_samples_s": {"baseline": "125", "no-plan": "100", "no-sync": "156.25"}[variant.name],
"raptor_power_mw": "100", "raptor_energy_pj": "100"})
with path.open("w", newline="") as stream:
writer = csv.DictWriter(stream, fieldnames=list(rows[0]))
writer.writeheader()
writer.writerows(rows)
output, failures = ablation.aggregate_results(tmp_path, variants)
assert not failures
with output.open(newline="") as stream:
rows = list(csv.DictReader(stream))
assert len(rows) == 3
latency, plan, sync = rows
assert (latency["comparison"], latency["latency_gain_percent"]) == ("plan", "+20.00%")
assert (plan["comparison"], plan["throughput_gain_percent"]) == ("plan", "+25.00%")
assert (sync["comparison"], sync["throughput_gain_percent"]) == ("sync", "-20.00%")
assert plan["baseline_variant"] == "no-plan"
assert sync["baseline_variant"] == "no-sync"
assert sync["baseline_functional_validation"] == "FAIL"
assert sync["functional_validation"] == "PASS"
@pytest.mark.parametrize("variant", [None, "baseline", "no-plan", "no-sync"])
def test_unconverged_fast_throughput_is_available_in_comparison_and_ablation(tmp_path, variant):
raw = """Fast throughput convergence failed before sim_time; using legacy full-duration throughput result
Chip Simulation Result:
- output count: 22 samples
- throughput: 22 samples/s
- average latency: 45.45454545 ms
- average power: 300 mW
- average energy: 10000000000 pJ/it
"""
metrics = comparator.parse_pimsim_nn_metrics(raw)
assert metrics["fast_throughput_converged"] is False
assert metrics["throughput_measurement"] == "full_duration_fallback"
assert "unconverged" in comparator.perf_status(metrics)
report = suite.result_dir(tmp_path, "vgg8", "arch-b", "throughput", 4, variant) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
data = {"validation_policy": suite.VALIDATION_POLICY}
for side in ("raptor", "pimcomp"):
data[side + "_validation"] = {"status": "done", "passed": True}
# Older reports can contain only the raw simulator output.
data[side + "_performance"] = {"raw_output": raw, "fast_throughput_converged": False}
report.write_text(json.dumps(data))
output = suite.write_results_csv(tmp_path, "arch-b", ["vgg8"],
(("throughput", 4, "batch"),), ablation_variant=variant)
with output.open(newline="") as stream:
row = next(csv.DictReader(stream))
for side in ("raptor", "pimcomp"):
assert float(row[side + "_throughput_samples_s"]) == 22
assert float(row[side + "_latency_ms"]) == pytest.approx(1000 / 22)
if variant is not None:
enabled = ablation.variant_output_dir(tmp_path, ablation.VARIANT_BY_NAME["baseline"]) / "results_comparison.csv"
enabled.parent.mkdir(parents=True, exist_ok=True)
enabled.write_text(output.read_text())
aggregate, failures = ablation.aggregate_results(tmp_path, ablation.selected_variants([variant]))
assert not failures
with aggregate.open(newline="") as stream:
for gain in csv.DictReader(stream):
assert gain["throughput_gain_percent"] == "+0.00%"
def test_fast_throughput_convergence_metadata():
metrics = comparator.parse_pimsim_nn_metrics("Fast throughput converged; stopping early\nthroughput: 100 samples/s")
assert metrics["fast_throughput_converged"] is True
assert metrics["throughput_measurement"] == "converged_windows"
assert metrics["throughput"] == 100
@pytest.fixture
@@ -56,7 +258,9 @@ def recorded_failure(tmp_path):
pimcomp_config=config, pimsim_mode="throughput", pimsim_time_ms=1000,
pimcomp_pipeline="batch", pimcomp_replication="GA", pimcomp_model_name=None,
seed=0, batch_size=64, threshold=1e-3, rtol=1e-4, no_fast=False,
skip_pimsim_nn=False,
skip_pimsim_nn=False, pimcomp_dir=tmp_path / "pimcomp",
pimsim_nn_build_dir=tmp_path / "oracle", pim_simulator_dir=tmp_path / "functional",
raptor_path=tmp_path / "raptor", raptor_extra_arg=["--pipeline=2"],
)
hardware = {"core_count": 168, "crossbar_size": 128}
context = comparator.pimcomp_reuse_context(args, model, functional, hardware)
@@ -66,7 +270,7 @@ def recorded_failure(tmp_path):
"error": "Unsupported model"},
"pimcomp_performance": {"skipped": True, "reason": "Compilation failed"},
"pimcomp_instruction_summary": comparator.empty_instruction_summary("No artifact"),
"paths": {}, "steps": [],
"paths": {}, "steps": [], "artifact_signatures": {"pimcomp": {}},
"failures": [{"stage": "Compile Pimcomp frontend", "error": "Unsupported model"}],
}
path = tmp_path / "comparison_report.json"
@@ -84,6 +288,117 @@ def test_failed_pimcomp_report_needs_no_simulation_artifacts(recorded_failure):
assert failures == report["failures"]
@pytest.mark.parametrize("tool", ["compiler", "oracle", "exporter"])
def test_reuse_checks_tool_content(recorded_failure, tool):
path, report, args, model, functional, hardware = recorded_failure
original = report["pimcomp_reuse_context"]["files"][tool]["path"]
tool_path = Path(original)
tool_path.parent.mkdir(parents=True, exist_ok=True)
tool_path.write_bytes(b"changed executable or exporter")
expected = comparator.pimcomp_reuse_context(args, model, functional, hardware)
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
comparator.load_reused_pimcomp_report(path, expected)
@pytest.mark.parametrize("mutation", ["instructions", "weights", "missing"])
def test_reuse_checks_artifact_content(recorded_failure, mutation):
path, report, *_ = recorded_failure
artifact = path.parent / "artifact"
artifact.mkdir()
(artifact / "core_0.json").write_text('[{"op":"nop"}]')
(artifact / "weights.bin").write_bytes(b"weights")
report["paths"]["pimcomp_exported_pim"] = str(artifact)
report["artifact_signatures"]["pimcomp"] = comparator.artifact_signatures(report, "pimcomp")
path.write_text(json.dumps(report))
assert comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"])
if mutation == "missing":
(artifact / "core_0.json").unlink()
elif mutation == "weights":
(artifact / "weights.bin").write_bytes(b"wrong weights")
else:
(artifact / "core_0.json").write_text('[{"op":"wait"}]')
with pytest.raises(ValueError, match="artifacts differ"):
comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"])
def test_raptor_context_covers_pipeline_and_ignores_pimcomp_pipeline(recorded_failure):
_, _, args, model, functional, hardware = recorded_failure
expected = comparator.experiment_context(args, model, functional, hardware, "raptor")
args.pimcomp_pipeline = "element"
assert expected == comparator.experiment_context(args, model, functional, hardware, "raptor")
args.raptor_extra_arg = ["--pipeline=8"]
assert expected != comparator.experiment_context(args, model, functional, hardware, "raptor")
def test_reuse_rejects_previous_validation_policy(recorded_failure):
path, report, *_ = recorded_failure
context = copy.deepcopy(report["pimcomp_reuse_context"])
del report["pimcomp_reuse_context"]["validation_policy"]
path.write_text(json.dumps(report))
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
comparator.load_reused_pimcomp_report(path, context)
def test_raptor_reuse_rejects_previous_validation_policy(recorded_failure, monkeypatch):
path, _, _, model, _, _ = recorded_failure
path.write_text('{"validation_policy": "legacy"}')
monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {})
monkeypatch.setattr(comparator, "load_model_inputs", lambda *args: ([], [], []))
monkeypatch.setattr(sys, "argv", [
"compare_raptor_pimcomp_model.py", "--model", str(model),
"--out-dir", str(path.parent / "out"), "--pimcomp-config", str(path.parent / "config.json"),
"--reuse-raptor-report", str(path),
])
with pytest.raises(ValueError, match="validation policy differs"):
comparator.main()
@pytest.mark.parametrize("tolerances", [None, {"atol": 0.01, "rtol": 0.0001},
{"atol": 0.001, "rtol": 0.01}])
def test_raptor_reuse_checks_actual_tolerances(recorded_failure, monkeypatch, tolerances):
path, _, _, model, _, _ = recorded_failure
path.write_text(json.dumps({"validation_policy": comparator.VALIDATION_POLICY,
"validation_tolerances": tolerances}))
monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {})
monkeypatch.setattr(comparator, "load_model_inputs", lambda *args: ([], [], []))
monkeypatch.setattr(sys, "argv", [
"compare_raptor_pimcomp_model.py", "--model", str(model),
"--out-dir", str(path.parent / "out"), "--pimcomp-config", str(path.parent / "config.json"),
"--reuse-raptor-report", str(path),
])
with pytest.raises(ValueError, match="validation tolerances differ"):
comparator.main()
def test_failed_raptor_compilation_is_not_reported_as_skipped(tmp_path, monkeypatch):
model = tmp_path / "model.onnx"
model.write_bytes(b"test model")
runner = tmp_path / "runner"
runner.touch()
monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {"core_count": 1})
monkeypatch.setattr(comparator, "load_model_inputs", lambda *args:
([(0, "input", 1, [1])], [(0, "output", 1, [1])], [np.zeros(1, np.float32)]))
monkeypatch.setattr(comparator, "ensure_reference_runner", lambda *args: runner)
monkeypatch.setattr(comparator, "generate_reference_outputs", lambda *args, **kwargs: tmp_path)
def fail_compile(*args):
raise RuntimeError("unsupported test operation")
monkeypatch.setattr(comparator, "compile_raptor_target", fail_compile)
out = tmp_path / "out"
monkeypatch.setattr(sys, "argv", [
"compare_raptor_pimcomp_model.py", "--model", str(model), "--out-dir", str(out),
"--raptor-only", "--skip-pimsim-nn", "--fail-on-error",
])
with pytest.raises(SystemExit) as exit_code:
comparator.main()
assert exit_code.value.code == 1
report = json.loads((out / "pimcomp/comparison_report.json").read_text())
assert report["raptor_validation"]["status"] == "failed"
assert not report["raptor_validation"]["passed"]
assert "unsupported test operation" in report["failures"][0]["error"]
@pytest.mark.parametrize("setting,value", [
("pimsim_mode", "latency"), ("pimcomp_pipeline", "element"),
("pimsim_time_ms", 2000), ("pimcomp_replication", "balance"),
@@ -177,6 +492,65 @@ def test_missing_pimcomp_report_is_not_reused(tmp_path):
assert not suite.pimcomp_report_ready(tmp_path / "output", None)
@pytest.mark.parametrize("model", ["yolo11n", "googlenet"])
def test_comparison_uses_same_model_for_reference_and_execution(tmp_path, monkeypatch, model):
commands = []
monkeypatch.setattr(suite, "run", lambda command, **kwargs: commands.append(command) or 0)
monkeypatch.setattr(sys, "argv", [
"run_pimcomp_models.py", "--out-dir", str(tmp_path), "--models", model,
"--archs", "arch-b", "--mode", "throughput", "--pipeline", "8",
"--raptor-only", "--dry-run",
])
assert suite.main() == 0
reference, comparison = commands
expected = str(suite.MODELS[model])
assert reference[reference.index("--model") + 1] == expected
assert comparison[comparison.index("--model") + 1] == expected
assert comparison[comparison.index("--functional-model") + 1] == expected
assert Path(comparison[comparison.index("--common-dir") + 1]).name == "common-full-model"
def test_yolo_uses_full_model_without_moving_comparison_artifacts():
assert suite.MODELS["yolo11n"] == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx"
assert suite.model_dir(None, "yolo11n") == suite.SUITE / "yolo11n"
assert suite.common_dir(None, "yolo11n") == suite.SUITE / "yolo11n/artifacts/common-full-model"
def test_googlenet_removes_only_lrn():
original = onnx.load(suite.SUITE / "googlenet/googlenet-12.onnx")
actual = onnx.load(suite.MODELS["googlenet"])
removed = {node.output[0]: node.input[0] for node in original.graph.node if node.op_type == "LRN"}
assert len(removed) == 2
retained = []
for node in original.graph.node:
if node.op_type == "LRN":
continue
for i, value in enumerate(node.input):
while value in removed:
value = removed[value]
node.input[i] = value
retained.append(copy.deepcopy(node))
del original.graph.node[:]
original.graph.node.extend(retained)
assert actual.SerializeToString() == original.SerializeToString()
assert sum(node.op_type == "Softmax" for node in actual.graph.node) == 1
assert [output.name for output in actual.graph.output] == ["prob_1"]
onnx.checker.check_model(actual)
assert suite.common_dir(None, "googlenet").name == "common-full-model"
@pytest.mark.parametrize("model,counts", [
("yolo11n", {"Softmax": 2, "Split": 11, "Resize": 2}),
("resnet18", {"BatchNormalization": 20, "Conv": 20}),
("resnet34", {"BatchNormalization": 36, "Conv": 36}),
("vgg8", {"Conv": 6, "Gemm": 2, "Relu": 7, "MaxPool": 3}),
])
def test_comparison_models_retain_required_operations(model, counts):
graph = onnx.load(suite.MODELS[model]).graph
for operation, expected in counts.items():
assert sum(node.op_type == operation for node in graph.node) == expected
def test_missing_pimcomp_report_is_rejected_at_load(recorded_failure):
path, report, *_ = recorded_failure
with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"):
@@ -341,7 +715,8 @@ def test_zero_outputs_preserve_only_measured_average_power():
def test_functional_pass_survives_unavailable_performance(tmp_path):
report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True},
report.write_text(json.dumps({"validation_policy": suite.VALIDATION_POLICY,
"raptor_validation": {"status": "done", "passed": True},
"raptor_performance": {"error": "timeout", "throughput": 0}}))
output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),))
with output.open(newline="") as stream:
@@ -350,6 +725,15 @@ def test_functional_pass_survives_unavailable_performance(tmp_path):
assert all(row[field] == "NA" for field, _ in ablation.PERCENTAGE_FIELDS)
def test_aggregate_rejects_previous_validation_policy(tmp_path):
report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json"
report.parent.mkdir(parents=True)
report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True}}))
with pytest.raises(ValueError, match="different validation policy"):
suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),))
assert not (tmp_path / "results_comparison.csv").exists()
def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path):
variants = ablation.selected_variants(["no-plan"])
for variant in variants:
@@ -368,5 +752,7 @@ def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path):
assert not failures and output == destination
with output.open(newline="") as stream:
rows = list(csv.DictReader(stream))
assert rows[1]["functional_validation"] == "PASS"
assert all(rows[1][metric] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS)
assert len(rows) == 1
assert rows[0]["functional_validation"] == "PASS"
assert rows[0]["baseline_functional_validation"] == "PASS"
assert all(rows[0][f"{metric}_gain_percent"] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS)
+470 -34
View File
@@ -53,13 +53,167 @@ def test_comparison_plot_cli_input_paths(comparison, tmp_path, monkeypatch, cust
raise InputCaptured
monkeypatch.setattr(comparison, "load_results", capture)
monkeypatch.setattr(comparison, "configure_style", lambda: None)
monkeypatch.setattr(comparison, "configure_style", lambda **kwargs: None)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom else [])])
with pytest.raises(InputCaptured):
comparison.main()
assert seen == [expected]
@pytest.mark.parametrize("custom", [False, True])
def test_comparison_plot_cli_output_paths(comparison, tmp_path, monkeypatch, custom):
expected = tmp_path / "custom" if custom else TOOLS / "pimcomp/compare/plots/default"
seen = []
class OutputCaptured(Exception):
pass
def capture(rows, output):
seen.append(output)
raise OutputCaptured
monkeypatch.setattr(comparison, "draw_validation", capture)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--output-dir", str(expected)] if custom else [])])
with pytest.raises(OutputCaptured):
comparison.main()
assert seen == [expected]
@pytest.mark.parametrize("flags,colors,anonymize", [([], True, False),
(["--colors", "--no-anonymize"], True, False), (["--no-colors"], False, False),
(["--anonymize"], True, True), (["--no-colors", "--anonymize"], False, True),
(["--no-colors", "--colors", "--anonymize", "--no-anonymize"], True, False)])
def test_plot_display_flags(comparison, monkeypatch, flags, colors, anonymize):
captured = []
class StyleCaptured(Exception):
pass
def capture(**options):
captured.append(options)
raise StyleCaptured
monkeypatch.setattr(comparison, "configure_style", capture)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *flags])
with pytest.raises(StyleCaptured):
comparison.main()
assert captured == [{"colors": colors, "anonymize": anonymize}]
@pytest.mark.parametrize("flags,borders", [([], True), (["--marker-borders"], True),
(["--no-marker-borders"], False), (["--no-marker-borders", "--marker-borders"], True)])
def test_marker_border_cli_flags(comparison, monkeypatch, flags, borders):
captured = []
def capture(*args, marker_borders):
captured.append(marker_borders)
for name in ("draw_validation", "draw_metric", "draw_absolute"):
monkeypatch.setattr(comparison, name, lambda *args, **kwargs: None)
for name in ("draw_compact_summary", "draw_pipeline_effect"):
monkeypatch.setattr(comparison, name, capture)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", *flags])
comparison.main()
assert captured == [borders, borders]
@pytest.mark.parametrize("borders", [True, False])
def test_marker_borders_reach_data_legends_and_svg(comparison, tmp_path, monkeypatch, borders):
from xml.etree import ElementTree as ET
from matplotlib.lines import Line2D
from matplotlib.colors import to_rgba
comparison.configure_style()
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
save = comparison.save_figure
def capture(fig, name, output):
try:
size = 6.5 if name == "performance_energy_compact" else 7
for ax in fig.axes:
points = [line for line in ax.lines if line.get_marker() in ("o", "s", "^", "D")]
assert points
for point in points:
assert point.get_markersize() == size
assert (point.get_markeredgewidth() > 0) == borders
assert to_rgba(point.get_markerfacecolor())[3] == 1
if not borders:
assert to_rgba(point.get_markerfacecolor()) == to_rgba(point.get_markeredgecolor())
handles = [handle for handle in fig.legends[0].legend_handles if isinstance(handle, Line2D)]
assert handles
for handle in handles:
assert handle.get_markersize() == size
assert (handle.get_markeredgewidth() > 0) == borders
if not borders:
assert to_rgba(handle.get_markerfacecolor()) not in ((1, 1, 1, 1), (0, 0, 0, 0))
assert to_rgba(handle.get_markerfacecolor()) == to_rgba(handle.get_markeredgecolor())
save(fig, name, output)
finally:
comparison.plt.close(fig)
root = ET.parse(output / f"svg/{name}.svg")
paths = [node for node in root.findall('.//{http://www.w3.org/2000/svg}defs/{http://www.w3.org/2000/svg}path')
if node.get("id", "").startswith("m")]
assert paths
assert all(("stroke:" in node.get("style", "")) == borders for node in paths)
monkeypatch.setattr(comparison, "save_figure", capture)
comparison.draw_compact_summary(rows, tmp_path, marker_borders=borders)
comparison.draw_pipeline_effect(rows, tmp_path, marker_borders=borders)
@pytest.mark.parametrize("colors", [True, False])
@pytest.mark.parametrize("anonymize", [True, False])
def test_plot_display_variant_exports(comparison, tmp_path, colors, anonymize):
import re
from PIL import Image, ImageChops
try:
comparison.configure_style(colors=colors, anonymize=anonymize)
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
comparison.draw_compact_summary(rows, tmp_path)
selected = comparison.select_compact_speedups([row for row in rows if row.model != "yolo11n"])
comparison.draw_metric(selected, "speedup", "Speedup (x)", "speedup_compact_comparison",
tmp_path, compact=True)
for name in ("performance_energy_compact", "speedup_compact_comparison"):
svg = (tmp_path / f"svg/{name}.svg").read_text()
if name == "performance_energy_compact":
assert ("Anonimized better" in svg) == anonymize
assert ("Raptor better" in svg) == (not anonymize)
svg_colors = re.findall(r"(?:fill|stroke):\s*#([0-9a-fA-F]{6})", svg)
assert svg_colors
assert any(color[:2] != color[2:4] or color[2:4] != color[4:] for color in svg_colors) == colors
with Image.open(tmp_path / f"png/{name}.png") as picture:
red, green, blue = picture.convert("RGB").split()
chromatic = (ImageChops.difference(red, green).getbbox() is not None or
ImageChops.difference(green, blue).getbbox() is not None)
assert chromatic == colors
assert (tmp_path / f"pdf/{name}.pdf").read_bytes().startswith(b"%PDF-")
finally:
comparison.configure_style()
def test_anonymization_reaches_all_plot_labels(comparison, monkeypatch):
from matplotlib.text import Text
captured = {}
def capture(fig, name, output):
captured[name] = [text.get_text() for text in fig.findobj(Text)]
comparison.plt.close(fig)
monkeypatch.setattr(comparison, "save_figure", capture)
monkeypatch.setattr(sys, "argv", ["generate_plots.py", "--anonymize"])
try:
comparison.main()
assert len(captured) == 30
assert all("Raptor" not in label for labels in captured.values() for label in labels)
assert "Anonimized pipeline stages" in captured["raptor_pipeline_throughput"]
assert "Anonimized better →" in captured["performance_energy_compact"]
assert "Anonimized\n2 stages" in captured["validation_status"]
finally:
comparison.configure_style()
def test_exact_comparison_configuration_matrix(comparison, comparison_csv, tmp_path):
fields, rows = comparison_csv
results = comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows))
@@ -185,7 +339,7 @@ def test_compact_validation_combines_pipeline_statuses(comparison, tmp_path, mon
assert [label.get_text() for label in ax.get_yticklabels()] == ([
"VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet", "YOLO11n"] if index == 0 else [])
assert len(ax.patches) == len(ax.texts) == 4 * len(comparison.MODELS)
assert [label.get_text() for label in ax.get_xticklabels()] == ["Latency", "Throughput"]
assert [label.get_text() for label in ax.get_xticklabels()] == ["Latency-mode", "Throughput-mode"]
assert all(text.get_fontsize() == 12 and text.get_weight() == "normal" for text in ax.texts)
assert all(text.get_color() == "#1D2530" for text in ax.texts)
ax = fig.axes[0]
@@ -211,15 +365,21 @@ def test_compact_speedups_select_best_pipeline(comparison, rates, stage):
@pytest.mark.parametrize("diverging", [False, True])
@pytest.mark.parametrize("metric,ylabel", [("speedup", "Speedup (x)"),
("energy", "Energy-per-inference advantage (x)"), ("power", "Average power advantage (x)")])
("energy", "Energy reduction (x)"), ("power", "Average power reduction (x)")])
def test_compact_metric_bars_and_units(comparison, tmp_path, monkeypatch, diverging, metric, ylabel):
from matplotlib.colors import to_rgba
template = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")[0]
rows = [replace(template, arch=arch, mode=mode, pipeline=stage, latency_speedup=0.5,
energy=0.5 if mode == "latency" else 3, power=0.5 if mode == "latency" else 3,
raptor_throughput=300, pimcomp_throughput=100, pimcomp_validation="FAIL")
for arch in comparison.ARCHITECTURES for mode, stage in (("latency", 1), ("throughput", 4))]
rows = [replace(template, model=model, arch=arch, mode=mode, pipeline=stage,
latency_speedup=0.5 if arch == "arch-a" else 0.25,
energy=(0.5 if arch == "arch-a" else 0.25) if mode == "latency" else
(3 if arch == "arch-a" else 4),
power=(0.5 if arch == "arch-a" else 0.25) if mode == "latency" else
(3 if arch == "arch-a" else 4),
raptor_throughput=300 if arch == "arch-a" else 400,
pimcomp_throughput=100, pimcomp_validation="FAIL")
for arch in reversed(comparison.ARCHITECTURES) for model in ("resnet18", "vgg8")
for mode, stage in (("throughput", 4), ("latency", 1))]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, name, path: captured.append(fig))
comparison.draw_metric(rows, metric, ylabel, "compact", tmp_path,
@@ -227,30 +387,84 @@ def test_compact_metric_bars_and_units(comparison, tmp_path, monkeypatch, diverg
fig = captured[0]
try:
assert fig.get_size_inches() == pytest.approx((8.5, 4.4))
expected_legend = {
"speedup": ["Single-inference latency", "Best-case throughput"],
"energy": ["Single-inference energy", "Best-case throughput energy"],
"power": ["Single-inference power", "Best-case throughput power"],
expected_panels = {
"speedup": ["Latency-mode", "Throughput-mode"],
"energy": ["Latency-mode energy", "Throughput-mode energy"],
"power": ["Latency-mode power", "Throughput-mode power"],
}
assert [text.get_text() for text in fig.legends[0].get_texts()] == expected_legend[metric]
assert [text.get_text() for text in fig.legends[0].get_texts()] == ["Arch-A", "Arch-B"]
for index, ax in enumerate(fig.axes):
latency, throughput = (container.patches[0] for container in ax.containers)
bars = [container.patches[0] for container in ax.containers]
assert len(bars) == 4
assert ax.get_xlabel() == expected_panels[metric][index]
assert [text.get_text() for text in ax.get_xticklabels()] == ["VGG-8", "ResNet-18"]
assert list(ax.get_xticks()) == [0.5, 3.5]
lower, upper = ax.get_xlim()
assert latency.get_x() - lower == pytest.approx(0.4)
assert upper - throughput.get_x() - throughput.get_width() == pytest.approx(0.4)
assert latency.get_x() + latency.get_width() == throughput.get_x()
assert latency.get_facecolor() == to_rgba(comparison.METRIC_STYLES["latency"][0])
assert throughput.get_facecolor() == to_rgba(comparison.METRIC_STYLES["throughput"][0])
assert latency.get_edgecolor() == to_rgba(comparison.METRIC_STYLES["latency"][1])
assert throughput.get_edgecolor() == to_rgba(comparison.METRIC_STYLES["throughput"][1])
assert not latency.get_hatch() and throughput.get_hatch() == "//"
assert latency.get_height() == (-1 if diverging else 0.5)
assert throughput.get_height() == (2 if diverging else 3)
assert bars[0].get_x() - lower == pytest.approx(0.4)
assert upper - bars[-1].get_x() - bars[-1].get_width() == pytest.approx(0.4)
for offset in (0, 2):
arch_a, arch_b = bars[offset:offset + 2]
assert arch_a.get_x() + arch_a.get_width() == arch_b.get_x()
for bar, arch in zip((arch_a, arch_b), comparison.ARCHITECTURES):
fill, border, pattern = comparison.ARCH_STYLES[arch]
assert bar.get_facecolor() == to_rgba(fill)
assert bar.get_edgecolor() == to_rgba(border)
assert bar.get_hatch() == pattern
assert arch_a.get_height() == ((-1 if diverging else 0.5) if index == 0 else (2 if diverging else 3))
assert arch_b.get_height() == ((-3 if diverging else 0.25) if index == 0 else (3 if diverging else 4))
assert bars[1].get_x() + bars[1].get_width() < bars[2].get_x()
assert ax.get_ylabel() == (ylabel if index == 0 else "")
assert bool(ax.get_yticklabels()) == (index == 0)
assert all("×" not in label.get_text() for label in ax.get_yticklabels())
assert [text.get_text() for text in ax.texts] == [
"2.00×" if diverging else "0.50×", "P. INVALID", "3.00×", "P. INVALID"]
values = (["2.00×", "4.00×"] if diverging else ["0.50×", "0.25×"]) if index == 0 else ["3.00×", "4.00×"]
assert [text.get_text() for text in ax.texts] == [value for _ in range(2) for number in values
for value in (number, "P. INVALID")]
finally:
comparison.plt.close(fig)
@pytest.mark.parametrize("minimum,expected", [(0.4, 0.25), (0.5, 0.5), (0.8, 0.5)])
@pytest.mark.parametrize("compact", [False, True])
def test_ratio_bar_minimum_keeps_values_and_labels_visible(comparison, tmp_path, monkeypatch,
minimum, expected, compact):
template = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")[0]
rows = [replace(template, arch=arch, mode=mode, pipeline=stage,
power=minimum if arch == "arch-a" else 2 if mode == "latency" else math.nan)
for arch in comparison.ARCHITECTURES for mode, stage in (("latency", 1), ("throughput", 4))]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_metric(rows, "power", "Average power reduction (x)", "power", tmp_path, compact=compact)
fig = captured[0]
try:
fig.canvas.draw()
renderer = fig.canvas.get_renderer()
assert fig.axes[0].get_ylim() == fig.axes[1].get_ylim()
for index, ax in enumerate(fig.axes):
assert ax.get_ylim()[0] == expected
if index == 0:
assert f"{expected:g}" in [text.get_text() for text in ax.get_yticklabels(which="both")]
else:
assert not ax.get_yticklabels(which="both")
bounds = ax.get_window_extent(renderer)
for text in ax.texts:
extent = text.get_window_extent(renderer)
assert bounds.y0 <= extent.y0 <= extent.y1 <= bounds.y1
assert any(text.get_text() == "N/A" for ax in fig.axes for text in ax.texts)
finally:
comparison.plt.close(fig)
def test_compact_summary_uses_lower_minimum_when_needed(comparison, tmp_path, monkeypatch):
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
rows = [replace(row, energy=0.4) if row.model == "vgg8" and row.mode == "latency" else row for row in rows]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_compact_summary(rows, tmp_path)
fig = captured[0]
try:
fig.canvas.draw()
assert all(ax.get_xlim()[0] == 0.25 for ax in fig.axes)
assert "0.25" in [text.get_text() for text in fig.axes[0].get_xticklabels()]
finally:
comparison.plt.close(fig)
@@ -270,7 +484,7 @@ def test_compact_metrics_use_speedup_pipeline_selection(comparison, monkeypatch,
captured.append((selected, field, basename, diverging))
monkeypatch.setattr(comparison, "load_results", lambda path: rows)
for name in ("configure_style", "draw_validation", "draw_pipeline_effect", "draw_absolute"):
for name in ("configure_style", "draw_validation", "draw_pipeline_effect", "draw_absolute", "draw_compact_summary"):
monkeypatch.setattr(comparison, name, lambda *args, **kwargs: None)
monkeypatch.setattr(comparison, "draw_metric", capture)
monkeypatch.setattr(sys, "argv", ["generate_plots.py"])
@@ -293,8 +507,8 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat
assert comparison.COMPILER_STYLES == {
"raptor": ("#B3D5CA", "#4B7C6A"), "pimcomp": ("#E1BC8B", "#856A44")}
assert comparison.METRIC_STYLES == {
"latency": ("#B3CCE3", "#456B90"), "throughput": ("#B3D5CA", "#4B7C6A")}
assert comparison.ARCH_STYLES == {
"arch-a": ("#B3CCE3", "#456B90", ""), "arch-b": ("#B3D5CA", "#4B7C6A", "//")}
assert comparison.VALIDATION_STYLES["PASS"] == ("#B3D5CA", "#4B7C6A")
assert comparison.TEXT_COLOR == "#222222"
assert comparison.NETWORK_STYLES["yolo11n"] == ("#FF9494", "#C52020")
@@ -304,7 +518,7 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat
for row in rows if row.model == "vgg8" and row.mode == "latency"]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_metric(selected, "power", "Average power advantage (x)", "power", tmp_path, diverging=True)
comparison.draw_metric(selected, "power", "Average power reduction (x)", "power", tmp_path, diverging=True)
comparison.draw_absolute(selected, "latency", "Single-inference latency (ms)", "latency", tmp_path)
try:
for ax, compiler in zip(captured[0].axes, ("raptor", "pimcomp")):
@@ -319,6 +533,44 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat
comparison.plt.close(fig)
@pytest.mark.parametrize("hatch", ["//", "\\\\", ".."])
@pytest.mark.parametrize("width,height", [(20, 130), (160, 15), (20, 1)])
@pytest.mark.parametrize("clipped", [False, True])
def test_hatch_geometry_stays_inside_bar(comparison, monkeypatch, hatch, width, height, clipped):
from matplotlib.backends.backend_agg import RendererAgg
from matplotlib.path import Path as PlotPath
from matplotlib.transforms import Affine2D, Bbox
from plot_style import CenteredHatch
renderer = RendererAgg(400, 300, 100)
gc = renderer.new_gc()
gc.set_hatch(hatch)
gc.set_hatch_color((0, 0, 0, 1))
path = PlotPath.unit_rectangle()
transform = Affine2D().scale(width, height).translate(50, 70)
bar = transform.transform_path(path).get_extents()
if clipped:
gc.set_clip_rectangle(Bbox.from_extents(55, 70.25, 180, 180))
bar = Bbox.intersection(bar, gc.get_clip_rectangle())
drawn = []
def capture(context, geometry, affine, face=None):
drawn.append((context.get_hatch(), affine.transform_path(geometry).get_extents()))
def capture_markers(context, marker, marker_transform, centers, affine, face=None):
for x, y in affine.transform_path(centers).vertices:
capture(context, marker, marker_transform + Affine2D().translate(x, y), face)
monkeypatch.setattr(renderer, "draw_path", capture)
monkeypatch.setattr(renderer, "draw_markers", capture_markers)
CenteredHatch().draw_path(renderer, gc, path, transform, (1, 1, 1, 1))
assert len(drawn) > 1
assert all(pattern is None for pattern, _ in drawn)
for _, bounds in drawn[1:]:
assert bar.x0 - 1e-8 <= bounds.x0 <= bounds.x1 <= bar.x1 + 1e-8
assert bar.y0 - 1e-8 <= bounds.y0 <= bounds.y1 <= bar.y1 + 1e-8
def test_tick_rotation_changes_only_overlapping_labels(comparison):
fig, ax = comparison.plt.subplots(figsize=(4, 3), layout="constrained")
try:
@@ -341,6 +593,14 @@ def test_all_plot_terminology(comparison, monkeypatch):
def capture(fig, name, output):
fig.canvas.draw()
if "compact" in name:
compact_text = "\n".join(text.get_text().lower() for text in fig.findobj(Text))
assert "single-inference" not in compact_text
assert "best-case throughput" not in compact_text
if name == "performance_energy_compact":
labels.update(text.get_text() for text in fig.findobj(Text))
plotter.plt.close(fig)
return
if name != "raptor_pipeline_throughput":
assert fig.axes[1].get_ylabel() == ""
assert fig.axes[1].get_yticklabels() == []
@@ -393,13 +653,14 @@ def test_all_plot_terminology(comparison, monkeypatch):
plotter.main()
text = "\n".join(labels)
for old in ("Pimcomp", "ARCHITECTURE", "VGG8", "RESNET", "GOOGLENET", "YOLO11N",
"Single inference", "Best case", "Latency, 1 stage", "Energy per inference advantage"):
"Single inference", "Best case", "Latency, 1 stage", "Energy per inference advantage",
"Energy-per-inference advantage", "Energy consumption reduction", "power advantage"):
assert old not in text
assert {"VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet"} <= labels
assert "Arch-A" in text and "Arch-B" in text
assert {"Raptor", "PIMCOMP", "YOLO11n", "Single-inference", "2 stages", "4 stages", "8 stages",
"Single-inference latency (ms)", "Throughput (inferences/s)", "Energy per inference (mJ)",
"Energy-per-inference advantage (x)", "Derived as 1 / latency",
"Energy reduction (x)", "Average power reduction (x)", "Derived as 1 / latency",
"Throughput measurement"} <= labels
@@ -432,7 +693,7 @@ def test_advantage_throughput_groups_touch(comparison, tmp_path, monkeypatch, di
selected = [row for row in rows if row.model in ("vgg8", "resnet18")]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_metric(selected, "energy", "Energy-per-inference advantage (x)", "energy", tmp_path,
comparison.draw_metric(selected, "energy", "Energy reduction (x)", "energy", tmp_path,
diverging=diverging)
fig = captured[0]
try:
@@ -466,8 +727,10 @@ def test_grouped_absolute_plot_scheduling_labels(comparison, tmp_path, monkeypat
comparison.plt.close(fig)
def test_pipeline_plot_connects_derived_and_measured_rates(comparison, tmp_path, monkeypatch):
@pytest.mark.parametrize("status", ["PASS", "FAIL", "SKIPPED", "NA"])
def test_pipeline_plot_connects_derived_and_measured_rates(comparison, tmp_path, monkeypatch, status):
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
rows = [replace(row, raptor_validation=status, pimcomp_validation=status) for row in rows]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_pipeline_effect(rows, tmp_path)
@@ -518,6 +781,7 @@ def test_plotter_exports_png_svg_and_pdf(comparison, tmp_path):
for row in rows if row.model == "vgg8" and row.mode == "latency"]
comparison.draw_absolute(selected, "latency", "Latency (ms)", "comparison-smoke", tmp_path)
comparison.draw_validation(rows, tmp_path, compact=True)
comparison.draw_compact_summary(rows, tmp_path)
compact_names = []
for metric in ("speedup", "energy", "power"):
for suffix, diverging in (("comparison", False), ("diverging", True)):
@@ -525,10 +789,182 @@ def test_plotter_exports_png_svg_and_pdf(comparison, tmp_path):
compact_names.append(name)
comparison.draw_metric(comparison.select_compact_speedups(rows), metric, "Advantage (x)",
name, tmp_path, diverging=diverging, compact=True)
for name in ("comparison-smoke", "validation_status_compact", *compact_names):
for name in ("comparison-smoke", "validation_status_compact", "performance_energy_compact", *compact_names):
assert (tmp_path / f"png/{name}.png").read_bytes().startswith(b"\x89PNG\r\n\x1a\n")
svg = (tmp_path / f"svg/{name}.svg").read_text()
assert "<svg" in svg
assert (tmp_path / f"pdf/{name}.pdf").read_bytes().startswith(b"%PDF-")
assert "N/A" in (tmp_path / "svg/comparison-smoke.svg").read_text()
assert "P. INVALID" in (tmp_path / "svg/comparison-smoke.svg").read_text()
def test_compact_summary_ratios_validity_and_layout(comparison, tmp_path, monkeypatch):
from matplotlib.path import Path as PlotPath
from matplotlib.transforms import Affine2D, Bbox
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
selected = comparison.select_compact_speedups([row for row in rows if row.model != "yolo11n"])
lookup = {(row.model, row.arch, row.mode): row for row in selected}
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, name, path: captured.append((fig, name)))
comparison.draw_compact_summary(rows, tmp_path)
fig, name = captured[0]
try:
assert name == "performance_energy_compact"
assert len(fig.axes) == 4
fig.canvas.draw()
renderer = fig.canvas.get_renderer()
keys = [(model, arch) for model in comparison.MODELS[:-1] for arch in comparison.ARCHITECTURES]
for ax, mode, field in zip(fig.axes, ("latency", "throughput", "latency", "throughput"),
("latency_speedup", "throughput_speedup", "energy", "energy")):
assert ax.get_xscale() == "linear"
low, high = ax.get_xlim()
assert low == 0.5 < 1 < high
points = [line for line in ax.lines if line.get_marker() in ("o", "s", "^", "D")]
point_keys = [(*key, mode) for key in keys]
assert len(points) == len(point_keys)
expected, invalid_flags = [], []
for point, key in zip(points, point_keys):
row = lookup[key]
ratio = getattr(row, field)
expected.append(ratio)
assert list(point.get_xdata()) == pytest.approx([ratio])
assert list(point.get_ydata()) == [0 if row.arch == "arch-a" else 2]
assert point.get_marker() == {"vgg8": "o", "resnet18": "s", "resnet34": "^", "googlenet": "D"}[row.model]
fill, border = comparison.NETWORK_STYLES[row.model]
assert point.get_markerfacecolor() == fill
assert point.get_markeredgecolor() == border
invalid = row.mode == "throughput" or row.pimcomp_validation == "FAIL"
invalid_flags.append(invalid)
labels = [text for text in ax.texts if getattr(text, "xycoords", None) == "data"]
groups = []
for text, key, ratio, invalid in zip(labels, point_keys, expected, invalid_flags):
assert text.get_text() == f"{ratio:.2f}×"
assert text.get_math_fontfamily() == "cm"
assert text.get_color() == comparison.TEXT_COLOR
assert text.get_rotation() == 90
assert text.arrow_patch is None
assert abs(text.get_position()[0]) <= 8
assert (text.get_position()[1] > 0) == (key[1] == "arch-a")
children = [child for child in ax.texts if getattr(child, "xycoords", None) is text]
assert len(children) == 1
for dagger in children:
assert dagger.get_text() == r"$\boldsymbol{\dagger}$"
assert dagger.get_color() == comparison.DAGGER_COLOR == "#AA1C1C"
assert dagger.get_fontsize() == 9
assert dagger.get_fontsize() > text.get_fontsize()
assert dagger.get_alpha() == int(invalid)
groups.append([text, *children])
for lane in (0, 2):
offsets = [text.get_window_extent(renderer).y1 - ax.transData.transform(text.xy)[1]
for text in labels if text.xy[1] == lane]
assert max(offsets) == pytest.approx(min(offsets))
assert len(labels) == 8
assert list(ax.get_yticks()) == [0, 2]
assert [text.get_text() for text in ax.get_yticklabels()] == ["Arch-A", "Arch-B"]
assert ax.get_title() == ""
assert ax.yaxis.label.get_rotation() == 90
assert ax.yaxis.label._get_multialignment() == "center"
assert ax.yaxis.label.get_fontsize() == 8.5
assert len(ax.get_ylabel().splitlines()) <= 2
assert ax.yaxis.label.get_window_extent(renderer).x1 < ax.get_window_extent(renderer).x0
assert bool(ax.get_xticklabels()) == (ax is fig.axes[0])
if ax is fig.axes[0]:
assert "0.5" in [text.get_text() for text in ax.get_xticklabels()]
assert "0" not in [text.get_text() for text in ax.get_xticklabels()]
assert all(tick.label2.get_visible() and not tick.label1.get_visible()
for tick in ax.xaxis.get_major_ticks())
assert not any(text.get_text() in ("Raptor / PIMCOMP", "PIMCOMP / Raptor") for text in ax.texts)
baseline = next(line for line in ax.lines if list(line.get_xdata()) == [1, 1])
assert baseline.get_linestyle() == "--"
bounds = ax.get_window_extent(renderer)
paths = []
for line in ax.lines:
box = line.get_window_extent(renderer).padded(1)
paths.append(PlotPath.unit_rectangle().transformed(
Affine2D().scale(box.width, box.height).translate(box.x0, box.y0)))
for group in groups:
text = group[0]
box = Bbox.union([part.get_window_extent(renderer) for part in group])
assert bounds.x0 <= box.x0 <= box.x1 <= bounds.x1
assert bounds.y0 <= box.y0 <= box.y1 <= bounds.y1
path = PlotPath.unit_rectangle().transformed(
Affine2D().scale(box.width, box.height).translate(box.x0, box.y0))
assert not any(path.intersects_path(other, filled=True) for other in paths), text.get_text()
paths.append(path)
assert "YOLO11n" not in [text.get_text() for text in ax.get_yticklabels()]
assert {text.get_position()[1] > 0 for ax in fig.axes for text in ax.texts
if getattr(text, "xycoords", None) == "data"} == {False, True}
assert all(ax.get_xlim() == fig.axes[0].get_xlim() for ax in fig.axes)
assert [ax.get_ylabel() for ax in fig.axes] == [
"Latency-mode\nspeedup", "Throughput-mode\nspeedup",
"Latency-mode\nenergy reduction", "Throughput-mode\nenergy reduction"]
assert fig.axes[0].xaxis.get_label_position() == "top"
xlabel = fig.axes[0].xaxis.label.get_window_extent(renderer)
legend = fig.legends[0].get_window_extent(renderer)
assert fig.axes[0].get_window_extent(renderer).y1 < xlabel.y0 < xlabel.y1 < legend.y0
tick_top = max(text.get_window_extent(renderer).y1 for text in fig.axes[0].get_xticklabels())
assert 0 < xlabel.y0 - tick_top < renderer.points_to_pixels(12)
axis_bounds = fig.axes[0].get_window_extent(renderer)
assert (xlabel.x0 + xlabel.x1) / 2 == pytest.approx((axis_bounds.x0 + axis_bounds.x1) / 2)
assert fig.axes[0].get_xlabel() == "Ratio (x)"
assert {"← PIMCOMP better", "Raptor better →"} <= {text.get_text() for text in fig.axes[0].texts}
left, right = [text.get_window_extent(renderer) for text in fig.axes[0].texts if not hasattr(text, "xy")]
assert left.y0 == right.y0 == pytest.approx(xlabel.y0)
assert left.x1 < xlabel.x0 < xlabel.x1 < right.x0
legend_labels = {text.get_text() for text in fig.legends[0].get_texts() if text.get_visible()}
assert legend_labels == {"VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet", "PIMCOMP invalid"}
for handle, text in zip(fig.legends[0].legend_handles, fig.legends[0].get_texts()):
if text.get_text() in comparison.MODEL_NAMES.values():
model = next(model for model, name in comparison.MODEL_NAMES.items() if name == text.get_text())
assert handle.get_marker() == {"vgg8": "o", "resnet18": "s", "resnet34": "^", "googlenet": "D"}[model]
assert handle.get_markerfacecolor() == comparison.NETWORK_STYLES[model][0]
elif text.get_text() == "PIMCOMP invalid":
assert handle.get_text() == r"$\boldsymbol{\dagger}$"
assert handle.get_fontsize() == 9
assert handle.get_math_fontfamily() == "cm"
assert handle.get_color() == comparison.DAGGER_COLOR
result_dagger = next(child for child in fig.axes[0].texts
if getattr(getattr(child, "xycoords", None), "xycoords", None) == "data"
and child.get_alpha() == 1)
result_box = result_dagger.get_window_extent(renderer)
legend_box = handle.get_window_extent(renderer)
assert legend_box.width == pytest.approx(result_box.height)
assert legend_box.height == pytest.approx(result_box.width)
ratios = [line.get_xdata()[0] for ax in fig.axes for line in ax.lines
if line.get_marker() in ("o", "s", "^", "D")]
lower, upper = fig.axes[0].get_xlim()
padding = (max(ratios) - min(ratios)) * 0.04
assert lower == 0.5
assert upper - max(ratios) == pytest.approx(padding)
finally:
comparison.plt.close(fig)
def test_compact_summary_rejects_missing_ratio(comparison, tmp_path):
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
rows = [replace(row, energy=math.nan) if row.mode == "throughput" and row.model == "vgg8"
else row for row in rows]
with pytest.raises(ValueError, match="requires a positive energy ratio"):
comparison.draw_compact_summary(rows, tmp_path)
@pytest.mark.parametrize("compiler,status,label", [("pimcomp", "FAIL", r"$\boldsymbol{\dagger}$"), ("raptor", "FAIL", "R. INVALID"),
("pimcomp", "SKIPPED", "NOT VALIDATED"), ("pimcomp", "NA", "NOT VALIDATED")])
def test_compact_summary_validation_labels(comparison, tmp_path, monkeypatch, compiler, status, label):
rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")
rows = [replace(row, **{f"{compiler}_validation": status})
if row.model == "vgg8" and row.arch == "arch-a" and row.mode == "latency" else row for row in rows]
captured = []
monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig))
comparison.draw_compact_summary(rows, tmp_path)
fig = captured[0]
try:
ax = fig.axes[0]
if compiler == "pimcomp" and status == "FAIL":
assert any(label in text.get_text() and text.get_color() == comparison.DAGGER_COLOR for text in ax.texts)
else:
assert label in ax.texts[0].get_text()
assert ax.texts[0].get_color() == comparison.TEXT_COLOR
finally:
comparison.plt.close(fig)
+44 -13
View File
@@ -12,10 +12,14 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
| Variant | Raptor options |
|---|---|
| `baseline` | None |
| `no-sync` | `--pim-disable-synchronization` |
| `no-plan` | `--pim-disable-spatial-planning` |
| `no-sync-no-plan` | Both options |
| `baseline` | None; planning and synchronization enabled |
| `no-sync` | `--pim-disable-synchronization`; throughput only |
| `no-plan` | `--pim-disable-spatial-planning`; synchronization remains enabled |
Latency runs compare `baseline` with `no-plan`. Throughput runs compare
`baseline` with each disabled feature separately. No run disables both features.
The `baseline` artifact name identifies the fully enabled program, not the
denominator used for percentage gains.
## Options and outputs
@@ -25,7 +29,9 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`.
Other [suite-runner options](../pimcomp/compare/README.md), including `--models`,
`--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline`
replaces the default latency/pipeline-4 case set. `--only` is rejected because
replaces the default latency/pipeline-4 case set. Mode-only selection runs all
suite pipelines for that mode. Pipeline-only selection uses its matching mode.
`--only` is rejected because
the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers.
Artifacts use `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`;
@@ -37,19 +43,44 @@ By default, the combined [results_ablation.csv](results_ablation.csv) is written
beside this script and printed after the run. An explicit `--out-dir PATH`
writes `PATH/results_ablation.csv`, keeping independent experiments separate.
Default model artifacts and per-variant summaries remain in the locations above.
The aggregate records case identifiers,
`functional_validation`, and signed percentage changes in latency, throughput,
power, and energy against the matching baseline case. Positive means higher;
baseline values are zero.
The aggregate contains one row per comparison:
| Mode | `comparison` | `baseline_variant` | Main gain |
|---|---|---|---|
| Latency | `plan` | `no-plan` | `latency_gain_percent` |
| Throughput | `plan` | `no-plan` | `throughput_gain_percent` |
| Throughput | `sync` | `no-sync` | `throughput_gain_percent` |
Throughput gain is `100 × (enabled / disabled − 1)`.
Latency, power, and energy gains are reductions: `100 × (1 − enabled / disabled)`.
The corresponding columns use the suffix `_gain_percent`. Positive means improvement;
negative means a cost. Synchronization gain can be negative. No sign is forced.
Both programs' functional statuses are recorded as `functional_validation`
and `baseline_functional_validation`. The latter belongs to the disabled feature.
Disabling synchronization removes host-buffer lifetime and pipeline restart
guarantees, so those variants may fail numerical validation and exit nonzero.
Baseline and `no-plan` must pass. Percentages from failing variants are not
valid speedups; deadline averages do not prove steady-state convergence.
Baseline and `no-plan` must pass. Percentages involving failing programs remain
diagnostic comparisons, not valid speedups.
When fast-mode convergence fails, positive fallback throughput remains available.
Its output interval uses the full simulation window. The per-case JSON report
records `throughput_measurement: full_duration_fallback` and
`fast_throughput_converged: false`. This result does not prove steady-state convergence.
Zero completed outputs still produce `NA` for per-inference metrics.
The `no-sync` compiler option keeps the normal selected schedule and omits
SYNC/WAIT instructions during code generation. Selection costs describe the
`synchronized-reference` program. Each backend attempt also reports a separate
`synchronization_ablation_cost`, with zero SYNC/WAIT work and no synchronization
dependencies. This diagnostic estimate is not used for selection. Its complete-state
recurrence can remain unproved because cores can advance independently.
Numerical validation still checks the emitted program and can fail.
## Result data
The checked-in `results_ablation.csv` is an intentional result dataset for tabular
reporting. Ablation experiments and aggregation remain available through the
The checked-in `results_ablation.csv` is a historical dataset until the next run
regenerates it with the comparison schema above. Existing measurements are preserved.
Ablation experiments and aggregation remain available through the
runner above. Percentages from failing variants are not valid speedups. See the
[comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation).
+69 -26
View File
@@ -37,10 +37,6 @@ VARIANTS = (
Variant("baseline", ()),
Variant("no-sync", ("--pim-disable-synchronization",)),
Variant("no-plan", ("--pim-disable-spatial-planning",)),
Variant(
"no-sync-no-plan",
("--pim-disable-synchronization", "--pim-disable-spatial-planning"),
),
)
VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS}
COMPARISON_RESULTS_FILENAME = "results_comparison.csv"
@@ -53,17 +49,19 @@ PERCENTAGE_FIELDS = (
("raptor_power_mw", "power"),
("raptor_energy_pj", "energy"),
)
RESULT_FIELDS = (*CASE_FIELDS, "functional_validation", *(target for _, target in PERCENTAGE_FIELDS))
RESULT_FIELDS = (
"comparison", *CASE_FIELDS, "baseline_variant", "baseline_functional_validation",
"functional_validation", *(f"{target}_gain_percent" for _, target in PERCENTAGE_FIELDS),
)
DEFAULT_CASE_ARGUMENTS = (
("latency", ("--mode", "latency")),
("throughput/pipeline4", ("--mode", "throughput", "--pipeline", "4")),
)
DEFAULT_CASE_KEYS = frozenset({("latency", "1"), ("throughput", "4")})
def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[str]]:
parser = argparse.ArgumentParser(
description="Run the complete compiler ablation matrix on the Pimcomp model suite.",
description="Measure planning and synchronization gains on the Pimcomp model suite.",
allow_abbrev=False,
)
parser.add_argument(
@@ -97,12 +95,30 @@ def has_option(arguments: list[str], option: str) -> bool:
def runner_argument_sets(forwarded: list[str]) -> tuple[tuple[str, list[str]], ...]:
if has_option(forwarded, "--mode") or has_option(forwarded, "--pipeline"):
return (("requested", forwarded),)
return tuple(
(label, [*forwarded, *case_arguments])
for label, case_arguments in DEFAULT_CASE_ARGUMENTS
)
if not has_option(forwarded, "--mode") and not has_option(forwarded, "--pipeline"):
return tuple(
(label, [*forwarded, *case_arguments])
for label, case_arguments in DEFAULT_CASE_ARGUMENTS
)
parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
parser.add_argument("--mode", nargs="+", choices=("latency", "throughput"))
parser.add_argument("--pipeline", type=int, choices=(1, 2, 4, 8))
args, remaining = parser.parse_known_args(forwarded)
cases = [(mode, pipeline) for mode, pipeline in
(("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8))
if (args.mode is None or mode in args.mode)
and (args.pipeline is None or pipeline == args.pipeline)]
if not cases:
parser.error("--pipeline does not belong to the selected --mode")
return tuple((f"{mode}/pipeline{pipeline}",
[*remaining, "--mode", mode, "--pipeline", str(pipeline)])
for mode, pipeline in cases)
def case_key(arguments: list[str]) -> tuple[str, str]:
mode = arguments[arguments.index("--mode") + 1]
pipeline = arguments[arguments.index("--pipeline") + 1] if "--pipeline" in arguments else "1"
return mode, pipeline
def variant_output_dir(out_dir: Path, variant: Variant) -> Path:
@@ -154,7 +170,9 @@ def terminate_process_group(process: subprocess.Popen[bytes]) -> None:
process.wait()
def percentage_difference(value: str | None, reference: str | None) -> str:
def percentage_difference(
value: str | None, reference: str | None, *, lower_is_better: bool = False
) -> str:
try:
current = float(value) if value is not None else math.nan
baseline = float(reference) if reference is not None else math.nan
@@ -162,7 +180,8 @@ def percentage_difference(value: str | None, reference: str | None) -> str:
return "NA"
if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0:
return "NA"
change = (current / baseline - 1) * 100
ratio = current / baseline
change = (1 - ratio if lower_is_better else ratio - 1) * 100
return f"{change:+.2f}%" if math.isfinite(change) else "NA"
@@ -202,25 +221,43 @@ def aggregate_results(
selected_rows.append(row)
if fields is None:
return None, failures
baseline_rows = {
enabled_rows = {
tuple(row.get(field, "") for field in CASE_FIELDS): row
for row in rows_by_variant.get("baseline", [])
}
if "baseline" not in rows_by_variant:
failures.append("baseline: missing percentage reference results")
failures.append("baseline: missing fully enabled results")
result_rows = []
variant_order = {variant.name: index for index, variant in enumerate(VARIANTS)}
for variant in variants:
if variant.name == "baseline":
continue
disabled_keys = {
tuple(row.get(field, "") for field in CASE_FIELDS)
for row in rows_by_variant.get(variant.name, [])
}
for key, row in enabled_rows.items():
if variant.name == "no-sync" and row.get("mode") != "throughput":
continue
if key not in disabled_keys:
failures.append(f"{variant.name}: missing baseline case " + "/".join(key))
for row in rows_by_variant.get(variant.name, []):
reference = baseline_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS))
if variant.name == "no-sync" and row.get("mode") != "throughput":
continue
enabled = enabled_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS))
if enabled is None:
failures.append(f"{variant.name}: missing fully enabled case "
+ "/".join(row.get(field, "") for field in CASE_FIELDS))
result_rows.append(
{
"variant": variant.name,
"comparison": "plan" if variant.name == "no-plan" else "sync",
**{field: row.get(field, "") for field in CASE_FIELDS},
"functional_validation": row.get("raptor_functional_validation", "NA"),
"baseline_variant": variant.name,
"baseline_functional_validation": row.get("raptor_functional_validation", "NA"),
"functional_validation": enabled.get("raptor_functional_validation", "NA") if enabled else "NA",
**{
target: percentage_difference(
row.get(source), reference.get(source) if reference else None
f"{target}_gain_percent": percentage_difference(
enabled.get(source) if enabled else None, row.get(source),
lower_is_better=target != "throughput",
)
for source, target in PERCENTAGE_FIELDS
},
@@ -231,14 +268,14 @@ def aggregate_results(
row["arch"],
row["model"],
row["mode"],
variant_order[row["variant"]],
int(row["raptor_pipeline"]),
row["comparison"],
)
)
output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME
output.parent.mkdir(parents=True, exist_ok=True)
with output.open("w", newline="", encoding="utf-8") as stream:
writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n")
writer = csv.DictWriter(stream, fieldnames=RESULT_FIELDS, lineterminator="\n")
writer.writeheader()
writer.writerows(result_rows)
return output, failures
@@ -252,9 +289,13 @@ def main(argv: list[str] | None = None) -> int:
variants = selected_variants(args.variants)
out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR
case_sets = runner_argument_sets(forwarded)
if not any(case_key(arguments)[0] == "throughput" for _, arguments in case_sets):
variants = [variant for variant in variants if variant.name != "no-sync"]
if args.dry_run:
for variant in variants:
for _, case_forwarded in case_sets:
if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput":
continue
print(
shlex.join(
runner_command(
@@ -275,6 +316,8 @@ def main(argv: list[str] | None = None) -> int:
try:
for variant in variants:
for case_label, case_forwarded in case_sets:
if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput":
continue
command = runner_command(
out_dir, variant, args.models, case_forwarded, out_dir, False
)
@@ -296,7 +339,7 @@ def main(argv: list[str] | None = None) -> int:
return 130
remove_lock_files(out_dir)
selected_cases = DEFAULT_CASE_KEYS if len(case_sets) > 1 else None
selected_cases = frozenset(case_key(arguments) for _, arguments in case_sets)
output, aggregation_failures = aggregate_results(
out_dir,
variants,
+96 -7
View File
@@ -39,13 +39,20 @@ beside this script. An explicit `--out-dir PATH` writes
Default model artifacts remain under `validation/networks/pimcomp_models`.
Artifacts use
`<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/`, with shared references
under `artifacts/common/` (`common-functional/` for YOLO's canonical reference)
under `artifacts/common/`
and `comparison_report.{md,json}` under each comparison's `pimcomp/`.
YOLO and GoogLeNet use `artifacts/common-full-model/` to avoid reusing historical Softmax-free references.
Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly
written report, including failures without simulation artifacts. Missing or
unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a
compiler, simulation, or functional comparison fails.
If fast-mode convergence fails, the reports and CSV retain valid full-duration
fallback throughput and output interval values. This is not a converged steady-state result.
Per-case JSON records `throughput_measurement: full_duration_fallback` and
`fast_throughput_converged: false`; the text report labels the fallback.
Zero completed outputs still give `NA` for per-inference metrics.
Reuse checks model/configuration paths and SHA-256 contents, effective hardware,
mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags.
Malformed, incomplete, mismatched, or legacy reports without fingerprints fail
@@ -56,6 +63,14 @@ exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity.
Tool/build revisions and manually edited reference artifacts are not hashed;
rerun the relevant compiler after those change.
Functional validation uses the same ONNX model and Raptor program as the performance comparison.
YOLO uses the full model, including both internal Softmax operations.
The functional simulator executes Softmax; Pimsim-nn and the queue estimator omit its timing cost.
Reported timing and energy exclude Softmax work. Other instructions and transfers retain their costs.
GoogLeNet removes only its two LRN nodes and retains terminal Softmax.
VGG8 and both ResNet graphs have no removed operators.
The native reference uses the same model as each compiled program.
## One-model comparator
`compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`.
@@ -63,6 +78,13 @@ Use `--help` for hardware and simulation options. The suite runner supplies
these automatically.
The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
`validate.py` uses `1e-3`/`1e-5`.
Both use the [same output checks](../../../../README.md#performance-and-result-interpretation),
including a relative L2 limit for each nonzero reference output.
`--rtol=0` disables that additional limit in the comparator.
Reports from an older validation policy require regeneration before reuse.
An elementwise failure with relative L2 agreement remains `FAIL`.
Its report notes possible numerical sensitivity. That note does not establish correctness.
Every batch iteration must pass. Reported errors are maxima across iterations for each output.
## Comparison charts
@@ -71,19 +93,43 @@ The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`;
```
Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied
`results_comparison.csv` beside the script and writes `plots/png/` and
`plots/svg/` and `plots/pdf/` there. Use `--input PATH` for a runner-generated CSV and
`results_comparison.csv` beside the script and writes `plots/default/png/`,
`plots/default/svg/`, and `plots/default/pdf/` there. Use `--input PATH` for a runner-generated CSV and
`--output-dir PATH` to override the destination. The plotter requires the full
40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput
pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp
configurations are rejected before plots are written.
Charts use the pastel palette and the Raptor name by default.
Use `--no-colors` for grayscale output; `--colors` explicitly enables colors.
Use `--anonymize` to replace displayed `Raptor` labels with `Anonimized`;
`--no-anonymize` explicitly keeps the Raptor name. The flags are independent.
Use `--no-marker-borders` for solid markers without outlines in the compact dot
plot and line chart, including their legends. These markers use the darker
border colors as solid fills. `--marker-borders` keeps the default outlines.
Line colors and bar patterns remain unchanged.
Input keys and output basenames remain stable. Use these output directories
to keep the grayscale and colored anonymized versions:
```bash
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
--no-colors --anonymize \
--output-dir validation/tools/pim/pimcomp/compare/plots/gray_anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
--anonymize \
--output-dir validation/tools/pim/pimcomp/compare/plots/anonimized
.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \
--anonymize --no-marker-borders \
--output-dir validation/tools/pim/pimcomp/compare/plots/no_border_anonimized
```
| Plot family | Meaning |
|---|---|
| `validation_status` | Recorded functional status for each compiler and experiment. |
| `validation_status_compact` | Compact functional status matrix. |
| `speedup_compact_*` | Touching single-inference latency and best-pipeline throughput bars for each network. |
| `energy_compact_*`, `power_compact_*` | Energy and power advantages for the same cases selected by the compact speedup plots. |
| `performance_energy_compact` | Four horizontal dot panels: latency-mode speedup, throughput-mode speedup, and separate energy panels for both modes. |
| `speedup_compact_*` | Latency panel on the left and best-pipeline throughput panel on the right, with touching Arch-A/B bars per network. |
| `energy_compact_*`, `power_compact_*` | Energy reduction and power reduction for the same cases selected by the compact speedup plots. |
| `latency_*` | Single-inference latency, latency mode only. |
| `throughput_*` | Inferences/s, throughput mode only. |
| `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. |
@@ -95,17 +141,47 @@ Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only);
advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for
latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use
winning factor minus one, with parity at zero.
Direct ratio axes start at 0.5, or 0.25 when a result is below 0.5.
Panels in each figure share the same minimum.
The shared `plot_style.py` defines the manuscript palette and display names.
Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns,
outlines, marker shapes, and status text preserve distinctions in grayscale.
Compact plots use blue for the single-inference case and hatched sage
green for the throughput case.
Compact bar plots use plain blue for Arch-A and hatched sage green for Arch-B.
The left panel contains latency-mode measurements. The right panel contains
throughput-mode measurements. Each network has touching Arch-A/B bars.
All compact figures use the display names `Latency-mode` and `Throughput-mode`.
Each architecture/network pair uses the highest recorded throughput across pipelines.
Energy and power use those same cases, without selecting a different pipeline for each metric.
Validation labels remain visible for the selected cases. Units appear in axis
titles and bar values; axis tick labels contain numbers only.
The `performance_energy_compact` figure includes the eight Arch-A/B comparisons
for VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It excludes YOLO11n.
All panels share a linear ratio axis from 0.5, or 0.25 when needed, through the largest ratio,
with a small right margin. A dashed line marks parity at 1.
The shared axis title and ticks appear at the top, below the legend.
The `Ratio (x)` title is centered over the axis. Panel titles appear to the left,
rotate 90° counterclockwise, and use centered text on at most two lines.
Titles start with `Latency-mode` or `Throughput-mode`, followed by the metric.
The PIMCOMP-better and Raptor-better indicators flank the shared axis title.
The figure has no mean markers.
Labels give ratios to two decimal places; marker positions use unrounded ratios.
Each panel has separate Arch-A and Arch-B lanes. The legend identifies models
through marker shapes and pastel colors. Result labels contain ratios and use
vertical text above or below their architecture line, without connectors.
Placement prefers above for Arch-A and below for Arch-B.
Lane spacing and label placement avoid other labels, markers, and reference lines.
Larger dark-red daggers mark invalid PIMCOMP comparisons, including every batch comparison,
even when the CSV records PASS. Marker borders retain the model colors;
ratio values use normal text colors. The legend dagger matches the result daggers.
Both use the same font and size. Labels without daggers reserve the same space,
so numeric values align within each lane.
These simulated ratios remain visible.
The two energy panels show latency-mode energy and throughput-selected energy.
The throughput energy uses the selected throughput pipeline, without a separate
energy optimization.
Throughput-stage bars touch within each network in both absolute and advantage
plots. Bar-label widths are measured before export; dense figures expand only
as needed to contain values and invalidity annotations. Legends use the available
@@ -116,6 +192,7 @@ pipeline plot retains its own scale, with y-axis ticks on both architecture rows
Only the bottom row shows the shared x-axis ticks. Network names appear above
the top row; architecture names appear on the right. That plot
trims unused vertical range and is intended to span both manuscript columns.
The line chart shows throughput only; validation status does not change its markers or colors.
Failed validation is labeled `P. INVALID`; other non-PASS statuses keep their
names, and missing measurements show `N/A`. Relative bars require both
@@ -130,3 +207,15 @@ publication provenance. The checked-in dataset is intentional. See the
See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation)
for measurement windows, units, numerical validity, and result provenance.
Report reuse checks model and configuration hashes, compiler and simulator
fingerprints, validation policy, actual tolerances, seed, batch size, and
simulation options. Artifact signatures check instructions, weights, and memory
images. Missing or changed artifacts require regeneration. Redundant Raptor
JSON instruction dumps do not affect signatures when native binaries exist.
`better_compiler` is available only when both compilers pass functional
validation. `speedup` retains the diagnostic timing ratio when both timings
exist. A timing ratio for a failed implementation does not identify a better
correct implementation. Valid-only plot aggregates require both validations
to pass.
@@ -18,7 +18,7 @@ import time
import types
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from dataclasses import asdict, dataclass
from dataclasses import asdict, dataclass, field
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import Any
@@ -45,6 +45,7 @@ from raptor_validation.onnx_utils import (
write_inputs_to_memory_bin,
)
from raptor_validation.raptor import compile_with_raptor
from raptor_validation.output_comparison import VALIDATION_POLICY, compare_output_arrays
from raptor_validation.pimsim_nn import (
export_raptor_pimsim_artifact,
parse_pimsim_nn_metrics,
@@ -75,6 +76,7 @@ class CompareResult:
max_diffs: dict[str, float]
status: str = "done"
error: str | None = None
relative_l2_errors: dict[str, float | None] = field(default_factory=dict)
def load_pimcomp_exporter():
@@ -264,20 +266,29 @@ def compare_simulator_outputs(
channel_last: bool = False,
) -> CompareResult:
sim_arrays = parse_pim_simulator_outputs(output_bin, outputs_desc)
if len(sim_arrays) != len(outputs_desc):
raise ValueError("simulator output count differs from the model output count")
max_diffs: dict[str, float] = {}
relative_errors: dict[str, float | None] = {}
passed = True
for sim_array, (idx, name, _, shape) in zip(sim_arrays, outputs_desc):
error = None
for sim_array, (idx, name, elem_type, shape) in zip(sim_arrays, outputs_desc):
if channel_last and len(shape) == 4:
n, c, h, w = shape
sim_array = sim_array.reshape(n, h, w, c).transpose(0, 3, 1, 2)
csv_name = reference_dir / f"output{idx}_{sanitize_output_name(name)}.csv"
ref = np.loadtxt(csv_name, delimiter=",", dtype=np.float32).reshape(shape)
diff = np.abs(sim_array.astype(np.float64) - ref.astype(np.float64))
allowed = threshold + rtol * np.abs(ref.astype(np.float64))
max_diffs[name] = float(np.max(diff))
if not np.all(diff <= allowed):
passed = False
return CompareResult(passed=passed, max_diffs=max_diffs)
ref = np.loadtxt(csv_name, delimiter=",", dtype=_ONNX_TO_NP[elem_type]).reshape(shape)
comparison = compare_output_arrays(sim_array, ref, atol=threshold, rtol=rtol)
max_diffs[name] = comparison.max_abs_diff
relative_errors[name] = comparison.relative_l2_error
passed &= comparison.passed
if not comparison.passed and error is None:
if (comparison.relative_l2_error is not None
and comparison.relative_l2_error <= rtol and not comparison.pointwise_passed):
error = f"{name}: elementwise mismatch despite relative L2 agreement; possible numerical sensitivity, not proof of correctness"
else:
error = f"{name}: output tolerance exceeded"
return CompareResult(passed=passed, max_diffs=max_diffs, relative_l2_errors=relative_errors, error=error)
def sanitize_output_name(name: str) -> str:
@@ -652,7 +663,9 @@ def run_functional_validation(
stage="Run functional simulation",
)
max_diffs: dict[str, float] = {}
relative_errors: dict[str, float | None] = {}
failed_iterations = []
failure_detail = None
for index, reference_dir in enumerate(reference_dirs):
result = compare_simulator_outputs(
batch_output_dir / f"output_{index:06d}.bin",
@@ -664,12 +677,20 @@ def run_functional_validation(
)
if not result.passed:
failed_iterations.append(index)
if failure_detail is None:
failure_detail = result.error
for name, diff in result.max_diffs.items():
max_diffs[name] = max(max_diffs.get(name, 0.0), diff)
for name, relative in result.relative_l2_errors.items():
if relative is not None:
relative_errors[name] = max(relative_errors.get(name) or 0.0, relative)
else:
relative_errors.setdefault(name, None)
return CompareResult(
passed=not failed_iterations,
max_diffs=max_diffs,
error=(f"batch iterations failed: {failed_iterations}" if failed_iterations else None),
error=(f"batch iterations failed: {failed_iterations}; {failure_detail}" if failed_iterations else None),
relative_l2_errors=relative_errors,
)
@@ -1126,6 +1147,8 @@ def perf_status(perf: dict[str, Any]) -> str:
return "SKIPPED"
if perf.get("error"):
return "FAILED"
if perf.get("throughput_measurement") == "full_duration_fallback":
return "DONE (full-duration fallback; unconverged)"
return "DONE"
@@ -1163,29 +1186,77 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None:
return Path(value) if value else None
def pimcomp_reuse_context(
args: argparse.Namespace,
model: Path,
functional_model: Path,
hardware: dict[str, int],
) -> dict[str, Any]:
files = {"model": model, "functional_model": functional_model,
"config": args.pimcomp_config}
digests = {}
for path in set(files.values()):
with path.open("rb") as stream:
digests[path] = hashlib.file_digest(stream, "sha256").hexdigest()
return {
"version": 1,
"files": {name: {"path": str(path), "sha256": digests[path]}
for name, path in files.items()},
"hardware": hardware,
"settings": {name: getattr(args, name) for name in (
"pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline",
"pimcomp_replication", "pimcomp_model_name", "seed", "batch_size",
"threshold", "rtol", "no_fast", "skip_pimsim_nn",
)},
}
def fingerprint_files(files: dict[str, Path]) -> dict[str, Any]:
fingerprints = {}
for name, path in files.items():
digest = None
if path.is_file():
with path.open("rb") as stream:
digest = hashlib.file_digest(stream, "sha256").hexdigest()
fingerprints[name] = {"path": str(path), "sha256": digest}
return fingerprints
def experiment_context(args, model, functional_model, hardware, side):
files = {"model": model, "functional_model": functional_model, "config": args.pimcomp_config,
"oracle": args.pimsim_nn_build_dir / "ChipTest",
"comparison": Path(__file__),
"validation": VALIDATION_DIR / "raptor_validation/output_comparison.py",
"functional_simulator": args.pim_simulator_dir / "target/release/pim-simulator"}
if side == "raptor":
files["compiler"] = args.raptor_path
else:
files["compiler"] = args.pimcomp_dir / "build/PIMCOMP-NN"
files["exporter"] = args.pimcomp_dir / "verification/export_to_pim_simulator.py"
files.update({f"frontend/{p.relative_to(args.pimcomp_dir / 'frontend')}": p
for p in sorted((args.pimcomp_dir / "frontend").rglob("*.py"))})
settings = {name: getattr(args, name) for name in (
"pimsim_mode", "pimsim_time_ms", "seed", "batch_size", "threshold", "rtol",
"no_fast", "skip_pimsim_nn")}
settings.update({name: getattr(args, name) for name in (
("raptor_extra_arg",) if side == "raptor" else
("pimcomp_pipeline", "pimcomp_replication", "pimcomp_model_name"))})
return {"version": 2, "validation_policy": VALIDATION_POLICY,
"files": fingerprint_files(files), "hardware": hardware, "settings": settings}
def pimcomp_reuse_context(args, model, functional_model, hardware):
return experiment_context(args, model, functional_model, hardware, "pimcomp")
def artifact_signatures(report, side):
names = (("raptor_pim", "raptor_pimsim_nn", "reference_runner", "reference_library",
"reference_inputs", "batch_inputs") if side == "raptor" else
("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn"))
signatures = {}
content = {}
for name in names:
root = path_from_report(report, name)
if root is None:
continue
if not root.exists():
signatures[name] = None
continue
paths = [root] if root.is_file() else sorted(p for p in root.rglob("*") if p.is_file())
binary = root.is_dir() and any(root.glob("core_*.pim"))
digest = hashlib.sha256()
for path in paths:
# JSON instruction dumps are redundant when native binaries exist.
if binary and path.parent == root and path.match("core_*.json"):
continue
resolved = path.resolve()
if resolved not in content:
with path.open("rb") as stream:
content[resolved] = hashlib.file_digest(stream, "sha256").hexdigest()
digest.update(str(path.relative_to(root) if root.is_dir() else path.name).encode())
digest.update(b"\0" + content[resolved].encode() + b"\0")
signatures[name] = digest.hexdigest()
return signatures
def verify_reused_artifacts(report, side):
if report.get("artifact_signatures", {}).get(side) != artifact_signatures(report, side):
raise ValueError(f"Reused {side} artifacts differ; regenerate the report")
def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]:
@@ -1212,6 +1283,7 @@ def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> di
restore_side_records(report, "pimcomp", [], [])
for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"):
path_from_report(report, name)
verify_reused_artifacts(report, "pimcomp")
return report
except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc:
raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc
@@ -1334,14 +1406,19 @@ def write_report(
if pimcomp_validation.error:
lines.append(f"- Pimcomp validation note: `{pimcomp_validation.error.splitlines()[0]}`")
lines.extend(["", "### Max output differences", ""])
lines.extend(["", "### Output errors", ""])
diff_names = sorted(set(raptor_validation.max_diffs) | set(pimcomp_validation.max_diffs))
if diff_names:
lines.extend(["| Output | Raptor max diff | Pimcomp max diff |", "|---|---:|---:|"])
lines.extend(["| Output | Raptor max diff | Pimcomp max diff | Raptor relative L2 | Pimcomp relative L2 |",
"|---|---:|---:|---:|---:|"])
for name in diff_names:
relative = [result.relative_l2_errors.get(name)
for result in (raptor_validation, pimcomp_validation)]
relative_text = [f"{value:.6e}" if value is not None else "N/A" for value in relative]
lines.append(
f"| `{name}` | {raptor_validation.max_diffs.get(name, float('nan')):.6e} | "
f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} |"
f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} | "
f"{relative_text[0]} | {relative_text[1]} |"
)
else:
lines.append("No output differences are available because validation did not run or failed before comparison.")
@@ -1358,24 +1435,24 @@ def write_report(
if pimsim_mode == "throughput":
lines.extend(
[
"| Compiler | Status | Avg latency | Throughput | Avg power | Avg energy | Output count |",
"| Compiler | Status | Output interval (ms) | Throughput | Average power (mW) | Energy per inference (pJ/inference) | Output count |",
"|---|---|---:|---:|---:|---:|---:|",
f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'average_latency_ms', 'ms')} | {perf_value(raptor_perf, 'throughput', 'samples/s')} | "
f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(raptor_perf, 'output_count')} |",
f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(raptor_perf, 'output_count')} |",
f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'average_latency_ms', 'ms')} | {perf_value(pimcomp_perf, 'throughput', 'samples/s')} | "
f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(pimcomp_perf, 'output_count')} |",
f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(pimcomp_perf, 'output_count')} |",
"",
]
)
else:
lines.extend(
[
"| Compiler | Status | Latency | Avg power | Avg energy |",
"| Compiler | Status | Latency (ms) | Average power (mW) | Energy per inference (pJ/inference) |",
"|---|---|---:|---:|---:|",
f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'latency_ms', 'ms')} | "
f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ')} |",
f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} |",
f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'latency_ms', 'ms')} | "
f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ')} |",
f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} |",
"",
]
)
@@ -1471,8 +1548,10 @@ def main():
parser.add_argument("--pim-simulator-dir", default=REPO / "backend-simulators/pim/pim-simulator", type=Path)
parser.add_argument("--pimsim-nn-build-dir", default=REPO / "backend-simulators/pim/pimsim-nn/build", type=Path)
parser.add_argument("--seed", type=int, default=0)
parser.add_argument("--threshold", type=float, default=1e-3)
parser.add_argument("--rtol", type=float, default=1e-4)
parser.add_argument("--threshold", type=float, default=1e-3,
help="Absolute tolerance for per-element output comparison.")
parser.add_argument("--rtol", type=float, default=1e-4,
help="Relative tolerance for per-element and output L2 comparison; 0 disables the L2 check.")
parser.add_argument(
"--timeout-seconds",
type=float,
@@ -1671,6 +1750,14 @@ def main():
reuse_report_path = args.reuse_raptor_report.resolve()
with open(reuse_report_path, "r", encoding="utf-8") as f:
reused = json.load(f)
if reused.get("validation_policy") != VALIDATION_POLICY:
raise ValueError("Reused Raptor validation policy differs; regenerate the Raptor report")
if reused.get("validation_tolerances") != {"atol": args.threshold, "rtol": args.rtol}:
raise ValueError("Reused Raptor validation tolerances differ; regenerate the Raptor report")
expected = experiment_context(args, model_path, functional_model_path, hardware, "raptor")
if reused.get("raptor_reuse_context") != expected:
raise ValueError("Reused Raptor model, compiler, simulation settings, or inputs differ; regenerate the report")
verify_reused_artifacts(reused, "raptor")
reused_hardware = reused["hardware"]
if reused_hardware != hardware:
raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
@@ -1848,13 +1935,13 @@ def main():
)
raptor_validation = validation if validation is not None else failed_validation("Raptor validation failed")
elif not reference_dirs:
raptor_validation = skipped_validation("Reference outputs are not available")
raptor_validation = failed_validation("Reference outputs are not available")
elif not outputs_desc:
raptor_validation = skipped_validation("Output descriptors are not available")
raptor_validation = failed_validation("Output descriptors are not available")
else:
raptor_validation = skipped_validation("Raptor input materialization failed")
raptor_validation = failed_validation("Raptor input materialization failed")
elif not reuse_raptor:
raptor_validation = skipped_validation("Raptor Pim compilation did not produce a Pim directory")
raptor_validation = failed_validation("Raptor Pim compilation did not produce a Pim directory")
pimcomp_model_path = model_path
@@ -2058,6 +2145,8 @@ def main():
)
json_report = {
"validation_policy": VALIDATION_POLICY,
"validation_tolerances": {"atol": args.threshold, "rtol": args.rtol},
"model": str(model_path),
"functional_model": str(functional_model_path),
"hardware": hardware,
@@ -2066,7 +2155,9 @@ def main():
"pimcomp_pipeline": args.pimcomp_pipeline,
"pimcomp_replication": args.pimcomp_replication,
"pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx",
"pimcomp_reuse_context": reuse_context,
"pimcomp_reuse_context": (reuse_context if reuse_pimcomp else
None if args.raptor_only else pimcomp_reuse_context(args, model_path, functional_model_path, hardware)),
"raptor_reuse_context": experiment_context(args, model_path, functional_model_path, hardware, "raptor"),
"pimcomp_config": str(args.pimcomp_config),
"raptor_extra_args": args.raptor_extra_arg,
"reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
@@ -2087,6 +2178,8 @@ def main():
"batch_inputs": optional_path(out_dir / "inputs.csv"),
"batch_outputs": optional_path(out_dir / "simulation/out_iterations"),
"reference_runner": optional_path(runner_path),
"reference_library": optional_path(common_dir / "runner" / f"{functional_model_path.stem}.so")
if (common_dir / "runner" / f"{functional_model_path.stem}.so").is_file() else None,
"raptor_pim": optional_path(raptor_pim_dir),
"raptor_functional_pim": optional_path(raptor_functional_pim_dir),
"raptor_pimsim_nn": optional_path(raptor_pimsim_dir),
@@ -2097,6 +2190,8 @@ def main():
"report_markdown": str(report_path),
},
}
json_report["artifact_signatures"] = {
side: artifact_signatures(json_report, side) for side in ("raptor", "pimcomp")}
json_path = out_dir / "pimcomp/comparison_report.json"
with open(json_path, "w", encoding="utf-8") as f:
json.dump(json_report, f, indent=2)
@@ -11,6 +11,7 @@ beside it, with outputs grouped in png/, svg/, and pdf/ under the output directo
YOLO models without PIMCOMP measurements get separate Raptor-only charts.
Comparisons use vertical bars for ratios, diverging advantages, and absolute
values, with compiler colors and patterns for the Raptor pipeline stages.
A compact dot-plot summary combines latency, throughput, and both energy cases.
Throughput-mode latency is the output interval, a bottleneck-stage proxy,
not end-to-end inference latency. Separate throughput plots use inferences/s.
Energy is reported per inference; power is averaged over the simulation.
@@ -36,11 +37,13 @@ matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.patches import Patch, Rectangle
from matplotlib.lines import Line2D
from matplotlib.text import Text
from matplotlib.ticker import FuncFormatter, MultipleLocator
from matplotlib.transforms import Bbox
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from plot_style import (ARCH_NAMES, BAND_COLOR, COMPILER_NAMES, COMPILER_STYLES,
ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR, METRIC_STYLES,
from plot_style import (ARCH_NAMES, ARCH_STYLES, BAND_COLOR, COMPACT_MODE_NAMES, COMPILER_NAMES, COMPILER_STYLES,
DAGGER_COLOR, ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR,
MODE_NAMES, MODEL_NAMES, NETWORK_STYLES,
PALETTE, TEXT_COLOR, add_legend, configure_style, fit_value_labels,
fit_bar_widths, fit_x_tick_labels, save_figure, value_label)
@@ -276,7 +279,7 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal
gap = 0.4 if index and mode != configurations[index - 1][0] else 0
positions.append(positions[-1] + 1 + gap if index else 0)
configuration_x = dict(zip(configurations, positions))
headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact
headers = [f"{COMPILER_NAMES['raptor']}\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact
else "PIMCOMP\nbatch" if mode == "throughput" and compiler == "pimcomp" and not compact
else COMPILER_NAMES[compiler]
for mode, stage, compiler in configurations]
@@ -288,7 +291,8 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal
for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]):
group_positions = [x for _, x in group]
centers.append((group_positions[0] + group_positions[-1]) / 2)
modes.append("Latency" if mode == "latency" else MODE_NAMES[mode])
modes.append(COMPACT_MODE_NAMES[mode] if compact else
"Latency" if mode == "latency" else MODE_NAMES[mode])
ax.set_xticks(centers, modes, fontsize=10)
ax.set_xticks(positions, headers, minor=True)
ax.tick_params(axis="x", which="minor", top=True, labeltop=True,
@@ -320,6 +324,140 @@ def select_compact_speedups(results: list[Result]) -> list[Result]:
return selected
def draw_compact_summary(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None:
selected = select_compact_speedups([row for row in results if row.model != "yolo11n"])
lookup = {(row.model, row.arch, row.mode): row for row in selected}
keys = [(model, arch) for model in MODELS if model != "yolo11n" for arch in ARCHITECTURES]
panels = ((f"{COMPACT_MODE_NAMES['latency']}\nspeedup", "latency", "latency_speedup"),
(f"{COMPACT_MODE_NAMES['throughput']}\nspeedup", "throughput", "throughput_speedup"),
(f"{COMPACT_MODE_NAMES['latency']}\nenergy reduction", "latency", "energy"),
(f"{COMPACT_MODE_NAMES['throughput']}\nenergy reduction", "throughput", "energy"))
measurements = []
for _, mode, field in panels:
points = []
for model, arch in keys:
row = lookup[(model, arch, mode)]
ratio = getattr(row, field)
if not math.isfinite(ratio) or ratio <= 0:
raise ValueError(f"Compact summary requires a positive {field} ratio for {arch}/{model}")
pimcomp_invalid = row.pimcomp_schedule == "batch" or row.pimcomp_validation == "FAIL"
status = ("P. INVALID" if pimcomp_invalid else
f"{COMPILER_NAMES['raptor'][0]}. INVALID" if row.raptor_validation == "FAIL" else
"NOT VALIDATED" if not row.comparison_passed else "")
points.append((row, ratio, status))
measurements.append(points)
ratios = [ratio for points in measurements for _, ratio, _ in points]
padding = max(max(ratios) - min(ratios), 1) * 0.04
limits = (0.25 if min(ratios) < 0.5 else 0.5, max(1, *ratios) + padding)
fig, axes = plt.subplots(4, 1, sharex=True, figsize=(6.8, 7.6), layout="constrained")
markers = dict(zip((model for model in MODELS if model != "yolo11n"), ("o", "s", "^", "D")))
lanes = {arch: index * 2 for index, arch in enumerate(ARCHITECTURES)}
daggers = {}
for ax, (title, _, _), points in zip(axes, panels, measurements):
ax.set_xlim(*limits)
ax.set_ylim(4, -2)
ax.set_ylabel(title, rotation=90, ha="center", va="bottom", multialignment="center",
fontsize=8.5, labelpad=10)
ax.set_yticks(list(lanes.values()), [ARCH_NAMES[arch] for arch in lanes], fontsize=8)
ax.set_xticks([limits[0], *range(1, math.floor(limits[1]) + 1)])
ax.xaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{value:g}"))
ax.tick_params(axis="both", length=0, pad=4, labelsize=8.5)
ax.tick_params(axis="x", bottom=False, labelbottom=False,
top=ax is axes[0], labeltop=ax is axes[0])
ax.grid(axis="x", linewidth=0.6)
ax.axvline(1, color=TEXT_COLOR, linewidth=0.9, linestyle="--", zorder=1)
for lane in lanes.values():
ax.axhline(lane, color=LINE_COLOR, linewidth=0.6)
ax.spines["left"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["top"].set_visible(ax is axes[0])
for row, ratio, status in points:
fill, border = NETWORK_STYLES[row.model]
lane = lanes[row.arch]
ax.plot(ratio, lane, marker=markers[row.model], markersize=6.5, linestyle="none",
markerfacecolor=fill if marker_borders else border, markeredgecolor=border,
markeredgewidth=1.1 if marker_borders else 0)
suffix = f" [{status}]" if status and status != "P. INVALID" else ""
label = f"{ratio:.2f}×{suffix}"
annotation = ax.annotate(label, (ratio, lane), xytext=(0, -5), textcoords="offset points",
rotation=90, rotation_mode="anchor", ha="right", va="center",
fontsize=7.5, color=TEXT_COLOR, math_fontfamily="cm")
annotation.set_in_layout(False)
dagger = ax.annotate(r"$\boldsymbol{\dagger}$", (0.5, 1), xycoords=annotation,
xytext=(0, 1), textcoords="offset points", rotation=90,
rotation_mode="anchor", ha="left", va="center", fontsize=9,
color=DAGGER_COLOR, math_fontfamily="cm",
alpha=1 if status == "P. INVALID" else 0)
dagger.set_in_layout(False)
daggers[annotation] = dagger
handles = [Line2D([], [], marker=marker, color=NETWORK_STYLES[model][1],
markerfacecolor=NETWORK_STYLES[model][0 if marker_borders else 1], markersize=6.5,
markeredgewidth=1 if marker_borders else 0,
linestyle="none", label=MODEL_NAMES[model])
for model, marker in markers.items()]
handles.append(Text(0, 0, r"$\boldsymbol{\dagger}$", color=DAGGER_COLOR, fontsize=9,
math_fontfamily="cm", label="PIMCOMP invalid"))
add_legend(fig, handles)
axes[0].xaxis.set_label_position("top")
axes[0].set_xlabel("Ratio (x)", fontsize=9, labelpad=7, va="bottom")
axes[0].xaxis.set_label_coords(0.5, 1.18)
axes[0].text(0, 1.18, "← PIMCOMP better", transform=axes[0].transAxes, ha="left", va="bottom", fontsize=8)
axes[0].text(1, 1.18, f"{COMPILER_NAMES['raptor']} better →", transform=axes[0].transAxes,
ha="right", va="bottom", fontsize=8)
fig.canvas.draw()
renderer = fig.canvas.get_renderer()
def label_bounds(text):
return Bbox.union([text.get_window_extent(renderer), daggers[text].get_window_extent(renderer)])
def position_label(text, dx, dy):
text.set_ha("left" if dy > 0 else "right")
extra = (daggers[text].get_window_extent(renderer).height + renderer.points_to_pixels(1)
if dy < 0 else 0)
text.set_position((dx, dy - extra * 72 / fig.dpi))
labels_by_axis = [[text for text in ax.texts if getattr(text, "xycoords", None) == "data"] for ax in axes]
label_height = max(label_bounds(text).height for labels in labels_by_axis for text in labels)
lane_gap = min(abs(ax.transData.transform((0, 0))[1] - ax.transData.transform((0, 2))[1])
for ax in axes)
height_scale = (label_height + renderer.points_to_pixels(5)) / lane_gap
if height_scale > 1:
fig.set_size_inches(fig.get_figwidth(), fig.get_figheight() * height_scale)
fig.canvas.draw()
renderer = fig.canvas.get_renderer()
for ax, labels in zip(axes, labels_by_axis):
placed = []
for line in ax.lines:
placed.append(line.get_window_extent(renderer).padded(1))
bounds = ax.get_window_extent(renderer)
choices = []
for text in labels:
preferred_dy = 5 if text.xy[1] == lanes[ARCHITECTURES[0]] else -5
candidates = []
for dx in (0, -3, 3, -6, 6, -8, 8):
for dy in (-5, 5):
position_label(text, dx, dy)
box = label_bounds(text)
overlap = any(box.overlaps(other) for other in placed)
outside = not (bounds.x0 <= box.x0 <= box.x1 <= bounds.x1 and
bounds.y0 <= box.y0 <= box.y1 <= bounds.y1)
if not outside and not overlap:
candidates.append(((dy != preferred_dy, abs(dx) + abs(dy)), dx, dy, box))
choices.append((text, candidates))
layouts = [((0, 0), [])]
for text, candidates in sorted(choices, key=lambda choice: len(choice[1])):
layouts = sorted(
(((cost[0] + distance[0], cost[1] + distance[1]), assignments + [(text, dx, dy, box)])
for cost, assignments in layouts for distance, dx, dy, box in candidates
if not any(box.overlaps(other[3]) for other in assignments)),
key=lambda layout: layout[0])[:32]
if not layouts:
raise ValueError("Dot labels do not fit without overlap; increase lane spacing or figure width")
for text, dx, dy, _ in layouts[0][1]:
position_label(text, dx, dy)
save_figure(fig, "performance_energy_compact", output_dir)
def draw_metric(results: list[Result], field: str, ylabel: str,
basename: str, output_dir: Path, diverging: bool = False,
compact: bool = False) -> None:
@@ -327,48 +465,52 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
for row in results]
available = [v for v in data if math.isfinite(v)]
upper = max(0, max(available, default=1.0))
lower = min(0, min(available, default=0)) if diverging else 0
lower = (min(0, min(available, default=0)) if diverging else
0.25 if min(available, default=1) < 0.5 else 0.5)
padding = max(upper - lower, 1.0) * 0.3
y_max = math.ceil((upper + padding) * 2) / 2
y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0
y_min = math.floor((lower - padding) * 2) / 2 if diverging else lower
y_interval = 1.0 if y_max - y_min > 4.5 else 0.5
fig, axes = plt.subplots(1, 2, sharey=True, figsize=(8.5 if compact else 12, 4.4),
layout="constrained")
if compact:
single_metric = "latency" if field == "speedup" else field
throughput_suffix = "" if field == "speedup" else f" {field}"
handles = [Patch(facecolor=METRIC_STYLES["latency"][0],
edgecolor=METRIC_STYLES["latency"][1],
label=f"Single-inference {single_metric}"),
Patch(facecolor=METRIC_STYLES["throughput"][0],
edgecolor=METRIC_STYLES["throughput"][1], hatch="//",
label=f"Best-case throughput{throughput_suffix}")]
metric_suffix = "" if field == "speedup" else f" {field}"
panel_labels = {mode: label + metric_suffix for mode, label in COMPACT_MODE_NAMES.items()}
handles = [Patch(facecolor=fill, edgecolor=border, hatch=pattern, label=ARCH_NAMES[arch])
for arch, (fill, border, pattern) in ARCH_STYLES.items()]
for ax, arch in zip(axes, ARCHITECTURES):
selected = [row for row in results if row.arch == arch]
panels = ("latency", "throughput") if compact else ARCHITECTURES
for panel_index, (ax, panel) in enumerate(zip(axes, panels)):
selected = [row for row in results if getattr(row, "mode" if compact else "arch") == panel]
if compact:
selected.sort(key=lambda row: (MODELS.index(row.model), ARCHITECTURES.index(row.arch)))
positions = [index + index // 2 for index in range(len(selected))] if compact else grouped_positions(selected)
ax.set_xlabel(ARCH_NAMES[arch],
ax.set_xlabel(panel_labels[panel] if compact else ARCH_NAMES[panel],
fontsize=11, weight="bold", labelpad=8)
ax.set_axisbelow(True)
ax.set_ylim(y_min, y_max)
ax.set_xlim(-0.9, positions[-1] + 0.9 if positions else 0.9)
label_groups(ax, selected, positions)
ax.tick_params(axis="y", labelleft=arch == ARCHITECTURES[0], pad=5, length=0)
ax.tick_params(axis="y", labelleft=panel_index == 0, pad=5, length=0)
ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6)
ax.yaxis.set_major_locator(MultipleLocator(y_interval))
if diverging:
ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}"))
else:
ax.set_yticks([y_min], minor=True)
ax.yaxis.set_minor_formatter(FuncFormatter(lambda value, _: f"{value:g}"))
ax.tick_params(axis="y", which="minor", labelleft=panel_index == 0, pad=5, length=0)
ax.grid(axis="y")
ax.axhline(0 if diverging else 1, color=TEXT_COLOR, linewidth=1.0,
linestyle="--", zorder=3)
if arch == ARCHITECTURES[0]:
if panel_index == 0:
ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6)
for position, row in zip(positions, selected):
ratio = getattr(row, field)
if not math.isfinite(ratio):
ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
ax.text(position, (0 if diverging else y_min) + (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom",
fontsize=9.5, color=TEXT_COLOR, fontstyle="italic")
continue
compiler = advantage_compiler(ratio)
@@ -377,8 +519,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
pattern = STAGE_PATTERNS[row.pipeline]
color, border = COMPILER_STYLES[compiler] if compiler else ("white", TEXT_COLOR)
if compact:
pattern = "//" if row.mode == "throughput" else ""
color, border = METRIC_STYLES[row.mode]
color, border, pattern = ARCH_STYLES[row.arch]
measurement = diverging_ratio(ratio) if diverging else ratio
ax.bar(position, measurement, width=1.0,
color=color,
@@ -395,7 +536,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str,
else:
figure_legend(fig, results, field)
fit_x_tick_labels(fig)
fit_value_labels(fig)
fit_value_labels(fig, lower_limit=None if diverging else y_min)
save_figure(fig, basename, output_dir)
@@ -463,7 +604,7 @@ def draw_absolute(results: list[Result], field: str, ylabel: str,
save_figure(fig, basename, output_dir)
def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
def draw_pipeline_effect(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None:
rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1]
stages = sorted({row.pipeline for row in rows})
panels = [model for model in MODELS if any(row.model == model for row in rows)]
@@ -479,15 +620,12 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
ax = axes[index, column]
keys = [(arch, model, stage) for stage in stages]
values = [rates.get(key, math.nan) for key in keys]
valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys]
ax.plot(stages, values, color=border, linewidth=1.8)
for stage, value, passed in zip(stages, values, valid):
for stage, value in zip(stages, values):
if math.isfinite(value):
marker = ("s" if stage == 1 else "o") if passed else "x"
status = lookup[(arch, model, stage)].raptor_validation
ax.plot(stage, value, color=ERROR_COLOR if status == "FAIL" else border,
markerfacecolor=fill, marker=marker,
markersize=6, linestyle="none")
ax.plot(stage, value, color=border,
markerfacecolor=fill if marker_borders else border, marker="s" if stage == 1 else "o",
markeredgewidth=1 if marker_borders else 0, markersize=7, linestyle="none")
ax.set_xticks(stages)
if index == 0:
ax.set_title(MODEL_NAMES[model], fontsize=10.5, weight="normal", pad=8)
@@ -500,15 +638,13 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None:
ax.set_ylabel(ARCH_NAMES[arch], fontsize=10.5, weight="bold", rotation=270, labelpad=24)
for ax in axes.flat:
ax.margins(y=0.08)
handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white",
marker=marker, linestyle="none", label=label)
handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white" if marker_borders else TEXT_COLOR,
markeredgewidth=1 if marker_borders else 0,
marker=marker, markersize=7, linestyle="none", label=label)
for marker, label in (("s", "Derived as 1 / latency"),
("o", "Throughput measurement"))]
if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()):
handles.append(Line2D([], [], color=TEXT_COLOR, marker="x", linestyle="none",
label="Raptor validation not PASS"))
add_legend(fig, handles)
fig.supxlabel("Raptor pipeline stages", fontsize=11)
fig.supxlabel(f"{COMPILER_NAMES['raptor']} pipeline stages", fontsize=11)
fit_x_tick_labels(fig)
save_figure(fig, "raptor_pipeline_throughput", output_dir)
@@ -517,16 +653,22 @@ def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
script_dir = Path(__file__).resolve().parent
parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv")
parser.add_argument("--output-dir", type=Path, default=script_dir / "plots")
parser.add_argument("--output-dir", type=Path, default=script_dir / "plots/default")
parser.add_argument("--colors", action=argparse.BooleanOptionalAction, default=True,
help="Use the pastel palette (default); --no-colors exports grayscale charts")
parser.add_argument("--anonymize", action=argparse.BooleanOptionalAction, default=False,
help="Replace the displayed Raptor name with Anonimized")
parser.add_argument("--marker-borders", action=argparse.BooleanOptionalAction, default=True,
help="Outline markers (default); --no-marker-borders uses solid markers in dot and line charts")
args = parser.parse_args()
configure_style()
configure_style(colors=args.colors, anonymize=args.anonymize)
results = load_results(args.input)
valid = sum(row.comparison_passed for row in results)
print(f"Results: {len(results)} configurations, {valid} valid comparisons, "
f"{len(results)-valid} comparisons without two PASS statuses")
draw_validation(results, args.output_dir)
draw_validation(results, args.output_dir, compact=True)
draw_pipeline_effect(results, args.output_dir)
draw_pipeline_effect(results, args.output_dir, marker_borders=args.marker_borders)
yolo_only_models = {
model for model in MODELS if model.startswith("yolo")
and not any(math.isfinite(getattr(row, f"pimcomp_{field}"))
@@ -536,9 +678,10 @@ def main() -> None:
comparisons = [row for row in results if row.model not in yolo_only_models]
yolo_only = [row for row in results if row.model in yolo_only_models]
compact_speedups = select_compact_speedups(comparisons)
draw_compact_summary(comparisons, args.output_dir, marker_borders=args.marker_borders)
for metric, label in (("speedup", "Speedup (x)"),
("energy", "Energy-per-inference advantage (x)"),
("power", "Average power advantage (x)")):
("energy", "Energy reduction (x)"),
("power", "Average power reduction (x)")):
for suffix, diverging in (("comparison", False), ("diverging", True)):
draw_metric(compact_speedups, metric, label,
f"{metric}_compact_{suffix}", args.output_dir, diverging=diverging, compact=True)
@@ -546,8 +689,8 @@ def main() -> None:
("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None),
("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"),
("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"),
("power", "power", "Average power advantage (x)", None),
("energy", "energy", "Energy-per-inference advantage (x)", None),
("power", "power", "Average power reduction (x)", None),
("energy", "energy", "Energy reduction (x)", None),
]:
selected = [row for row in comparisons if mode is None or row.mode == mode]
draw_metric(selected, metric, label,
@@ -1,41 +1,41 @@
arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,429.31,482.60,407972136.20,541691691.00,raptor,1.14
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.78,1.04,1280.00,965.00,457.04,482.60,415761605.20,541691691.00,raptor,1.33
arch-a,vgg8,throughput,8,batch,PASS,PASS,0.78,1.04,1290.00,965.00,464.07,482.60,396424007.70,541691691.00,raptor,1.34
arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394478521.12,13868866808.12,raptor,4.15
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.21,13.46,52.10,74.30,380.10,481.60,7957033431.00,6563451004.00,pimcomp,1.43
arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.93,13.46,50.20,74.30,361.80,481.60,7763795670.00,6563451004.00,pimcomp,1.48
arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.27,23.58,43.00,42.40,471.72,515.68,13055037600.00,12283627410.00,raptor,1.01
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.47,23.58,36.40,42.40,422.22,515.68,14899055210.00,12283627410.00,pimcomp,1.16
arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.84,23.58,25.10,42.40,352.28,515.68,17062658890.00,12283627410.00,pimcomp,1.69
arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.87,231.19,5513262106.92,14547744283.24,raptor,6.44
arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.42,18.26,119.00,54.80,610.75,435.11,5777456331.00,8164787803.00,raptor,2.17
arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.73,18.26,93.20,54.80,528.03,435.11,6592892213.00,8164787803.00,raptor,1.70
arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.12,18.26,98.80,54.80,523.81,435.11,6521737843.00,8164787803.00,raptor,1.80
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,102.34,NA,9.77,NA,337.88,NA,38308060150.00,NA,NA,NA
arch-a,yolo11n,throughput,4,batch,PASS,FAIL,87.91,NA,11.40,NA,358.20,NA,38729846260.00,NA,NA,NA
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,83.68,NA,11.90,NA,365.64,NA,38623589740.00,NA,NA,NA
arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
arch-b,vgg8,throughput,2,batch,PASS,PASS,0.69,1.07,1460.00,932.00,456.04,409.47,403694238.30,439288835.80,raptor,1.57
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.67,1.07,1490.00,932.00,448.90,409.47,413526781.30,439288835.80,raptor,1.60
arch-b,vgg8,throughput,8,batch,PASS,PASS,0.75,1.07,1330.00,932.00,425.74,409.47,394457860.80,439288835.80,raptor,1.43
arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.12,13.43,82.50,74.40,461.41,484.80,6295321671.00,6569655551.00,raptor,1.11
arch-b,resnet18,throughput,4,batch,PASS,FAIL,12.68,13.43,78.90,74.40,451.19,484.80,7096979122.00,6569655551.00,raptor,1.06
arch-b,resnet18,throughput,8,batch,PASS,FAIL,20.59,13.43,48.60,74.40,321.63,484.80,7162570047.00,6569655551.00,pimcomp,1.53
arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.09,22.85,34.40,43.80,364.96,493.33,16328513230.00,11392215660.00,pimcomp,1.27
arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.85,21.00,43.80,289.82,493.33,16346419570.00,11392215660.00,pimcomp,2.09
arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422781257.92,9182062336.24,raptor,3.38
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21,21.39,109.00,46.80,545.49,417.87,5593397699.00,9014726090.00,raptor,2.33
arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56,21.39,117.00,46.80,555.26,417.87,5863121048.00,9014726090.00,raptor,2.50
arch-b,googlenet,throughput,8,batch,PASS,FAIL,16.19,21.39,61.80,46.80,355.70,417.87,6726458739.00,9014726090.00,raptor,1.32
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,104.49,NA,9.57,NA,302.12,NA,35801517850.00,NA,NA,NA
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,108.25,NA,9.24,NA,297.48,NA,37629699620.00,NA,NA,NA
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,100.13,NA,9.99,NA,308.83,NA,37180785770.00,NA,NA,NA
arch-a,vgg8,latency,1,element,PASS,PASS,1.51833,7.98507,NA,NA,321.046,200.111,4.87455e+08,1.5979e+09,raptor,5.26
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.906321,1.0364,1100,965,429.311,482.597,4.07972e+08,5.41692e+08,raptor,1.14
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.778242,1.0364,1280,965,457.039,482.597,4.15762e+08,5.41692e+08,raptor,1.33
arch-a,vgg8,throughput,8,batch,PASS,PASS,0.778094,1.0364,1290,965,464.071,482.597,3.96424e+08,5.41692e+08,raptor,1.34
arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,raptor,4.15
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.7314,13.4585,85.2,74.3,503.111,481.604,6.62807e+09,6.56345e+09,raptor,1.15
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.2072,13.4585,52.1,74.3,380.1,481.604,7.95703e+09,6.56345e+09,pimcomp,1.43
arch-a,resnet18,throughput,8,batch,PASS,FAIL,14.1578,13.4585,70.6,74.3,453.416,481.604,6.77909e+09,6.56345e+09,pimcomp,1.05
arch-a,resnet34,latency,1,element,PASS,FAIL,29.7186,66.7114,NA,NA,411.806,277.485,1.22383e+10,1.85114e+10,raptor,2.24
arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.2705,23.5817,43,42.4,471.717,515.68,1.3055e+10,1.22836e+10,raptor,1.01
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.468,23.5817,36.4,42.4,422.223,515.68,1.48991e+10,1.22836e+10,pimcomp,1.16
arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.8392,23.5817,25.1,42.4,352.28,515.68,1.70627e+10,1.22836e+10,pimcomp,1.69
arch-a,googlenet,latency,1,element,PASS,PASS,9.77772,62.9248,NA,NA,563.864,231.193,5.5133e+09,1.45477e+10,raptor,6.44
arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.41545,18.2643,119,54.8,610.746,435.107,5.77746e+09,8.16479e+09,raptor,2.17
arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.7292,18.2643,93.2,54.8,528.026,435.107,6.59289e+09,8.16479e+09,raptor,1.70
arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.1219,18.2643,98.8,54.8,523.808,435.107,6.52174e+09,8.16479e+09,raptor,1.80
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.174,NA,4.27977e+10,NA,NA,NA
arch-a,yolo11n,throughput,2,batch,SKIPPED,FAIL,102.337,NA,9.77,NA,337.882,NA,3.83081e+10,NA,NA,NA
arch-a,yolo11n,throughput,4,batch,SKIPPED,FAIL,87.9103,NA,11.4,NA,358.201,NA,3.87298e+10,NA,NA,NA
arch-a,yolo11n,throughput,8,batch,SKIPPED,FAIL,82.9508,NA,12.1,NA,366.725,NA,3.85347e+10,NA,NA,NA
arch-b,vgg8,latency,1,element,PASS,PASS,1.5333,7.15212,NA,NA,292.559,173.769,4.48582e+08,1.24281e+09,raptor,4.66
arch-b,vgg8,throughput,2,batch,PASS,PASS,0.685556,1.07282,1460,932,456.044,409.47,4.03694e+08,4.39289e+08,raptor,1.57
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.673192,1.07282,1490,932,448.898,409.47,4.13527e+08,4.39289e+08,raptor,1.60
arch-b,vgg8,throughput,8,batch,PASS,PASS,0.751106,1.07282,1330,932,425.742,409.47,3.94458e+08,4.39289e+08,raptor,1.43
arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,raptor,4.19
arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.1214,13.434,82.5,74.4,461.407,484.804,6.29532e+09,6.56966e+09,raptor,1.11
arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3601,13.434,88,74.4,479.333,484.804,6.98745e+09,6.56966e+09,raptor,1.18
arch-b,resnet18,throughput,8,batch,PASS,FAIL,13.1216,13.434,76.2,74.4,441.248,484.804,6.38271e+09,6.56966e+09,raptor,1.02
arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,raptor,3.83
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.282,22.8454,44.9,43.8,438.238,493.332,1.42187e+10,1.13922e+10,raptor,1.03
arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.0906,22.8454,34.4,43.8,364.956,493.332,1.63285e+10,1.13922e+10,pimcomp,1.27
arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.8454,21,43.8,289.821,493.332,1.63464e+10,1.13922e+10,pimcomp,2.09
arch-b,googlenet,latency,1,element,PASS,PASS,11.2212,37.9367,NA,NA,483.266,242.037,5.42282e+09,9.18206e+09,raptor,3.38
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21209,21.3902,109,46.8,545.488,417.866,5.5934e+09,9.01473e+09,raptor,2.33
arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56194,21.3902,117,46.8,555.264,417.866,5.86312e+09,9.01473e+09,raptor,2.50
arch-b,googlenet,throughput,8,batch,PASS,FAIL,10.8652,21.3902,92,46.8,483.477,417.866,6.12683e+09,9.01473e+09,raptor,1.97
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.144,NA,NA,NA,245.314,NA,4.1248e+10,NA,NA,NA
arch-b,yolo11n,throughput,2,batch,SKIPPED,FAIL,104.494,NA,9.57,NA,302.121,NA,3.58015e+10,NA,NA,NA
arch-b,yolo11n,throughput,4,batch,SKIPPED,FAIL,108.247,NA,9.24,NA,297.484,NA,3.76297e+10,NA,NA,NA
arch-b,yolo11n,throughput,8,batch,SKIPPED,FAIL,100.129,NA,9.99,NA,308.833,NA,3.71808e+10,NA,NA,NA
1 arch model mode raptor_pipeline pimcomp_pipeline raptor_functional_validation pimcomp_functional_validation raptor_latency_ms pimcomp_latency_ms raptor_throughput_samples_s pimcomp_throughput_samples_s raptor_power_mw pimcomp_power_mw raptor_energy_pj pimcomp_energy_pj better_compiler speedup
2 arch-a vgg8 latency 1 element PASS PASS 1.52 1.51833 7.99 7.98507 NA NA 321.05 321.046 200.11 200.111 487455139.04 4.87455e+08 1597904071.12 1.5979e+09 raptor 5.26
3 arch-a vgg8 throughput 2 batch PASS PASS 0.91 0.906321 1.04 1.0364 1100.00 1100 965.00 965 429.31 429.311 482.60 482.597 407972136.20 4.07972e+08 541691691.00 5.41692e+08 raptor 1.14
4 arch-a vgg8 throughput 4 batch PASS PASS 0.78 0.778242 1.04 1.0364 1280.00 1280 965.00 965 457.04 457.039 482.60 482.597 415761605.20 4.15762e+08 541691691.00 5.41692e+08 raptor 1.33
5 arch-a vgg8 throughput 8 batch PASS PASS 0.78 0.778094 1.04 1.0364 1290.00 1290 965.00 965 464.07 464.071 482.60 482.597 396424007.70 3.96424e+08 541691691.00 5.41692e+08 raptor 1.34
6 arch-a resnet18 latency 1 element PASS FAIL 14.03 14.0332 58.18 58.1775 NA NA 455.67 238.39 238.389 6394478521.12 6.3945e+09 13868866808.12 1.38689e+10 raptor 4.15
7 arch-a resnet18 throughput 2 batch PASS FAIL 11.73 11.7314 13.46 13.4585 85.20 85.2 74.30 74.3 503.11 503.111 481.60 481.604 6628073894.00 6.62807e+09 6563451004.00 6.56345e+09 raptor 1.15
8 arch-a resnet18 throughput 4 batch PASS FAIL 19.21 19.2072 13.46 13.4585 52.10 52.1 74.30 74.3 380.10 380.1 481.60 481.604 7957033431.00 7.95703e+09 6563451004.00 6.56345e+09 pimcomp 1.43
9 arch-a resnet18 throughput 8 batch PASS FAIL 19.93 14.1578 13.46 13.4585 50.20 70.6 74.30 74.3 361.80 453.416 481.60 481.604 7763795670.00 6.77909e+09 6563451004.00 6.56345e+09 pimcomp 1.48 1.05
10 arch-a resnet34 latency 1 element PASS FAIL 29.72 29.7186 66.71 66.7114 NA NA 411.81 411.806 277.49 277.485 12238321706.68 1.22383e+10 18511425563.68 1.85114e+10 raptor 2.24
11 arch-a resnet34 throughput 2 batch PASS FAIL 23.27 23.2705 23.58 23.5817 43.00 43 42.40 42.4 471.72 471.717 515.68 13055037600.00 1.3055e+10 12283627410.00 1.22836e+10 raptor 1.01
12 arch-a resnet34 throughput 4 batch PASS FAIL 27.47 27.468 23.58 23.5817 36.40 36.4 42.40 42.4 422.22 422.223 515.68 14899055210.00 1.48991e+10 12283627410.00 1.22836e+10 pimcomp 1.16
13 arch-a resnet34 throughput 8 batch PASS FAIL 39.84 39.8392 23.58 23.5817 25.10 25.1 42.40 42.4 352.28 515.68 17062658890.00 1.70627e+10 12283627410.00 1.22836e+10 pimcomp 1.69
14 arch-a googlenet latency 1 element PASS PASS 9.78 9.77772 62.92 62.9248 NA NA 563.87 563.864 231.19 231.193 5513262106.92 5.5133e+09 14547744283.24 1.45477e+10 raptor 6.44
15 arch-a googlenet throughput 2 batch PASS FAIL 8.42 8.41545 18.26 18.2643 119.00 119 54.80 54.8 610.75 610.746 435.11 435.107 5777456331.00 5.77746e+09 8164787803.00 8.16479e+09 raptor 2.17
16 arch-a googlenet throughput 4 batch PASS FAIL 10.73 10.7292 18.26 18.2643 93.20 93.2 54.80 54.8 528.03 528.026 435.11 435.107 6592892213.00 6.59289e+09 8164787803.00 8.16479e+09 raptor 1.70
17 arch-a googlenet throughput 8 batch PASS FAIL 10.12 10.1219 18.26 18.2643 98.80 98.8 54.80 54.8 523.81 523.808 435.11 435.107 6521737843.00 6.52174e+09 8164787803.00 8.16479e+09 raptor 1.80
18 arch-a yolo11n latency 1 element PASS FAIL 147.49 NA NA NA 290.17 290.174 NA 42797722129.00 4.27977e+10 NA NA NA
19 arch-a yolo11n throughput 2 batch PASS SKIPPED FAIL 102.34 102.337 NA 9.77 NA 337.88 337.882 NA 38308060150.00 3.83081e+10 NA NA NA
20 arch-a yolo11n throughput 4 batch PASS SKIPPED FAIL 87.91 87.9103 NA 11.40 11.4 NA 358.20 358.201 NA 38729846260.00 3.87298e+10 NA NA NA
21 arch-a yolo11n throughput 8 batch PASS SKIPPED FAIL 83.68 82.9508 NA 11.90 12.1 NA 365.64 366.725 NA 38623589740.00 3.85347e+10 NA NA NA
22 arch-b vgg8 latency 1 element PASS PASS 1.53 1.5333 7.15 7.15212 NA NA 292.56 292.559 173.77 173.769 448581679.04 4.48582e+08 1242814988.12 1.24281e+09 raptor 4.66
23 arch-b vgg8 throughput 2 batch PASS PASS 0.69 0.685556 1.07 1.07282 1460.00 1460 932.00 932 456.04 456.044 409.47 403694238.30 4.03694e+08 439288835.80 4.39289e+08 raptor 1.57
24 arch-b vgg8 throughput 4 batch PASS PASS 0.67 0.673192 1.07 1.07282 1490.00 1490 932.00 932 448.90 448.898 409.47 413526781.30 4.13527e+08 439288835.80 4.39289e+08 raptor 1.60
25 arch-b vgg8 throughput 8 batch PASS PASS 0.75 0.751106 1.07 1.07282 1330.00 1330 932.00 932 425.74 425.742 409.47 394457860.80 3.94458e+08 439288835.80 4.39289e+08 raptor 1.43
26 arch-b resnet18 latency 1 element PASS FAIL 15.38 15.3758 64.38 64.3818 NA NA 400.67 400.661 201.66 201.663 6160441550.12 6.16048e+09 12983432281.12 1.29834e+10 raptor 4.19
27 arch-b resnet18 throughput 2 batch PASS FAIL 12.12 12.1214 13.43 13.434 82.50 82.5 74.40 74.4 461.41 461.407 484.80 484.804 6295321671.00 6.29532e+09 6569655551.00 6.56966e+09 raptor 1.11
28 arch-b resnet18 throughput 4 batch PASS FAIL 12.68 11.3601 13.43 13.434 78.90 88 74.40 74.4 451.19 479.333 484.80 484.804 7096979122.00 6.98745e+09 6569655551.00 6.56966e+09 raptor 1.06 1.18
29 arch-b resnet18 throughput 8 batch PASS FAIL 20.59 13.1216 13.43 13.434 48.60 76.2 74.40 74.4 321.63 441.248 484.80 484.804 7162570047.00 6.38271e+09 6569655551.00 6.56966e+09 pimcomp raptor 1.53 1.02
30 arch-b resnet34 latency 1 element PASS FAIL 27.95 27.9541 107.20 107.2 NA NA 397.21 397.206 206.48 206.476 11103517952.68 1.11035e+10 22134174654.68 2.21342e+10 raptor 3.83
31 arch-b resnet34 throughput 2 batch PASS FAIL 22.28 22.282 22.85 22.8454 44.90 44.9 43.80 43.8 438.24 438.238 493.33 493.332 14218683390.00 1.42187e+10 11392215660.00 1.13922e+10 raptor 1.03
32 arch-b resnet34 throughput 4 batch PASS FAIL 29.09 29.0906 22.85 22.8454 34.40 34.4 43.80 43.8 364.96 364.956 493.33 493.332 16328513230.00 1.63285e+10 11392215660.00 1.13922e+10 pimcomp 1.27
33 arch-b resnet34 throughput 8 batch PASS FAIL 47.68 22.85 22.8454 21.00 21 43.80 43.8 289.82 289.821 493.33 493.332 16346419570.00 1.63464e+10 11392215660.00 1.13922e+10 pimcomp 2.09
34 arch-b googlenet latency 1 element PASS PASS 11.22 11.2212 37.94 37.9367 NA NA 483.27 483.266 242.04 242.037 5422781257.92 5.42282e+09 9182062336.24 9.18206e+09 raptor 3.38
35 arch-b googlenet throughput 2 batch PASS FAIL 9.21 9.21209 21.39 21.3902 109.00 109 46.80 46.8 545.49 545.488 417.87 417.866 5593397699.00 5.5934e+09 9014726090.00 9.01473e+09 raptor 2.33
36 arch-b googlenet throughput 4 batch PASS FAIL 8.56 8.56194 21.39 21.3902 117.00 117 46.80 46.8 555.26 555.264 417.87 417.866 5863121048.00 5.86312e+09 9014726090.00 9.01473e+09 raptor 2.50
37 arch-b googlenet throughput 8 batch PASS FAIL 16.19 10.8652 21.39 21.3902 61.80 92 46.80 46.8 355.70 483.477 417.87 417.866 6726458739.00 6.12683e+09 9014726090.00 9.01473e+09 raptor 1.32 1.97
38 arch-b yolo11n latency 1 element PASS FAIL 168.14 168.144 NA NA NA 245.31 245.314 NA 41248047514.00 4.1248e+10 NA NA NA
39 arch-b yolo11n throughput 2 batch PASS SKIPPED FAIL 104.49 104.494 NA 9.57 NA 302.12 302.121 NA 35801517850.00 3.58015e+10 NA NA NA
40 arch-b yolo11n throughput 4 batch PASS SKIPPED FAIL 108.25 108.247 NA 9.24 NA 297.48 297.484 NA 37629699620.00 3.76297e+10 NA NA NA
41 arch-b yolo11n throughput 8 batch PASS SKIPPED FAIL 100.13 100.129 NA 9.99 NA 308.83 308.833 NA 37180785770.00 3.71808e+10 NA NA NA
@@ -23,13 +23,13 @@ REPO = Path(__file__).resolve().parents[5]
sys.path.insert(0, str(REPO / "validation"))
from raptor_validation.pimcomp_models import (
FUNCTIONAL_MODELS,
MODELS,
SUITE,
add_models_argument,
)
from raptor_validation.artifacts import artifacts_dir, remove_lock_files
from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics
from raptor_validation.output_comparison import VALIDATION_POLICY
from raptor_validation.validate_one import STAGE_COLORS
PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
@@ -63,7 +63,7 @@ class ComparisonSpec:
def model_dir(root: Path | None, name: str) -> Path:
return root / name if root is not None else MODELS[name].parent
return (root or SUITE) / name
def result_dir(
@@ -83,9 +83,9 @@ def result_dir(
def common_dir(root: Path | None, name: str, common_root: Path | None = None) -> Path:
suffix = "common" if FUNCTIONAL_MODELS[name] == MODELS[name] else "common-functional"
base = common_root / name if common_root is not None else model_dir(root, name)
return artifacts_dir(base) / suffix
# Keep references that retain Softmax separate from historical outputs.
return artifacts_dir(base) / ("common-full-model" if name in ("yolo11n", "googlenet") else "common")
def clean_artifacts(
@@ -199,6 +199,8 @@ def write_results_csv(
}
if report_path.exists():
report = json.loads(report_path.read_text(encoding="utf-8"))
if report.get("validation_policy") != VALIDATION_POLICY:
raise ValueError(f"Report uses a different validation policy: {report_path}. Regenerate this comparison.")
raptor_values = performance_values(report.get("raptor_performance") or {})
pimcomp_values = performance_values(report.get("pimcomp_performance") or {})
raptor_metric = raptor_values["throughput"] if comparison_mode == "throughput" else raptor_values["latency"]
@@ -222,7 +224,11 @@ def write_results_csv(
if raptor_metric is not None and pimcomp_metric is not None:
min_metric = min(raptor_metric, pimcomp_metric)
row.update(
better_compiler=comparison_winner(comparison_mode, raptor_metric, pimcomp_metric),
better_compiler=(
comparison_winner(comparison_mode, raptor_metric, pimcomp_metric)
if row["raptor_functional_validation"] == row["pimcomp_functional_validation"] == "PASS"
else "NA"
),
speedup=(
f"{max(raptor_metric, pimcomp_metric) / min_metric:.2f}"
if min_metric > 0
@@ -684,10 +690,9 @@ def main() -> int:
)
missing = [
str(path)
str(MODELS[name])
for name in args.models
for path in (MODELS[name], FUNCTIONAL_MODELS[name])
if not path.exists()
if not MODELS[name].exists()
]
if missing:
parser.error(f"missing model(s): {', '.join(missing)}")
@@ -733,7 +738,7 @@ def main() -> int:
try:
run(
prepare_common_command(
FUNCTIONAL_MODELS[name],
MODELS[name],
common_dir(out_dir, name, common_root),
args.timeout_seconds,
reference_batch_size,
@@ -770,7 +775,7 @@ def main() -> int:
ComparisonSpec(
label=label,
model=MODELS[name],
functional_model=FUNCTIONAL_MODELS[name],
functional_model=MODELS[name],
output_dir=model_result_dir,
common_dir=common_dir(out_dir, name, common_root),
config=configs[mode],
@@ -49,6 +49,13 @@ loaded provider tensors have no store producer; prefill reduces maximum
reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's
verifier contract; it does not repair the standalone batch program.
The reconstructed VGG8 batch program also loads intermediate Pool and Flatten
buffers without corresponding store producers. Its output can remain zero.
The small reference output can fit inside an absolute tolerance despite this
missing computation. Comparison therefore requires a relative L2 check in
addition to the elementwise check. A zero output against a nonzero reference
has relative L2 error one and must fail the default relative tolerance.
## ResNet BatchNorm gap
Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535)
+64 -16
View File
@@ -6,10 +6,14 @@ from math import ceil
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.colors import to_rgba
from matplotlib.legend_handler import HandlerBase
from matplotlib.lines import Line2D
from matplotlib.patches import Patch
from matplotlib.path import Path as PlotPath
from matplotlib.patheffects import AbstractPathEffect
from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath
from matplotlib.text import Text
from matplotlib.transforms import Affine2D, Bbox, IdentityTransform, TransformedPath
PALETTE = {
@@ -25,11 +29,12 @@ PALETTE = {
TEXT_COLOR = "#222222"
MATRIX_TEXT_COLOR = "#1D2530"
ERROR_COLOR = PALETTE["red"][1]
DAGGER_COLOR = "#AA1C1C"
GRID_COLOR = "#E6E9EC"
LINE_COLOR = "#C9CED3"
BAND_COLOR = "#F7F9FA"
COMPILER_STYLES = {"raptor": PALETTE["sage"], "pimcomp": PALETTE["sand"]}
METRIC_STYLES = {"latency": PALETTE["blue"], "throughput": PALETTE["sage"]}
ARCH_STYLES = {"arch-a": (*PALETTE["blue"], ""), "arch-b": (*PALETTE["sage"], "//")}
NETWORK_STYLES = {"vgg8": PALETTE["blue"], "resnet18": PALETTE["sage"],
"resnet34": PALETTE["sand"], "googlenet": PALETTE["purple"],
"yolo11n": PALETTE["red"]}
@@ -38,6 +43,8 @@ ARCH_NAMES = {"arch-a": "Arch-A", "arch-b": "Arch-B"}
MODEL_NAMES = {"vgg8": "VGG-8", "resnet18": "ResNet-18", "resnet34": "ResNet-34",
"googlenet": "GoogLeNet", "yolo11n": "YOLO11n"}
MODE_NAMES = {"latency": "Single-inference", "throughput": "Throughput"}
COMPACT_MODE_NAMES = {"latency": "Latency-mode", "throughput": "Throughput-mode"}
_USE_COLORS = True
class CenteredHatch(AbstractPathEffect):
@@ -47,11 +54,14 @@ class CenteredHatch(AbstractPathEffect):
hatch = gc.get_hatch() or ""
base_gc = renderer.new_gc()
base_gc.copy_properties(gc)
base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\"))
base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\") or None)
renderer.draw_path(base_gc, tpath, affine, rgbFace)
base_gc.restore()
bounds = affine.transform_path(tpath).get_extents()
clip_bounds = bounds if gc.get_clip_rectangle() is None else Bbox.intersection(bounds, gc.get_clip_rectangle())
if clip_bounds is None:
return
center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2
vertices = []
diagonal = 2 ** -0.5
@@ -61,13 +71,16 @@ class CenteredHatch(AbstractPathEffect):
continue
spacing = renderer.points_to_pixels(12 / count)
half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2
half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2
extent = ceil(half_span / spacing)
for step in range(-extent, extent + 1):
x = center_x - dy * step * spacing
y = center_y + dx * step * spacing
vertices.extend(((x - dx * half_length, y - dy * half_length),
(x + dx * half_length, y + dy * half_length)))
y_limits = sorted(((clip_bounds.y0 - y) / dy, (clip_bounds.y1 - y) / dy))
start = max((clip_bounds.x0 - x) / dx, y_limits[0])
end = min((clip_bounds.x1 - x) / dx, y_limits[1])
if start < end:
vertices.extend(((x + dx * start, y + dy * start),
(x + dx * end, y + dy * end)))
line_gc = renderer.new_gc()
line_gc.copy_properties(gc)
line_gc.set_hatch(None)
@@ -81,17 +94,22 @@ class CenteredHatch(AbstractPathEffect):
spacing = renderer.points_to_pixels(12 / hatch.count("."))
x_extent = ceil(bounds.width / (2 * spacing))
y_extent = ceil(bounds.height / (2 * spacing))
centers = [(center_x + x * spacing, center_y + y * spacing)
for x in range(-x_extent, x_extent + 1)
for y in range(-y_extent, y_extent + 1)]
# Trim geometry too: SVG importers can discard clip paths.
dots = [PlotPath.unit_circle().transformed(
Affine2D().scale(renderer.points_to_pixels(0.9)).translate(
center_x + x * spacing, center_y + y * spacing)).clip_to_bbox(clip_bounds)
for x in range(-x_extent, x_extent + 1)
for y in range(-y_extent, y_extent + 1)]
line_gc.set_linewidth(0)
renderer.draw_markers(line_gc, PlotPath.unit_circle(),
Affine2D().scale(renderer.points_to_pixels(0.9)),
PlotPath(centers), IdentityTransform(), gc.get_hatch_color())
renderer.draw_path(line_gc, PlotPath.make_compound_path(*dots),
IdentityTransform(), gc.get_hatch_color())
line_gc.restore()
def configure_style() -> None:
def configure_style(*, colors: bool = True, anonymize: bool = False) -> None:
global _USE_COLORS
_USE_COLORS = colors
COMPILER_NAMES["raptor"] = "Anonimized" if anonymize else "Raptor"
plt.rcParams.update({
"font.family": "DejaVu Sans",
"axes.spines.top": False,
@@ -109,6 +127,14 @@ def configure_style() -> None:
})
class TextLegendHandler(HandlerBase):
def create_artists(self, legend, original, xdescent, ydescent, width, height, fontsize, transform):
return [Text(width / 2 - xdescent, height / 2 - ydescent, original.get_text(),
color=original.get_color(), fontproperties=original.get_fontproperties(),
math_fontfamily=original.get_math_fontfamily(), ha="center", va="center",
transform=transform)]
def add_legend(fig, handles) -> None:
"""Use the fewest balanced legend rows that fit the figure width."""
renderer = fig.canvas.get_renderer()
@@ -123,6 +149,7 @@ def add_legend(fig, handles) -> None:
ordered = [handle for column in zip(*grid) for handle in column]
legend = fig.legend(handles=[handle if handle is not None else Patch(visible=False) for handle in ordered],
labels=[handle.get_label() if handle is not None else "" for handle in ordered],
handler_map={Text: TextLegendHandler()},
loc="outside upper center", ncols=columns, frameon=False, fontsize=9.5,
handlelength=1.7, columnspacing=1.5, handletextpad=0.6, labelspacing=0.4)
for handle, patch, text in zip(ordered, legend.legend_handles, legend.get_texts()):
@@ -197,7 +224,7 @@ def fit_bar_widths(fig) -> None:
fig.set_size_inches(fig.get_figwidth() * scale, fig.get_figheight())
def fit_value_labels(fig) -> None:
def fit_value_labels(fig, *, lower_limit: float | None = None) -> None:
"""Fit common vertical limits to the data and rendered value labels."""
for ax in fig.axes:
for text in ax.texts:
@@ -215,18 +242,39 @@ def fit_value_labels(fig) -> None:
labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height,
max(0, extent.y1 - anchor + 3) / bounds.height))
limits.append(ax.dataLim.intervaly if ax.has_data() else ax.get_ylim())
lower = min(0, *(lower for lower, _ in limits))
lower = min(0, *(lower for lower, _ in limits)) if lower_limit is None else lower_limit
upper = max(0, *(upper for _, upper in limits))
if lower == upper:
upper = lower + 1
for _ in range(12):
lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
if lower_limit is None:
lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)])
upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)])
for ax in fig.axes:
ax.set_ylim(lower, upper)
def grayscale_figure(fig) -> None:
def gray(color):
red, green, blue, alpha = to_rgba(color)
level = 0.2126 * red + 0.7152 * green + 0.0722 * blue
return level, level, level, alpha
for artist in fig.findobj():
if isinstance(artist, Patch):
artist.set_facecolor(gray(artist.get_facecolor()))
artist.set_edgecolor(gray(artist.get_edgecolor()))
elif isinstance(artist, Line2D):
artist.set_color(gray(artist.get_color()))
artist.set_markerfacecolor(gray(artist.get_markerfacecolor()))
artist.set_markeredgecolor(gray(artist.get_markeredgecolor()))
elif isinstance(artist, Text):
artist.set_color(gray(artist.get_color()))
def save_figure(fig, basename: str, output_dir: Path) -> None:
if not _USE_COLORS:
grayscale_figure(fig)
for patch in fig.findobj(Patch):
hatch = patch.get_hatch() or ""
if any(symbol in hatch for symbol in "./\\"):
+4 -4
View File
@@ -225,7 +225,7 @@ def main():
ap.add_argument("--threshold", type=float, default=1e-3,
help="Absolute tolerance for per-element output comparison.")
ap.add_argument("--relative-threshold", type=float, default=1e-5,
help="Relative tolerance for per-element output comparison.")
help="Relative tolerance for per-element and output L2 comparison; 0 disables the L2 check.")
ap.add_argument("--seed", type=int, default=0, help="RNG seed for generated validation inputs.")
ap.add_argument("--crossbar-size", type=int, default=default_hardware["crossbar_rows"],
help=f"Crossbar size (default: {default_hardware['crossbar_rows']}).")
@@ -416,10 +416,10 @@ def main():
n_passed = sum(1 for result in results.values() if result.passed)
n_total = len(results)
latency_headers = ("Operation", "Arch", "Result", "Compile", "Host mem", "Cores mem",
"Cores", "Xbars", "Latency", "Power", "Energy")
"Cores", "Xbars", "Latency", "Average power", "Energy per inference")
throughput_headers = ("Operation", "Arch", "Result", "Compile", "Host mem", "Cores mem",
"Cores", "Xbars", "Throughput", "Avg latency", "Avg power",
"Avg energy")
"Cores", "Xbars", "Throughput", "Output interval", "Average power",
"Energy per inference")
csv_headers = (
"Operation", "Arch", "Result (l)", "Result (t)",
"Compile (l)", "Host mem (l)", "Cores mem (l)", "Cores (l)", "Xbars (l)",