finally fast googlenet with correct latency artifacts for fair comparison

validation/.gitignore

@@ -13,3 +13,4 @@ networks/**/real_image_val
 networks/**/*.png
 networks/**/*.jpg
 networks/**/*.csv
+!networks/pimcomp_models/results.csv

validation/README.md

@@ -8,7 +8,7 @@ model it can:
 3. compile PIM artifacts with Raptor;
 4. run the reference implementation and functional PIM simulator;
 5. compare their outputs;
-6. run `pimsim-nn` to report latency and power.
+6. run `pimsim-nn` to report latency, power, and energy.
 
 Run the script from the repository root with the repository Python environment.
 
@@ -65,7 +65,7 @@ The script discovers them recursively.
 The PIMCOMP paper-model suite has a one-command Arch-A comparison:
 
 ```bash
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
 ```
 
 The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
@@ -134,7 +134,7 @@ count with `-j` or `--jobs`:
 | `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
 | `--non-functional-simulator-build-dir PATH` | `pimsim-nn` build directory. Defaults to the in-tree build. |
 | `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
-| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency and power measurement. |
+| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency, power, and energy measurement. |
 | `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
 | `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
 | `--seed INT` | Seed for generated inputs. Defaults to `0`. |
@@ -165,17 +165,17 @@ prints the incompatible values; functional validation still runs.
 The checked-in profiles are under
 `validation/pimsim_configs/pimcomp/<profile>/latency_config.json`.
 
-Use `--skip-non-functional-simulation` when latency and power are not required.
+Use `--skip-non-functional-simulation` when latency, power, and energy are not required.
 The summary reports non-functional results as measured, failed, unsupported, or
 skipped.
 
 Overall PASS/FAIL is determined by compilation and functional output
 comparison. A non-functional simulation failure remains visible as `ERROR` in
-the latency and power columns but does not change a functional PASS.
+the latency, power, and energy columns but does not change a functional PASS.
 
 `pimsim-nn` does not currently implement the `vsoftmax` instruction. When its
 explicit unsupported-op diagnostic is encountered, Softmax validations retain
-their functional PASS and show `UNSUPPORTED` in both non-functional columns.
+their functional PASS and show `UNSUPPORTED` in the non-functional columns.
 Other `pimsim-nn` failures remain `ERROR`.
 
 ## Generated artifacts
@@ -186,7 +186,7 @@ Artifacts are written beside each model:
 |---|---|
 | `inputs/` | Generated input CSV files. |
 | `outputs/` | ONNX-MLIR reference output CSV files. |
-| `raptor/` | Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts. |
+| `raptor/` | Exported MLIR, dialect snapshots, reports, final FP32 `pim/` artifacts, and the int8-equivalent `pimsim_nn/` latency view. |
 | `runner/` | Generated reference runner source, build tree, and shared library. |
 | `simulation/` | Functional simulator outputs used for numerical comparison. |
 | `pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |

validation/networks/pimcomp_models/README.md

@@ -4,7 +4,8 @@ This directory contains the four networks evaluated in
 [PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
 VGG-8, ResNet-18, ResNet-34, and GoogLeNet.
 
-See [RESULTS.md](RESULTS.md) for the current latency-only result status.
+See the runner-generated [results.csv](results.csv) for the current latency
+and energy results.
 
 ## Models and provenance
 
@@ -15,11 +16,11 @@ See [RESULTS.md](RESULTS.md) for the current latency-only result status.
 | `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). |
 | `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Deterministic compiler workload with six convolution and two fully connected layers. |
 
-`googlenet/googlenet-12-no-softmax.onnx` is a derived latency model that
-exposes the original model's final FC logits (`loss3/classifier_1`) as its
-output. This matches PIMCOMP's instruction stream, which records but does not
-schedule the terminal `OP_SOFTMAX`. Keep `googlenet-12.onnx` for full-model
-functional validation.
+`googlenet/googlenet-12-latency.onnx` is the explicit common latency model.
+It exposes the original model's final FC logits (`loss3/classifier_1`) and
+removes its two LRN nodes and terminal Softmax so the comparison covers only
+operations scheduled by PIMCOMP. Keep `googlenet-12.onnx` as the unmodified
+source model.
 
 The PIMCOMP authors did not publish the ONNX checkpoints used by the paper.
 Running PIMCOMP's frontend on the three Model Zoo files above produces JSON
@@ -38,7 +39,7 @@ Current SHA-256 checksums:
 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f  resnet18/resnet18-v1-7.onnx
 c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096  resnet34/resnet34-v1-7.onnx
 c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e  googlenet/googlenet-12.onnx
-a35bad96441efbee28699cb61d1656cca7f7281f14040cf01699c3d0cfd8b202  googlenet/googlenet-12-no-softmax.onnx
+a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982  googlenet/googlenet-12-latency.onnx
 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef  vgg8/vgg8-mnist-reconstructed.onnx
 ```
 
@@ -145,17 +146,17 @@ uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.
 ## Compare Raptor and PIMCOMP
 
 The comparison driver uses one random input and one native ONNX-MLIR reference,
-compiles both instruction streams, runs both through `pimsim-nn`, validates
-Raptor through the Rust simulator, and writes Markdown and JSON reports.
-PIMCOMP Rust validation also runs when its optional exporter is available.
+compiles both instruction streams, runs both through `pimsim-nn`, runs
+functional validation through `pim-simulator`, and writes Markdown and JSON
+reports.
 
 To reproduce the complete Arch-A latency experiment, use the model-by-model
 runner. It verifies the paper GA settings, builds Raptor and the existing
 `third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
-for one model at a time:
+for one model at a time and regenerates `results.csv` from the JSON reports:
 
 ```bash
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
 ```
 
 Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
@@ -176,7 +177,7 @@ Arch-A low-latency example:
 ```bash
 RAPTOR_ROOT=$PWD
 
-"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/compare_raptor_pimcomp.py" \
+"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \
   --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
   --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
   --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
@@ -202,12 +203,36 @@ The comparison runner enables `--fail-on-error`, so a failed compiler,
 simulation, or semantic validation makes the command fail while preserving the
 generated report.
 
+### Numeric precision and simulator artifacts
+
+The functional and non-functional simulators intentionally consume different
+artifacts:
+
+- Raptor and PIMCOMP are validated against the native ONNX-MLIR reference as
+  FP32 programs in the Rust simulator. Raptor's emitted program is already
+  FP32. The PIMCOMP-to-Rust export expands its element-addressed storage and
+  byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector
+  `imm_len` fields as element counts.
+- PIMCOMP's original `SimulationInfo.gz` is copied unchanged for `pimsim-nn`.
+  PIMCOMP hardcodes `setbw 8, 8` and one byte per element without performing
+  numerical quantization; this artifact is used only for latency estimation.
+- Raptor's original FP32 artifact remains unchanged for functional validation.
+  A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its
+  byte-addressed storage and transfer sizes from four bytes to one byte per
+  element. Vector `imm_len` fields remain element counts. Like PIMCOMP's
+  artifact, this view is not numerically valid and is used only for a fair
+  non-functional comparison.
+
+The ISA defines vector lengths in elements, while `ld`, `st`, `lldi`, `lmv`,
+`send`, `recv`, addresses, and non-vector offsets are byte-based. Do not use
+either latency-only artifact for semantic validation.
+
 Current Raptor status:
 
 - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
-- Use `googlenet-12-no-softmax.onnx` for the paper-matched latency comparison.
-  The original model's final `vsoftmax` is supported by Raptor's functional
-  simulator but not by `pimsim-nn`; PIMCOMP does not schedule that operation.
+- Use `googlenet-12-latency.onnx` for the paper-matched latency comparison.
+  It removes the two LRN nodes and terminal softmax that PIMCOMP does not
+  schedule.
 - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
   `512x1024` arrays can therefore be compiled by PIMCOMP but not compared
   exactly with Raptor.
@@ -227,10 +252,8 @@ rsync -azL validation/networks/pimcomp_models/ \
   "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
 rsync -az validation/pimsim_configs/pimcomp/ \
   "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
-rsync -az validation/tools/compare_raptor_pimcomp.py \
-  "monolith:$REMOTE_REPO/validation/tools/compare_raptor_pimcomp.py"
-rsync -az validation/tools/run_pimcomp_paper_latency.py \
-  "monolith:$REMOTE_REPO/validation/tools/run_pimcomp_paper_latency.py"
+rsync -az validation/tools/pimcomp/ \
+  "monolith:$REMOTE_REPO/validation/tools/pimcomp/"
 rsync -az --exclude=.git --exclude=build --exclude=output \
   third_party/PIMCOMP-NN/ \
   "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
@@ -247,7 +270,7 @@ python3 -m venv .venv
 .venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
 
 # Run every latency comparison serially.
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
 ```
 
 Copy reports back without transferring large compiler artifacts:

validation/networks/pimcomp_models/RESULTS.md

@@ -1,32 +0,0 @@
-# Raptor vs PIMCOMP latency results
-
-## GoogLeNet, Arch-A, low latency
-
-Measured with `googlenet-12-no-softmax.onnx`, batch 1, Raptor's best current
-schedule, and PIMCOMP's GA/element artifacts. Both instruction streams were
-simulated by the same `pimsim-nn` build using the complete Arch-A timing and
-precision configuration.
-
-| Compiler | Latency (ms) | Instructions | Sends | Receives | MVMUL |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| Raptor | 1132.458315 | 77,717,248 | 29,115 | 29,115 | 157,158 |
-| PIMCOMP | 41.790450 | 3,398,070 | 110,334 | 110,334 | 113,639 |
-
-PIMCOMP is 27.10x faster in this latency simulation.
-
-Semantic validation did not pass the comparison driver's strict default
-tolerance: the maximum logit differences from the native ONNX reference were
-`0.01995039` for Raptor and `7.768404` for PIMCOMP. Treat these as performance
-results, not as a correctness-equivalent comparison.
-
-No throughput experiment was run.
-
-## Reproduce
-
-```bash
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
-  --models googlenet
-```
-
-See [README.md](README.md) for model provenance, limitations, and monolith
-instructions.

validation/networks/pimcomp_models/results.csv

@@ -0,0 +1,5 @@
+model,raptor_latency_ms,pimcomp_latency_ms,raptor_energy_pj,pimcomp_energy_pj,faster_compiler,speedup
+vgg8,2.451664,7.985074,647608825.120001,1597904071.120000,raptor,3.26
+resnet18,113.483357,58.853733,23204566467.119949,13983168748.119974,pimcomp,1.93
+resnet34,120.655292,91.607980,27614808203.679939,22722922369.680016,pimcomp,1.32
+googlenet,22.936609,62.923463,7798109358.239990,14547526780.240000,raptor,2.74

validation/raptor_validation/pimsim_nn.py

@@ -0,0 +1,73 @@
+import json
+import re
+import shutil
+from pathlib import Path
+
+
+_METRIC_PATTERNS = {
+    "output_count": r"output count:\s+([0-9]+)\s+samples",
+    "throughput": r"throughput:\s+([0-9.eE+-]+)\s+samples/s",
+    "average_latency_ms": r"average latency:\s+([0-9.eE+-]+)\s+ms",
+    "latency_ms": r"latency:\s+([0-9.eE+-]+)\s+ms",
+    "average_power_mw": r"average power:\s+([0-9.eE+-]+)\s+mW",
+    "average_energy_pj": r"average energy:\s+([0-9.eE+-]+)\s+pJ(?:/it)?",
+}
+
+
+def parse_pimsim_nn_metrics(output):
+    metrics = {"raw_output": output}
+    for name, pattern in _METRIC_PATTERNS.items():
+        match = re.search(pattern, output)
+        if match:
+            value = match.group(1)
+            metrics[name] = int(value) if name == "output_count" else float(value)
+    return metrics
+
+
+def export_raptor_latency_artifact(pim_dir, output_dir):
+    pim_dir = Path(pim_dir)
+    output_dir = Path(output_dir)
+    if output_dir.exists():
+        shutil.rmtree(output_dir)
+    output_dir.mkdir(parents=True)
+
+    def int8_bytes(value, field):
+        if value % 4:
+            raise ValueError(f"Raptor {field}={value} is not aligned to its fp32 element width")
+        return value // 4
+
+    with open(pim_dir / "config.json", encoding="utf-8") as f:
+        config = json.load(f)
+    for field in ("inputs_addresses", "outputs_addresses"):
+        if field in config:
+            config[field] = [int8_bytes(value, field) for value in config[field]]
+    with open(output_dir / "config.json", "w", encoding="utf-8") as f:
+        json.dump(config, f, separators=(",", ":"))
+        f.write("\n")
+
+    byte_size_fields = {
+        "ld": "size",
+        "st": "size",
+        "lldi": "len",
+        "lmv": "len",
+        "send": "size",
+        "recv": "size",
+    }
+    for source in sorted(pim_dir.glob("core_*.json"), key=lambda path: int(path.stem.split("_")[1])):
+        with open(source, encoding="utf-8") as f:
+            instructions = json.load(f)
+        for instruction in instructions:
+            op = instruction["op"]
+            if op == "setbw":
+                instruction["ibiw"] = 8
+                instruction["obiw"] = 8
+            elif op == "sldi":
+                instruction["imm"] = int8_bytes(instruction["imm"], "address")
+            if field := byte_size_fields.get(op):
+                instruction[field] = int8_bytes(instruction[field], f"{op} {field}")
+                if offset := instruction.get("offset"):
+                    offset["offset_value"] = int8_bytes(offset["offset_value"], f"{op} offset")
+        with open(output_dir / source.name, "w", encoding="utf-8") as f:
+            json.dump(instructions, f, separators=(",", ":"))
+            f.write("\n")
+    return output_dir

validation/raptor_validation/validate_one.py

@@ -1,6 +1,5 @@
 import json
 import os
-import re
 import shutil
 import subprocess
 import sys
@@ -11,6 +10,7 @@ from colorama import Style, Fore
 from .gen_network_runner import gen_network_runner
 from .onnx_utils import gen_random_inputs, save_inputs_to_files, onnx_io, write_inputs_to_memory_bin, _ONNX_TO_NP
 from .raptor import compile_with_raptor
+from .pimsim_nn import export_raptor_latency_artifact, parse_pimsim_nn_metrics
 from .subprocess_utils import run_command_with_reporter
 
 STAGE_TITLES = (
@@ -61,9 +61,7 @@ PIMSIM_FAILED = "ERROR"
 PIMSIM_UNSUPPORTED = "UNSUPPORTED"
 PIMSIM_SKIPPED = "SKIP"
 PIMSIM_NOT_RUN = "-"
-PIMSIM_UNSUPPORTED_VSOFTMAX = "pimsim-nn does not support binary opcode vsoftmax"
-PIMSIM_LATENCY_RE = re.compile(r"\blatency:\s+([0-9.eE+-]+)\s+ms")
-PIMSIM_POWER_RE = re.compile(r"\baverage power:\s+([0-9.eE+-]+)\s+mW")
+PIMSIM_UNSUPPORTED_VSOFTMAX = "pimsim-nn does not support opcode vsoftmax"
 
 
 class PimSimUnsupportedError(RuntimeError):
@@ -80,6 +78,7 @@ class ValidationResult:
     pim_pass_timings: dict[str, float] = field(default_factory=dict)
     pimsim_latency_ms: float | None = None
     pimsim_power_mw: float | None = None
+    pimsim_energy_pj: float | None = None
     pimsim_status: str = PIMSIM_SKIPPED
 
 
@@ -262,9 +261,10 @@ def pimcomp_compatibility_errors(config_path, *, core_count, crossbar_count, cro
 
 
 def run_pimsim_nn(pimsim_nn_build_dir, pim_dir, config_path, reporter=None, timeout_sec=None):
+    latency_artifact = export_raptor_latency_artifact(pim_dir, Path(pim_dir).parent / "pimsim_nn")
     try:
         output = run_command(
-            [pimsim_nn_build_dir / "ChipTest", pim_dir, config_path, "--gui=false"],
+            [pimsim_nn_build_dir / "ChipTest", latency_artifact, config_path, "--gui=false"],
             cwd=pimsim_nn_build_dir,
             reporter=reporter,
             timeout_sec=timeout_sec,
@@ -275,11 +275,11 @@ def run_pimsim_nn(pimsim_nn_build_dir, pim_dir, config_path, reporter=None, time
         if PIMSIM_UNSUPPORTED_VSOFTMAX in error_output:
             raise PimSimUnsupportedError(PIMSIM_UNSUPPORTED_VSOFTMAX) from exc
         raise
-    latency_match = PIMSIM_LATENCY_RE.search(output)
-    power_match = PIMSIM_POWER_RE.search(output)
-    if not latency_match or not power_match:
-        raise RuntimeError("pimsim-nn output did not contain latency and average power")
-    return float(latency_match.group(1)), float(power_match.group(1))
+    metrics = parse_pimsim_nn_metrics(output)
+    required = ("latency_ms", "average_power_mw", "average_energy_pj")
+    if any(name not in metrics for name in required):
+        raise RuntimeError("pimsim-nn output did not contain latency, average power, and average energy")
+    return tuple(metrics[name] for name in required)
 
 
 def clean_workspace_artifacts(workspace_dir, model_stem):
@@ -416,6 +416,8 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
         pimsim_nn_build_dir = Path(pimsim_nn_build_dir).resolve()
         pimsim_config_path = Path(pimsim_config_path).resolve()
     compile_extra_args = list(raptor_extra_args or [])
+    if pimsim_enabled and "--pim-emit-json" not in compile_extra_args:
+        compile_extra_args.append("--pim-emit-json")
     owns_reporter = reporter is None
     reporter = reporter or ProgressReporter(model_total, stages_per_model=len(MODE_STAGE_TITLES[mode]), verbose=verbose)
 
@@ -540,10 +542,11 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
             "Run Non-functional Simulation")
         pimsim_latency_ms = None
         pimsim_power_mw = None
+        pimsim_energy_pj = None
         pimsim_status = PIMSIM_SKIPPED
         if pimsim_enabled:
             try:
-                pimsim_latency_ms, pimsim_power_mw = run_pimsim_nn(
+                pimsim_latency_ms, pimsim_power_mw, pimsim_energy_pj = run_pimsim_nn(
                     pimsim_nn_build_dir,
                     pim_dir,
                     pimsim_config_path,
@@ -554,7 +557,8 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
                 print_info(
                     reporter,
                     f"Latency: {pimsim_latency_ms:.6f} ms, "
-                    f"Power: {pimsim_power_mw:.6f} mW")
+                    f"Power: {pimsim_power_mw:.6f} mW, "
+                    f"Energy: {pimsim_energy_pj:.6f} pJ")
             except PimSimUnsupportedError as exc:
                 pimsim_status = PIMSIM_UNSUPPORTED
                 print_info(reporter, str(exc))
@@ -581,6 +585,7 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
             pim_pass_timings=pim_pass_timings,
             pimsim_latency_ms=pimsim_latency_ms,
             pimsim_power_mw=pimsim_power_mw,
+            pimsim_energy_pj=pimsim_energy_pj,
             pimsim_status=pimsim_status,
         )
     except Exception:

validation/tools/pimcomp/compare_raptor_pimcomp.py

@@ -24,9 +24,10 @@ import onnx
 from colorama import Fore, Style
 
 
-REPO = Path(__file__).resolve().parents[2]
+REPO = Path(__file__).resolve().parents[3]
 VALIDATION_DIR = REPO / "validation"
 PIMSIM_CONFIG_DIR = VALIDATION_DIR / "pimsim_configs/pimcomp"
+PIMCOMP_OUTPUT_FILES = ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
 sys.path.insert(0, str(VALIDATION_DIR))
 
 from raptor_validation.gen_network_runner import gen_network_runner  # noqa: E402
@@ -38,6 +39,10 @@ from raptor_validation.onnx_utils import (  # noqa: E402
     write_inputs_to_memory_bin,
 )
 from raptor_validation.raptor import compile_with_raptor  # noqa: E402
+from raptor_validation.pimsim_nn import (  # noqa: E402
+    export_raptor_latency_artifact,
+    parse_pimsim_nn_metrics,
+)
 from raptor_validation.validate_one import (  # noqa: E402
     STAGE_COLORS,
     build_dump_ranges,
@@ -70,7 +75,12 @@ def load_pimcomp_exporter():
     module = importlib.util.module_from_spec(spec)
     assert spec is not None and spec.loader is not None
     sys.modules.setdefault("cv2", types.ModuleType("cv2"))
-    spec.loader.exec_module(module)
+    write_bytecode = sys.dont_write_bytecode
+    sys.dont_write_bytecode = True
+    try:
+        spec.loader.exec_module(module)
+    finally:
+        sys.dont_write_bytecode = write_bytecode
     return module
 
 
@@ -183,37 +193,21 @@ def run_logged(
     return proc.stdout
 
 
-def remove_tree(path: Path) -> None:
-    if not path.exists() and not path.is_symlink():
-        return
-    if path.is_symlink() or path.is_file():
-        path.unlink()
-        return
-    while True:
-        children = list(path.iterdir())
-        if not children:
-            break
-        for child in children:
-            remove_tree(child)
-    path.rmdir()
-
-
 def load_model_inputs(model_path: Path, seed: int):
     inputs_desc, outputs_desc = onnx_io(model_path)
     arrays_in_order, _ = gen_random_inputs(inputs_desc, seed=seed)
-    return inputs_desc, outputs_desc, arrays_in_order, arrays_in_order
+    return inputs_desc, outputs_desc, arrays_in_order
 
 
 def load_saved_inputs(
-    model_path: Path,
     inputs_desc: list[tuple[int, str, int, list[int]]],
     inputs_dir: Path,
-) -> tuple[list[np.ndarray], list[np.ndarray]]:
+) -> list[np.ndarray]:
     arrays = []
     for idx, name, elem_type, shape in inputs_desc:
         array = np.loadtxt(inputs_dir / f"in{idx}.csv", delimiter=",", dtype=_ONNX_TO_NP[elem_type]).reshape(shape)
         arrays.append(array)
-    return arrays, arrays
+    return arrays
 
 
 def prepare_pimcomp_model(model_path: Path, out_dir: Path) -> Path:
@@ -328,7 +322,7 @@ def compile_reference(
     runner_base = runner_dir / stem
 
     run_logged(
-        "Reference Emit ONNX IR",
+        "Compile Reference ONNX IR",
         [str(args.raptor_path), str(model_path), "-o", str(onnx_ir_base), "--EmitONNXIR",
          "--mlir-elide-elementsattrs-if-larger=16", "--enable-conv-opt-pass=false"],
         cwd=REPO,
@@ -337,7 +331,7 @@ def compile_reference(
         stage="Compile ONNX",
     )
     run_logged(
-        "Reference Native Compile",
+        "Compile Reference Native",
         [str(args.raptor_path), "-O3", str(model_path), "-o", str(runner_base)],
         cwd=REPO,
         timeout_sec=args.timeout_seconds,
@@ -416,13 +410,18 @@ def compile_raptor_target(
         f"--crossbar-size={hardware['crossbar_size']}",
         f"--crossbar-count={hardware['crossbar_count']}",
         f"--core-count={hardware['core_count']}",
+        f"--pim-target-config={args.pimcomp_config}",
         "--pim-emit-json",
         *args.raptor_extra_arg,
     ]
     print_step("Compile Raptor PIM", cmd, REPO, "Compile PIM")
     start = time.perf_counter()
     command = shell_join(cmd)
-    raptor_extra_args = ["--pim-emit-json", *args.raptor_extra_arg]
+    raptor_extra_args = [
+        f"--pim-target-config={args.pimcomp_config}",
+        "--pim-emit-json",
+        *args.raptor_extra_arg,
+    ]
     try:
         timings = compile_with_raptor(
             model_path,
@@ -459,7 +458,7 @@ def compile_raptor_target(
     return out_dir / "pim", timings
 
 
-def run_rust_validation(
+def run_functional_validation(
     label: str,
     pim_dir: Path,
     config_path: Path,
@@ -510,7 +509,7 @@ def run_rust_validation(
 
 def copy_pimcomp_outputs(source_dir: Path, out_dir: Path):
     out_dir.mkdir(parents=True, exist_ok=True)
-    for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt"):
+    for name in PIMCOMP_OUTPUT_FILES:
         shutil.copy2(source_dir / name, out_dir / name)
 
 
@@ -527,7 +526,7 @@ def compile_pimcomp(
         shutil.copy2(args.pimcomp_config, runtime_config)
     pimcomp_output_dir = out_dir / "output"
     pimcomp_output_dir.mkdir(parents=True, exist_ok=True)
-    for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt"):
+    for name in PIMCOMP_OUTPUT_FILES:
         (pimcomp_output_dir / name).unlink(missing_ok=True)
     model_name = args.pimcomp_model_name or f"compare_{model_path.stem}"
     frontend_json = frontend_json_dir / f"{model_name}.json"
@@ -540,7 +539,7 @@ def compile_pimcomp(
         str(frontend_json),
     ]
     run_logged(
-        "PIMCOMP Frontend",
+        "Compile PIMCOMP Frontend",
         frontend_cmd,
         cwd=args.pimcomp_dir / "frontend",
         timeout_sec=args.timeout_seconds,
@@ -556,20 +555,20 @@ def compile_pimcomp(
         "-s=YES",
     ]
     run_logged(
-        "PIMCOMP Backend",
+        "Compile PIMCOMP Backend",
         backend_cmd,
         cwd=frontend_json_dir.parent,
         timeout_sec=args.timeout_seconds,
         steps=steps,
         stage="Compile PIM",
     )
-    remove_tree(frontend_json_dir.parent)
+    shutil.rmtree(frontend_json_dir.parent)
     return pimcomp_output_dir / "VerificationInfo.json", pimcomp_output_dir / "SimulationInfo.gz"
 
 
 def export_pimcomp_for_pimsim_nn(simulation_info: Path, output_dir: Path) -> Path:
     if output_dir.exists():
-        remove_tree(output_dir)
+        shutil.rmtree(output_dir)
     with gzip.open(simulation_info, "rt", encoding="utf-8") as f:
         sim_info = json.load(f)
 
@@ -598,7 +597,7 @@ def export_pimcomp_for_pimsim_nn(simulation_info: Path, output_dir: Path) -> Pat
 
     for core_idx in core_indices:
         core_key = f"core{core_idx}"
-        instructions = sim_info.get(core_key, []) or [{"op": "lldi", "rd": 0, "imm": 0, "len": 0}]
+        instructions = sim_info.get(core_key, [])
         with open(output_dir / f"core_{core_idx}.json", "w", encoding="utf-8") as f:
             json.dump(instructions, f, separators=(",", ":"))
             f.write("\n")
@@ -622,7 +621,7 @@ def export_pimcomp_for_rust(
     if len(runtime_inputs) != 1:
         raise ValueError("PIMCOMP export currently requires exactly one runtime input tensor")
     if output_dir.exists():
-        remove_tree(output_dir)
+        shutil.rmtree(output_dir)
     exporter = load_pimcomp_exporter()
     with open(verification_info, "r", encoding="utf-8") as f:
         final_info = json.load(f)
@@ -727,7 +726,7 @@ def export_pimcomp_for_rust(
         for sim_inst in sim_info.get(core_name, []) or []:
             op = sim_inst["op"]
             if op == "setbw":
-                instructions.append(sim_inst)
+                instructions.append({"op": "setbw", "ibiw": 32, "obiw": 32})
                 continue
             if op == "sldi":
                 translated = {"op": "sldi", "rd": sim_inst["rd"], "imm": exporter.byte_offset(sim_inst["imm"])}
@@ -782,10 +781,12 @@ def export_pimcomp_for_rust(
                         "offset": sim_inst["offset"],
                     }
                 )
-            elif op in ("lmv", "vvadd", "vvmul", "vvmax", "vrelu"):
+            elif op == "lmv":
                 translated = dict(sim_inst)
                 translated["len"] = exporter.byte_offset(sim_inst["len"])
                 instructions.append(translated)
+            elif op in ("vvadd", "vvmul", "vvmax", "vrelu"):
+                instructions.append(sim_inst)
             elif op in ("send", "recv"):
                 translated = dict(sim_inst)
                 translated["size"] = exporter.byte_offset(sim_inst["size"])
@@ -822,24 +823,6 @@ def export_pimcomp_for_rust(
     return output_dir
 
 
-def parse_pimsim_nn_report(output: str) -> dict[str, float | int | str]:
-    patterns = {
-        "output_count": r"output count:\s+([0-9]+)\s+samples",
-        "throughput": r"throughput:\s+([0-9.eE+-]+)\s+samples/s",
-        "average_latency_ms": r"average latency:\s+([0-9.eE+-]+)\s+ms",
-        "latency_ms": r"latency:\s+([0-9.eE+-]+)\s+ms",
-        "average_power_mw": r"average power:\s+([0-9.eE+-]+)\s+mW",
-        "average_energy_pj": r"average energy:\s+([0-9.eE+-]+)\s+pJ/it",
-    }
-    result: dict[str, float | int | str] = {"raw_output": output}
-    for key, pattern in patterns.items():
-        match = re.search(pattern, output)
-        if match:
-            value = match.group(1)
-            result[key] = int(value) if key == "output_count" else float(value)
-    return result
-
-
 def run_pimsim_nn(
     label: str,
     inst_path: Path,
@@ -861,7 +844,7 @@ def run_pimsim_nn(
         steps=steps,
         stage="Run Non-functional Simulation",
     )
-    return parse_pimsim_nn_report(output)
+    return parse_pimsim_nn_metrics(output)
 
 
 def parse_raptor_instructions(pim_dir: Path) -> dict[str, Any]:
@@ -1069,7 +1052,7 @@ def write_report(
 
     lines.extend(
         [
-            "## Semantic Validation",
+            "## Functional Validation",
             "",
             f"- Raptor via `pim-simulator`: `{validation_status(raptor_validation)}`",
             f"- PIMCOMP via exported `pim-simulator`: `{validation_status(pimcomp_validation)}`",
@@ -1268,6 +1251,7 @@ def main():
     runner_path: Path | None = None
     reference_dir: Path | None = None
     raptor_pim_dir: Path | None = None
+    raptor_pimsim_dir: Path | None = None
     raptor_pass_timings: dict[str, float] = {}
     verification_info: Path | None = None
     simulation_info: Path | None = None
@@ -1289,7 +1273,8 @@ def main():
 
     model_io = try_stage(failures, "Load model inputs", load_model_inputs, model_path, args.seed)
     if model_io is not None:
-        inputs_desc, outputs_desc, arrays_in_order, runtime_inputs = model_io
+        inputs_desc, outputs_desc, arrays_in_order = model_io
+        runtime_inputs = arrays_in_order
 
     if reuse_raptor and model_io is not None:
         reuse_report_path = args.reuse_raptor_report.resolve()
@@ -1300,11 +1285,11 @@ def main():
             raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
         reference_dir = Path(reused["paths"]["reference_outputs"])
         raptor_pim_dir = Path(reused["paths"]["raptor_pim"])
-        arrays_in_order, runtime_inputs = load_saved_inputs(
-            model_path,
+        arrays_in_order = load_saved_inputs(
             inputs_desc,
             reference_dir.parent / "inputs",
         )
+        runtime_inputs = arrays_in_order
         raptor_validation = CompareResult(**reused["raptor_validation"])
         raptor_perf = reused["raptor_performance"]
         raptor_instr = reused["raptor_instruction_summary"]
@@ -1392,9 +1377,9 @@ def main():
         if wrote_inputs and reference_dir is not None and outputs_desc:
             validation = try_stage(
                 failures,
-                "Rust Validation Raptor",
-                run_rust_validation,
-                "Rust Validation Raptor",
+                "Functional Validation Raptor",
+                run_functional_validation,
+                "Functional Validation Raptor",
                 raptor_pim_dir,
                 raptor_pim_dir / "config.json",
                 out_dir / "simulation/out.bin",
@@ -1451,7 +1436,7 @@ def main():
     if verification_info is not None and simulation_info is not None and model_io is not None:
         exported = try_stage(
             failures,
-            "Export PIMCOMP for Rust",
+            "Export PIMCOMP for Functional Validation",
             export_pimcomp_for_rust,
             pimcomp_model_path,
             verification_info,
@@ -1464,22 +1449,22 @@ def main():
     elif verification_info is None or simulation_info is None:
         record_failure(
             failures,
-            "Export PIMCOMP for Rust",
-            "PIMCOMP Rust export failed because PIMCOMP did not produce VerificationInfo.json and SimulationInfo.gz.",
+            "Export PIMCOMP for Functional Validation",
+            "PIMCOMP functional export failed because PIMCOMP did not produce VerificationInfo.json and SimulationInfo.gz.",
         )
     else:
         record_failure(
             failures,
-            "Export PIMCOMP for Rust",
-            "PIMCOMP Rust export failed because model inputs are not available.",
+            "Export PIMCOMP for Functional Validation",
+            "PIMCOMP functional export failed because model inputs are not available.",
         )
 
     if pimcomp_export_dir is not None and reference_dir is not None and outputs_desc:
         validation = try_stage(
             failures,
-            "Rust Validation PIMCOMP",
-            run_rust_validation,
-            "Rust Validation PIMCOMP",
+            "Functional Validation PIMCOMP",
+            run_functional_validation,
+            "Functional Validation PIMCOMP",
             pimcomp_export_dir,
             pimcomp_export_dir / "config.json",
             out_dir / "simulation/pimcomp.out.bin",
@@ -1491,7 +1476,7 @@ def main():
         )
         pimcomp_validation = validation if validation is not None else failed_validation("PIMCOMP validation failed")
     elif pimcomp_export_dir is None:
-        pimcomp_validation = failed_validation("PIMCOMP Rust export is not available")
+        pimcomp_validation = failed_validation("PIMCOMP functional export is not available")
     elif reference_dir is None:
         pimcomp_validation = failed_validation("Reference outputs are not available")
     else:
@@ -1524,17 +1509,27 @@ def main():
         pimcomp_perf = skipped_perf("pimsim-nn config is not available")
     else:
         if not reuse_raptor and raptor_pim_dir is not None:
-            perf = try_stage(
+            raptor_pimsim_dir = try_stage(
                 failures,
-                "pimsim-nn Raptor",
-                run_pimsim_nn,
-                "pimsim-nn Raptor",
+                "Export Raptor for pimsim-nn",
+                export_raptor_latency_artifact,
                 raptor_pim_dir,
-                pimsim_config,
-                steps,
-                args,
+                out_dir / "raptor/pimsim_nn",
             )
-            raptor_perf = perf if perf is not None else failed_perf("pimsim-nn Raptor failed")
+            if raptor_pimsim_dir is not None:
+                perf = try_stage(
+                    failures,
+                    "Non-Functional Simulation Raptor",
+                    run_pimsim_nn,
+                    "Non-Functional Simulation Raptor",
+                    raptor_pimsim_dir,
+                    pimsim_config,
+                    steps,
+                    args,
+                )
+                raptor_perf = perf if perf is not None else failed_perf("pimsim-nn Raptor failed")
+            else:
+                raptor_perf = failed_perf("Raptor pimsim-nn export failed")
         elif not reuse_raptor:
             raptor_perf = skipped_perf("Raptor PIM directory is not available")
 
@@ -1549,9 +1544,9 @@ def main():
             if pimcomp_pimsim_dir is not None:
                 perf = try_stage(
                     failures,
-                    "pimsim-nn PIMCOMP",
+                    "Non-Functional Simulation PIMCOMP",
                     run_pimsim_nn,
-                    "pimsim-nn PIMCOMP",
+                    "Non-Functional Simulation PIMCOMP",
                     pimcomp_pimsim_dir,
                     pimsim_config,
                     steps,
@@ -1613,6 +1608,7 @@ def main():
         "paths": {
             "reference_outputs": optional_path(reference_dir),
             "raptor_pim": optional_path(raptor_pim_dir),
+            "raptor_pimsim_nn": optional_path(raptor_pimsim_dir),
             "pimcomp_simulation_info": optional_path(simulation_info),
             "pimcomp_exported_pim": optional_path(pimcomp_export_dir),
             "pimsim_config": optional_path(pimsim_config),
@@ -1625,11 +1621,11 @@ def main():
         f.write("\n")
 
     failed_steps = any(step.status != "passed" for step in steps)
-    semantic_failure = any(
+    functional_failure = any(
         result.status == "done" and not result.passed
         for result in (raptor_validation, pimcomp_validation)
     )
-    failed = bool(failures or failed_steps or semantic_failure)
+    failed = bool(failures or failed_steps or functional_failure)
     result = "FAIL" if args.fail_on_error and failed else "DONE" if failed else "PASS"
     color = Fore.RED if result == "FAIL" else Fore.YELLOW if result == "DONE" else Fore.GREEN
     print("\n" + Style.BRIGHT + f"Result: {color}{result}" + Style.RESET_ALL)

validation/tools/pimcomp/run_pimcomp_paper_latency.py

@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+from colorama import Fore, Style
+
+
+REPO = Path(__file__).resolve().parents[3]
+SUITE = REPO / "validation/networks/pimcomp_models"
+sys.path.insert(0, str(REPO / "validation"))
+
+from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics  # noqa: E402
+from raptor_validation.validate_one import STAGE_COLORS  # noqa: E402
+
+PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
+PIMCOMP_CONFIG = REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json"
+COMPARE = Path(__file__).resolve().with_name("compare_raptor_pimcomp.py")
+MODELS = {
+    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
+    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
+    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
+    "googlenet": SUITE / "googlenet/googlenet-12-latency.onnx",
+}
+
+
+def result_dir(root: Path | None, name: str) -> Path:
+    return root / name if root is not None else MODELS[name].parent
+
+
+def write_results_csv(root: Path | None) -> Path:
+    output = (root or SUITE) / "results.csv"
+    fields = (
+        "model",
+        "raptor_latency_ms",
+        "pimcomp_latency_ms",
+        "raptor_energy_pj",
+        "pimcomp_energy_pj",
+        "faster_compiler",
+        "speedup",
+    )
+    rows = []
+    for name in MODELS:
+        report_path = result_dir(root, name) / "pimcomp/comparison_report.json"
+        if not report_path.exists():
+            continue
+        report = json.loads(report_path.read_text(encoding="utf-8"))
+        raptor = report.get("raptor_performance") or {}
+        pimcomp = report.get("pimcomp_performance") or {}
+        raptor_latency = raptor.get("latency_ms")
+        pimcomp_latency = pimcomp.get("latency_ms")
+        if raptor_latency is None or pimcomp_latency is None:
+            continue
+        raptor_energy = (raptor.get("average_energy_pj")
+                          or parse_pimsim_nn_metrics(raptor.get("raw_output", "")).get("average_energy_pj"))
+        pimcomp_energy = (pimcomp.get("average_energy_pj")
+                           or parse_pimsim_nn_metrics(pimcomp.get("raw_output", "")).get("average_energy_pj"))
+        faster = "raptor" if raptor_latency < pimcomp_latency else "pimcomp"
+        rows.append({
+            "model": name,
+            "raptor_latency_ms": f"{raptor_latency:.6f}",
+            "pimcomp_latency_ms": f"{pimcomp_latency:.6f}",
+            "raptor_energy_pj": "" if raptor_energy is None else f"{raptor_energy:.6f}",
+            "pimcomp_energy_pj": "" if pimcomp_energy is None else f"{pimcomp_energy:.6f}",
+            "faster_compiler": faster,
+            "speedup": f"{max(raptor_latency, pimcomp_latency) / min(raptor_latency, pimcomp_latency):.2f}",
+        })
+    with open(output, "w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
+        writer.writeheader()
+        writer.writerows(rows)
+    return output
+
+
+def print_stage(title: str, color: str) -> None:
+    print("\n" + Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL, flush=True)
+
+
+def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
+    print(f"  cwd: {REPO}", flush=True)
+    print(f"  $ {shlex.join(command)}", flush=True)
+    if dry_run:
+        return 0
+    return subprocess.run(command, cwd=REPO, check=check).returncode
+
+
+def validate_pimcomp_source() -> None:
+    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
+    source = header.read_text(encoding="utf-8")
+    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
+        if setting not in source:
+            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
+
+
+def comparison_command(model: Path, result_dir: Path, timeout: float) -> list[str]:
+    return [
+        sys.executable,
+        str(COMPARE),
+        "--model",
+        str(model),
+        "--out-dir",
+        str(result_dir),
+        "--pimcomp-dir",
+        str(PIMCOMP_SOURCE),
+        "--pimcomp-config",
+        str(PIMCOMP_CONFIG),
+        "--core-count",
+        "168",
+        "--crossbar-count",
+        "96",
+        "--crossbar-size",
+        "128",
+        "--mesh-rows",
+        "12",
+        "--mesh-cols",
+        "14",
+        "--pimsim-mode",
+        "latency",
+        "--pimcomp-pipeline",
+        "element",
+        "--pimcomp-replication",
+        "GA",
+        "--timeout-seconds",
+        str(timeout),
+        "--fail-on-error",
+    ]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Reproduce the serial Arch-A latency comparison from the PIMCOMP paper."
+    )
+    parser.add_argument(
+        "--out-dir",
+        type=Path,
+        help="Result root (default: artifacts beside each model under validation/).",
+    )
+    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
+    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
+    parser.add_argument(
+        "--resume",
+        action="store_true",
+        help="Skip models with a completed JSON report.",
+    )
+    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
+    args = parser.parse_args()
+
+    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
+
+    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
+    if missing:
+        parser.error(f"missing model(s): {', '.join(missing)}")
+
+    validate_pimcomp_source()
+    if out_dir is not None and not args.dry_run:
+        out_dir.mkdir(parents=True, exist_ok=True)
+
+    print(Style.BRIGHT + f"Found {len(args.models)} PIMCOMP model(s) to compare." + Style.RESET_ALL)
+    print(f"Results root: {out_dir or SUITE}")
+    print("=" * 72)
+
+    print_stage("Build Raptor", STAGE_COLORS["Build Runner"])
+    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
+
+    print_stage("Build PIMCOMP", STAGE_COLORS["Build Runner"])
+    run(
+        ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
+        dry_run=args.dry_run,
+    )
+
+    failed = []
+    for index, name in enumerate(args.models, start=1):
+        model_result_dir = result_dir(out_dir, name)
+        print(
+            "\n" + Fore.CYAN + f"[{index}/{len(args.models)}]" + Style.RESET_ALL
+            + f" {Style.BRIGHT}Comparing {name}{Style.RESET_ALL}",
+            flush=True,
+        )
+        if args.resume and (model_result_dir / "pimcomp/comparison_report.json").exists():
+            print(
+                Fore.YELLOW + "  Completed report exists; skipping" + Style.RESET_ALL,
+                flush=True,
+            )
+            continue
+        returncode = run(
+            comparison_command(MODELS[name], model_result_dir, args.timeout_seconds),
+            dry_run=args.dry_run,
+            check=False,
+        )
+        if returncode:
+            failed.append(name)
+
+    if args.dry_run:
+        return 1 if failed else 0
+
+    results_path = write_results_csv(out_dir)
+    print_stage("Results", STAGE_COLORS["Compare Outputs"])
+    print(results_path.read_text(encoding="utf-8"), end="")
+    print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
+    print(Style.BRIGHT + f"Passed: {len(args.models) - len(failed)}" + Style.RESET_ALL)
+    print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
+    print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
+    if failed:
+        print(
+            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

validation/tools/run_pimcomp_paper_latency.py

@@ -1,148 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import shlex
-import subprocess
-import sys
-from pathlib import Path
-
-from colorama import Fore, Style
-
-
-REPO = Path(__file__).resolve().parents[2]
-SUITE = REPO / "validation/networks/pimcomp_models"
-PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
-PIMCOMP_CONFIG = REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json"
-COMPARE = REPO / "validation/tools/compare_raptor_pimcomp.py"
-MODELS = {
-    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
-    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
-    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
-    "googlenet": SUITE / "googlenet/googlenet-12-no-softmax.onnx",
-}
-
-
-def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
-    print(Fore.CYAN + "$ " + Style.RESET_ALL + shlex.join(command), flush=True)
-    if dry_run:
-        return 0
-    return subprocess.run(command, cwd=REPO, check=check).returncode
-
-
-def validate_pimcomp_source() -> None:
-    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
-    source = header.read_text(encoding="utf-8")
-    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
-        if setting not in source:
-            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
-
-
-def comparison_command(model: Path, result_dir: Path, timeout: float) -> list[str]:
-    return [
-        sys.executable,
-        str(COMPARE),
-        "--model",
-        str(model),
-        "--out-dir",
-        str(result_dir),
-        "--pimcomp-dir",
-        str(PIMCOMP_SOURCE),
-        "--pimcomp-config",
-        str(PIMCOMP_CONFIG),
-        "--core-count",
-        "168",
-        "--crossbar-count",
-        "96",
-        "--crossbar-size",
-        "128",
-        "--mesh-rows",
-        "12",
-        "--mesh-cols",
-        "14",
-        "--pimsim-mode",
-        "latency",
-        "--pimcomp-pipeline",
-        "element",
-        "--pimcomp-replication",
-        "GA",
-        "--timeout-seconds",
-        str(timeout),
-        "--fail-on-error",
-    ]
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(
-        description="Reproduce the serial Arch-A latency comparison from the PIMCOMP paper."
-    )
-    parser.add_argument(
-        "--out-dir",
-        type=Path,
-        help="Result root (default: artifacts beside each model under validation/).",
-    )
-    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
-    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
-    parser.add_argument(
-        "--resume",
-        action="store_true",
-        help="Skip models with a completed JSON report.",
-    )
-    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
-    args = parser.parse_args()
-
-    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
-
-    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
-    if missing:
-        parser.error(f"missing model(s): {', '.join(missing)}")
-
-    validate_pimcomp_source()
-    if out_dir is not None and not args.dry_run:
-        out_dir.mkdir(parents=True, exist_ok=True)
-
-    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
-    run(
-        ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
-        dry_run=args.dry_run,
-    )
-
-    failed = []
-    for name in args.models:
-        result_dir = out_dir / name if out_dir is not None else MODELS[name].parent
-        if args.resume and (result_dir / "pimcomp/comparison_report.json").exists():
-            print(
-                Fore.YELLOW + f"[{name}] completed report exists; skipping" + Style.RESET_ALL,
-                flush=True,
-            )
-            continue
-        print(
-            "\n" + Fore.CYAN + f"[{name}]" + Style.RESET_ALL
-            + f" {Style.BRIGHT}Arch-A latency comparison{Style.RESET_ALL}",
-            flush=True,
-        )
-        returncode = run(
-            comparison_command(MODELS[name], result_dir, args.timeout_seconds),
-            dry_run=args.dry_run,
-            check=False,
-        )
-        if returncode:
-            failed.append(name)
-
-    if failed:
-        print(
-            "\n" + Style.BRIGHT + Fore.RED + "Result: FAIL" + Style.RESET_ALL,
-            file=sys.stderr,
-        )
-        print(
-            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
-            file=sys.stderr,
-        )
-        return 1
-    if not args.dry_run:
-        print("\n" + Style.BRIGHT + f"Result: {Fore.GREEN}PASS" + Style.RESET_ALL)
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

validation/validate.py

@@ -255,6 +255,10 @@ def main():
     total_timing_sum = 0.0
     timed_benchmark_count = 0
     reporter = ProgressReporter(len(onnx_files), stages_per_model=1, verbose=a.verbose)
+    raptor_extra_args = list(a.raptor_extra_arg)
+    if not any(str(arg).startswith("--pim-target-config=") for arg in raptor_extra_args):
+        raptor_extra_args.append(f"--pim-target-config={pimsim_config_path}")
+
     validation_kwargs = {
         "raptor_path": a.raptor_path,
         "onnx_include_dir": a.onnx_include_dir,
@@ -262,7 +266,7 @@ def main():
         "crossbar_size": a.crossbar_size,
         "crossbar_count": a.crossbar_count,
         "core_count": a.core_count,
-        "raptor_extra_args": a.raptor_extra_arg,
+        "raptor_extra_args": raptor_extra_args,
         "pimsim_nn_build_dir": pimsim_nn_build_dir,
         "pimsim_config_path": selected_pimsim_config,
         "command_timeout_seconds": a.command_timeout_seconds,
@@ -337,28 +341,31 @@ def main():
         rel: (
             format_pimsim_metric(result, result.pimsim_latency_ms, "ms"),
             format_pimsim_metric(result, result.pimsim_power_mw, "mW"),
+            format_pimsim_metric(result, result.pimsim_energy_pj, "pJ"),
         )
         for rel, result in results.items()
     }
     latency_width = max(len("Latency"), *(len(metrics[0]) for metrics in formatted_metrics.values()))
     power_width = max(len("Power"), *(len(metrics[1]) for metrics in formatted_metrics.values()))
+    energy_width = max(len("Energy"), *(len(metrics[2]) for metrics in formatted_metrics.values()))
     separator = (
         f"+-{'-' * path_width}-+-{'-' * status_width}-+-{'-' * latency_width}"
-        f"-+-{'-' * power_width}-+")
+        f"-+-{'-' * power_width}-+-{'-' * energy_width}-+")
     print(separator)
     print(
         f"| {'Operation'.ljust(path_width)} | {'Result'.ljust(status_width)} | "
-        f"{'Latency'.rjust(latency_width)} | {'Power'.rjust(power_width)} |"
+        f"{'Latency'.rjust(latency_width)} | {'Power'.rjust(power_width)} | "
+        f"{'Energy'.rjust(energy_width)} |"
     )
     print(separator)
     for rel, result in results.items():
         plain_status = "PASS" if result.passed else "FAIL"
         status = Fore.GREEN + plain_status.ljust(status_width) + Style.RESET_ALL if result.passed else \
             Fore.RED + plain_status.ljust(status_width) + Style.RESET_ALL
-        latency, power = formatted_metrics[rel]
+        latency, power, energy = formatted_metrics[rel]
         print(
             f"| {rel.ljust(path_width)} | {status} | {latency.rjust(latency_width)} | "
-            f"{power.rjust(power_width)} |")
+            f"{power.rjust(power_width)} | {energy.rjust(energy_width)} |")
     print(separator)
     print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
     print(Style.BRIGHT + f"Passed: {n_passed}" + Style.RESET_ALL)