simplify pimcomp compare workflow

validation/.gitignore

@@ -6,6 +6,7 @@ operations/**/simulation
 networks/**/inputs
 networks/**/outputs
 networks/**/raptor
+networks/**/pimcomp
 networks/**/runner
 networks/**/simulation
 networks/**/real_image_val

validation/README.md

@@ -60,6 +60,27 @@ Validate a network or network slice:
 `--operations-dir` may point to any directory tree containing `.onnx` files.
 The script discovers them recursively.
 
+## Raptor vs PIMCOMP comparison
+
+The PIMCOMP paper-model suite has a one-command Arch-A comparison:
+
+```bash
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
+```
+
+The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
+builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, and compares
+the four paper models one at a time. Each PIMCOMP GA run evaluates candidates
+in parallel; set `OMP_NUM_THREADS` to control its worker count. Use
+`--models vgg8` for one model or `--dry-run` to print the commands.
+
+Generated artifacts are stored beside each model and ignored by Git. The
+comparison reuses the same model-level `inputs/`, `outputs/`, `runner/`,
+`raptor/`, and `simulation/` paths as regular validation. PIMCOMP-only
+artifacts and `comparison_report.{md,json}` live under `pimcomp/`. See
+[`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for
+profiles, model provenance, limitations, and remote execution.
+
 ## Validation modes
 
 The default mode performs the complete workflow.
@@ -167,7 +188,8 @@ Artifacts are written beside each model:
 | `outputs/` | ONNX-MLIR reference output CSV files. |
 | `raptor/` | Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts. |
 | `runner/` | Generated reference runner source, build tree, and shared library. |
-| `simulation/out.bin` | Functional simulator output used for comparison. |
+| `simulation/` | Functional simulator outputs used for numerical comparison. |
+| `pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
 
 The `raptor/` directory may include `spatial0.mlir`,
 `spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,

validation/networks/pimcomp_models/README.md

@@ -136,12 +136,11 @@ random placement code occasionally segfaults; an unchanged retry succeeded in
 the observed cases.
 
 The paper's optimizer uses a genetic algorithm with population 200 and up to
-1000 iterations. Select it with `-r=GA` for optimizer studies. The released
-source keeps population 200 but sets `max_iteration = 3`, so reproducing the
-paper's optimization search also requires changing that value in
-`backend/GeneticAlgorithm.h`. GA allocates roughly 32 GB in its fast evaluator;
-use monolith below instead of reducing cores or crossbars when local RAM is
-insufficient.
+1000 iterations. The checked-out PIMCOMP submodule already has both paper
+settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness
+evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS`
+to control its parallelism; otherwise OpenMP uses the available CPUs. The GA
+uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.
 
 ## Compare Raptor and PIMCOMP
 
@@ -150,34 +149,37 @@ compiles both instruction streams, runs both through `pimsim-nn`, validates
 Raptor through the Rust simulator, and writes Markdown and JSON reports.
 PIMCOMP Rust validation also runs when its optional exporter is available.
 
-To reproduce the complete Arch-A latency experiment, use the serial experiment
-runner. It creates an isolated PIMCOMP build with population 200 and 1000 GA
-iterations, runs only the `element`/batch-1 latency pipeline, and invokes the
-comparison driver for one model at a time:
+To reproduce the complete Arch-A latency experiment, use the model-by-model
+runner. It verifies the paper GA settings, builds Raptor and the existing
+`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
+for one model at a time:
 
 ```bash
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
-  --out-dir /tmp/raptor-pimcomp-paper-latency
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
 ```
 
-Reports are written under `<out-dir>/<model>/comparison_report.{md,json}`.
-Use `--models vgg8` to run one model, `--resume` after an interruption, or
-`--dry-run` to inspect every command. The runner continues after a failed model
-so all reports are produced.
+Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
+`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and
+`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates
+one isolated `models/JSON/` graph because PIMCOMP requires that relative
+layout; it is removed after a successful backend run and the shared submodule
+model directory is never modified. Models requiring BatchNormalization folding
+also receive a prepared ONNX file; other models use the original ONNX directly.
+PIMCOMP's source tree and build directory remain unchanged at runtime. Use
+`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run`
+to inspect every command, or `--out-dir PATH` to keep results outside
+`validation/`. The runner continues after a failed model so all reports are
+produced.
 
 Arch-A low-latency example:
 
 ```bash
 RAPTOR_ROOT=$PWD
-PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
-CONFIG_BACKUP=$(mktemp)
-cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
-trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
-cp "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" "$PIMCOMP/config.json"
 
 "$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/compare_raptor_pimcomp.py" \
   --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
-  --out-dir /tmp/compare-resnet34-arch-a-ll \
+  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
+  --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
   --core-count 168 \
   --crossbar-count 96 \
   --crossbar-size 128 \
@@ -196,11 +198,9 @@ If only semantic and instruction comparison is required, add
 `--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both
 semantic validations with maximum output differences below `5e-10`.
 
-The current PIMCOMP-NN submodule does not include the optional
-`verification/export_to_pim_simulator.py` helper. The driver therefore records
-PIMCOMP Rust semantic validation as skipped, while PIMCOMP compilation,
-instruction reporting, and `pimsim-nn` latency still run. Use
-`--fail-on-error` only when that semantic export helper is available.
+The comparison runner enables `--fail-on-error`, so a failed compiler,
+simulation, or semantic validation makes the command fail while preserving the
+generated report.
 
 Current Raptor status:
 
@@ -247,14 +247,13 @@ python3 -m venv .venv
 .venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
 
 # Run every latency comparison serially.
-.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
-  --out-dir /tmp/raptor-pimcomp-paper-latency
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py
 ```
 
 Copy reports back without transferring large compiler artifacts:
 
 ```bash
 rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \
-  monolith:/tmp/raptor-pimcomp-paper-latency/ \
-  /tmp/raptor-pimcomp-paper-latency/
+  "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \
+  validation/networks/pimcomp_models/
 ```

validation/networks/pimcomp_models/RESULTS.md

@@ -25,7 +25,6 @@ No throughput experiment was run.
 
 ```bash
 .venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
-  --out-dir /tmp/raptor-pimcomp-paper-latency \
   --models googlenet
 ```
 

validation/raptor_validation/raptor.py

@@ -1,5 +1,6 @@
 import re
 import shlex
+import shutil
 import subprocess
 from pathlib import Path
 from colorama import Fore, Style
@@ -43,6 +44,10 @@ def _format_command(cmd):
 def compile_with_raptor(network_path, raptor_onnx_path: Path, output_base: Path,
                         crossbar_size, crossbar_count, core_count,
                         raptor_extra_args, cwd, verbose, reporter, timeout_sec):
+    pim_dir = Path(output_base).parent / "pim"
+    if pim_dir.exists():
+        shutil.rmtree(pim_dir)
+
     # Define the arguments, with the possibility to set crossbar size and count
     args = [
         network_path,

validation/raptor_validation/validate_one.py

@@ -23,6 +23,16 @@ STAGE_TITLES = (
     "Compare Outputs",
     "Run Non-functional Simulation",
 )
+STAGE_COLORS = {
+    STAGE_TITLES[0]: Fore.BLUE,
+    STAGE_TITLES[1]: Fore.MAGENTA,
+    STAGE_TITLES[2]: Fore.YELLOW,
+    STAGE_TITLES[3]: Fore.GREEN,
+    STAGE_TITLES[4]: Fore.CYAN,
+    STAGE_TITLES[5]: Fore.MAGENTA,
+    STAGE_TITLES[6]: Fore.YELLOW,
+    STAGE_TITLES[7]: Fore.BLUE,
+}
 STAGE_COUNT = len(STAGE_TITLES)
 GENERATED_DIR_NAMES = ("inputs", "outputs", "raptor", "runner", "simulation")
 
@@ -294,17 +304,7 @@ def clean_workspace_artifacts(workspace_dir, model_stem):
 
 
 def print_stage(reporter, model_index, model_total, model_name, title):
-    stage_colors = {
-        STAGE_TITLES[0]: Fore.BLUE,
-        STAGE_TITLES[1]: Fore.MAGENTA,
-        STAGE_TITLES[2]: Fore.YELLOW,
-        STAGE_TITLES[3]: Fore.GREEN,
-        STAGE_TITLES[4]: Fore.CYAN,
-        STAGE_TITLES[5]: Fore.MAGENTA,
-        STAGE_TITLES[6]: Fore.YELLOW,
-        STAGE_TITLES[7]: Fore.BLUE,
-    }
-    color = stage_colors.get(title, Fore.WHITE)
+    color = STAGE_COLORS.get(title, Fore.WHITE)
     reporter.log(Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL)
     reporter.set_stage(model_index, model_total, model_name, title)
 

validation/tools/compare_raptor_pimcomp.py

@@ -21,6 +21,7 @@ from typing import Any
 
 import numpy as np
 import onnx
+from colorama import Fore, Style
 
 
 REPO = Path(__file__).resolve().parents[2]
@@ -37,7 +38,11 @@ from raptor_validation.onnx_utils import (  # noqa: E402
     write_inputs_to_memory_bin,
 )
 from raptor_validation.raptor import compile_with_raptor  # noqa: E402
-from raptor_validation.validate_one import build_dump_ranges, parse_pim_simulator_outputs  # noqa: E402
+from raptor_validation.validate_one import (  # noqa: E402
+    STAGE_COLORS,
+    build_dump_ranges,
+    parse_pim_simulator_outputs,
+)
 
 
 @dataclass
@@ -73,8 +78,14 @@ def shell_join(cmd: list[str]) -> str:
     return shlex.join(str(arg) for arg in cmd)
 
 
-def print_step(name: str, cmd: list[str] | None = None, cwd: Path | None = None):
-    print(f"\n[{name}]")
+def print_step(
+    name: str,
+    cmd: list[str] | None = None,
+    cwd: Path | None = None,
+    stage: str | None = None,
+):
+    color = STAGE_COLORS.get(stage or name, Fore.WHITE)
+    print("\n" + Style.BRIGHT + color + f"[{name}]" + Style.RESET_ALL)
     if cmd is not None:
         print(f"  cwd: {cwd or REPO}")
         print(f"  $ {shell_join(cmd)}")
@@ -108,9 +119,12 @@ def exception_message(exc: BaseException) -> str:
 
 def print_failure(name: str, exc: BaseException | str) -> None:
     message = exc if isinstance(exc, str) else exception_message(exc)
-    print(f"\n[{name} FAILED]")
+    print(
+        "\n" + Style.BRIGHT + Fore.RED + f"[{name} FAILED]" + Style.RESET_ALL,
+        file=sys.stderr,
+    )
     for line in message.splitlines()[:20]:
-        print(f"  {line}")
+        print(Fore.RED + f"  {line}" + Style.RESET_ALL, file=sys.stderr)
 
 
 def run_logged(
@@ -120,8 +134,9 @@ def run_logged(
     cwd: Path,
     timeout_sec: float,
     steps: list[StepRecord],
+    stage: str | None = None,
 ) -> str:
-    print_step(name, cmd, cwd)
+    print_step(name, cmd, cwd, stage)
     start = time.perf_counter()
     command = shell_join(cmd)
     try:
@@ -202,29 +217,29 @@ def load_saved_inputs(
 
 
 def prepare_pimcomp_model(model_path: Path, out_dir: Path) -> Path:
+    model = onnx.load(model_path)
+    if not any(node.op_type == "BatchNormalization" for node in model.graph.node):
+        return model_path
+
     out_dir.mkdir(parents=True, exist_ok=True)
     output_path = out_dir / f"{model_path.stem}_pimcomp.onnx"
-    model = onnx.load(model_path)
+    from onnxsim import simplify
+
+    model, equivalent = simplify(model, check_n=1)
+    if not equivalent:
+        raise RuntimeError("Conv+BatchNormalization folding changed the model output")
     if any(node.op_type == "BatchNormalization" for node in model.graph.node):
-        from onnxsim import simplify
+        import onnxruntime as ort
 
-        model, equivalent = simplify(model, check_n=1)
-        if not equivalent:
-            raise RuntimeError("Conv+BatchNormalization folding changed the model output")
+        options = ort.SessionOptions()
+        options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
+        options.optimized_model_filepath = str(output_path)
+        ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"])
+        model = onnx.load(output_path)
         if any(node.op_type == "BatchNormalization" for node in model.graph.node):
-            import onnxruntime as ort
-
-            options = ort.SessionOptions()
-            options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
-            options.optimized_model_filepath = str(output_path)
-            ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"])
-            model = onnx.load(output_path)
-            if any(node.op_type == "BatchNormalization" for node in model.graph.node):
-                raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization")
-        else:
-            onnx.save(model, output_path)
+            raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization")
     else:
-        shutil.copy2(model_path, output_path)
+        onnx.save(model, output_path)
     return output_path
 
 
@@ -259,8 +274,7 @@ def sanitize_output_name(name: str) -> str:
 
 
 def load_effective_hardware(args: argparse.Namespace) -> dict[str, int]:
-    config_path = args.pimcomp_dir / "config.json"
-    with open(config_path, "r", encoding="utf-8") as f:
+    with open(args.pimcomp_config, "r", encoding="utf-8") as f:
         config = json.load(f)
     rows, cols = config["chip_config"]["network_config"]["layout"]
     xbar_h, xbar_w = config["chip_config"]["core_config"]["matrix_config"]["xbar_size"]
@@ -304,7 +318,7 @@ def compile_reference(
     work_dir: Path,
     steps: list[StepRecord],
 ) -> Path:
-    raptor_dir = work_dir / "reference"
+    raptor_dir = work_dir / "raptor"
     runner_dir = work_dir / "runner"
     build_dir = runner_dir / "build"
     raptor_dir.mkdir(parents=True, exist_ok=True)
@@ -320,6 +334,7 @@ def compile_reference(
         cwd=REPO,
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Compile ONNX",
     )
     run_logged(
         "Reference Native Compile",
@@ -327,10 +342,11 @@ def compile_reference(
         cwd=REPO,
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Compile ONNX",
     )
     network_so = runner_base.with_suffix(".so")
 
-    print_step("Generate Runner Source")
+    print_step("Generate Runner Source", stage="Build Runner")
     gen_network_runner(
         model_path,
         network_so,
@@ -346,6 +362,7 @@ def compile_reference(
         cwd=build_dir,
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Build Runner",
     )
     run_logged(
         "Build Runner",
@@ -367,7 +384,7 @@ def generate_reference_outputs(
     out_dir: Path,
 ) -> Path:
     inputs_dir = out_dir / "inputs"
-    reference_dir = out_dir / "reference_outputs"
+    reference_dir = out_dir / "outputs"
     inputs_dir.mkdir(parents=True, exist_ok=True)
     reference_dir.mkdir(parents=True, exist_ok=True)
     flags, _ = save_inputs_to_files(model_path, arrays_in_order, inputs_dir)
@@ -402,7 +419,7 @@ def compile_raptor_target(
         "--pim-emit-json",
         *args.raptor_extra_arg,
     ]
-    print_step("Compile Raptor PIM", cmd, REPO)
+    print_step("Compile Raptor PIM", cmd, REPO, "Compile PIM")
     start = time.perf_counter()
     command = shell_join(cmd)
     raptor_extra_args = ["--pim-emit-json", *args.raptor_extra_arg]
@@ -446,6 +463,7 @@ def run_rust_validation(
     label: str,
     pim_dir: Path,
     config_path: Path,
+    output_bin: Path,
     outputs_desc: list[tuple[int, str, int, list[int]]],
     reference_dir: Path,
     steps: list[StepRecord],
@@ -453,7 +471,6 @@ def run_rust_validation(
     *,
     channel_last: bool = False,
 ) -> CompareResult:
-    output_bin = pim_dir.parent / "semantic_validation" / "out.bin"
     dump_ranges = build_dump_ranges(config_path, outputs_desc)
     cmd = [
         "cargo",
@@ -472,14 +489,14 @@ def run_rust_validation(
         "-d",
         dump_ranges,
     ]
-    simulation_dir = pim_dir.parent / "semantic_validation"
-    simulation_dir.mkdir(parents=True, exist_ok=True)
+    output_bin.parent.mkdir(parents=True, exist_ok=True)
     run_logged(
         label,
         cmd,
         cwd=args.pim_simulator_dir,
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Run Functional Simulation",
     )
     return compare_simulator_outputs(
         output_bin,
@@ -503,16 +520,20 @@ def compile_pimcomp(
     out_dir: Path,
     steps: list[StepRecord],
 ) -> tuple[Path, Path]:
-    out_dir.mkdir(parents=True, exist_ok=True)
-    pimcomp_output_dir = args.pimcomp_dir / "output"
+    frontend_json_dir = out_dir / "models/JSON"
+    frontend_json_dir.mkdir(parents=True, exist_ok=True)
+    runtime_config = out_dir / "config.json"
+    if args.pimcomp_config != runtime_config:
+        shutil.copy2(args.pimcomp_config, runtime_config)
+    pimcomp_output_dir = out_dir / "output"
     pimcomp_output_dir.mkdir(parents=True, exist_ok=True)
     for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt"):
         (pimcomp_output_dir / name).unlink(missing_ok=True)
     model_name = args.pimcomp_model_name or f"compare_{model_path.stem}"
-    frontend_json = args.pimcomp_dir / "models/JSON" / f"{model_name}.json"
+    frontend_json = frontend_json_dir / f"{model_name}.json"
     frontend_cmd = [
         sys.executable,
-        "frontend.py",
+        str(args.pimcomp_dir / "frontend/frontend.py"),
         "--model_path",
         str(model_path),
         "--save_path",
@@ -524,6 +545,7 @@ def compile_pimcomp(
         cwd=args.pimcomp_dir / "frontend",
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Compile PIM",
     )
     backend_cmd = [
         str(args.pimcomp_dir / "build" / "PIMCOMP-NN"),
@@ -536,12 +558,13 @@ def compile_pimcomp(
     run_logged(
         "PIMCOMP Backend",
         backend_cmd,
-        cwd=args.pimcomp_dir / "build",
+        cwd=frontend_json_dir.parent,
         timeout_sec=args.timeout_seconds,
         steps=steps,
+        stage="Compile PIM",
     )
-    copy_pimcomp_outputs(pimcomp_output_dir, out_dir)
-    return out_dir / "VerificationInfo.json", out_dir / "SimulationInfo.gz"
+    remove_tree(frontend_json_dir.parent)
+    return pimcomp_output_dir / "VerificationInfo.json", pimcomp_output_dir / "SimulationInfo.gz"
 
 
 def export_pimcomp_for_pimsim_nn(simulation_info: Path, output_dir: Path) -> Path:
@@ -836,6 +859,7 @@ def run_pimsim_nn(
         cwd=args.pimsim_nn_build_dir,
         timeout_sec=args.timeout_seconds * 10.0,
         steps=steps,
+        stage="Run Non-functional Simulation",
     )
     return parse_pimsim_nn_report(output)
 
@@ -1170,6 +1194,11 @@ def main():
     parser.add_argument("--raptor-path", default=REPO / "build_release/Release/bin/onnx-mlir", type=Path)
     parser.add_argument("--onnx-include-dir", default=REPO / "onnx-mlir/include", type=Path)
     parser.add_argument("--pimcomp-dir", default=REPO / "third_party/PIMCOMP-NN", type=Path)
+    parser.add_argument(
+        "--pimcomp-config",
+        type=Path,
+        help="PIMCOMP hardware config (default: <pimcomp-dir>/config.json).",
+    )
     parser.add_argument("--pim-simulator-dir", default=REPO / "backend-simulators/pim/pim-simulator", type=Path)
     parser.add_argument("--pimsim-nn-build-dir", default=REPO / "backend-simulators/pim/pimsim-nn/build", type=Path)
     parser.add_argument("--seed", type=int, default=0)
@@ -1213,6 +1242,12 @@ def main():
         args.pimcomp_pipeline = "element" if args.pimsim_mode == "latency" else "batch"
 
     model_path = args.model.resolve()
+    args.pimcomp_dir = args.pimcomp_dir.resolve()
+    args.pimcomp_config = (
+        args.pimcomp_config.resolve()
+        if args.pimcomp_config is not None
+        else args.pimcomp_dir / "config.json"
+    )
     out_dir = args.out_dir.resolve()
     out_dir.mkdir(parents=True, exist_ok=True)
 
@@ -1265,14 +1300,19 @@ def main():
             raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
         reference_dir = Path(reused["paths"]["reference_outputs"])
         raptor_pim_dir = Path(reused["paths"]["raptor_pim"])
-        arrays_in_order, runtime_inputs = load_saved_inputs(model_path, inputs_desc, reuse_report_path.parent / "inputs")
+        arrays_in_order, runtime_inputs = load_saved_inputs(
+            model_path,
+            inputs_desc,
+            reference_dir.parent / "inputs",
+        )
         raptor_validation = CompareResult(**reused["raptor_validation"])
         raptor_perf = reused["raptor_performance"]
         raptor_instr = reused["raptor_instruction_summary"]
         raptor_pass_timings = reused["raptor_pass_timings"]
-        print(f"\n[Reuse Raptor]\n  Report: {reuse_report_path}")
+        print_step("Reuse Raptor")
+        print(f"  Report: {reuse_report_path}")
 
-    expected_runner_path = out_dir / "runner" / "build" / "runner"
+    expected_runner_path = out_dir / "runner/build/runner"
 
     if not reuse_raptor:
         reference_compile = try_stage(
@@ -1289,7 +1329,14 @@ def main():
         else:
             if expected_runner_path.exists():
                 runner_path = expected_runner_path
-                print(f"\n[Continue] Reusing partial runner: {runner_path}")
+                print(
+                    "\n"
+                    + Style.BRIGHT
+                    + Fore.YELLOW
+                    + "[Continue]"
+                    + Style.RESET_ALL
+                    + f" Reusing partial runner: {runner_path}"
+                )
 
     if not reuse_raptor and runner_path is not None and runner_path.exists() and model_io is not None:
         generated_reference = try_stage(
@@ -1350,6 +1397,7 @@ def main():
                 "Rust Validation Raptor",
                 raptor_pim_dir,
                 raptor_pim_dir / "config.json",
+                out_dir / "simulation/out.bin",
                 outputs_desc,
                 reference_dir,
                 steps,
@@ -1370,7 +1418,7 @@ def main():
         "Prepare PIMCOMP model",
         prepare_pimcomp_model,
         model_path,
-        out_dir / "pimcomp_model",
+        out_dir / "pimcomp/model",
     )
 
     if args.reuse_pimcomp_dir is not None:
@@ -1385,7 +1433,8 @@ def main():
         if copied_pimcomp:
             verification_info = out_dir / "pimcomp/VerificationInfo.json"
             simulation_info = out_dir / "pimcomp/SimulationInfo.gz"
-            print(f"\n[Reuse PIMCOMP]\n  Directory: {reused_pimcomp_dir}")
+            print_step("Reuse PIMCOMP")
+            print(f"  Directory: {reused_pimcomp_dir}")
     else:
         compiled_pimcomp = try_stage(
             failures,
@@ -1408,7 +1457,7 @@ def main():
             verification_info,
             simulation_info,
             runtime_inputs,
-            out_dir / "pimcomp_exported",
+            out_dir / "pimcomp/exported",
         )
         if exported is not None:
             pimcomp_export_dir = exported
@@ -1433,6 +1482,7 @@ def main():
             "Rust Validation PIMCOMP",
             pimcomp_export_dir,
             pimcomp_export_dir / "config.json",
+            out_dir / "simulation/pimcomp.out.bin",
             outputs_desc,
             reference_dir,
             steps,
@@ -1494,7 +1544,7 @@ def main():
                 "Export PIMCOMP for pimsim-nn",
                 export_pimcomp_for_pimsim_nn,
                 simulation_info,
-                out_dir / "pimcomp_pimsim_nn",
+                out_dir / "pimcomp/pimsim_nn",
             )
             if pimcomp_pimsim_dir is not None:
                 perf = try_stage(
@@ -1525,7 +1575,7 @@ def main():
     else:
         pimcomp_instr = empty_instruction_summary("PIMCOMP SimulationInfo.gz is not available")
 
-    report_path = out_dir / "comparison_report.md"
+    report_path = out_dir / "pimcomp/comparison_report.md"
     write_report(
         report_path,
         model_path=model_path,
@@ -1569,22 +1619,30 @@ def main():
             "report_markdown": str(report_path),
         },
     }
-    json_path = out_dir / "comparison_report.json"
+    json_path = out_dir / "pimcomp/comparison_report.json"
     with open(json_path, "w", encoding="utf-8") as f:
         json.dump(json_report, f, indent=2)
         f.write("\n")
 
-    print(f"\n[Done]")
-    print(f"  Report: {report_path}")
-    print(f"  JSON:   {json_path}")
-    if failures or any(step.status != "passed" for step in steps):
-        print(f"  Completed with {len(failures)} recorded failure/skipped stage(s).")
-
+    failed_steps = any(step.status != "passed" for step in steps)
     semantic_failure = any(
         result.status == "done" and not result.passed
         for result in (raptor_validation, pimcomp_validation)
     )
-    if args.fail_on_error and (failures or any(step.status != "passed" for step in steps) or semantic_failure):
+    failed = bool(failures or failed_steps or semantic_failure)
+    result = "FAIL" if args.fail_on_error and failed else "DONE" if failed else "PASS"
+    color = Fore.RED if result == "FAIL" else Fore.YELLOW if result == "DONE" else Fore.GREEN
+    print("\n" + Style.BRIGHT + f"Result: {color}{result}" + Style.RESET_ALL)
+    print(f"  Report: {report_path}")
+    print(f"  JSON:   {json_path}")
+    if failures or failed_steps:
+        print(
+            Fore.YELLOW
+            + f"  Completed with {len(failures)} recorded failure/skipped stage(s)."
+            + Style.RESET_ALL
+        )
+
+    if args.fail_on_error and failed:
         raise SystemExit(1)
 
 

validation/tools/run_pimcomp_paper_latency.py

@@ -2,17 +2,18 @@
 from __future__ import annotations
 
 import argparse
-import re
 import shlex
-import shutil
 import subprocess
 import sys
 from pathlib import Path
 
+from colorama import Fore, Style
+
 
 REPO = Path(__file__).resolve().parents[2]
 SUITE = REPO / "validation/networks/pimcomp_models"
 PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
+PIMCOMP_CONFIG = REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json"
 COMPARE = REPO / "validation/tools/compare_raptor_pimcomp.py"
 MODELS = {
     "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
@@ -23,38 +24,21 @@ MODELS = {
 
 
 def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
-    print(f"$ {shlex.join(command)}", flush=True)
+    print(Fore.CYAN + "$ " + Style.RESET_ALL + shlex.join(command), flush=True)
     if dry_run:
         return 0
     return subprocess.run(command, cwd=REPO, check=check).returncode
 
 
-def prepare_pimcomp(work_dir: Path) -> None:
-    shutil.copytree(
-        PIMCOMP_SOURCE,
-        work_dir,
-        dirs_exist_ok=True,
-        ignore=shutil.ignore_patterns(".git", "build", "output"),
-    )
-    header = work_dir / "backend/GeneticAlgorithm.h"
+def validate_pimcomp_source() -> None:
+    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
     source = header.read_text(encoding="utf-8")
-    if "int population_num = 200;" not in source:
-        raise RuntimeError("PIMCOMP GA population is not 200")
-    source, replacements = re.subn(
-        r"int max_iteration = \d+;",
-        "int max_iteration = 1000;",
-        source,
-    )
-    if replacements != 1:
-        raise RuntimeError("Could not set PIMCOMP GA max_iteration")
-    header.write_text(source, encoding="utf-8")
-    shutil.copy2(
-        REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json",
-        work_dir / "config.json",
-    )
+    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
+        if setting not in source:
+            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
 
 
-def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout: float) -> list[str]:
+def comparison_command(model: Path, result_dir: Path, timeout: float) -> list[str]:
     return [
         sys.executable,
         str(COMPARE),
@@ -63,7 +47,9 @@ def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout
         "--out-dir",
         str(result_dir),
         "--pimcomp-dir",
-        str(pimcomp_dir),
+        str(PIMCOMP_SOURCE),
+        "--pimcomp-config",
+        str(PIMCOMP_CONFIG),
         "--core-count",
         "168",
         "--crossbar-count",
@@ -82,6 +68,7 @@ def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout
         "GA",
         "--timeout-seconds",
         str(timeout),
+        "--fail-on-error",
     ]
 
 
@@ -89,51 +76,53 @@ def main() -> int:
     parser = argparse.ArgumentParser(
         description="Reproduce the serial Arch-A latency comparison from the PIMCOMP paper."
     )
-    parser.add_argument("--out-dir", required=True, type=Path)
+    parser.add_argument(
+        "--out-dir",
+        type=Path,
+        help="Result root (default: artifacts beside each model under validation/).",
+    )
     parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
     parser.add_argument("--timeout-seconds", type=float, default=3600.0)
     parser.add_argument(
         "--resume",
         action="store_true",
-        help="Keep the existing work tree and skip models with a completed JSON report.",
+        help="Skip models with a completed JSON report.",
     )
     parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
     args = parser.parse_args()
 
-    out_dir = args.out_dir.resolve()
-    work_dir = out_dir / "pimcomp-ga1000"
-    if not args.dry_run and out_dir.exists() and any(out_dir.iterdir()) and not args.resume:
-        parser.error(f"{out_dir} is not empty; choose a fresh directory or pass --resume")
+    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
 
     missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
     if missing:
         parser.error(f"missing model(s): {', '.join(missing)}")
 
-    if args.dry_run:
-        print(f"# prepare isolated PIMCOMP GA build in {work_dir}")
-    else:
+    validate_pimcomp_source()
+    if out_dir is not None and not args.dry_run:
         out_dir.mkdir(parents=True, exist_ok=True)
-        prepare_pimcomp(work_dir)
 
     run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
     run(
-        ["cmake", "-S", str(work_dir), "-B", str(work_dir / "build")],
-        dry_run=args.dry_run,
-    )
-    run(
-        ["cmake", "--build", str(work_dir / "build"), "--target", "PIMCOMP-NN"],
+        ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
         dry_run=args.dry_run,
     )
 
     failed = []
     for name in args.models:
-        result_dir = out_dir / name
-        if args.resume and (result_dir / "comparison_report.json").exists():
-            print(f"[{name}] completed report exists; skipping", flush=True)
+        result_dir = out_dir / name if out_dir is not None else MODELS[name].parent
+        if args.resume and (result_dir / "pimcomp/comparison_report.json").exists():
+            print(
+                Fore.YELLOW + f"[{name}] completed report exists; skipping" + Style.RESET_ALL,
+                flush=True,
+            )
             continue
-        print(f"\n[{name}] Arch-A latency comparison", flush=True)
+        print(
+            "\n" + Fore.CYAN + f"[{name}]" + Style.RESET_ALL
+            + f" {Style.BRIGHT}Arch-A latency comparison{Style.RESET_ALL}",
+            flush=True,
+        )
         returncode = run(
-            comparison_command(MODELS[name], result_dir, work_dir, args.timeout_seconds),
+            comparison_command(MODELS[name], result_dir, args.timeout_seconds),
             dry_run=args.dry_run,
             check=False,
         )
@@ -141,8 +130,17 @@ def main() -> int:
             failed.append(name)
 
     if failed:
-        print(f"\nCompleted with failed comparisons: {', '.join(failed)}", file=sys.stderr)
+        print(
+            "\n" + Style.BRIGHT + Fore.RED + "Result: FAIL" + Style.RESET_ALL,
+            file=sys.stderr,
+        )
+        print(
+            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
+            file=sys.stderr,
+        )
         return 1
+    if not args.dry_run:
+        print("\n" + Style.BRIGHT + f"Result: {Fore.GREEN}PASS" + Style.RESET_ALL)
     return 0