diff --git a/third_party/PIMCOMP-NN b/third_party/PIMCOMP-NN index 2750c3d..5095788 160000 --- a/third_party/PIMCOMP-NN +++ b/third_party/PIMCOMP-NN @@ -1 +1 @@ -Subproject commit 2750c3d87327af81e3be8e2b56b99e4b85ace9eb +Subproject commit 5095788c00f4c2156fa0cd7f95309c962621dba5 diff --git a/validation/.gitignore b/validation/.gitignore index dd52fc0..9d556a1 100644 --- a/validation/.gitignore +++ b/validation/.gitignore @@ -6,6 +6,7 @@ operations/**/simulation networks/**/inputs networks/**/outputs networks/**/raptor +networks/**/pimcomp networks/**/runner networks/**/simulation networks/**/real_image_val diff --git a/validation/README.md b/validation/README.md index cd10b9f..0902dc3 100644 --- a/validation/README.md +++ b/validation/README.md @@ -60,6 +60,27 @@ Validate a network or network slice: `--operations-dir` may point to any directory tree containing `.onnx` files. The script discovers them recursively. +## Raptor vs PIMCOMP comparison + +The PIMCOMP paper-model suite has a one-command Arch-A comparison: + +```bash +.venv/bin/python validation/tools/run_pimcomp_paper_latency.py +``` + +The runner verifies PIMCOMP's population-200, 1000-iteration GA settings, +builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, and compares +the four paper models one at a time. Each PIMCOMP GA run evaluates candidates +in parallel; set `OMP_NUM_THREADS` to control its worker count. Use +`--models vgg8` for one model or `--dry-run` to print the commands. + +Generated artifacts are stored beside each model and ignored by Git. The +comparison reuses the same model-level `inputs/`, `outputs/`, `runner/`, +`raptor/`, and `simulation/` paths as regular validation. PIMCOMP-only +artifacts and `comparison_report.{md,json}` live under `pimcomp/`. See +[`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for +profiles, model provenance, limitations, and remote execution. + ## Validation modes The default mode performs the complete workflow. @@ -167,7 +188,8 @@ Artifacts are written beside each model: | `outputs/` | ONNX-MLIR reference output CSV files. | | `raptor/` | Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts. | | `runner/` | Generated reference runner source, build tree, and shared library. | -| `simulation/out.bin` | Functional simulator output used for comparison. | +| `simulation/` | Functional simulator outputs used for numerical comparison. | +| `pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. | The `raptor/` directory may include `spatial0.mlir`, `spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`, diff --git a/validation/networks/pimcomp_models/README.md b/validation/networks/pimcomp_models/README.md index 2decac1..73d3f95 100644 --- a/validation/networks/pimcomp_models/README.md +++ b/validation/networks/pimcomp_models/README.md @@ -136,12 +136,11 @@ random placement code occasionally segfaults; an unchanged retry succeeded in the observed cases. The paper's optimizer uses a genetic algorithm with population 200 and up to -1000 iterations. Select it with `-r=GA` for optimizer studies. The released -source keeps population 200 but sets `max_iteration = 3`, so reproducing the -paper's optimization search also requires changing that value in -`backend/GeneticAlgorithm.h`. GA allocates roughly 32 GB in its fast evaluator; -use monolith below instead of reducing cores or crossbars when local RAM is -insufficient. +1000 iterations. The checked-out PIMCOMP submodule already has both paper +settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness +evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS` +to control its parallelism; otherwise OpenMP uses the available CPUs. The GA +uses the fixed seed `1`, so repeated serial and parallel runs are reproducible. ## Compare Raptor and PIMCOMP @@ -150,34 +149,37 @@ compiles both instruction streams, runs both through `pimsim-nn`, validates Raptor through the Rust simulator, and writes Markdown and JSON reports. PIMCOMP Rust validation also runs when its optional exporter is available. -To reproduce the complete Arch-A latency experiment, use the serial experiment -runner. It creates an isolated PIMCOMP build with population 200 and 1000 GA -iterations, runs only the `element`/batch-1 latency pipeline, and invokes the -comparison driver for one model at a time: +To reproduce the complete Arch-A latency experiment, use the model-by-model +runner. It verifies the paper GA settings, builds Raptor and the existing +`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison +for one model at a time: ```bash -.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \ - --out-dir /tmp/raptor-pimcomp-paper-latency +.venv/bin/python validation/tools/run_pimcomp_paper_latency.py ``` -Reports are written under `//comparison_report.{md,json}`. -Use `--models vgg8` to run one model, `--resume` after an interruption, or -`--dry-run` to inspect every command. The runner continues after a failed model -so all reports are produced. +Each model directory reuses regular validation's ignored `inputs/`, `outputs/`, +`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and +`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates +one isolated `models/JSON/` graph because PIMCOMP requires that relative +layout; it is removed after a successful backend run and the shared submodule +model directory is never modified. Models requiring BatchNormalization folding +also receive a prepared ONNX file; other models use the original ONNX directly. +PIMCOMP's source tree and build directory remain unchanged at runtime. Use +`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run` +to inspect every command, or `--out-dir PATH` to keep results outside +`validation/`. The runner continues after a failed model so all reports are +produced. Arch-A low-latency example: ```bash RAPTOR_ROOT=$PWD -PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN" -CONFIG_BACKUP=$(mktemp) -cp "$PIMCOMP/config.json" "$CONFIG_BACKUP" -trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT -cp "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" "$PIMCOMP/config.json" "$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/compare_raptor_pimcomp.py" \ --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \ - --out-dir /tmp/compare-resnet34-arch-a-ll \ + --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \ + --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \ --core-count 168 \ --crossbar-count 96 \ --crossbar-size 128 \ @@ -196,11 +198,9 @@ If only semantic and instruction comparison is required, add `--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both semantic validations with maximum output differences below `5e-10`. -The current PIMCOMP-NN submodule does not include the optional -`verification/export_to_pim_simulator.py` helper. The driver therefore records -PIMCOMP Rust semantic validation as skipped, while PIMCOMP compilation, -instruction reporting, and `pimsim-nn` latency still run. Use -`--fail-on-error` only when that semantic export helper is available. +The comparison runner enables `--fail-on-error`, so a failed compiler, +simulation, or semantic validation makes the command fail while preserving the +generated report. Current Raptor status: @@ -247,14 +247,13 @@ python3 -m venv .venv .venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama # Run every latency comparison serially. -.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \ - --out-dir /tmp/raptor-pimcomp-paper-latency +.venv/bin/python validation/tools/run_pimcomp_paper_latency.py ``` Copy reports back without transferring large compiler artifacts: ```bash rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \ - monolith:/tmp/raptor-pimcomp-paper-latency/ \ - /tmp/raptor-pimcomp-paper-latency/ + "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \ + validation/networks/pimcomp_models/ ``` diff --git a/validation/networks/pimcomp_models/RESULTS.md b/validation/networks/pimcomp_models/RESULTS.md index 8d53254..c1d8738 100644 --- a/validation/networks/pimcomp_models/RESULTS.md +++ b/validation/networks/pimcomp_models/RESULTS.md @@ -25,7 +25,6 @@ No throughput experiment was run. ```bash .venv/bin/python validation/tools/run_pimcomp_paper_latency.py \ - --out-dir /tmp/raptor-pimcomp-paper-latency \ --models googlenet ``` diff --git a/validation/raptor_validation/raptor.py b/validation/raptor_validation/raptor.py index 0ec8529..f23a549 100644 --- a/validation/raptor_validation/raptor.py +++ b/validation/raptor_validation/raptor.py @@ -1,5 +1,6 @@ import re import shlex +import shutil import subprocess from pathlib import Path from colorama import Fore, Style @@ -43,6 +44,10 @@ def _format_command(cmd): def compile_with_raptor(network_path, raptor_onnx_path: Path, output_base: Path, crossbar_size, crossbar_count, core_count, raptor_extra_args, cwd, verbose, reporter, timeout_sec): + pim_dir = Path(output_base).parent / "pim" + if pim_dir.exists(): + shutil.rmtree(pim_dir) + # Define the arguments, with the possibility to set crossbar size and count args = [ network_path, diff --git a/validation/raptor_validation/validate_one.py b/validation/raptor_validation/validate_one.py index 262a48d..3420758 100644 --- a/validation/raptor_validation/validate_one.py +++ b/validation/raptor_validation/validate_one.py @@ -23,6 +23,16 @@ STAGE_TITLES = ( "Compare Outputs", "Run Non-functional Simulation", ) +STAGE_COLORS = { + STAGE_TITLES[0]: Fore.BLUE, + STAGE_TITLES[1]: Fore.MAGENTA, + STAGE_TITLES[2]: Fore.YELLOW, + STAGE_TITLES[3]: Fore.GREEN, + STAGE_TITLES[4]: Fore.CYAN, + STAGE_TITLES[5]: Fore.MAGENTA, + STAGE_TITLES[6]: Fore.YELLOW, + STAGE_TITLES[7]: Fore.BLUE, +} STAGE_COUNT = len(STAGE_TITLES) GENERATED_DIR_NAMES = ("inputs", "outputs", "raptor", "runner", "simulation") @@ -294,17 +304,7 @@ def clean_workspace_artifacts(workspace_dir, model_stem): def print_stage(reporter, model_index, model_total, model_name, title): - stage_colors = { - STAGE_TITLES[0]: Fore.BLUE, - STAGE_TITLES[1]: Fore.MAGENTA, - STAGE_TITLES[2]: Fore.YELLOW, - STAGE_TITLES[3]: Fore.GREEN, - STAGE_TITLES[4]: Fore.CYAN, - STAGE_TITLES[5]: Fore.MAGENTA, - STAGE_TITLES[6]: Fore.YELLOW, - STAGE_TITLES[7]: Fore.BLUE, - } - color = stage_colors.get(title, Fore.WHITE) + color = STAGE_COLORS.get(title, Fore.WHITE) reporter.log(Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL) reporter.set_stage(model_index, model_total, model_name, title) diff --git a/validation/tools/compare_raptor_pimcomp.py b/validation/tools/compare_raptor_pimcomp.py index dd743c7..817b985 100755 --- a/validation/tools/compare_raptor_pimcomp.py +++ b/validation/tools/compare_raptor_pimcomp.py @@ -21,6 +21,7 @@ from typing import Any import numpy as np import onnx +from colorama import Fore, Style REPO = Path(__file__).resolve().parents[2] @@ -37,7 +38,11 @@ from raptor_validation.onnx_utils import ( # noqa: E402 write_inputs_to_memory_bin, ) from raptor_validation.raptor import compile_with_raptor # noqa: E402 -from raptor_validation.validate_one import build_dump_ranges, parse_pim_simulator_outputs # noqa: E402 +from raptor_validation.validate_one import ( # noqa: E402 + STAGE_COLORS, + build_dump_ranges, + parse_pim_simulator_outputs, +) @dataclass @@ -73,8 +78,14 @@ def shell_join(cmd: list[str]) -> str: return shlex.join(str(arg) for arg in cmd) -def print_step(name: str, cmd: list[str] | None = None, cwd: Path | None = None): - print(f"\n[{name}]") +def print_step( + name: str, + cmd: list[str] | None = None, + cwd: Path | None = None, + stage: str | None = None, +): + color = STAGE_COLORS.get(stage or name, Fore.WHITE) + print("\n" + Style.BRIGHT + color + f"[{name}]" + Style.RESET_ALL) if cmd is not None: print(f" cwd: {cwd or REPO}") print(f" $ {shell_join(cmd)}") @@ -108,9 +119,12 @@ def exception_message(exc: BaseException) -> str: def print_failure(name: str, exc: BaseException | str) -> None: message = exc if isinstance(exc, str) else exception_message(exc) - print(f"\n[{name} FAILED]") + print( + "\n" + Style.BRIGHT + Fore.RED + f"[{name} FAILED]" + Style.RESET_ALL, + file=sys.stderr, + ) for line in message.splitlines()[:20]: - print(f" {line}") + print(Fore.RED + f" {line}" + Style.RESET_ALL, file=sys.stderr) def run_logged( @@ -120,8 +134,9 @@ def run_logged( cwd: Path, timeout_sec: float, steps: list[StepRecord], + stage: str | None = None, ) -> str: - print_step(name, cmd, cwd) + print_step(name, cmd, cwd, stage) start = time.perf_counter() command = shell_join(cmd) try: @@ -202,29 +217,29 @@ def load_saved_inputs( def prepare_pimcomp_model(model_path: Path, out_dir: Path) -> Path: + model = onnx.load(model_path) + if not any(node.op_type == "BatchNormalization" for node in model.graph.node): + return model_path + out_dir.mkdir(parents=True, exist_ok=True) output_path = out_dir / f"{model_path.stem}_pimcomp.onnx" - model = onnx.load(model_path) + from onnxsim import simplify + + model, equivalent = simplify(model, check_n=1) + if not equivalent: + raise RuntimeError("Conv+BatchNormalization folding changed the model output") if any(node.op_type == "BatchNormalization" for node in model.graph.node): - from onnxsim import simplify + import onnxruntime as ort - model, equivalent = simplify(model, check_n=1) - if not equivalent: - raise RuntimeError("Conv+BatchNormalization folding changed the model output") + options = ort.SessionOptions() + options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC + options.optimized_model_filepath = str(output_path) + ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"]) + model = onnx.load(output_path) if any(node.op_type == "BatchNormalization" for node in model.graph.node): - import onnxruntime as ort - - options = ort.SessionOptions() - options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC - options.optimized_model_filepath = str(output_path) - ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"]) - model = onnx.load(output_path) - if any(node.op_type == "BatchNormalization" for node in model.graph.node): - raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization") - else: - onnx.save(model, output_path) + raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization") else: - shutil.copy2(model_path, output_path) + onnx.save(model, output_path) return output_path @@ -259,8 +274,7 @@ def sanitize_output_name(name: str) -> str: def load_effective_hardware(args: argparse.Namespace) -> dict[str, int]: - config_path = args.pimcomp_dir / "config.json" - with open(config_path, "r", encoding="utf-8") as f: + with open(args.pimcomp_config, "r", encoding="utf-8") as f: config = json.load(f) rows, cols = config["chip_config"]["network_config"]["layout"] xbar_h, xbar_w = config["chip_config"]["core_config"]["matrix_config"]["xbar_size"] @@ -304,7 +318,7 @@ def compile_reference( work_dir: Path, steps: list[StepRecord], ) -> Path: - raptor_dir = work_dir / "reference" + raptor_dir = work_dir / "raptor" runner_dir = work_dir / "runner" build_dir = runner_dir / "build" raptor_dir.mkdir(parents=True, exist_ok=True) @@ -320,6 +334,7 @@ def compile_reference( cwd=REPO, timeout_sec=args.timeout_seconds, steps=steps, + stage="Compile ONNX", ) run_logged( "Reference Native Compile", @@ -327,10 +342,11 @@ def compile_reference( cwd=REPO, timeout_sec=args.timeout_seconds, steps=steps, + stage="Compile ONNX", ) network_so = runner_base.with_suffix(".so") - print_step("Generate Runner Source") + print_step("Generate Runner Source", stage="Build Runner") gen_network_runner( model_path, network_so, @@ -346,6 +362,7 @@ def compile_reference( cwd=build_dir, timeout_sec=args.timeout_seconds, steps=steps, + stage="Build Runner", ) run_logged( "Build Runner", @@ -367,7 +384,7 @@ def generate_reference_outputs( out_dir: Path, ) -> Path: inputs_dir = out_dir / "inputs" - reference_dir = out_dir / "reference_outputs" + reference_dir = out_dir / "outputs" inputs_dir.mkdir(parents=True, exist_ok=True) reference_dir.mkdir(parents=True, exist_ok=True) flags, _ = save_inputs_to_files(model_path, arrays_in_order, inputs_dir) @@ -402,7 +419,7 @@ def compile_raptor_target( "--pim-emit-json", *args.raptor_extra_arg, ] - print_step("Compile Raptor PIM", cmd, REPO) + print_step("Compile Raptor PIM", cmd, REPO, "Compile PIM") start = time.perf_counter() command = shell_join(cmd) raptor_extra_args = ["--pim-emit-json", *args.raptor_extra_arg] @@ -446,6 +463,7 @@ def run_rust_validation( label: str, pim_dir: Path, config_path: Path, + output_bin: Path, outputs_desc: list[tuple[int, str, int, list[int]]], reference_dir: Path, steps: list[StepRecord], @@ -453,7 +471,6 @@ def run_rust_validation( *, channel_last: bool = False, ) -> CompareResult: - output_bin = pim_dir.parent / "semantic_validation" / "out.bin" dump_ranges = build_dump_ranges(config_path, outputs_desc) cmd = [ "cargo", @@ -472,14 +489,14 @@ def run_rust_validation( "-d", dump_ranges, ] - simulation_dir = pim_dir.parent / "semantic_validation" - simulation_dir.mkdir(parents=True, exist_ok=True) + output_bin.parent.mkdir(parents=True, exist_ok=True) run_logged( label, cmd, cwd=args.pim_simulator_dir, timeout_sec=args.timeout_seconds, steps=steps, + stage="Run Functional Simulation", ) return compare_simulator_outputs( output_bin, @@ -503,16 +520,20 @@ def compile_pimcomp( out_dir: Path, steps: list[StepRecord], ) -> tuple[Path, Path]: - out_dir.mkdir(parents=True, exist_ok=True) - pimcomp_output_dir = args.pimcomp_dir / "output" + frontend_json_dir = out_dir / "models/JSON" + frontend_json_dir.mkdir(parents=True, exist_ok=True) + runtime_config = out_dir / "config.json" + if args.pimcomp_config != runtime_config: + shutil.copy2(args.pimcomp_config, runtime_config) + pimcomp_output_dir = out_dir / "output" pimcomp_output_dir.mkdir(parents=True, exist_ok=True) for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt"): (pimcomp_output_dir / name).unlink(missing_ok=True) model_name = args.pimcomp_model_name or f"compare_{model_path.stem}" - frontend_json = args.pimcomp_dir / "models/JSON" / f"{model_name}.json" + frontend_json = frontend_json_dir / f"{model_name}.json" frontend_cmd = [ sys.executable, - "frontend.py", + str(args.pimcomp_dir / "frontend/frontend.py"), "--model_path", str(model_path), "--save_path", @@ -524,6 +545,7 @@ def compile_pimcomp( cwd=args.pimcomp_dir / "frontend", timeout_sec=args.timeout_seconds, steps=steps, + stage="Compile PIM", ) backend_cmd = [ str(args.pimcomp_dir / "build" / "PIMCOMP-NN"), @@ -536,12 +558,13 @@ def compile_pimcomp( run_logged( "PIMCOMP Backend", backend_cmd, - cwd=args.pimcomp_dir / "build", + cwd=frontend_json_dir.parent, timeout_sec=args.timeout_seconds, steps=steps, + stage="Compile PIM", ) - copy_pimcomp_outputs(pimcomp_output_dir, out_dir) - return out_dir / "VerificationInfo.json", out_dir / "SimulationInfo.gz" + remove_tree(frontend_json_dir.parent) + return pimcomp_output_dir / "VerificationInfo.json", pimcomp_output_dir / "SimulationInfo.gz" def export_pimcomp_for_pimsim_nn(simulation_info: Path, output_dir: Path) -> Path: @@ -836,6 +859,7 @@ def run_pimsim_nn( cwd=args.pimsim_nn_build_dir, timeout_sec=args.timeout_seconds * 10.0, steps=steps, + stage="Run Non-functional Simulation", ) return parse_pimsim_nn_report(output) @@ -1170,6 +1194,11 @@ def main(): parser.add_argument("--raptor-path", default=REPO / "build_release/Release/bin/onnx-mlir", type=Path) parser.add_argument("--onnx-include-dir", default=REPO / "onnx-mlir/include", type=Path) parser.add_argument("--pimcomp-dir", default=REPO / "third_party/PIMCOMP-NN", type=Path) + parser.add_argument( + "--pimcomp-config", + type=Path, + help="PIMCOMP hardware config (default: /config.json).", + ) parser.add_argument("--pim-simulator-dir", default=REPO / "backend-simulators/pim/pim-simulator", type=Path) parser.add_argument("--pimsim-nn-build-dir", default=REPO / "backend-simulators/pim/pimsim-nn/build", type=Path) parser.add_argument("--seed", type=int, default=0) @@ -1213,6 +1242,12 @@ def main(): args.pimcomp_pipeline = "element" if args.pimsim_mode == "latency" else "batch" model_path = args.model.resolve() + args.pimcomp_dir = args.pimcomp_dir.resolve() + args.pimcomp_config = ( + args.pimcomp_config.resolve() + if args.pimcomp_config is not None + else args.pimcomp_dir / "config.json" + ) out_dir = args.out_dir.resolve() out_dir.mkdir(parents=True, exist_ok=True) @@ -1265,14 +1300,19 @@ def main(): raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}") reference_dir = Path(reused["paths"]["reference_outputs"]) raptor_pim_dir = Path(reused["paths"]["raptor_pim"]) - arrays_in_order, runtime_inputs = load_saved_inputs(model_path, inputs_desc, reuse_report_path.parent / "inputs") + arrays_in_order, runtime_inputs = load_saved_inputs( + model_path, + inputs_desc, + reference_dir.parent / "inputs", + ) raptor_validation = CompareResult(**reused["raptor_validation"]) raptor_perf = reused["raptor_performance"] raptor_instr = reused["raptor_instruction_summary"] raptor_pass_timings = reused["raptor_pass_timings"] - print(f"\n[Reuse Raptor]\n Report: {reuse_report_path}") + print_step("Reuse Raptor") + print(f" Report: {reuse_report_path}") - expected_runner_path = out_dir / "runner" / "build" / "runner" + expected_runner_path = out_dir / "runner/build/runner" if not reuse_raptor: reference_compile = try_stage( @@ -1289,7 +1329,14 @@ def main(): else: if expected_runner_path.exists(): runner_path = expected_runner_path - print(f"\n[Continue] Reusing partial runner: {runner_path}") + print( + "\n" + + Style.BRIGHT + + Fore.YELLOW + + "[Continue]" + + Style.RESET_ALL + + f" Reusing partial runner: {runner_path}" + ) if not reuse_raptor and runner_path is not None and runner_path.exists() and model_io is not None: generated_reference = try_stage( @@ -1350,6 +1397,7 @@ def main(): "Rust Validation Raptor", raptor_pim_dir, raptor_pim_dir / "config.json", + out_dir / "simulation/out.bin", outputs_desc, reference_dir, steps, @@ -1370,7 +1418,7 @@ def main(): "Prepare PIMCOMP model", prepare_pimcomp_model, model_path, - out_dir / "pimcomp_model", + out_dir / "pimcomp/model", ) if args.reuse_pimcomp_dir is not None: @@ -1385,7 +1433,8 @@ def main(): if copied_pimcomp: verification_info = out_dir / "pimcomp/VerificationInfo.json" simulation_info = out_dir / "pimcomp/SimulationInfo.gz" - print(f"\n[Reuse PIMCOMP]\n Directory: {reused_pimcomp_dir}") + print_step("Reuse PIMCOMP") + print(f" Directory: {reused_pimcomp_dir}") else: compiled_pimcomp = try_stage( failures, @@ -1408,7 +1457,7 @@ def main(): verification_info, simulation_info, runtime_inputs, - out_dir / "pimcomp_exported", + out_dir / "pimcomp/exported", ) if exported is not None: pimcomp_export_dir = exported @@ -1433,6 +1482,7 @@ def main(): "Rust Validation PIMCOMP", pimcomp_export_dir, pimcomp_export_dir / "config.json", + out_dir / "simulation/pimcomp.out.bin", outputs_desc, reference_dir, steps, @@ -1494,7 +1544,7 @@ def main(): "Export PIMCOMP for pimsim-nn", export_pimcomp_for_pimsim_nn, simulation_info, - out_dir / "pimcomp_pimsim_nn", + out_dir / "pimcomp/pimsim_nn", ) if pimcomp_pimsim_dir is not None: perf = try_stage( @@ -1525,7 +1575,7 @@ def main(): else: pimcomp_instr = empty_instruction_summary("PIMCOMP SimulationInfo.gz is not available") - report_path = out_dir / "comparison_report.md" + report_path = out_dir / "pimcomp/comparison_report.md" write_report( report_path, model_path=model_path, @@ -1569,22 +1619,30 @@ def main(): "report_markdown": str(report_path), }, } - json_path = out_dir / "comparison_report.json" + json_path = out_dir / "pimcomp/comparison_report.json" with open(json_path, "w", encoding="utf-8") as f: json.dump(json_report, f, indent=2) f.write("\n") - print(f"\n[Done]") - print(f" Report: {report_path}") - print(f" JSON: {json_path}") - if failures or any(step.status != "passed" for step in steps): - print(f" Completed with {len(failures)} recorded failure/skipped stage(s).") - + failed_steps = any(step.status != "passed" for step in steps) semantic_failure = any( result.status == "done" and not result.passed for result in (raptor_validation, pimcomp_validation) ) - if args.fail_on_error and (failures or any(step.status != "passed" for step in steps) or semantic_failure): + failed = bool(failures or failed_steps or semantic_failure) + result = "FAIL" if args.fail_on_error and failed else "DONE" if failed else "PASS" + color = Fore.RED if result == "FAIL" else Fore.YELLOW if result == "DONE" else Fore.GREEN + print("\n" + Style.BRIGHT + f"Result: {color}{result}" + Style.RESET_ALL) + print(f" Report: {report_path}") + print(f" JSON: {json_path}") + if failures or failed_steps: + print( + Fore.YELLOW + + f" Completed with {len(failures)} recorded failure/skipped stage(s)." + + Style.RESET_ALL + ) + + if args.fail_on_error and failed: raise SystemExit(1) diff --git a/validation/tools/run_pimcomp_paper_latency.py b/validation/tools/run_pimcomp_paper_latency.py old mode 100644 new mode 100755 index 2d7726c..4b54dae --- a/validation/tools/run_pimcomp_paper_latency.py +++ b/validation/tools/run_pimcomp_paper_latency.py @@ -2,17 +2,18 @@ from __future__ import annotations import argparse -import re import shlex -import shutil import subprocess import sys from pathlib import Path +from colorama import Fore, Style + REPO = Path(__file__).resolve().parents[2] SUITE = REPO / "validation/networks/pimcomp_models" PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN" +PIMCOMP_CONFIG = REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json" COMPARE = REPO / "validation/tools/compare_raptor_pimcomp.py" MODELS = { "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx", @@ -23,38 +24,21 @@ MODELS = { def run(command: list[str], *, dry_run: bool, check: bool = True) -> int: - print(f"$ {shlex.join(command)}", flush=True) + print(Fore.CYAN + "$ " + Style.RESET_ALL + shlex.join(command), flush=True) if dry_run: return 0 return subprocess.run(command, cwd=REPO, check=check).returncode -def prepare_pimcomp(work_dir: Path) -> None: - shutil.copytree( - PIMCOMP_SOURCE, - work_dir, - dirs_exist_ok=True, - ignore=shutil.ignore_patterns(".git", "build", "output"), - ) - header = work_dir / "backend/GeneticAlgorithm.h" +def validate_pimcomp_source() -> None: + header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h" source = header.read_text(encoding="utf-8") - if "int population_num = 200;" not in source: - raise RuntimeError("PIMCOMP GA population is not 200") - source, replacements = re.subn( - r"int max_iteration = \d+;", - "int max_iteration = 1000;", - source, - ) - if replacements != 1: - raise RuntimeError("Could not set PIMCOMP GA max_iteration") - header.write_text(source, encoding="utf-8") - shutil.copy2( - REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json", - work_dir / "config.json", - ) + for setting in ("int population_num = 200;", "int max_iteration = 1000;"): + if setting not in source: + raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}") -def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout: float) -> list[str]: +def comparison_command(model: Path, result_dir: Path, timeout: float) -> list[str]: return [ sys.executable, str(COMPARE), @@ -63,7 +47,9 @@ def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout "--out-dir", str(result_dir), "--pimcomp-dir", - str(pimcomp_dir), + str(PIMCOMP_SOURCE), + "--pimcomp-config", + str(PIMCOMP_CONFIG), "--core-count", "168", "--crossbar-count", @@ -82,6 +68,7 @@ def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout "GA", "--timeout-seconds", str(timeout), + "--fail-on-error", ] @@ -89,51 +76,53 @@ def main() -> int: parser = argparse.ArgumentParser( description="Reproduce the serial Arch-A latency comparison from the PIMCOMP paper." ) - parser.add_argument("--out-dir", required=True, type=Path) + parser.add_argument( + "--out-dir", + type=Path, + help="Result root (default: artifacts beside each model under validation/).", + ) parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS)) parser.add_argument("--timeout-seconds", type=float, default=3600.0) parser.add_argument( "--resume", action="store_true", - help="Keep the existing work tree and skip models with a completed JSON report.", + help="Skip models with a completed JSON report.", ) parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.") args = parser.parse_args() - out_dir = args.out_dir.resolve() - work_dir = out_dir / "pimcomp-ga1000" - if not args.dry_run and out_dir.exists() and any(out_dir.iterdir()) and not args.resume: - parser.error(f"{out_dir} is not empty; choose a fresh directory or pass --resume") + out_dir = args.out_dir.resolve() if args.out_dir is not None else None missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()] if missing: parser.error(f"missing model(s): {', '.join(missing)}") - if args.dry_run: - print(f"# prepare isolated PIMCOMP GA build in {work_dir}") - else: + validate_pimcomp_source() + if out_dir is not None and not args.dry_run: out_dir.mkdir(parents=True, exist_ok=True) - prepare_pimcomp(work_dir) run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run) run( - ["cmake", "-S", str(work_dir), "-B", str(work_dir / "build")], - dry_run=args.dry_run, - ) - run( - ["cmake", "--build", str(work_dir / "build"), "--target", "PIMCOMP-NN"], + ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"], dry_run=args.dry_run, ) failed = [] for name in args.models: - result_dir = out_dir / name - if args.resume and (result_dir / "comparison_report.json").exists(): - print(f"[{name}] completed report exists; skipping", flush=True) + result_dir = out_dir / name if out_dir is not None else MODELS[name].parent + if args.resume and (result_dir / "pimcomp/comparison_report.json").exists(): + print( + Fore.YELLOW + f"[{name}] completed report exists; skipping" + Style.RESET_ALL, + flush=True, + ) continue - print(f"\n[{name}] Arch-A latency comparison", flush=True) + print( + "\n" + Fore.CYAN + f"[{name}]" + Style.RESET_ALL + + f" {Style.BRIGHT}Arch-A latency comparison{Style.RESET_ALL}", + flush=True, + ) returncode = run( - comparison_command(MODELS[name], result_dir, work_dir, args.timeout_seconds), + comparison_command(MODELS[name], result_dir, args.timeout_seconds), dry_run=args.dry_run, check=False, ) @@ -141,8 +130,17 @@ def main() -> int: failed.append(name) if failed: - print(f"\nCompleted with failed comparisons: {', '.join(failed)}", file=sys.stderr) + print( + "\n" + Style.BRIGHT + Fore.RED + "Result: FAIL" + Style.RESET_ALL, + file=sys.stderr, + ) + print( + Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL, + file=sys.stderr, + ) return 1 + if not args.dry_run: + print("\n" + Style.BRIGHT + f"Result: {Fore.GREEN}PASS" + Style.RESET_ALL) return 0