more complete pimcomp comparison scripts

update pimsim-nn submodule
2026-08-06 21:49:54 +02:00
parent 4acd3b0c81
commit 2e76164aed
20 changed files with 1626 additions and 548 deletions
@@ -63,22 +63,24 @@ that directory while retaining the terminal table.
 ## Raptor vs PIMCOMP comparison
-The PIMCOMP paper-model suite has a one-command Arch-A comparison:
+The PIMCOMP paper-model suite has a one-command multi-architecture comparison:
 ```bash
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
 The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
 builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, and compares
-the four paper models one at a time. Each PIMCOMP GA run evaluates candidates
+the four paper models in parallel. Set `--jobs` to control comparison workers;
-in parallel; set `OMP_NUM_THREADS` to control its worker count. Use
+the runner leaves `OMP_NUM_THREADS` at its environment default. Use
 `--models vgg8` for one model or `--dry-run` to print the commands.
 Generated artifacts are stored beside each model and ignored by Git. The
-comparison reuses the same model-level `inputs/`, `outputs/`, `runner/`,
+comparison reuses model-level `common/inputs/`, `common/outputs/`,
-`raptor/`, and `simulation/` paths as regular validation. PIMCOMP-only
+`common/runner/` across architectures, modes, and pipelines. Pimsim configs and
-artifacts and `comparison_report.{md,json}` live under `pimcomp/`. See
+network meshes are referenced directly from
 `validation/pimsim_configs/pimcomp/`. Architecture- and pipeline-specific `raptor/`, `simulation/`, and
 PIMCOMP artifacts stay under each comparison directory. See
 [`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for
 profiles, model provenance, limitations, and remote execution.
@@ -164,7 +166,9 @@ The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
 prints the incompatible values; functional validation still runs.
 The checked-in profiles are under
-`validation/pimsim_configs/pimcomp/<profile>/latency_config.json`.
+`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
 `latency_config.json`, while throughput uses
 `throughput_config_<time>ms.json` and the mesh beside it.
 Use `--skip-non-functional-simulation` when latency, power, and energy are not required.
 The summary reports non-functional results as measured, failed, unsupported, or
@@ -185,14 +189,14 @@ Artifacts are written beside each model:
 | Path | Contents |
 |---|---|
-| `inputs/` | Generated input CSV files. |
+| `common/inputs/` | Shared generated input CSV files. |
-| `outputs/` | ONNX-MLIR reference output CSV files. |
+| `common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
-| `raptor/` | Exported MLIR, dialect snapshots, reports, final FP32 `pim/` artifacts, and the int8-equivalent `pimsim_nn/` latency view. |
+| `common/runner/` | Shared reference runner source, build tree, and library. |
-| `runner/` | Generated reference runner source, build tree, and shared library. |
+| `<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and PIM artifacts. |
-| `simulation/` | Functional simulator outputs used for numerical comparison. |
+| `<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
-| `pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
+| `<arch>/<mode>[/pipelineN]/pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
-The `raptor/` directory may include `spatial0.mlir`,
+Each comparison's `raptor/` directory may include `spatial0.mlir`,
 `spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
 `spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
 `pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.
@@ -5,8 +5,12 @@ This directory contains the four networks evaluated in
 VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
 additional compiler comparison model.
-See the runner-generated [results.csv](results.csv) for the current latency
+See the runner-generated [results.csv](results.csv) for the current comparison
-and energy results.
+results. Rows are retained separately for each model, architecture, mode, and
 pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
 the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
 Running the runner with `--arch arch-b` or `--arch arch-c` appends those
 architecture rows without replacing the existing `arch-a` entries.
 ## Models and provenance
@@ -18,11 +22,11 @@ and energy results.
 | `vgg8/`      | VGG-8 reconstruction | `1x1x28x28`   | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers.                                                            |
 | `yolo11n/`   | YOLO11n detection    | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
-`googlenet/googlenet-12-latency.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
+`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
 It removes the two LRN nodes and terminal Softmax from the original model,
 so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
-`yolo11n/yolo11n-latency.onnx` is the explicit pimsim-nn-ready YOLO11n model.
+`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
 It removes the Softmax nodes from the original model,
 so that the compiled artifact can be simulated in pimsim-nn.
@@ -98,9 +102,9 @@ Current SHA-256 checksums:
 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f  resnet18/resnet18-v1-7.onnx
 c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096  resnet34/resnet34-v1-7.onnx
 c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e  googlenet/googlenet-12.onnx
-a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982  googlenet/googlenet-12-latency.onnx
+a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982  googlenet/googlenet-12-pimsim-nn.onnx
 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef  vgg8/vgg8-mnist-reconstructed.onnx
-229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f  yolo11n/yolo11n-latency.onnx
+229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f  yolo11n/yolo11n-pimsim-nn.onnx
 ```
 ## Paper hardware profiles
@@ -109,8 +113,8 @@ The files in
 [`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
 encode Table V's explicit resource parameters.
 Each profile subdirectory contains pre-generated latency and throughput
-`pimsim-nn` configs plus its matching mesh; comparison and validation select
+`pimsim-nn` configs plus its matching mesh; comparison and validation reference
-these checked-in artifacts without generating configs at runtime.
+these canonical artifacts directly.
 | Config                       |             Cores | Crossbars/core | Crossbar   |  Cell | PIMCOMP layout  |
 |------------------------------|------------------:|---------------:|------------|------:|-----------------|
@@ -210,36 +214,57 @@ compiles both instruction streams, runs both through `pimsim-nn`, runs
 functional validation through `pim-simulator`, and writes Markdown and JSON
 reports.
-To reproduce the complete Arch-A latency experiment, use the model-by-model
+To reproduce all configured architectures and both latency/throughput modes,
-runner. It verifies the paper GA settings, builds Raptor and the existing
+use the model-by-model runner. It verifies the paper GA settings, builds Raptor
-`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
+and the existing `third_party/PIMCOMP-NN/build` tree, then runs the comparisons
-for one model at a time and regenerates `results.csv` from the JSON reports:
+in parallel and regenerates `results.csv` from the JSON reports:
 ```bash
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
-Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
+Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
-`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and
+
-`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates
+Each model directory has a shared ignored `common/` directory containing
 `inputs/`, `outputs/`, and the native `runner/`.
 Pimsim-nn configs and network meshes remain canonical under
 `validation/pimsim_configs/pimcomp/` and are referenced in place.
 Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
 artifacts remain under each comparison directory; PIMCOMP outputs are prepared
 once per model/architecture/mode and linked into the other pipeline directories;
 `comparison_report.{md,json}`
 live under its `pimcomp/`. The frontend regenerates
 one isolated `models/JSON/` graph because PIMCOMP requires that relative
 layout; it is removed after a successful backend run and the shared submodule
-model directory is never modified. Models requiring BatchNormalization folding
+model directory is never modified. The original ONNX model is passed to the
-also receive a prepared ONNX file; other models use the original ONNX directly.
+frontend unchanged.
 PIMCOMP's source tree and build directory remain unchanged at runtime. Use
-`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run`
+`--models vgg8` to run one model, `--mode throughput` to select one mode,
 `--pipeline 4` to select one throughput pipeline, `--only raptor` or
 `--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
 to inspect every command, or `--out-dir PATH` to keep results outside
-`validation/`. The runner continues after a failed model so all reports are
+`validation/`. Use `--clean` to remove generated comparison artifacts and
-produced.
+summaries. Selecting a subset replaces only those comparison rows and
 recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
 reference runner are generated even for an isolated run. Use `--jobs 4` to cap
 parallel comparisons. The per-stage timeout is unlimited by default; pass a
 positive `--timeout-seconds` value to impose one.
 PIMCOMP receives the original ONNX model, and its frontend applies native
 BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
 The runner continues after a failed model so all reports are produced.
 The known PIMCOMP batch-scheduling correctness issue and a reproducible
 reference-intermediate prefill experiment are documented in
 [`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
 Arch-A low-latency example:
 ```bash
 RAPTOR_ROOT=$PWD
-"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \
+"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
  --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
-  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
+  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
  --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
  --core-count 168 \
  --crossbar-count 96 \
@@ -251,7 +276,7 @@ RAPTOR_ROOT=$PWD
 ```
 Use the same command with
-`yolo11n/yolo11n-latency.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
+`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
 compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
 for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
 as YOLO11n would produce a misleading latency, so no PIMCOMP number is
@@ -297,7 +322,7 @@ either latency-only artifact for semantic validation.
 Current Raptor status:
 - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
- Use `googlenet-12-latency.onnx` for the paper-matched latency comparison.
+- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
  It removes the two LRN nodes and terminal softmax that PIMCOMP does not
  schedule.
 - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
@@ -319,8 +344,8 @@ rsync -azL validation/networks/pimcomp_models/ \
  "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
 rsync -az validation/pimsim_configs/pimcomp/ \
  "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
-rsync -az validation/tools/pimcomp/ \
+rsync -az validation/tools/pim/ \
-  "monolith:$REMOTE_REPO/validation/tools/pimcomp/"
+  "monolith:$REMOTE_REPO/validation/tools/pim/"
 rsync -az --exclude=.git --exclude=build --exclude=output \
  third_party/PIMCOMP-NN/ \
  "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
@@ -334,10 +359,10 @@ cd /home/gmagnani/Project/Raptor
 # One-time setup if the repository virtual environment is absent.
 python3 -m venv .venv
-.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
+.venv/bin/python -m pip install numpy onnx onnxruntime colorama
-# Run every latency comparison serially.
+# Run every configured comparison in parallel.
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
 Copy reports back without transferring large compiler artifacts:
@@ -1,5 +1,41 @@
-model,raptor_latency_ms,pimcomp_latency_ms,raptor_energy_pj,pimcomp_energy_pj,faster_compiler,speedup
+model,arch,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_latency_ms,pimcomp_latency_ms,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
-vgg8,1.465778,7.985074,477298145.040001,1597904071.120000,raptor,5.45
+vgg8,arch-a,latency,1,element,PASS,PASS,NA,NA,1.465778,7.985074,325.627854,200.111367,477298145.040001,1597904071.120000,raptor,5.45
-resnet18,28.099952,58.853733,8781611766.119984,13983168748.119974,raptor,2.09
+vgg8,arch-b,latency,1,element,PASS,PASS,NA,NA,1.438869,7.152125,304.673458,173.768633,438385194.040001,1242814988.120001,raptor,4.97
-resnet34,45.781486,91.607980,14962940227.679951,22722922369.680016,raptor,2.00
+vgg8,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-googlenet,13.371204,62.923463,6117835798.919991,14547526780.240000,raptor,4.71
+vgg8,arch-a,throughput,2,batch,PASS,PASS,1480.000000,1380.000000,0.674197,0.725548,456.064662,475.694103,307477506.200000,345138738.300000,raptor,1.07
 vgg8,arch-b,throughput,2,batch,PASS,PASS,1160.000000,1080.000000,0.861527,0.921878,333.045167,408.540967,286927542.600000,376624916.800000,raptor,1.07
 vgg8,arch-c,throughput,2,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 vgg8,arch-a,throughput,4,batch,PASS,PASS,2160.000000,1380.000000,0.462342,0.725548,446.528150,475.694103,206448818.800000,345138738.300000,raptor,1.57
 vgg8,arch-c,throughput,4,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 vgg8,arch-a,throughput,8,batch,PASS,PASS,831.000000,1380.000000,1.202894,0.725548,331.341726,475.694103,398569128.700000,345138738.300000,pimcomp,1.66
 vgg8,arch-c,throughput,8,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet18,arch-a,latency,1,element,PASS,PASS,NA,NA,28.099951,58.855175,312.513413,237.590194,8781611597.119984,13983412446.119972,raptor,2.09
 resnet18,arch-b,latency,1,element,PASS,PASS,NA,NA,34.938563,65.084209,254.439797,200.986958,8889760875.119959,13081077194.119965,raptor,1.86
 resnet18,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet18,arch-a,throughput,2,batch,PASS,FAIL,20.000000,76.000000,50.000000,13.149606,263.148477,479.788072,13157423870.000000,6309024249.000000,pimcomp,3.80
 resnet18,arch-b,throughput,2,batch,PASS,FAIL,26.300000,83.200000,38.016529,12.020906,250.553072,483.029728,9525158116.000000,5806454916.000000,pimcomp,3.16
 resnet18,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet18,arch-a,throughput,4,batch,PASS,FAIL,31.700000,76.000000,31.578947,13.149606,336.949886,479.788072,10640522720.000000,6309024249.000000,pimcomp,2.40
 resnet18,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet18,arch-a,throughput,8,batch,PASS,FAIL,46.400000,76.000000,21.566110,13.149606,319.292391,479.788072,6885894954.000000,6309024249.000000,pimcomp,1.64
 resnet18,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet34,arch-a,latency,1,element,PASS,PASS,NA,NA,45.781484,91.608751,326.833876,248.044564,14962939889.679951,22723052668.680016,raptor,2.00
 resnet34,arch-b,latency,1,element,PASS,PASS,NA,NA,72.119607,94.519582,239.192092,215.513439,17250439680.679901,20370240175.680019,raptor,1.31
 resnet34,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet34,arch-a,throughput,2,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
 resnet34,arch-b,throughput,2,batch,PASS,FAIL,11.600000,43.300000,86.250000,23.076923,260.049856,489.500435,22429300100.000000,11296163880.000000,pimcomp,3.73
 resnet34,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet34,arch-a,throughput,4,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
 resnet34,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 resnet34,arch-a,throughput,8,batch,PASS,FAIL,24.300000,40.800000,41.176471,24.522761,271.814969,506.131154,11192381060.000000,12411733160.000000,pimcomp,1.68
 resnet34,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 googlenet,arch-a,latency,1,element,PASS,PASS,NA,NA,13.032305,62.923369,465.072002,231.194088,6060960174.919998,14547510894.240002,raptor,4.83
 googlenet,arch-b,latency,1,element,PASS,PASS,NA,NA,16.086822,37.935747,378.951815,242.039077,6096130396.919978,9181933205.239973,raptor,2.36
 googlenet,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 googlenet,arch-a,throughput,2,batch,PASS,FAIL,55.000000,66.200000,18.181818,15.094340,346.696726,434.959232,6303576840.000000,6565422375.000000,pimcomp,1.20
 googlenet,arch-b,throughput,2,batch,PASS,FAIL,55.000000,72.000000,18.181818,13.888889,314.645370,417.992095,5720824912.000000,5805445765.000000,pimcomp,1.31
 googlenet,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 googlenet,arch-a,throughput,4,batch,PASS,FAIL,111.000000,66.200000,9.012016,15.094340,373.195227,434.959232,3363241363.000000,6565422375.000000,raptor,1.68
 googlenet,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
 googlenet,arch-a,throughput,8,batch,PASS,FAIL,61.300000,66.200000,16.310680,15.094340,341.592923,434.959232,5571612719.000000,6565422375.000000,pimcomp,1.08
 googlenet,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
@@ -2,5 +2,5 @@ Operation,Result,Compile,Host mem,Cores mem,Cores,Xbars,Latency,Power,Energy
 vgg8-mnist-reconstructed,PASS,1.009 s,1.37 MiB,3.14 MiB,141,761,1.465778 ms,325.627854 mW,477298145.040001 pJ
 resnet18-v1-7,PASS,11.548 s,9.89 MiB,40.24 MiB,168,7676,28.099952 ms,312.513408 mW,8781611766.119984 pJ
 resnet34-v1-7,PASS,28.495 s,9.90 MiB,48.89 MiB,168,15292,45.781486 ms,326.833870 mW,14962940227.679951 pJ
-googlenet-12-latency,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
+googlenet-12-pimsim-nn,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
-yolo11n-latency,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ
+yolo11n-pimsim-nn,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ
@@ -0,0 +1,61 @@
 {
  "chip_config": {
    "core_config": {
      "period": 1,
      "matrix_config": {
        "xbar_array_count": 96,
        "period": 1,
        "pipeline_mode": true,
        "dac_resolution": 1,
        "dac_count": 128,
        "xbar_size": [
          128,
          128
        ],
        "cell_precision": 2,
        "xbar_latency": 7,
        "xbar_read_power": 22.24,
        "sample_hold_latency_cycle": 1,
        "adc_resolution": 8,
        "adc_latency_cycle": 20,
        "adc_static_power": 0.322,
        "adc_dynamic_power": 1.135,
        "adc_count": 16,
        "shift_adder_latency_cycle": 1,
        "output_buffer_latency_cycle": 1
      },
      "vector_width": 32,
      "vector_latency_cycle": 4,
      "local_memory_config": {
        "data_width": 64,
        "period": 1,
        "write_latency_cycle": 30,
        "read_latency_cycle": 20
      },
      "global_memory_switch_id": -10,
      "rob_size": 1
    },
    "global_memory_config": {
      "data_width": 32,
      "period": 1,
      "write_latency_cycle": 50,
      "read_latency_cycle": 40
    },
    "network_config": {
      "bus_topology": "mesh",
      "bus_width": 32,
      "layout": [
        12,
        14
      ],
      "net_config_file_path": "network_mesh_168.json"
    },
    "core_cnt": 168,
    "global_memory_switch_id": -10
  },
  "sim_config": {
    "sim_mode": 0,
    "sim_time": 100,
    "report_verbose_level": 0
  }
 }
@@ -0,0 +1,61 @@
 {
  "chip_config": {
    "core_config": {
      "period": 1,
      "matrix_config": {
        "xbar_array_count": 128,
        "period": 1,
        "pipeline_mode": true,
        "dac_resolution": 1,
        "dac_count": 128,
        "xbar_size": [
          128,
          128
        ],
        "cell_precision": 2,
        "xbar_latency": 7,
        "xbar_read_power": 22.24,
        "sample_hold_latency_cycle": 1,
        "adc_resolution": 8,
        "adc_latency_cycle": 20,
        "adc_static_power": 0.322,
        "adc_dynamic_power": 1.135,
        "adc_count": 16,
        "shift_adder_latency_cycle": 1,
        "output_buffer_latency_cycle": 1
      },
      "vector_width": 32,
      "vector_latency_cycle": 4,
      "local_memory_config": {
        "data_width": 64,
        "period": 1,
        "write_latency_cycle": 30,
        "read_latency_cycle": 20
      },
      "global_memory_switch_id": -10,
      "rob_size": 1
    },
    "global_memory_config": {
      "data_width": 32,
      "period": 1,
      "write_latency_cycle": 50,
      "read_latency_cycle": 40
    },
    "network_config": {
      "bus_topology": "mesh",
      "bus_width": 32,
      "layout": [
        6,
        23
      ],
      "net_config_file_path": "network_mesh_138.json"
    },
    "core_cnt": 138,
    "global_memory_switch_id": -10
  },
  "sim_config": {
    "sim_mode": 0,
    "sim_time": 100,
    "report_verbose_level": 0
  }
 }
@@ -0,0 +1,61 @@
 {
  "chip_config": {
    "core_config": {
      "period": 1,
      "matrix_config": {
        "xbar_array_count": 8,
        "period": 1,
        "pipeline_mode": true,
        "dac_resolution": 1,
        "dac_count": 128,
        "xbar_size": [
          512,
          1024
        ],
        "cell_precision": 2,
        "xbar_latency": 7,
        "xbar_read_power": 22.24,
        "sample_hold_latency_cycle": 1,
        "adc_resolution": 8,
        "adc_latency_cycle": 20,
        "adc_static_power": 0.322,
        "adc_dynamic_power": 1.135,
        "adc_count": 16,
        "shift_adder_latency_cycle": 1,
        "output_buffer_latency_cycle": 1
      },
      "vector_width": 32,
      "vector_latency_cycle": 4,
      "local_memory_config": {
        "data_width": 64,
        "period": 1,
        "write_latency_cycle": 30,
        "read_latency_cycle": 20
      },
      "global_memory_switch_id": -10,
      "rob_size": 1
    },
    "global_memory_config": {
      "data_width": 32,
      "period": 1,
      "write_latency_cycle": 50,
      "read_latency_cycle": 40
    },
    "network_config": {
      "bus_topology": "mesh",
      "bus_width": 32,
      "layout": [
        8,
        8
      ],
      "net_config_file_path": "network_mesh_64.json"
    },
    "core_cnt": 64,
    "global_memory_switch_id": -10
  },
  "sim_config": {
    "sim_mode": 0,
    "sim_time": 100,
    "report_verbose_level": 0
  }
 }
@@ -17,14 +17,14 @@ import types
 from collections import Counter
 from dataclasses import asdict, dataclass
 from pathlib import Path
 from tempfile import TemporaryDirectory
 from typing import Any
 import numpy as np
 import onnx
 from colorama import Fore, Style
-REPO = Path(__file__).resolve().parents[3]
+REPO = Path(__file__).resolve().parents[5]
 VALIDATION_DIR = REPO / "validation"
 PIMSIM_CONFIG_DIR = VALIDATION_DIR / "pimsim_configs/pimcomp"
 PIMCOMP_OUTPUT_FILES = ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
@@ -156,7 +156,7 @@ def run_logged(
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
-            timeout=timeout_sec,
+            timeout=None if timeout_sec == 0 else timeout_sec,
        )
    except subprocess.TimeoutExpired as exc:
        duration = time.perf_counter() - start
@@ -210,31 +210,24 @@ def load_saved_inputs(
    return arrays
-def prepare_pimcomp_model(model_path: Path, out_dir: Path) -> Path:
+def reference_outputs_exist(
-    model = onnx.load(model_path)
+    outputs_desc: list[tuple[int, str, int, list[int]]],
-    if not any(node.op_type == "BatchNormalization" for node in model.graph.node):
+    reference_dir: Path,
-        return model_path
+) -> bool:
    return reference_dir.is_dir() and all(
        (reference_dir / f"output{idx}_{sanitize_output_name(name)}.csv").is_file()
        for idx, name, _, _ in outputs_desc
    )
    out_dir.mkdir(parents=True, exist_ok=True)
    output_path = out_dir / f"{model_path.stem}_pimcomp.onnx"
    from onnxsim import simplify
-    model, equivalent = simplify(model, check_n=1)
+def reference_inputs_exist(
-    if not equivalent:
+    inputs_desc: list[tuple[int, str, int, list[int]]],
-        raise RuntimeError("Conv+BatchNormalization folding changed the model output")
+    inputs_dir: Path,
-    if any(node.op_type == "BatchNormalization" for node in model.graph.node):
+) -> bool:
-        import onnxruntime as ort
+    return inputs_dir.is_dir() and all(
-
+        (inputs_dir / f"in{idx}.csv").is_file()
-        options = ort.SessionOptions()
+        for idx, _, _, _ in inputs_desc
-        options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
+    )
        options.optimized_model_filepath = str(output_path)
        ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"])
        model = onnx.load(output_path)
        if any(node.op_type == "BatchNormalization" for node in model.graph.node):
            raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization")
    else:
        onnx.save(model, output_path)
    return output_path
 def compare_simulator_outputs(
@@ -285,8 +278,12 @@ def load_effective_hardware(args: argparse.Namespace) -> dict[str, int]:
 def select_pimsim_config(args: argparse.Namespace, hardware: dict[str, int]) -> Path:
-    fallback: Path | None = None
+    filename = (
-    for path in sorted(PIMSIM_CONFIG_DIR.glob(f"*/{args.pimsim_mode}_config.json")):
+        "latency_config.json"
        if args.pimsim_mode == "latency"
        else f"throughput_config_{args.pimsim_time_ms}ms.json"
    )
    for path in sorted(PIMSIM_CONFIG_DIR.glob(f"*/{filename}")):
        with open(path, encoding="utf-8") as f:
            config = json.load(f)
        chip = config["chip_config"]
@@ -299,41 +296,18 @@ def select_pimsim_config(args: argparse.Namespace, hardware: dict[str, int]) ->
            and network["layout"] == [hardware["mesh_rows"], hardware["mesh_cols"]]
            and config["sim_config"]["sim_mode"] == (1 if args.pimsim_mode == "latency" else 0)
        ):
-            if config["sim_config"]["sim_time"] == args.pimsim_time_ms:
+            if args.pimsim_mode == "latency" or config["sim_config"]["sim_time"] == args.pimsim_time_ms:
                return path
            fallback = fallback or path
    if fallback is not None:
        return fallback
    raise ValueError(
-        f"No pre-generated {args.pimsim_mode} pimsim-nn config matches {hardware}"
+        f"No pre-generated {args.pimsim_mode} pimsim-nn config for {args.pimsim_time_ms} ms matches {hardware}"
    )
 def prepare_pimsim_config(
    args: argparse.Namespace,
    hardware: dict[str, int],
    out_dir: Path,
 ) -> Path:
-    source = select_pimsim_config(args, hardware)
+    return select_pimsim_config(args, hardware)
    with open(source, encoding="utf-8") as f:
        config = json.load(f)
    if config["sim_config"]["sim_time"] == args.pimsim_time_ms:
        return source
    config["sim_config"]["sim_time"] = args.pimsim_time_ms
    target = out_dir / "pimsim_config.json"
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8")
    network_path = Path(config["chip_config"]["network_config"]["net_config_file_path"])
    if not network_path.is_absolute():
        network_path = source.parent / network_path
    target_network = target.parent / Path(
        config["chip_config"]["network_config"]["net_config_file_path"]
    )
    target_network.parent.mkdir(parents=True, exist_ok=True)
    shutil.copy2(network_path, target_network)
    return target
 def compile_reference(
@@ -395,6 +369,7 @@ def compile_reference(
        timeout_sec=args.timeout_seconds,
        steps=steps,
    )
    shutil.rmtree(raptor_dir)
    return build_dir / "runner"
@@ -422,6 +397,33 @@ def generate_reference_outputs(
    return reference_dir
 def prepare_common_artifacts(
    args: argparse.Namespace,
    model_path: Path,
    common_dir: Path,
 ) -> None:
    inputs_desc, outputs_desc, arrays_in_order = load_model_inputs(model_path, args.seed)
    inputs_dir = common_dir / "inputs"
    outputs_dir = common_dir / "outputs"
    runner_path = common_dir / "runner/build/runner"
    steps: list[StepRecord] = []
    if not runner_path.exists():
        runner_path = compile_reference(args, model_path, common_dir, steps)
    inputs_ready = reference_inputs_exist(inputs_desc, inputs_dir)
    outputs_ready = reference_outputs_exist(outputs_desc, outputs_dir)
    if inputs_ready:
        arrays_in_order = load_saved_inputs(inputs_desc, inputs_dir)
    if not (inputs_ready and outputs_ready):
        generate_reference_outputs(
            runner_path,
            runner_path.parent,
            model_path,
            arrays_in_order,
            steps,
            args,
            common_dir,
        )
 def compile_raptor_target(
    model_path: Path,
    out_dir: Path,
@@ -538,9 +540,13 @@ def run_functional_validation(
 def copy_pimcomp_outputs(source_dir: Path, out_dir: Path):
    if source_dir.resolve() == out_dir.resolve():
        return
    out_dir.mkdir(parents=True, exist_ok=True)
    for name in PIMCOMP_OUTPUT_FILES:
-        shutil.copy2(source_dir / name, out_dir / name)
+        destination = out_dir / name
        destination.unlink(missing_ok=True)
        destination.symlink_to(source_dir / name)
 def compile_pimcomp(
@@ -560,22 +566,27 @@ def compile_pimcomp(
        (pimcomp_output_dir / name).unlink(missing_ok=True)
    model_name = args.pimcomp_model_name or f"compare_{model_path.stem}"
    frontend_json = frontend_json_dir / f"{model_name}.json"
-    frontend_cmd = [
+    # The original PIMCOMP frontend rewrites its input ONNX while loading it.
-        sys.executable,
+    # Isolate that mutation without changing the model sent through PIMCOMP.
-        str(args.pimcomp_dir / "frontend/frontend.py"),
+    with TemporaryDirectory(prefix="pimcomp-model-") as temp_dir:
-        "--model_path",
+        frontend_model = Path(temp_dir) / model_path.name
-        str(model_path),
+        shutil.copy2(model_path, frontend_model)
-        "--save_path",
+        frontend_cmd = [
-        str(frontend_json),
+            sys.executable,
-    ]
+            str(args.pimcomp_dir / "frontend/frontend.py"),
-    run_logged(
+            "--model_path",
-        "Compile PIMCOMP Frontend",
+            str(frontend_model),
-        frontend_cmd,
+            "--save_path",
-        cwd=args.pimcomp_dir / "frontend",
+            str(frontend_json),
-        timeout_sec=args.timeout_seconds,
+        ]
-        steps=steps,
+        run_logged(
-        stage="Compile PIM",
+            "Compile PIMCOMP Frontend",
-    )
+            frontend_cmd,
            cwd=args.pimcomp_dir / "frontend",
            timeout_sec=args.timeout_seconds,
            steps=steps,
            stage="Compile PIM",
        )
    backend_cmd = [
        str(args.pimcomp_dir / "build" / "PIMCOMP-NN"),
        f"-m={model_name}",
@@ -584,15 +595,17 @@ def compile_pimcomp(
        "-v=YES",
        "-s=YES",
    ]
-    run_logged(
+    try:
-        "Compile PIMCOMP Backend",
+        run_logged(
-        backend_cmd,
+            "Compile PIMCOMP Backend",
-        cwd=frontend_json_dir.parent,
+            backend_cmd,
-        timeout_sec=args.timeout_seconds,
+            cwd=frontend_json_dir.parent,
-        steps=steps,
+            timeout_sec=args.timeout_seconds,
-        stage="Compile PIM",
+            steps=steps,
-    )
+            stage="Compile PIM",
-    shutil.rmtree(frontend_json_dir.parent)
+        )
    finally:
        shutil.rmtree(frontend_json_dir.parent, ignore_errors=True)
    return pimcomp_output_dir / "VerificationInfo.json", pimcomp_output_dir / "SimulationInfo.gz"
@@ -1001,6 +1014,16 @@ def optional_path(path: Path | None) -> str | None:
    return str(path) if path is not None else None
 def default_common_dir(out_dir: Path) -> Path:
    if out_dir.name == "latency":
        return out_dir.parents[1] / "common"
    if out_dir.parent.name == "throughput":
        return out_dir.parents[2] / "common"
    if out_dir.name.startswith("arch-"):
        return out_dir.parent / "common"
    return out_dir / "common"
 def record_failure(failures: list[dict[str, str]], stage: str, exc: BaseException | str) -> None:
    message = exc if isinstance(exc, str) else exception_message(exc)
    failures.append({"stage": stage, "error": message})
@@ -1204,6 +1227,16 @@ def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", required=True, type=Path)
    parser.add_argument("--out-dir", required=True, type=Path)
    parser.add_argument(
        "--common-dir",
        type=Path,
        help="Shared per-model directory for reference inputs, outputs, runner, and prepared model.",
    )
    parser.add_argument(
        "--prepare-common",
        action="store_true",
        help="Prepare shared model artifacts and exit before a comparison.",
    )
    parser.add_argument("--raptor-path", default=REPO / "build_release/Release/bin/onnx-mlir", type=Path)
    parser.add_argument("--onnx-include-dir", default=REPO / "onnx-mlir/include", type=Path)
    parser.add_argument("--pimcomp-dir", default=REPO / "third_party/PIMCOMP-NN", type=Path)
@@ -1217,7 +1250,12 @@ def main():
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--threshold", type=float, default=1e-3)
    parser.add_argument("--rtol", type=float, default=1e-5)
-    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
+    parser.add_argument(
        "--timeout-seconds",
        type=float,
        default=0.0,
        help="Per-stage timeout in seconds; 0 means no timeout (default).",
    )
    parser.add_argument("--core-count", type=int)
    parser.add_argument("--crossbar-count", type=int)
    parser.add_argument("--crossbar-size", type=int)
@@ -1253,6 +1291,8 @@ def main():
    args = parser.parse_args()
    if args.pimsim_time_ms <= 0:
        parser.error("--pimsim-time-ms must be positive")
    if args.timeout_seconds < 0:
        parser.error("--timeout-seconds must be non-negative")
    if args.pimcomp_pipeline is None:
        args.pimcomp_pipeline = "element" if args.pimsim_mode == "latency" else "batch"
@@ -1265,6 +1305,18 @@ def main():
    )
    out_dir = args.out_dir.resolve()
    out_dir.mkdir(parents=True, exist_ok=True)
    common_dir = (
        args.common_dir.resolve()
        if args.common_dir is not None
        else default_common_dir(out_dir).resolve()
    )
    common_dir.mkdir(parents=True, exist_ok=True)
    if args.prepare_common:
        print_step("Prepare shared artifacts")
        prepare_common_artifacts(args, model_path, common_dir)
        print(f"Shared artifacts: {common_dir}")
        return
    failures: list[dict[str, str]] = []
    steps: list[StepRecord] = []
@@ -1308,6 +1360,22 @@ def main():
        inputs_desc, outputs_desc, arrays_in_order = model_io
        runtime_inputs = arrays_in_order
    reuse_pimcomp = args.reuse_pimcomp_dir is not None
    if reuse_pimcomp:
        reused_pimcomp_report_path = args.reuse_pimcomp_dir.resolve().parent / "comparison_report.json"
        if not reused_pimcomp_report_path.exists():
            raise ValueError(f"Missing PIMCOMP report beside {args.reuse_pimcomp_dir}")
        with open(reused_pimcomp_report_path, "r", encoding="utf-8") as f:
            reused_pimcomp = json.load(f)
        if reused_pimcomp.get("pimcomp_model_source") != "original_onnx":
            raise ValueError("Reused PIMCOMP artifacts were not generated from the original ONNX model")
        pimcomp_validation = CompareResult(**reused_pimcomp["pimcomp_validation"])
        pimcomp_perf = reused_pimcomp["pimcomp_performance"]
        pimcomp_instr = reused_pimcomp["pimcomp_instruction_summary"]
        reused_config = reused_pimcomp.get("paths", {}).get("pimsim_config")
        if reused_config:
            pimsim_config = Path(reused_config)
    if reuse_raptor and model_io is not None:
        reuse_report_path = args.reuse_raptor_report.resolve()
        with open(reuse_report_path, "r", encoding="utf-8") as f:
@@ -1315,11 +1383,12 @@ def main():
        reused_hardware = reused["hardware"]
        if reused_hardware != hardware:
            raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
-        reference_dir = Path(reused["paths"]["reference_outputs"])
+        reference_dir = common_dir / "outputs"
-        raptor_pim_dir = Path(reused["paths"]["raptor_pim"])
+        reused_raptor_pim = reused["paths"].get("raptor_pim")
        raptor_pim_dir = Path(reused_raptor_pim) if reused_raptor_pim else None
        arrays_in_order = load_saved_inputs(
            inputs_desc,
-            reference_dir.parent / "inputs",
+            common_dir / "inputs",
        )
        runtime_inputs = arrays_in_order
        raptor_validation = CompareResult(**reused["raptor_validation"])
@@ -1329,53 +1398,66 @@ def main():
        print_step("Reuse Raptor")
        print(f"  Report: {reuse_report_path}")
-    expected_runner_path = out_dir / "runner/build/runner"
+    expected_runner_path = common_dir / "runner/build/runner"
    common_inputs_dir = common_dir / "inputs"
    common_reference_dir = common_dir / "outputs"
    if not reuse_raptor:
-        reference_compile = try_stage(
+        inputs_ready = reference_inputs_exist(inputs_desc, common_inputs_dir)
-            failures,
+        if inputs_ready and model_io is not None:
-            "Compile reference",
+            saved_inputs = try_stage(
-            compile_reference,
+                failures,
-            args,
+                "Load shared reference inputs",
-            model_path,
+                load_saved_inputs,
-            out_dir,
+                inputs_desc,
-            steps,
+                common_inputs_dir,
-        )
+            )
-        if reference_compile is not None:
+            if saved_inputs is not None:
-            runner_path = reference_compile
+                arrays_in_order = saved_inputs
-        else:
+                runtime_inputs = arrays_in_order
-            if expected_runner_path.exists():
+            else:
-                runner_path = expected_runner_path
+                inputs_ready = False
                print(
                    "\n"
                    + Style.BRIGHT
                    + Fore.YELLOW
                    + "[Continue]"
                    + Style.RESET_ALL
                    + f" Reusing partial runner: {runner_path}"
                )
-    if not reuse_raptor and runner_path is not None and runner_path.exists() and model_io is not None:
+        if expected_runner_path.exists():
-        generated_reference = try_stage(
+            runner_path = expected_runner_path
-            failures,
+        else:
-            "Run reference",
+            reference_compile = try_stage(
-            generate_reference_outputs,
+                failures,
-            runner_path,
+                "Compile reference",
-            runner_path.parent,
+                compile_reference,
-            model_path,
+                args,
-            arrays_in_order,
+                model_path,
-            steps,
+                common_dir,
-            args,
+                steps,
-            out_dir,
+            )
-        )
+            if reference_compile is not None:
-        if generated_reference is not None:
+                runner_path = reference_compile
-            reference_dir = generated_reference
+
-    elif not reuse_raptor:
+        if runner_path is not None and runner_path.exists() and model_io is not None:
-        record_failure(
+            if inputs_ready and reference_outputs_exist(outputs_desc, common_reference_dir):
-            failures,
+                reference_dir = common_reference_dir
-            "Skip reference outputs",
+                print_step("Reuse shared reference")
-            "Reference outputs were skipped because the native runner or model inputs are not available.",
+            else:
-        )
+                generated_reference = try_stage(
                    failures,
                    "Run reference",
                    generate_reference_outputs,
                    runner_path,
                    runner_path.parent,
                    model_path,
                    arrays_in_order,
                    steps,
                    args,
                    common_dir,
                )
                if generated_reference is not None:
                    reference_dir = generated_reference
        else:
            record_failure(
                failures,
                "Skip reference outputs",
                "Reference outputs were skipped because the native runner or model inputs are not available.",
            )
    if not reuse_raptor and model_path.exists() and hardware["core_count"] > 0:
        compiled_raptor = try_stage(
@@ -1430,13 +1512,7 @@ def main():
    elif not reuse_raptor:
        raptor_validation = skipped_validation("Raptor PIM compilation did not produce a PIM directory")
-    pimcomp_model_path = try_stage(
+    pimcomp_model_path = model_path
        failures,
        "Prepare PIMCOMP model",
        prepare_pimcomp_model,
        model_path,
        out_dir / "pimcomp/model",
    )
    if args.reuse_pimcomp_dir is not None:
        reused_pimcomp_dir = args.reuse_pimcomp_dir.resolve()
@@ -1465,7 +1541,9 @@ def main():
        if compiled_pimcomp is not None:
            verification_info, simulation_info = compiled_pimcomp
-    if verification_info is not None and simulation_info is not None and model_io is not None:
+    if reuse_pimcomp:
        pass
    elif verification_info is not None and simulation_info is not None and model_io is not None:
        exported = try_stage(
            failures,
            "Export PIMCOMP for Functional Validation",
@@ -1491,7 +1569,7 @@ def main():
            "PIMCOMP functional export failed because model inputs are not available.",
        )
-    if pimcomp_export_dir is not None and reference_dir is not None and outputs_desc:
+    if not reuse_pimcomp and pimcomp_export_dir is not None and reference_dir is not None and outputs_desc:
        validation = try_stage(
            failures,
            "Functional Validation PIMCOMP",
@@ -1507,6 +1585,8 @@ def main():
            channel_last=True,
        )
        pimcomp_validation = validation if validation is not None else failed_validation("PIMCOMP validation failed")
    elif reuse_pimcomp:
        pass
    elif pimcomp_export_dir is None:
        pimcomp_validation = failed_validation("PIMCOMP functional export is not available")
    elif reference_dir is None:
@@ -1521,7 +1601,6 @@ def main():
            prepare_pimsim_config,
            args,
            hardware,
            out_dir,
        )
        if written_config is not None:
            pimsim_config = written_config
@@ -1566,7 +1645,7 @@ def main():
        elif not reuse_raptor:
            raptor_perf = skipped_perf("Raptor PIM directory is not available")
-        if simulation_info is not None:
+        if not reuse_pimcomp and simulation_info is not None:
            pimcomp_pimsim_dir = try_stage(
                failures,
                "Export PIMCOMP for pimsim-nn",
@@ -1597,7 +1676,7 @@ def main():
    elif not reuse_raptor:
        raptor_instr = empty_instruction_summary("Raptor PIM directory is not available")
-    if simulation_info is not None and simulation_info.exists():
+    if not reuse_pimcomp and simulation_info is not None and simulation_info.exists():
        parsed = try_stage(failures, "Parse PIMCOMP instructions", parse_pimcomp_instructions, simulation_info)
        pimcomp_instr = parsed if parsed is not None else empty_instruction_summary(error="Failed to parse PIMCOMP instructions")
    else:
@@ -1629,6 +1708,7 @@ def main():
        "pimsim_time_ms": args.pimsim_time_ms,
        "pimcomp_pipeline": args.pimcomp_pipeline,
        "pimcomp_replication": args.pimcomp_replication,
        "pimcomp_model_source": "original_onnx",
        "pimcomp_config": str(args.pimcomp_config),
        "raptor_extra_args": args.raptor_extra_arg,
        "reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
@@ -1642,7 +1722,10 @@ def main():
        "pimcomp_instruction_summary": pimcomp_instr,
        "raptor_pass_timings": raptor_pass_timings,
        "paths": {
            "common_dir": str(common_dir),
            "reference_inputs": optional_path(common_dir / "inputs"),
            "reference_outputs": optional_path(reference_dir),
            "reference_runner": optional_path(runner_path),
            "raptor_pim": optional_path(raptor_pim_dir),
            "raptor_pimsim_nn": optional_path(raptor_pimsim_dir),
            "pimcomp_simulation_info": optional_path(simulation_info),
@@ -0,0 +1,755 @@
 #!/usr/bin/env python3
 from __future__ import annotations
 import argparse
 import csv
 import json
 import os
 import shlex
 import shutil
 import subprocess
 import sys
 from concurrent.futures import ProcessPoolExecutor, as_completed
 from contextlib import nullcontext, redirect_stderr, redirect_stdout
 from dataclasses import dataclass
 from pathlib import Path
 from tempfile import TemporaryDirectory
 from colorama import Fore, Style
 REPO = Path(__file__).resolve().parents[5]
 SUITE = REPO / "validation/networks/pimcomp_models"
 sys.path.insert(0, str(REPO / "validation"))
 from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics  # noqa: E402
 from raptor_validation.validate_one import STAGE_COLORS  # noqa: E402
 PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
 PIMCOMP_CONFIGS = REPO / "validation/pimsim_configs/pimcomp"
 COMPARE = Path(__file__).resolve().with_name("compare_raptor_pimcomp.py")
 ARCHES = tuple(sorted(path.name for path in PIMCOMP_CONFIGS.iterdir() if path.is_dir()))
 MODELS = {
    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
    "googlenet": SUITE / "googlenet/googlenet-12-pimsim-nn.onnx",
    "yolo11n": SUITE / "yolo11n/yolo11n-pimsim-nn.onnx",
 }
 COMPARISONS = (
    ("latency", 1, "element"),
    ("throughput", 2, "batch"),
    ("throughput", 4, "batch"),
    ("throughput", 8, "batch"),
 )
@dataclass(frozen=True)
 class ComparisonSpec:
    label: str
    model: Path
    output_dir: Path
    common_dir: Path
    config: Path
    mode: str
    pipeline: int
    pimcomp_pipeline: str
    shared_pimcomp_dir: Path
    anchor: bool
 def model_dir(root: Path | None, name: str) -> Path:
    return root / name if root is not None else MODELS[name].parent
 def result_dir(root: Path | None, name: str, arch: str, mode: str, pipeline: int) -> Path:
    base = model_dir(root, name)
    suffix = "latency" if mode == "latency" else f"throughput/pipeline{pipeline}"
    return base / arch / suffix
 def clean_artifacts(root: Path | None, models: list[str], arches: list[str]) -> int:
    removed = 0
    for name in models:
        base = model_dir(root, name)
        for arch in arches:
            for path in (base / arch / "latency", base / arch / "throughput"):
                if path.is_dir() and not path.is_symlink():
                    shutil.rmtree(path)
                    removed += 1
        common = base / "common"
        if common.is_dir() and not common.is_symlink():
            shutil.rmtree(common)
            removed += 1
    for path in (
        (root or SUITE) / "results.csv",
        (root or SUITE) / "results_latency.csv",
        (root or SUITE) / "results_throughput.csv",
    ):
        if path.is_file() or path.is_symlink():
            path.unlink()
            removed += 1
    return removed
 def write_results_csv(
    root: Path | None,
    arch: str,
    models: list[str],
    comparisons: tuple[tuple[str, int, str], ...] = COMPARISONS,
 ) -> Path:
    output = (root or SUITE) / "results.csv"
    fields = (
        "model",
        "arch",
        "mode",
        "raptor_pipeline",
        "pimcomp_pipeline",
        "raptor_functional_validation",
        "pimcomp_functional_validation",
        "raptor_throughput_samples_s",
        "pimcomp_throughput_samples_s",
        "raptor_latency_ms",
        "pimcomp_latency_ms",
        "raptor_power_mw",
        "pimcomp_power_mw",
        "raptor_energy_pj",
        "pimcomp_energy_pj",
        "better_compiler",
        "speedup",
    )
    selected = {
        (name, arch, mode, str(pipeline))
        for name in models
        for mode, pipeline, _ in comparisons
    }
    rows = []
    if output.exists():
        with open(output, newline="", encoding="utf-8") as f:
            reader = csv.DictReader(f)
            if reader.fieldnames and "model" in reader.fieldnames:
                for row in reader:
                    key = (
                        row.get("model"),
                        row.get("arch"),
                        row.get("mode"),
                        row.get("raptor_pipeline"),
                    )
                    if key in selected:
                        continue
                    row.pop("status", None)
                    rows.append({field: row.get(field, "NA") for field in fields})
    for name in models:
        for comparison_mode, pipeline, pimcomp_pipeline in comparisons:
            report_path = result_dir(root, name, arch, comparison_mode, pipeline) / "pimcomp/comparison_report.json"
            row = {
                "model": name,
                "arch": arch,
                "mode": comparison_mode,
                "raptor_pipeline": pipeline,
                "pimcomp_pipeline": pimcomp_pipeline,
                "raptor_functional_validation": "NA",
                "pimcomp_functional_validation": "NA",
                "raptor_throughput_samples_s": "NA",
                "pimcomp_throughput_samples_s": "NA",
                "raptor_latency_ms": "NA",
                "pimcomp_latency_ms": "NA",
                "raptor_power_mw": "NA",
                "pimcomp_power_mw": "NA",
                "raptor_energy_pj": "NA",
                "pimcomp_energy_pj": "NA",
                "better_compiler": "NA",
                "speedup": "NA",
            }
            if report_path.exists():
                report = json.loads(report_path.read_text(encoding="utf-8"))
                raptor_values = performance_values(report.get("raptor_performance") or {})
                pimcomp_values = performance_values(report.get("pimcomp_performance") or {})
                raptor_metric = raptor_values["throughput"] if comparison_mode == "throughput" else raptor_values["latency"]
                pimcomp_metric = pimcomp_values["throughput"] if comparison_mode == "throughput" else pimcomp_values["latency"]
                row.update(
                    raptor_functional_validation=functional_validation_status(
                        report.get("raptor_validation")
                    ),
                    pimcomp_functional_validation=functional_validation_status(
                        report.get("pimcomp_validation")
                    ),
                    raptor_throughput_samples_s=format_value(raptor_values["throughput"]),
                    pimcomp_throughput_samples_s=format_value(pimcomp_values["throughput"]),
                    raptor_latency_ms=format_value(raptor_values["latency"]),
                    pimcomp_latency_ms=format_value(pimcomp_values["latency"]),
                    raptor_power_mw=format_value(raptor_values["power"]),
                    pimcomp_power_mw=format_value(pimcomp_values["power"]),
                    raptor_energy_pj=format_value(raptor_values["energy"]),
                    pimcomp_energy_pj=format_value(pimcomp_values["energy"]),
                )
                if raptor_metric is not None and pimcomp_metric is not None:
                    row.update(
                        better_compiler=comparison_winner(comparison_mode, raptor_metric, pimcomp_metric),
                        speedup=f"{max(raptor_metric, pimcomp_metric) / min(raptor_metric, pimcomp_metric):.2f}",
                    )
            rows.append(row)
    rows.sort(key=lambda row: (list(MODELS).index(row["model"]), row["mode"], int(row["raptor_pipeline"])))
    with open(output, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
        writer.writeheader()
        writer.writerows(rows)
    return output
 def performance_values(performance: dict) -> dict[str, float | None]:
    parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
    return {
        "throughput": performance.get("throughput") or parsed.get("throughput"),
        "latency": (
            performance.get("latency_ms")
            or performance.get("average_latency_ms")
            or parsed.get("latency_ms")
            or parsed.get("average_latency_ms")
        ),
        "power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
        "energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
    }
 def comparison_passed(report: dict) -> bool:
    if report.get("failures"):
        return False
    for key in ("raptor_validation", "pimcomp_validation"):
        result = report.get(key) or {}
        if result.get("status") != "done" or not result.get("passed"):
            return False
    for key in ("raptor_performance", "pimcomp_performance"):
        performance = report.get(key) or {}
        if performance.get("error") or performance.get("skipped"):
            return False
    return True
 def functional_validation_status(result: dict | None) -> str:
    if result is None:
        return "NA"
    return "PASS" if result.get("status") == "done" and result.get("passed") else "FAIL"
 def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
    if raptor == pimcomp:
        return "tie"
    if mode == "throughput":
        return "raptor" if raptor > pimcomp else "pimcomp"
    return "raptor" if raptor < pimcomp else "pimcomp"
 def format_value(value: float | None) -> str:
    return "NA" if value is None else f"{value:.6f}"
 def print_stage(title: str, color: str) -> None:
    print("\n" + Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL, flush=True)
 def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
    print(f"  cwd: {REPO}", flush=True)
    print(f"  $ {shlex.join(command)}", flush=True)
    if dry_run:
        return 0
    return subprocess.run(command, cwd=REPO, check=check).returncode
 def run_comparison_job(job: tuple[str, list[str], Path | None]) -> tuple[str, int, str | None]:
    label, command, log_path = job
    sys.stdout.flush()
    sys.stderr.flush()
    if log_path is None:
        return label, run(command, dry_run=False, check=False), None
    saved_stdout = os.dup(1)
    saved_stderr = os.dup(2)
    try:
        with open(log_path, "w", encoding="utf-8", buffering=1) as log:
            os.dup2(log.fileno(), 1)
            os.dup2(log.fileno(), 2)
            with redirect_stdout(log), redirect_stderr(log):
                returncode = run(command, dry_run=False, check=False)
    finally:
        os.dup2(saved_stdout, 1)
        os.dup2(saved_stderr, 2)
        os.close(saved_stdout)
        os.close(saved_stderr)
    return label, returncode, str(log_path)
 def validate_pimcomp_source() -> None:
    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
    source = header.read_text(encoding="utf-8")
    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
        if setting not in source:
            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
 def comparison_command(
    model: Path,
    result_dir: Path,
    common_dir: Path,
    config: Path,
    mode: str,
    pipeline: int,
    pimcomp_pipeline: str,
    pimsim_time_ms: int,
    timeout: float,
    reuse_raptor_report: Path | None = None,
    reuse_pimcomp_dir: Path | None = None,
 ) -> list[str]:
    time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
    reuse_args = []
    if reuse_raptor_report is not None:
        reuse_args.extend(["--reuse-raptor-report", str(reuse_raptor_report)])
    if reuse_pimcomp_dir is not None:
        reuse_args.extend(["--reuse-pimcomp-dir", str(reuse_pimcomp_dir)])
    return [
        sys.executable,
        str(COMPARE),
        "--model",
        str(model),
        "--out-dir",
        str(result_dir),
        "--common-dir",
        str(common_dir),
        "--pimcomp-dir",
        str(PIMCOMP_SOURCE),
        "--pimcomp-config",
        str(config),
        "--pimsim-mode",
        mode,
        *time_args,
        "--pimcomp-pipeline",
        pimcomp_pipeline,
        "--pimcomp-replication",
        "GA",
        f"--raptor-extra-arg=--pipeline={pipeline}",
        "--timeout-seconds",
        str(timeout),
        "--fail-on-error",
        *reuse_args,
    ]
 def prepare_common_command(model: Path, common_dir: Path, timeout: float) -> list[str]:
    return [
        sys.executable,
        str(COMPARE),
        "--model",
        str(model),
        "--out-dir",
        str(common_dir),
        "--common-dir",
        str(common_dir),
        "--prepare-common",
        "--timeout-seconds",
        str(timeout),
    ]
 def comparison_command_for(
    spec: ComparisonSpec,
    args: argparse.Namespace,
    *,
    reuse_shared_pimcomp: bool,
 ) -> list[str]:
    reuse_pimcomp_dir = None
    if reuse_shared_pimcomp:
        reuse_pimcomp_dir = spec.shared_pimcomp_dir
    elif args.only == "raptor":
        reuse_pimcomp_dir = spec.output_dir / "pimcomp/output"
    return comparison_command(
        spec.model,
        spec.output_dir,
        spec.common_dir,
        spec.config,
        spec.mode,
        spec.pipeline,
        spec.pimcomp_pipeline,
        args.pimsim_time_ms,
        args.timeout_seconds,
        reuse_raptor_report=(
            spec.output_dir / "pimcomp/comparison_report.json"
            if args.only == "pimcomp"
            else None
        ),
        reuse_pimcomp_dir=reuse_pimcomp_dir,
    )
 def pimcomp_artifact_ready(output_dir: Path) -> bool:
    return all(
        (output_dir / name).is_file()
        for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
    ) and (output_dir.parent / "comparison_report.json").is_file()
 def run_comparison_jobs(
    comparison_jobs: list[tuple[str, list[str], Path | None]],
    max_workers: int,
    log_dir: Path | None,
    log_offset: int,
 ) -> tuple[list[str], int]:
    if not comparison_jobs:
        return [], log_offset
    print_directly = min(max_workers, len(comparison_jobs)) == 1
    jobs = [
        (
            label,
            command,
            None if print_directly or log_dir is None else log_dir / f"{log_offset + index}.log",
        )
        for index, (label, command, _) in enumerate(comparison_jobs)
    ]
    with (nullcontext(None) if print_directly else ProcessPoolExecutor(max_workers=max_workers)) as executor:
        completed = (
            map(run_comparison_job, jobs)
            if print_directly
            else (
                future.result()
                for future in as_completed(executor.submit(run_comparison_job, job) for job in jobs)
            )
        )
        failed = []
        for label, returncode, result_log in completed:
            if result_log:
                output = Path(result_log).read_text(encoding="utf-8", errors="replace")
                if output:
                    print(output, end="" if output.endswith("\n") else "\n")
            if returncode:
                failed.append(label)
    return failed, log_offset + len(jobs)
 def ensure_throughput_config(arch: str, sim_time_ms: int) -> Path:
    source = PIMCOMP_CONFIGS / arch / "throughput_config_1000ms.json"
    target = PIMCOMP_CONFIGS / arch / f"throughput_config_{sim_time_ms}ms.json"
    if target.exists():
        return target
    if sim_time_ms == 1000:
        return source
    config = json.loads(source.read_text(encoding="utf-8"))
    config["sim_config"]["sim_time"] = sim_time_ms
    target.write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8")
    return target
 def config_path(arch: str, mode: str, sim_time_ms: int, *, write: bool) -> Path:
    path = (
        PIMCOMP_CONFIGS / arch / "latency_config.json"
        if mode == "latency"
        else PIMCOMP_CONFIGS / arch / f"throughput_config_{sim_time_ms}ms.json"
    )
    if mode == "throughput" and write:
        path = ensure_throughput_config(arch, sim_time_ms)
    if not path.exists() and (write or mode == "latency"):
        raise ValueError(f"{arch} has no {mode} config: {path}")
    return path
 def core_count(config: Path) -> int:
    if not config.exists() and config.name.startswith("throughput_config_"):
        config = config.with_name("throughput_config_1000ms.json")
    with open(config, encoding="utf-8") as f:
        return int(json.load(f)["chip_config"]["core_cnt"])
 def main() -> int:
    parser = argparse.ArgumentParser(
        description="Compare supported PIMCOMP models with Raptor latency and throughput schedules."
    )
    parser.add_argument(
        "--out-dir",
        type=Path,
        help="Result root (default: artifacts beside each model under validation/).",
    )
    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
    parser.add_argument(
        "--arch",
        nargs="+",
        choices=ARCHES,
        default=list(ARCHES),
        help="PIM architectures to run (default: all architectures).",
    )
    parser.add_argument(
        "--mode",
        nargs="+",
        choices=("latency", "throughput"),
        default=["latency", "throughput"],
        help="Pimsim modes to run (default: both).",
    )
    parser.add_argument(
        "--only",
        choices=("raptor", "pimcomp"),
        help="Re-run only this compiler and reuse the other side's existing artifacts.",
    )
    parser.add_argument(
        "--pipeline",
        type=int,
        choices=(1, 2, 4, 8),
        help="Run only this Raptor pipeline within the selected mode.",
    )
    parser.add_argument(
        "--pimsim-time-ms",
        type=int,
        default=100,
        help="throughput pimsim-nn horizon in ms (default: 100).",
    )
    parser.add_argument(
        "--timeout-seconds",
        type=float,
        default=0.0,
        help="Per-stage timeout in seconds; 0 means no timeout (default).",
    )
    parser.add_argument(
        "-j",
        "--jobs",
        type=int,
        default=os.cpu_count() or 1,
        help="Number of comparisons to run in parallel (default: all available CPUs).",
    )
    parser.add_argument(
        "--clean",
        action="store_true",
        help="Remove generated comparison artifacts and result summaries, then exit.",
    )
    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
    args = parser.parse_args()
    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
    args.arch = list(dict.fromkeys(args.arch))
    args.mode = list(dict.fromkeys(args.mode))
    if args.clean:
        print(f"Removed {clean_artifacts(out_dir, args.models, args.arch)} comparison artifact path(s).")
        return 0
    if args.jobs < 1:
        parser.error("--jobs must be at least 1")
    if args.pimsim_time_ms <= 0:
        parser.error("--pimsim-time-ms must be positive")
    if args.timeout_seconds < 0:
        parser.error("--timeout-seconds must be non-negative")
    comparisons_by_arch: dict[str, tuple[tuple[str, int, str], ...]] = {}
    configs_by_arch: dict[str, dict[str, Path]] = {}
    for arch in args.arch:
        comparisons = tuple(
            comparison for comparison in COMPARISONS
            if comparison[0] in args.mode
            and (args.pipeline is None or comparison[1] == args.pipeline)
        )
        if not comparisons:
            parser.error("--pipeline does not belong to the selected --mode")
        configs = {
            mode: config_path(arch, mode, args.pimsim_time_ms, write=not args.dry_run)
            for mode, _, _ in comparisons
        }
        unsupported = [
            pipeline for mode, pipeline, _ in comparisons
            if mode == "throughput" and core_count(configs[mode]) % pipeline
        ]
        if unsupported:
            if args.pipeline is not None:
                parser.error(
                    f"{arch} has {core_count(configs['throughput'])} cores; "
                    f"throughput pipelines must divide that count (invalid: {unsupported})"
                )
            print(
                Fore.YELLOW
                + f"Skipping unsupported {arch} throughput pipeline(s): {unsupported}"
                + Style.RESET_ALL
            )
            comparisons = tuple(comparison for comparison in comparisons if comparison[1] not in unsupported)
        comparisons_by_arch[arch] = comparisons
        configs_by_arch[arch] = configs
    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
    if missing:
        parser.error(f"missing model(s): {', '.join(missing)}")
    if args.only is not None:
        missing_reuse = []
        for arch in args.arch:
            for name in args.models:
                for mode, pipeline, _ in comparisons_by_arch[arch]:
                    comparison_dir = result_dir(out_dir, name, arch, mode, pipeline)
                    required = (
                        comparison_dir / "pimcomp/comparison_report.json"
                        if args.only == "pimcomp"
                        else comparison_dir / "pimcomp/output/SimulationInfo.gz"
                    )
                    if not required.exists():
                        missing_reuse.append(str(required))
                    elif args.only == "raptor":
                        report = comparison_dir / "pimcomp/comparison_report.json"
                        if not report.exists() or json.loads(report.read_text(encoding="utf-8")).get(
                            "pimcomp_model_source"
                        ) != "original_onnx":
                            missing_reuse.append(f"{report} (not generated from the original ONNX model)")
        if missing_reuse:
            parser.error(
                f"--only {args.only} needs existing comparison artifacts: "
                + ", ".join(missing_reuse)
            )
    validate_pimcomp_source()
    if out_dir is not None and not args.dry_run:
        out_dir.mkdir(parents=True, exist_ok=True)
    print(Style.BRIGHT + f"Found {len(args.models)} PIMCOMP model(s) to compare." + Style.RESET_ALL)
    print(f"Architectures: {', '.join(args.arch)}")
    print(f"Modes: {', '.join(args.mode)}")
    print(f"Throughput pimsim time: {args.pimsim_time_ms} ms")
    print(f"Max parallel jobs: {args.jobs}")
    print(f"Results root: {out_dir or SUITE}")
    print("=" * 72)
    print_stage("Build Raptor", STAGE_COLORS["Build Runner"])
    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
    if args.only != "raptor":
        print_stage("Build PIMCOMP", STAGE_COLORS["Build Runner"])
        run(
            ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
            dry_run=args.dry_run,
        )
    print_stage("Prepare shared artifacts", STAGE_COLORS["Build Runner"])
    for name in args.models:
        run(
            prepare_common_command(
                MODELS[name],
                model_dir(out_dir, name) / "common",
                args.timeout_seconds,
            ),
            dry_run=args.dry_run,
        )
    failed = []
    comparison_specs: list[ComparisonSpec] = []
    shared_pimcomp_by_group: dict[tuple[str, str, str], Path] = {}
    for arch in args.arch:
        comparisons = comparisons_by_arch[arch]
        configs = configs_by_arch[arch]
        for index, name in enumerate(args.models, start=1):
            for mode, pipeline, pimcomp_pipeline in comparisons:
                model_result_dir = result_dir(out_dir, name, arch, mode, pipeline)
                label = f"{arch}/{name}/{mode}/pipeline{pipeline}"
                group = (arch, name, mode)
                shared_pimcomp_dir = shared_pimcomp_by_group.get(group)
                anchor = shared_pimcomp_dir is None
                if anchor:
                    shared_pimcomp_dir = model_result_dir / "pimcomp/output"
                    shared_pimcomp_by_group[group] = shared_pimcomp_dir
                print(
                    "\n" + Fore.CYAN + f"[{arch} {index}/{len(args.models)}]" + Style.RESET_ALL
                    + f" {Style.BRIGHT}Comparing {name} ({mode}, pipeline={pipeline}){Style.RESET_ALL}",
                    flush=True,
                )
                comparison_specs.append(
                    ComparisonSpec(
                        label=label,
                        model=MODELS[name],
                        output_dir=model_result_dir,
                        common_dir=model_dir(out_dir, name) / "common",
                        config=configs[mode],
                        mode=mode,
                        pipeline=pipeline,
                        pimcomp_pipeline=pimcomp_pipeline,
                        shared_pimcomp_dir=shared_pimcomp_dir,
                        anchor=anchor,
                    )
                )
    if args.dry_run:
        for spec in comparison_specs:
            command = comparison_command_for(
                spec,
                args,
                reuse_shared_pimcomp=not spec.anchor,
            )
            if run(command, dry_run=True, check=False):
                failed.append(spec.label)
    elif comparison_specs:
        print(f"Running {len(comparison_specs)} comparison(s)")
        anchor_jobs = [
            (
                spec.label,
                comparison_command_for(spec, args, reuse_shared_pimcomp=False),
                None,
            )
            for spec in comparison_specs
            if spec.anchor
        ]
        dependent_specs = [spec for spec in comparison_specs if not spec.anchor]
        print_directly = min(args.jobs, len(comparison_specs)) == 1
        with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-pimcomp-")) as log_dir:
            anchor_failed, log_offset = run_comparison_jobs(
                anchor_jobs,
                args.jobs,
                Path(log_dir) if log_dir else None,
                0,
            )
            failed.extend(anchor_failed)
            dependent_jobs = [
                (
                    spec.label,
                    comparison_command_for(
                        spec,
                        args,
                        reuse_shared_pimcomp=pimcomp_artifact_ready(spec.shared_pimcomp_dir),
                    ),
                    None,
                )
                for spec in dependent_specs
            ]
            dependent_failed, _ = run_comparison_jobs(
                dependent_jobs,
                args.jobs,
                Path(log_dir) if log_dir else None,
                log_offset,
            )
            failed.extend(dependent_failed)
    if args.dry_run:
        return 1 if failed else 0
    results_path = None
    for arch in args.arch:
        results_path = write_results_csv(
            out_dir,
            arch,
            args.models,
            comparisons_by_arch[arch],
        )
    assert results_path is not None
    print_stage(results_path.name, STAGE_COLORS["Compare Outputs"])
    print(results_path.read_text(encoding="utf-8"), end="")
    for arch in args.arch:
        for name in args.models:
            for mode, pipeline, _ in comparisons_by_arch[arch]:
                label = f"{arch}/{name}/{mode}/pipeline{pipeline}"
                report_path = result_dir(out_dir, name, arch, mode, pipeline) / "pimcomp/comparison_report.json"
                if not report_path.exists() or not comparison_passed(
                    json.loads(report_path.read_text(encoding="utf-8"))
                ):
                    if label not in failed:
                        failed.append(label)
    print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
    total_jobs = sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())
    print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL)
    print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
    print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
    if failed:
        print(
            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
            file=sys.stderr,
        )
        return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,49 @@
 # PIMCOMP batch correctness reproduction
 PIMCOMP's batch scheduler currently emits an incomplete standalone program for
 models containing post operations. The generated `VerificationInfo.json` uses a
 negative `source_address` to identify the preceding node, but the batch
 verifier resolves that address by copying the provider tensor directly from
 ONNX Runtime's `intermediate_result` table. It does not read the simulated
 global-memory buffer.
 The corresponding post-operation scheduling call is commented out in
 [`BatchPipelineSchedule.cpp`](../../../../../third_party/PIMCOMP-NN/backend/BatchPipelineSchedule.cpp#L1535).
 Consequently, Pool, ReLU, Concat, and Reshape provider tensors can be consumed
 by later `LD` instructions without any generated `ST` producer. The comparison
 exporter initializes those host-side intermediate buffers to zero, which makes
 the missing computation visible in Rust functional validation.
 For the checked-in GoogLeNet throughput/pipeline2 artifact, 39 provider tensors
 are referenced by batch loads. Nineteen have generated stores; twenty are
 never written. Preloading the provider tensors with the same ONNX Runtime
 intermediates used by PIMCOMP's verifier makes the exported program pass. This
 reproduces the verifier's input contract; it does not repair PIMCOMP's batch
 schedule.
 Run the default reproduction from the repository root:
 ```bash
 .venv/bin/python validation/tools/pim/pimcomp/correctness/run_prefill_experiment.py
 ```
 The launcher accepts an alternate comparison directory, model, and work
 work directory, and shared reference-artifact directory:
 ```bash
 .venv/bin/python validation/tools/pim/pimcomp/correctness/run_prefill_experiment.py \
  validation/networks/pimcomp_models/googlenet/arch-a/throughput/pipeline2 \
  validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx \
  /tmp/pimcomp-prefill-googlenet \
  validation/networks/pimcomp_models/googlenet/common
 ```
 It runs the exported artifact once with its original memory image and once
 with [`prefill_batch_memory.py`](prefill_batch_memory.py), then compares both
 outputs with the recorded native reference. The expected GoogLeNet result is a
 baseline maximum difference near `6.70705` and a prefilled maximum difference
 near `4.05e-6`.
 The issue is in PIMCOMP batch scheduling/validation semantics, not in the Rust
 simulator's vector-length interpretation. Vector lengths remain element counts
 as specified by the reference ISA.
@@ -0,0 +1,171 @@
 #!/usr/bin/env python3
 """Populate the host-side intermediate buffers expected by PIMCOMP batch mode."""
 from __future__ import annotations
 import argparse
 import json
 import shutil
 from pathlib import Path
 import numpy as np
 import onnx
 import onnxruntime as ort
 def sanitize_output_name(name: str) -> str:
    return "".join(char if char.isalnum() or char in "_.-" else "_" for char in name[:255])
 def flatten_reference(value: np.ndarray) -> np.ndarray:
    if value.ndim == 4:
        value = value.transpose(0, 2, 3, 1)
    elif value.ndim == 2:
        value = value.transpose()
    else:
        raise ValueError(f"PIMCOMP batch verification only flattens 2D/4D tensors, got {value.shape}")
    return value.astype(np.float32, copy=False).reshape(-1)
 def provider_indices(verification_info: dict) -> list[int]:
    indices = set()
    for core in verification_info["instruction"]["core_list"]:
        for instruction in core or []:
            if (
                instruction["operation"] == "LD"
                and instruction.get("stage") in ("INPUT", "POST")
                and instruction["node_index"] != 1
            ):
                source_address = instruction["source_address"]
                if source_address >= 0:
                    raise ValueError(f"Expected a negative provider address, got {source_address}")
                indices.add(-source_address)
    return sorted(indices)
 def output_base(
    model: onnx.ModelProto,
    config: dict,
    node_list: list[dict],
    max_output: int,
 ) -> tuple[int, list[dict]]:
    if len(model.graph.output) != len(config["outputs_addresses"]):
        raise ValueError("Exported output addresses do not match the ONNX graph outputs")
    nodes_by_name = {node["name"]: node for node in node_list}
    bases = []
    outputs = []
    for index, graph_output in enumerate(model.graph.output):
        node = nodes_by_name[graph_output.name]
        address = config["outputs_addresses"][index]
        base = int(address - node["new_node_index"] * max_output * 4)
        bases.append(base)
        outputs.append({"name": graph_output.name, "address": address})
    if len(set(bases)) != 1:
        raise ValueError(f"Exported output addresses use different memory bases: {bases}")
    return bases[0], outputs
 def prefill_batch_memory(
    model_path: Path,
    comparison_dir: Path,
    input_path: Path,
    output_memory: Path,
    metadata_path: Path,
 ) -> None:
    comparison_dir = comparison_dir.resolve()
    model = onnx.load(model_path)
    verification_info = json.loads(
        (comparison_dir / "pimcomp/output/VerificationInfo.json").read_text(encoding="utf-8")
    )
    config = json.loads((comparison_dir / "pimcomp/exported/config.json").read_text(encoding="utf-8"))
    node_list = verification_info["node_list"]
    max_output = max(int(np.prod(node["output_dim"], dtype=int)) for node in node_list)
    base, outputs = output_base(model, config, node_list, max_output)
    providers = provider_indices(verification_info)
    runtime_model = onnx.ModelProto()
    runtime_model.CopyFrom(model)
    existing_outputs = {output.name for output in runtime_model.graph.output}
    for index in providers:
        name = node_list[index]["name"]
        if name not in existing_outputs:
            runtime_model.graph.output.append(onnx.ValueInfoProto(name=name))
    session = ort.InferenceSession(runtime_model.SerializeToString(), providers=["CPUExecutionProvider"])
    session_inputs = session.get_inputs()
    if len(session_inputs) != 1:
        raise ValueError("PIMCOMP export currently requires exactly one runtime input tensor")
    input_meta = session_inputs[0]
    input_tensor = np.loadtxt(input_path, delimiter=",", dtype=np.float32).reshape(input_meta.shape)
    provider_names = [node_list[index]["name"] for index in providers]
    provider_values = session.run(provider_names, {input_meta.name: input_tensor})
    output_memory = output_memory.resolve()
    metadata_path = metadata_path.resolve()
    if output_memory == (comparison_dir / "pimcomp/exported/memory.bin").resolve():
        raise ValueError("Refusing to overwrite the original exported memory.bin")
    output_memory.parent.mkdir(parents=True, exist_ok=True)
    shutil.copy2(comparison_dir / "pimcomp/exported/memory.bin", output_memory)
    nodes_by_index = {index: node for index, node in enumerate(node_list)}
    with output_memory.open("r+b") as memory:
        memory_size = output_memory.stat().st_size
        for index, value in zip(providers, provider_values):
            flattened = flatten_reference(value)
            node = nodes_by_index[index]
            if node["operation"] in ("OP_CONV", "OP_FC") and node.get("with_act") == 1:
                flattened = np.maximum(flattened, 0)
            address = base + index * max_output * 4
            end = address + flattened.nbytes
            if end > memory_size:
                raise ValueError(f"Provider {index} exceeds exported memory: {end} > {memory_size}")
            memory.seek(address)
            memory.write(flattened.tobytes())
    output_metadata = {output["name"]: output for output in outputs}
    for index, output_meta in enumerate(session.get_outputs()):
        if output_meta.name not in output_metadata:
            continue
        output = output_metadata[output_meta.name]
        output["dump"] = f"{output['address']},{int(np.prod(output_meta.shape, dtype=int)) * 4}"
        output["reference"] = str(
            comparison_dir / f"outputs/output{index}_{sanitize_output_name(output_meta.name)}.csv"
        )
    metadata_path.parent.mkdir(parents=True, exist_ok=True)
    metadata_path.write_text(
        json.dumps(
            {
                "memory": str(output_memory),
                "output_base": base,
                "provider_indices": providers,
                "outputs": outputs,
            },
            indent=2,
        )
        + "\n",
        encoding="utf-8",
    )
    print(f"Prefilled {len(providers)} provider tensor(s) into {output_memory}")
    print(f"Metadata: {metadata_path}")
 def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--model", type=Path, required=True)
    parser.add_argument("--comparison-dir", type=Path, required=True)
    parser.add_argument("--input", type=Path, required=True)
    parser.add_argument("--output-memory", type=Path, required=True)
    parser.add_argument("--metadata", type=Path, required=True)
    args = parser.parse_args()
    prefill_batch_memory(
        args.model,
        args.comparison_dir,
        args.input,
        args.output_memory,
        args.metadata,
    )
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,127 @@
 #!/usr/bin/env python3
 """Reproduce the PIMCOMP batch prefill correctness experiment."""
 from __future__ import annotations
 import argparse
 import json
 import subprocess
 import sys
 import tempfile
 from pathlib import Path
 import numpy as np
 from prefill_batch_memory import prefill_batch_memory
 SCRIPT_DIR = Path(__file__).resolve().parent
 REPO_ROOT = SCRIPT_DIR.parents[4]
 DEFAULT_COMPARISON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/arch-a/throughput/pipeline2"
 DEFAULT_MODEL = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx"
 DEFAULT_COMMON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/common"
 SIMULATOR_MANIFEST = REPO_ROOT / "backend-simulators/pim/pim-simulator/Cargo.toml"
 def run_simulator(
    comparison_dir: Path,
    memory: Path,
    output: Path,
    dump: str,
 ) -> None:
    subprocess.run(
        [
            "cargo",
            "run",
            "--manifest-path",
            str(SIMULATOR_MANIFEST),
            "--no-default-features",
            "--release",
            "--bin",
            "pim-simulator",
            "--",
            "-f",
            str(comparison_dir / "pimcomp/exported"),
            "--memory",
            str(memory),
            "-o",
            str(output),
            "-d",
            dump,
        ],
        cwd=REPO_ROOT,
        check=True,
    )
 def compare_outputs(
    baseline_path: Path,
    prefilled_path: Path,
    reference_path: Path,
    work_dir: Path,
 ) -> None:
    baseline = np.fromfile(baseline_path, dtype=np.float32)
    prefilled = np.fromfile(prefilled_path, dtype=np.float32)
    reference = np.loadtxt(reference_path, delimiter=",", dtype=np.float32).reshape(-1)
    baseline_diff = np.max(np.abs(baseline - reference))
    prefilled_diff = np.max(np.abs(prefilled - reference))
    print(f"work directory: {work_dir}")
    print(f"baseline max diff: {baseline_diff:.9g}")
    print(f"prefilled max diff: {prefilled_diff:.9g}")
    if np.allclose(baseline, reference, atol=1e-4, rtol=1e-3):
        raise RuntimeError("baseline unexpectedly passes")
    if not np.allclose(prefilled, reference, atol=1e-4, rtol=1e-3):
        raise RuntimeError("prefilled run does not pass")
 def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("comparison_dir", nargs="?", type=Path, default=DEFAULT_COMPARISON_DIR)
    parser.add_argument("model", nargs="?", type=Path, default=DEFAULT_MODEL)
    parser.add_argument("work_dir", nargs="?", type=Path)
    parser.add_argument("common_dir", nargs="?", type=Path, default=DEFAULT_COMMON_DIR)
    args = parser.parse_args()
    comparison_dir = args.comparison_dir.resolve()
    model = args.model.resolve()
    common_dir = args.common_dir.resolve()
    if args.work_dir is None:
        work_dir = Path(tempfile.mkdtemp(prefix="pimcomp-prefill."))
    else:
        work_dir = args.work_dir.resolve()
        work_dir.mkdir(parents=True, exist_ok=True)
    input_path = common_dir / "inputs/in0.csv"
    if not input_path.is_file():
        input_path = comparison_dir / "inputs/in0.csv"
    prefilled_memory = work_dir / "prefilled_memory.bin"
    metadata_path = work_dir / "metadata.json"
    baseline_output = work_dir / "baseline.out.bin"
    prefilled_output = work_dir / "prefilled.out.bin"
    prefill_batch_memory(
        model,
        comparison_dir,
        input_path,
        prefilled_memory,
        metadata_path,
    )
    metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
    output = metadata["outputs"][0]
    dump = output["dump"]
    reference = Path(output["reference"])
    run_simulator(
        comparison_dir,
        comparison_dir / "pimcomp/exported/memory.bin",
        baseline_output,
        dump,
    )
    run_simulator(comparison_dir, prefilled_memory, prefilled_output, dump)
    compare_outputs(baseline_output, prefilled_output, reference, work_dir)
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -1,355 +0,0 @@
 #!/usr/bin/env python3
 from __future__ import annotations
 import argparse
 import csv
 import json
 import shlex
 import subprocess
 import sys
 from pathlib import Path
 from colorama import Fore, Style
 REPO = Path(__file__).resolve().parents[3]
 SUITE = REPO / "validation/networks/pimcomp_models"
 sys.path.insert(0, str(REPO / "validation"))
 from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics  # noqa: E402
 from raptor_validation.validate_one import STAGE_COLORS  # noqa: E402
 PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
 PIMCOMP_CONFIGS = REPO / "validation/pimsim_configs/pimcomp"
 COMPARE = Path(__file__).resolve().with_name("compare_raptor_pimcomp.py")
 ARCHES = tuple(sorted(path.name for path in PIMCOMP_CONFIGS.iterdir() if path.is_dir()))
 MODELS = {
    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
    "googlenet": SUITE / "googlenet/googlenet-12-latency.onnx",
 }
 COMPARISONS = (
    ("latency", 1, "element"),
    ("throughput", 2, "batch"),
    ("throughput", 4, "batch"),
    ("throughput", 8, "batch"),
 )
 def result_dir(root: Path | None, name: str, mode: str, pipeline: int) -> Path:
    base = root / name if root is not None else MODELS[name].parent
    suffix = "latency" if mode == "latency" else f"throughput/pipeline{pipeline}"
    return base / suffix
 def write_results_csv(root: Path | None, arch: str, models: list[str]) -> Path:
    output = (root or SUITE) / "results.csv"
    fields = (
        "model",
        "arch",
        "mode",
        "raptor_pipeline",
        "pimcomp_pipeline",
        "status",
        "raptor_throughput_samples_s",
        "pimcomp_throughput_samples_s",
        "raptor_latency_ms",
        "pimcomp_latency_ms",
        "raptor_power_mw",
        "pimcomp_power_mw",
        "raptor_energy_pj",
        "pimcomp_energy_pj",
        "better_compiler",
        "speedup",
    )
    rows = []
    for name in models:
        for mode, pipeline, pimcomp_pipeline in COMPARISONS:
            report_path = result_dir(root, name, mode, pipeline) / "pimcomp/comparison_report.json"
            if not report_path.exists():
                continue
            report = json.loads(report_path.read_text(encoding="utf-8"))
            raptor = report.get("raptor_performance") or {}
            pimcomp = report.get("pimcomp_performance") or {}
            raptor_values = performance_values(raptor)
            pimcomp_values = performance_values(pimcomp)
            raptor_metric = raptor_values["throughput"] if mode == "throughput" else raptor_values["latency"]
            pimcomp_metric = pimcomp_values["throughput"] if mode == "throughput" else pimcomp_values["latency"]
            status = "PASS" if comparison_passed(report) else "FAIL"
            if raptor_metric is None or pimcomp_metric is None:
                better = ""
                speedup = ""
            else:
                better = comparison_winner(mode, raptor_metric, pimcomp_metric)
                speedup = f"{max(raptor_metric, pimcomp_metric) / min(raptor_metric, pimcomp_metric):.2f}"
            rows.append({
                "model": name,
                "arch": arch,
                "mode": mode,
                "raptor_pipeline": pipeline,
                "pimcomp_pipeline": pimcomp_pipeline,
                "status": status,
                "raptor_throughput_samples_s": format_value(raptor_values["throughput"]),
                "pimcomp_throughput_samples_s": format_value(pimcomp_values["throughput"]),
                "raptor_latency_ms": format_value(raptor_values["latency"]),
                "pimcomp_latency_ms": format_value(pimcomp_values["latency"]),
                "raptor_power_mw": format_value(raptor_values["power"]),
                "pimcomp_power_mw": format_value(pimcomp_values["power"]),
                "raptor_energy_pj": format_value(raptor_values["energy"]),
                "pimcomp_energy_pj": format_value(pimcomp_values["energy"]),
                "better_compiler": better,
                "speedup": speedup,
            })
    with open(output, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
        writer.writeheader()
        writer.writerows(rows)
    return output
 def performance_values(performance: dict) -> dict[str, float | None]:
    parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
    return {
        "throughput": performance.get("throughput") or parsed.get("throughput"),
        "latency": (
            performance.get("latency_ms")
            or performance.get("average_latency_ms")
            or parsed.get("latency_ms")
            or parsed.get("average_latency_ms")
        ),
        "power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
        "energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
    }
 def comparison_passed(report: dict) -> bool:
    if report.get("failures"):
        return False
    for key in ("raptor_validation", "pimcomp_validation"):
        result = report.get(key) or {}
        if result.get("status") != "done" or not result.get("passed"):
            return False
    for key in ("raptor_performance", "pimcomp_performance"):
        performance = report.get(key) or {}
        if performance.get("error") or performance.get("skipped"):
            return False
    return True
 def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
    if raptor == pimcomp:
        return "tie"
    if mode == "throughput":
        return "raptor" if raptor > pimcomp else "pimcomp"
    return "raptor" if raptor < pimcomp else "pimcomp"
 def format_value(value: float | None) -> str:
    return "" if value is None else f"{value:.6f}"
 def print_stage(title: str, color: str) -> None:
    print("\n" + Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL, flush=True)
 def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
    print(f"  cwd: {REPO}", flush=True)
    print(f"  $ {shlex.join(command)}", flush=True)
    if dry_run:
        return 0
    return subprocess.run(command, cwd=REPO, check=check).returncode
 def validate_pimcomp_source() -> None:
    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
    source = header.read_text(encoding="utf-8")
    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
        if setting not in source:
            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
 def comparison_command(
    model: Path,
    result_dir: Path,
    config: Path,
    mode: str,
    pipeline: int,
    pimcomp_pipeline: str,
    pimsim_time_ms: int,
    timeout: float,
 ) -> list[str]:
    time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
    return [
        sys.executable,
        str(COMPARE),
        "--model",
        str(model),
        "--out-dir",
        str(result_dir),
        "--pimcomp-dir",
        str(PIMCOMP_SOURCE),
        "--pimcomp-config",
        str(config),
        "--pimsim-mode",
        mode,
        *time_args,
        "--pimcomp-pipeline",
        pimcomp_pipeline,
        "--pimcomp-replication",
        "GA",
        f"--raptor-extra-arg=--pipeline={pipeline}",
        "--timeout-seconds",
        str(timeout),
        "--fail-on-error",
    ]
 def config_path(arch: str, mode: str) -> Path:
    path = PIMCOMP_CONFIGS / arch / f"{mode}_config.json"
    if not path.exists():
        raise ValueError(f"{arch} has no {mode} config: {path}")
    return path
 def core_count(config: Path) -> int:
    with open(config, encoding="utf-8") as f:
        return int(json.load(f)["chip_config"]["core_cnt"])
 def completed_report(path: Path, mode: str, pipeline: int, config: Path, pimsim_time_ms: int) -> bool:
    if not path.exists():
        return False
    report = json.loads(path.read_text(encoding="utf-8"))
    return (
        report.get("pimsim_mode") == mode
        and report.get("pimcomp_pipeline") == ("element" if mode == "latency" else "batch")
        and report.get("pimsim_time_ms") == pimsim_time_ms
        and report.get("pimcomp_config") == str(config.resolve())
        and f"--pipeline={pipeline}" in report.get("raptor_extra_args", [])
    )
 def main() -> int:
    parser = argparse.ArgumentParser(
        description="Compare supported PIMCOMP models with Raptor latency and throughput schedules."
    )
    parser.add_argument(
        "--out-dir",
        type=Path,
        help="Result root (default: artifacts beside each model under validation/).",
    )
    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
    parser.add_argument(
        "--arch", choices=ARCHES, default="arch-a", help="PIM architecture (default: arch-a)."
    )
    parser.add_argument(
        "--pimsim-time-ms",
        type=int,
        default=100,
        help="throughput pimsim-nn horizon in ms (default: 100).",
    )
    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
    parser.add_argument(
        "--resume",
        action="store_true",
        help="Skip models with a completed JSON report.",
    )
    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
    args = parser.parse_args()
    if args.pimsim_time_ms <= 0:
        parser.error("--pimsim-time-ms must be positive")
    configs = {mode: config_path(args.arch, mode) for mode, _, _ in COMPARISONS}
    unsupported = [pipeline for mode, pipeline, _ in COMPARISONS if core_count(configs[mode]) % pipeline]
    if unsupported:
        parser.error(
            f"{args.arch} has {core_count(configs['throughput'])} cores; "
            f"throughput pipelines must divide that count (invalid: {unsupported})"
        )
    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
    if missing:
        parser.error(f"missing model(s): {', '.join(missing)}")
    validate_pimcomp_source()
    if out_dir is not None and not args.dry_run:
        out_dir.mkdir(parents=True, exist_ok=True)
    print(Style.BRIGHT + f"Found {len(args.models)} PIMCOMP model(s) to compare." + Style.RESET_ALL)
    print(f"Architecture: {args.arch}")
    print(f"Throughput pimsim time: {args.pimsim_time_ms} ms")
    print(f"Results root: {out_dir or SUITE}")
    print("=" * 72)
    print_stage("Build Raptor", STAGE_COLORS["Build Runner"])
    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
    print_stage("Build PIMCOMP", STAGE_COLORS["Build Runner"])
    run(
        ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
        dry_run=args.dry_run,
    )
    failed = []
    for index, name in enumerate(args.models, start=1):
        for mode, pipeline, pimcomp_pipeline in COMPARISONS:
            model_result_dir = result_dir(out_dir, name, mode, pipeline)
            print(
                "\n" + Fore.CYAN + f"[{index}/{len(args.models)}]" + Style.RESET_ALL
                + f" {Style.BRIGHT}Comparing {name} ({mode}, pipeline={pipeline}){Style.RESET_ALL}",
                flush=True,
            )
            if args.resume and completed_report(
                model_result_dir / "pimcomp/comparison_report.json",
                mode,
                pipeline,
                configs[mode],
                args.pimsim_time_ms,
            ):
                print(
                    Fore.YELLOW + "  Completed report exists; skipping" + Style.RESET_ALL,
                    flush=True,
                )
                continue
            returncode = run(
                comparison_command(
                    MODELS[name],
                    model_result_dir,
                    configs[mode],
                    mode,
                    pipeline,
                    pimcomp_pipeline,
                    args.pimsim_time_ms,
                    args.timeout_seconds,
                ),
                dry_run=args.dry_run,
                check=False,
            )
            if returncode:
                failed.append(f"{name}/{mode}/pipeline{pipeline}")
    if args.dry_run:
        return 1 if failed else 0
    results_path = write_results_csv(out_dir, args.arch, args.models)
    print_stage("Results", STAGE_COLORS["Compare Outputs"])
    print(results_path.read_text(encoding="utf-8"), end="")
    print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
    total_jobs = len(args.models) * len(COMPARISONS)
    print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL)
    print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
    print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
    if failed:
        print(
            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
            file=sys.stderr,
        )
        return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())