more complete pimcomp comparison scripts

update pimsim-nn submodule

validation/README.md

@@ -63,22 +63,24 @@ that directory while retaining the terminal table.
 
 ## Raptor vs PIMCOMP comparison
 
-The PIMCOMP paper-model suite has a one-command Arch-A comparison:
+The PIMCOMP paper-model suite has a one-command multi-architecture comparison:
 
 ```bash
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
 
 The runner verifies PIMCOMP's population-200, 1000-iteration GA settings,
 builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, and compares
-the four paper models one at a time. Each PIMCOMP GA run evaluates candidates
+the four paper models in parallel. Set `--jobs` to control comparison workers;
-in parallel; set `OMP_NUM_THREADS` to control its worker count. Use
+the runner leaves `OMP_NUM_THREADS` at its environment default. Use
 `--models vgg8` for one model or `--dry-run` to print the commands.
 
 Generated artifacts are stored beside each model and ignored by Git. The
-comparison reuses the same model-level `inputs/`, `outputs/`, `runner/`,
+comparison reuses model-level `common/inputs/`, `common/outputs/`,
-`raptor/`, and `simulation/` paths as regular validation. PIMCOMP-only
+`common/runner/` across architectures, modes, and pipelines. Pimsim configs and
-artifacts and `comparison_report.{md,json}` live under `pimcomp/`. See
+network meshes are referenced directly from
+`validation/pimsim_configs/pimcomp/`. Architecture- and pipeline-specific `raptor/`, `simulation/`, and
+PIMCOMP artifacts stay under each comparison directory. See
 [`networks/pimcomp_models/README.md`](networks/pimcomp_models/README.md) for
 profiles, model provenance, limitations, and remote execution.
 
@@ -164,7 +166,9 @@ The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
 prints the incompatible values; functional validation still runs.
 
 The checked-in profiles are under
-`validation/pimsim_configs/pimcomp/<profile>/latency_config.json`.
+`validation/pimsim_configs/pimcomp/<profile>/`; latency uses
+`latency_config.json`, while throughput uses
+`throughput_config_<time>ms.json` and the mesh beside it.
 
 Use `--skip-non-functional-simulation` when latency, power, and energy are not required.
 The summary reports non-functional results as measured, failed, unsupported, or
@@ -185,14 +189,14 @@ Artifacts are written beside each model:
 
 | Path | Contents |
 |---|---|
-| `inputs/` | Generated input CSV files. |
+| `common/inputs/` | Shared generated input CSV files. |
-| `outputs/` | ONNX-MLIR reference output CSV files. |
+| `common/outputs/` | Shared ONNX-MLIR reference output CSV files. |
-| `raptor/` | Exported MLIR, dialect snapshots, reports, final FP32 `pim/` artifacts, and the int8-equivalent `pimsim_nn/` latency view. |
+| `common/runner/` | Shared reference runner source, build tree, and library. |
-| `runner/` | Generated reference runner source, build tree, and shared library. |
+| `<arch>/<mode>[/pipelineN]/raptor/` | Architecture- and pipeline-specific Raptor MLIR and PIM artifacts. |
-| `simulation/` | Functional simulator outputs used for numerical comparison. |
+| `<arch>/<mode>[/pipelineN]/simulation/` | Functional simulator outputs for that comparison. |
-| `pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
+| `<arch>/<mode>[/pipelineN]/pimcomp/` | PIMCOMP graph, instruction, simulator, and comparison-report artifacts. |
 
-The `raptor/` directory may include `spatial0.mlir`,
+Each comparison's `raptor/` directory may include `spatial0.mlir`,
 `spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
 `spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
 `pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.

validation/networks/pimcomp_models/README.md

@@ -5,8 +5,12 @@ This directory contains the four networks evaluated in
 VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
 additional compiler comparison model.
 
-See the runner-generated [results.csv](results.csv) for the current latency
+See the runner-generated [results.csv](results.csv) for the current comparison
-and energy results.
+results. Rows are retained separately for each model, architecture, mode, and
+pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
+the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
+Running the runner with `--arch arch-b` or `--arch arch-c` appends those
+architecture rows without replacing the existing `arch-a` entries.
 
 ## Models and provenance
 
@@ -18,11 +22,11 @@ and energy results.
 | `vgg8/`      | VGG-8 reconstruction | `1x1x28x28`   | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers.                                                            |
 | `yolo11n/`   | YOLO11n detection    | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
 
-`googlenet/googlenet-12-latency.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
+`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
 It removes the two LRN nodes and terminal Softmax from the original model,
 so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
 
-`yolo11n/yolo11n-latency.onnx` is the explicit pimsim-nn-ready YOLO11n model.
+`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
 It removes the Softmax nodes from the original model,
 so that the compiled artifact can be simulated in pimsim-nn.
 
@@ -98,9 +102,9 @@ Current SHA-256 checksums:
 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f  resnet18/resnet18-v1-7.onnx
 c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096  resnet34/resnet34-v1-7.onnx
 c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e  googlenet/googlenet-12.onnx
-a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982  googlenet/googlenet-12-latency.onnx
+a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982  googlenet/googlenet-12-pimsim-nn.onnx
 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef  vgg8/vgg8-mnist-reconstructed.onnx
-229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f  yolo11n/yolo11n-latency.onnx
+229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f  yolo11n/yolo11n-pimsim-nn.onnx
 ```
 
 ## Paper hardware profiles
@@ -109,8 +113,8 @@ The files in
 [`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
 encode Table V's explicit resource parameters.
 Each profile subdirectory contains pre-generated latency and throughput
-`pimsim-nn` configs plus its matching mesh; comparison and validation select
+`pimsim-nn` configs plus its matching mesh; comparison and validation reference
-these checked-in artifacts without generating configs at runtime.
+these canonical artifacts directly.
 
 | Config                       |             Cores | Crossbars/core | Crossbar   |  Cell | PIMCOMP layout  |
 |------------------------------|------------------:|---------------:|------------|------:|-----------------|
@@ -210,36 +214,57 @@ compiles both instruction streams, runs both through `pimsim-nn`, runs
 functional validation through `pim-simulator`, and writes Markdown and JSON
 reports.
 
-To reproduce the complete Arch-A latency experiment, use the model-by-model
+To reproduce all configured architectures and both latency/throughput modes,
-runner. It verifies the paper GA settings, builds Raptor and the existing
+use the model-by-model runner. It verifies the paper GA settings, builds Raptor
-`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
+and the existing `third_party/PIMCOMP-NN/build` tree, then runs the comparisons
-for one model at a time and regenerates `results.csv` from the JSON reports:
+in parallel and regenerates `results.csv` from the JSON reports:
 
 ```bash
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
 
-Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
+Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
-`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and
+
-`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates
+Each model directory has a shared ignored `common/` directory containing
+`inputs/`, `outputs/`, and the native `runner/`.
+Pimsim-nn configs and network meshes remain canonical under
+`validation/pimsim_configs/pimcomp/` and are referenced in place.
+Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
+artifacts remain under each comparison directory; PIMCOMP outputs are prepared
+once per model/architecture/mode and linked into the other pipeline directories;
+`comparison_report.{md,json}`
+live under its `pimcomp/`. The frontend regenerates
 one isolated `models/JSON/` graph because PIMCOMP requires that relative
 layout; it is removed after a successful backend run and the shared submodule
-model directory is never modified. Models requiring BatchNormalization folding
+model directory is never modified. The original ONNX model is passed to the
-also receive a prepared ONNX file; other models use the original ONNX directly.
+frontend unchanged.
 PIMCOMP's source tree and build directory remain unchanged at runtime. Use
-`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run`
+`--models vgg8` to run one model, `--mode throughput` to select one mode,
+`--pipeline 4` to select one throughput pipeline, `--only raptor` or
+`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
 to inspect every command, or `--out-dir PATH` to keep results outside
-`validation/`. The runner continues after a failed model so all reports are
+`validation/`. Use `--clean` to remove generated comparison artifacts and
-produced.
+summaries. Selecting a subset replaces only those comparison rows and
+recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
+reference runner are generated even for an isolated run. Use `--jobs 4` to cap
+parallel comparisons. The per-stage timeout is unlimited by default; pass a
+positive `--timeout-seconds` value to impose one.
+PIMCOMP receives the original ONNX model, and its frontend applies native
+BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
+The runner continues after a failed model so all reports are produced.
+
+The known PIMCOMP batch-scheduling correctness issue and a reproducible
+reference-intermediate prefill experiment are documented in
+[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
 
 Arch-A low-latency example:
 
 ```bash
 RAPTOR_ROOT=$PWD
 
-"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \
+"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
   --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
-  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
+  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
   --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
   --core-count 168 \
   --crossbar-count 96 \
@@ -251,7 +276,7 @@ RAPTOR_ROOT=$PWD
 ```
 
 Use the same command with
-`yolo11n/yolo11n-latency.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
+`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
 compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
 for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
 as YOLO11n would produce a misleading latency, so no PIMCOMP number is
@@ -297,7 +322,7 @@ either latency-only artifact for semantic validation.
 Current Raptor status:
 
 - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
-- Use `googlenet-12-latency.onnx` for the paper-matched latency comparison.
+- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
   It removes the two LRN nodes and terminal softmax that PIMCOMP does not
   schedule.
 - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
@@ -319,8 +344,8 @@ rsync -azL validation/networks/pimcomp_models/ \
   "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
 rsync -az validation/pimsim_configs/pimcomp/ \
   "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
-rsync -az validation/tools/pimcomp/ \
+rsync -az validation/tools/pim/ \
-  "monolith:$REMOTE_REPO/validation/tools/pimcomp/"
+  "monolith:$REMOTE_REPO/validation/tools/pim/"
 rsync -az --exclude=.git --exclude=build --exclude=output \
   third_party/PIMCOMP-NN/ \
   "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
@@ -334,10 +359,10 @@ cd /home/gmagnani/Project/Raptor
 
 # One-time setup if the repository virtual environment is absent.
 python3 -m venv .venv
-.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
+.venv/bin/python -m pip install numpy onnx onnxruntime colorama
 
-# Run every latency comparison serially.
+# Run every configured comparison in parallel.
-.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
 ```
 
 Copy reports back without transferring large compiler artifacts:

validation/networks/pimcomp_models/results.csv

@@ -1,5 +1,41 @@
-model,raptor_latency_ms,pimcomp_latency_ms,raptor_energy_pj,pimcomp_energy_pj,faster_compiler,speedup
+model,arch,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_latency_ms,pimcomp_latency_ms,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
-vgg8,1.465778,7.985074,477298145.040001,1597904071.120000,raptor,5.45
+vgg8,arch-a,latency,1,element,PASS,PASS,NA,NA,1.465778,7.985074,325.627854,200.111367,477298145.040001,1597904071.120000,raptor,5.45
-resnet18,28.099952,58.853733,8781611766.119984,13983168748.119974,raptor,2.09
+vgg8,arch-b,latency,1,element,PASS,PASS,NA,NA,1.438869,7.152125,304.673458,173.768633,438385194.040001,1242814988.120001,raptor,4.97
-resnet34,45.781486,91.607980,14962940227.679951,22722922369.680016,raptor,2.00
+vgg8,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-googlenet,13.371204,62.923463,6117835798.919991,14547526780.240000,raptor,4.71
+vgg8,arch-a,throughput,2,batch,PASS,PASS,1480.000000,1380.000000,0.674197,0.725548,456.064662,475.694103,307477506.200000,345138738.300000,raptor,1.07
+vgg8,arch-b,throughput,2,batch,PASS,PASS,1160.000000,1080.000000,0.861527,0.921878,333.045167,408.540967,286927542.600000,376624916.800000,raptor,1.07
+vgg8,arch-c,throughput,2,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+vgg8,arch-a,throughput,4,batch,PASS,PASS,2160.000000,1380.000000,0.462342,0.725548,446.528150,475.694103,206448818.800000,345138738.300000,raptor,1.57
+vgg8,arch-c,throughput,4,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+vgg8,arch-a,throughput,8,batch,PASS,PASS,831.000000,1380.000000,1.202894,0.725548,331.341726,475.694103,398569128.700000,345138738.300000,pimcomp,1.66
+vgg8,arch-c,throughput,8,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet18,arch-a,latency,1,element,PASS,PASS,NA,NA,28.099951,58.855175,312.513413,237.590194,8781611597.119984,13983412446.119972,raptor,2.09
+resnet18,arch-b,latency,1,element,PASS,PASS,NA,NA,34.938563,65.084209,254.439797,200.986958,8889760875.119959,13081077194.119965,raptor,1.86
+resnet18,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet18,arch-a,throughput,2,batch,PASS,FAIL,20.000000,76.000000,50.000000,13.149606,263.148477,479.788072,13157423870.000000,6309024249.000000,pimcomp,3.80
+resnet18,arch-b,throughput,2,batch,PASS,FAIL,26.300000,83.200000,38.016529,12.020906,250.553072,483.029728,9525158116.000000,5806454916.000000,pimcomp,3.16
+resnet18,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet18,arch-a,throughput,4,batch,PASS,FAIL,31.700000,76.000000,31.578947,13.149606,336.949886,479.788072,10640522720.000000,6309024249.000000,pimcomp,2.40
+resnet18,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet18,arch-a,throughput,8,batch,PASS,FAIL,46.400000,76.000000,21.566110,13.149606,319.292391,479.788072,6885894954.000000,6309024249.000000,pimcomp,1.64
+resnet18,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet34,arch-a,latency,1,element,PASS,PASS,NA,NA,45.781484,91.608751,326.833876,248.044564,14962939889.679951,22723052668.680016,raptor,2.00
+resnet34,arch-b,latency,1,element,PASS,PASS,NA,NA,72.119607,94.519582,239.192092,215.513439,17250439680.679901,20370240175.680019,raptor,1.31
+resnet34,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet34,arch-a,throughput,2,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
+resnet34,arch-b,throughput,2,batch,PASS,FAIL,11.600000,43.300000,86.250000,23.076923,260.049856,489.500435,22429300100.000000,11296163880.000000,pimcomp,3.73
+resnet34,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet34,arch-a,throughput,4,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
+resnet34,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+resnet34,arch-a,throughput,8,batch,PASS,FAIL,24.300000,40.800000,41.176471,24.522761,271.814969,506.131154,11192381060.000000,12411733160.000000,pimcomp,1.68
+resnet34,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+googlenet,arch-a,latency,1,element,PASS,PASS,NA,NA,13.032305,62.923369,465.072002,231.194088,6060960174.919998,14547510894.240002,raptor,4.83
+googlenet,arch-b,latency,1,element,PASS,PASS,NA,NA,16.086822,37.935747,378.951815,242.039077,6096130396.919978,9181933205.239973,raptor,2.36
+googlenet,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+googlenet,arch-a,throughput,2,batch,PASS,FAIL,55.000000,66.200000,18.181818,15.094340,346.696726,434.959232,6303576840.000000,6565422375.000000,pimcomp,1.20
+googlenet,arch-b,throughput,2,batch,PASS,FAIL,55.000000,72.000000,18.181818,13.888889,314.645370,417.992095,5720824912.000000,5805445765.000000,pimcomp,1.31
+googlenet,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+googlenet,arch-a,throughput,4,batch,PASS,FAIL,111.000000,66.200000,9.012016,15.094340,373.195227,434.959232,3363241363.000000,6565422375.000000,raptor,1.68
+googlenet,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
+googlenet,arch-a,throughput,8,batch,PASS,FAIL,61.300000,66.200000,16.310680,15.094340,341.592923,434.959232,5571612719.000000,6565422375.000000,pimcomp,1.08
+googlenet,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA

validation/networks/pimcomp_models/validation_results.csv

@@ -2,5 +2,5 @@ Operation,Result,Compile,Host mem,Cores mem,Cores,Xbars,Latency,Power,Energy
 vgg8-mnist-reconstructed,PASS,1.009 s,1.37 MiB,3.14 MiB,141,761,1.465778 ms,325.627854 mW,477298145.040001 pJ
 resnet18-v1-7,PASS,11.548 s,9.89 MiB,40.24 MiB,168,7676,28.099952 ms,312.513408 mW,8781611766.119984 pJ
 resnet34-v1-7,PASS,28.495 s,9.90 MiB,48.89 MiB,168,15292,45.781486 ms,326.833870 mW,14962940227.679951 pJ
-googlenet-12-latency,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
+googlenet-12-pimsim-nn,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
-yolo11n-latency,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ
+yolo11n-pimsim-nn,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ

validation/pimsim_configs/pimcomp/arch-a/throughput_config_100ms.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 96,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          128,
+          128
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        12,
+        14
+      ],
+      "net_config_file_path": "network_mesh_168.json"
+    },
+    "core_cnt": 168,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 100,
+    "report_verbose_level": 0
+  }
+}

validation/pimsim_configs/pimcomp/arch-b/throughput_config_100ms.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 128,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          128,
+          128
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        6,
+        23
+      ],
+      "net_config_file_path": "network_mesh_138.json"
+    },
+    "core_cnt": 138,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 100,
+    "report_verbose_level": 0
+  }
+}

validation/pimsim_configs/pimcomp/arch-c/throughput_config_100ms.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 8,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          512,
+          1024
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        8,
+        8
+      ],
+      "net_config_file_path": "network_mesh_64.json"
+    },
+    "core_cnt": 64,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 100,
+    "report_verbose_level": 0
+  }
+}

validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py

@@ -17,14 +17,14 @@ import types
 from collections import Counter
 from dataclasses import asdict, dataclass
 from pathlib import Path
+from tempfile import TemporaryDirectory
 from typing import Any
 
 import numpy as np
-import onnx
 from colorama import Fore, Style
 
 
-REPO = Path(__file__).resolve().parents[3]
+REPO = Path(__file__).resolve().parents[5]
 VALIDATION_DIR = REPO / "validation"
 PIMSIM_CONFIG_DIR = VALIDATION_DIR / "pimsim_configs/pimcomp"
 PIMCOMP_OUTPUT_FILES = ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
@@ -156,7 +156,7 @@ def run_logged(
             stdout=subprocess.PIPE,
             stderr=subprocess.STDOUT,
             text=True,
-            timeout=timeout_sec,
+            timeout=None if timeout_sec == 0 else timeout_sec,
         )
     except subprocess.TimeoutExpired as exc:
         duration = time.perf_counter() - start
@@ -210,31 +210,24 @@ def load_saved_inputs(
     return arrays
 
 
-def prepare_pimcomp_model(model_path: Path, out_dir: Path) -> Path:
+def reference_outputs_exist(
-    model = onnx.load(model_path)
+    outputs_desc: list[tuple[int, str, int, list[int]]],
-    if not any(node.op_type == "BatchNormalization" for node in model.graph.node):
+    reference_dir: Path,
-        return model_path
+) -> bool:
+    return reference_dir.is_dir() and all(
+        (reference_dir / f"output{idx}_{sanitize_output_name(name)}.csv").is_file()
+        for idx, name, _, _ in outputs_desc
+    )
 
-    out_dir.mkdir(parents=True, exist_ok=True)
-    output_path = out_dir / f"{model_path.stem}_pimcomp.onnx"
-    from onnxsim import simplify
 
-    model, equivalent = simplify(model, check_n=1)
+def reference_inputs_exist(
-    if not equivalent:
+    inputs_desc: list[tuple[int, str, int, list[int]]],
-        raise RuntimeError("Conv+BatchNormalization folding changed the model output")
+    inputs_dir: Path,
-    if any(node.op_type == "BatchNormalization" for node in model.graph.node):
+) -> bool:
-        import onnxruntime as ort
+    return inputs_dir.is_dir() and all(
-
+        (inputs_dir / f"in{idx}.csv").is_file()
-        options = ort.SessionOptions()
+        for idx, _, _, _ in inputs_desc
-        options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC
+    )
-        options.optimized_model_filepath = str(output_path)
-        ort.InferenceSession(str(model_path), options, providers=["CPUExecutionProvider"])
-        model = onnx.load(output_path)
-        if any(node.op_type == "BatchNormalization" for node in model.graph.node):
-            raise RuntimeError("PIMCOMP model preparation did not eliminate BatchNormalization")
-    else:
-        onnx.save(model, output_path)
-    return output_path
 
 
 def compare_simulator_outputs(
@@ -285,8 +278,12 @@ def load_effective_hardware(args: argparse.Namespace) -> dict[str, int]:
 
 
 def select_pimsim_config(args: argparse.Namespace, hardware: dict[str, int]) -> Path:
-    fallback: Path | None = None
+    filename = (
-    for path in sorted(PIMSIM_CONFIG_DIR.glob(f"*/{args.pimsim_mode}_config.json")):
+        "latency_config.json"
+        if args.pimsim_mode == "latency"
+        else f"throughput_config_{args.pimsim_time_ms}ms.json"
+    )
+    for path in sorted(PIMSIM_CONFIG_DIR.glob(f"*/{filename}")):
         with open(path, encoding="utf-8") as f:
             config = json.load(f)
         chip = config["chip_config"]
@@ -299,41 +296,18 @@ def select_pimsim_config(args: argparse.Namespace, hardware: dict[str, int]) ->
             and network["layout"] == [hardware["mesh_rows"], hardware["mesh_cols"]]
             and config["sim_config"]["sim_mode"] == (1 if args.pimsim_mode == "latency" else 0)
         ):
-            if config["sim_config"]["sim_time"] == args.pimsim_time_ms:
+            if args.pimsim_mode == "latency" or config["sim_config"]["sim_time"] == args.pimsim_time_ms:
                 return path
-            fallback = fallback or path
-    if fallback is not None:
-        return fallback
     raise ValueError(
-        f"No pre-generated {args.pimsim_mode} pimsim-nn config matches {hardware}"
+        f"No pre-generated {args.pimsim_mode} pimsim-nn config for {args.pimsim_time_ms} ms matches {hardware}"
     )
 
 
 def prepare_pimsim_config(
     args: argparse.Namespace,
     hardware: dict[str, int],
-    out_dir: Path,
 ) -> Path:
-    source = select_pimsim_config(args, hardware)
+    return select_pimsim_config(args, hardware)
-    with open(source, encoding="utf-8") as f:
-        config = json.load(f)
-    if config["sim_config"]["sim_time"] == args.pimsim_time_ms:
-        return source
-
-    config["sim_config"]["sim_time"] = args.pimsim_time_ms
-    target = out_dir / "pimsim_config.json"
-    target.parent.mkdir(parents=True, exist_ok=True)
-    target.write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8")
-
-    network_path = Path(config["chip_config"]["network_config"]["net_config_file_path"])
-    if not network_path.is_absolute():
-        network_path = source.parent / network_path
-    target_network = target.parent / Path(
-        config["chip_config"]["network_config"]["net_config_file_path"]
-    )
-    target_network.parent.mkdir(parents=True, exist_ok=True)
-    shutil.copy2(network_path, target_network)
-    return target
 
 
 def compile_reference(
@@ -395,6 +369,7 @@ def compile_reference(
         timeout_sec=args.timeout_seconds,
         steps=steps,
     )
+    shutil.rmtree(raptor_dir)
     return build_dir / "runner"
 
 
@@ -422,6 +397,33 @@ def generate_reference_outputs(
     return reference_dir
 
 
+def prepare_common_artifacts(
+    args: argparse.Namespace,
+    model_path: Path,
+    common_dir: Path,
+) -> None:
+    inputs_desc, outputs_desc, arrays_in_order = load_model_inputs(model_path, args.seed)
+    inputs_dir = common_dir / "inputs"
+    outputs_dir = common_dir / "outputs"
+    runner_path = common_dir / "runner/build/runner"
+    steps: list[StepRecord] = []
+    if not runner_path.exists():
+        runner_path = compile_reference(args, model_path, common_dir, steps)
+
+    inputs_ready = reference_inputs_exist(inputs_desc, inputs_dir)
+    outputs_ready = reference_outputs_exist(outputs_desc, outputs_dir)
+    if inputs_ready:
+        arrays_in_order = load_saved_inputs(inputs_desc, inputs_dir)
+    if not (inputs_ready and outputs_ready):
+        generate_reference_outputs(
+            runner_path,
+            runner_path.parent,
+            model_path,
+            arrays_in_order,
+            steps,
+            args,
+            common_dir,
+        )
 def compile_raptor_target(
     model_path: Path,
     out_dir: Path,
@@ -538,9 +540,13 @@ def run_functional_validation(
 
 
 def copy_pimcomp_outputs(source_dir: Path, out_dir: Path):
+    if source_dir.resolve() == out_dir.resolve():
+        return
     out_dir.mkdir(parents=True, exist_ok=True)
     for name in PIMCOMP_OUTPUT_FILES:
-        shutil.copy2(source_dir / name, out_dir / name)
+        destination = out_dir / name
+        destination.unlink(missing_ok=True)
+        destination.symlink_to(source_dir / name)
 
 
 def compile_pimcomp(
@@ -560,11 +566,16 @@ def compile_pimcomp(
         (pimcomp_output_dir / name).unlink(missing_ok=True)
     model_name = args.pimcomp_model_name or f"compare_{model_path.stem}"
     frontend_json = frontend_json_dir / f"{model_name}.json"
+    # The original PIMCOMP frontend rewrites its input ONNX while loading it.
+    # Isolate that mutation without changing the model sent through PIMCOMP.
+    with TemporaryDirectory(prefix="pimcomp-model-") as temp_dir:
+        frontend_model = Path(temp_dir) / model_path.name
+        shutil.copy2(model_path, frontend_model)
         frontend_cmd = [
             sys.executable,
             str(args.pimcomp_dir / "frontend/frontend.py"),
             "--model_path",
-        str(model_path),
+            str(frontend_model),
             "--save_path",
             str(frontend_json),
         ]
@@ -584,6 +595,7 @@ def compile_pimcomp(
         "-v=YES",
         "-s=YES",
     ]
+    try:
         run_logged(
             "Compile PIMCOMP Backend",
             backend_cmd,
@@ -592,7 +604,8 @@ def compile_pimcomp(
             steps=steps,
             stage="Compile PIM",
         )
-    shutil.rmtree(frontend_json_dir.parent)
+    finally:
+        shutil.rmtree(frontend_json_dir.parent, ignore_errors=True)
     return pimcomp_output_dir / "VerificationInfo.json", pimcomp_output_dir / "SimulationInfo.gz"
 
 
@@ -1001,6 +1014,16 @@ def optional_path(path: Path | None) -> str | None:
     return str(path) if path is not None else None
 
 
+def default_common_dir(out_dir: Path) -> Path:
+    if out_dir.name == "latency":
+        return out_dir.parents[1] / "common"
+    if out_dir.parent.name == "throughput":
+        return out_dir.parents[2] / "common"
+    if out_dir.name.startswith("arch-"):
+        return out_dir.parent / "common"
+    return out_dir / "common"
+
+
 def record_failure(failures: list[dict[str, str]], stage: str, exc: BaseException | str) -> None:
     message = exc if isinstance(exc, str) else exception_message(exc)
     failures.append({"stage": stage, "error": message})
@@ -1204,6 +1227,16 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument("--model", required=True, type=Path)
     parser.add_argument("--out-dir", required=True, type=Path)
+    parser.add_argument(
+        "--common-dir",
+        type=Path,
+        help="Shared per-model directory for reference inputs, outputs, runner, and prepared model.",
+    )
+    parser.add_argument(
+        "--prepare-common",
+        action="store_true",
+        help="Prepare shared model artifacts and exit before a comparison.",
+    )
     parser.add_argument("--raptor-path", default=REPO / "build_release/Release/bin/onnx-mlir", type=Path)
     parser.add_argument("--onnx-include-dir", default=REPO / "onnx-mlir/include", type=Path)
     parser.add_argument("--pimcomp-dir", default=REPO / "third_party/PIMCOMP-NN", type=Path)
@@ -1217,7 +1250,12 @@ def main():
     parser.add_argument("--seed", type=int, default=0)
     parser.add_argument("--threshold", type=float, default=1e-3)
     parser.add_argument("--rtol", type=float, default=1e-5)
-    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
+    parser.add_argument(
+        "--timeout-seconds",
+        type=float,
+        default=0.0,
+        help="Per-stage timeout in seconds; 0 means no timeout (default).",
+    )
     parser.add_argument("--core-count", type=int)
     parser.add_argument("--crossbar-count", type=int)
     parser.add_argument("--crossbar-size", type=int)
@@ -1253,6 +1291,8 @@ def main():
     args = parser.parse_args()
     if args.pimsim_time_ms <= 0:
         parser.error("--pimsim-time-ms must be positive")
+    if args.timeout_seconds < 0:
+        parser.error("--timeout-seconds must be non-negative")
     if args.pimcomp_pipeline is None:
         args.pimcomp_pipeline = "element" if args.pimsim_mode == "latency" else "batch"
 
@@ -1265,6 +1305,18 @@ def main():
     )
     out_dir = args.out_dir.resolve()
     out_dir.mkdir(parents=True, exist_ok=True)
+    common_dir = (
+        args.common_dir.resolve()
+        if args.common_dir is not None
+        else default_common_dir(out_dir).resolve()
+    )
+    common_dir.mkdir(parents=True, exist_ok=True)
+
+    if args.prepare_common:
+        print_step("Prepare shared artifacts")
+        prepare_common_artifacts(args, model_path, common_dir)
+        print(f"Shared artifacts: {common_dir}")
+        return
 
     failures: list[dict[str, str]] = []
     steps: list[StepRecord] = []
@@ -1308,6 +1360,22 @@ def main():
         inputs_desc, outputs_desc, arrays_in_order = model_io
         runtime_inputs = arrays_in_order
 
+    reuse_pimcomp = args.reuse_pimcomp_dir is not None
+    if reuse_pimcomp:
+        reused_pimcomp_report_path = args.reuse_pimcomp_dir.resolve().parent / "comparison_report.json"
+        if not reused_pimcomp_report_path.exists():
+            raise ValueError(f"Missing PIMCOMP report beside {args.reuse_pimcomp_dir}")
+        with open(reused_pimcomp_report_path, "r", encoding="utf-8") as f:
+            reused_pimcomp = json.load(f)
+        if reused_pimcomp.get("pimcomp_model_source") != "original_onnx":
+            raise ValueError("Reused PIMCOMP artifacts were not generated from the original ONNX model")
+        pimcomp_validation = CompareResult(**reused_pimcomp["pimcomp_validation"])
+        pimcomp_perf = reused_pimcomp["pimcomp_performance"]
+        pimcomp_instr = reused_pimcomp["pimcomp_instruction_summary"]
+        reused_config = reused_pimcomp.get("paths", {}).get("pimsim_config")
+        if reused_config:
+            pimsim_config = Path(reused_config)
+
     if reuse_raptor and model_io is not None:
         reuse_report_path = args.reuse_raptor_report.resolve()
         with open(reuse_report_path, "r", encoding="utf-8") as f:
@@ -1315,11 +1383,12 @@ def main():
         reused_hardware = reused["hardware"]
         if reused_hardware != hardware:
             raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}")
-        reference_dir = Path(reused["paths"]["reference_outputs"])
+        reference_dir = common_dir / "outputs"
-        raptor_pim_dir = Path(reused["paths"]["raptor_pim"])
+        reused_raptor_pim = reused["paths"].get("raptor_pim")
+        raptor_pim_dir = Path(reused_raptor_pim) if reused_raptor_pim else None
         arrays_in_order = load_saved_inputs(
             inputs_desc,
-            reference_dir.parent / "inputs",
+            common_dir / "inputs",
         )
         runtime_inputs = arrays_in_order
         raptor_validation = CompareResult(**reused["raptor_validation"])
@@ -1329,33 +1398,46 @@ def main():
         print_step("Reuse Raptor")
         print(f"  Report: {reuse_report_path}")
 
-    expected_runner_path = out_dir / "runner/build/runner"
+    expected_runner_path = common_dir / "runner/build/runner"
+    common_inputs_dir = common_dir / "inputs"
+    common_reference_dir = common_dir / "outputs"
 
     if not reuse_raptor:
+        inputs_ready = reference_inputs_exist(inputs_desc, common_inputs_dir)
+        if inputs_ready and model_io is not None:
+            saved_inputs = try_stage(
+                failures,
+                "Load shared reference inputs",
+                load_saved_inputs,
+                inputs_desc,
+                common_inputs_dir,
+            )
+            if saved_inputs is not None:
+                arrays_in_order = saved_inputs
+                runtime_inputs = arrays_in_order
+            else:
+                inputs_ready = False
+
+        if expected_runner_path.exists():
+            runner_path = expected_runner_path
+        else:
             reference_compile = try_stage(
                 failures,
                 "Compile reference",
                 compile_reference,
                 args,
                 model_path,
-            out_dir,
+                common_dir,
                 steps,
             )
             if reference_compile is not None:
                 runner_path = reference_compile
-        else:
-            if expected_runner_path.exists():
-                runner_path = expected_runner_path
-                print(
-                    "\n"
-                    + Style.BRIGHT
-                    + Fore.YELLOW
-                    + "[Continue]"
-                    + Style.RESET_ALL
-                    + f" Reusing partial runner: {runner_path}"
-                )
 
-    if not reuse_raptor and runner_path is not None and runner_path.exists() and model_io is not None:
+        if runner_path is not None and runner_path.exists() and model_io is not None:
+            if inputs_ready and reference_outputs_exist(outputs_desc, common_reference_dir):
+                reference_dir = common_reference_dir
+                print_step("Reuse shared reference")
+            else:
                 generated_reference = try_stage(
                     failures,
                     "Run reference",
@@ -1366,11 +1448,11 @@ def main():
                     arrays_in_order,
                     steps,
                     args,
-            out_dir,
+                    common_dir,
                 )
                 if generated_reference is not None:
                     reference_dir = generated_reference
-    elif not reuse_raptor:
+        else:
             record_failure(
                 failures,
                 "Skip reference outputs",
@@ -1430,13 +1512,7 @@ def main():
     elif not reuse_raptor:
         raptor_validation = skipped_validation("Raptor PIM compilation did not produce a PIM directory")
 
-    pimcomp_model_path = try_stage(
+    pimcomp_model_path = model_path
-        failures,
-        "Prepare PIMCOMP model",
-        prepare_pimcomp_model,
-        model_path,
-        out_dir / "pimcomp/model",
-    )
 
     if args.reuse_pimcomp_dir is not None:
         reused_pimcomp_dir = args.reuse_pimcomp_dir.resolve()
@@ -1465,7 +1541,9 @@ def main():
         if compiled_pimcomp is not None:
             verification_info, simulation_info = compiled_pimcomp
 
-    if verification_info is not None and simulation_info is not None and model_io is not None:
+    if reuse_pimcomp:
+        pass
+    elif verification_info is not None and simulation_info is not None and model_io is not None:
         exported = try_stage(
             failures,
             "Export PIMCOMP for Functional Validation",
@@ -1491,7 +1569,7 @@ def main():
             "PIMCOMP functional export failed because model inputs are not available.",
         )
 
-    if pimcomp_export_dir is not None and reference_dir is not None and outputs_desc:
+    if not reuse_pimcomp and pimcomp_export_dir is not None and reference_dir is not None and outputs_desc:
         validation = try_stage(
             failures,
             "Functional Validation PIMCOMP",
@@ -1507,6 +1585,8 @@ def main():
             channel_last=True,
         )
         pimcomp_validation = validation if validation is not None else failed_validation("PIMCOMP validation failed")
+    elif reuse_pimcomp:
+        pass
     elif pimcomp_export_dir is None:
         pimcomp_validation = failed_validation("PIMCOMP functional export is not available")
     elif reference_dir is None:
@@ -1521,7 +1601,6 @@ def main():
             prepare_pimsim_config,
             args,
             hardware,
-            out_dir,
         )
         if written_config is not None:
             pimsim_config = written_config
@@ -1566,7 +1645,7 @@ def main():
         elif not reuse_raptor:
             raptor_perf = skipped_perf("Raptor PIM directory is not available")
 
-        if simulation_info is not None:
+        if not reuse_pimcomp and simulation_info is not None:
             pimcomp_pimsim_dir = try_stage(
                 failures,
                 "Export PIMCOMP for pimsim-nn",
@@ -1597,7 +1676,7 @@ def main():
     elif not reuse_raptor:
         raptor_instr = empty_instruction_summary("Raptor PIM directory is not available")
 
-    if simulation_info is not None and simulation_info.exists():
+    if not reuse_pimcomp and simulation_info is not None and simulation_info.exists():
         parsed = try_stage(failures, "Parse PIMCOMP instructions", parse_pimcomp_instructions, simulation_info)
         pimcomp_instr = parsed if parsed is not None else empty_instruction_summary(error="Failed to parse PIMCOMP instructions")
     else:
@@ -1629,6 +1708,7 @@ def main():
         "pimsim_time_ms": args.pimsim_time_ms,
         "pimcomp_pipeline": args.pimcomp_pipeline,
         "pimcomp_replication": args.pimcomp_replication,
+        "pimcomp_model_source": "original_onnx",
         "pimcomp_config": str(args.pimcomp_config),
         "raptor_extra_args": args.raptor_extra_arg,
         "reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None,
@@ -1642,7 +1722,10 @@ def main():
         "pimcomp_instruction_summary": pimcomp_instr,
         "raptor_pass_timings": raptor_pass_timings,
         "paths": {
+            "common_dir": str(common_dir),
+            "reference_inputs": optional_path(common_dir / "inputs"),
             "reference_outputs": optional_path(reference_dir),
+            "reference_runner": optional_path(runner_path),
             "raptor_pim": optional_path(raptor_pim_dir),
             "raptor_pimsim_nn": optional_path(raptor_pimsim_dir),
             "pimcomp_simulation_info": optional_path(simulation_info),

validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py

@@ -0,0 +1,755 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import os
+import shlex
+import shutil
+import subprocess
+import sys
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from contextlib import nullcontext, redirect_stderr, redirect_stdout
+from dataclasses import dataclass
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+from colorama import Fore, Style
+
+
+REPO = Path(__file__).resolve().parents[5]
+SUITE = REPO / "validation/networks/pimcomp_models"
+sys.path.insert(0, str(REPO / "validation"))
+
+from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics  # noqa: E402
+from raptor_validation.validate_one import STAGE_COLORS  # noqa: E402
+
+PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
+PIMCOMP_CONFIGS = REPO / "validation/pimsim_configs/pimcomp"
+COMPARE = Path(__file__).resolve().with_name("compare_raptor_pimcomp.py")
+ARCHES = tuple(sorted(path.name for path in PIMCOMP_CONFIGS.iterdir() if path.is_dir()))
+MODELS = {
+    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
+    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
+    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
+    "googlenet": SUITE / "googlenet/googlenet-12-pimsim-nn.onnx",
+    "yolo11n": SUITE / "yolo11n/yolo11n-pimsim-nn.onnx",
+}
+COMPARISONS = (
+    ("latency", 1, "element"),
+    ("throughput", 2, "batch"),
+    ("throughput", 4, "batch"),
+    ("throughput", 8, "batch"),
+)
+
+
+@dataclass(frozen=True)
+class ComparisonSpec:
+    label: str
+    model: Path
+    output_dir: Path
+    common_dir: Path
+    config: Path
+    mode: str
+    pipeline: int
+    pimcomp_pipeline: str
+    shared_pimcomp_dir: Path
+    anchor: bool
+
+
+def model_dir(root: Path | None, name: str) -> Path:
+    return root / name if root is not None else MODELS[name].parent
+
+
+def result_dir(root: Path | None, name: str, arch: str, mode: str, pipeline: int) -> Path:
+    base = model_dir(root, name)
+    suffix = "latency" if mode == "latency" else f"throughput/pipeline{pipeline}"
+    return base / arch / suffix
+
+
+def clean_artifacts(root: Path | None, models: list[str], arches: list[str]) -> int:
+    removed = 0
+    for name in models:
+        base = model_dir(root, name)
+        for arch in arches:
+            for path in (base / arch / "latency", base / arch / "throughput"):
+                if path.is_dir() and not path.is_symlink():
+                    shutil.rmtree(path)
+                    removed += 1
+        common = base / "common"
+        if common.is_dir() and not common.is_symlink():
+            shutil.rmtree(common)
+            removed += 1
+    for path in (
+        (root or SUITE) / "results.csv",
+        (root or SUITE) / "results_latency.csv",
+        (root or SUITE) / "results_throughput.csv",
+    ):
+        if path.is_file() or path.is_symlink():
+            path.unlink()
+            removed += 1
+    return removed
+
+
+def write_results_csv(
+    root: Path | None,
+    arch: str,
+    models: list[str],
+    comparisons: tuple[tuple[str, int, str], ...] = COMPARISONS,
+) -> Path:
+    output = (root or SUITE) / "results.csv"
+    fields = (
+        "model",
+        "arch",
+        "mode",
+        "raptor_pipeline",
+        "pimcomp_pipeline",
+        "raptor_functional_validation",
+        "pimcomp_functional_validation",
+        "raptor_throughput_samples_s",
+        "pimcomp_throughput_samples_s",
+        "raptor_latency_ms",
+        "pimcomp_latency_ms",
+        "raptor_power_mw",
+        "pimcomp_power_mw",
+        "raptor_energy_pj",
+        "pimcomp_energy_pj",
+        "better_compiler",
+        "speedup",
+    )
+    selected = {
+        (name, arch, mode, str(pipeline))
+        for name in models
+        for mode, pipeline, _ in comparisons
+    }
+    rows = []
+    if output.exists():
+        with open(output, newline="", encoding="utf-8") as f:
+            reader = csv.DictReader(f)
+            if reader.fieldnames and "model" in reader.fieldnames:
+                for row in reader:
+                    key = (
+                        row.get("model"),
+                        row.get("arch"),
+                        row.get("mode"),
+                        row.get("raptor_pipeline"),
+                    )
+                    if key in selected:
+                        continue
+                    row.pop("status", None)
+                    rows.append({field: row.get(field, "NA") for field in fields})
+    for name in models:
+        for comparison_mode, pipeline, pimcomp_pipeline in comparisons:
+            report_path = result_dir(root, name, arch, comparison_mode, pipeline) / "pimcomp/comparison_report.json"
+            row = {
+                "model": name,
+                "arch": arch,
+                "mode": comparison_mode,
+                "raptor_pipeline": pipeline,
+                "pimcomp_pipeline": pimcomp_pipeline,
+                "raptor_functional_validation": "NA",
+                "pimcomp_functional_validation": "NA",
+                "raptor_throughput_samples_s": "NA",
+                "pimcomp_throughput_samples_s": "NA",
+                "raptor_latency_ms": "NA",
+                "pimcomp_latency_ms": "NA",
+                "raptor_power_mw": "NA",
+                "pimcomp_power_mw": "NA",
+                "raptor_energy_pj": "NA",
+                "pimcomp_energy_pj": "NA",
+                "better_compiler": "NA",
+                "speedup": "NA",
+            }
+            if report_path.exists():
+                report = json.loads(report_path.read_text(encoding="utf-8"))
+                raptor_values = performance_values(report.get("raptor_performance") or {})
+                pimcomp_values = performance_values(report.get("pimcomp_performance") or {})
+                raptor_metric = raptor_values["throughput"] if comparison_mode == "throughput" else raptor_values["latency"]
+                pimcomp_metric = pimcomp_values["throughput"] if comparison_mode == "throughput" else pimcomp_values["latency"]
+                row.update(
+                    raptor_functional_validation=functional_validation_status(
+                        report.get("raptor_validation")
+                    ),
+                    pimcomp_functional_validation=functional_validation_status(
+                        report.get("pimcomp_validation")
+                    ),
+                    raptor_throughput_samples_s=format_value(raptor_values["throughput"]),
+                    pimcomp_throughput_samples_s=format_value(pimcomp_values["throughput"]),
+                    raptor_latency_ms=format_value(raptor_values["latency"]),
+                    pimcomp_latency_ms=format_value(pimcomp_values["latency"]),
+                    raptor_power_mw=format_value(raptor_values["power"]),
+                    pimcomp_power_mw=format_value(pimcomp_values["power"]),
+                    raptor_energy_pj=format_value(raptor_values["energy"]),
+                    pimcomp_energy_pj=format_value(pimcomp_values["energy"]),
+                )
+                if raptor_metric is not None and pimcomp_metric is not None:
+                    row.update(
+                        better_compiler=comparison_winner(comparison_mode, raptor_metric, pimcomp_metric),
+                        speedup=f"{max(raptor_metric, pimcomp_metric) / min(raptor_metric, pimcomp_metric):.2f}",
+                    )
+            rows.append(row)
+    rows.sort(key=lambda row: (list(MODELS).index(row["model"]), row["mode"], int(row["raptor_pipeline"])))
+    with open(output, "w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
+        writer.writeheader()
+        writer.writerows(rows)
+    return output
+
+
+def performance_values(performance: dict) -> dict[str, float | None]:
+    parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
+    return {
+        "throughput": performance.get("throughput") or parsed.get("throughput"),
+        "latency": (
+            performance.get("latency_ms")
+            or performance.get("average_latency_ms")
+            or parsed.get("latency_ms")
+            or parsed.get("average_latency_ms")
+        ),
+        "power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
+        "energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
+    }
+
+
+def comparison_passed(report: dict) -> bool:
+    if report.get("failures"):
+        return False
+    for key in ("raptor_validation", "pimcomp_validation"):
+        result = report.get(key) or {}
+        if result.get("status") != "done" or not result.get("passed"):
+            return False
+    for key in ("raptor_performance", "pimcomp_performance"):
+        performance = report.get(key) or {}
+        if performance.get("error") or performance.get("skipped"):
+            return False
+    return True
+
+
+def functional_validation_status(result: dict | None) -> str:
+    if result is None:
+        return "NA"
+    return "PASS" if result.get("status") == "done" and result.get("passed") else "FAIL"
+
+
+def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
+    if raptor == pimcomp:
+        return "tie"
+    if mode == "throughput":
+        return "raptor" if raptor > pimcomp else "pimcomp"
+    return "raptor" if raptor < pimcomp else "pimcomp"
+
+
+def format_value(value: float | None) -> str:
+    return "NA" if value is None else f"{value:.6f}"
+
+
+def print_stage(title: str, color: str) -> None:
+    print("\n" + Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL, flush=True)
+
+
+def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
+    print(f"  cwd: {REPO}", flush=True)
+    print(f"  $ {shlex.join(command)}", flush=True)
+    if dry_run:
+        return 0
+    return subprocess.run(command, cwd=REPO, check=check).returncode
+
+
+def run_comparison_job(job: tuple[str, list[str], Path | None]) -> tuple[str, int, str | None]:
+    label, command, log_path = job
+    sys.stdout.flush()
+    sys.stderr.flush()
+    if log_path is None:
+        return label, run(command, dry_run=False, check=False), None
+
+    saved_stdout = os.dup(1)
+    saved_stderr = os.dup(2)
+    try:
+        with open(log_path, "w", encoding="utf-8", buffering=1) as log:
+            os.dup2(log.fileno(), 1)
+            os.dup2(log.fileno(), 2)
+            with redirect_stdout(log), redirect_stderr(log):
+                returncode = run(command, dry_run=False, check=False)
+    finally:
+        os.dup2(saved_stdout, 1)
+        os.dup2(saved_stderr, 2)
+        os.close(saved_stdout)
+        os.close(saved_stderr)
+    return label, returncode, str(log_path)
+
+
+def validate_pimcomp_source() -> None:
+    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
+    source = header.read_text(encoding="utf-8")
+    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
+        if setting not in source:
+            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
+
+
+def comparison_command(
+    model: Path,
+    result_dir: Path,
+    common_dir: Path,
+    config: Path,
+    mode: str,
+    pipeline: int,
+    pimcomp_pipeline: str,
+    pimsim_time_ms: int,
+    timeout: float,
+    reuse_raptor_report: Path | None = None,
+    reuse_pimcomp_dir: Path | None = None,
+) -> list[str]:
+    time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
+    reuse_args = []
+    if reuse_raptor_report is not None:
+        reuse_args.extend(["--reuse-raptor-report", str(reuse_raptor_report)])
+    if reuse_pimcomp_dir is not None:
+        reuse_args.extend(["--reuse-pimcomp-dir", str(reuse_pimcomp_dir)])
+    return [
+        sys.executable,
+        str(COMPARE),
+        "--model",
+        str(model),
+        "--out-dir",
+        str(result_dir),
+        "--common-dir",
+        str(common_dir),
+        "--pimcomp-dir",
+        str(PIMCOMP_SOURCE),
+        "--pimcomp-config",
+        str(config),
+        "--pimsim-mode",
+        mode,
+        *time_args,
+        "--pimcomp-pipeline",
+        pimcomp_pipeline,
+        "--pimcomp-replication",
+        "GA",
+        f"--raptor-extra-arg=--pipeline={pipeline}",
+        "--timeout-seconds",
+        str(timeout),
+        "--fail-on-error",
+        *reuse_args,
+    ]
+
+
+def prepare_common_command(model: Path, common_dir: Path, timeout: float) -> list[str]:
+    return [
+        sys.executable,
+        str(COMPARE),
+        "--model",
+        str(model),
+        "--out-dir",
+        str(common_dir),
+        "--common-dir",
+        str(common_dir),
+        "--prepare-common",
+        "--timeout-seconds",
+        str(timeout),
+    ]
+
+
+def comparison_command_for(
+    spec: ComparisonSpec,
+    args: argparse.Namespace,
+    *,
+    reuse_shared_pimcomp: bool,
+) -> list[str]:
+    reuse_pimcomp_dir = None
+    if reuse_shared_pimcomp:
+        reuse_pimcomp_dir = spec.shared_pimcomp_dir
+    elif args.only == "raptor":
+        reuse_pimcomp_dir = spec.output_dir / "pimcomp/output"
+    return comparison_command(
+        spec.model,
+        spec.output_dir,
+        spec.common_dir,
+        spec.config,
+        spec.mode,
+        spec.pipeline,
+        spec.pimcomp_pipeline,
+        args.pimsim_time_ms,
+        args.timeout_seconds,
+        reuse_raptor_report=(
+            spec.output_dir / "pimcomp/comparison_report.json"
+            if args.only == "pimcomp"
+            else None
+        ),
+        reuse_pimcomp_dir=reuse_pimcomp_dir,
+    )
+
+
+def pimcomp_artifact_ready(output_dir: Path) -> bool:
+    return all(
+        (output_dir / name).is_file()
+        for name in ("SimulationInfo.gz", "VerificationInfo.json", "MappingResult.txt")
+    ) and (output_dir.parent / "comparison_report.json").is_file()
+
+
+def run_comparison_jobs(
+    comparison_jobs: list[tuple[str, list[str], Path | None]],
+    max_workers: int,
+    log_dir: Path | None,
+    log_offset: int,
+) -> tuple[list[str], int]:
+    if not comparison_jobs:
+        return [], log_offset
+    print_directly = min(max_workers, len(comparison_jobs)) == 1
+    jobs = [
+        (
+            label,
+            command,
+            None if print_directly or log_dir is None else log_dir / f"{log_offset + index}.log",
+        )
+        for index, (label, command, _) in enumerate(comparison_jobs)
+    ]
+    with (nullcontext(None) if print_directly else ProcessPoolExecutor(max_workers=max_workers)) as executor:
+        completed = (
+            map(run_comparison_job, jobs)
+            if print_directly
+            else (
+                future.result()
+                for future in as_completed(executor.submit(run_comparison_job, job) for job in jobs)
+            )
+        )
+        failed = []
+        for label, returncode, result_log in completed:
+            if result_log:
+                output = Path(result_log).read_text(encoding="utf-8", errors="replace")
+                if output:
+                    print(output, end="" if output.endswith("\n") else "\n")
+            if returncode:
+                failed.append(label)
+    return failed, log_offset + len(jobs)
+
+
+def ensure_throughput_config(arch: str, sim_time_ms: int) -> Path:
+    source = PIMCOMP_CONFIGS / arch / "throughput_config_1000ms.json"
+    target = PIMCOMP_CONFIGS / arch / f"throughput_config_{sim_time_ms}ms.json"
+    if target.exists():
+        return target
+    if sim_time_ms == 1000:
+        return source
+    config = json.loads(source.read_text(encoding="utf-8"))
+    config["sim_config"]["sim_time"] = sim_time_ms
+    target.write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8")
+    return target
+
+
+def config_path(arch: str, mode: str, sim_time_ms: int, *, write: bool) -> Path:
+    path = (
+        PIMCOMP_CONFIGS / arch / "latency_config.json"
+        if mode == "latency"
+        else PIMCOMP_CONFIGS / arch / f"throughput_config_{sim_time_ms}ms.json"
+    )
+    if mode == "throughput" and write:
+        path = ensure_throughput_config(arch, sim_time_ms)
+    if not path.exists() and (write or mode == "latency"):
+        raise ValueError(f"{arch} has no {mode} config: {path}")
+    return path
+
+
+def core_count(config: Path) -> int:
+    if not config.exists() and config.name.startswith("throughput_config_"):
+        config = config.with_name("throughput_config_1000ms.json")
+    with open(config, encoding="utf-8") as f:
+        return int(json.load(f)["chip_config"]["core_cnt"])
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Compare supported PIMCOMP models with Raptor latency and throughput schedules."
+    )
+    parser.add_argument(
+        "--out-dir",
+        type=Path,
+        help="Result root (default: artifacts beside each model under validation/).",
+    )
+    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
+    parser.add_argument(
+        "--arch",
+        nargs="+",
+        choices=ARCHES,
+        default=list(ARCHES),
+        help="PIM architectures to run (default: all architectures).",
+    )
+    parser.add_argument(
+        "--mode",
+        nargs="+",
+        choices=("latency", "throughput"),
+        default=["latency", "throughput"],
+        help="Pimsim modes to run (default: both).",
+    )
+    parser.add_argument(
+        "--only",
+        choices=("raptor", "pimcomp"),
+        help="Re-run only this compiler and reuse the other side's existing artifacts.",
+    )
+    parser.add_argument(
+        "--pipeline",
+        type=int,
+        choices=(1, 2, 4, 8),
+        help="Run only this Raptor pipeline within the selected mode.",
+    )
+    parser.add_argument(
+        "--pimsim-time-ms",
+        type=int,
+        default=100,
+        help="throughput pimsim-nn horizon in ms (default: 100).",
+    )
+    parser.add_argument(
+        "--timeout-seconds",
+        type=float,
+        default=0.0,
+        help="Per-stage timeout in seconds; 0 means no timeout (default).",
+    )
+    parser.add_argument(
+        "-j",
+        "--jobs",
+        type=int,
+        default=os.cpu_count() or 1,
+        help="Number of comparisons to run in parallel (default: all available CPUs).",
+    )
+    parser.add_argument(
+        "--clean",
+        action="store_true",
+        help="Remove generated comparison artifacts and result summaries, then exit.",
+    )
+    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
+    args = parser.parse_args()
+
+    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
+    args.arch = list(dict.fromkeys(args.arch))
+    args.mode = list(dict.fromkeys(args.mode))
+    if args.clean:
+        print(f"Removed {clean_artifacts(out_dir, args.models, args.arch)} comparison artifact path(s).")
+        return 0
+
+    if args.jobs < 1:
+        parser.error("--jobs must be at least 1")
+    if args.pimsim_time_ms <= 0:
+        parser.error("--pimsim-time-ms must be positive")
+    if args.timeout_seconds < 0:
+        parser.error("--timeout-seconds must be non-negative")
+    comparisons_by_arch: dict[str, tuple[tuple[str, int, str], ...]] = {}
+    configs_by_arch: dict[str, dict[str, Path]] = {}
+    for arch in args.arch:
+        comparisons = tuple(
+            comparison for comparison in COMPARISONS
+            if comparison[0] in args.mode
+            and (args.pipeline is None or comparison[1] == args.pipeline)
+        )
+        if not comparisons:
+            parser.error("--pipeline does not belong to the selected --mode")
+        configs = {
+            mode: config_path(arch, mode, args.pimsim_time_ms, write=not args.dry_run)
+            for mode, _, _ in comparisons
+        }
+        unsupported = [
+            pipeline for mode, pipeline, _ in comparisons
+            if mode == "throughput" and core_count(configs[mode]) % pipeline
+        ]
+        if unsupported:
+            if args.pipeline is not None:
+                parser.error(
+                    f"{arch} has {core_count(configs['throughput'])} cores; "
+                    f"throughput pipelines must divide that count (invalid: {unsupported})"
+                )
+            print(
+                Fore.YELLOW
+                + f"Skipping unsupported {arch} throughput pipeline(s): {unsupported}"
+                + Style.RESET_ALL
+            )
+            comparisons = tuple(comparison for comparison in comparisons if comparison[1] not in unsupported)
+        comparisons_by_arch[arch] = comparisons
+        configs_by_arch[arch] = configs
+
+    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
+    if missing:
+        parser.error(f"missing model(s): {', '.join(missing)}")
+
+    if args.only is not None:
+        missing_reuse = []
+        for arch in args.arch:
+            for name in args.models:
+                for mode, pipeline, _ in comparisons_by_arch[arch]:
+                    comparison_dir = result_dir(out_dir, name, arch, mode, pipeline)
+                    required = (
+                        comparison_dir / "pimcomp/comparison_report.json"
+                        if args.only == "pimcomp"
+                        else comparison_dir / "pimcomp/output/SimulationInfo.gz"
+                    )
+                    if not required.exists():
+                        missing_reuse.append(str(required))
+                    elif args.only == "raptor":
+                        report = comparison_dir / "pimcomp/comparison_report.json"
+                        if not report.exists() or json.loads(report.read_text(encoding="utf-8")).get(
+                            "pimcomp_model_source"
+                        ) != "original_onnx":
+                            missing_reuse.append(f"{report} (not generated from the original ONNX model)")
+        if missing_reuse:
+            parser.error(
+                f"--only {args.only} needs existing comparison artifacts: "
+                + ", ".join(missing_reuse)
+            )
+
+    validate_pimcomp_source()
+    if out_dir is not None and not args.dry_run:
+        out_dir.mkdir(parents=True, exist_ok=True)
+
+    print(Style.BRIGHT + f"Found {len(args.models)} PIMCOMP model(s) to compare." + Style.RESET_ALL)
+    print(f"Architectures: {', '.join(args.arch)}")
+    print(f"Modes: {', '.join(args.mode)}")
+    print(f"Throughput pimsim time: {args.pimsim_time_ms} ms")
+    print(f"Max parallel jobs: {args.jobs}")
+    print(f"Results root: {out_dir or SUITE}")
+    print("=" * 72)
+
+    print_stage("Build Raptor", STAGE_COLORS["Build Runner"])
+    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
+
+    if args.only != "raptor":
+        print_stage("Build PIMCOMP", STAGE_COLORS["Build Runner"])
+        run(
+            ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
+            dry_run=args.dry_run,
+        )
+
+    print_stage("Prepare shared artifacts", STAGE_COLORS["Build Runner"])
+    for name in args.models:
+        run(
+            prepare_common_command(
+                MODELS[name],
+                model_dir(out_dir, name) / "common",
+                args.timeout_seconds,
+            ),
+            dry_run=args.dry_run,
+        )
+
+    failed = []
+    comparison_specs: list[ComparisonSpec] = []
+    shared_pimcomp_by_group: dict[tuple[str, str, str], Path] = {}
+    for arch in args.arch:
+        comparisons = comparisons_by_arch[arch]
+        configs = configs_by_arch[arch]
+        for index, name in enumerate(args.models, start=1):
+            for mode, pipeline, pimcomp_pipeline in comparisons:
+                model_result_dir = result_dir(out_dir, name, arch, mode, pipeline)
+                label = f"{arch}/{name}/{mode}/pipeline{pipeline}"
+                group = (arch, name, mode)
+                shared_pimcomp_dir = shared_pimcomp_by_group.get(group)
+                anchor = shared_pimcomp_dir is None
+                if anchor:
+                    shared_pimcomp_dir = model_result_dir / "pimcomp/output"
+                    shared_pimcomp_by_group[group] = shared_pimcomp_dir
+                print(
+                    "\n" + Fore.CYAN + f"[{arch} {index}/{len(args.models)}]" + Style.RESET_ALL
+                    + f" {Style.BRIGHT}Comparing {name} ({mode}, pipeline={pipeline}){Style.RESET_ALL}",
+                    flush=True,
+                )
+                comparison_specs.append(
+                    ComparisonSpec(
+                        label=label,
+                        model=MODELS[name],
+                        output_dir=model_result_dir,
+                        common_dir=model_dir(out_dir, name) / "common",
+                        config=configs[mode],
+                        mode=mode,
+                        pipeline=pipeline,
+                        pimcomp_pipeline=pimcomp_pipeline,
+                        shared_pimcomp_dir=shared_pimcomp_dir,
+                        anchor=anchor,
+                    )
+                )
+
+    if args.dry_run:
+        for spec in comparison_specs:
+            command = comparison_command_for(
+                spec,
+                args,
+                reuse_shared_pimcomp=not spec.anchor,
+            )
+            if run(command, dry_run=True, check=False):
+                failed.append(spec.label)
+    elif comparison_specs:
+        print(f"Running {len(comparison_specs)} comparison(s)")
+        anchor_jobs = [
+            (
+                spec.label,
+                comparison_command_for(spec, args, reuse_shared_pimcomp=False),
+                None,
+            )
+            for spec in comparison_specs
+            if spec.anchor
+        ]
+        dependent_specs = [spec for spec in comparison_specs if not spec.anchor]
+        print_directly = min(args.jobs, len(comparison_specs)) == 1
+        with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-pimcomp-")) as log_dir:
+            anchor_failed, log_offset = run_comparison_jobs(
+                anchor_jobs,
+                args.jobs,
+                Path(log_dir) if log_dir else None,
+                0,
+            )
+            failed.extend(anchor_failed)
+            dependent_jobs = [
+                (
+                    spec.label,
+                    comparison_command_for(
+                        spec,
+                        args,
+                        reuse_shared_pimcomp=pimcomp_artifact_ready(spec.shared_pimcomp_dir),
+                    ),
+                    None,
+                )
+                for spec in dependent_specs
+            ]
+            dependent_failed, _ = run_comparison_jobs(
+                dependent_jobs,
+                args.jobs,
+                Path(log_dir) if log_dir else None,
+                log_offset,
+            )
+            failed.extend(dependent_failed)
+
+    if args.dry_run:
+        return 1 if failed else 0
+
+    results_path = None
+    for arch in args.arch:
+        results_path = write_results_csv(
+            out_dir,
+            arch,
+            args.models,
+            comparisons_by_arch[arch],
+        )
+    assert results_path is not None
+    print_stage(results_path.name, STAGE_COLORS["Compare Outputs"])
+    print(results_path.read_text(encoding="utf-8"), end="")
+    for arch in args.arch:
+        for name in args.models:
+            for mode, pipeline, _ in comparisons_by_arch[arch]:
+                label = f"{arch}/{name}/{mode}/pipeline{pipeline}"
+                report_path = result_dir(out_dir, name, arch, mode, pipeline) / "pimcomp/comparison_report.json"
+                if not report_path.exists() or not comparison_passed(
+                    json.loads(report_path.read_text(encoding="utf-8"))
+                ):
+                    if label not in failed:
+                        failed.append(label)
+    print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
+    total_jobs = sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())
+    print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL)
+    print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
+    print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
+    if failed:
+        print(
+            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

validation/tools/pim/pimcomp/correctness/README.md

@@ -0,0 +1,49 @@
+# PIMCOMP batch correctness reproduction
+
+PIMCOMP's batch scheduler currently emits an incomplete standalone program for
+models containing post operations. The generated `VerificationInfo.json` uses a
+negative `source_address` to identify the preceding node, but the batch
+verifier resolves that address by copying the provider tensor directly from
+ONNX Runtime's `intermediate_result` table. It does not read the simulated
+global-memory buffer.
+
+The corresponding post-operation scheduling call is commented out in
+[`BatchPipelineSchedule.cpp`](../../../../../third_party/PIMCOMP-NN/backend/BatchPipelineSchedule.cpp#L1535).
+Consequently, Pool, ReLU, Concat, and Reshape provider tensors can be consumed
+by later `LD` instructions without any generated `ST` producer. The comparison
+exporter initializes those host-side intermediate buffers to zero, which makes
+the missing computation visible in Rust functional validation.
+
+For the checked-in GoogLeNet throughput/pipeline2 artifact, 39 provider tensors
+are referenced by batch loads. Nineteen have generated stores; twenty are
+never written. Preloading the provider tensors with the same ONNX Runtime
+intermediates used by PIMCOMP's verifier makes the exported program pass. This
+reproduces the verifier's input contract; it does not repair PIMCOMP's batch
+schedule.
+
+Run the default reproduction from the repository root:
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/correctness/run_prefill_experiment.py
+```
+
+The launcher accepts an alternate comparison directory, model, and work
+work directory, and shared reference-artifact directory:
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/correctness/run_prefill_experiment.py \
+  validation/networks/pimcomp_models/googlenet/arch-a/throughput/pipeline2 \
+  validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx \
+  /tmp/pimcomp-prefill-googlenet \
+  validation/networks/pimcomp_models/googlenet/common
+```
+
+It runs the exported artifact once with its original memory image and once
+with [`prefill_batch_memory.py`](prefill_batch_memory.py), then compares both
+outputs with the recorded native reference. The expected GoogLeNet result is a
+baseline maximum difference near `6.70705` and a prefilled maximum difference
+near `4.05e-6`.
+
+The issue is in PIMCOMP batch scheduling/validation semantics, not in the Rust
+simulator's vector-length interpretation. Vector lengths remain element counts
+as specified by the reference ISA.

validation/tools/pim/pimcomp/correctness/prefill_batch_memory.py

@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Populate the host-side intermediate buffers expected by PIMCOMP batch mode."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+from pathlib import Path
+
+import numpy as np
+import onnx
+import onnxruntime as ort
+
+
+def sanitize_output_name(name: str) -> str:
+    return "".join(char if char.isalnum() or char in "_.-" else "_" for char in name[:255])
+
+
+def flatten_reference(value: np.ndarray) -> np.ndarray:
+    if value.ndim == 4:
+        value = value.transpose(0, 2, 3, 1)
+    elif value.ndim == 2:
+        value = value.transpose()
+    else:
+        raise ValueError(f"PIMCOMP batch verification only flattens 2D/4D tensors, got {value.shape}")
+    return value.astype(np.float32, copy=False).reshape(-1)
+
+
+def provider_indices(verification_info: dict) -> list[int]:
+    indices = set()
+    for core in verification_info["instruction"]["core_list"]:
+        for instruction in core or []:
+            if (
+                instruction["operation"] == "LD"
+                and instruction.get("stage") in ("INPUT", "POST")
+                and instruction["node_index"] != 1
+            ):
+                source_address = instruction["source_address"]
+                if source_address >= 0:
+                    raise ValueError(f"Expected a negative provider address, got {source_address}")
+                indices.add(-source_address)
+    return sorted(indices)
+
+
+def output_base(
+    model: onnx.ModelProto,
+    config: dict,
+    node_list: list[dict],
+    max_output: int,
+) -> tuple[int, list[dict]]:
+    if len(model.graph.output) != len(config["outputs_addresses"]):
+        raise ValueError("Exported output addresses do not match the ONNX graph outputs")
+    nodes_by_name = {node["name"]: node for node in node_list}
+    bases = []
+    outputs = []
+    for index, graph_output in enumerate(model.graph.output):
+        node = nodes_by_name[graph_output.name]
+        address = config["outputs_addresses"][index]
+        base = int(address - node["new_node_index"] * max_output * 4)
+        bases.append(base)
+        outputs.append({"name": graph_output.name, "address": address})
+    if len(set(bases)) != 1:
+        raise ValueError(f"Exported output addresses use different memory bases: {bases}")
+    return bases[0], outputs
+
+
+def prefill_batch_memory(
+    model_path: Path,
+    comparison_dir: Path,
+    input_path: Path,
+    output_memory: Path,
+    metadata_path: Path,
+) -> None:
+    comparison_dir = comparison_dir.resolve()
+    model = onnx.load(model_path)
+    verification_info = json.loads(
+        (comparison_dir / "pimcomp/output/VerificationInfo.json").read_text(encoding="utf-8")
+    )
+    config = json.loads((comparison_dir / "pimcomp/exported/config.json").read_text(encoding="utf-8"))
+    node_list = verification_info["node_list"]
+    max_output = max(int(np.prod(node["output_dim"], dtype=int)) for node in node_list)
+    base, outputs = output_base(model, config, node_list, max_output)
+    providers = provider_indices(verification_info)
+
+    runtime_model = onnx.ModelProto()
+    runtime_model.CopyFrom(model)
+    existing_outputs = {output.name for output in runtime_model.graph.output}
+    for index in providers:
+        name = node_list[index]["name"]
+        if name not in existing_outputs:
+            runtime_model.graph.output.append(onnx.ValueInfoProto(name=name))
+
+    session = ort.InferenceSession(runtime_model.SerializeToString(), providers=["CPUExecutionProvider"])
+    session_inputs = session.get_inputs()
+    if len(session_inputs) != 1:
+        raise ValueError("PIMCOMP export currently requires exactly one runtime input tensor")
+    input_meta = session_inputs[0]
+    input_tensor = np.loadtxt(input_path, delimiter=",", dtype=np.float32).reshape(input_meta.shape)
+    provider_names = [node_list[index]["name"] for index in providers]
+    provider_values = session.run(provider_names, {input_meta.name: input_tensor})
+
+    output_memory = output_memory.resolve()
+    metadata_path = metadata_path.resolve()
+    if output_memory == (comparison_dir / "pimcomp/exported/memory.bin").resolve():
+        raise ValueError("Refusing to overwrite the original exported memory.bin")
+    output_memory.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copy2(comparison_dir / "pimcomp/exported/memory.bin", output_memory)
+
+    nodes_by_index = {index: node for index, node in enumerate(node_list)}
+    with output_memory.open("r+b") as memory:
+        memory_size = output_memory.stat().st_size
+        for index, value in zip(providers, provider_values):
+            flattened = flatten_reference(value)
+            node = nodes_by_index[index]
+            if node["operation"] in ("OP_CONV", "OP_FC") and node.get("with_act") == 1:
+                flattened = np.maximum(flattened, 0)
+            address = base + index * max_output * 4
+            end = address + flattened.nbytes
+            if end > memory_size:
+                raise ValueError(f"Provider {index} exceeds exported memory: {end} > {memory_size}")
+            memory.seek(address)
+            memory.write(flattened.tobytes())
+
+    output_metadata = {output["name"]: output for output in outputs}
+    for index, output_meta in enumerate(session.get_outputs()):
+        if output_meta.name not in output_metadata:
+            continue
+        output = output_metadata[output_meta.name]
+        output["dump"] = f"{output['address']},{int(np.prod(output_meta.shape, dtype=int)) * 4}"
+        output["reference"] = str(
+            comparison_dir / f"outputs/output{index}_{sanitize_output_name(output_meta.name)}.csv"
+        )
+    metadata_path.parent.mkdir(parents=True, exist_ok=True)
+    metadata_path.write_text(
+        json.dumps(
+            {
+                "memory": str(output_memory),
+                "output_base": base,
+                "provider_indices": providers,
+                "outputs": outputs,
+            },
+            indent=2,
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    print(f"Prefilled {len(providers)} provider tensor(s) into {output_memory}")
+    print(f"Metadata: {metadata_path}")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", type=Path, required=True)
+    parser.add_argument("--comparison-dir", type=Path, required=True)
+    parser.add_argument("--input", type=Path, required=True)
+    parser.add_argument("--output-memory", type=Path, required=True)
+    parser.add_argument("--metadata", type=Path, required=True)
+    args = parser.parse_args()
+    prefill_batch_memory(
+        args.model,
+        args.comparison_dir,
+        args.input,
+        args.output_memory,
+        args.metadata,
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

validation/tools/pim/pimcomp/correctness/run_prefill_experiment.py

@@ -0,0 +1,127 @@
+#!/usr/bin/env python3
+"""Reproduce the PIMCOMP batch prefill correctness experiment."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from prefill_batch_memory import prefill_batch_memory
+
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+REPO_ROOT = SCRIPT_DIR.parents[4]
+DEFAULT_COMPARISON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/arch-a/throughput/pipeline2"
+DEFAULT_MODEL = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx"
+DEFAULT_COMMON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/common"
+SIMULATOR_MANIFEST = REPO_ROOT / "backend-simulators/pim/pim-simulator/Cargo.toml"
+
+
+def run_simulator(
+    comparison_dir: Path,
+    memory: Path,
+    output: Path,
+    dump: str,
+) -> None:
+    subprocess.run(
+        [
+            "cargo",
+            "run",
+            "--manifest-path",
+            str(SIMULATOR_MANIFEST),
+            "--no-default-features",
+            "--release",
+            "--bin",
+            "pim-simulator",
+            "--",
+            "-f",
+            str(comparison_dir / "pimcomp/exported"),
+            "--memory",
+            str(memory),
+            "-o",
+            str(output),
+            "-d",
+            dump,
+        ],
+        cwd=REPO_ROOT,
+        check=True,
+    )
+
+
+def compare_outputs(
+    baseline_path: Path,
+    prefilled_path: Path,
+    reference_path: Path,
+    work_dir: Path,
+) -> None:
+    baseline = np.fromfile(baseline_path, dtype=np.float32)
+    prefilled = np.fromfile(prefilled_path, dtype=np.float32)
+    reference = np.loadtxt(reference_path, delimiter=",", dtype=np.float32).reshape(-1)
+    baseline_diff = np.max(np.abs(baseline - reference))
+    prefilled_diff = np.max(np.abs(prefilled - reference))
+    print(f"work directory: {work_dir}")
+    print(f"baseline max diff: {baseline_diff:.9g}")
+    print(f"prefilled max diff: {prefilled_diff:.9g}")
+    if np.allclose(baseline, reference, atol=1e-4, rtol=1e-3):
+        raise RuntimeError("baseline unexpectedly passes")
+    if not np.allclose(prefilled, reference, atol=1e-4, rtol=1e-3):
+        raise RuntimeError("prefilled run does not pass")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("comparison_dir", nargs="?", type=Path, default=DEFAULT_COMPARISON_DIR)
+    parser.add_argument("model", nargs="?", type=Path, default=DEFAULT_MODEL)
+    parser.add_argument("work_dir", nargs="?", type=Path)
+    parser.add_argument("common_dir", nargs="?", type=Path, default=DEFAULT_COMMON_DIR)
+    args = parser.parse_args()
+
+    comparison_dir = args.comparison_dir.resolve()
+    model = args.model.resolve()
+    common_dir = args.common_dir.resolve()
+    if args.work_dir is None:
+        work_dir = Path(tempfile.mkdtemp(prefix="pimcomp-prefill."))
+    else:
+        work_dir = args.work_dir.resolve()
+        work_dir.mkdir(parents=True, exist_ok=True)
+
+    input_path = common_dir / "inputs/in0.csv"
+    if not input_path.is_file():
+        input_path = comparison_dir / "inputs/in0.csv"
+
+    prefilled_memory = work_dir / "prefilled_memory.bin"
+    metadata_path = work_dir / "metadata.json"
+    baseline_output = work_dir / "baseline.out.bin"
+    prefilled_output = work_dir / "prefilled.out.bin"
+
+    prefill_batch_memory(
+        model,
+        comparison_dir,
+        input_path,
+        prefilled_memory,
+        metadata_path,
+    )
+    metadata = json.loads(metadata_path.read_text(encoding="utf-8"))
+    output = metadata["outputs"][0]
+    dump = output["dump"]
+    reference = Path(output["reference"])
+
+    run_simulator(
+        comparison_dir,
+        comparison_dir / "pimcomp/exported/memory.bin",
+        baseline_output,
+        dump,
+    )
+    run_simulator(comparison_dir, prefilled_memory, prefilled_output, dump)
+    compare_outputs(baseline_output, prefilled_output, reference, work_dir)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

validation/tools/pimcomp/run_pimcomp_paper_latency.py

@@ -1,355 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import csv
-import json
-import shlex
-import subprocess
-import sys
-from pathlib import Path
-
-from colorama import Fore, Style
-
-
-REPO = Path(__file__).resolve().parents[3]
-SUITE = REPO / "validation/networks/pimcomp_models"
-sys.path.insert(0, str(REPO / "validation"))
-
-from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics  # noqa: E402
-from raptor_validation.validate_one import STAGE_COLORS  # noqa: E402
-
-PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN"
-PIMCOMP_CONFIGS = REPO / "validation/pimsim_configs/pimcomp"
-COMPARE = Path(__file__).resolve().with_name("compare_raptor_pimcomp.py")
-ARCHES = tuple(sorted(path.name for path in PIMCOMP_CONFIGS.iterdir() if path.is_dir()))
-MODELS = {
-    "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx",
-    "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
-    "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
-    "googlenet": SUITE / "googlenet/googlenet-12-latency.onnx",
-}
-COMPARISONS = (
-    ("latency", 1, "element"),
-    ("throughput", 2, "batch"),
-    ("throughput", 4, "batch"),
-    ("throughput", 8, "batch"),
-)
-
-
-def result_dir(root: Path | None, name: str, mode: str, pipeline: int) -> Path:
-    base = root / name if root is not None else MODELS[name].parent
-    suffix = "latency" if mode == "latency" else f"throughput/pipeline{pipeline}"
-    return base / suffix
-
-
-def write_results_csv(root: Path | None, arch: str, models: list[str]) -> Path:
-    output = (root or SUITE) / "results.csv"
-    fields = (
-        "model",
-        "arch",
-        "mode",
-        "raptor_pipeline",
-        "pimcomp_pipeline",
-        "status",
-        "raptor_throughput_samples_s",
-        "pimcomp_throughput_samples_s",
-        "raptor_latency_ms",
-        "pimcomp_latency_ms",
-        "raptor_power_mw",
-        "pimcomp_power_mw",
-        "raptor_energy_pj",
-        "pimcomp_energy_pj",
-        "better_compiler",
-        "speedup",
-    )
-    rows = []
-    for name in models:
-        for mode, pipeline, pimcomp_pipeline in COMPARISONS:
-            report_path = result_dir(root, name, mode, pipeline) / "pimcomp/comparison_report.json"
-            if not report_path.exists():
-                continue
-            report = json.loads(report_path.read_text(encoding="utf-8"))
-            raptor = report.get("raptor_performance") or {}
-            pimcomp = report.get("pimcomp_performance") or {}
-            raptor_values = performance_values(raptor)
-            pimcomp_values = performance_values(pimcomp)
-            raptor_metric = raptor_values["throughput"] if mode == "throughput" else raptor_values["latency"]
-            pimcomp_metric = pimcomp_values["throughput"] if mode == "throughput" else pimcomp_values["latency"]
-            status = "PASS" if comparison_passed(report) else "FAIL"
-            if raptor_metric is None or pimcomp_metric is None:
-                better = ""
-                speedup = ""
-            else:
-                better = comparison_winner(mode, raptor_metric, pimcomp_metric)
-                speedup = f"{max(raptor_metric, pimcomp_metric) / min(raptor_metric, pimcomp_metric):.2f}"
-            rows.append({
-                "model": name,
-                "arch": arch,
-                "mode": mode,
-                "raptor_pipeline": pipeline,
-                "pimcomp_pipeline": pimcomp_pipeline,
-                "status": status,
-                "raptor_throughput_samples_s": format_value(raptor_values["throughput"]),
-                "pimcomp_throughput_samples_s": format_value(pimcomp_values["throughput"]),
-                "raptor_latency_ms": format_value(raptor_values["latency"]),
-                "pimcomp_latency_ms": format_value(pimcomp_values["latency"]),
-                "raptor_power_mw": format_value(raptor_values["power"]),
-                "pimcomp_power_mw": format_value(pimcomp_values["power"]),
-                "raptor_energy_pj": format_value(raptor_values["energy"]),
-                "pimcomp_energy_pj": format_value(pimcomp_values["energy"]),
-                "better_compiler": better,
-                "speedup": speedup,
-            })
-    with open(output, "w", newline="", encoding="utf-8") as f:
-        writer = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
-        writer.writeheader()
-        writer.writerows(rows)
-    return output
-
-
-def performance_values(performance: dict) -> dict[str, float | None]:
-    parsed = parse_pimsim_nn_metrics(performance.get("raw_output", ""))
-    return {
-        "throughput": performance.get("throughput") or parsed.get("throughput"),
-        "latency": (
-            performance.get("latency_ms")
-            or performance.get("average_latency_ms")
-            or parsed.get("latency_ms")
-            or parsed.get("average_latency_ms")
-        ),
-        "power": performance.get("average_power_mw") or parsed.get("average_power_mw"),
-        "energy": performance.get("average_energy_pj") or parsed.get("average_energy_pj"),
-    }
-
-
-def comparison_passed(report: dict) -> bool:
-    if report.get("failures"):
-        return False
-    for key in ("raptor_validation", "pimcomp_validation"):
-        result = report.get(key) or {}
-        if result.get("status") != "done" or not result.get("passed"):
-            return False
-    for key in ("raptor_performance", "pimcomp_performance"):
-        performance = report.get(key) or {}
-        if performance.get("error") or performance.get("skipped"):
-            return False
-    return True
-
-
-def comparison_winner(mode: str, raptor: float, pimcomp: float) -> str:
-    if raptor == pimcomp:
-        return "tie"
-    if mode == "throughput":
-        return "raptor" if raptor > pimcomp else "pimcomp"
-    return "raptor" if raptor < pimcomp else "pimcomp"
-
-
-def format_value(value: float | None) -> str:
-    return "" if value is None else f"{value:.6f}"
-
-
-def print_stage(title: str, color: str) -> None:
-    print("\n" + Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL, flush=True)
-
-
-def run(command: list[str], *, dry_run: bool, check: bool = True) -> int:
-    print(f"  cwd: {REPO}", flush=True)
-    print(f"  $ {shlex.join(command)}", flush=True)
-    if dry_run:
-        return 0
-    return subprocess.run(command, cwd=REPO, check=check).returncode
-
-
-def validate_pimcomp_source() -> None:
-    header = PIMCOMP_SOURCE / "backend/GeneticAlgorithm.h"
-    source = header.read_text(encoding="utf-8")
-    for setting in ("int population_num = 200;", "int max_iteration = 1000;"):
-        if setting not in source:
-            raise RuntimeError(f"PIMCOMP paper setting is missing: {setting}")
-
-
-def comparison_command(
-    model: Path,
-    result_dir: Path,
-    config: Path,
-    mode: str,
-    pipeline: int,
-    pimcomp_pipeline: str,
-    pimsim_time_ms: int,
-    timeout: float,
-) -> list[str]:
-    time_args = ["--pimsim-time-ms", str(pimsim_time_ms)] if mode == "throughput" else []
-    return [
-        sys.executable,
-        str(COMPARE),
-        "--model",
-        str(model),
-        "--out-dir",
-        str(result_dir),
-        "--pimcomp-dir",
-        str(PIMCOMP_SOURCE),
-        "--pimcomp-config",
-        str(config),
-        "--pimsim-mode",
-        mode,
-        *time_args,
-        "--pimcomp-pipeline",
-        pimcomp_pipeline,
-        "--pimcomp-replication",
-        "GA",
-        f"--raptor-extra-arg=--pipeline={pipeline}",
-        "--timeout-seconds",
-        str(timeout),
-        "--fail-on-error",
-    ]
-
-
-def config_path(arch: str, mode: str) -> Path:
-    path = PIMCOMP_CONFIGS / arch / f"{mode}_config.json"
-    if not path.exists():
-        raise ValueError(f"{arch} has no {mode} config: {path}")
-    return path
-
-
-def core_count(config: Path) -> int:
-    with open(config, encoding="utf-8") as f:
-        return int(json.load(f)["chip_config"]["core_cnt"])
-
-
-def completed_report(path: Path, mode: str, pipeline: int, config: Path, pimsim_time_ms: int) -> bool:
-    if not path.exists():
-        return False
-    report = json.loads(path.read_text(encoding="utf-8"))
-    return (
-        report.get("pimsim_mode") == mode
-        and report.get("pimcomp_pipeline") == ("element" if mode == "latency" else "batch")
-        and report.get("pimsim_time_ms") == pimsim_time_ms
-        and report.get("pimcomp_config") == str(config.resolve())
-        and f"--pipeline={pipeline}" in report.get("raptor_extra_args", [])
-    )
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(
-        description="Compare supported PIMCOMP models with Raptor latency and throughput schedules."
-    )
-    parser.add_argument(
-        "--out-dir",
-        type=Path,
-        help="Result root (default: artifacts beside each model under validation/).",
-    )
-    parser.add_argument("--models", nargs="+", choices=MODELS, default=list(MODELS))
-    parser.add_argument(
-        "--arch", choices=ARCHES, default="arch-a", help="PIM architecture (default: arch-a)."
-    )
-    parser.add_argument(
-        "--pimsim-time-ms",
-        type=int,
-        default=100,
-        help="throughput pimsim-nn horizon in ms (default: 100).",
-    )
-    parser.add_argument("--timeout-seconds", type=float, default=3600.0)
-    parser.add_argument(
-        "--resume",
-        action="store_true",
-        help="Skip models with a completed JSON report.",
-    )
-    parser.add_argument("--dry-run", action="store_true", help="Print commands without modifying files.")
-    args = parser.parse_args()
-
-    if args.pimsim_time_ms <= 0:
-        parser.error("--pimsim-time-ms must be positive")
-    configs = {mode: config_path(args.arch, mode) for mode, _, _ in COMPARISONS}
-    unsupported = [pipeline for mode, pipeline, _ in COMPARISONS if core_count(configs[mode]) % pipeline]
-    if unsupported:
-        parser.error(
-            f"{args.arch} has {core_count(configs['throughput'])} cores; "
-            f"throughput pipelines must divide that count (invalid: {unsupported})"
-        )
-
-    out_dir = args.out_dir.resolve() if args.out_dir is not None else None
-
-    missing = [str(MODELS[name]) for name in args.models if not MODELS[name].exists()]
-    if missing:
-        parser.error(f"missing model(s): {', '.join(missing)}")
-
-    validate_pimcomp_source()
-    if out_dir is not None and not args.dry_run:
-        out_dir.mkdir(parents=True, exist_ok=True)
-
-    print(Style.BRIGHT + f"Found {len(args.models)} PIMCOMP model(s) to compare." + Style.RESET_ALL)
-    print(f"Architecture: {args.arch}")
-    print(f"Throughput pimsim time: {args.pimsim_time_ms} ms")
-    print(f"Results root: {out_dir or SUITE}")
-    print("=" * 72)
-
-    print_stage("Build Raptor", STAGE_COLORS["Build Runner"])
-    run(["cmake", "--build", str(REPO / "build_release")], dry_run=args.dry_run)
-
-    print_stage("Build PIMCOMP", STAGE_COLORS["Build Runner"])
-    run(
-        ["cmake", "--build", str(PIMCOMP_SOURCE / "build"), "--target", "PIMCOMP-NN"],
-        dry_run=args.dry_run,
-    )
-
-    failed = []
-    for index, name in enumerate(args.models, start=1):
-        for mode, pipeline, pimcomp_pipeline in COMPARISONS:
-            model_result_dir = result_dir(out_dir, name, mode, pipeline)
-            print(
-                "\n" + Fore.CYAN + f"[{index}/{len(args.models)}]" + Style.RESET_ALL
-                + f" {Style.BRIGHT}Comparing {name} ({mode}, pipeline={pipeline}){Style.RESET_ALL}",
-                flush=True,
-            )
-            if args.resume and completed_report(
-                model_result_dir / "pimcomp/comparison_report.json",
-                mode,
-                pipeline,
-                configs[mode],
-                args.pimsim_time_ms,
-            ):
-                print(
-                    Fore.YELLOW + "  Completed report exists; skipping" + Style.RESET_ALL,
-                    flush=True,
-                )
-                continue
-            returncode = run(
-                comparison_command(
-                    MODELS[name],
-                    model_result_dir,
-                    configs[mode],
-                    mode,
-                    pipeline,
-                    pimcomp_pipeline,
-                    args.pimsim_time_ms,
-                    args.timeout_seconds,
-                ),
-                dry_run=args.dry_run,
-                check=False,
-            )
-            if returncode:
-                failed.append(f"{name}/{mode}/pipeline{pipeline}")
-
-    if args.dry_run:
-        return 1 if failed else 0
-
-    results_path = write_results_csv(out_dir, args.arch, args.models)
-    print_stage("Results", STAGE_COLORS["Compare Outputs"])
-    print(results_path.read_text(encoding="utf-8"), end="")
-    print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
-    total_jobs = len(args.models) * len(COMPARISONS)
-    print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL)
-    print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
-    print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
-    if failed:
-        print(
-            Fore.RED + f"Failed comparisons: {', '.join(failed)}" + Style.RESET_ALL,
-            file=sys.stderr,
-        )
-        return 1
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())