add non-functional validation

refactors

.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md

@@ -29,8 +29,13 @@ Compiler representations may share analysis, planning, or code-generation
 work only when the emitted execution retains the same independent work,
 schedule semantics, scheduling granularity, and available parallelism.
 
-An optimization is acceptable only when its static runtime proxies are equal or
-better.
+Static runtime proxies are useful for screening changes, but they are not the
+latency authority. When comparable validation results are available for the
+same model, inputs, hardware configuration, and non-functional simulation
+configuration, the `Latency` reported by `validation/validate.py` is
+authoritative. An optimization is acceptable only when its static runtime
+proxies are equal or better, unless such a matched validation comparison
+demonstrates equal or better latency.
 
 ## Asymptotic cost
 
@@ -64,6 +69,7 @@ Do not:
 
 Every performance change must report before and after:
 
+- matched validation `Latency` when the changed execution can be simulated;
 - compiler wall time;
 - compiler peak RSS;
 - IR operation and value counts at the changed stage;
@@ -82,4 +88,5 @@ Every performance change must report before and after:
 ## Stop rule
 
 If compile time or memory improves but a runtime proxy worsens, stop and reject
-the change.
+the change unless a matched validation comparison demonstrates equal or better
+`Latency`. Proxy improvements do not override a validation latency regression.

CMakeLists.txt

@@ -145,4 +145,24 @@ raptor_apply_patch(
   "Skip output emission for PIM accelerator"
 )
 
+# Count the PIM status messages instead of the unused standard backend phases
+raptor_apply_patch(
+  "${ONNX_MLIR_DIR}/src/Compiler/CompilerUtils.cpp"
+  "void showCompilePhase(std::string msg) {"
+  [=[void showCompilePhase(std::string msg) {
+  if (llvm::is_contained(maccel, accel::Accelerator::Kind::PIM)) {
+    if (pimOnlyCodegen)
+      TOTAL_COMPILE_PHASE = 2;
+    else if (pimEmissionTarget == EmitSpatial)
+      TOTAL_COMPILE_PHASE = 3;
+    else if (pimEmissionTarget == EmitPim)
+      TOTAL_COMPILE_PHASE = 4;
+    else if (pimEmissionTarget == EmitPimBufferized)
+      TOTAL_COMPILE_PHASE = 5;
+    else
+      TOTAL_COMPILE_PHASE = 9;
+  }]=]
+  "Use PIM compile phase count"
+)
+
 add_subdirectory(onnx-mlir)

README.md

@@ -29,7 +29,8 @@ lowering, scheduling, memory layout, and code-generation optimizations.
 - `backend-simulators/pim/pim-simulator` is the in-tree Rust functional
   simulator used by validation. It reads Raptor's `pim/` artifact directory and
   compares simulator output against native ONNX-MLIR execution.
-- `backend-simulators/pim/pimsim-nn` is the performance simulator submodule.
+- `backend-simulators/pim/pimsim-nn` is the non-functional simulator submodule
+  used internally by validation for latency, power, and energy.
   The helper scripts in `pimcomp_utils/` are for comparison with PIMCOMP-NN and
   contain local paths; treat them as local utilities, not portable workflows.
 
@@ -155,7 +156,7 @@ This writes PIM artifacts under `/tmp/raptor/pim/`.
 
 Functional validation lives in `validation/`. It compiles ONNX models, builds a
 native ONNX-MLIR reference runner, generates random inputs, runs Raptor, runs
-the Rust PIM simulator, and compares outputs.
+the Rust functional simulator, and compares outputs.
 
 Python dependencies used by the validation scripts are `numpy`, `onnx`, and
 `colorama`. The simulator requires the Rust toolchain.
@@ -167,9 +168,6 @@ Per-operation validation from the repository root:
   --raptor-path build_release/Release/bin/onnx-mlir \
   --onnx-include-dir onnx-mlir/include \
   --operations-dir validation/operations \
-  --crossbar-count 64 \
-  --crossbar-size 128 \
-  --core-count 144 \
   --verbose \
   --raptor-extra-arg=--pim-detect-communication-deadlock \
   --raptor-extra-arg=--pim-export-spatial-dataflow=none
@@ -183,17 +181,24 @@ containing `.onnx` files:
   --raptor-path build_release/Release/bin/onnx-mlir \
   --onnx-include-dir onnx-mlir/include \
   --operations-dir validation/networks/yolo11n/depth_04 \
-  --crossbar-count 64 \
-  --crossbar-size 128 \
-  --core-count 144 \
   --verbose \
   --raptor-extra-arg=--pim-detect-communication-deadlock \
   --raptor-extra-arg=--pim-export-spatial-dataflow=none
 ```
 
 Useful validation options:
-- `--simulator-dir <path>` - override the auto-detected
+- `--simulator-dir <path>` - override the auto-detected functional
   `backend-simulators/pim/pim-simulator` path.
+- `--pimcomp-config <arch-a|arch-b|arch-c>` - select the checked-in
+  non-functional simulation profile. Arch-A is the default; the hardware
+  argument defaults match it (`168` cores, `96` crossbars/core, `128x128`
+  crossbars).
+  Selecting another profile also requires its matching hardware arguments.
+- `--non-functional-simulator-build-dir <path>` - override the auto-detected
+  non-functional simulator build directory.
+- `--skip-non-functional-simulation` - skip non-functional simulation. If
+  explicit core/crossbar arguments do not match the selected profile, validation
+  reports the mismatch and skips only non-functional simulation.
 - `--threshold <float>` - maximum allowed per-element output difference.
 - `--seed <int>` - RNG seed for generated inputs.
 - `--command-timeout-seconds <float>` - timeout for compiler, runner, and
@@ -208,7 +213,14 @@ Each validation run writes artifacts in the model workspace, for example under
 - `raptor/` - compiler artifacts, including `*.onnx.mlir`, dialect dumps under
   `dialects/`, reports under `reports/`, and final PIM artifacts under `pim/`.
 - `runner/` - generated reference runner source, build tree, and shared library.
-- `simulation/out.bin` - raw simulator output used for comparison.
+- `simulation/out.bin` - raw functional simulation output used for comparison.
+
+By default, validation also runs non-functional simulation. Per-model `Latency`
+is shown in the result table alongside `Power`, with measured, failed, and
+skipped counts plus total latency in the summary. Use these validation results
+for performance comparisons; the simulation backend does not need to be invoked
+separately. The selected pre-generated configs and meshes live under
+`validation/pimsim_configs/pimcomp/`.
 
 The compiler currently dumps dialect snapshots such as `spatial0.mlir`,
 `spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
@@ -216,8 +228,8 @@ The compiler currently dumps dialect snapshots such as `spatial0.mlir`,
 `pim2_folded.mlir`, and `pim3_memory_planned.mlir` when an output directory is
 available.
 
-To rerun the simulator manually with tracing after validation has produced a
-`raptor/pim/` directory:
+To rerun the functional simulator manually with tracing after validation has
+produced a `raptor/pim/` directory:
 
 ```bash
 cd backend-simulators/pim/pim-simulator
@@ -360,7 +372,7 @@ cargo test
 - `validation/` - functional validation scripts, ONNX operation tests, network
   slices, and pimsim config generation.
 - `backend-simulators/pim/pim-simulator/` - in-tree Rust functional simulator.
-- `backend-simulators/pim/pimsim-nn/` - performance simulator submodule.
+- `backend-simulators/pim/pimsim-nn/` - non-functional simulator submodule.
 - `pimcomp_utils/` - local comparison helpers for PIMCOMP-NN.
 - `.github/actions/` and `.github/workflows/validate_operations.yml` - CI setup
   for MLIR/Protobuf caching, building Raptor, and validation.

requirements.txt

@@ -1,4 +1,5 @@
 colorama>=0.4.6,<1
 numpy>=1.26.4,<3
 onnx>=1.17,<2
+onnxsim>=0.6.5,<1
 -e ./tools/raptor_graph_explorer[dev]

validation/networks/pimcomp_models/README.md

@@ -44,13 +44,18 @@ a35bad96441efbee28699cb61d1656cca7f7281f14040cf01699c3d0cfd8b202  googlenet/goog
 
 ## Paper hardware profiles
 
-The files in `configs/` encode Table V's explicit resource parameters.
+The files in
+[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
+encode Table V's explicit resource parameters.
+Each profile subdirectory contains pre-generated latency and throughput
+`pimsim-nn` configs plus its matching mesh; comparison and validation select
+these checked-in artifacts without generating configs at runtime.
 
 | Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout |
 | --- | ---: | ---: | --- | ---: | --- |
-| `arch-a.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
-| `arch-b.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
-| `arch-c.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` |
+| `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
+| `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
+| `arch-c/latency_config.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` |
 
 `adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
 The paper does not give a two-dimensional core topology for Arch-A/B, so the
@@ -67,7 +72,7 @@ reproducible, but absolute paper power and energy numbers are not.
 From the Raptor repository root:
 
 ```bash
-.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
+.venv/bin/python -m pip install -r requirements.txt
 cmake --build ./build_release
 cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
 
@@ -85,11 +90,11 @@ select one paper profile, and restore it when the shell exits:
 ```bash
 RAPTOR_ROOT=$PWD
 PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
-PAPER_MODELS="$RAPTOR_ROOT/validation/networks/pimcomp_models"
+PIMCOMP_CONFIGS="$RAPTOR_ROOT/validation/pimsim_configs/pimcomp"
 CONFIG_BACKUP=$(mktemp)
 cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
 trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
-cp "$PAPER_MODELS/configs/arch-a.json" "$PIMCOMP/config.json"
+cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json"
 ```
 
 The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile
@@ -124,7 +129,8 @@ cd third_party/PIMCOMP-NN/build
 ./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES
 ```
 
-Repeat after selecting `arch-b.json` and `arch-c.json`. All four models were
+Repeat after selecting `arch-b/latency_config.json` and
+`arch-c/latency_config.json`. All four models were
 compiled successfully in both modes with all three configs. The released
 random placement code occasionally segfaults; an unchanged retry succeeded in
 the observed cases.
@@ -140,8 +146,9 @@ insufficient.
 ## Compare Raptor and PIMCOMP
 
 The comparison driver uses one random input and one native ONNX-MLIR reference,
-compiles both instruction streams, validates both through Raptor's Rust
-simulator, and writes Markdown and JSON reports.
+compiles both instruction streams, runs both through `pimsim-nn`, validates
+Raptor through the Rust simulator, and writes Markdown and JSON reports.
+PIMCOMP Rust validation also runs when its optional exporter is available.
 
 To reproduce the complete Arch-A latency experiment, use the serial experiment
 runner. It creates an isolated PIMCOMP build with population 200 and 1000 GA
@@ -156,8 +163,7 @@ comparison driver for one model at a time:
 Reports are written under `<out-dir>/<model>/comparison_report.{md,json}`.
 Use `--models vgg8` to run one model, `--resume` after an interruption, or
 `--dry-run` to inspect every command. The runner continues after a failed model
-so all reports are produced, then returns a nonzero status if any comparison
-failed.
+so all reports are produced.
 
 Arch-A low-latency example:
 
@@ -167,28 +173,34 @@ PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
 CONFIG_BACKUP=$(mktemp)
 cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
 trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
-cp "$RAPTOR_ROOT/validation/networks/pimcomp_models/configs/arch-a.json" "$PIMCOMP/config.json"
+cp "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" "$PIMCOMP/config.json"
 
 "$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/compare_raptor_pimcomp.py" \
-  --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx" \
-  --out-dir /tmp/compare-vgg8-arch-a-ll \
+  --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
+  --out-dir /tmp/compare-resnet34-arch-a-ll \
   --core-count 168 \
   --crossbar-count 96 \
   --crossbar-size 128 \
   --mesh-rows 12 \
   --mesh-cols 14 \
   --pimsim-mode latency \
-  --pimcomp-pipeline element \
-  --fail-on-error
+  --pimcomp-pipeline element
 ```
 
 For Arch-A high throughput, use `--pimsim-mode throughput
 --pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a
-`6x23` mesh, and `configs/arch-b.json`.
+`6x23` mesh, and
+`validation/pimsim_configs/pimcomp/arch-b/latency_config.json`.
 
 If only semantic and instruction comparison is required, add
-`--skip-pimsim-nn`. This exact VGG-8 Arch-A LL smoke test passed both semantic
-validations with maximum output differences below `5e-10`.
+`--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both
+semantic validations with maximum output differences below `5e-10`.
+
+The current PIMCOMP-NN submodule does not include the optional
+`verification/export_to_pim_simulator.py` helper. The driver therefore records
+PIMCOMP Rust semantic validation as skipped, while PIMCOMP compilation,
+instruction reporting, and `pimsim-nn` latency still run. Use
+`--fail-on-error` only when that semantic export helper is available.
 
 Current Raptor status:
 
@@ -213,6 +225,8 @@ the canonical `resnetv2/depth_68` file may not exist remotely:
 REMOTE_REPO=/home/gmagnani/Project/Raptor
 rsync -azL validation/networks/pimcomp_models/ \
   "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
+rsync -az validation/pimsim_configs/pimcomp/ \
+  "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
 rsync -az validation/tools/compare_raptor_pimcomp.py \
   "monolith:$REMOTE_REPO/validation/tools/compare_raptor_pimcomp.py"
 rsync -az validation/tools/run_pimcomp_paper_latency.py \

validation/operations/README.md

@@ -32,9 +32,6 @@ Run the complete suite with deadlock detection:
   --raptor-path build_release/Release/bin/onnx-mlir \
   --onnx-include-dir onnx-mlir/include \
   --operations-dir validation/operations \
-  --crossbar-count 64 \
-  --crossbar-size 128 \
-  --core-count 144 \
   --raptor-extra-arg=--pim-detect-communication-deadlock \
   --raptor-extra-arg=--pim-export-spatial-dataflow=none
 ```

validation/pimsim_configs/pimcomp/arch-a/latency_config.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 96,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          128,
+          128
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        12,
+        14
+      ],
+      "net_config_file_path": "network_mesh_168.json"
+    },
+    "core_cnt": 168,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 1,
+    "sim_time": 1000,
+    "report_verbose_level": 0
+  }
+}

validation/pimsim_configs/pimcomp/arch-a/throughput_config.json

@@ -8,7 +8,10 @@
         "pipeline_mode": true,
         "dac_resolution": 1,
         "dac_count": 128,
-        "xbar_size": [128, 128],
+        "xbar_size": [
+          128,
+          128
+        ],
         "cell_precision": 2,
         "xbar_latency": 7,
         "xbar_read_power": 22.24,
@@ -29,7 +32,8 @@
         "write_latency_cycle": 30,
         "read_latency_cycle": 20
       },
-      "global_memory_switch_id": -10
+      "global_memory_switch_id": -10,
+      "rob_size": 1
     },
     "global_memory_config": {
       "data_width": 32,
@@ -40,7 +44,10 @@
     "network_config": {
       "bus_topology": "mesh",
       "bus_width": 32,
-      "layout": [12, 14],
+      "layout": [
+        12,
+        14
+      ],
       "net_config_file_path": "network_mesh_168.json"
     },
     "core_cnt": 168,
@@ -48,7 +55,7 @@
   },
   "sim_config": {
     "sim_mode": 0,
-    "sim_time": 200,
+    "sim_time": 1000,
     "report_verbose_level": 0
   }
 }

validation/pimsim_configs/pimcomp/arch-b/latency_config.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 128,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          128,
+          128
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        6,
+        23
+      ],
+      "net_config_file_path": "network_mesh_138.json"
+    },
+    "core_cnt": 138,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 1,
+    "sim_time": 1000,
+    "report_verbose_level": 0
+  }
+}

validation/pimsim_configs/pimcomp/arch-b/throughput_config.json

@@ -8,7 +8,10 @@
         "pipeline_mode": true,
         "dac_resolution": 1,
         "dac_count": 128,
-        "xbar_size": [128, 128],
+        "xbar_size": [
+          128,
+          128
+        ],
         "cell_precision": 2,
         "xbar_latency": 7,
         "xbar_read_power": 22.24,
@@ -29,7 +32,8 @@
         "write_latency_cycle": 30,
         "read_latency_cycle": 20
       },
-      "global_memory_switch_id": -10
+      "global_memory_switch_id": -10,
+      "rob_size": 1
     },
     "global_memory_config": {
       "data_width": 32,
@@ -40,7 +44,10 @@
     "network_config": {
       "bus_topology": "mesh",
       "bus_width": 32,
-      "layout": [6, 23],
+      "layout": [
+        6,
+        23
+      ],
       "net_config_file_path": "network_mesh_138.json"
     },
     "core_cnt": 138,
@@ -48,7 +55,7 @@
   },
   "sim_config": {
     "sim_mode": 0,
-    "sim_time": 200,
+    "sim_time": 1000,
     "report_verbose_level": 0
   }
 }

validation/pimsim_configs/pimcomp/arch-c/latency_config.json

@@ -0,0 +1,61 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 8,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [
+          512,
+          1024
+        ],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10,
+      "rob_size": 1
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [
+        8,
+        8
+      ],
+      "net_config_file_path": "network_mesh_64.json"
+    },
+    "core_cnt": 64,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 1,
+    "sim_time": 1000,
+    "report_verbose_level": 0
+  }
+}

validation/pimsim_configs/pimcomp/arch-c/throughput_config.json

@@ -8,7 +8,10 @@
         "pipeline_mode": true,
         "dac_resolution": 1,
         "dac_count": 128,
-        "xbar_size": [512, 1024],
+        "xbar_size": [
+          512,
+          1024
+        ],
         "cell_precision": 2,
         "xbar_latency": 7,
         "xbar_read_power": 22.24,
@@ -29,7 +32,8 @@
         "write_latency_cycle": 30,
         "read_latency_cycle": 20
       },
-      "global_memory_switch_id": -10
+      "global_memory_switch_id": -10,
+      "rob_size": 1
     },
     "global_memory_config": {
       "data_width": 32,
@@ -40,7 +44,10 @@
     "network_config": {
       "bus_topology": "mesh",
       "bus_width": 32,
-      "layout": [8, 8],
+      "layout": [
+        8,
+        8
+      ],
       "net_config_file_path": "network_mesh_64.json"
     },
     "core_cnt": 64,
@@ -48,7 +55,7 @@
   },
   "sim_config": {
     "sim_mode": 0,
-    "sim_time": 200,
+    "sim_time": 1000,
     "report_verbose_level": 0
   }
 }

validation/raptor_validation/__init__.py

@@ -0,0 +1 @@
+"""Reusable Raptor validation support."""

validation/raptor_validation/gen_network_runner.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 import argparse, os, pathlib, textwrap
-from onnx_utils import onnx_io
+from .onnx_utils import onnx_io
 from onnx import TensorProto
 
 # ONNX dtype -> (ctype, printf, ONNX_TYPE_*)
@@ -207,7 +207,7 @@ int main(int argc, char **argv) {{
 }}
 """
 
-def gen_network_runner(network_onnx, network_so, onnx_include_dir, entry="run_main_graph", out=None, verbose=True):
+def gen_network_runner(network_onnx, network_so, onnx_include_dir, entry, out, verbose):
     ins, outs = onnx_io(network_onnx)
     out_c = out or "runner.c"
     so_abs = os.path.abspath(network_so)
@@ -240,4 +240,4 @@ if __name__=="__main__":
     ap.add_argument("--out", default=None)
     a=ap.parse_args()
 
-    gen_network_runner(a.network_onnx, a.network_so, a.onnx_include_dir, a.entry, a.out)
+    gen_network_runner(a.network_onnx, a.network_so, a.onnx_include_dir, a.entry, a.out, True)

validation/raptor_validation/onnx_utils.py

@@ -30,9 +30,12 @@ def onnx_io(path):
         return s
 
     ins, outs = [], []
-    for i, v in enumerate(g.input):
+    initializer_names = {initializer.name for initializer in g.initializer}
+    for v in g.input:
+        if v.name in initializer_names:
+            continue
         t = v.type.tensor_type
-        ins.append((i, v.name, t.elem_type, shp(t)))
+        ins.append((len(ins), v.name, t.elem_type, shp(t)))
     for i, v in enumerate(g.output):
         t = v.type.tensor_type
         outs.append((i, v.name, t.elem_type, shp(t)))

validation/raptor_validation/raptor.py

@@ -3,7 +3,7 @@ import shlex
 import subprocess
 from pathlib import Path
 from colorama import Fore, Style
-from subprocess_utils import run_command_with_reporter
+from .subprocess_utils import run_command_with_reporter
 
 PIM_PASS_LABELS = (
     ("ONNXToSpatialPass", "ONNX to Spatial"),
@@ -41,9 +41,8 @@ def _format_command(cmd):
 
 
 def compile_with_raptor(network_path, raptor_onnx_path: Path, output_base: Path,
-                        crossbar_size, crossbar_count, core_count=None,
-                        raptor_extra_args=None, cwd=None, verbose=False,
-                        reporter=None, timeout_sec=None):
+                        crossbar_size, crossbar_count, core_count,
+                        raptor_extra_args, cwd, verbose, reporter, timeout_sec):
     # Define the arguments, with the possibility to set crossbar size and count
     args = [
         network_path,

validation/raptor_validation/validate_one.py

@@ -1,15 +1,16 @@
 import json
 import os
-import numpy as np
+import re
 import shutil
 import sys
+import numpy as np
 from dataclasses import dataclass, field
 from pathlib import Path
 from colorama import Style, Fore
-from onnx_utils import gen_random_inputs, save_inputs_to_files, onnx_io, write_inputs_to_memory_bin, _ONNX_TO_NP
-from raptor import compile_with_raptor
-from gen_network_runner import gen_network_runner
-from subprocess_utils import run_command_with_reporter
+from .gen_network_runner import gen_network_runner
+from .onnx_utils import gen_random_inputs, save_inputs_to_files, onnx_io, write_inputs_to_memory_bin, _ONNX_TO_NP
+from .raptor import compile_with_raptor
+from .subprocess_utils import run_command_with_reporter
 
 STAGE_TITLES = (
     "Compile ONNX",
@@ -17,8 +18,9 @@ STAGE_TITLES = (
     "Generate Inputs",
     "Run Reference",
     "Compile PIM",
-    "Run Simulator",
+    "Run Functional Simulation",
     "Compare Outputs",
+    "Run Non-functional Simulation",
 )
 STAGE_COUNT = len(STAGE_TITLES)
 GENERATED_DIR_NAMES = ("inputs", "outputs", "raptor", "runner", "simulation")
@@ -37,11 +39,19 @@ MODE_STAGE_TITLES = {
     MODE_RUN_ONLY: (
         "Generate Inputs",
         "Run Reference",
-        "Run Simulator",
+        "Run Functional Simulation",
         "Compare Outputs",
+        "Run Non-functional Simulation",
     ),
 }
 
+PIMSIM_DONE = "DONE"
+PIMSIM_FAILED = "ERROR"
+PIMSIM_SKIPPED = "SKIP"
+PIMSIM_NOT_RUN = "-"
+PIMSIM_LATENCY_RE = re.compile(r"\blatency:\s+([0-9.eE+-]+)\s+ms")
+PIMSIM_POWER_RE = re.compile(r"\baverage power:\s+([0-9.eE+-]+)\s+mW")
+
 
 def sanitize_output_name(name):
     return "".join(ch if ch.isalnum() or ch in "_.-" else "_" for ch in name[:255])
@@ -51,6 +61,9 @@ def sanitize_output_name(name):
 class ValidationResult:
     passed: bool
     pim_pass_timings: dict[str, float] = field(default_factory=dict)
+    pimsim_latency_ms: float | None = None
+    pimsim_power_mw: float | None = None
+    pimsim_status: str = PIMSIM_SKIPPED
 
 
 class ProgressReporter:
@@ -180,8 +193,70 @@ class ProgressReporter:
             self._clear()
 
 
-def run_command(cmd, cwd=None, reporter=None, timeout_sec=None):
-    run_command_with_reporter(cmd, cwd=cwd, reporter=reporter, timeout_sec=timeout_sec)
+def run_command(cmd, cwd=None, reporter=None, timeout_sec=None, capture_output=False):
+    return run_command_with_reporter(
+        cmd,
+        cwd=cwd,
+        reporter=reporter,
+        timeout_sec=timeout_sec,
+        capture_output=capture_output,
+    )
+
+
+def load_pimcomp_hardware(config_path):
+    with open(config_path, encoding="utf-8") as f:
+        config = json.load(f)
+    matrix = config["chip_config"]["core_config"]["matrix_config"]
+    rows, cols = config["chip_config"]["network_config"]["layout"]
+    xbar_rows, xbar_cols = matrix["xbar_size"]
+    return {
+        "core_count": config["chip_config"]["core_cnt"],
+        "crossbar_count": matrix["xbar_array_count"],
+        "crossbar_rows": xbar_rows,
+        "crossbar_cols": xbar_cols,
+        "mesh_rows": rows,
+        "mesh_cols": cols,
+    }
+
+
+def pimcomp_compatibility_errors(config_path, *, core_count, crossbar_count, crossbar_size):
+    hardware = load_pimcomp_hardware(config_path)
+    errors = []
+    if hardware["mesh_rows"] * hardware["mesh_cols"] != hardware["core_count"]:
+        errors.append(
+            f"config layout {hardware['mesh_rows']}x{hardware['mesh_cols']} does not match "
+            f"{hardware['core_count']} cores"
+        )
+    if core_count != hardware["core_count"]:
+        errors.append(f"--core-count={core_count}, config requires {hardware['core_count']}")
+    if crossbar_count != hardware["crossbar_count"]:
+        errors.append(
+            f"--crossbar-count={crossbar_count}, config requires {hardware['crossbar_count']}"
+        )
+    if (
+        hardware["crossbar_rows"] != hardware["crossbar_cols"]
+        or crossbar_size != hardware["crossbar_rows"]
+    ):
+        errors.append(
+            f"--crossbar-size={crossbar_size}, config requires "
+            f"{hardware['crossbar_rows']}x{hardware['crossbar_cols']}"
+        )
+    return errors
+
+
+def run_pimsim_nn(pimsim_nn_build_dir, pim_dir, config_path, reporter=None, timeout_sec=None):
+    output = run_command(
+        [pimsim_nn_build_dir / "ChipTest", pim_dir, config_path, "false"],
+        cwd=pimsim_nn_build_dir,
+        reporter=reporter,
+        timeout_sec=timeout_sec,
+        capture_output=True,
+    )
+    latency_match = PIMSIM_LATENCY_RE.search(output)
+    power_match = PIMSIM_POWER_RE.search(output)
+    if not latency_match or not power_match:
+        raise RuntimeError("pimsim-nn output did not contain latency and average power")
+    return float(latency_match.group(1)), float(power_match.group(1))
 
 
 def clean_workspace_artifacts(workspace_dir, model_stem):
@@ -214,6 +289,7 @@ def print_stage(reporter, model_index, model_total, model_name, title):
         STAGE_TITLES[4]: Fore.CYAN,
         STAGE_TITLES[5]: Fore.MAGENTA,
         STAGE_TITLES[6]: Fore.YELLOW,
+        STAGE_TITLES[7]: Fore.BLUE,
     }
     color = stage_colors.get(title, Fore.WHITE)
     reporter.log(Style.BRIGHT + color + f"[{title}]" + Style.RESET_ALL)
@@ -278,7 +354,7 @@ def parse_pim_simulator_outputs(output_bin_path, outputs_descriptor):
     return arrays
 
 
-def validate_outputs(sim_arrays, runner_out_dir, outputs_descriptor, threshold=1e-3, rtol=1e-5, verbose=False):
+def validate_outputs(sim_arrays, runner_out_dir, outputs_descriptor, threshold, rtol, verbose):
     all_passed = True
     rows = []
     for sim_array, (oi, name, _, shape) in zip(sim_arrays, outputs_descriptor):
@@ -312,15 +388,23 @@ def validate_outputs(sim_arrays, runner_out_dir, outputs_descriptor, threshold=1
 
 
 def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
-                     simulator_dir, crossbar_size=128, crossbar_count=64, core_count=144,
-                     raptor_extra_args=None,
-                     threshold=1e-3, rtol=1e-5,
-                     seed=0, reporter=None, model_index=1, model_total=1, verbose=False,
-                     command_timeout_seconds=60.0, mode=MODE_FULL):
+                     simulator_dir, crossbar_size, crossbar_count, core_count,
+                     raptor_extra_args,
+                     pimsim_nn_build_dir, pimsim_config_path,
+                     threshold, rtol,
+                     seed, reporter, model_index, model_total, verbose,
+                     command_timeout_seconds, mode):
     network_onnx_path = Path(network_onnx_path).resolve()
     raptor_path = Path(raptor_path).resolve()
     onnx_include_dir = Path(onnx_include_dir).resolve()
     simulator_dir = Path(simulator_dir).resolve()
+    pimsim_enabled = pimsim_nn_build_dir is not None and pimsim_config_path is not None
+    if pimsim_enabled:
+        pimsim_nn_build_dir = Path(pimsim_nn_build_dir).resolve()
+        pimsim_config_path = Path(pimsim_config_path).resolve()
+    compile_extra_args = list(raptor_extra_args or [])
+    if pimsim_enabled and "--pim-emit-json" not in compile_extra_args:
+        compile_extra_args.append("--pim-emit-json")
     owns_reporter = reporter is None
     reporter = reporter or ProgressReporter(model_total, stages_per_model=len(MODE_STAGE_TITLES[mode]), verbose=verbose)
 
@@ -356,8 +440,14 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
             reporter.advance()
 
             print_stage(reporter, model_index, model_total, network_onnx_path.name, "Build Runner")
-            gen_network_runner(network_onnx_path, network_so_path, onnx_include_dir, out=runner_dir / "runner.c",
-                               verbose=False)
+            gen_network_runner(
+                network_onnx_path,
+                network_so_path,
+                onnx_include_dir,
+                entry="run_main_graph",
+                out=runner_dir / "runner.c",
+                verbose=False,
+            )
             runner_path = build_onnx_runner(runner_dir, runner_build_dir, reporter=reporter,
                                             timeout_sec=command_timeout_seconds)
             print_info(reporter, f"Runner built at {runner_path}")
@@ -368,7 +458,7 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
                 pim_pass_timings = compile_with_raptor(
                     network_onnx_path, raptor_path, pim_output_base, crossbar_size, crossbar_count,
                     core_count=core_count,
-                    raptor_extra_args=raptor_extra_args,
+                    raptor_extra_args=compile_extra_args,
                     cwd=raptor_dir, verbose=verbose, reporter=reporter, timeout_sec=command_timeout_seconds)
                 print_info(reporter, f"PIM artifacts saved to {raptor_dir / 'pim'}")
                 reporter.advance()
@@ -408,12 +498,14 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
             pim_pass_timings = compile_with_raptor(
                 network_onnx_path, raptor_path, pim_output_base, crossbar_size, crossbar_count,
                 core_count=core_count,
-                raptor_extra_args=raptor_extra_args,
+                raptor_extra_args=compile_extra_args,
                 cwd=raptor_dir, verbose=verbose, reporter=reporter, timeout_sec=command_timeout_seconds)
             print_info(reporter, f"PIM artifacts saved to {raptor_dir / 'pim'}")
             reporter.advance()
 
-        print_stage(reporter, model_index, model_total, network_onnx_path.name, "Run Simulator")
+        print_stage(
+            reporter, model_index, model_total, network_onnx_path.name,
+            "Run Functional Simulation")
         pim_dir = raptor_dir / "pim"
         write_inputs_to_memory_bin(pim_dir / "memory.bin", pim_dir / "config.json", inputs_list)
         simulation_dir = workspace_dir / "simulation"
@@ -422,7 +514,7 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
         output_bin_path = simulation_dir / "out.bin"
         run_pim_simulator(simulator_dir, pim_dir, output_bin_path, dump_ranges, reporter=reporter,
                           timeout_sec=command_timeout_seconds)
-        print_info(reporter, f"Simulator output saved to {output_bin_path}")
+        print_info(reporter, f"Functional simulation output saved to {output_bin_path}")
         reporter.advance()
 
         print_stage(reporter, model_index, model_total, network_onnx_path.name, "Compare Outputs")
@@ -431,10 +523,53 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
         passed = validate_outputs(sim_arrays, out_dir, outputs_descriptor, threshold, rtol=rtol, verbose=verbose)
         reporter.resume()
         reporter.advance()
+
+        print_stage(
+            reporter, model_index, model_total, network_onnx_path.name,
+            "Run Non-functional Simulation")
+        pimsim_latency_ms = None
+        pimsim_power_mw = None
+        pimsim_status = PIMSIM_SKIPPED
+        if pimsim_enabled:
+            try:
+                pimsim_latency_ms, pimsim_power_mw = run_pimsim_nn(
+                    pimsim_nn_build_dir,
+                    pim_dir,
+                    pimsim_config_path,
+                    reporter=reporter,
+                    timeout_sec=command_timeout_seconds,
+                )
+                pimsim_status = PIMSIM_DONE
+                print_info(
+                    reporter,
+                    f"Latency: {pimsim_latency_ms:.6f} ms, "
+                    f"Power: {pimsim_power_mw:.6f} mW")
+            except Exception as exc:
+                pimsim_status = PIMSIM_FAILED
+                reporter.suspend()
+                print(
+                    Fore.RED
+                    + f"pimsim-nn non-functional simulation failed: {type(exc).__name__}: {exc}"
+                    + Style.RESET_ALL,
+                    file=sys.stderr,
+                    flush=True,
+                )
+                reporter.resume()
+        else:
+            print_info(reporter, "pimsim-nn non-functional simulation skipped")
+        reporter.advance()
+
+        passed = passed and pimsim_status != PIMSIM_FAILED
         reporter.record_result(passed)
         status = Fore.GREEN + "PASS" + Style.RESET_ALL if passed else Fore.RED + "FAIL" + Style.RESET_ALL
         reporter.log(Style.BRIGHT + f"Result: {status}" + Style.RESET_ALL)
-        return ValidationResult(passed=passed, pim_pass_timings=pim_pass_timings)
+        return ValidationResult(
+            passed=passed,
+            pim_pass_timings=pim_pass_timings,
+            pimsim_latency_ms=pimsim_latency_ms,
+            pimsim_power_mw=pimsim_power_mw,
+            pimsim_status=pimsim_status,
+        )
     except Exception:
         failed_with_exception = True
         reporter.record_result(False)

validation/tools/classification_local_image_validation.py

@@ -15,8 +15,8 @@ REPO_ROOT = VALIDATION_DIR.parent
 if str(VALIDATION_DIR) not in sys.path:
     sys.path.insert(0, str(VALIDATION_DIR))
 
-from onnx_utils import _ONNX_TO_NP, onnx_io, write_inputs_to_memory_bin
-from validate_one import (
+from raptor_validation.onnx_utils import _ONNX_TO_NP, onnx_io, write_inputs_to_memory_bin
+from raptor_validation.validate_one import (
     MODE_COMPILE_ONLY,
     build_dump_ranges,
     parse_pim_simulator_outputs,
@@ -59,6 +59,15 @@ def ensure_local_artifacts(args, model_path: Path):
         crossbar_size=args.crossbar_size,
         crossbar_count=args.crossbar_count,
         core_count=args.core_count,
+        raptor_extra_args=[],
+        pimsim_nn_build_dir=None,
+        pimsim_config_path=None,
+        threshold=1e-3,
+        rtol=1e-5,
+        seed=0,
+        reporter=None,
+        model_index=1,
+        model_total=1,
         command_timeout_seconds=args.command_timeout_seconds,
         mode=MODE_COMPILE_ONLY,
         verbose=args.verbose,

validation/tools/compare_raptor_pimcomp.py

@@ -25,12 +25,19 @@ import onnx
 
 REPO = Path(__file__).resolve().parents[2]
 VALIDATION_DIR = REPO / "validation"
+PIMSIM_CONFIG_DIR = VALIDATION_DIR / "pimsim_configs/pimcomp"
 sys.path.insert(0, str(VALIDATION_DIR))
 
-from gen_network_runner import gen_network_runner  # noqa: E402
-from onnx_utils import _ONNX_TO_NP, gen_random_inputs, onnx_io, save_inputs_to_files, write_inputs_to_memory_bin  # noqa: E402
-from validate_one import build_dump_ranges, parse_pim_simulator_outputs  # noqa: E402
-from raptor import compile_with_raptor  # noqa: E402
+from raptor_validation.gen_network_runner import gen_network_runner  # noqa: E402
+from raptor_validation.onnx_utils import (  # noqa: E402
+    _ONNX_TO_NP,
+    gen_random_inputs,
+    onnx_io,
+    save_inputs_to_files,
+    write_inputs_to_memory_bin,
+)
+from raptor_validation.raptor import compile_with_raptor  # noqa: E402
+from raptor_validation.validate_one import build_dump_ranges, parse_pim_simulator_outputs  # noqa: E402
 
 
 @dataclass
@@ -62,15 +69,6 @@ def load_pimcomp_exporter():
     return module
 
 
-def load_mesh_builder():
-    path = REPO / "validation/pimsim-configs/generate_mesh_config.py"
-    spec = importlib.util.spec_from_file_location("mesh_builder", path)
-    module = importlib.util.module_from_spec(spec)
-    assert spec is not None and spec.loader is not None
-    spec.loader.exec_module(module)
-    return module
-
-
 def shell_join(cmd: list[str]) -> str:
     return shlex.join(str(arg) for arg in cmd)
 
@@ -278,41 +276,26 @@ def load_effective_hardware(args: argparse.Namespace) -> dict[str, int]:
     return hardware
 
 
-def write_pimsim_config(args: argparse.Namespace, out_dir: Path, hardware: dict[str, int]) -> Path:
-    mesh_builder = load_mesh_builder()
-    with open(args.pimcomp_dir / "config.json", "r", encoding="utf-8") as f:
-        config = json.load(f)
-    config["chip_config"]["core_config"].setdefault("rob_size", 1)
-    config["chip_config"]["core_config"]["matrix_config"]["xbar_array_count"] = hardware["crossbar_count"]
-    config["chip_config"]["core_config"]["matrix_config"]["xbar_size"] = [
-        hardware["crossbar_size"],
-        hardware["crossbar_size"],
-    ]
-    config["chip_config"]["network_config"]["layout"] = [
-        hardware["mesh_rows"],
-        hardware["mesh_cols"],
-    ]
-    config["chip_config"]["network_config"]["net_config_file_path"] = f"network_mesh_{hardware['core_count']}.json"
-    config["chip_config"]["core_cnt"] = hardware["core_count"]
-    config["sim_config"]["sim_mode"] = 1 if args.pimsim_mode == "latency" else 0
-    config["sim_config"]["sim_time"] = args.pimsim_time_ms
-    out_dir.mkdir(parents=True, exist_ok=True)
-    config_path = out_dir / f"{args.pimsim_mode}_config.json"
-    network_path = out_dir / f"network_mesh_{hardware['core_count']}.json"
-    with open(config_path, "w", encoding="utf-8") as f:
-        json.dump(config, f, indent=2)
-        f.write("\n")
-    with open(network_path, "w", encoding="utf-8") as f:
-        json.dump(
-            mesh_builder.build_network(
-                hardware["core_count"],
-                (hardware["mesh_rows"], hardware["mesh_cols"]),
-            ),
-            f,
-            separators=(",", ":"),
-        )
-        f.write("\n")
-    return config_path
+def select_pimsim_config(args: argparse.Namespace, hardware: dict[str, int]) -> Path:
+    for path in sorted(PIMSIM_CONFIG_DIR.glob(f"*/{args.pimsim_mode}_config.json")):
+        with open(path, encoding="utf-8") as f:
+            config = json.load(f)
+        chip = config["chip_config"]
+        matrix = chip["core_config"]["matrix_config"]
+        network = chip["network_config"]
+        if (
+            chip["core_cnt"] == hardware["core_count"]
+            and matrix["xbar_array_count"] == hardware["crossbar_count"]
+            and matrix["xbar_size"] == [hardware["crossbar_size"]] * 2
+            and network["layout"] == [hardware["mesh_rows"], hardware["mesh_cols"]]
+            and config["sim_config"]["sim_mode"] == (1 if args.pimsim_mode == "latency" else 0)
+            and config["sim_config"]["sim_time"] == args.pimsim_time_ms
+        ):
+            return path
+    raise ValueError(
+        f"No pre-generated {args.pimsim_mode} pimsim-nn config matches "
+        f"{hardware} with sim_time={args.pimsim_time_ms}"
+    )
 
 
 def compile_reference(
@@ -348,7 +331,14 @@ def compile_reference(
     network_so = runner_base.with_suffix(".so")
 
     print_step("Generate Runner Source")
-    gen_network_runner(model_path, network_so, args.onnx_include_dir, out=runner_dir / "runner.c", verbose=False)
+    gen_network_runner(
+        model_path,
+        network_so,
+        args.onnx_include_dir,
+        entry="run_main_graph",
+        out=runner_dir / "runner.c",
+        verbose=False,
+    )
 
     run_logged(
         "Configure Runner",
@@ -427,6 +417,7 @@ def compile_raptor_target(
             raptor_extra_args=raptor_extra_args,
             cwd=out_dir,
             verbose=args.verbose_raptor_compile,
+            reporter=None,
             timeout_sec=args.timeout_seconds,
         )
     except Exception as exc:
@@ -1453,18 +1444,17 @@ def main():
     else:
         pimcomp_validation = skipped_validation("Output descriptors are not available")
 
-    if hardware["core_count"] > 0:
+    if not args.skip_pimsim_nn and hardware["core_count"] > 0:
         written_config = try_stage(
             failures,
-            "Write pimsim-nn config",
-            write_pimsim_config,
+            "Select pimsim-nn config",
+            select_pimsim_config,
             args,
-            out_dir / "pimsim_config",
             hardware,
         )
         if written_config is not None:
             pimsim_config = written_config
-    else:
+    elif not args.skip_pimsim_nn:
         record_failure(
             failures,
             "Skip pimsim-nn config",

validation/tools/run_pimcomp_paper_latency.py

@@ -48,7 +48,10 @@ def prepare_pimcomp(work_dir: Path) -> None:
     if replacements != 1:
         raise RuntimeError("Could not set PIMCOMP GA max_iteration")
     header.write_text(source, encoding="utf-8")
-    shutil.copy2(SUITE / "configs/arch-a.json", work_dir / "config.json")
+    shutil.copy2(
+        REPO / "validation/pimsim_configs/pimcomp/arch-a/latency_config.json",
+        work_dir / "config.json",
+    )
 
 
 def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout: float) -> list[str]:
@@ -79,7 +82,6 @@ def comparison_command(model: Path, result_dir: Path, pimcomp_dir: Path, timeout
         "GA",
         "--timeout-seconds",
         str(timeout),
-        "--fail-on-error",
     ]
 
 

validation/tools/yolo_local_image_validation.py

@@ -21,8 +21,14 @@ if sys.version_info < (3, 10):
         "Run it with a newer interpreter, for example your project venv Python."
     )
 
-from onnx_utils import _ONNX_TO_NP, onnx_io, write_inputs_to_memory_bin
-from validate_one import MODE_COMPILE_ONLY, build_dump_ranges, run_pim_simulator, sanitize_output_name, validate_network
+from raptor_validation.onnx_utils import _ONNX_TO_NP, onnx_io, write_inputs_to_memory_bin
+from raptor_validation.validate_one import (
+    MODE_COMPILE_ONLY,
+    build_dump_ranges,
+    run_pim_simulator,
+    sanitize_output_name,
+    validate_network,
+)
 from yolo_real_image_validation import (
     IMAGE_CASES,
     decode_yolo_output,
@@ -80,6 +86,15 @@ def ensure_local_artifacts(args, network_onnx_path: Path):
         crossbar_size=args.crossbar_size,
         crossbar_count=args.crossbar_count,
         core_count=args.core_count,
+        raptor_extra_args=[],
+        pimsim_nn_build_dir=None,
+        pimsim_config_path=None,
+        threshold=1e-3,
+        rtol=1e-5,
+        seed=0,
+        reporter=None,
+        model_index=1,
+        model_total=1,
         command_timeout_seconds=args.command_timeout_seconds,
         mode=MODE_COMPILE_ONLY,
         verbose=args.verbose,

validation/validate.py

@@ -6,16 +6,26 @@ import subprocess
 import sys
 from pathlib import Path
 from colorama import Style, Fore
-from validate_one import (
+from raptor_validation.validate_one import (
     MODE_COMPILE_ONLY,
     MODE_FULL,
     MODE_RUN_ONLY,
     MODE_STAGE_TITLES,
+    PIMSIM_DONE,
+    PIMSIM_FAILED,
+    PIMSIM_NOT_RUN,
+    PIMSIM_SKIPPED,
     ProgressReporter,
+    ValidationResult,
     clean_workspace_artifacts,
+    load_pimcomp_hardware,
+    pimcomp_compatibility_errors,
     validate_network,
 )
-from raptor import PIM_PASS_LABELS
+from raptor_validation.raptor import PIM_PASS_LABELS
+
+DEFAULT_PIMCOMP_CONFIG = "arch-a"
+PIMCOMP_CONFIG_CHOICES = ("arch-a", "arch-b", "arch-c")
 
 
 def format_return_status(returncode):
@@ -60,41 +70,70 @@ def print_average_pim_pass_timings(pass_timing_sums, pass_timing_counts, total_t
     print(f"  {'Total'.ljust(28)} {total_timing_sum / timed_benchmark_count:.4f}s")
 
 
+def format_pimsim_metric(result, value, unit):
+    if result.pimsim_status == PIMSIM_DONE:
+        return f"{value:.6f} {unit}"
+    return result.pimsim_status
+
+
 def main():
+    script_dir = Path(__file__).parent.resolve()
+    pimcomp_configs_dir = script_dir / "pimsim_configs" / "pimcomp"
+    default_pimsim_config = pimcomp_configs_dir / DEFAULT_PIMCOMP_CONFIG / "latency_config.json"
+    default_hardware = load_pimcomp_hardware(default_pimsim_config)
     ap = argparse.ArgumentParser(description="Validate all ONNX operations under the operations/ directory.")
     ap.add_argument("--raptor-path", help="Path to the Raptor compiler binary.")
     ap.add_argument("--onnx-include-dir", help="Path to OnnxMlirRuntime include directory.")
     ap.add_argument("--operations-dir", default=None, help="Root of the operations tree (default: operations).")
     ap.add_argument("--simulator-dir", default=None,
-                    help="Path to pim-simulator crate root (default: auto-detected relative to script).")
+                    help="Path to the functional pim-simulator crate root "
+                         "(default: auto-detected relative to script).")
+    ap.add_argument("--non-functional-simulator-build-dir", metavar="PATH", default=None,
+                    help="Path to the non-functional simulator build directory "
+                         "(default: auto-detected relative to script).")
+    ap.add_argument("--pimcomp-config", choices=PIMCOMP_CONFIG_CHOICES, default=DEFAULT_PIMCOMP_CONFIG,
+                    help="Hardware/timing profile for non-functional simulation "
+                         "(default: arch-a).")
+    ap.add_argument("--skip-non-functional-simulation", action="store_true",
+                    help="Skip non-functional simulation.")
     ap.add_argument("--threshold", type=float, default=1e-3,
                     help="Absolute tolerance for per-element output comparison.")
     ap.add_argument("--relative-threshold", type=float, default=1e-5,
                     help="Relative tolerance for per-element output comparison.")
     ap.add_argument("--seed", type=int, default=0, help="RNG seed for generated validation inputs.")
-    ap.add_argument("--crossbar-size", type=int, default=128)
-    ap.add_argument("--crossbar-count", type=int, default=64)
-    ap.add_argument("--core-count", type=int, default=144)
+    ap.add_argument("--crossbar-size", type=int, default=default_hardware["crossbar_rows"],
+                    help=f"Crossbar size (default: {default_hardware['crossbar_rows']}).")
+    ap.add_argument("--crossbar-count", type=int, default=default_hardware["crossbar_count"],
+                    help=f"Crossbars per core (default: {default_hardware['crossbar_count']}).")
+    ap.add_argument("--core-count", type=int, default=default_hardware["core_count"],
+                    help=f"Core count (default: {default_hardware['core_count']}).")
     ap.add_argument("--raptor-extra-arg", action="append", default=[],
                     help="Additional argument to pass through to the Raptor compiler. Repeat as needed.")
     ap.add_argument("--command-timeout-seconds", type=float, default=1000000.0,
-                    help="Per-subprocess timeout in seconds for compiler, runner, and simulator commands.")
+                    help="Per-subprocess timeout in seconds for compiler, runner, and simulation commands.")
     ap.add_argument("--clean", action="store_true",
                     help="Remove generated validation artifacts under each model workspace and exit.")
     mode_group = ap.add_mutually_exclusive_group()
     mode_group.add_argument("--compile-only", action="store_true",
-                            help="Compile reference and PIM artifacts only; do not run reference, simulator, or compare.")
+                            help="Compile reference and PIM artifacts only; do not run reference execution, "
+                                 "simulations, or comparison.")
     mode_group.add_argument("--run-only", action="store_true",
-                            help="Reuse existing compiled artifacts and only run inputs/reference/simulator/compare.")
+                            help="Reuse existing compiled artifacts and only run inputs, reference execution, "
+                                 "simulations, and comparison.")
     ap.add_argument("--verbose", action="store_true",
                     help="Print per-stage progress and subprocess logs for passing validations too.")
     a = ap.parse_args()
 
-    script_dir = Path(__file__).parent.resolve()
     operations_dir = Path(a.operations_dir).resolve() if a.operations_dir else script_dir / "operations"
     simulator_dir = Path(a.simulator_dir).resolve() if a.simulator_dir else (
             script_dir / ".." / "backend-simulators" / "pim" / "pim-simulator"
     )
+    pimsim_nn_build_dir = (
+        Path(a.non_functional_simulator_build_dir).resolve()
+        if a.non_functional_simulator_build_dir else
+        script_dir / ".." / "backend-simulators" / "pim" / "pimsim-nn" / "build"
+    )
+    pimsim_config_path = pimcomp_configs_dir / a.pimcomp_config / "latency_config.json"
 
     if not operations_dir.is_dir():
         print(Fore.RED + f"Operations directory not found: {operations_dir}" + Style.RESET_ALL)
@@ -130,7 +169,26 @@ def main():
     elif a.run_only:
         mode = MODE_RUN_ONLY
 
-    results = {}  # relative_path -> passed
+    selected_pimsim_config = None
+    if not a.skip_non_functional_simulation:
+        compatibility_errors = pimcomp_compatibility_errors(
+            pimsim_config_path,
+            core_count=a.core_count,
+            crossbar_count=a.crossbar_count,
+            crossbar_size=a.crossbar_size,
+        )
+        if compatibility_errors:
+            print(
+                Fore.RED
+                + "Non-functional simulation disabled: "
+                + "; ".join(compatibility_errors)
+                + Style.RESET_ALL,
+                file=sys.stderr,
+            )
+        else:
+            selected_pimsim_config = pimsim_config_path
+
+    results = {}  # relative_path -> ValidationResult
     pass_timing_sums = {label: 0.0 for _, label in PIM_PASS_LABELS}
     pass_timing_counts = {label: 0 for _, label in PIM_PASS_LABELS}
     total_timing_sum = 0.0
@@ -143,6 +201,8 @@ def main():
                 onnx_path, a.raptor_path, a.onnx_include_dir, simulator_dir,
                 crossbar_size=a.crossbar_size, crossbar_count=a.crossbar_count, core_count=a.core_count,
                 raptor_extra_args=a.raptor_extra_arg,
+                pimsim_nn_build_dir=pimsim_nn_build_dir,
+                pimsim_config_path=selected_pimsim_config,
                 command_timeout_seconds=a.command_timeout_seconds,
                 threshold=a.threshold,
                 rtol=a.relative_threshold,
@@ -153,7 +213,7 @@ def main():
                 verbose=a.verbose,
                 mode=mode,
             )
-            results[str(rel)] = result.passed
+            results[str(rel)] = result
             if result.pim_pass_timings:
                 benchmark_total = 0.0
                 for label, duration in result.pim_pass_timings.items():
@@ -163,32 +223,73 @@ def main():
                 total_timing_sum += benchmark_total
                 timed_benchmark_count += 1
         except subprocess.CalledProcessError as exc:
-            results[str(rel)] = False
+            results[str(rel)] = ValidationResult(False, pimsim_status=PIMSIM_NOT_RUN)
             print_validation_error(reporter, rel, exc)
         except Exception as exc:
-            results[str(rel)] = False
+            results[str(rel)] = ValidationResult(False, pimsim_status=PIMSIM_NOT_RUN)
             print_validation_error(reporter, rel, exc)
 
     reporter.finish()
 
     # Summary
-    n_passed = sum(1 for passed in results.values() if passed)
+    n_passed = sum(1 for result in results.values() if result.passed)
     n_total = len(results)
     status_width = len("Result")
     path_width = max(len("Operation"), *(len(rel) for rel in results))
-    separator = f"+-{'-' * path_width}-+-{'-' * status_width}-+"
+    formatted_metrics = {
+        rel: (
+            format_pimsim_metric(result, result.pimsim_latency_ms, "ms"),
+            format_pimsim_metric(result, result.pimsim_power_mw, "mW"),
+        )
+        for rel, result in results.items()
+    }
+    latency_width = max(len("Latency"), *(len(metrics[0]) for metrics in formatted_metrics.values()))
+    power_width = max(len("Power"), *(len(metrics[1]) for metrics in formatted_metrics.values()))
+    separator = (
+        f"+-{'-' * path_width}-+-{'-' * status_width}-+-{'-' * latency_width}"
+        f"-+-{'-' * power_width}-+")
     print(separator)
-    print(f"| {'Operation'.ljust(path_width)} | {'Result'.ljust(status_width)} |")
+    print(
+        f"| {'Operation'.ljust(path_width)} | {'Result'.ljust(status_width)} | "
+        f"{'Latency'.ljust(latency_width)} | {'Power'.ljust(power_width)} |"
+    )
     print(separator)
-    for rel, passed in results.items():
-        plain_status = "PASS" if passed else "FAIL"
-        status = Fore.GREEN + plain_status.ljust(status_width) + Style.RESET_ALL if passed else \
+    for rel, result in results.items():
+        plain_status = "PASS" if result.passed else "FAIL"
+        status = Fore.GREEN + plain_status.ljust(status_width) + Style.RESET_ALL if result.passed else \
             Fore.RED + plain_status.ljust(status_width) + Style.RESET_ALL
-        print(f"| {rel.ljust(path_width)} | {status} |")
+        latency, power = formatted_metrics[rel]
+        print(
+            f"| {rel.ljust(path_width)} | {status} | {latency.ljust(latency_width)} | "
+            f"{power.ljust(power_width)} |")
     print(separator)
     print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
     print(Style.BRIGHT + f"Passed: {n_passed}" + Style.RESET_ALL)
     print(Style.BRIGHT + f"Failed: {n_total - n_passed}" + Style.RESET_ALL)
+    measured_latencies = [
+        result.pimsim_latency_ms
+        for result in results.values()
+        if result.pimsim_status == PIMSIM_DONE
+    ]
+    pimsim_failed = sum(
+        result.pimsim_status == PIMSIM_FAILED for result in results.values()
+    )
+    pimsim_skipped = sum(
+        result.pimsim_status in (PIMSIM_SKIPPED, PIMSIM_NOT_RUN)
+        for result in results.values()
+    )
+    print(
+        Style.BRIGHT
+        + f"pimsim-nn: {len(measured_latencies)} measured, "
+        f"{pimsim_failed} failed, {pimsim_skipped} skipped"
+        + Style.RESET_ALL
+    )
+    if measured_latencies:
+        print(
+            Style.BRIGHT
+            + f"Total latency: {sum(measured_latencies):.6f} ms"
+            + Style.RESET_ALL
+        )
     if a.verbose:
         print_average_pim_pass_timings(
             pass_timing_sums,