Python script for compare

validation/networks/pimcomp_models/README.md

@@ -0,0 +1,246 @@
+# PIMCOMP paper models
+
+This directory contains the four networks evaluated in
+[PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
+VGG-8, ResNet-18, ResNet-34, and GoogLeNet.
+
+See [RESULTS.md](RESULTS.md) for the current latency-only result status.
+
+## Models and provenance
+
+| Directory | Model | Input | Provenance |
+| --- | --- | --- | --- |
+| `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete ONNX Model Zoo model already present at `../resnetv2/depth_68/resnetv2_depth_68.onnx`. |
+| `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as PIMCOMP's frontend does. |
+| `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). |
+| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Deterministic compiler workload with six convolution and two fully connected layers. |
+
+`googlenet/googlenet-12-no-softmax.onnx` is a derived latency model that
+exposes the original model's final FC logits (`loss3/classifier_1`) as its
+output. This matches PIMCOMP's instruction stream, which records but does not
+schedule the terminal `OP_SOFTMAX`. Keep `googlenet-12.onnx` for full-model
+functional validation.
+
+The PIMCOMP authors did not publish the ONNX checkpoints used by the paper.
+Running PIMCOMP's frontend on the three Model Zoo files above produces JSON
+graphs exactly equal to PIMCOMP-NN's bundled `resnet18.json`, `resnet34.json`,
+and `googlenet.json`.
+
+There is no VGG-8 artifact in the ONNX Model Zoo or any PIMCOMP-NN revision.
+The included VGG-8 therefore has deterministic random weights and is suitable
+for compiler and simulator comparison, not paper-accuracy reproduction. The
+paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the
+published PIMCOMP graphs and ResNet Model Zoo artifacts use ImageNet shapes.
+
+Current SHA-256 checksums:
+
+```text
+788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f  resnet18/resnet18-v1-7.onnx
+c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096  resnet34/resnet34-v1-7.onnx
+c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e  googlenet/googlenet-12.onnx
+a35bad96441efbee28699cb61d1656cca7f7281f14040cf01699c3d0cfd8b202  googlenet/googlenet-12-no-softmax.onnx
+396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef  vgg8/vgg8-mnist-reconstructed.onnx
+```
+
+## Paper hardware profiles
+
+The files in `configs/` encode Table V's explicit resource parameters.
+
+| Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout |
+| --- | ---: | ---: | --- | ---: | --- |
+| `arch-a.json` | 168 | 96 | `128x128` | 2-bit | `12x14` |
+| `arch-b.json` | 138 | 128 | `128x128` | 2-bit | `6x23` |
+| `arch-c.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` |
+
+`adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
+The paper does not give a two-dimensional core topology for Arch-A/B, so the
+factorizations above preserve core count but cannot reproduce unpublished NoC
+placement details. Released PIMCOMP-NN has no chip-count field; Arch-C is
+therefore flattened to 64 cores and does not model chip boundaries.
+
+The remaining latency and power values come from PIMCOMP-NN's released default
+configuration. Consequently, instruction/resource comparisons are
+reproducible, but absolute paper power and energy numbers are not.
+
+## Build and validate the ONNX files
+
+From the Raptor repository root:
+
+```bash
+.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
+cmake --build ./build_release
+cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
+
+.venv/bin/python -c \
+  'from pathlib import Path; import onnx; [onnx.checker.check_model(onnx.load(p)) for p in Path("validation/networks/pimcomp_models").glob("*/*.onnx")]'
+```
+
+Do not build either project with `ninja` directly.
+
+## Compile with PIMCOMP
+
+PIMCOMP-NN reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
+select one paper profile, and restore it when the shell exits:
+
+```bash
+RAPTOR_ROOT=$PWD
+PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
+PAPER_MODELS="$RAPTOR_ROOT/validation/networks/pimcomp_models"
+CONFIG_BACKUP=$(mktemp)
+cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
+trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
+cp "$PAPER_MODELS/configs/arch-a.json" "$PIMCOMP/config.json"
+```
+
+The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile
+them directly:
+
+```bash
+cd "$PIMCOMP/build"
+
+# High-throughput mode; the paper evaluates batches of 128 samples.
+./PIMCOMP-NN -m=resnet18 -r=balance -p=batch -o=YES -v=YES -s=YES
+./PIMCOMP-NN -m=resnet34 -r=balance -p=batch -o=YES -v=YES -s=YES
+./PIMCOMP-NN -m=googlenet -r=balance -p=batch -o=YES -v=YES -s=YES
+
+# Low-latency mode; the paper uses batch size 1.
+./PIMCOMP-NN -m=resnet18 -r=balance -p=element -o=YES -v=YES -s=YES
+./PIMCOMP-NN -m=resnet34 -r=balance -p=element -o=YES -v=YES -s=YES
+./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES
+```
+
+VGG-8 first needs PIMCOMP's JSON frontend. Use a temporary ONNX copy because
+the released frontend rewrites the input batch dimension in place:
+
+```bash
+cd /path/to/Raptor
+cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx /tmp/vgg8-pimcomp.onnx
+.venv/bin/python third_party/PIMCOMP-NN/frontend/frontend.py \
+  --model_path /tmp/vgg8-pimcomp.onnx \
+  --save_path third_party/PIMCOMP-NN/models/JSON/vgg8_paper_reconstructed.json
+
+cd third_party/PIMCOMP-NN/build
+./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=batch -o=YES -v=YES -s=YES
+./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES
+```
+
+Repeat after selecting `arch-b.json` and `arch-c.json`. All four models were
+compiled successfully in both modes with all three configs. The released
+random placement code occasionally segfaults; an unchanged retry succeeded in
+the observed cases.
+
+The paper's optimizer uses a genetic algorithm with population 200 and up to
+1000 iterations. Select it with `-r=GA` for optimizer studies. The released
+source keeps population 200 but sets `max_iteration = 3`, so reproducing the
+paper's optimization search also requires changing that value in
+`backend/GeneticAlgorithm.h`. GA allocates roughly 32 GB in its fast evaluator;
+use monolith below instead of reducing cores or crossbars when local RAM is
+insufficient.
+
+## Compare Raptor and PIMCOMP
+
+The comparison driver uses one random input and one native ONNX-MLIR reference,
+compiles both instruction streams, validates both through Raptor's Rust
+simulator, and writes Markdown and JSON reports.
+
+To reproduce the complete Arch-A latency experiment, use the serial experiment
+runner. It creates an isolated PIMCOMP build with population 200 and 1000 GA
+iterations, runs only the `element`/batch-1 latency pipeline, and invokes the
+comparison driver for one model at a time:
+
+```bash
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
+  --out-dir /tmp/raptor-pimcomp-paper-latency
+```
+
+Reports are written under `<out-dir>/<model>/comparison_report.{md,json}`.
+Use `--models vgg8` to run one model, `--resume` after an interruption, or
+`--dry-run` to inspect every command. The runner continues after a failed model
+so all reports are produced, then returns a nonzero status if any comparison
+failed.
+
+Arch-A low-latency example:
+
+```bash
+RAPTOR_ROOT=$PWD
+PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
+CONFIG_BACKUP=$(mktemp)
+cp "$PIMCOMP/config.json" "$CONFIG_BACKUP"
+trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
+cp "$RAPTOR_ROOT/validation/networks/pimcomp_models/configs/arch-a.json" "$PIMCOMP/config.json"
+
+"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/compare_raptor_pimcomp.py" \
+  --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx" \
+  --out-dir /tmp/compare-vgg8-arch-a-ll \
+  --core-count 168 \
+  --crossbar-count 96 \
+  --crossbar-size 128 \
+  --mesh-rows 12 \
+  --mesh-cols 14 \
+  --pimsim-mode latency \
+  --pimcomp-pipeline element \
+  --fail-on-error
+```
+
+For Arch-A high throughput, use `--pimsim-mode throughput
+--pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a
+`6x23` mesh, and `configs/arch-b.json`.
+
+If only semantic and instruction comparison is required, add
+`--skip-pimsim-nn`. This exact VGG-8 Arch-A LL smoke test passed both semantic
+validations with maximum output differences below `5e-10`.
+
+Current Raptor status:
+
+- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
+- Use `googlenet-12-no-softmax.onnx` for the paper-matched latency comparison.
+  The original model's final `vsoftmax` is supported by Raptor's functional
+  simulator but not by `pimsim-nn`; PIMCOMP does not schedule that operation.
+- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
+  `512x1024` arrays can therefore be compiled by PIMCOMP but not compared
+  exactly with Raptor.
+
+Do not change the hardware profile to bypass either limitation; that would no
+longer be a paper-matched comparison.
+
+## Monolith fallback
+
+The local `monolith` SSH alias points to the high-memory host. Copy only this
+suite and the comparison driver; `-L` materializes the ResNet-18 symlink because
+the canonical `resnetv2/depth_68` file may not exist remotely:
+
+```bash
+REMOTE_REPO=/home/gmagnani/Project/Raptor
+rsync -azL validation/networks/pimcomp_models/ \
+  "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
+rsync -az validation/tools/compare_raptor_pimcomp.py \
+  "monolith:$REMOTE_REPO/validation/tools/compare_raptor_pimcomp.py"
+rsync -az validation/tools/run_pimcomp_paper_latency.py \
+  "monolith:$REMOTE_REPO/validation/tools/run_pimcomp_paper_latency.py"
+rsync -az --exclude=.git --exclude=build --exclude=output \
+  third_party/PIMCOMP-NN/ \
+  "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
+```
+
+Then use the same commands over SSH:
+
+```bash
+ssh monolith
+cd /home/gmagnani/Project/Raptor
+
+# One-time setup if the repository virtual environment is absent.
+python3 -m venv .venv
+.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
+
+# Run every latency comparison serially.
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
+  --out-dir /tmp/raptor-pimcomp-paper-latency
+```
+
+Copy reports back without transferring large compiler artifacts:
+
+```bash
+rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \
+  monolith:/tmp/raptor-pimcomp-paper-latency/ \
+  /tmp/raptor-pimcomp-paper-latency/
+```

validation/networks/pimcomp_models/RESULTS.md

@@ -0,0 +1,33 @@
+# Raptor vs PIMCOMP latency results
+
+## GoogLeNet, Arch-A, low latency
+
+Measured with `googlenet-12-no-softmax.onnx`, batch 1, Raptor's best current
+schedule, and PIMCOMP's GA/element artifacts. Both instruction streams were
+simulated by the same `pimsim-nn` build using the complete Arch-A timing and
+precision configuration.
+
+| Compiler | Latency (ms) | Instructions | Sends | Receives | MVMUL |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Raptor | 1132.458315 | 77,717,248 | 29,115 | 29,115 | 157,158 |
+| PIMCOMP | 41.790450 | 3,398,070 | 110,334 | 110,334 | 113,639 |
+
+PIMCOMP is 27.10x faster in this latency simulation.
+
+Semantic validation did not pass the comparison driver's strict default
+tolerance: the maximum logit differences from the native ONNX reference were
+`0.01995039` for Raptor and `7.768404` for PIMCOMP. Treat these as performance
+results, not as a correctness-equivalent comparison.
+
+No throughput experiment was run.
+
+## Reproduce
+
+```bash
+.venv/bin/python validation/tools/run_pimcomp_paper_latency.py \
+  --out-dir /tmp/raptor-pimcomp-paper-latency \
+  --models googlenet
+```
+
+See [README.md](README.md) for model provenance, limitations, and monolith
+instructions.

validation/networks/pimcomp_models/configs/arch-a.json

@@ -0,0 +1,54 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 96,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [128, 128],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [12, 14],
+      "net_config_file_path": "network_mesh_168.json"
+    },
+    "core_cnt": 168,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 200,
+    "report_verbose_level": 0
+  }
+}

validation/networks/pimcomp_models/configs/arch-b.json

@@ -0,0 +1,54 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 128,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [128, 128],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [6, 23],
+      "net_config_file_path": "network_mesh_138.json"
+    },
+    "core_cnt": 138,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 200,
+    "report_verbose_level": 0
+  }
+}

validation/networks/pimcomp_models/configs/arch-c.json

@@ -0,0 +1,54 @@
+{
+  "chip_config": {
+    "core_config": {
+      "period": 1,
+      "matrix_config": {
+        "xbar_array_count": 8,
+        "period": 1,
+        "pipeline_mode": true,
+        "dac_resolution": 1,
+        "dac_count": 128,
+        "xbar_size": [512, 1024],
+        "cell_precision": 2,
+        "xbar_latency": 7,
+        "xbar_read_power": 22.24,
+        "sample_hold_latency_cycle": 1,
+        "adc_resolution": 8,
+        "adc_latency_cycle": 20,
+        "adc_static_power": 0.322,
+        "adc_dynamic_power": 1.135,
+        "adc_count": 16,
+        "shift_adder_latency_cycle": 1,
+        "output_buffer_latency_cycle": 1
+      },
+      "vector_width": 32,
+      "vector_latency_cycle": 4,
+      "local_memory_config": {
+        "data_width": 64,
+        "period": 1,
+        "write_latency_cycle": 30,
+        "read_latency_cycle": 20
+      },
+      "global_memory_switch_id": -10
+    },
+    "global_memory_config": {
+      "data_width": 32,
+      "period": 1,
+      "write_latency_cycle": 50,
+      "read_latency_cycle": 40
+    },
+    "network_config": {
+      "bus_topology": "mesh",
+      "bus_width": 32,
+      "layout": [8, 8],
+      "net_config_file_path": "network_mesh_64.json"
+    },
+    "core_cnt": 64,
+    "global_memory_switch_id": -10
+  },
+  "sim_config": {
+    "sim_mode": 0,
+    "sim_time": 200,
+    "report_verbose_level": 0
+  }
+}

validation/networks/pimcomp_models/resnet18/resnet18-v1-7.onnx

@@ -0,0 +1 @@
+../../resnetv2/depth_68/resnetv2_depth_68.onnx