add ablation study

normalize names and artifact paths
2026-08-20 17:58:02 +02:00
parent add20e56eb
commit b009e1ff08
67 changed files with 1573 additions and 993 deletions
@@ -1,38 +1,38 @@
-# PIMCOMP comparison models
+# Pimcomp comparison models

 This directory contains the four networks evaluated in
-[PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
+[Pimcomp: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159):
 VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
 additional compiler comparison model.

-See the runner-generated [results.csv](results.csv) for the current comparison
+See the runner-generated [results_comparison.csv](results_comparison.csv) for the current comparison
 results. Rows are retained separately for each model, architecture, mode, and
 pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
-the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
-Running the runner with `--arch arch-b` or `--arch arch-c` appends those
-architecture rows without replacing the existing `arch-a` entries.
+the Raptor and Pimcomp artifacts; rows without a generated report contain `NA`.
+Use `--archs` to select the architecture rows to generate; existing rows for
+other architectures remain unchanged.

 ## Models and provenance

 | Directory    | Model                | Input         | Provenance                                                                                                                                                                                                 |
 |--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `resnet18/`  | ResNet-18 v1         | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`.                        |
-| `resnet34/`  | ResNet-34 v1         | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as PIMCOMP's frontend does.                                                        |
+| `resnet34/`  | ResNet-34 v1         | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as Pimcomp's frontend does.                                                        |
 | `googlenet/` | GoogLeNet            | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12).                                                                                                              |
-| `vgg8/`      | VGG-8 reconstruction | `1x1x28x28`   | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers.                                                            |
+| `vgg8/`      | VGG-8 reconstruction | `1x1x28x28`   | Reconstruction of the [Pimcomp VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers.                                                            |
 | `yolo11n/`   | YOLO11n detection    | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |

-`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
+`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit Pimsim-ready GoogLeNet model.
 It removes the two LRN nodes and terminal Softmax from the original model,
-so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
+so that the comparison covers only operations scheduled by Pimcomp and supported by Pimsim.

-`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
+`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit Pimsim-ready YOLO11n model.
 It removes the Softmax nodes from the original model,
-so that the compiled artifact can be simulated in pimsim-nn.
+so that the compiled artifact can be simulated in Pimsim.

 ## Unsupported and ignored operations

-PIMCOMP's frontend accepts exactly these ONNX operations:
+Pimcomp's frontend accepts exactly these ONNX operations:

 ```text
 Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
@@ -40,14 +40,14 @@ Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
 Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
 ```

-`Constant` is consumed as frontend metadata rather than emitted as a PIMCOMP
+`Constant` is consumed as frontend metadata rather than emitted as a Pimcomp
 node. Every other ONNX operation is unsupported: the frontend prints
 `operation: <type> not considered` and stops at the first occurrence. Thus the
 complete unsupported set is the complement of the allowlist above for the
 model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and
 `Resize` nodes.

-PIMCOMP's low-latency scheduler, hierarchy mapper, and genetic algorithm use
+Pimcomp's low-latency scheduler, hierarchy mapper, and genetic algorithm use
 this complete explicit no-consider set:

 ```text
@@ -63,7 +63,7 @@ specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its
 consumer. These transformations do not make an otherwise standalone ignored
 operation timed.

-`pimsim-nn` consumes PIM ISA instructions. It supports every named opcode
+Pimsim consumes Pim ISA instructions. It supports every named opcode
 in the shared serialized range except `vsoftmax` (opcode 21),
 which is rejected explicitly in both JSON and binary input. It
 silently ignores no opcode; unknown names and numbers are errors.
@@ -71,12 +71,12 @@ silently ignores no opcode; unknown names and numbers are errors.
 These boundaries explain the dedicated artifacts:

 - GoogLeNet's two LRN nodes and terminal Softmax perform real computation but
-  are ignored by PIMCOMP, so the common latency artifact removes them. Its
+  are ignored by Pimcomp, so the common latency artifact removes them. Its
  inference Dropout and shape-only Reshape can remain without adding compute.
 - YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can
-  run in `pimsim-nn`. Every other node, including MatMul, Transpose, and the
+  run in Pimsim. Every other node, including MatMul, Transpose, and the
  final detection-decoding tail, remains present and timed by Raptor. No
-  PIMCOMP latency is reported because its frontend stops at `Split` and also
+  Pimcomp latency is reported because its frontend stops at `Split` and also
  lacks `Resize`; compiling that prefix would not represent YOLO11n.

 The authoritative lists are in
@@ -85,16 +85,16 @@ The authoritative lists are in
 [`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and
 [`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp).

-The PIMCOMP authors did not publish the ONNX checkpoints used by the paper.
-Running PIMCOMP's frontend on the three Model Zoo files above produces JSON
-graphs exactly equal to PIMCOMP-NN's bundled `resnet18.json`, `resnet34.json`,
+The Pimcomp authors did not publish the ONNX checkpoints used by the paper.
+Running Pimcomp's frontend on the three Model Zoo files above produces JSON
+graphs exactly equal to Pimcomp's bundled `resnet18.json`, `resnet34.json`,
 and `googlenet.json`.

-There is no VGG-8 artifact in the ONNX Model Zoo or any PIMCOMP-NN revision.
+There is no VGG-8 artifact in the ONNX Model Zoo or any Pimcomp revision.
 The included VGG-8 therefore has deterministic random weights and is suitable
 for compiler and simulator comparison, not paper-accuracy reproduction. The
 paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the
-published PIMCOMP graphs and ResNet Model Zoo artifacts use ImageNet shapes.
+published Pimcomp graphs and ResNet Model Zoo artifacts use ImageNet shapes.

 Current SHA-256 checksums:

@@ -113,10 +113,10 @@ The files in
 [`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
 encode Table V's explicit resource parameters.
 Each profile subdirectory contains pre-generated latency and throughput
-`pimsim-nn` configs plus its matching mesh; comparison and validation reference
+Pimsim configs plus its matching mesh; comparison and validation reference
 these canonical artifacts directly.

-| Config                       |             Cores | Crossbars/core | Crossbar   |  Cell | PIMCOMP layout  |
+| Config                       |             Cores | Crossbars/core | Crossbar   |  Cell | Pimcomp layout  |
 |------------------------------|------------------:|---------------:|------------|------:|-----------------|
 | `arch-a/latency_config.json` |               168 |             96 | `128x128`  | 2-bit | `12x14`         |
 | `arch-b/latency_config.json` |               138 |            128 | `128x128`  | 2-bit | `6x23`          |
@@ -125,10 +125,10 @@ these canonical artifacts directly.
 `adc_count` is 16, matching the paper's 16-bit fixed-point weight precision.
 The paper does not give a two-dimensional core topology for Arch-A/B, so the
 factorizations above preserve core count but cannot reproduce unpublished NoC
-placement details. Released PIMCOMP-NN has no chip-count field; Arch-C is
+placement details. Released Pimcomp has no chip-count field; Arch-C is
 therefore flattened to 64 cores and does not model chip boundaries.

-The remaining latency and power values come from PIMCOMP-NN's released default
+The remaining latency and power values come from Pimcomp's released default
 configuration. Consequently, instruction/resource comparisons are
 reproducible, but absolute paper power and energy numbers are not.

@@ -147,9 +147,9 @@ cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN

 Do not build either project with `ninja` directly.

-## Compile with PIMCOMP
+## Compile with Pimcomp

-PIMCOMP-NN reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
+Pimcomp reads `third_party/PIMCOMP-NN/config.json` directly. Back it up,
 select one paper profile, and restore it when the shell exits:

 ```bash
@@ -162,7 +162,7 @@ trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT
 cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json"
 ```

-The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile
+The Model Zoo files map exactly to Pimcomp's bundled model names, so compile
 them directly:

 ```bash
@@ -179,7 +179,7 @@ cd "$PIMCOMP/build"
 ./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES
 ```

-VGG-8 first needs PIMCOMP's JSON frontend. Use a temporary ONNX copy because
+VGG-8 first needs Pimcomp's JSON frontend. Use a temporary ONNX copy because
 the released frontend rewrites the input batch dimension in place:

 ```bash
@@ -201,62 +201,66 @@ random placement code occasionally segfaults; an unchanged retry succeeded in
 the observed cases.

 The paper's optimizer uses a genetic algorithm with population 200 and up to
-1000 iterations. The checked-out PIMCOMP submodule already has both paper
+1000 iterations. The checked-out Pimcomp submodule already has both paper
 settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness
 evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS`
 to control its parallelism; otherwise OpenMP uses the available CPUs. The GA
 uses the fixed seed `1`, so repeated serial and parallel runs are reproducible.

-## Compare Raptor and PIMCOMP
+## Compare Raptor and Pimcomp

 The comparison driver uses one random input and one native ONNX-MLIR reference,
-compiles both instruction streams, runs both through `pimsim-nn`, runs
+compiles both instruction streams, runs both through Pimsim, runs
 functional validation through `pim-simulator`, and writes Markdown and JSON
 reports.

-To reproduce all configured architectures and both latency/throughput modes,
-use the model-by-model runner. It verifies the paper GA settings, expects Raptor
+To reproduce the default `arch-a`/`arch-b` architectures and both
+latency/throughput modes, use the model-by-model runner. Use `--archs` to
+specify a different architecture set. It verifies the paper GA settings, expects Raptor
 and the existing `third_party/PIMCOMP-NN/build` tree to already be built, then
-runs the comparisons in parallel and regenerates `results.csv` from the JSON
+runs the comparisons in parallel and regenerates `results_comparison.csv` from the JSON
 reports:

 ```bash
-.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
 ```

-Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
+Use `--archs arch-a --mode latency` for only the Arch-A latency experiment, or
+`--archs arch-a arch-b arch-c` to run all three architectures.

-Each model directory has a shared ignored `common/` directory containing
+Each model directory has a shared ignored `artifacts/common/` directory containing
 `inputs/`, `outputs/`, and the native `runner/`.
-Pimsim-nn configs and network meshes remain canonical under
+Pimsim configs and network meshes remain canonical under
 `validation/pimsim_configs/pimcomp/` and are referenced in place.
-Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
-artifacts remain under each comparison directory; PIMCOMP outputs are prepared
+Architecture- and pipeline-specific `raptor/`, `simulation/`, and Pimcomp
+artifacts remain under each `artifacts/<arch>/<mode>[/pipelineN]/` comparison
+directory; ablation variants use
+`artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>/`. Pimcomp outputs are prepared
 once per model/architecture/mode and linked into the other pipeline directories;
 `comparison_report.{md,json}`
 live under its `pimcomp/`. The frontend regenerates
-one isolated `models/JSON/` graph because PIMCOMP requires that relative
+one isolated `models/JSON/` graph because Pimcomp requires that relative
 layout; it is removed after a successful backend run and the shared submodule
 model directory is never modified. The original ONNX model is passed to the
 frontend unchanged.
-PIMCOMP's source tree and build directory remain unchanged at runtime. Use
+Pimcomp's source tree and build directory remain unchanged at runtime. Use
 `--models vgg8` to run one model, `--mode throughput` to select one mode,
 `--pipeline 4` to select one throughput pipeline, `--only raptor` or
 `--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
 to inspect every command, or `--out-dir PATH` to keep results outside
 `validation/`. Use `--clean` to remove generated comparison artifacts and
-summaries. Selecting a subset replaces only those comparison rows and
-recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
+summaries, including stale reference lock files. Selecting a subset replaces only those comparison rows and
+recomputes the aggregate `results_comparison.csv`; missing shared inputs, outputs, or the
 reference runner are generated even for an isolated run. Use `--jobs 4` to cap
 parallel comparisons. The per-stage timeout is unlimited by default; pass a
 positive `--timeout-seconds` value to impose one.
 Throughput comparisons default to `pimsim-nn --fast` with a 1000 ms
 convergence deadline. Add `--no-fast` for authoritative full-duration runs.
-PIMCOMP receives the original ONNX model, and its frontend applies native
+Pimcomp receives the original ONNX model, and its frontend applies native
 BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
 The runner continues after a failed model so all reports are produced.

-The known PIMCOMP batch-scheduling correctness issue and a reproducible
+The known Pimcomp batch-scheduling correctness issue and a reproducible
 reference-intermediate prefill experiment are documented in
 [`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).

@@ -265,9 +269,9 @@ Arch-A low-latency example:
 ```bash
 RAPTOR_ROOT=$PWD

-"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
+"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py" \
  --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
-  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
+  --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/artifacts/arch-a/latency" \
  --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
  --core-count 168 \
  --crossbar-count 96 \
@@ -279,10 +283,10 @@ RAPTOR_ROOT=$PWD
 ```

 Use the same command with
-`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
+`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released Pimcomp cannot
 compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
 for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
-as YOLO11n would produce a misleading latency, so no PIMCOMP number is
+as YOLO11n would produce a misleading latency, so no Pimcomp number is
 reported for this model.

 For Arch-A high throughput, use `--pimsim-mode throughput
@@ -303,18 +307,18 @@ generated report.
 The functional and non-functional simulators intentionally consume different
 artifacts:

- Raptor and PIMCOMP are validated against the native ONNX-MLIR reference as
+- Raptor and Pimcomp are validated against the native ONNX-MLIR reference as
  FP32 programs in the Rust simulator. Raptor's emitted program is already
-  FP32. The PIMCOMP-to-Rust export expands its element-addressed storage and
+  FP32. The Pimcomp-to-Rust export expands its element-addressed storage and
  byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector
  `imm_len` fields as element counts.
- PIMCOMP's original `SimulationInfo.gz` is copied unchanged for `pimsim-nn`.
-  PIMCOMP hardcodes `setbw 8, 8` and one byte per element without performing
+- Pimcomp's original `SimulationInfo.gz` is copied unchanged for Pimsim.
+  Pimcomp hardcodes `setbw 8, 8` and one byte per element without performing
  numerical quantization; this artifact is used only for latency estimation.
 - Raptor's original FP32 artifact remains unchanged for functional validation.
  A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its
  byte-addressed storage and transfer sizes from four bytes to one byte per
-  element. Vector `imm_len` fields remain element counts. Like PIMCOMP's
+  element. Vector `imm_len` fields remain element counts. Like Pimcomp's
  artifact, this view is not numerically valid and is used only for a fair
  non-functional comparison.

@@ -326,10 +330,10 @@ Current Raptor status:

 - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
 - Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
-  It removes the two LRN nodes and terminal softmax that PIMCOMP does not
+  It removes the two LRN nodes and terminal softmax that Pimcomp does not
  schedule.
 - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
-  `512x1024` arrays can therefore be compiled by PIMCOMP but not compared
+  `512x1024` arrays can therefore be compiled by Pimcomp but not compared
  exactly with Raptor.

 Do not change the hardware profile to bypass either limitation; that would no
@@ -365,7 +369,7 @@ python3 -m venv .venv
 .venv/bin/python -m pip install numpy onnx onnxruntime colorama

 # Run every configured comparison in parallel.
-.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
 ```

 Copy reports back without transferring large compiler artifacts:
@@ -1,61 +0,0 @@
-arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
-arch-a,vgg8,latency,1,element,PASS,PASS,1.47,7.99,NA,NA,325.67,200.11,477232782.04,1597904071.12,raptor,5.45
-arch-a,vgg8,throughput,2,batch,PASS,PASS,0.94,1.04,1060.00,965.00,377.54,482.60,474325940.70,541691691.00,raptor,1.10
-arch-a,vgg8,throughput,4,batch,PASS,PASS,1.19,1.04,839.00,965.00,310.32,482.60,500148858.20,541691691.00,pimcomp,1.15
-arch-a,vgg8,throughput,8,batch,PASS,PASS,1.52,1.04,658.00,965.00,279.37,482.60,554348483.90,541691691.00,pimcomp,1.47
-arch-a,resnet18,latency,1,element,PASS,FAIL,15.65,58.18,NA,NA,425.87,238.39,6666000553.12,13869146503.12,raptor,3.72
-arch-a,resnet18,throughput,2,batch,PASS,FAIL,30.55,13.46,32.70,74.30,295.90,481.60,10218790630.00,6562535800.00,pimcomp,2.27
-arch-a,resnet18,throughput,4,batch,PASS,FAIL,26.60,13.46,37.60,74.30,313.14,481.60,10993189350.00,6562535800.00,pimcomp,1.98
-arch-a,resnet18,throughput,8,batch,PASS,FAIL,35.51,13.46,28.20,74.30,275.95,481.60,12844245960.00,6562535800.00,pimcomp,2.63
-arch-a,resnet34,latency,1,element,PASS,FAIL,32.37,66.71,NA,NA,391.82,277.48,12684227802.68,18511540483.68,raptor,2.06
-arch-a,resnet34,throughput,2,batch,FAIL,FAIL,NA,23.58,NA,42.40,NA,515.66,NA,12277699310.00,NA,NA
-arch-a,resnet34,throughput,4,batch,FAIL,FAIL,NA,23.58,NA,42.40,NA,515.66,NA,12277699310.00,NA,NA
-arch-a,resnet34,throughput,8,batch,PASS,FAIL,115.11,23.58,8.69,42.40,229.85,515.66,29881854960.00,12277699310.00,pimcomp,4.88
-arch-a,googlenet,latency,1,element,PASS,PASS,13.03,62.92,NA,NA,465.11,231.19,6060646341.92,14547510894.24,raptor,4.83
-arch-a,googlenet,throughput,2,batch,PASS,FAIL,24.01,18.27,41.70,54.70,328.17,435.11,8529823770.00,8164469406.00,pimcomp,1.31
-arch-a,googlenet,throughput,4,batch,PASS,FAIL,27.01,18.27,37.00,54.70,305.68,435.11,10498372450.00,8164469406.00,pimcomp,1.48
-arch-a,googlenet,throughput,8,batch,PASS,FAIL,40.03,18.27,25.00,54.70,268.61,435.11,15290418970.00,8164469406.00,pimcomp,2.19
-arch-a,yolo11n,latency,1,element,PASS,FAIL,267.73,NA,NA,NA,235.73,NA,63111612279.00,NA,NA,NA
-arch-a,yolo11n,throughput,2,batch,PASS,FAIL,500.00,NA,2.00,NA,227.74,NA,113867559100.00,NA,NA,NA
-arch-a,yolo11n,throughput,4,batch,PASS,FAIL,500.00,NA,2.00,NA,233.16,NA,116581237400.00,NA,NA,NA
-arch-a,yolo11n,throughput,8,batch,PASS,FAIL,500.00,NA,2.00,NA,253.59,NA,126797224900.00,NA,NA,NA
-arch-b,vgg8,latency,1,element,PASS,PASS,1.44,7.15,NA,NA,304.72,173.77,438329594.04,1242814988.12,raptor,4.97
-arch-b,vgg8,throughput,2,batch,PASS,PASS,1.56,1.07,640.00,932.00,287.50,409.47,478592086.00,439288835.80,pimcomp,1.46
-arch-b,vgg8,throughput,4,batch,PASS,PASS,1.14,1.07,874.00,932.00,323.79,409.47,457617420.30,439288835.80,pimcomp,1.07
-arch-b,vgg8,throughput,8,batch,PASS,PASS,1.48,1.07,676.00,932.00,282.34,409.47,437422264.40,439288835.80,pimcomp,1.38
-arch-b,resnet18,latency,1,element,PASS,FAIL,17.98,64.38,NA,NA,362.63,201.66,6520518736.12,12983607699.12,raptor,3.58
-arch-b,resnet18,throughput,2,batch,PASS,FAIL,33.84,13.43,29.60,74.40,253.32,484.80,9423636948.00,6569655551.00,pimcomp,2.51
-arch-b,resnet18,throughput,4,batch,PASS,FAIL,26.14,13.43,38.30,74.40,286.86,484.80,9493311384.00,6569655551.00,pimcomp,1.94
-arch-b,resnet18,throughput,8,batch,PASS,FAIL,28.29,13.43,35.30,74.40,272.66,484.80,11314980340.00,6569655551.00,pimcomp,2.11
-arch-b,resnet34,latency,1,element,PASS,FAIL,49.51,107.20,NA,NA,284.72,206.47,14097763205.68,22134333948.68,raptor,2.17
-arch-b,resnet34,throughput,2,batch,PASS,FAIL,93.13,22.80,10.70,43.90,215.38,493.39,21278771500.00,11381213070.00,pimcomp,4.10
-arch-b,resnet34,throughput,4,batch,PASS,FAIL,67.55,22.80,14.80,43.90,237.97,493.39,19346594270.00,11381213070.00,pimcomp,2.97
-arch-b,resnet34,throughput,8,batch,PASS,FAIL,115.01,22.80,8.70,43.90,197.27,493.39,26678372930.00,11381213070.00,pimcomp,5.05
-arch-b,googlenet,latency,1,element,PASS,PASS,16.08,37.94,NA,NA,378.99,242.04,6095729659.92,9181933205.24,raptor,2.36
-arch-b,googlenet,throughput,2,batch,PASS,FAIL,28.37,21.39,35.20,46.70,267.15,417.83,8684222064.00,9014655684.00,pimcomp,1.33
-arch-b,googlenet,throughput,4,batch,PASS,FAIL,26.16,21.39,38.20,46.70,280.21,417.83,9300527914.00,9014655684.00,pimcomp,1.22
-arch-b,googlenet,throughput,8,batch,PASS,FAIL,46.45,21.39,21.50,46.70,237.73,417.83,14078049200.00,9014655684.00,pimcomp,2.17
-arch-b,yolo11n,latency,1,element,PASS,FAIL,316.70,NA,NA,NA,195.43,NA,61892225295.00,NA,NA,NA
-arch-b,yolo11n,throughput,2,batch,PASS,FAIL,333.33,NA,3.00,NA,214.01,NA,71336401290.00,NA,NA,NA
-arch-b,yolo11n,throughput,4,batch,PASS,FAIL,333.33,NA,3.00,NA,216.24,NA,72079194110.00,NA,NA,NA
-arch-b,yolo11n,throughput,8,batch,PASS,FAIL,500.00,NA,2.00,NA,218.42,NA,109211119900.00,NA,NA,NA
-arch-c,vgg8,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,vgg8,throughput,2,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,vgg8,throughput,4,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,vgg8,throughput,8,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet18,latency,1,element,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet18,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet18,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet18,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet34,latency,1,element,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet34,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet34,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,resnet34,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,googlenet,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,googlenet,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,googlenet,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,googlenet,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,yolo11n,latency,1,element,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,yolo11n,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,yolo11n,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
-arch-c,yolo11n,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
@@ -1,6 +0,0 @@
-Operation,Result,Compile,Host mem,Cores mem,Cores,Xbars,Latency,Power,Energy
-vgg8-mnist-reconstructed,PASS,1.009 s,1.37 MiB,3.14 MiB,141,761,1.465778 ms,325.627854 mW,477298145.040001 pJ
-resnet18-v1-7,PASS,11.548 s,9.89 MiB,40.24 MiB,168,7676,28.099952 ms,312.513408 mW,8781611766.119984 pJ
-resnet34-v1-7,PASS,28.495 s,9.90 MiB,48.89 MiB,168,15292,45.781486 ms,326.833870 mW,14962940227.679951 pJ
-googlenet-12-pimsim-nn,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
-yolo11n-pimsim-nn,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ