Raptor/docs/evaluation/pimcomp-reproducibility.md

# Pimcomp reproduction reference

The [model README](../../validation/networks/pimcomp_models/README.md) owns provenance, checksums, hardware profiles, and
numeric-artifact limitations. The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
owns suite commands, reports, reuse rules, and metric interpretation.

## Frontend acceptance versus scheduled computation

The current [frontend mapping](../../third_party/PIMCOMP-NN/frontend/frontend.py)
accepts exactly these ONNX node types:

```text
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
```

`Constant` is consumed as metadata rather than emitted as a backend node.
Other types cause `operation: <type> not considered` and stop conversion at
the first occurrence. In particular, YOLO11n's Split and Resize are unsupported;
the emitted prefix cannot represent a complete inference.

The latency scheduler's explicit no-consider set, also used by hierarchy
mapping and GA post-operation accounting, is:

```text
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
Softmax, Squeeze, Transpose
```

See [ElementPipelineSchedule.cpp](../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
[HierarchyMapping.cpp](../../third_party/PIMCOMP-NN/backend/HierarchyMapping.cpp),
and [GeneticAlgorithm.cpp](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.cpp).
These accepted nodes receive no standalone latency instructions while remaining
in the backend graph. Acceptance therefore does not establish timed computation.

Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into
Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific
Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer.
Those transformations differ from both frontend rejection and leaving a node
unscheduled. In particular, the [BatchNorm correctness gap](../../validation/tools/pim/pimcomp/correctness/README.md#resnet-batchnorm-gap)
shows why a fusion label is not proof of numeric equivalence.

Pimsim accepts ISA instructions rather than ONNX operators. Its
[instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp)
rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must
not be approximated in the oracle. This is why the dedicated GoogLeNet artifact
removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other
operations, including MatMul/Transpose and detection decoding, for Raptor.

## Standalone frontend/backend reproduction

Build Pimcomp with CMake and an OpenMP-capable C++ compiler:

```bash
cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
```

The backend resolves `../config.json`, `../models/JSON/<name>.json`, and
`../output/` relative to its working directory, not its executable. The frontend
rewrites its input ONNX batch dimension in place. Use an isolated workspace and
a model copy, as the comparison driver does, to preserve source/config files.
From the Raptor root, after installing the [frontend dependencies](../../third_party/PIMCOMP-NN/README.md):

```bash
RAPTOR_ROOT=$PWD
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
PIMCOMP_RUN=$(mktemp -d)
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
  --model_path "$PIMCOMP_RUN/vgg8.onnx" \
  --save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
(
  cd "$PIMCOMP_RUN/work"
  OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
    -m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
)
```

Outputs remain under `$PIMCOMP_RUN/output`. `-v=YES` emits verification data;
`-s=YES` emits `SimulationInfo.gz`; `-o=YES` saves the evaluation report.
Use `-p=batch` for throughput scheduling, with the matching throughput config
when running Pimsim. The paper's batch experiment used 128 samples; the current
comparison runner's functional batch default is 64. Element mode represents one
inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular
crossbars remain unsupported by Raptor.

For the bundled Model Zoo graphs, copy `models/JSON/resnet18.json`,
`resnet34.json`, or `googlenet.json` from Pimcomp to the workspace and change
`-m` accordingly. That reproduces bundled graph scheduling; use the suite
runner to regenerate graphs from the exact checked-in performance ONNX files.

## Genetic algorithm and reproducibility limits

[`GeneticAlgorithm.h`](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.h)
sets population 200 and at most 1000 iterations. `-r=GA` selects it; standalone
backend default replication is `balance`, whereas the suite explicitly uses GA.
GA initialization calls `srand(1)`. Fitness evaluation uses OpenMP with static
work distribution; `OMP_NUM_THREADS` bounds its parallelism (use 1 for serial
reproduction), otherwise the OpenMP runtime chooses the thread count.

The fixed GA seed does not make every backend stage deterministic:
`ElementPipelineSchedule.cpp` separately calls `srand(time(NULL))`. Keep model
hashes, source/build revisions, profile, mode, and thread count with experiment
results. Compare correctness
against the original ONNX reference, including the known batch and BatchNorm
failures; neither omitted operators nor prefilled reference intermediates repair
those programs.