Raptor/docs/evaluation/pimcomp-reproducibility.md

Pimcomp reproduction reference

The model README owns provenance, checksums, hardware profiles, and numeric-artifact limitations. The comparison guide owns suite commands, reports, reuse rules, and metric interpretation.

Frontend acceptance versus scheduled computation

The current frontend mapping accepts exactly these ONNX node types:

Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze

Constant is consumed as metadata rather than emitted as a backend node. Other types cause operation: <type> not considered and stop conversion at the first occurrence. In particular, YOLO11n's Split and Resize are unsupported; the emitted prefix cannot represent a complete inference.

The latency scheduler's explicit no-consider set, also used by hierarchy mapping and GA post-operation accounting, is:

Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
Softmax, Squeeze, Transpose

See ElementPipelineSchedule.cpp, HierarchyMapping.cpp, and GeneticAlgorithm.cpp. These accepted nodes receive no standalone latency instructions while remaining in the backend graph. Acceptance therefore does not establish timed computation.

Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer. Those transformations differ from both frontend rejection and leaving a node unscheduled. In particular, the BatchNorm correctness gap shows why a fusion label is not proof of numeric equivalence.

Pimsim accepts ISA instructions rather than ONNX operators. Its instruction decoder rejects vsoftmax (opcode 21) and unknown opcodes; unsupported operations must not be approximated in the oracle. This is why the dedicated GoogLeNet artifact removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other operations, including MatMul/Transpose and detection decoding, for Raptor.

Standalone frontend/backend reproduction

Build Pimcomp with CMake and an OpenMP-capable C++ compiler:

cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN

The backend resolves ../config.json, ../models/JSON/<name>.json, and ../output/ relative to its working directory, not its executable. The frontend rewrites its input ONNX batch dimension in place. Use an isolated workspace and a model copy, as the comparison driver does, to preserve source/config files. From the Raptor root, after installing the frontend dependencies:

RAPTOR_ROOT=$PWD
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
PIMCOMP_RUN=$(mktemp -d)
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
  --model_path "$PIMCOMP_RUN/vgg8.onnx" \
  --save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
(
  cd "$PIMCOMP_RUN/work"
  OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
    -m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
)

Outputs remain under $PIMCOMP_RUN/output. -v=YES emits verification data; -s=YES emits SimulationInfo.gz; -o=YES saves the evaluation report. Use -p=batch for throughput scheduling, with the matching throughput config when running Pimsim. The paper's batch experiment used 128 samples; the current comparison runner's functional batch default is 64. Element mode represents one inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular crossbars remain unsupported by Raptor.

For the bundled Model Zoo graphs, copy models/JSON/resnet18.json, resnet34.json, or googlenet.json from Pimcomp to the workspace and change -m accordingly. That reproduces bundled graph scheduling; use the suite runner to regenerate graphs from the exact checked-in performance ONNX files.

Genetic algorithm and reproducibility limits

GeneticAlgorithm.h sets population 200 and at most 1000 iterations. -r=GA selects it; standalone backend default replication is balance, whereas the suite explicitly uses GA. GA initialization calls srand(1). Fitness evaluation uses OpenMP with static work distribution; OMP_NUM_THREADS bounds its parallelism (use 1 for serial reproduction), otherwise the OpenMP runtime chooses the thread count.

The fixed GA seed does not make every backend stage deterministic: ElementPipelineSchedule.cpp separately calls srand(time(NULL)). Keep model hashes, source/build revisions, profile, mode, and thread count with experiment results. Compare correctness against the original ONNX reference, including the known batch and BatchNorm failures; neither omitted operators nor prefilled reference intermediates repair those programs.