nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

rework documentation
Validate Operations / validate-operations (push) Failing after 3h10m55s
Details

Browse Source 
add plot generation scripts
adjust validation tools
add up-to-date results
This commit is contained in:
NiccoloN
2026-10-19 16:34:01 +02:00
parent 9053cc5e8b
commit 034c197260
33 changed files with 3077 additions and 1798 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+39
View File
@@ -0,0 +1,39 @@
# Raptor documentation
Start with the [build and quick start](../README.md). Commands in these guides
run from the repository root unless stated otherwise.
## Compiler
- [Architecture, compilation pipeline, source layout, and simulator tests](compiler/architecture.md)
- [Pim compiler options and defaults](compiler/options.md)
- [Spatial physical layouts and compatibility matrix](compiler/spatial-layouts.md)
- [Scheduling ownership, costs, capacity, and complexity](compiler/scheduling.md)
- [Scheduling implementation contracts and model limitations](compiler/scheduling-implementation.md)
## Evaluation
- [Measurement methodology and canonical result datasets](evaluation/methodology.md)
- [Pimcomp operation support, standalone reproduction, and GA controls](evaluation/pimcomp-reproducibility.md)
- [Validation commands and defaults](../validation/README.md)
- [Pimcomp comparison and plotting](../validation/tools/pim/pimcomp/compare/README.md)
- [Raptor ablation and plotting](../validation/tools/pim/ablation/README.md)
- [Pimcomp correctness experiments](../validation/tools/pim/pimcomp/correctness/README.md)
## Component references
- [Operation suite](../validation/operations/README.md) and [all 177 cases](../validation/operations/INVENTORY.md)
- [Comparison models, provenance, checksums, and hardware profiles](../validation/networks/pimcomp_models/README.md)
- [Graph Explorer usage](../tools/raptor_graph_explorer/README.md) and [architecture/API](../tools/raptor_graph_explorer/ARCHITECTURE.md)
## Normative engineering contracts
These contracts stay in `.agents/invariants/`; general documentation links to
them rather than restating their requirements:
- [Graph compute batch fragments](../.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md)
- [Compiler performance](../.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md)
- [Pimsim performance oracle](../.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md)
- [PIM synchronization](../.agents/invariants/PIM_SYNCHRONIZATION_INVARIANT.md)
- [Pipeline scheduling](../.agents/invariants/PIPELINE_SCHEDULING_INVARIANT.md)
- [Spatial target generality](../.agents/invariants/SPATIAL_TARGET_GENERALITY_INVARIANT.md)
docs/compiler/architecture.md
+51
View File
@@ -0,0 +1,51 @@
# Compiler architecture and pipeline
See the [documentation index](../README.md) for related compiler and evaluation
references. Source paths below are relative to the repository root.
## Compilation pipeline
```text
ONNX-MLIR -> Spatial -> Pim (tensor) -> Pim (bufferized) -> Pim artifacts
```
| Stage | Responsibility and source |
|---|---|
| ONNX → Spatial | Operation lowering, target-neutral layout planning, and plan materialization in `src/PIM/Conversion/ONNXToSpatial`. |
| Spatial scheduling | Graph merging, scheduling, verification, and communication realization in `src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes`; see [scheduling ownership](scheduling.md). |
| Spatial → Pim | Cores, communication, packing, and global tensors in `src/PIM/Conversion/SpatialToPim`. |
| Bufferization | Writable destinations, one-shot analysis, memory normalization, and verification in `src/PIM/Dialect/Pim/Passes/Transforms/Bufferization`. |
| Local-memory planning | Whole-core lifetimes and address reuse in `src/PIM/Dialect/Pim/Passes/Transforms/LocalMemoryPlanning`. |
| Verification/codegen | Verify the explicit memory plan and emit artifacts in `src/PIM/Passes/PimCodegen` and `src/PIM/Compiler`. |
Spatial plans select physical layouts and insert explicit materializations at
boundaries. Dense NCHW is the default; compatible operations can use row-strip
layouts, while Flatten can reassemble fragmented inputs. Function results
remain dense NCHW. See the [layout compatibility reference](spatial-layouts.md)
for operation alternatives, legality checks, and planner behavior.
## Validation and tests
[Validation instructions](../../validation/README.md) cover ONNX-MLIR reference
comparison, functional simulation, performance measurement, and artifacts.
The Rust functional simulator reads Raptor's artifact directory;
`pimsim-nn` supplies latency, power, and energy estimates.
```bash
cd backend-simulators/pim/pim-simulator
cargo test
```
## Repository layout
- `src/PIM/`: accelerator, dialects, conversions, analyses, and codegen.
- `test/PIM/`: compiler unit tests.
- `validation/`: operation/network suites and simulator configuration.
- `backend-simulators/pim/pim-simulator/`: Rust functional simulator.
- `backend-simulators/pim/pimsim-nn/`: performance simulator submodule.
- `tools/raptor_graph_explorer/`: [Spatial CSV viewer](../../tools/raptor_graph_explorer/README.md).
- `pimcomp_utils/`: comparison helpers containing local paths.
- `.github/`: CI build and validation setup.
CMake exposes `src/PIM` and `test/PIM` through generated shims under
`onnx-mlir/src/Accelerators/PIM` and `onnx-mlir/test/accelerators/PIM`.
docs/compiler/options.md
+28
View File
@@ -0,0 +1,28 @@
# Pim compiler options
See the [quick start](../../README.md#compile-a-model) for a complete command
and the [documentation index](../README.md) for the compilation pipeline.
Pass these to `onnx-mlir`; `--help` also lists inherited ONNX-MLIR options.
Boolean switches default to off unless stated otherwise.
| Option | Purpose / default |
|---|---|
| `--maccel=PIM` | Select the PIM accelerator. |
| `--EmitSpatial`, `--EmitPim`, `--EmitPimBufferized`, `--EmitPimCodegen` | Stop at a pipeline stage; default: codegen. |
| `--core-count=N`, `--crossbar-size=N`, `--crossbar-count=N` | Required hardware resources; crossbars are square. |
| `--pipeline=N` | Throughput stages; default: 1 (latency scheduling). |
| `--pim-target-config=PATH` | Target cost/topology configuration; resources must match hardware flags. Default: built-in model. |
| `--pim-memory-report=summary\|none` | Memory report at `reports/memory_report.txt`; default: `summary`. |
| `--pim-only-codegen` | Run the codegen tail on bufferized Pim IR. |
| `--pim-emit-json` | Also emit per-core JSON instructions. |
| `--pim-export-spatial-dataflow=none\|spatial1\|spatial2\|spatial3\|spatial4\|all` | Export graph, trivially merged, scheduled, and realized CSV snapshots under `reports/`; default: `none`. |
| `--pim-conv-lowering=STRATEGY` | `auto` (default), `legacy`, `depthwise`, `packed-im2col`, `streamed-patch`, `streamed-packed`, `output-channel-tiled`, `input-k-tiled`, or `tiled-2d`. |
| `--pim-conv-im2col-max-elements=N` | Global im2col limit before streaming; default: 1048576. |
| `--pim-conv-stream-chunk-positions=N` | Output positions per streamed chunk; default: 1024. |
| `--pim-report-conv-lowering=true\|false` | Bounded lowering report; default: true. |
| `--pim-disable-spatial-planning` | Choose each plan's first dense alternative; ONNX rewrites and graph merging remain enabled. |
| `--pim-disable-synchronization` | Omit wait/sync for ablation; removes correctness guarantees for buffer reuse and restart. |
| `--pim-detect-communication-deadlock` | Check expanded blocking send/receive ordering. |
| `--pim-verify-bufferization-copy-freedom` | Run the expensive pre-bufferization copy-freedom proof. |
docs/compiler/scheduling-implementation.md
+98
View File
@@ -0,0 +1,98 @@
# Scheduling implementation notes
The [scheduling overview](scheduling.md) maps ownership and summarizes costs/complexity.
Normative graph, pipeline, synchronization, and target contracts are linked
there; the details below describe current implementation choices.
## Publication, order, and local materialization
[`materializeScheduledCompute`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledComputeMaterialization.cpp) records
graph IDs for publication identity, not execution order. Class lanes retain
graph encounter order independently of physical core IDs, preserving regular
source-lane mappings. Within each core, instances sort by verified scheduler
slots. Only consecutive compatible instances form homogeneous runs; class-lane
ordering must not reorder a core's work.
`insertLocalFragment` replaces a complete singleton insertion with a leading
unit-dimension reshape view only when:
- source and destination have static shapes;
- destination rank is source rank + 1, with leading dimension 1 and remaining
dimensions exactly equal to the source shape;
- element type and encoding match;
- offset/size/stride lists all match destination rank; and
- every offset is statically zero, every size is the full corresponding
destination dimension, and every stride is statically one.
This includes scalar sources. Other insertions retain `tensor.insert_slice`
semantics rather than being converted into views.
Receive forwarding belongs to
[`PimBufferizationPass.cpp`](../../src/PIM/Dialect/Pim/Passes/Transforms/Bufferization/PimBufferizationPass.cpp),
after copy lowering and contiguity normalization. It requires a private
allocation and single-use receive/copy chain in the same block, zero source and
receive offsets, and equal sizes. The receive can sink to the copy target past
memory-effect-free operations and `memref.load` (including lane lookup loads).
Writes, synchronization, and other effectful operations are barriers. Forwarding
does not establish synchronization policy.
## Transfer accounting and transpose costs
[`ComputeGraph.cpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ComputeGraph.cpp) uses the same cached publication map for
dependency extraction and projected transfer volume. Physical slots map back
to graph lanes; graph-range overlap is not a byte estimate. Blueprint edges
belong to this shared graph for both scheduling strategies.
Exact projections charge selected physical fragments individually, including
each fragment's byte, endpoint-memory, and packet rounding. Conservative batch
dependencies charge a producer chunk as one aggregate: scale element count by
chunk lanes, multiply by effective bitwidth, then round to bytes once before
memory-access/flit rounding. Dividing an already rounded whole-result byte
count would lose sub-byte payloads. These two granularities are deliberate.
Spatial can analyze sub-byte storage; PIM communication lowering requires
byte-sized elements. Blueprint assembly charges per fragment. A full fragment
receive directly into assembly storage omits duplicate endpoint-memory edge
cost; later copy elimination can leave this estimate conservative.
Execution transpose costs share `getTransposeRun` with lowering: round memory
reads/writes per run and add vector execution when source stride is not one.
The [Pimsim vector unit](../../backend-simulators/pim/pimsim-nn/src/core/component/VectorUnit.cpp)
charges vector payload bytes, not the full strided address footprint. The
compiler model follows that accounting; instruction setup/issue and contention
remain approximations, not extra inferred memory traffic.
## Priorities, placement, and recovery limits
[`PipelineScheduling.cpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp) caches a per-template
adjustment between physical execution cost and nominal volume priority, scaling
it by chunk lanes. Partitioning and packing use the nominal priorities;
transposes count as bulk moves there. PEFT, final schedule evaluation, and
reported per-core demand use physical estimates. These quantities must not be
substituted for one another when interpreting results.
Placement compares maximum per-core serial compute plus incident same-stage
transfer demand, then completion/span and compute load. Each participating core
is charged incident transfer demand as a placement proxy; those charges are not
summed twice into elapsed time. Within 1% demand difference, span breaks the tie.
Ordered packing plus dependency-tail priority biases cuts away from long
dependent groups; it is not a throughput prediction. Measured simulator periods
remain authoritative, including when a proxy improves.
Core budgets come from stage allocation and bounded capacity recovery. Failed
packing budgets are not lower bounds. [PEFT reservations](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeftScheduler.cpp)
prefer weight reuse when distinct weights exceed half the fleet's one-copy
resident capacity; otherwise they favor communication/load. The reuse-first
constructor supplies feasibility recovery without replacing a successful
reservation, and can concentrate work. No exact combinatorial solver proves
infeasibility after heuristic failure.
[`ResidentWeightPacking.hpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp) adds at most 17
constructions after initial failure: one reverse traversal, then up to 16
single-choice replays sharing 65,536 candidate checks. Scratch buffers are reused;
an extra `O(T)` failed-assignment snapshot is created only after both initial
orders fail. Weight indexing still has `O(R W)` worst-case equality work; partition
DP and latency insertion-gap searches remain superlinear as listed in the
scheduling overview. Cached PEFT center distances avoid a topology scan per task/candidate;
projection and priority analyses share/cache work rather than revisiting every
template body per chunk. Ready-transfer signature searches remain another
potential compile-time hotspot.
docs/compiler/scheduling.md
+101
View File
@@ -0,0 +1,101 @@
# Spatial scheduling
`ScheduleAndRealizeSpatial` constructs a shared `ComputeGraph`, then selects
latency PEFT or pipeline scheduling. Pipeline mode does not run PEFT.
See [implementation notes](scheduling-implementation.md) for fragment-view eligibility,
receive forwarding, transfer rounding, cost-model boundaries, and recovery limits.
## Ownership
| Responsibility | Owner |
|---|---|
| Instances, dependencies, execution/transfer costs, resident weights | `ComputeGraph`, `ComputeInstanceUtils` |
| Target resources and physical link costs | Compiler target adapter via `SchedulingTarget` |
| Latency priorities, reservations, placement, per-core order | `PeftScheduler` |
| Capacity-driven workload splitting | `PipelineWorkloadPreparation` |
| Groups, stage partitioning, core budgets | `buildPipelineGroups`, `partitionPipelineGroups`, `assignPipelineStages` |
| Topology clustering and stage-local packing | `PipelineCoreLayout`, `packPipelineStages`, `IndexedResidentWeights` |
| Timing evaluation and publication equivalence | `evaluateSchedule`, `populateScheduleEquivalenceClasses` |
| Selected execution order and homogeneous runs | Scheduled compute planning/materialization |
| Routing, forwarding, blocking communication order | Deferred transfer planning and communication scheduling |
| READY/FREE counts, registers, overwrite/restart boundaries | Deferred boundary planning/realization |
Both strategies produce `MergeScheduleResult`: core assignments, contiguous
per-core slots, estimated starts, last-task markers, equivalence classes, and
cost summaries. Pipeline results also assign each core a stage. Materialization
uses verified slots for execution order and merges only consecutive compatible
instances. Graph/class lane identity does not imply core identity or execution
order.
Dependencies and transfer costs share `getGraphBatchPublicationMap`.
Statically proven physical-slot projections support many-to-many edges;
unproven projections retain complete producer dependencies, including region
captures. Malformed publication metadata fails graph construction.
Normative contracts:
[graph fragments](../../.agents/invariants/GRAPH_COMPUTE_BATCH_INVARIANT.md),
[pipeline stages](../../.agents/invariants/PIPELINE_SCHEDULING_INVARIANT.md),
[target boundary](../../.agents/invariants/SPATIAL_TARGET_GENERALITY_INVARIANT.md),
[synchronization](../../.agents/invariants/PIM_SYNCHRONIZATION_INVARIANT.md),
and [performance](../../.agents/invariants/PERFORMANCE_OPTIMIZATION_INVARIANT.md).
## Costs and interpretation
Execution and fixed transfer costs use integer nanoseconds, converted from
target cycles. Bytes, elements, flits, weight slots, and lanes remain resource
quantities. Memory work is rounded at its actual fragment/run granularity;
runtime-free views cost zero. Transposes charge memory traffic and strided
vector work; concatenation charges per-input blocks per outer row. Direct
receives into assembly storage avoid duplicate endpoint-memory charges.
Sub-byte storage is analyzable in Spatial but rejected by byte-addressed PIM
communication lowering.
The evaluator separates serial work, per-core work, dependency critical path,
summed transfer work, single-inference completion, and stage spans. Dependency
paths exclude core-order constraints; completion includes them. Stage spans
exclude cross-stage routing, restart/host synchronization, and contention.
Simulator latency and measured throughput remain the performance gates.
Pipeline partitioning uses ordered-packing/dependency-tail priorities and
nominal movement volumes. PEFT and final evaluation use physical cost estimates.
Placement ranks maximum per-core compute/incident-transfer demand, then stage
completion and compute load; within 1% demand, stage span breaks the tie.
These are placement heuristics, not a steady-state timing model.
## Capacity and recovery
Distinct-weight union / per-core capacity is a lower bound; an oversized task
is infeasible. Ordered pack counts estimate budgets, and failed heuristic
packing does not prove infeasibility. Exhausted budgets restart from lower
bounds and constructive packing establishes feasibility.
Latency reservations try load/communication-aware placement, then
largest-weight-first and shared reuse-first best fit on failure. Pipeline
packing compares arrival/load-aware candidates and uses constructive recovery.
The shared constructor retries in reverse order, then at most 16 single-choice
deviations within 65,536 core-candidate checks. Successful placements are
preserved. Exhausted search produces a bounded diagnostic, not a claim of
proved hardware infeasibility.
## Complexity and limits
Let `T` be tasks, `E` dependencies, `P` cores, `G` groups, `S` stages,
`R` weight references, `W` distinct weights, and `C` modeled work.
| Analysis | Time / storage |
|---|---|
| Schedule evaluation | `O((T + E) log T + P)` / `O(T + E + P)` |
| Weight indexing | Up to `O(R W)` equality comparisons |
| Weight lower bounds | `O(R + W)` / `O(W)` bits |
| Constructive best fit | `O(T log T + P R + P W)` / `O(P W)` bits plus `O(T + P)`; bounded retries only on failure |
| Partition DP | Up to `O(S G² ((T + R) log C + W))` plus dependency traversal / `O(S G²)` cache |
| Topology clustering / cached PEFT distances | `O(P²)` time |
| Concat cost extraction | `O(A D)` for `A` inputs of rank `D`; constant auxiliary space |
| Cached pipeline priorities | `O(T + D)` time / `O(T + G)` space for `D` template-body operations |
| Exact projections | `O(Q + L)` time/temporary space for `Q` slot visits and `L` producer lanes; publication maps cached |
| Conservative producer tracing | Linear in the input projection DAG, including captures |
Latency insertion-gap searches can remain quadratic. Issue overhead,
eliminated assembly copies, contention, and pipeline synchronization remain
cost approximations. Changes require matched execution measurements.
docs/compiler/spatial-layouts.md
+63
View File
@@ -0,0 +1,63 @@
# Spatial layout capabilities
This reference describes the alternatives advertised by
[`SpatialLayoutCapabilities.cpp`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Analyses/SpatialLayoutCapabilities.cpp),
through `SpatialLayoutCapabilityInterface`. Each alternative specifies all
operand layouts and one result layout; it is not a global promise that every
shape or target can use that layout.
## Physical layouts
Logical tensor coordinates remain NCHW. The physical enum is defined in
[`Spatial.td`](../../src/PIM/Dialect/Spatial/Spatial.td).
| Layout | Contract and current use |
|---|---|
| `DenseNCHW` | Ordinary dense storage; first/default alternative for every layout-aware plan. |
| `NHWCRowStrip` | Physical row strips of channel vectors for logically NCHW tensors; compatible operations can preserve this representation through a chain. |
| `Fragmented` | Existing fragmented input accepted by Flatten for dense reconstruction; no current capability advertises it as a result alternative. |
| `NCHWRowStrip` | Reserved NCHW-oriented row-strip enum; no current capability advertises it as an alternative. |
## Compatibility matrix
Every operation below also advertises all-dense operands → `DenseNCHW`.
Here `R` means `NHWCRowStrip`, `D` means `DenseNCHW`; Conv's other operands
retain the layouts supplied to its interface on the `R` path.
| Plan operation | Additional operand → result alternatives | Interface gate |
|---|---|---|
| Conv2D | Dense input → R; R input → R | `canLowerConvPlanToRowStrip`; `canConsumeAndProduceRowStrip`, respectively. |
| Flatten | Fragmented → D; R → D | Fragmented input already present; `canLowerFlattenFromRowStrip` for R. |
| Relu | R → R | Input already R. |
| SiLU | R → R | Input already R; intrinsic cost −3 instead of the generic R cost −2. |
| ResizeNearest | R → R | Input already R and `canLowerResizeNearestPlanToRowStrip`. |
| MaxPool2D | D → R; R → R | `canLowerMaxPoolPlanToRowStrip`. |
| GlobalAveragePool | D → R; R → R | `canLowerGlobalAveragePoolPlanToRowStrip`. |
| BiasAdd | (R data, D bias) → R | Ranked result, R data input, and `isSupportedBiasAddValue` for the bias shape. |
| Add | (R, R) → R | Both data operands already R. |
| Concat | All R inputs → R | Nonempty operand list, all already R. |
The named predicates live in the [plan-lowering implementation](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms)
and [bias helper](../../src/PIM/Conversion/ONNXToSpatial/Common/BiasAddUtils.hpp). They check operation geometry and
target resources; the matrix does not replace those legality checks.
## Selection and materialization
[`SpatialLayoutPlanningPass.cpp`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms/SpatialLayoutPlanningPass.cpp)
starts every plan at its dense alternative. It scores intrinsic cost, operand
layout mismatches, and downstream inability to accept the candidate layout.
Strictly cheaper alternatives replace the current selection; ties preserve it.
Traversal alternates forward/reverse order for at most `2 × plan_count + 1`
rounds. Nonconvergence fails the pass with a diagnostic.
Results returned from the function must remain dense. The pass records selected
layouts and inserts explicit `spat.materialize_layout` operations where selected
producer/consumer contracts disagree, including non-layout-aware consumers of
non-dense values. It verifies exact operand/result compatibility before
[`LowerSpatialPlans`](../../src/PIM/Conversion/ONNXToSpatial/Passes/Transforms/LowerSpatialPlansPass.cpp) lowers plans
and materializations.
`--pim-disable-spatial-planning` skips cost selection, choosing alternative 0
for every plan. The planning/materialization/verification pass still runs.
ONNX rewrites and later graph merging are unchanged; disabling layout planning
does not request fusion or removal of separate elementwise graph nodes.
docs/evaluation/methodology.md
+75
View File
@@ -0,0 +1,75 @@
# Evaluation methodology
The [documentation index](../README.md) links execution guides, model provenance,
reproduction details, and correctness experiments.
## Canonical result datasets and output roots
The authoritative default aggregate CSVs are:
- [Comparison results](../../validation/tools/pim/pimcomp/compare/results_comparison.csv)
- [Ablation results](../../validation/tools/pim/ablation/results_ablation.csv)
They are intentionally checked in beside their scripts. Default benchmark runs
update the corresponding canonical CSV; default plotting reads its current
contents. Plot generation does not run benchmarks or establish publication
provenance. Preserve the model/configuration/source/build revisions, invocation,
reference inputs, and original per-case reports when publishing measurements.
An explicit custom `--out-dir PATH` keeps aggregate CSVs under that experiment
root. Model artifacts still default to `validation/networks/pimcomp_models`;
changing aggregate placement does not change artifact or shared-reference paths.
Per-variant ablation summaries remain `<artifact-root>/ablation/<variant>/results_comparison.csv`.
Use `--input PATH` to plot a custom aggregate. Plot destinations can be selected
with `--output-dir PATH`. Cleanup of default model artifacts preserves the two
canonical datasets.
## Functional validity and performance availability
Functional PASS/FAIL and performance availability are independent. A numerically
correct program can fail performance simulation or complete no inference before
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
per-inference measurements become `NA` in aggregates, not zero. With zero
completed outputs, interval, throughput, and energy per inference are unavailable;
measured average chip power can still be reported. Functional status is retained.
Ablation percentages require positive finite values for both the variant and
its matching baseline. Tiny positive measurements retain significant digits
instead of being rounded to zero in the comparison CSV. An extreme valid
decrease can round to `-100.00%`; that value alone is not evidence of failure.
Reciprocal output interval and throughput cannot both have that decrease.
The ablation plotter warns and displays `N/A` for such invalid paired percentages,
while leaving the CSV and functional status unchanged.
## Report reuse and implementation revisions
The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
describes fingerprint validation and failed-report reuse. Fingerprints cover
model/configuration contents and experiment settings, but do not identify
compiler/tool implementations, built binaries, or manually changed reference
artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant
compiler after those change, and record build/source revisions for publication.
No revision cache is inferred from Git HEAD, which would miss dirty sources,
external tools, and binaries built from another checkout.
## Metric interpretation
Metrics come from Pimsim's
[`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp)
and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp).
- Latency mode measures time to complete one inference.
- Throughput-mode latency is measurement duration / completed iterations,
equal to `1000 / throughput` before rounding. It is not end-to-end pipelined
inference latency; multiplying it by depth does not establish that quantity.
- Average power is total modeled static + dynamic chip energy / elapsed time.
- Energy per inference is total energy / completed iterations (one in latency
mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size
division is applied.
Fast mode measures throughput over the last two convergence windows but power
and energy over the entire run up to early stop. Energy includes startup and
work in flight, so it need not equal average power / reported throughput.
Use `--no-fast` for full-duration measurements. Consult the
[correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows
as performance comparisons.
docs/evaluation/pimcomp-reproducibility.md
+111
View File
@@ -0,0 +1,111 @@
# Pimcomp reproduction reference
The [model README](../../validation/networks/pimcomp_models/README.md) owns provenance, checksums, hardware profiles, and
numeric-artifact limitations. The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
owns suite commands, reports, reuse rules, and metric interpretation.
## Frontend acceptance versus scheduled computation
The current [frontend mapping](../../third_party/PIMCOMP-NN/frontend/frontend.py)
accepts exactly these ONNX node types:
```text
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
```
`Constant` is consumed as metadata rather than emitted as a backend node.
Other types cause `operation: <type> not considered` and stop conversion at
the first occurrence. In particular, YOLO11n's Split and Resize are unsupported;
the emitted prefix cannot represent a complete inference.
The latency scheduler's explicit no-consider set, also used by hierarchy
mapping and GA post-operation accounting, is:
```text
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
Softmax, Squeeze, Transpose
```
See [ElementPipelineSchedule.cpp](../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
[HierarchyMapping.cpp](../../third_party/PIMCOMP-NN/backend/HierarchyMapping.cpp),
and [GeneticAlgorithm.cpp](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.cpp).
These accepted nodes receive no standalone latency instructions while remaining
in the backend graph. Acceptance therefore does not establish timed computation.
Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into
Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific
Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer.
Those transformations differ from both frontend rejection and leaving a node
unscheduled. In particular, the [BatchNorm correctness gap](../../validation/tools/pim/pimcomp/correctness/README.md#resnet-batchnorm-gap)
shows why a fusion label is not proof of numeric equivalence.
Pimsim accepts ISA instructions rather than ONNX operators. Its
[instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp)
rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must
not be approximated in the oracle. This is why the dedicated GoogLeNet artifact
removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other
operations, including MatMul/Transpose and detection decoding, for Raptor.
## Standalone frontend/backend reproduction
Build Pimcomp with CMake and an OpenMP-capable C++ compiler:
```bash
cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
```
The backend resolves `../config.json`, `../models/JSON/<name>.json`, and
`../output/` relative to its working directory, not its executable. The frontend
rewrites its input ONNX batch dimension in place. Use an isolated workspace and
a model copy, as the comparison driver does, to preserve source/config files.
From the Raptor root, after installing the [frontend dependencies](../../third_party/PIMCOMP-NN/README.md):
```bash
RAPTOR_ROOT=$PWD
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
PIMCOMP_RUN=$(mktemp -d)
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
--model_path "$PIMCOMP_RUN/vgg8.onnx" \
--save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
(
cd "$PIMCOMP_RUN/work"
OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
-m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
)
```
Outputs remain under `$PIMCOMP_RUN/output`. `-v=YES` emits verification data;
`-s=YES` emits `SimulationInfo.gz`; `-o=YES` saves the evaluation report.
Use `-p=batch` for throughput scheduling, with the matching throughput config
when running Pimsim. The paper's batch experiment used 128 samples; the current
comparison runner's functional batch default is 64. Element mode represents one
inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular
crossbars remain unsupported by Raptor.
For the bundled Model Zoo graphs, copy `models/JSON/resnet18.json`,
`resnet34.json`, or `googlenet.json` from Pimcomp to the workspace and change
`-m` accordingly. That reproduces bundled graph scheduling; use the suite
runner to regenerate graphs from the exact checked-in performance ONNX files.
## Genetic algorithm and reproducibility limits
[`GeneticAlgorithm.h`](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.h)
sets population 200 and at most 1000 iterations. `-r=GA` selects it; standalone
backend default replication is `balance`, whereas the suite explicitly uses GA.
GA initialization calls `srand(1)`. Fitness evaluation uses OpenMP with static
work distribution; `OMP_NUM_THREADS` bounds its parallelism (use 1 for serial
reproduction), otherwise the OpenMP runtime chooses the thread count.
The fixed GA seed does not make every backend stage deterministic:
`ElementPipelineSchedule.cpp` separately calls `srand(time(NULL))`. Keep model
hashes, source/build revisions, profile, mode, and thread count with experiment
results. Compare correctness
against the original ONNX reference, including the known batch and BatchNorm
failures; neither omitted operators nor prefilled reference intermediates repair
those programs.