diff --git a/.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md b/.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md index 8980da5..6f45f57 100644 --- a/.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md +++ b/.agents/invariants/PIMSIM_NN_ORACLE_INVARIANT.md @@ -9,18 +9,35 @@ behavior defines the hardware model used for Raptor/PIMCOMP comparisons. - Changes to pimsim-nn must preserve simulation behavior exactly. Acceptable changes are limited to behavior-neutral maintenance proven not to alter simulated timing, scheduling, power, energy, or supported input programs. -- Unsupported pimsim-nn operations must remain unsupported; do not approximate - their timing or map them onto another operation. +- Other unsupported or unknown pimsim-nn operations remain unsupported. Do not + approximate their timing or map them onto another operation. - Validation must report an explicit unsupported-op diagnostic as `UNSUPPORTED`, not as a simulator failure. - A compiled artifact with zero active PIM cores has no meaningful non-functional simulation and must be reported as `SKIP`. -- Adapt compiler inputs to the oracle instead. For YOLO, use - `validation/networks/pimcomp_models/yolo11n/yolo11n-pimsim-nn.onnx`, the - dedicated pimsim-ready performance artifact with Softmax operations removed. - Use `validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx` for YOLO - functional validation; removing Softmax changes the model's numerical - behavior, so the pimsim-ready artifact is not a correctness reference. + +## Explicit `vsoftmax` exception + +`vsoftmax` has no Pimsim-nn hardware timing model. This one explicitly +authorized exception does not permit other simulator approximations. + +- Pimsim-nn omits `vsoftmax` while loading JSON or binary instructions, before + fetch. Omitted instructions consume no fetch, issue, execution, memory, or + dynamic-energy service. +- Pimsim-nn warns once per process that Softmax timing and energy are excluded. +- Functional simulation executes Softmax normally. +- Raptor's queue estimator applies the same zero-cost omission. All surrounding + instructions, materialization, transfers, and synchronization remain modeled. +- `state.instructions` intentionally counts `vsoftmax`: it is a real encoded + PIM instruction, although Pimsim-nn does not simulate it. +- No additional Pimsim-nn behavior changes are authorized. + +Current comparisons use the full YOLO11n model, including both Softmax +operations, at `validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx`. +GoogLeNet uses +`validation/networks/pimcomp_models/googlenet/googlenet-12-no-lrn.onnx`, +with only its two unsupported LRN nodes bypassed. Terminal Softmax is retained. +Historical Softmax-free artifacts are not matched baselines for these graphs. Any proposed pimsim-nn behavior change requires explicit user authorization and must not be introduced as part of a compiler optimization. diff --git a/docs/compiler/options.md b/docs/compiler/options.md index da6fe4a..242c572 100644 --- a/docs/compiler/options.md +++ b/docs/compiler/options.md @@ -24,7 +24,7 @@ Boolean switches default to off unless stated otherwise. | `--pim-conv-stream-chunk-positions=N` | Output positions per streamed chunk; default: 1024. | | `--pim-report-conv-lowering=true\|false` | Bounded lowering report; default: true. | | `--pim-disable-spatial-planning` | Choose each plan's first dense alternative; ONNX rewrites and graph merging remain enabled. | -| `--pim-disable-synchronization` | Omit wait/sync for ablation; removes correctness guarantees for buffer reuse and restart. | +| `--pim-disable-synchronization` | Keep the normal selected schedule and omit wait/sync during code generation. Report a separate zero-cost synchronization estimate for ablation. Removes correctness guarantees for buffer reuse and restart. | | `--pim-detect-communication-deadlock` | Check expanded blocking send/receive ordering. | | `--pim-verify-bufferization-copy-freedom` | Run the expensive pre-bufferization copy-freedom proof. | diff --git a/docs/compiler/scheduling-implementation.md b/docs/compiler/scheduling-implementation.md index f05ef69..1dbcca6 100644 --- a/docs/compiler/scheduling-implementation.md +++ b/docs/compiler/scheduling-implementation.md @@ -136,12 +136,10 @@ remain approximations, not extra inferred memory traffic. [`PipelineScheduling.cpp`](../../src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp) caches a per-template adjustment between physical execution cost and nominal volume priority, scaling -it by chunk lanes. The original partitioning and packing candidate uses nominal -priorities; transposes count as bulk moves there. The memory-aware candidate -uses execution estimates for partitioning, retaining the existing core-budget -and packing heuristics. Final -selection evaluates bounded alternatives after physical placement, retaining -the original candidate when the combined resource estimates worsen. PEFT, +it by chunk lanes. Initialization uses nominal priorities; transposes count as +bulk moves there. One joint prefix/resource DP constructs the initial partition. +Bounded beam refinement changes partitions and their core allocations together. +Queue-aware Spatial evaluation scores every admitted physical placement. PEFT, final schedule evaluation, and reported per-core demand use physical estimates. These quantities must not be substituted for measured simulator periods. @@ -220,7 +218,7 @@ Consumed prefixes, capacities, and compaction indices cannot affect future timin An idle controller's past completion cannot delay future arrivals. These normalizations require causal, monotonic transitions, which the proof checks. -Proof construction uses at most 256 generations and 16 MiB of snapshot storage. +Proof construction uses at most 1024 generations and 64 MiB of snapshot storage. Each checkpoint scans events, dependencies, and remaining tokens, then orders a copy of the pending frontier. Exact comparisons use bounded history; hashes only filter comparisons. Unproved recurrence remains explicit and cannot guide multi-candidate selection. diff --git a/docs/compiler/scheduling.md b/docs/compiler/scheduling.md index e9ea623..af3fe02 100644 --- a/docs/compiler/scheduling.md +++ b/docs/compiler/scheduling.md @@ -2,7 +2,7 @@ `prepareSpatialSchedules` constructs the shared `ComputeGraph`. Latency mode uses PEFT. For `--pipeline > 1`, the PIM compiler selects among bounded pipeline -candidates with static queue-aware instruction analysis. Pipeline mode does not run PEFT. +candidates with static queue-aware Spatial execution analysis. Pipeline mode does not run PEFT. See [implementation notes](scheduling-implementation.md) for fragment-view eligibility, receive forwarding, transfer rounding, cost-model boundaries, and recovery limits. @@ -15,7 +15,7 @@ receive forwarding, transfer rounding, cost-model boundaries, and recovery limit | Target resources and physical link costs | Compiler target adapter via `SchedulingTarget` | | Latency priorities, reservations, placement, per-core order | `PeftScheduler` | | Capacity-driven workload splitting | `PipelineWorkloadPreparation` | -| Groups, stage partitioning, core budgets | `buildPipelineGroups`, `partitionPipelineGroups`, `assignPipelineStages` | +| Groups, stage partitioning, core budgets | `buildPipelineGroups`, `initialPipelinePartition`, `optimizePipelineSchedule` | | Topology clustering and stage-local packing | `PipelineCoreLayout`, `packPipelineStages`, `IndexedResidentWeights` | | Timing evaluation and publication equivalence | `evaluateSchedule`, `populateScheduleEquivalenceClasses` | | Selected execution order and homogeneous runs | Scheduled compute planning/materialization | @@ -58,14 +58,61 @@ paths exclude core-order constraints; completion includes them. Stage spans exclude cross-stage routing, restart/host synchronization, and contention. Simulator latency and measured throughput remain the performance gates. -Pipeline candidate generation retains the original ordered-packing/dependency-tail -heuristic and the memory-aware and sharing-aware searches. The partition DP -keeps at most eight compute/memory trade-offs per state. Four final alternatives -and the initial partition undergo core allocation, topology placement, and -capacity verification. The previous compute/memory guards identify candidate -zero, the original fallback. They do not reject legal alternatives before queue -analysis. All candidates preserve candidate zero's non-empty pipeline stages. -PEFT is unchanged. +Throughput scheduling uses one bounded joint optimizer. Initialization groups all +lanes of an operation and computes one ordered partition with a prefix DP. +The DP jointly assigns operation prefixes and at most sixteen units of spare +core capacity. Each configured stage reserves one physical core. Quantization +limits initialization only; refinement uses geometric core transfers, including one core. +Quantization can miss a feasible allocation. Initialization failure reports a +bounded-search limitation, not proved hardware infeasibility. +The seed minimizes maximum stage packing/dependency priority over the legal +active-stage counts within the configured depth. These priorities provide +a starting point, not a timing guarantee. Refinement can remove stage boundaries. +No empty stage is inserted between dependent operations. + +Every proposed partition receives a minimax core budget for its changed stages. +Unchanged stages retain their budgets. The allocator builds +constructive ordered-packing curves and uses reuse-first resident-weight packing +when order alone cannot realize a capacity bound. A feasible packing remains +feasible with idle extra cores. All placements pass the existing topology-aware +physical packing, operation-group, adjacency and capacity checks. + +Refinement uses a beam of two schedules ranked by the larger queue estimate. +Its ordered neighborhood moves operation groups across adjacent stage boundaries, +inserts a boundary, or removes consecutive boundaries. +Refinement preserves contiguous groups in the initialization's topological order. +Coalescing compacts later stages. Each changed block redistributes its core pool. +Core coordinates also transfer powers of two between stages, including one core. +These moves cross packing plateaus within a bounded number of evaluations. +The configured pipeline depth and total physical resources remain fixed. + +A cheap proxy ranks the neighborhood before physical packing and queue analysis. +It uses ordered-packing curves, dependency spans and aggregate shared-memory demand. +Refinement uses the target's existing per-operation execution costs, rather than +redistributing aggregate processor work across operations. +Processor service and critical recurrence contributions guide ties toward affected bottlenecks. +The PIM adapter supplies target-neutral processor guidance from the shared periodic +graph. PIM timing semantics remain outside Spatial. + +The optimizer evaluates at most thirty-two complete Spatial queue costs and attempts +at most 128 constructive allocations and physical packings. +Each round evaluates at most two new placements. +Each retained parent receives one queue-evaluation opportunity per round. +The first round can evaluate two neighbors of its single parent. +The two best evaluated schedules form the next beam. +All evaluated schedules remain available for final comparison. +Search stops at the evaluation or +packing budget, or when the beam's legal unvisited neighborhood is exhausted. +Canonical stage/budget vectors, physical core order and evaluation index break +ties deterministically. Actual physical placements, including inactive-core +stage identities, are deduplicated. The final 2.1% near-equivalence policy compares +the retained queue estimates. There are no independent legacy, sharing-aware, +allocator-only or boundary-shift candidate-generation policies. + +Unavailable timing, unsupported static forms or unproved recurrence explicitly +stop a costed search. They are not silently discarded. One operation group or +one distinct neighborhood schedule skips optional queue comparison and lowers +once. Final backend resource retries remain restricted to already costed schedules. `ComputeGraph` retains physical publication and consumer identities separately from aggregated dependency edges. Cross-stage demands charge one store per @@ -247,27 +294,28 @@ pending tokens. The event heap contains at most one event per node. FIFO token storage retains pending generations, rather than instruction-stream copies. The diagnostic limits each event to 64 generations and processed events to `16(V + E)`. It reports `unavailable` if either bound prevents convergence. -The two tie-order runs reuse the same graph. Production uses this queue execution directly and omits maximum-cycle-ratio analysis. +The two tie-order runs reuse the same graph. Production uses queue execution for costs and maximum-cycle-ratio analysis for bounded search guidance. The standalone frontend retains the recurrence witness. Neither analysis changes latency PEFT. ## Default throughput selection Full PIM code generation uses Spatial-only queue-aware selection for `--pipeline > 1`. Latency PEFT and `--pim-only-codegen` retain their existing behavior. -Intermediate emission retains the original schedule and stops at its requested stage. +Intermediate emission retains the initialized schedule and stops at its requested stage. Its schedule can differ from the final program. The compiler performs these steps: 1. Run the frontend, layout planning, and graph preparation once. -2. Generate at most eleven legal schedules from the existing searches. -3. Remove duplicate physical placements, including stage identities and core order. -4. Realize each candidate in an independent clone of the prepared checkpoint. +2. Construct one legal initial partition, allocate cores, and place its work. +3. Rank legal joint partition/allocation moves with inexpensive resource proxies. +4. Realize each admitted schedule in an independent clone of the prepared checkpoint. 5. Query the target adapter over realized Spatial IR, then release the clone. -6. Require supported static forms, available costs, and proved modeled recurrence for every candidate. -7. Select a candidate with the existing 2.1% selection margin. -8. Realize the selected schedule again from the unchanged checkpoint. -9. Lower, allocate, verify, and emit only the selected program. +6. Require supported static forms, available costs, and proved modeled recurrence. +7. Refine a two-schedule beam within thirty-two total queue evaluations. +8. Apply the existing 2.1% near-equivalence rule to the evaluated schedules. +9. Realize the selected schedule again from the unchanged checkpoint. +10. Lower, allocate, verify, and emit only the selected program. A single candidate skips event extraction, queue analysis, recurrence proof, and comparison. It is realized and lowered once, even when optional timing is unavailable. @@ -283,14 +331,17 @@ The target supplies model evidence. Spatial applies the 2.1% selection policy ma Selection does not read JSON diagnostics. Reports serialize the typed results. Spatial owns candidate generation, legality, placement, and target-neutral comparison. The PIM adapter owns execution-event extraction, instruction timing, and resource contention. +Pimsim-nn omits `vsoftmax` before fetch, so the PIM queue model assigns it no timed work. +Input materialization, surrounding instructions, transfers, and synchronization retain their costs. +Functional simulation still executes Softmax. Estimated timing and energy exclude Softmax work. The shared queue engine models controller arrival arbitration and cross-generation dependencies. The compiler does not execute candidate programs or invoke a simulator. Both arbitration policies must establish complete-state recurrence within bounded analysis budgets. Their estimates are sensitivity checks, not formal bounds. -The candidate with the lowest larger estimate must separate from the original schedule. +The candidate with the lowest larger estimate must separate from the initialized schedule. Comparison uses `[min(ties)/1.021, max(ties)/0.979]`. -Close alternatives retain candidate zero. The 2.1% margin is a policy heuristic. +Close alternatives retain the initialized schedule. The 2.1% margin is a policy heuristic. Spatial timing predictions have no formal accuracy bound. The search finds the best distinguishable candidate in its bounded set, without global optimality. @@ -310,7 +361,7 @@ Code-generation failures also stop compilation when recovery cannot be justified If every candidate fails, one diagnostic lists all attempted candidates and causes. `reports/pipeline_selection.json` records Spatial costs, eligibility, selection, and backend attempts. -It also records candidate generation, realization, extraction, expansion, queue, lowering, and finalization times. +It also records search parents, compact task-stage runs, core budgets, recurrence contributions, termination, attempts, queue budget, and phase times. `--enable-timing` adds per-pass lowering times and IR operation/value counts. Final instruction generation uses the normal compiler timing report. Explicit Spatial exports and IR dumps reconstruct pre-lowering artifacts for the selected schedule. @@ -328,8 +379,19 @@ Optional diagnostics expose resource work, recurrence, and ordered-request finge Offline Pimsim fast-mode runs validate predictions and emitted programs. Simulator measurements never enter production selection. -For `C <= 11` candidates, scheduling metadata uses `O(C(T+P))` storage for `T` compute instances and `P` cores. -Placement deduplication costs `O(C²T)` time. +For `C <= 32` candidates, scheduling metadata uses `O(C(T+P))` storage for `T` compute instances and `P` cores. +Placement deduplication costs `O(C²(T+P))` time. The beam retains at most two schedules. +A round streams legal boundary and core moves into a shortlist of at most `2C`. +Each parent builds stage resource and dependency summaries in `O(T+E+P)` time and storage. +Unchanged-stage packing costs use an `O(SP)` cache. Changed task lists are not cached. +Each parent proposes `O(G + S² log P)` group, cut, and core moves. +For `M` proposed assignments, cheap allocation and scoring cost +`O(M (T log T + S P + E + F + S Kpack))`, plus bounded shortlist comparisons, +where `Kpack` is one existing ordered-packing query's cost. +Constructive allocation builds at most `S P` ordered-packing queries per admitted +partition. The graph and weight indices are reused. Search performs at most +128 physical packings; queue costs remain capped at thirty-two. +Recurrence guidance uses the shared bounded maximum-cycle-ratio solver. Candidate realization and extraction are sequential; complete candidate modules are not retained. Normal successful compilation performs one candidate-specific backend lowering. Recovery performs at most `C` lowerings. Retry selection costs `O(C²)` time and `O(C)` metadata. @@ -354,7 +416,7 @@ proved hardware infeasibility. Let `T` be tasks, `E` dependencies, `P` cores, `G` groups, `S` stages, `R` weight references, `W` distinct weights, `C` modeled work, `F` physical -fragment/consumer demands, and bounded frontier size `K = 8`. +fragment/consumer demands, and resource quantization `Q <= 16`. | Analysis | Time / storage | |---|---| @@ -362,8 +424,8 @@ fragment/consumer demands, and bounded frontier size `K = 8`. | Weight indexing | Up to `O(R W)` equality comparisons | | Weight lower bounds | `O(R + W)` / `O(W)` bits | | Constructive best fit | `O(T log T + P R + P W)` / `O(P W)` bits plus `O(T + P)`; bounded retries only on failure | -| Partition DP | Up to `O(S G² ((T + R) log C + W + K³))` plus dependency traversal / `O(S G² + S G K)`; at most five additional physical placements per memory objective | -| Cut memory service | Expected `O(F + G)` time / `O(F + G)` scratch and retained demands | +| Joint initialization DP | `O(G² Q Kpack + S G² Q²)` time / `O(G² Q + S G Q)` storage; `Kpack` is one ordered-pack cost query | +| Neighborhood memory service | Expected `O(T + F)` per partition / `O(T + F)` scratch | | Topology clustering / cached PEFT distances | `O(P²)` time | | Concat cost extraction | `O(A D)` for `A` inputs of rank `D`; constant auxiliary space | | Cached pipeline priorities | `O(T + D)` time / `O(T + G)` space for `D` template-body operations | diff --git a/docs/evaluation/methodology.md b/docs/evaluation/methodology.md index a4c9c60..56546c4 100644 --- a/docs/evaluation/methodology.md +++ b/docs/evaluation/methodology.md @@ -33,13 +33,29 @@ per-inference measurements become `NA` in aggregates, not zero. With zero completed outputs, interval, throughput, and energy per inference are unavailable; measured average chip power can still be reported. Functional status is retained. +Floating-point correctness requires both elementwise absolute/relative tolerance +and a relative L2 limit for each nonzero reference output. The L2 limit prevents +small reference outputs from accepting all-zero results under the absolute floor. +It cannot substitute for the elementwise check. Zero references use the absolute +check; integer outputs require exact agreement. Nonfinite outputs are rejected. +Validation reports record both maximum absolute and relative L2 errors. +Report reuse includes the validation-policy identity. + +Removing normalization operations can amplify floating-point sensitivity and +change the computation. Such variants must use their own native reference. +Agreement in a global norm alone does not establish elementwise correctness. +Independent reference implementations and higher-precision calculations can +help diagnose rounding sensitivity without relaxing the recorded pass criteria. + Ablation percentages require positive finite values for both the variant and its matching baseline. Tiny positive measurements retain significant digits instead of being rounded to zero in the comparison CSV. An extreme valid decrease can round to `-100.00%`; that value alone is not evidence of failure. Reciprocal output interval and throughput cannot both have that decrease. -The ablation plotter warns and displays `N/A` for such invalid paired percentages, -while leaving the CSV and functional status unchanged. +The ablation report uses each disabled feature as its comparison baseline. +Planning gain uses `no-plan`; throughput synchronization gain uses `no-sync`. +Throughput gain measures the relative increase. Latency, power, and energy gains +measure relative reductions. Both programs' functional statuses remain separate. ## Report reuse and implementation revisions @@ -58,6 +74,12 @@ Metrics come from Pimsim's [`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp) and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp). +Pimsim-nn omits `vsoftmax` before instruction fetch. Reported performance and +energy exclude its work. The PIM queue estimator uses the same omission. +Functional validation executes Softmax and compares the same full model with +its native reference. This timing approximation does not establish hardware +support for Softmax or include its execution energy. + - Latency mode measures time to complete one inference. - Throughput-mode latency is measurement duration / completed iterations, equal to `1000 / throughput` before rounding. It is not end-to-end pipelined @@ -67,9 +89,16 @@ and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCou mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size division is applied. +Existing `average_energy_pj`, `raptor_energy_pj`, and validation CSV energy +columns retain their names for compatibility. They contain energy per inference. + Fast mode measures throughput over the last two convergence windows but power and energy over the entire run up to early stop. Energy includes startup and work in flight, so it need not equal average power / reported throughput. +If convergence fails, comparison and ablation reports retain the simulator's +full-duration fallback throughput. This measurement includes transient execution. +It does not establish steady-state convergence. The per-case JSON records its +measurement type and convergence status. Missing or invalid measurements remain `NA`. Use `--no-fast` for full-duration measurements. Consult the [correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows as performance comparisons. diff --git a/docs/evaluation/pimcomp-reproducibility.md b/docs/evaluation/pimcomp-reproducibility.md index a70d77f..55353a4 100644 --- a/docs/evaluation/pimcomp-reproducibility.md +++ b/docs/evaluation/pimcomp-reproducibility.md @@ -43,10 +43,13 @@ shows why a fusion label is not proof of numeric equivalence. Pimsim accepts ISA instructions rather than ONNX operators. Its [instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp) -rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must -not be approximated in the oracle. This is why the dedicated GoogLeNet artifact -removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other -operations, including MatMul/Transpose and detection decoding, for Raptor. +omits `vsoftmax` (binary opcode 21) before fetch and warns once per process +because it has no timing model. Other unknown opcodes remain errors. +Reported timing and energy exclude Softmax work. Functional validation still +executes Softmax. The comparison uses full YOLO and GoogLeNet with only its two +LRN nodes bypassed. VGG8 and both ResNet graphs retain all operators. +The historical `*-pimsim-nn.onnx` files retain the earlier Softmax-free graphs +for result reproduction; they are no longer the suite's selected inputs. ## Standalone frontend/backend reproduction diff --git a/src/PIM/Compiler/PimPeriodicAnalysis.cpp b/src/PIM/Compiler/PimPeriodicAnalysis.cpp index e14c843..a661fc0 100644 --- a/src/PIM/Compiler/PimPeriodicAnalysis.cpp +++ b/src/PIM/Compiler/PimPeriodicAnalysis.cpp @@ -35,7 +35,7 @@ struct Notice { size_t arrival; int64_t source; }; struct Wait { size_t end; int64_t expected; }; struct Core { size_t entry, tail, firstNetwork, lastNetwork; - Cost pending = 0, work = 0, instructions = 0; + Cost pending = 0, work = 0, instructions = 0, directWork = 0, constantWork = 0; std::array service{}; std::array traceWork{}; llvm::SHA256 requests{}, blockingOrder{}; @@ -80,10 +80,13 @@ public: stateWords = state.size(); return true; } + // Snapshots are immutable; do not retain geometric construction capacity. + state.shrink_to_fit(); + generations.shrink_to_fit(); size_t added = add(mul(add(state.capacity(), generations.capacity()), sizeof(uint64_t)), sizeof(Snapshot)); - if (add(bytes, added) > 16 * 1024 * 1024) { + if (add(bytes, added) > 64 * 1024 * 1024) { exhausted = true; - evidence.reason = "periodic-state history exceeds 16 MiB"; + evidence.reason = "periodic-state history exceeds 64 MiB"; return false; } bytes += added; @@ -141,7 +144,7 @@ PimQueuePolicyResult estimateQueuedPeriod(const std::vector &events, pending.push({sub(time, service[node]), node, memoryAccess[node]}); }; for (size_t node = 0; node < nodes.size(); ++node) schedule(node); - const size_t maximumRounds = verifyState ? 256 : 64; + const size_t maximumRounds = verifyState ? 1024 : 64; std::vector roundEnd(maximumRounds); std::vector roundCount(maximumRounds); std::map> queueWait; @@ -462,7 +465,7 @@ private: // ROB retirement needs another edge; matrix also has AG/commit registers. return issue ? mul(matrix ? 3 : 1, target.processorPeriodNs) : 0; } - void memoryAccess(int64_t core, uint32_t pc, bool write, Cost bytes); + void memoryAccess(int64_t core, uint32_t pc, bool write, Cost bytes, bool constant = false); bool communication(int64_t core, uint32_t pc, Opcode op, int32_t peer, int32_t reg, int32_t count, Cost bytes); }; @@ -518,14 +521,17 @@ bool Analysis::communication(int64_t core, uint32_t pc, Opcode op, target.getLinkServiceNs(target.controlMessageBytes, core, peer, arithmetic)); receives[{peer, core}].push_back({complete, permission.arrival, permission.begin, bytes}); } - cores.at(core).work = add(cores.at(core).work, add(local(bytes, op == Opcode::recv), retirement())); - cores.at(core).service[2] = add(cores.at(core).service[2], add(local(bytes, op == Opcode::recv), retirement())); - opcodeService[static_cast(op)] = add(opcodeService[static_cast(op)], add(local(bytes, op == Opcode::recv), retirement())); + Cost endpointWork = add(local(bytes, op == Opcode::recv), retirement()); + auto &state = cores.at(core); + state.work = add(state.work, endpointWork); + state.service[2] = add(state.service[2], endpointWork); + state.directWork = add(state.directWork, endpointWork); + opcodeService[static_cast(op)] = add(opcodeService[static_cast(op)], endpointWork); } return true; } -void Analysis::memoryAccess(int64_t core, uint32_t pc, bool write, Cost bytes) { +void Analysis::memoryAccess(int64_t core, uint32_t pc, bool write, Cost bytes, bool constant) { Core &state = cores.at(core); recordBlocking(core, write ? Opcode::st : Opcode::ld, bytes); if (options.workBreakdown) { @@ -548,6 +554,7 @@ void Analysis::memoryAccess(int64_t core, uint32_t pc, bool write, Cost bytes) { Cost cost = add(add(add(write ? local(bytes, false) : 0, demand.serviceNs), completion), add(target.getSharedMemoryLinkServiceNs(write ? bytes : target.controlMessageBytes, core, false, arithmetic), target.getSharedMemoryLinkServiceNs(!write && target.sharedMemoryReadReturnsPayload ? bytes : 0, core, true, arithmetic))); + if (constant && !write) state.constantWork = add(state.constantWork, cost); state.service[write ? 4 : 3] = add(state.service[write ? 4 : 3], cost); opcodeService[static_cast(write ? Opcode::st : Opcode::ld)] = add(opcodeService[static_cast(write ? Opcode::st : Opcode::ld)], cost); if (memoryQueue) { @@ -618,6 +625,8 @@ bool Analysis::readCore(int64_t core, llvm::StringRef data, llvm::ArrayRef(op)] = add(opcodeService[static_cast(op)], cost); continue; } - if (length < 0 || op > Opcode::sync || op == Opcode::vsoftmax) { + if (length < 0 || op > Opcode::sync) { error = "unsupported instruction opcode or negative transfer length"; return false; } Cost bytes = length; @@ -781,7 +790,7 @@ bool Analysis::readExecution(int64_t core, const std::vector } index = found->second.end - 1; } else if (item.kind == PimExecutionEvent::Load || item.kind == PimExecutionEvent::Store) { - memoryAccess(core, position, item.kind == PimExecutionEvent::Store, item.amount); + memoryAccess(core, position, item.kind == PimExecutionEvent::Store, item.amount, item.constant); } else { for (int64_t value : {item.peer, item.reg, item.count}) if (value < 0 || value > std::numeric_limits::max()) { @@ -839,7 +848,7 @@ EvaluatedAnalysis Analysis::evaluate() { if (cores.empty()) result.error = "artifact has no active repeating instruction stream"; else if (!connect()) result.error = error; if (!result.error.empty()) { result.cost.reason = result.error; return result; } - if (options.detailed) { + if (options.detailed || options.searchGuidance) { auto recurrence = analyzePeriodicDependencies(events.size(), edges, result.error); if (mlir::failed(recurrence)) { result.cost.reason = result.error; return result; } result.recurrence = std::move(*recurrence); @@ -859,6 +868,23 @@ EvaluatedAnalysis Analysis::evaluate() { std::chrono::steady_clock::now() - start).count(); result.cost = queue.cost(); } + if (options.searchGuidance) { + result.cost.processorTaskWorkNs.resize(target.processorCount); + result.cost.processorCriticalDelayNs.resize(target.processorCount); + for (const auto &[core, state] : cores) + result.cost.processorTaskWorkNs[core] = add(state.constantWork, + add(add(state.service[0], state.service[1]), + add(sub(state.service[2], state.directWork), state.service[5]))); + Cost distance = 0; + for (size_t index : result.recurrence->criticalCycle) + distance = add(distance, edges[index].distance); + if (distance) + for (size_t index : result.recurrence->criticalCycle) { + const auto &edge = edges[index]; + result.cost.processorCriticalDelayNs[events[edge.source].core] + += static_cast(edge.delayNs) / distance; + } + } return result; } diff --git a/src/PIM/Compiler/PimPeriodicAnalysis.hpp b/src/PIM/Compiler/PimPeriodicAnalysis.hpp index 4838f79..38fb528 100644 --- a/src/PIM/Compiler/PimPeriodicAnalysis.hpp +++ b/src/PIM/Compiler/PimPeriodicAnalysis.hpp @@ -89,8 +89,11 @@ struct PimPeriodicOptions { bool instructionIssue = true; bool scalarIssue = true; bool ignoreLocalCopies = false; + // Spatial ablation mirrors codegen's omission of SYNC/WAIT instructions. + bool omitSynchronization = false; bool workBreakdown = false; bool measureTime = false; + bool searchGuidance = false; // Optional ordered work/IO extraction for timing validation; no execution. PimExecutionProgram *executionTrace = nullptr; // Prove recurrence of the modeled state. Instruction diagnostics retain diff --git a/src/PIM/Compiler/PimPipelineSelection.cpp b/src/PIM/Compiler/PimPipelineSelection.cpp index 408e580..49985cd 100644 --- a/src/PIM/Compiler/PimPipelineSelection.cpp +++ b/src/PIM/Compiler/PimPipelineSelection.cpp @@ -1,6 +1,8 @@ #include "PimPipelineSelection.hpp" #include "PimCompilerOptions.hpp" #include "PimCompilerUtils.hpp" +#include "PimSpatialCost.hpp" +#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp" #include "src/Accelerators/PIM/Common/Support/DebugDump.hpp" #include "src/Accelerators/PIM/Common/Support/FileSystemUtils.hpp" #include "src/Accelerators/PIM/Common/Support/ReportUtils.hpp" @@ -108,83 +110,133 @@ struct PimPipelineSelectionPass final : PassWrapper &costs, + PreparedSpatialSchedules &prepared, std::vector &costs, llvm::json::Object &report) { llvm::json::Array summaries; - double realization = 0, extraction = 0, expansion = 0, queue = 0; - std::string errors; - for (size_t i = 0; i < costs.size(); ++i) { - llvm::json::Object summary{{"index", static_cast(i)}}; + double realization = 0, extraction = 0, expansion = 0, queue = 0, evaluation = 0; + auto evaluate = [&](const MergeScheduleResult &schedule) { + size_t index = summaries.size(); + llvm::json::Object summary{{"index", static_cast(index)}}; OwningOpRef candidate; + WholeScheduleCost cost; std::string scheduleReport; { ScopedCompilerOutputSuppression silence; - if (failed(realizeCandidate(checkpoint, prepared.candidates[i], candidate, scheduleReport, summary))) - costs[i].reason = "Spatial schedule realization failed verification"; - else costs[i] = spatialCost(*candidate, summary); + if (failed(realizeCandidate(checkpoint, schedule, candidate, scheduleReport, summary))) + cost.reason = "Spatial schedule realization failed verification"; + else cost = spatialCost(*candidate, summary); } realization += summary.getNumber("realization_seconds").value_or(0); - extraction += costs[i].timing.extractionSeconds; - expansion += costs[i].timing.expansionSeconds; - queue += costs[i].timing.evaluationSeconds; - bool supported = costs[i].loweringSupport == StaticLoweringSupport::Supported; - auto convergence = costs[i].convergenceProved(); - auto support = costs[i].loweringSupported(); - bool converged = convergence == true; - bool safe = costs[i].eligible(); - summary["cost_available"] = costs[i].available(); + evaluation += summary.getNumber("spatial_cost_seconds").value_or(0); + extraction += cost.timing.extractionSeconds; + expansion += cost.timing.expansionSeconds; + queue += cost.timing.evaluationSeconds; + auto convergence = cost.convergenceProved(), support = cost.loweringSupported(); + summary["cost_available"] = cost.available(); summary["convergence_proved"] = convergence ? llvm::json::Value(*convergence) : llvm::json::Value(nullptr); summary["lowering_supported"] = support ? llvm::json::Value(*support) : llvm::json::Value(nullptr); - summary["eligible"] = safe; - if (costs[i].primary) summary["interval_ns"] = *costs[i].primary; - if (costs[i].alternative) summary["alternative_tie_interval_ns"] = *costs[i].alternative; - if (!safe) { - std::string property; - if (!costs[i].available()) property += "unavailable timing; "; - if (!supported) property += "unsupported or unknown static lowering; "; - if (!converged) property += "unproved modeled-state convergence; "; - if (property.empty()) property = "unavailable selection margin"; - summary["reason"] = property + ": " + costs[i].reason; - errors += "\ncandidate " + std::to_string(i) + ": " + property + ": " + costs[i].reason; + summary["eligible"] = cost.eligible(); + if (cost.primary) summary["interval_ns"] = *cost.primary; + if (cost.alternative) summary["alternative_tie_interval_ns"] = *cost.alternative; + llvm::json::Array budgets, witness; + std::vector cores(stages); + std::vector pressure(stages); + for (size_t core = 0; core < schedule.processorStages.size(); ++core) { + size_t stage = schedule.processorStages[core]; + ++cores[stage]; + if (core < cost.processorCriticalDelayNs.size()) + pressure[stage] += cost.processorCriticalDelayNs[core]; } + for (size_t stage = 0; stage < stages; ++stage) { + budgets.push_back(static_cast(cores[stage])); + witness.push_back(pressure[stage]); + } + summary["stage_core_budgets"] = std::move(budgets); + summary["recurrence_stage_contributions_ns"] = std::move(witness); + llvm::json::Array runs; + for (size_t begin = 0; begin < prepared.graph.nodes.size();) { + auto stageFor = [&](size_t task) { + return schedule.processorStages[schedule.computeToCpuMap.lookup( + prepared.graph.nodes[task].instance)]; + }; + size_t end = begin + 1, stage = stageFor(begin); + while (end < prepared.graph.nodes.size() && stageFor(end) == stage) ++end; + runs.push_back(llvm::json::Array{static_cast(begin), + static_cast(end - begin), static_cast(stage)}); + begin = end; + } + summary["task_stage_runs"] = std::move(runs); + if (!cost.eligible()) summary["reason"] = cost.reason; summaries.push_back(std::move(summary)); - } - report["candidates"] = std::move(summaries); + return cost; + }; + std::string error; + auto start = Clock::now(); + auto search = optimizePipelineSchedule(prepared.graph, prepared.candidates.front(), + stages, target, evaluate, error); + report["search_seconds"] = elapsed(start); + report["search_refinement_seconds"] = std::max(0.0, elapsed(start) - realization - evaluation); + report["cost_evaluation_seconds"] = evaluation; report["candidate_realization_seconds"] = realization; report["event_extraction_seconds"] = extraction; report["event_expansion_seconds"] = expansion; report["queue_seconds"] = queue; - if (!errors.empty()) { + report["candidates"] = std::move(summaries); + if (failed(search)) { report["status"] = "incomplete-costs"; - report["reason"] = errors; - checkpoint.emitError("Spatial pipeline cannot compare the complete candidate set:") << errors; + report["reason"] = error; + checkpoint.emitError("Spatial pipeline cannot compare the complete candidate set: ") << error; return failure(); } + prepared.candidates = std::move(search->schedules); + costs = std::move(search->costs); + bool single = costs.empty(); + if (single) { + report["candidates"] = llvm::json::Array{llvm::json::Object{{"index", 0}}}; + } else { + auto *entries = report.getArray("candidates"); + for (size_t i = 0; i < entries->size(); ++i) + (*entries)[i].getAsObject()->insert({"parent", static_cast(search->parents[i])}); + } + report["generated_candidates"] = static_cast(prepared.candidates.size()); + report["single_candidate_analysis_skipped"] = single; + report["search_rounds"] = static_cast(search->rounds); + report["packing_attempts"] = static_cast(search->packingAttempts); + report["considered_assignments"] = static_cast(search->consideredAssignments); + report["search_termination"] = search->termination; + report["queue_evaluation_budget"] = static_cast(kPipelineQueueEvaluationBudget); + report["beam_width"] = static_cast(kPipelineBeamWidth); return success(); } void runOnOperation() override { ModuleOp checkpoint = getOperation(); auto start = Clock::now(); - auto prepared = prepareSpatialSchedules(checkpoint, target, stages, /*collectCandidates=*/true); + auto prepared = prepareSpatialSchedules(checkpoint, target, stages); if (failed(prepared)) { signalPassFailure(); return; } if (prepared->candidates.empty()) { checkpoint.emitError("Spatial pipeline requires at least one candidate schedule"); signalPassFailure(); return; } - bool single = prepared->candidates.size() == 1; llvm::json::Object report{{"strategy", "spatial-queue-aware"}, {"static_analysis", true}, {"candidate_generation_seconds", elapsed(start)}, {"generated_candidates", static_cast(prepared->candidates.size())}, - {"single_candidate_analysis_skipped", single}, {"selection_margin", kPipelineSelectionMargin}, + {"selection_margin", kPipelineSelectionMargin}, {"prediction_error_bound_available", false}, {"temporary_disk_bytes", 0}}; - std::vector costs(prepared->candidates.size()); - if (!single && failed(evaluateSpatialCandidates(checkpoint, *prepared, costs, report))) { + if (pimDisableSynchronization) { + report["synchronization_ablation"] = true; + report["cost_model_program"] = "synchronized-reference"; + llvm::errs() << "Synchronization ablation: keep the normal selected schedule; " + "report zero-cost SYNC/WAIT separately and omit them in code generation. " + "Buffer reuse and restart correctness are not guaranteed.\n"; + } + std::vector costs; + if (failed(evaluateSpatialCandidates(checkpoint, *prepared, costs, report))) { writeSelectionReport(std::move(report)); signalPassFailure(); return; } - if (single) report["candidates"] = llvm::json::Array{llvm::json::Object{{"index", 0}}}; + bool single = costs.empty(); OwningOpRef winner; std::string scheduleReport; llvm::json::Array attempts; @@ -204,6 +256,8 @@ struct PimPipelineSelectionPass final : PassWrappercandidates.size() << "; " << reason << "\n"; writeSelectionReport(std::move(report)); } diff --git a/src/PIM/Compiler/PimSpatialCost.cpp b/src/PIM/Compiler/PimSpatialCost.cpp index 80ceab6..1b6073f 100644 --- a/src/PIM/Compiler/PimSpatialCost.cpp +++ b/src/PIM/Compiler/PimSpatialCost.cpp @@ -47,13 +47,15 @@ class Extraction { public: const PimInstructionTiming ⌖ bool diagnostics; + bool omitSynchronization; PimExecutionProgram program; std::string error, loweringReason; StaticLoweringSupport loweringSupport = StaticLoweringSupport::Supported; std::map gaps; int64_t visited = 0, constantLoads = 0; - explicit Extraction(const PimInstructionTiming &target, bool diagnostics) - : target(target), diagnostics(diagnostics), program(target.processorCount) {} + explicit Extraction(const PimInstructionTiming &target, bool diagnostics, bool omitSynchronization) + : target(target), diagnostics(diagnostics), omitSynchronization(omitSynchronization), + program(target.processorCount) {} bool build(ModuleOp module); private: llvm::SmallDenseSet assignedCores; @@ -135,7 +137,7 @@ Residence Extraction::storage(Value value) { return Residence::Host; } if (isa_and_nonnull(value.getDefiningOp())) @@ -246,10 +248,13 @@ bool Extraction::operation(Operation &op, Sequence &sequence) { return *result; }; if (auto sync = dyn_cast(op)) { - sequence.push_back({Event::Signal, 0, index(sync.getTargetCoreId()), index(sync.getEventRegister())}); + if (!omitSynchronization) + sequence.push_back({Event::Signal, 0, index(sync.getTargetCoreId()), index(sync.getEventRegister())}); } else if (auto wait = dyn_cast(op)) { - int64_t count = index(wait.getWaitValue()); - if (count) sequence.push_back({Event::Wait, 0, 0, index(wait.getEventRegister()), count}); + if (!omitSynchronization) { + int64_t count = index(wait.getWaitValue()); + if (count) sequence.push_back({Event::Wait, 0, 0, index(wait.getEventRegister()), count}); + } } else if (auto send = dyn_cast(op)) { materialize(send.getInput(), sequence); sequence.push_back({Event::Send, bytes(send.getInput().getType()), index(send.getTargetCoreId())}); @@ -261,10 +266,13 @@ bool Extraction::operation(Operation &op, Sequence &sequence) { sequence.push_back({Event::Repeat, static_cast(plan->second.getRunCount()), 0, 0, 0, {{Event::Store, ceil(mul(plan->second.getRunElements(), target.computeBitwidth), 8)}}}); else sequence.push_back({Event::Store, bytes(store.getInput().getType())}); - sequence.push_back({Event::Signal, 0, index(store.getTargetCoreId()), index(store.getEventRegister())}); + if (!omitSynchronization) + sequence.push_back({Event::Signal, 0, index(store.getTargetCoreId()), index(store.getEventRegister())}); } else if (auto load = dyn_cast(op)) { - int64_t count = index(load.getWaitValue()); - if (count) sequence.push_back({Event::Wait, 0, 0, index(load.getEventRegister()), count}); + if (!omitSynchronization) { + int64_t count = index(load.getWaitValue()); + if (count) sequence.push_back({Event::Wait, 0, 0, index(load.getEventRegister()), count}); + } sequence.push_back({Event::Load, bytes(load.getOutput().getType())}); } else if (auto vmm = dyn_cast(op)) { materialize(vmm.getInput(), sequence); @@ -284,6 +292,9 @@ bool Extraction::operation(Operation &op, Sequence &sequence) { pim_binary::kMatrixWeightPrecisionBits, weight.getDimSize(1) / target.matrixColumns, ceil(target.computeBitwidth, 8)); work(sequence, getPimMatrixWorkNs(target, target.computeBitwidth, output), Event::Matrix); + } else if (auto softmax = dyn_cast(op)) { + materialize(softmax.getInput(), sequence); + // Pimsim-nn omits vsoftmax itself. Keep any preceding input materialization. } else if (isa(op)) { for (Value operand : op.getOperands()) materialize(operand, sequence); @@ -464,7 +475,7 @@ PimPeriodicEstimate onnx_mlir::evaluatePimSpatialSchedule(ModuleOp module, if (!target.hasPeriodicTransferTiming() || !target.computeBitwidth) return unavailableCost("unsupported abstract timing contract"); auto start = std::chrono::steady_clock::now(); - Extraction extraction(target, options.workBreakdown); + Extraction extraction(target, options.workBreakdown, options.omitSynchronization); bool available = extraction.build(module); double extractionMs = std::chrono::duration( std::chrono::steady_clock::now() - start).count(); @@ -587,8 +598,28 @@ llvm::json::Object onnx_mlir::analyzePimSpatialSchedule(ModuleOp module, return evaluatePimSpatialSchedule(module, target, options).report; } +llvm::json::Object onnx_mlir::analyzePimSpatialSynchronizationAblation(ModuleOp module) { + auto timing = getPimInstructionTiming(); + auto contract = getPimPeriodicTimingContract(); + if (!contract) return unavailableCost(llvm::toString(contract.takeError())).report; + timing.matrixCellBits = *contract; + PimPeriodicOptions options; + options.omitSynchronization = true; + options.verifyPeriodicState = true; + options.measureTime = true; + auto result = evaluatePimSpatialSchedule(module, timing, options); + result.report["diagnostic_only"] = true; + result.report["used_for_selection"] = false; + result.report["cost_model_program"] = "synchronization-omitted"; + return std::move(result.report); +} + void onnx_mlir::configurePimScheduleCosts(spatial::SchedulingTarget &target, PimInstructionTiming timing) { + // The stage-zero entry barrier joins the previous generation's work. + target.estimatePipelineDependencySpan = [](llvm::ArrayRef spans) { + return spans.empty() ? Cost{0} : spans.front(); + }; auto contract = getPimPeriodicTimingContract(); if (contract) timing.matrixCellBits = *contract; auto hardware = std::make_shared(std::move(timing)); @@ -601,8 +632,6 @@ void onnx_mlir::configurePimScheduleCosts(spatial::SchedulingTarget &target, if (!contract) target.wholeScheduleCostUnavailable = llvm::toString(contract.takeError()); if (!hardware->hasPeriodicTransferTiming()) target.wholeScheduleCostUnavailable = "target has no supported queue timing contract"; - if (pimDisableSynchronization) - target.wholeScheduleCostUnavailable = "synchronization ablation has no periodic queue timing contract"; if (!target.wholeScheduleCostUnavailable.empty()) return; target.estimateRealizedScheduleCost = [hardware](mlir::ModuleOp module, llvm::json::Object &summary) { @@ -610,6 +639,7 @@ void onnx_mlir::configurePimScheduleCosts(spatial::SchedulingTarget &target, options.workBreakdown = pimPeriodicDiagnostics; options.measureTime = true; options.verifyPeriodicState = true; + options.searchGuidance = true; auto estimate = evaluatePimSpatialSchedule(module, *hardware, options); auto cost = std::move(estimate.cost); summary = std::move(estimate.report); diff --git a/src/PIM/Compiler/PimSpatialCost.hpp b/src/PIM/Compiler/PimSpatialCost.hpp index 5f348c7..e0c8e63 100644 --- a/src/PIM/Compiler/PimSpatialCost.hpp +++ b/src/PIM/Compiler/PimSpatialCost.hpp @@ -12,5 +12,6 @@ PimPeriodicEstimate evaluatePimSpatialSchedule(mlir::ModuleOp module, const PimInstructionTiming &target, PimPeriodicOptions options = {}); llvm::json::Object analyzePimSpatialSchedule(mlir::ModuleOp module, const PimInstructionTiming &target, PimPeriodicOptions options = {}); +llvm::json::Object analyzePimSpatialSynchronizationAblation(mlir::ModuleOp module); } // namespace onnx_mlir diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.cpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.cpp index 5ed6088..0a644c0 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.cpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.cpp @@ -65,7 +65,7 @@ struct ScheduleAndRealizeSpatialPass final signalPassFailure(); return; } - auto prepared = prepareSpatialSchedules(moduleOp, target, pipelineStages, false); + auto prepared = prepareSpatialSchedules(moduleOp, target, pipelineStages); if (failed(prepared) || failed(realizeSpatialSchedule(moduleOp, prepared->candidates.front(), target, pipelineStages, exportStage, periodicDiagnostics ? &prepared->graph : nullptr))) @@ -83,7 +83,7 @@ private: } // namespace FailureOr prepareSpatialSchedules(ModuleOp moduleOp, - const SchedulingTarget &target, size_t pipelineStages, bool collectCandidates) { + const SchedulingTarget &target, size_t pipelineStages) { if (!hasValidTarget(target) || !PipelineCoreLayout(target.processorCount, pipelineStages).isValid() || (pipelineStages > 1 && !target.synchronizationRegisterCount)) return moduleOp.emitError("Spatial scheduling requires an explicit valid target and pipeline layout"), failure(); @@ -122,7 +122,7 @@ FailureOr prepareSpatialSchedules(ModuleOp moduleOp, break; } if (succeeded(runPipelineScheduler( - scheduledGraph, schedule, pipelineStages, target, pipelineError, collectCandidates ? &prepared.candidates : nullptr))) + scheduledGraph, schedule, pipelineStages, target, pipelineError))) break; moduleOp.emitError() << pipelineError; return failure(); diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp index 11fae77..d380a0e 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp @@ -14,7 +14,7 @@ struct PreparedSpatialSchedules { // Success always includes at least one candidate, including an empty compute graph. mlir::FailureOr prepareSpatialSchedules( mlir::ModuleOp module, const SchedulingTarget &target, - size_t pipelineStages, bool collectCandidates); + size_t pipelineStages); mlir::LogicalResult realizeSpatialSchedule(mlir::ModuleOp module, const MergeScheduleResult &schedule, const SchedulingTarget &target, size_t pipelineStages, SpatialDataflowExportStage exportStage, diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp index d7a2e86..7d1d5e1 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp @@ -4,10 +4,12 @@ #include "llvm/Support/raw_ostream.h" #include +#include #include #include #include #include +#include #include #include @@ -29,9 +31,8 @@ using TaskCosts = std::vector; using namespace mlir; struct PipelineTaskModel { - PipelineTaskModel(const ComputeGraph &graph, const SchedulingTarget &target, - bool sharePublications = true) - : weights(graph), graph(graph), target(target), sharePublications(sharePublications) { + PipelineTaskModel(const ComputeGraph &graph, const SchedulingTarget &target) + : weights(graph), target(target) { llvm::DenseMap transposeAdjustments; schedulingCosts.reserve(graph.nodes.size()); executionCosts.reserve(graph.nodes.size()); @@ -51,104 +52,13 @@ struct PipelineTaskModel { executionCosts.push_back(node.cost); } } - // Volume priorities preserve the calibrated pipeline heuristic; timing uses executionCosts. + // Placement priorities and target execution demand have different units. TaskCosts schedulingCosts; TaskCosts executionCosts; IndexedResidentWeights weights; - const ComputeGraph &graph; const SchedulingTarget ⌖ - bool sharePublications; }; -static Cost estimateOrderedPackCost( - const TaskCosts &taskCosts, - const std::vector &taskWeightIds, size_t weightCount, - const TaskList &tasks, size_t residentWeightCapacity, - size_t maximumPacks, TaskList *taskPacks = nullptr) { - Cost low = 0; - Cost high = 0; - for (size_t task : tasks) { - low = std::max(low, taskCosts[task]); - high = checkedAdd(high, taskCosts[task]); - } - low = std::max( - low, high / maximumPacks + (high % maximumPacks != 0)); - - std::vector seen(weightCount); - size_t generation = 0; - auto pack = [&](Cost maximumCost, TaskList *assignment) { - size_t packs = 1; - Cost cost = 0; - size_t packWeightCount = 0; - ++generation; - bool fits = true; - bool packEmpty = true; - for (size_t task : tasks) { - size_t addedWeights = 0; - for (size_t weight : taskWeightIds[task]) - addedWeights += seen[weight] != generation; - Cost taskCost = taskCosts[task]; - bool startsNewPack = !packEmpty - && (cost > maximumCost - taskCost - || packWeightCount + addedWeights > residentWeightCapacity); - if (startsNewPack) { - if (++packs > maximumPacks) { - fits = false; - break; - } - cost = 0; - packWeightCount = 0; - ++generation; - packEmpty = true; - } - cost = checkedAdd(cost, taskCost); - for (size_t weight : taskWeightIds[task]) - if (seen[weight] != generation) { - seen[weight] = generation; - ++packWeightCount; - } - if (assignment) - assignment->push_back(packs - 1); - packEmpty = false; - } - return fits; - }; - while (low < high) { - Cost middle = low + (high - low) / 2; - if (pack(middle, nullptr)) - high = middle; - else - low = middle + 1; - } - if (taskPacks) { - taskPacks->clear(); - if (!pack(low, taskPacks)) - taskPacks->clear(); - } - return low; -} - -static size_t estimateOrderedPackCount( - const std::vector &taskWeightIds, const TaskList &tasks, - size_t residentWeightCapacity) { - if (tasks.empty()) - return 0; - size_t packs = 1; - TaskList weights; - for (size_t task : tasks) { - size_t unionSize = weights.size(); - for (size_t weight : taskWeightIds[task]) - unionSize += !llvm::is_contained(weights, weight); - if (!weights.empty() && unionSize > residentWeightCapacity) { - ++packs; - weights.clear(); - } - for (size_t weight : taskWeightIds[task]) - if (!llvm::is_contained(weights, weight)) - weights.push_back(weight); - } - return packs; -} struct PipelineGroup { TaskList tasks; @@ -163,26 +73,6 @@ struct PipelineStageAssignment { std::vector stageSizes; }; -static TaskCosts getPipelineStageDependencyTails( - const std::vector &groups, ArrayRef groupOrder, - ArrayRef groupStages, const TaskCosts &groupCosts, size_t stageCount) { - TaskCosts groupEnds(groups.size()); - TaskCosts stageSpans(stageCount); - TaskCosts largestTasks(stageCount); - for (size_t group : groupOrder) { - size_t stage = groupStages[group]; - groupEnds[group] = checkedAdd(groupEnds[group], groupCosts[group]); - largestTasks[stage] = std::max(largestTasks[stage], groupCosts[group]); - stageSpans[stage] = std::max(stageSpans[stage], groupEnds[group]); - for (size_t successor : groups[group].successors) - if (groupStages[successor] == stage) - groupEnds[successor] = std::max(groupEnds[successor], groupEnds[group]); - } - for (size_t stage = 0; stage < stageCount; ++stage) - stageSpans[stage] -= largestTasks[stage]; - return stageSpans; -} - static bool consumesPipelineInput(const ComputeGraphNode &node) { if (!node.instance.op) return false; @@ -194,8 +84,7 @@ struct PipelineGroupModel { std::vector groups; std::vector taskToGroup; TaskList groupOrder; - TaskCosts groupCosts; - TaskCosts groupExecutionCosts; + TaskCosts groupPriorities; size_t minimumStageZeroEnd = 0; std::vector furthestSuccessorBefore; bool hasCrossingEdge = false; @@ -287,23 +176,16 @@ static FailureOr buildPipelineGroups( return failure(); } - TaskCosts groupCosts(groups.size()); - TaskCosts groupExecutionCosts(groups.size()); - TaskCosts internalEnds(graph.nodes.size()); - TaskCosts executionEnds(graph.nodes.size()); + TaskCosts groupPriorities(groups.size()), priorityEnds(graph.nodes.size()); for (size_t group : groupOrder) for (size_t task : groups[group].tasks) { - Cost start = 0; - Cost executionStart = 0; + Cost priorityStart = 0; for (const auto &[predecessor, transferCost] : graph.predecessors[task]) if (taskToGroup[predecessor] == group) { - start = std::max(start, internalEnds[predecessor]); - executionStart = std::max(executionStart, executionEnds[predecessor]); + priorityStart = std::max(priorityStart, priorityEnds[predecessor]); } - internalEnds[task] = checkedAdd(start, model.schedulingCosts[task]); - groupCosts[group] = std::max(groupCosts[group], internalEnds[task]); - executionEnds[task] = checkedAdd(executionStart, model.executionCosts[task]); - groupExecutionCosts[group] = std::max(groupExecutionCosts[group], executionEnds[task]); + priorityEnds[task] = checkedAdd(priorityStart, model.schedulingCosts[task]); + groupPriorities[group] = std::max(groupPriorities[group], priorityEnds[task]); } std::vector position(groups.size()); @@ -330,362 +212,109 @@ static FailureOr buildPipelineGroups( } return PipelineGroupModel{std::move(groups), std::move(taskToGroup), - std::move(groupOrder), std::move(groupCosts), std::move(groupExecutionCosts), minimumStageZeroEnd, + std::move(groupOrder), std::move(groupPriorities), minimumStageZeroEnd, std::move(furthestSuccessorBefore), hasCrossingEdge}; } struct PipelinePartitionCandidate { Cost priority = 0; - Cost execution = 0; - Cost memory = 0; size_t start = 0; - size_t parent = 0; + size_t units = 0; + bool valid = false; }; -using PartitionFrontier = SmallVector; - -static void retainPartitionCandidates(PartitionFrontier &frontier, - PipelinePartitionCandidate candidate, Cost baselineMemory, bool memoryAware) { - if (!memoryAware) { - if (frontier.empty() || candidate.priority < frontier.front().priority) - frontier.assign(1, candidate); - return; - } - frontier.push_back(candidate); - size_t legacy = 0; - for (size_t i = 1; i < frontier.size(); ++i) - if (frontier[i].priority < frontier[legacy].priority) - legacy = i; - PartitionFrontier kept; - for (size_t i = 0; i < frontier.size(); ++i) { - bool dominated = false; - for (size_t j = 0; j < frontier.size() && i != legacy; ++j) - if (i != j && frontier[j].execution <= frontier[i].execution - && frontier[j].memory <= frontier[i].memory - && (frontier[j].execution < frontier[i].execution - || frontier[j].memory < frontier[i].memory || j < i)) { - dominated = true; - break; - } - if (!dominated) - kept.push_back(frontier[i]); - } - // Bound the frontier, preserving the legacy path and both resource extremes. - constexpr size_t limit = 8; - if (kept.size() > limit) { - llvm::stable_sort(kept, [](const auto &a, const auto &b) { - return std::tie(a.execution, a.memory, a.priority) - < std::tie(b.execution, b.memory, b.priority); - }); - auto score = [&](const auto &a) { - return std::make_tuple(std::max(a.execution, checkedAdd(baselineMemory, a.memory)), - a.execution, a.memory, a.priority); - }; - size_t best = 0, lowestPriority = 0, lowestMemory = 0; - for (size_t i = 1; i < kept.size(); ++i) { - if (score(kept[i]) < score(kept[best])) best = i; - if (kept[i].priority < kept[lowestPriority].priority) lowestPriority = i; - if (kept[i].memory < kept[lowestMemory].memory) lowestMemory = i; - } - SmallVector indices; - auto add = [&](size_t i) { - if (!llvm::is_contained(indices, i) && indices.size() < limit) - indices.push_back(i); - }; - for (size_t i : {size_t{0}, lowestMemory, lowestPriority, best}) add(i); - for (size_t i = 0; i < limit; ++i) add(i * (kept.size() - 1) / (limit - 1)); - for (size_t i = 0; i < kept.size(); ++i) add(i); - frontier.clear(); - for (size_t i : indices) frontier.push_back(kept[i]); - } else { - frontier = std::move(kept); - } -} - -static TaskCosts getPipelineCutMemoryCosts(const PipelineGroupModel &groups, - const PipelineTaskModel &model) { - size_t count = groups.groups.size(); - std::vector position(count); - for (auto [index, group] : llvm::enumerate(groups.groupOrder)) position[group] = index; - TaskCosts starts(count + 2), ends(count + 2), costs(count + 1); - struct PublicationRange { size_t source, target; Cost service; }; - llvm::DenseMap publications; - for (const PublicationTransfer &transfer : model.graph.publicationTransfers) { - size_t source = position[groups.taskToGroup[transfer.source]]; - size_t target = position[groups.taskToGroup[transfer.target]]; - if (source == target) continue; - assert(source < target); - Cost cost = model.target.getSharedMemoryDemand( - transfer.bytes, transfer.count, false).serviceNs; - if (auto identity = model.sharePublications ? transfer.sharedIdentity() : std::nullopt) { - auto [entry, inserted] = publications.try_emplace(*identity, - PublicationRange{source, target, - transfer.getStoreMemoryDemand(model.target, 1).serviceNs}); - entry->second.target = std::max(entry->second.target, target); - } else { - cost = checkedAdd(cost, model.sharePublications - ? transfer.getStoreMemoryDemand(model.target, transfer.count).serviceNs - : model.target.getSharedMemoryDemand(transfer.bytes, transfer.count, true).serviceNs); - } - starts[source + 1] = checkedAdd(starts[source + 1], cost); - ends[target + 1] = checkedAdd(ends[target + 1], cost); - } - for (const auto &[identity, range] : publications) { - starts[range.source + 1] = checkedAdd(starts[range.source + 1], range.service); - ends[range.target + 1] = checkedAdd(ends[range.target + 1], range.service); - } - Cost active = 0; - for (size_t cut = 0; cut <= count; ++cut) { - active = checkedAdd(active, starts[cut]); - assert(active >= ends[cut]); - active -= ends[cut]; - costs[cut] = active; - } - return costs; -} - -static FailureOr>> partitionPipelineGroups( +static FailureOr initialPipelinePartition( const PipelineGroupModel &constraints, const PipelineTaskModel &model, - ArrayRef coreCounts, size_t residentWeightCapacity, - std::string &error, bool memoryAware) { - const auto &groups = constraints.groups; - const auto &groupOrder = constraints.groupOrder; - const auto &taskWeightIds = model.weights.taskWeightIds; - size_t count = groups.size(); - const Cost infinity = std::numeric_limits::max(); - std::vector> best(coreCounts.size() + 1, - std::vector(count + 1)); - using SegmentCost = std::pair; - std::vector stageCostCache(coreCounts.size()); - std::vector cachedCoreCounts; - std::vector>> caches; - for (auto [stage, coreCount] : llvm::enumerate(coreCounts)) { - auto cached = llvm::find(cachedCoreCounts, coreCount); - if (cached == cachedCoreCounts.end()) { - stageCostCache[stage] = caches.size(); - cachedCoreCounts.push_back(coreCount); - caches.emplace_back(count + 1, std::vector(count + 1, {infinity, infinity})); - } else { - stageCostCache[stage] = cached - cachedCoreCounts.begin(); - } - } - TaskCosts cutMemory = getPipelineCutMemoryCosts(constraints, model); - Cost baselineMemory = model.graph.baselineMemory.serviceNs; - best[0][0].push_back({}); - for (size_t stage = 0; stage < coreCounts.size(); ++stage) { - size_t coreCount = coreCounts[stage]; - size_t capacity = checkedMultiply(coreCount, residentWeightCapacity); - for (size_t start = 0; start < count; ++start) { - if (best[stage][start].empty()) continue; - TaskList segmentTasks; - TaskCosts priorityEnds(count), executionEnds(count); - Cost prioritySpan = 0, executionSpan = 0, largestPriority = 0, largestExecution = 0; - llvm::SmallBitVector weights(model.weights.weightCount); - size_t weightCount = 0; - for (size_t end = start + 1; end <= count; ++end) { - size_t index = groupOrder[end - 1]; - const PipelineGroup &group = groups[index]; - priorityEnds[index] = checkedAdd(priorityEnds[index], constraints.groupCosts[index]); - executionEnds[index] = checkedAdd(executionEnds[index], constraints.groupExecutionCosts[index]); - prioritySpan = std::max(prioritySpan, priorityEnds[index]); - executionSpan = std::max(executionSpan, executionEnds[index]); - largestPriority = std::max(largestPriority, constraints.groupCosts[index]); - largestExecution = std::max(largestExecution, constraints.groupExecutionCosts[index]); - for (size_t successor : group.successors) { - priorityEnds[successor] = std::max(priorityEnds[successor], priorityEnds[index]); - executionEnds[successor] = std::max(executionEnds[successor], executionEnds[index]); - } - llvm::append_range(segmentTasks, group.tasks); - for (size_t weight : group.weightIds) - if (!weights.test(weight)) { weights.set(weight); ++weightCount; } - if (weightCount > capacity) break; - if (stage == 0 && end < constraints.minimumStageZeroEnd) continue; - if (stage != 0 && constraints.hasCrossingEdge - && constraints.furthestSuccessorBefore[start] >= end) continue; - auto &[priority, execution] = caches[stageCostCache[stage]][start][end]; - if (priority == infinity) { - priority = checkedAdd(estimateOrderedPackCost(model.schedulingCosts, - taskWeightIds, model.weights.weightCount, segmentTasks, - residentWeightCapacity, coreCount), prioritySpan - largestPriority); - execution = memoryAware ? checkedAdd(estimateOrderedPackCost(model.executionCosts, - taskWeightIds, model.weights.weightCount, segmentTasks, - residentWeightCapacity, coreCount), executionSpan - largestExecution) : priority; - } - for (auto [parent, previous] : llvm::enumerate(best[stage][start])) - retainPartitionCandidates(best[stage + 1][end], { - std::max(previous.priority, priority), std::max(previous.execution, execution), - checkedAdd(previous.memory, cutMemory[start]), start, parent}, - baselineMemory, memoryAware); - } - } - } - using Terminal = std::pair; - std::vector terminals; - auto candidate = [&](Terminal terminal) -> const PipelinePartitionCandidate & { - return best[terminal.first][count][terminal.second]; - }; - auto score = [&](Terminal terminal) { - const auto &a = candidate(terminal); - return memoryAware ? std::make_tuple(std::max(a.execution, checkedAdd(baselineMemory, a.memory)), - a.execution, a.memory) : std::make_tuple(a.priority, Cost{0}, Cost{0}); - }; - for (size_t stages = 1; stages <= coreCounts.size(); ++stages) - for (size_t index = 0; index < best[stages][count].size(); ++index) - terminals.emplace_back(stages, index); - if (terminals.empty()) { - error = "pipeline scheduling cannot split operations into dependency-adjacent " - "stages within the physical crossbar limit"; - return failure(); - } - llvm::sort(terminals, [&](Terminal a, Terminal b) { - return score(a) != score(b) ? score(a) < score(b) : a > b; - }); - SmallVector selected{terminals.front()}; - auto add = [&](Terminal terminal) { - if (!llvm::is_contained(selected, terminal) && selected.size() < 4) - selected.push_back(terminal); - }; - if (memoryAware) { - for (unsigned resource = 0; resource < 3; ++resource) { - auto resourceScore = [&](Terminal terminal) { - const auto &a = candidate(terminal); - return resource == 0 ? std::make_tuple(a.execution, a.memory, a.priority) - : resource == 1 ? std::make_tuple(a.memory, a.execution, a.priority) - : std::make_tuple(a.priority, a.execution, a.memory); - }; - add(*llvm::min_element(terminals, [&](Terminal a, Terminal b) { - return resourceScore(a) < resourceScore(b); - })); - } - for (Terminal terminal : terminals) add(terminal); - } - std::vector> partitions; - for (auto [stages, index] : selected) { - std::vector groupStages(count); - size_t end = count; - for (size_t stage = stages; stage > 0; --stage) { - auto entry = best[stage][end][index]; - for (size_t position = entry.start; position < end; ++position) - groupStages[groupOrder[position]] = stage - 1; - end = entry.start; - index = entry.parent; - } - if (!llvm::is_contained(partitions, groupStages)) - partitions.push_back(std::move(groupStages)); - } - return partitions; -} - -static FailureOr> assignPipelineStages( - const ComputeGraph &graph, const PipelineTaskModel &model, const PipelineCoreLayout &layout, size_t residentWeightCapacity, - std::string &error, bool memoryAware) { - if (graph.nodes.empty()) - return std::vector{{ - {}, std::vector(layout.getStageSizes())}}; - auto constraints = buildPipelineGroups(graph, model, error); - if (failed(constraints)) - return failure(); - const auto &groups = constraints->groups; - const auto &groupOrder = constraints->groupOrder; - const auto &taskToGroup = constraints->taskToGroup; - const auto &taskWeightIds = model.weights.taskWeightIds; - auto partitionGroups = [&](ArrayRef coreCounts) { - return partitionPipelineGroups(*constraints, model, coreCounts, - residentWeightCapacity, error, memoryAware); - }; - auto initialPartitions = partitionGroups(layout.getStageSizes()); - if (failed(initialPartitions)) - return failure(); - const auto &initialGroupStages = initialPartitions->front(); - - std::vector tasksByStage(layout.getStageCount()); - for (size_t group : groupOrder) - llvm::append_range( - tasksByStage[initialGroupStages[group]], groups[group].tasks); - std::vector stageSizes(layout.getStageCount(), 1); - size_t assignedCores = stageSizes.size(); - for (size_t stage = 0; stage < stageSizes.size(); ++stage) { - for (size_t task : tasksByStage[stage]) - if (taskWeightIds[task].size() > residentWeightCapacity) { - error = "pipeline scheduling cannot fit one compute instance in a " - "physical core's crossbars"; - return failure(); - } - stageSizes[stage] = std::max( - stageSizes[stage], estimateOrderedPackCount( - taskWeightIds, tasksByStage[stage], - residentWeightCapacity)); - assignedCores += stageSizes[stage] - 1; - } - if (assignedCores > layout.getProcessorCount()) { - assignedCores = 0; - for (size_t stage = 0; stage < stageSizes.size(); ++stage) { - stageSizes[stage] = std::max(1, - model.weights.getPackCountLowerBound(tasksByStage[stage], residentWeightCapacity)); - assignedCores += stageSizes[stage]; - } - if (assignedCores > layout.getProcessorCount()) { - error = "selected pipeline partition exceeds physical capacity even " - "at the resident-weight lower bound"; - return failure(); - } - } - - TaskCosts stageDependencyTails = getPipelineStageDependencyTails( - groups, groupOrder, initialGroupStages, - constraints->groupCosts, stageSizes.size()); - auto getStageCost = [&](size_t stage, size_t coreCount) { - Cost schedulingCost = estimateOrderedPackCost( - model.schedulingCosts, taskWeightIds, model.weights.weightCount, - tasksByStage[stage], residentWeightCapacity, coreCount); - schedulingCost = checkedAdd(schedulingCost, stageDependencyTails[stage]); - return schedulingCost; - }; - std::vector stageCosts(stageSizes.size()); - std::vector nextStageCosts(stageSizes.size()); - TaskCosts stageWork(stageSizes.size()); - for (size_t stage = 0; stage < stageSizes.size(); ++stage) - for (size_t task : tasksByStage[stage]) - stageWork[stage] = checkedAdd(stageWork[stage], model.schedulingCosts[task]); - for (size_t stage = 0; stage < stageSizes.size(); ++stage) - stageCosts[stage] = getStageCost(stage, stageSizes[stage]); - for (size_t stage = 0; stage < stageSizes.size(); ++stage) - nextStageCosts[stage] = getStageCost(stage, stageSizes[stage] + 1); - bool canPlaceTasksSeparately = graph.nodes.size() <= layout.getProcessorCount(); - while (assignedCores < layout.getProcessorCount()) { - size_t bestStage = stageSizes.size(); - std::optional> bestScore; - for (size_t stage = 0; stage < stageSizes.size(); ++stage) { - Cost benefit = stageCosts[stage] - nextStageCosts[stage]; - Cost workPerCore = stageWork[stage] / stageSizes[stage]; - auto score = std::make_tuple(benefit, - benefit == 0 && canPlaceTasksSeparately ? workPerCore : 0, stageCosts[stage], - layout.getProcessorCount() - stageSizes[stage]); - if (!bestScore || score > *bestScore) { - bestStage = stage; - bestScore = score; + std::string &error) { + size_t count = constraints.groups.size(), stages = layout.getStageCount(); + if (!count) return PipelineStageAssignment{{}, + {layout.getStageSizes().begin(), layout.getStageSizes().end()}}; + size_t extra = layout.getProcessorCount() - stages; + // Resource quantization bounds seed construction only. Refinement uses + // individual physical cores and final allocation uses every available core. + size_t units = std::min(16, extra); + auto coreCount = [&](size_t amount) { return 1 + (units ? extra * amount / units : 0); }; + using Cell = PipelinePartitionCandidate; + std::vector>> best(stages + 1, + std::vector>(count + 1, std::vector(units + 1))); + const Cost infinity = std::numeric_limits::max(); + std::vector> costs(count + 1, + std::vector(count + 1, TaskCosts(units + 1, infinity))); + for (size_t start = 0; start < count; ++start) { + TaskList tasks; + TaskCosts ends(count); + Cost span = 0, largest = 0; + llvm::SmallBitVector weights(model.weights.weightCount); + for (size_t end = start + 1; end <= count; ++end) { + size_t index = constraints.groupOrder[end - 1]; + const auto &group = constraints.groups[index]; + ends[index] = checkedAdd(ends[index], constraints.groupPriorities[index]); + span = std::max(span, ends[index]); + largest = std::max(largest, constraints.groupPriorities[index]); + for (size_t successor : group.successors) + ends[successor] = std::max(ends[successor], ends[index]); + llvm::append_range(tasks, group.tasks); + for (size_t weight : group.weightIds) weights.set(weight); + for (size_t amount = 0; amount <= units; ++amount) { + size_t cores = coreCount(amount); + if (weights.count() > checkedMultiply(cores, residentWeightCapacity)) continue; + costs[start][end][amount] = checkedAdd(span - largest, + estimateOrderedPackCost(model.schedulingCosts, model.weights.taskWeightIds, + model.weights.weightCount, tasks, residentWeightCapacity, cores)); } } - ++stageSizes[bestStage]; - stageCosts[bestStage] = nextStageCosts[bestStage]; - nextStageCosts[bestStage] = - getStageCost(bestStage, stageSizes[bestStage] + 1); - ++assignedCores; } - auto refinedPartitions = partitionGroups(stageSizes); - if (failed(refinedPartitions)) + best[0][0][0].valid = true; + for (size_t stage = 0; stage < stages; ++stage) + for (size_t start = 0; start < count; ++start) + for (size_t used = 0; used <= units; ++used) { + const auto &previous = best[stage][start][used]; + if (!previous.valid) continue; + for (size_t end = start + 1; end <= count; ++end) { + if (stage == 0 && end < constraints.minimumStageZeroEnd) continue; + if (stage && constraints.hasCrossingEdge + && constraints.furthestSuccessorBefore[start] >= end) continue; + for (size_t amount = 0; used + amount <= units; ++amount) { + Cost priority = costs[start][end][amount]; + if (priority == infinity) continue; + Cell candidate{std::max(previous.priority, priority), start, amount, true}; + auto ¤t = best[stage + 1][end][used + amount]; + if (!current.valid || candidate.priority < current.priority) current = candidate; + } + } + } + size_t selected = 0; + for (size_t stage = 1; stage <= stages; ++stage) + if (best[stage][count][units].valid + && (!selected || best[stage][count][units].priority + < best[selected][count][units].priority)) + selected = stage; + if (!selected && count) { + error = "pipeline initialization found no dependency-adjacent operation partition within its bounded core-budget search"; return failure(); - if (memoryAware && !llvm::is_contained(*refinedPartitions, initialGroupStages)) - refinedPartitions->push_back(initialGroupStages); - std::vector assignments; - for (const auto &groupStages : *refinedPartitions) { - std::vector taskStages(graph.nodes.size()); - for (size_t task = 0; task < graph.nodes.size(); ++task) - taskStages[task] = groupStages[taskToGroup[task]]; - assignments.push_back({std::move(taskStages), stageSizes}); } - return assignments; + PipelineStageAssignment result; + result.stageSizes.assign(stages, 1); + result.taskStages.resize(constraints.taskToGroup.size()); + size_t end = count, remaining = units; + for (size_t stage = selected; stage; --stage) { + const auto &cell = best[stage][end][remaining]; + result.stageSizes[stage - 1] = coreCount(cell.units); + for (size_t position = cell.start; position < end; ++position) + for (size_t task : constraints.groups[constraints.groupOrder[position]].tasks) + result.taskStages[task] = stage - 1; + end = cell.start; + remaining -= cell.units; + } + size_t assigned = std::accumulate(result.stageSizes.begin(), result.stageSizes.end(), size_t{0}); + for (; assigned < layout.getProcessorCount(); ++assigned) { + size_t bestStage = 0; + for (size_t stage = 1; stage < std::max(1, selected); ++stage) + if (result.stageSizes[stage] < result.stageSizes[bestStage]) bestStage = stage; + ++result.stageSizes[bestStage]; + } + return result; } static bool packPipelineStage( @@ -1074,59 +703,207 @@ static LogicalResult packPipelineAssignment(const ComputeGraph &graph, schedule.costs.stageSerialDemandNs.push_back(std::get<0>(stage)); schedule.costs.baselineMemory = graph.baselineMemory; schedule.costs.boundaryMemory = getPipelineBoundaryMemoryDemand( - graph, assignment.taskStages, physicalTarget, taskModel.sharePublications); + graph, assignment.taskStages, physicalTarget); populateScheduleEquivalenceClasses(graph, tasksByPhysicalCpu, schedule); return mlir::success(); } - -struct PipelineCandidateCapture { - bool enabled = false; - std::vector packed; - - void consider(const MergeScheduleResult &candidate) { - if (enabled) - packed.push_back(candidate); +static std::optional allocatePipelineStageCores( + const ComputeGraph &graph, const PipelineTaskModel &model, ArrayRef taskStages, + size_t stageCount, const SchedulingTarget &target, ArrayRef coreBudgets, + ArrayRef adjustableStages, bool constructPacks = true) { + std::vector tasks(stageCount); + PipelineStageAssignment assignment; + for (size_t task = 0; task < graph.nodes.size(); ++task) { + size_t stage = taskStages[task]; + assignment.taskStages.push_back(stage); + tasks[stage].push_back(task); } -}; - -static LogicalResult assignPipelineCores(const ComputeGraph &graph, - MergeScheduleResult &schedule, size_t pipelineStages, - const SchedulingTarget &physicalTarget, std::string &error, bool memoryAware, - bool sharePublications = true, PipelineCandidateCapture *capture = nullptr) { - PipelineCoreLayout balancedLayout(physicalTarget.processorCount, pipelineStages); - if (!balancedLayout.isValid()) { - error = "pipeline scheduling received an incompatible physical core layout"; - return failure(); - } - PipelineTaskModel taskModel(graph, physicalTarget, sharePublications); - auto assignments = assignPipelineStages(graph, taskModel, balancedLayout, - physicalTarget.residentWeightCapacity, error, memoryAware); - if (failed(assignments)) return failure(); - bool packed = false; - for (const auto &assignment : *assignments) { - MergeScheduleResult candidate; - if (failed(packPipelineAssignment(graph, taskModel, assignment, candidate, - pipelineStages, physicalTarget, error))) + assignment.stageSizes.resize(stageCount); + std::vector adjustable(stageCount); + for (size_t stage : adjustableStages) adjustable[stage] = true; + if (adjustableStages.empty()) return std::nullopt; + std::vector costs(stageCount, TaskCosts(target.processorCount + 1)); + TaskCosts levels, work(stageCount); + size_t minimumCores = 0; + for (size_t stage = 0; stage < stageCount; ++stage) { + llvm::sort(tasks[stage], [&](size_t a, size_t b) { + return graph.nodes[a].originalOrder < graph.nodes[b].originalOrder; + }); + size_t minimum = std::max(1, model.weights.getPackCountLowerBound( + tasks[stage], target.residentWeightCapacity)); + if (!adjustable[stage]) { + if (coreBudgets[stage] < minimum) return std::nullopt; + assignment.stageSizes[stage] = coreBudgets[stage]; + minimumCores += coreBudgets[stage]; + if (minimumCores > target.processorCount) return std::nullopt; continue; - if (capture) - capture->consider(candidate); - auto maximumStageSpan = [](const MergeScheduleResult &result) { - return result.costs.stageSpanNs.empty() ? Cost{0} - : *llvm::max_element(result.costs.stageSpanNs); - }; - // Store savings do not cover a longer stage when placement-dependent - // baseline loads remain approximate. Keep the calibrated phase-1 cut. - bool preservesStageSpan = !sharePublications - || maximumStageSpan(candidate) <= maximumStageSpan(schedule); - if (!memoryAware || (preservesStageSpan - && isBetterPipelineSchedule(candidate.costs, schedule.costs))) - schedule = std::move(candidate); - packed = true; + } + assignment.stageSizes[stage] = minimum; + minimumCores += minimum; + if (minimumCores > target.processorCount) return std::nullopt; + Cost largest = 0; + for (size_t task : tasks[stage]) { + work[stage] = checkedAdd(work[stage], model.schedulingCosts[task]); + largest = std::max(largest, model.schedulingCosts[task]); + } + for (size_t cores = minimum; cores <= target.processorCount; ++cores) { + TaskList packs; + Cost packedCost = std::max(largest, work[stage] / cores + (work[stage] % cores != 0)); + if (constructPacks) + packedCost = estimateOrderedPackCost(model.schedulingCosts, + model.weights.taskWeightIds, model.weights.weightCount, tasks[stage], + target.residentWeightCapacity, cores, &packs); + if (constructPacks && packs.size() != tasks[stage].size()) { + auto constructive = model.weights.pack(tasks[stage], model.schedulingCosts, + target.residentWeightCapacity, cores); + packedCost = std::numeric_limits::max(); + if (constructive) { + TaskCosts loads(cores); + for (size_t i = 0; i < tasks[stage].size(); ++i) + loads[(*constructive)[i]] = checkedAdd(loads[(*constructive)[i]], + model.schedulingCosts[tasks[stage][i]]); + packedCost = *llvm::max_element(loads); + } + } + // A feasible assignment remains feasible when extra cores stay idle. + if (cores > minimum) packedCost = std::min(packedCost, costs[stage][cores - 1]); + costs[stage][cores] = packedCost; + if (packedCost != std::numeric_limits::max()) levels.push_back(packedCost); + } } - return success(packed); + if (levels.empty()) return std::nullopt; + llvm::sort(levels); + levels.erase(std::unique(levels.begin(), levels.end()), levels.end()); + auto required = [&](size_t stage, Cost interval) { + if (!adjustable[stage]) return coreBudgets[stage]; + auto begin = costs[stage].begin() + assignment.stageSizes[stage]; + return static_cast(std::lower_bound(begin, costs[stage].end(), + interval, std::greater()) - costs[stage].begin()); + }; + auto fits = [&](Cost interval) { + size_t cores = 0; + for (size_t stage = 0; stage < stageCount; ++stage) { + cores += required(stage, interval); + if (cores > target.processorCount) return false; + } + return true; + }; + if (!fits(levels.back())) return std::nullopt; + size_t low = 0, high = levels.size() - 1; + while (low < high) { + size_t middle = low + (high - low) / 2; + if (fits(levels[middle])) high = middle; + else low = middle + 1; + } + std::vector sizes(stageCount); + for (size_t stage = 0; stage < stageCount; ++stage) + sizes[stage] = required(stage, levels[low]); + size_t assigned = std::accumulate(sizes.begin(), sizes.end(), size_t{0}); + for (; assigned < target.processorCount; ++assigned) { + auto score = [&](size_t stage) { + return std::make_tuple(sizes[stage] < tasks[stage].size(), + costs[stage][sizes[stage]], work[stage] / sizes[stage]); + }; + size_t best = adjustableStages.front(); + for (size_t stage = 1; stage < stageCount; ++stage) + if (adjustable[stage] && score(stage) > score(best)) best = stage; + ++sizes[best]; + } + assignment.stageSizes = std::move(sizes); + return assignment; } + +static void visitAdjacentPipelinePartitions(const PipelineGroupModel &groups, + const PipelineStageAssignment &original, + const std::function &visit) { + size_t count = groups.groups.size(), stages = original.stageSizes.size(), active = 0; + TaskList current(count); + for (size_t group = 0; group < count; ++group) { + current[group] = original.taskStages[groups.groups[group].tasks.front()]; + active = std::max(active, current[group] + 1); + } + auto emit = [&](TaskList proposed) { + TaskList counts(stages), compact(stages); + for (size_t stage : proposed) ++counts[stage]; + size_t next = 0; + for (size_t stage = 0; stage < stages; ++stage) + if (counts[stage]) compact[stage] = next++; + for (size_t &stage : proposed) stage = compact[stage]; + if (proposed == current) return; + for (size_t group = 0; group < count; ++group) { + if (groups.groups[group].consumesPipelineInput && proposed[group] != 0) return; + for (size_t successor : groups.groups[group].successors) + if (proposed[successor] < proposed[group] + || proposed[successor] > proposed[group] + 1) return; + } + auto candidate = original; + std::vector flows(stages), incoming(stages); + for (size_t group = 0; group < count; ++group) + for (size_t task : groups.groups[group].tasks) + candidate.taskStages[task] = proposed[group]; + for (size_t group = 0; group < count; ++group) { + if (!llvm::is_contained(flows[current[group]], proposed[group])) + flows[current[group]].push_back(proposed[group]); + if (!llvm::is_contained(incoming[proposed[group]], current[group])) + incoming[proposed[group]].push_back(current[group]); + } + candidate.stageSizes.assign(stages, 1); + std::vector adjustable(stages); + size_t spare = 0; + for (size_t stage = 0; stage < stages; ++stage) { + if (flows[stage].empty()) { spare += original.stageSizes[stage] - 1; continue; } + size_t destination = flows[stage].front(); + candidate.stageSizes[destination] += original.stageSizes[stage] - 1; + if (flows[stage].size() > 1) + for (size_t moved : flows[stage]) adjustable[moved] = true; + if (next == active && destination != stage) adjustable[destination] = true; + } + TaskList changed; + for (size_t stage = 0; stage < stages; ++stage) { + if (incoming[stage].size() > 1) adjustable[stage] = true; + if (adjustable[stage]) changed.push_back(stage); + } + if (changed.empty()) return; + candidate.stageSizes[changed.front()] += spare; + visit(std::move(candidate), std::move(changed)); + }; + for (size_t stage = 0; stage + 1 < active; ++stage) { + TaskList members; + for (size_t group : groups.groupOrder) + if (current[group] == stage || current[group] == stage + 1) members.push_back(group); + for (size_t cut = 1; cut < members.size(); ++cut) { + auto proposed = current; + for (size_t index = 0; index < members.size(); ++index) + proposed[members[index]] = stage + (index >= cut); + emit(std::move(proposed)); + } + } + for (size_t begin = 0; begin + 1 < active; ++begin) + for (size_t end = begin + 1; end < active; ++end) { + auto proposed = current; + for (size_t &stage : proposed) + if (stage > end) stage -= end - begin; + else if (stage >= begin) stage = begin; + emit(std::move(proposed)); + } + if (active < stages) + for (size_t stage = 0; stage < active; ++stage) { + TaskList members; + for (size_t group : groups.groupOrder) + if (current[group] == stage) members.push_back(group); + for (size_t cut = 1; cut < members.size(); ++cut) { + auto proposed = current; + for (size_t &value : proposed) if (value > stage) ++value; + for (size_t index = cut; index < members.size(); ++index) proposed[members[index]] = stage + 1; + emit(std::move(proposed)); + } + } +} + + + } // namespace PipelineCoreLayout::PipelineCoreLayout(ArrayRef sizes, @@ -1171,73 +948,321 @@ PipelineCoreLayout::PipelineCoreLayout(ArrayRef sizes, } } -mlir::LogicalResult runPipelineScheduler(const ComputeGraph& graph, - MergeScheduleResult& schedule, - size_t pipelineStages, - const SchedulingTarget& physicalTarget, - std::string& error, - std::vector *candidates) { - PipelineCoreLayout pipelineLayout( - physicalTarget.processorCount, pipelineStages); - if (pipelineStages <= 1 || !pipelineLayout.isValid()) { - error = "pipeline scheduling requires a valid balanced physical core layout"; - return mlir::failure(); - } - PipelineCandidateCapture capture{candidates != nullptr, {}}; - if (failed(assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, error, false, false, &capture))) +mlir::LogicalResult runPipelineScheduler(const ComputeGraph &graph, + MergeScheduleResult &schedule, size_t pipelineStages, + const SchedulingTarget &target, std::string &error) { + PipelineCoreLayout layout(target.processorCount, pipelineStages); + if (pipelineStages <= 1 || !layout.isValid()) { + error = "pipeline scheduling requires a valid physical core layout"; return failure(); - std::string candidateError; - // Retain the calibrated phase-1 winner before evaluating sharing-aware cuts. - if (physicalTarget.sharedMemoryWidthBytes && !graph.publicationTransfers.empty()) { - (void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, false, &capture); - std::vector taskStages; - for (const auto &node : graph.nodes) - taskStages.push_back(schedule.processorStages[ - schedule.computeToCpuMap.lookup(node.instance)]); - schedule.costs.boundaryMemory = getPipelineBoundaryMemoryDemand( - graph, taskStages, physicalTarget); - (void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, true, &capture); } - if (candidates) { - auto activeStages = [&](const MergeScheduleResult &result) { - llvm::SmallBitVector stages(pipelineStages); - for (const auto &node : graph.nodes) - stages.set(result.processorStages[result.computeToCpuMap.lookup(node.instance)]); - return stages; - }; - auto samePlacement = [&](const MergeScheduleResult &a, const MergeScheduleResult &b) { - for (const auto &node : graph.nodes) { - size_t cpu = a.computeToCpuMap.lookup(node.instance); - if (cpu != b.computeToCpuMap.lookup(node.instance) - || a.computeToCpuSlotMap.lookup(node.instance) != b.computeToCpuSlotMap.lookup(node.instance) - || a.processorStages[cpu] != b.processorStages[cpu]) - return false; + PipelineTaskModel model(graph, target); + auto groups = buildPipelineGroups(graph, model, error); + if (failed(groups)) return failure(); + auto partition = initialPipelinePartition(*groups, model, layout, + target.residentWeightCapacity, error); + if (failed(partition)) return failure(); + return packPipelineAssignment(graph, model, *partition, schedule, + pipelineStages, target, error); +} + +namespace { + +static PipelineStageAssignment scheduleAssignment(const ComputeGraph &graph, + const MergeScheduleResult &schedule, size_t stages) { + PipelineStageAssignment assignment; + assignment.stageSizes.resize(stages); + for (size_t stage : schedule.processorStages) ++assignment.stageSizes[stage]; + for (const auto &node : graph.nodes) + assignment.taskStages.push_back(schedule.processorStages[ + schedule.computeToCpuMap.lookup(node.instance)]); + return assignment; +} + +static TaskList placementIdentity(const ComputeGraph &graph, + const MergeScheduleResult &schedule) { + TaskList identity = schedule.processorStages; + for (const auto &node : graph.nodes) { + identity.push_back(schedule.computeToCpuMap.lookup(node.instance)); + identity.push_back(schedule.computeToCpuSlotMap.lookup(node.instance)); + } + return identity; +} + +struct PartitionProxy { + std::vector tasks; + TaskCosts spans; + Cost memory = 0; +}; + +static PartitionProxy partitionProxy(const ComputeGraph &graph, + const PipelineGroupModel &groups, const PipelineTaskModel &model, + const PipelineStageAssignment &assignment) { + size_t stages = assignment.stageSizes.size(); + PartitionProxy result; + result.tasks.resize(stages); + result.spans.resize(stages); + TaskCosts ends(graph.nodes.size()); + for (size_t group : groups.groupOrder) + for (size_t task : groups.groups[group].tasks) { + size_t stage = assignment.taskStages[task]; + result.tasks[stage].push_back(task); + Cost start = 0; + for (const auto &[predecessor, transfer] : graph.predecessors[task]) + if (assignment.taskStages[predecessor] == stage) + start = std::max(start, ends[predecessor]); + ends[task] = checkedAdd(start, model.schedulingCosts[task]); + result.spans[stage] = std::max(result.spans[stage], ends[task]); + } + result.memory = checkedAdd(graph.baselineMemory.serviceNs, + getPipelineBoundaryMemoryDemand(graph, assignment.taskStages, model.target).serviceNs); + return result; +} + +struct PipelineProposal { + PipelineStageAssignment assignment; + size_t parent; + std::tuple score; + bool optimizeCores; + TaskList adjustableStages; +}; +using PipelineMove = std::tuple; + +static std::vector pipelineNeighborhood(const ComputeGraph &graph, + const PipelineTaskModel &baseModel, const PipelineGroupModel &baseGroups, + const PipelineSearchResult &search, ArrayRef beam, + const std::set &visited, size_t limit) { + const auto &target = baseModel.target; + std::vector proposals; + for (size_t parent : beam) { + std::vector neighborhood; + const auto &model = baseModel; + const auto &groups = baseGroups; + auto original = scheduleAssignment(graph, search.schedules[parent], + search.schedules[parent].costs.stageSpanNs.size()); + size_t stages = original.stageSizes.size(); + std::vector pressure(stages); + if (!search.costs.empty()) { + const auto &cost = search.costs[parent]; + for (size_t core = 0; core < target.processorCount; ++core) { + size_t stage = search.schedules[parent].processorStages[core]; + if (core < cost.processorCriticalDelayNs.size()) { + pressure[stage] += cost.processorCriticalDelayNs[core]; + } } - return true; + for (size_t core = 0; core < cost.processorTaskWorkNs.size(); ++core) + pressure[search.schedules[parent].processorStages[core]] = std::max( + pressure[search.schedules[parent].processorStages[core]], cost.processorTaskWorkNs[core]); + } + auto originalView = partitionProxy(graph, groups, model, original); + std::vector>> packCosts(stages, + std::vector>(target.processorCount + 1)); + auto packingCost = [&](const TaskList &tasks, size_t cores, size_t stage) { + auto *cached = tasks == originalView.tasks[stage] ? &packCosts[stage][cores] : nullptr; + if (cached && *cached) return **cached; + Cost value = estimateOrderedPackCost(model.schedulingCosts, + model.weights.taskWeightIds, model.weights.weightCount, tasks, + target.residentWeightCapacity, cores); + if (cached) *cached = value; + return value; }; - auto originalStages = activeStages(schedule); - SmallVector alternatives; - for (auto [index, candidate] : llvm::enumerate(capture.packed)) { - if (activeStages(candidate) != originalStages || samePlacement(candidate, schedule)) - continue; - if (llvm::none_of(alternatives, [&](size_t other) { - return samePlacement(candidate, capture.packed[other]); - })) - alternatives.push_back(index); - } - candidates->push_back(schedule); - for (size_t index : alternatives) { - auto &candidate = capture.packed[index]; - std::vector taskStages; - for (const auto &node : graph.nodes) - taskStages.push_back(candidate.processorStages[ - candidate.computeToCpuMap.lookup(node.instance)]); - candidate.costs.boundaryMemory = getPipelineBoundaryMemoryDemand( - graph, taskStages, physicalTarget); - candidates->push_back(std::move(candidate)); - } + Cost originalBound = originalView.memory; + if (target.estimatePipelineDependencySpan) + originalBound = std::max(originalBound, target.estimatePipelineDependencySpan(originalView.spans)); + for (size_t stage = 0; stage < stages; ++stage) + originalBound = std::max(originalBound, packingCost(originalView.tasks[stage], + original.stageSizes[stage], stage)); + auto consider = [&](PipelineStageAssignment assignment, const PartitionProxy &view, + bool optimizeCores = false, ArrayRef adjustable = {}) { + TaskList adjustableStages(adjustable.begin(), adjustable.end()); + auto identity = std::make_tuple(parent, assignment.taskStages, assignment.stageSizes, + optimizeCores, adjustableStages); + if (visited.count(identity) || llvm::any_of(neighborhood, [&](const auto &proposal) { + return proposal.parent == parent && proposal.optimizeCores == optimizeCores + && proposal.adjustableStages == adjustableStages + && proposal.assignment.taskStages == assignment.taskStages + && proposal.assignment.stageSizes == assignment.stageSizes; + })) return; + Cost maximum = 0, total = 0; + double affected = 0; + std::vector changed(stages); + for (size_t task = 0; task < graph.nodes.size(); ++task) + if (original.taskStages[task] != assignment.taskStages[task]) + changed[original.taskStages[task]] = changed[assignment.taskStages[task]] = true; + for (size_t stage = 0; stage < stages; ++stage) { + if (model.weights.getPackCountLowerBound(view.tasks[stage], + target.residentWeightCapacity) > assignment.stageSizes[stage]) return; + Cost work = packingCost(view.tasks[stage], assignment.stageSizes[stage], stage); + maximum = std::max(maximum, work); + // A within-generation span is latency, not an initiation-interval bound. + total = checkedAdd(total, view.spans[stage]); + if (changed[stage] || original.stageSizes[stage] != assignment.stageSizes[stage]) + affected += pressure[stage]; + } + double priority = maximum; + if (!search.costs.empty()) { + // Compare local proxy changes against the parent's queue cost. Absolute + // resource projections from different parents are not comparable. + const auto &cost = search.costs[parent]; + Cost resource = std::max(maximum, view.memory), dependency = 0; + if (target.estimatePipelineDependencySpan) + dependency = target.estimatePipelineDependencySpan(view.spans); + priority = std::max(*cost.primary, *cost.alternative) + + static_cast(std::max(resource, dependency)) - originalBound; + } + neighborhood.push_back({std::move(assignment), parent, + {priority, -affected, view.memory, total}, optimizeCores, std::move(adjustableStages)}); + llvm::sort(neighborhood, [](const auto &a, const auto &b) { + return std::tie(a.score, a.assignment.taskStages, a.assignment.stageSizes, a.parent) + < std::tie(b.score, b.assignment.taskStages, b.assignment.stageSizes, b.parent); + }); + if (neighborhood.size() > limit / beam.size()) neighborhood.pop_back(); + }; + visitAdjacentPipelinePartitions(groups, original, + [&](PipelineStageAssignment partition, TaskList adjustable) { + if (auto allocation = allocatePipelineStageCores(graph, model, + partition.taskStages, stages, target, partition.stageSizes, adjustable, false)) { + auto view = partitionProxy(graph, groups, model, *allocation); + consider(std::move(*allocation), view, true, adjustable); + } + }); + // Core transfers are coordinate moves in the same joint neighborhood. + std::vector active(stages); + for (size_t stage : original.taskStages) active[stage] = true; + for (size_t source = 0; source < stages; ++source) + for (size_t destination = 0; destination < stages; ++destination) + if (source != destination && (active[source] || active[destination]) + && original.stageSizes[source] > 1) { + for (size_t cores = 1; cores < original.stageSizes[source]; cores *= 2) { + auto allocation = original; + allocation.stageSizes[source] -= cores; + allocation.stageSizes[destination] += cores; + consider(std::move(allocation), originalView); + if (cores > original.stageSizes[source] / 2) break; + } + } + for (auto &proposal : neighborhood) proposals.push_back(std::move(proposal)); } - return success(); + llvm::sort(proposals, [](const auto &a, const auto &b) { + return std::tie(a.score, a.assignment.taskStages, a.assignment.stageSizes, a.parent) + < std::tie(b.score, b.assignment.taskStages, b.assignment.stageSizes, b.parent); + }); + return proposals; +} +} // namespace + +mlir::FailureOr optimizePipelineSchedule( + const ComputeGraph &graph, const MergeScheduleResult &initial, + size_t pipelineStages, const SchedulingTarget &target, + const PipelineCostQuery &evaluate, std::string &error, size_t evaluationBudget) { + error.clear(); + if (!evaluationBudget || evaluationBudget > kPipelineQueueEvaluationBudget) { + error = "pipeline queue-evaluation budget must be between one and thirty-two"; + return failure(); + } + PipelineTaskModel model(graph, target); + auto executionModel = model; + executionModel.schedulingCosts = model.executionCosts; + auto groups = buildPipelineGroups(graph, model, error); + if (failed(groups)) return failure(); + PipelineSearchResult result; + result.schedules.push_back(initial); result.parents.push_back(0); + if (groups->groups.size() <= 1) { + result.termination = "one operation group; queue comparison skipped"; + return result; + } + std::set visited; + std::set placements{placementIdentity(graph, initial)}; + std::vector beam{0}; + auto proposals = pipelineNeighborhood(graph, executionModel, *groups, result, beam, visited, 2 * evaluationBudget); + if (proposals.empty() || evaluationBudget == 1) { + result.termination = "one neighborhood schedule; queue comparison skipped"; + return result; + } + auto cost = [&](const MergeScheduleResult &schedule) { + auto value = evaluate(schedule); + for (const auto *guidance : {&value.processorTaskWorkNs, &value.processorCriticalDelayNs}) { + if (guidance->size() > target.processorCount) + error = "pipeline optimizer target guidance exceeds the physical processor count"; + else if (llvm::any_of(*guidance, [](double v) { + return !std::isfinite(v) || v < 0 + || static_cast(v) >= std::numeric_limits::max(); + })) error = "pipeline optimizer target guidance is unrepresentable"; + } + if (error.empty() && !value.eligible()) error = "pipeline optimizer candidate " + + std::to_string(result.costs.size()) + " cannot compare complete cost evidence: " + value.reason; + return value; + }; + result.costs.push_back(cost(initial)); + if (!error.empty()) return failure(); + constexpr size_t maximumPackingAttempts = 4 * kPipelineQueueEvaluationBudget; + while (result.costs.size() < evaluationBudget && result.packingAttempts < maximumPackingAttempts) { + proposals = pipelineNeighborhood(graph, executionModel, *groups, result, beam, visited, 2 * evaluationBudget); + if (proposals.empty()) break; + ++result.rounds; + size_t evaluated = 0; + std::set expandedParents; + std::vector ranked; + for (const auto &proposal : proposals) { + if (result.costs.size() == evaluationBudget || evaluated == kPipelineBeamWidth + || result.packingAttempts == maximumPackingAttempts) break; + if (beam.size() > 1 && expandedParents.count(proposal.parent)) continue; + ++result.consideredAssignments; + ++result.packingAttempts; + visited.emplace(proposal.parent, proposal.assignment.taskStages, + proposal.assignment.stageSizes, proposal.optimizeCores, proposal.adjustableStages); + auto assignment = proposal.assignment; + if (proposal.optimizeCores) { + auto allocated = allocatePipelineStageCores(graph, executionModel, assignment.taskStages, + pipelineStages, target, assignment.stageSizes, proposal.adjustableStages); + if (!allocated) continue; + assignment = std::move(*allocated); + } + MergeScheduleResult schedule; + std::string packingError; + if (failed(packPipelineAssignment(graph, model, assignment, schedule, + pipelineStages, target, packingError))) continue; + if (!placements.insert(placementIdentity(graph, schedule)).second) continue; + auto estimate = cost(schedule); + if (!error.empty()) return failure(); + result.parents.push_back(proposal.parent); + result.schedules.push_back(std::move(schedule)); + result.costs.push_back(std::move(estimate)); + expandedParents.insert(proposal.parent); + ++evaluated; + } + ranked.resize(result.costs.size()); + std::iota(ranked.begin(), ranked.end(), 0); + llvm::sort(ranked, [&](size_t a, size_t b) { + auto score = [&](size_t i) { + const auto &value = result.costs[i]; + return std::make_tuple(std::max(*value.primary, *value.alternative), + *value.primary, *value.alternative, i); + }; + return score(a) < score(b); + }); + ranked.resize(std::min(kPipelineBeamWidth, ranked.size())); + beam = std::move(ranked); + } + result.termination = result.costs.size() == evaluationBudget ? "queue-evaluation budget" + : result.packingAttempts == maximumPackingAttempts ? "physical-packing budget" : "beam neighborhood exhausted"; + return result; +} + +mlir::LogicalResult packPipelineSchedule(const ComputeGraph &graph, + ArrayRef taskStages, ArrayRef stageSizes, + const SchedulingTarget &target, MergeScheduleResult &schedule, std::string &error) { + if (taskStages.size() != graph.nodes.size() || stageSizes.empty() + || llvm::is_contained(stageSizes, 0) + || std::accumulate(stageSizes.begin(), stageSizes.end(), size_t{0}) != target.processorCount + || llvm::any_of(taskStages, [&](size_t stage) { return stage >= stageSizes.size(); })) { + error = "pipeline packing requires complete stage assignments and physical core budgets"; + return failure(); + } + PipelineTaskModel model(graph, target); + return packPipelineAssignment(graph, model, + {{taskStages.begin(), taskStages.end()}, {stageSizes.begin(), stageSizes.end()}}, + schedule, stageSizes.size(), target, error); } } // namespace onnx_mlir::spatial diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp index ce90a0f..9948b06 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp @@ -7,6 +7,7 @@ #include #include +#include #include #include #include @@ -32,25 +33,6 @@ inline bool isBetterStagePlacement(const StagePlacementScore &candidate, return candidate < current; } -inline bool isBetterPipelineSchedule(const ScheduleCostSummary &candidate, - const ScheduleCostSummary ¤t) { - auto score = [](const ScheduleCostSummary &result) { - Cost stage = 0; - for (Cost span : result.stageSpanNs) stage = std::max(stage, span); - Cost memory = checkedAdd(result.baselineMemory.serviceNs, result.boundaryMemory.serviceNs); - return std::make_tuple(std::max(stage, memory), stage, memory); - }; - auto currentScore = score(current); - auto candidateScore = score(candidate); - // Stage spans omit contention: do not exchange a larger compute regression - // for a smaller memory saving, even when memory remains the largest resource. - Cost currentDemand = checkedAdd(std::get<1>(currentScore), std::get<2>(currentScore)); - Cost candidateDemand = checkedAdd(std::get<1>(candidateScore), std::get<2>(candidateScore)); - Cost currentCoreDemand = checkedAdd(static_cast(current.maximumCoreWorkNs), std::get<2>(currentScore)); - Cost candidateCoreDemand = checkedAdd(static_cast(candidate.maximumCoreWorkNs), std::get<2>(candidateScore)); - return candidateScore < currentScore && candidateDemand <= currentDemand - && candidateCoreDemand <= currentCoreDemand; -} struct PipelineStageRange { size_t begin; @@ -119,13 +101,34 @@ private: std::vector physicalCoreStages; }; -// Produces a complete physical schedule directly from the shared compute graph. -mlir::LogicalResult runPipelineScheduler(const ComputeGraph& graph, - MergeScheduleResult& schedule, - size_t pipelineStages, - const SchedulingTarget& physicalTarget, - std::string& error, - std::vector *candidates = nullptr); +// One initialization procedure, also used for intermediate emission. +mlir::LogicalResult runPipelineScheduler(const ComputeGraph &graph, + MergeScheduleResult &schedule, size_t pipelineStages, + const SchedulingTarget &target, std::string &error); + +inline constexpr size_t kPipelineQueueEvaluationBudget = 32; +inline constexpr size_t kPipelineBeamWidth = 2; + +struct PipelineSearchResult { + std::vector schedules; + std::vector costs; + std::vector parents; + size_t rounds = 0, packingAttempts = 0, consideredAssignments = 0; + std::string termination; +}; + +// The callback realizes an independent Spatial clone; it must not mutate graph IR. +using PipelineCostQuery = std::function; +mlir::FailureOr optimizePipelineSchedule( + const ComputeGraph &graph, const MergeScheduleResult &initial, + size_t pipelineStages, const SchedulingTarget &target, + const PipelineCostQuery &evaluate, std::string &error, + size_t evaluationBudget = kPipelineQueueEvaluationBudget); + +// Constructive capacity/topology packing shared with the small-instance oracle. +mlir::LogicalResult packPipelineSchedule(const ComputeGraph &graph, + llvm::ArrayRef taskStages, llvm::ArrayRef stageSizes, + const SchedulingTarget &target, MergeScheduleResult &schedule, std::string &error); enum class PipelineWorkloadPreparation { Ready, diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp index c30dcef..d4578e9 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp @@ -10,6 +10,77 @@ namespace onnx_mlir::spatial { +// Greedy maximal prefixes minimize ordered pack count: both constraints are +// closed under taking subranges. The optimum cost cannot increase with packs. +inline Cost estimateOrderedPackCost( + llvm::ArrayRef taskCosts, + const std::vector> &taskWeightIds, size_t weightCount, + llvm::ArrayRef tasks, size_t residentWeightCapacity, + size_t maximumPacks, std::vector *taskPacks = nullptr) { + Cost low = 0; + Cost high = 0; + for (size_t task : tasks) { + low = std::max(low, taskCosts[task]); + high = checkedAdd(high, taskCosts[task]); + } + low = std::max( + low, high / maximumPacks + (high % maximumPacks != 0)); + + std::vector seen(weightCount); + size_t generation = 0; + auto pack = [&](Cost maximumCost, std::vector *assignment) { + size_t packs = 1; + Cost cost = 0; + size_t packWeightCount = 0; + ++generation; + bool fits = true; + bool packEmpty = true; + for (size_t task : tasks) { + size_t addedWeights = 0; + for (size_t weight : taskWeightIds[task]) + addedWeights += seen[weight] != generation; + Cost taskCost = taskCosts[task]; + bool startsNewPack = !packEmpty + && (cost > maximumCost - taskCost + || packWeightCount + addedWeights > residentWeightCapacity); + if (startsNewPack) { + if (++packs > maximumPacks) { + fits = false; + break; + } + cost = 0; + packWeightCount = 0; + ++generation; + packEmpty = true; + } + cost = checkedAdd(cost, taskCost); + for (size_t weight : taskWeightIds[task]) + if (seen[weight] != generation) { + seen[weight] = generation; + ++packWeightCount; + } + if (assignment) + assignment->push_back(packs - 1); + packEmpty = false; + } + return fits; + }; + while (low < high) { + Cost middle = low + (high - low) / 2; + if (pack(middle, nullptr)) + high = middle; + else + low = middle + 1; + } + if (taskPacks) { + taskPacks->clear(); + if (!pack(low, taskPacks)) + taskPacks->clear(); + } + return low; +} + + struct IndexedResidentWeights { std::vector> taskWeightIds; size_t weightCount = 0; diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp index d5e14e7..ac775ec 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp @@ -9,6 +9,7 @@ #include #include "mlir/Support/LogicalResult.h" +#include "llvm/ADT/ArrayRef.h" #include "Utils.hpp" @@ -61,6 +62,9 @@ struct WholeScheduleCost { std::optional selectionMargin; std::array recurrence; StaticLoweringSupport loweringSupport = StaticLoweringSupport::Unknown; + // Optional task-attached work and recurrence contributions by physical processor. + // These guide search; they are not additional charges or hardware guarantees. + std::vector processorTaskWorkNs, processorCriticalDelayNs; struct AnalysisTiming { double extractionSeconds = 0, expansionSeconds = 0, evaluationSeconds = 0; } timing; @@ -99,6 +103,9 @@ struct SchedulingTarget { std::function estimateRealizedScheduleCost; std::string wholeScheduleCostUnavailable; + // Optional recurrence priority over stage dependency spans. This screens + // moves only; the complete whole-schedule cost remains authoritative. + std::function)> estimatePipelineDependencySpan; std::function reportPeriodicSchedule; diff --git a/test/PIM/CMakeLists.txt b/test/PIM/CMakeLists.txt index 8b3b64c..855aa1c 100644 --- a/test/PIM/CMakeLists.txt +++ b/test/PIM/CMakeLists.txt @@ -55,6 +55,11 @@ add_pim_unittest(PimAbstractExecutionTest LINK_LIBS PRIVATE OMPimCompilerUtils ) +add_pim_unittest(PipelineOptimizerTest + PipelineOptimizerTest.cpp + LINK_LIBS PRIVATE OMPimCompilerUtils +) + add_pim_unittest(SpatialSchedulingTargetTest SpatialSchedulingTargetTest.cpp diff --git a/test/PIM/PimAbstractExecutionTest.cpp b/test/PIM/PimAbstractExecutionTest.cpp index b21abd6..cfdf3e5 100644 --- a/test/PIM/PimAbstractExecutionTest.cpp +++ b/test/PIM/PimAbstractExecutionTest.cpp @@ -47,6 +47,27 @@ static std::string binary(const std::vector &reco return result; } +static void testSearchGuidance(PimInstructionTiming target) { + PimExecutionProgram program{{{E::Work, 29}, {E::Load, 65}}}; + PimPeriodicOptions options; + options.verifyPeriodicState = true; + auto plain = evaluatePimExecution(program, target, options); + options.searchGuidance = true; + auto guided = evaluatePimExecution(program, target, options); + TEST_CHECK(plain.cost.primary == guided.cost.primary); + TEST_CHECK(plain.cost.alternative == guided.cost.alternative); + TEST_CHECK(plain.cost.convergenceProved() == guided.cost.convergenceProved()); + TEST_CHECK(guided.cost.processorTaskWorkNs.size() == target.processorCount); + TEST_CHECK(guided.cost.processorTaskWorkNs.front() == 29); + TEST_CHECK(guided.cost.processorCriticalDelayNs.size() == target.processorCount); + TEST_CHECK(guided.cost.processorCriticalDelayNs.front() > 0); + program.front()[1].constant = true; + auto constants = evaluatePimExecution(program, target, options); + TEST_CHECK(constants.cost.primary == plain.cost.primary); + TEST_CHECK(constants.cost.alternative == plain.cost.alternative); + TEST_CHECK(constants.cost.processorTaskWorkNs.front() > 29); +} + static void testTypedResult(PimInstructionTiming target) { PimPeriodicOptions options; options.verifyPeriodicState = true; @@ -496,10 +517,14 @@ static void testReadOnlySpatial(PimInstructionTiming target, bool batched) { TEST_CHECK((*classes)[E::LocalCopy].getAsInteger() == 0); TEST_CHECK((*classes)[E::ScalarIssue].getAsInteger() == 0); TEST_CHECK(totalWork == report.getInteger("compute_and_local_work_ns")); - auto softmax = spatial::SpatSoftmaxOp::create(builder, loc, type, input); + auto softmax = spatial::SpatSoftmaxOp::create(builder, loc, type, local); + auto secondSoftmax = spatial::SpatSoftmaxOp::create(builder, loc, type, softmax.getOutput()); TEST_CHECK(succeeded(verify(module))); - TEST_CHECK(analyzePimSpatialSchedule(module, target, spatialDetails()).getString("status") == "unavailable"); - softmax.erase(); + auto softmaxReport = analyzePimSpatialSchedule(module, target, spatialDetails()); + TEST_CHECK(softmaxReport.getString("status") == "estimated"); + TEST_CHECK(softmaxReport.getInteger("compute_and_local_work_ns") == totalWork); + TEST_CHECK(softmaxReport.getInteger("global_read_requests") == report.getInteger("global_read_requests")); + secondSoftmax.erase(); softmax.erase(); source = tensor::EmptyOp::create(builder, loc, ArrayRef{1, 0, 1, 224}, builder.getF32Type()); destination = tensor::EmptyOp::create(builder, loc, ArrayRef{1, 1, 224, 0}, builder.getF32Type()); linalg::TransposeOp::create(builder, loc, source, destination, ArrayRef{0, 2, 3, 1}); @@ -623,6 +648,40 @@ static void testPublicationAndOutput(PimInstructionTiming target) { TEST_CHECK((*(*runs)[0].getAsArray())[1].getAsInteger() == 60); TEST_CHECK((*(*runs)[1].getAsArray())[1].getAsInteger() == 5); TEST_CHECK((*(*runs)[1].getAsArray())[2].getAsInteger() == 4); + + // Ablation removes standalone and fused synchronization, not host accesses. + builder.setInsertionPoint(compute.getBody().front().getTerminator()); + Value readerId = arith::ConstantIndexOp::create(builder, loc, 1); + Value eventRegister = arith::ConstantIndexOp::create(builder, loc, 2); + spatial::SpatSyncOp::create(builder, loc, readerId, eventRegister); + builder.setInsertionPoint(reader.getBody().front().getTerminator()); + eventRegister = arith::ConstantIndexOp::create(builder, loc, 2); + spatial::SpatHostWaitLoadOp::create(builder, loc, slice.getType(), + zero, one, zero, eventRegister, one); + TEST_CHECK(succeeded(verify(module))); + PimExecutionProgram reference, omitted; + auto options = spatialDetails(); + options.executionTrace = &reference; + auto synchronized = evaluatePimSpatialSchedule(module, target, options); + TEST_CHECK(synchronized.cost.available()); + options.omitSynchronization = true; + options.executionTrace = &omitted; + auto ablation = evaluatePimSpatialSchedule(module, target, options); + TEST_CHECK(ablation.cost.available()); + TEST_CHECK(ablation.report.getObject("event_counts")->getInteger("sync") == 0); + TEST_CHECK(ablation.report.getObject("event_counts")->getInteger("wait") == 0); + for (const char *metric : {"global_read_requests", "global_read_bytes", + "global_write_requests", "global_write_bytes", "global_memory_service_ns"}) + TEST_CHECK(ablation.report.getInteger(metric) == synchronized.report.getInteger(metric)); + for (auto &sequence : reference) + llvm::erase_if(sequence, [](const E &event) { return event.kind == E::Signal || event.kind == E::Wait; }); + for (const auto &sequence : omitted) + for (const E &event : sequence) + TEST_CHECK(event.kind != E::Signal && event.kind != E::Wait); + options.executionTrace = nullptr; + auto expected = evaluatePimExecution(reference, target, options); + TEST_CHECK(ablation.cost.primary == expected.cost.primary); + TEST_CHECK(ablation.cost.alternative == expected.cost.alternative); module.erase(); } @@ -850,6 +909,7 @@ int main() { for (int64_t period : {1, 5}) { testLateRecurrenceAuthority(hardware(period)); testTypedResult(hardware(period)); + testSearchGuidance(hardware(period)); testSharedEngine(hardware(period)); testAccountingAndOverflow(hardware(period)); testMatrixGeometry(hardware(period)); diff --git a/test/PIM/PimPipelineSelectionTest.cpp b/test/PIM/PimPipelineSelectionTest.cpp index eef873f..f291c79 100644 --- a/test/PIM/PimPipelineSelectionTest.cpp +++ b/test/PIM/PimPipelineSelectionTest.cpp @@ -53,8 +53,8 @@ static void testEmptyGraphPreparation() { SchedulingTarget target; target.processorCount = target.residentWeightCapacity = 1; target.interProcessorLatencyNs = {0}; - for (bool collectCandidates : {false, true}) { - auto prepared = prepareSpatialSchedules(*module, target, 1, collectCandidates); + { + auto prepared = prepareSpatialSchedules(*module, target, 1); TEST_CHECK(succeeded(prepared) && prepared->candidates.size() == 1); TEST_CHECK(prepared->candidates.front().dominanceOrderCompute.empty()); } @@ -109,9 +109,9 @@ static void testSelectedLoweringFailure() { bufferization::func_ext::registerBufferizableOpInterfaceExternalModels(registry); context.appendDialectRegistry(registry); context.loadAllAvailableDialects(); - for (bool collectCandidates : {false, true}) { + { OwningOpRef checkpoint = cast((*module)->clone()); - auto prepared = prepareSpatialSchedules(*checkpoint, target, 2, collectCandidates); + auto prepared = prepareSpatialSchedules(*checkpoint, target, 2); TEST_CHECK(succeeded(prepared) && !prepared->candidates.empty()); } PassManager manager(&context); @@ -233,6 +233,14 @@ static void testResourceRetries() { }, reason); TEST_CHECK(!selected && attempts == 1); TEST_CHECK(reason.find("recovery is not justified") != std::string::npos); + // The original schedule succeeds alone. Adding unknown evidence must produce + // an explicit incomplete comparison, never silently drop the alternative. + attempts = 0; + std::vector single{original}; + selected = lowerSelectedPimCandidate(single, [&](size_t index) { + ++attempts; TEST_CHECK(index == 0); return PimCandidateResult::Success; + }, reason); + TEST_CHECK(selected == 0 && attempts == 1); for (unsigned unknown = 0; unknown < 4; ++unknown) { auto invalid = faster; if (unknown == 0) invalid.primary.reset(); @@ -246,6 +254,7 @@ static void testResourceRetries() { return PimCandidateResult::Success; }, reason); TEST_CHECK(!selected && attempts == 0); + TEST_CHECK(reason.find("cannot compare the complete candidate set") != std::string::npos); } coresCount = previousCores; crossbarSize = previousSize; crossbarCountInCore = previousCrossbars; } diff --git a/test/PIM/PipelineOptimizerTest.cpp b/test/PIM/PipelineOptimizerTest.cpp new file mode 100644 index 0000000..b528033 --- /dev/null +++ b/test/PIM/PipelineOptimizerTest.cpp @@ -0,0 +1,252 @@ +#include "TestCheck.hpp" +#include "src/Accelerators/PIM/Compiler/PimPeriodicAnalysis.hpp" +#include "src/Accelerators/PIM/Compiler/PimInstructionTiming.hpp" +#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp" +#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ScheduleEvaluation.hpp" +#include +#include +#include +#include +#include + +using namespace onnx_mlir; +using namespace onnx_mlir::spatial; +using E = PimExecutionEvent; +static bool sawJointMove = false, sawSuccessiveMoves = false; + +// Test-only workloads use stage-boundary host transfers and local direct links. +// The same queue +// engine and complete-state evidence rank reference and searched schedules. +static WholeScheduleCost queueCost(const ComputeGraph &graph, + const MergeScheduleResult &schedule, const PimInstructionTiming &target) { + PimExecutionProgram program(target.processorCount); + std::vector> tasks(target.processorCount); + for (size_t task = 0; task < graph.nodes.size(); ++task) + tasks[schedule.computeToCpuMap.lookup(graph.nodes[task].instance)].push_back(task); + for (auto &core : tasks) + llvm::sort(core, [&](size_t a, size_t b) { + return schedule.computeToCpuSlotMap.lookup(graph.nodes[a].instance) + < schedule.computeToCpuSlotMap.lookup(graph.nodes[b].instance); + }); + struct Transfer { size_t source, target, reader, writer, reg; bool host; }; + std::vector transfers; + std::vector> readers(target.processorCount), writers(target.processorCount); + // Branched synthetic workloads use host routing to avoid inventing a second + // blocking-transfer scheduler in this test-only objective. + bool branching = llvm::any_of(graph.successors, [](const auto &edges) { return edges.size() > 1; }); + for (size_t task = 0; task < graph.nodes.size(); ++task) + for (const auto &[predecessor, dependency] : graph.predecessors[task]) { + size_t writer = schedule.computeToCpuMap.lookup(graph.nodes[predecessor].instance); + size_t reader = schedule.computeToCpuMap.lookup(graph.nodes[task].instance); + if (writer == reader) continue; + bool host = branching || schedule.processorStages[writer] != schedule.processorStages[reader]; + transfers.push_back({predecessor, task, reader, writer, 1 + transfers.size(), host}); + if (host) { readers[writer].insert(reader); writers[reader].insert(writer); } + } + for (size_t core = 0; core < program.size(); ++core) { + auto &events = program[core]; + for (size_t writer : writers[core]) events.emplace_back(E::Signal, 0, writer, 0, 1); + if (schedule.processorStages[core] == 0) { + size_t peers = 0; + for (size_t peer = 0; peer < program.size(); ++peer) + if (peer != core && schedule.processorStages[peer] == 0) { + events.emplace_back(E::Signal, 0, peer, 100, 1); ++peers; + } + if (peers) events.emplace_back(E::Wait, 0, 0, 100, peers); + } + bool reuseWait = false; + for (size_t task : tasks[core]) { + for (const auto &transfer : transfers) + if (transfer.target == task) { + if (transfer.host) { + events.emplace_back(E::Wait, 0, 0, transfer.reg, 1); + events.emplace_back(E::Load, 64); + } else events.emplace_back(E::Receive, 64, transfer.writer); + } + E work(E::Work, graph.nodes[task].cost); work.workClass = E::Vector; + events.push_back(work); + for (const auto &transfer : transfers) + if (transfer.source == task) { + if (!transfer.host) { + events.emplace_back(E::Send, 64, transfer.reader); continue; + } + if (!reuseWait) { + events.emplace_back(E::Wait, 0, 0, 0, readers[core].size()); reuseWait = true; + } + events.emplace_back(E::Store, 64); + events.emplace_back(E::Signal, 0, transfer.reader, transfer.reg, 1); + } + } + } + PimPeriodicOptions options; + options.verifyPeriodicState = true; options.searchGuidance = true; + auto estimate = evaluatePimExecution(program, target, options); + auto cost = std::move(estimate.cost); + cost.loweringSupport = StaticLoweringSupport::Supported; + cost.selectionMargin = 0.021; + TEST_CHECK(cost.eligible()); + return cost; +} + +static PimInstructionTiming hardware(size_t profile) { + PimInstructionTiming target; + target.processorCount = 5; + target.residentWeightCapacity = 2; + target.estimatePipelineDependencySpan = [](llvm::ArrayRef spans) { return spans.front(); }; + target.peerPermissionThenSourceRead = true; + target.signalIssueNs = profile + 1; + target.controlMessageBytes = 1; + target.sharedMemoryWidthBytes = 64; + target.sharedMemoryReadLatencyNs = 17 + profile * 9; + target.sharedMemoryWriteLatencyNs = 23 + profile * 7; + target.synchronizationRegisterCount = 128; + target.sharedMemoryRequestLatencyNs.assign(5, 2); + target.sharedMemoryResponseLatencyNs.assign(5, 3); + target.interProcessorLatencyNs.resize(25); + for (size_t a = 0; a < 5; ++a) + for (size_t b = 0; b < 5; ++b) { + const size_t irregular[] = {0, 1, 5, 6, 20}; + size_t x = profile ? irregular[a] : a, y = profile ? irregular[b] : b; + target.interProcessorLatencyNs[a * 5 + b] = x > y ? x - y : y - x; + } + return target; +} + +static void runReference(size_t profile, size_t fixture) { + auto target = hardware(profile); + ComputeGraph graph; + const std::vector> work{{900, 70, 800, 90, 700}, + {30, 700, 110, 600, 90}, {1000, 50, 40, 900, 60}, + {90, 80, 100, 110, 120}, {100, 800, 100, 800, 100}, + {900, 800, 40, 30, 20}}; + size_t count = work[fixture].size(); + graph.predecessors.resize(count); graph.successors.resize(count); + for (size_t task = 0; task < count; ++task) { + ResidentWeight weight; weight.opaqueLane = fixture == 4 ? task % 3 : task % 2; + graph.nodes.push_back({{nullptr, static_cast(task), 1}, work[fixture][task], + {weight}, task}); + auto edge = [&](size_t source) { + graph.predecessors[task].push_back({source, {64, 1}}); + graph.successors[source].push_back({task, {64, 1}}); + graph.publicationTransfers.push_back({source, task, {}, 0, {}, 64, 1}); + }; + if (fixture == 1) { + if (task == 1 || task == 2) edge(0); + if (task == 3) { edge(1); edge(2); } + if (task == 4) edge(3); + } else if (task) edge(task - 1); + if (fixture == 2 && task == 4) edge(1); + } + if (fixture == 3) { + target.sharedMemoryReadLatencyNs = 400; + target.sharedMemoryWriteLatencyNs = 600; + } + std::string error; + MergeScheduleResult initial; + TEST_CHECK(mlir::succeeded(runPipelineScheduler(graph, initial, 3, target, error))); + auto evaluate = [&](const MergeScheduleResult &schedule) { return queueCost(graph, schedule, target); }; + auto begin = std::chrono::steady_clock::now(); + auto search = optimizePipelineSchedule(graph, initial, 3, target, evaluate, error); + double searchSeconds = std::chrono::duration(std::chrono::steady_clock::now() - begin).count(); + TEST_CHECK(mlir::succeeded(search)); + auto repeat = optimizePipelineSchedule(graph, initial, 3, target, evaluate, error); + TEST_CHECK(mlir::succeeded(repeat) && search->schedules.size() == repeat->schedules.size()); + for (size_t i = 0; i < search->schedules.size(); ++i) { + TEST_CHECK(search->schedules[i].computeToCpuMap == repeat->schedules[i].computeToCpuMap); + TEST_CHECK(search->schedules[i].computeToCpuSlotMap == repeat->schedules[i].computeToCpuSlotMap); + TEST_CHECK(search->schedules[i].processorStages == repeat->schedules[i].processorStages); + TEST_CHECK(search->costs.empty() || search->costs[i].primary == repeat->costs[i].primary); + TEST_CHECK(search->costs.empty() || search->costs[i].alternative == repeat->costs[i].alternative); + } + double reference = std::numeric_limits::infinity(); + size_t referenceEvaluations = 0; + std::vector stages(count), budgets(3, 1); + std::function partitions = [&](size_t task) { + if (task != count) { + for (size_t stage = 0; stage <= (task ? 2 : 0); ++stage) { + stages[task] = stage; partitions(task + 1); + } + return; + } + std::vector used(3); + for (size_t stage : stages) used[stage] = true; + if (used[2] && !used[1]) return; + for (size_t core0 = 1; core0 + 1 < target.processorCount; ++core0) + for (size_t core1 = 1; core0 + core1 < target.processorCount; ++core1) { + budgets = {core0, core1, target.processorCount - core0 - core1}; + MergeScheduleResult schedule; + if (mlir::failed(packPipelineSchedule(graph, stages, budgets, target, schedule, error))) continue; + auto cost = evaluate(schedule); ++referenceEvaluations; + reference = std::min(reference, std::max(*cost.primary, *cost.alternative)); + } + }; + auto referenceBegin = std::chrono::steady_clock::now(); + partitions(0); + double referenceSeconds = std::chrono::duration(std::chrono::steady_clock::now() - referenceBegin).count(); + std::string reason; + size_t winner = search->costs.empty() ? 0 : selectPipelineCandidate(search->costs, reason); + auto selected = evaluate(search->schedules[winner]); + double cost = std::max(*selected.primary, *selected.alternative); + double bestEvaluated = cost; + for (const auto &estimate : search->costs) + bestEvaluated = std::min(bestEvaluated, std::max(*estimate.primary, *estimate.alternative)); + TEST_CHECK(std::isfinite(reference) && cost + 1e-6 >= reference); + TEST_CHECK(bestEvaluated + 1e-6 >= reference); + TEST_CHECK(cost <= reference * (1 + kPipelineSelectionMargin) + 1e-6); + TEST_CHECK(search->costs.size() <= kPipelineQueueEvaluationBudget + && search->packingAttempts <= 4 * kPipelineQueueEvaluationBudget + && search->consideredAssignments <= 4 * kPipelineQueueEvaluationBudget); + size_t depth = 0; + for (size_t i = winner; i; i = search->parents[i]) { + ++depth; + const auto ¤t = search->schedules[i], &parent = search->schedules[search->parents[i]]; + bool cutChanged = false; + for (const auto &node : graph.nodes) + cutChanged |= current.processorStages[current.computeToCpuMap.lookup(node.instance)] + != parent.processorStages[parent.computeToCpuMap.lookup(node.instance)]; + std::vector currentBudgets(3), parentBudgets(3); + for (size_t core = 0; core < target.processorCount; ++core) { + ++currentBudgets[current.processorStages[core]]; + ++parentBudgets[parent.processorStages[core]]; + } + sawJointMove |= cutChanged && currentBudgets != parentBudgets; + } + sawSuccessiveMoves |= depth >= 2; + std::cout << "profile=" << profile << " fixture=" << fixture + << " reference_ns=" << reference << " selected_ns=" << cost + << " gap_percent=" << 100 * (cost / reference - 1) + << " best_evaluated_ns=" << bestEvaluated + << " best_evaluated_gap_percent=" << 100 * (bestEvaluated / reference - 1) + << " reference_evaluations=" << referenceEvaluations + << " search_evaluations=" << search->costs.size() << " refinement_depth=" << depth + << " search_seconds=" << searchSeconds << " reference_seconds=" << referenceSeconds + << " total_seconds=" << std::chrono::duration(std::chrono::steady_clock::now() - begin).count() + << '\n'; + auto unavailable = [&](const MergeScheduleResult &) { WholeScheduleCost value; value.reason = "test unavailable timing"; return value; }; + auto unknown = optimizePipelineSchedule(graph, initial, 3, target, unavailable, error); + TEST_CHECK(search->costs.empty() || (mlir::failed(unknown) && error.find("test unavailable timing") != std::string::npos)); + auto unproved = [&](const MergeScheduleResult &schedule) { + auto cost = evaluate(schedule); + cost.recurrence[0].status = ScheduleRecurrenceEvidence::Unproved; + cost.reason = "test unproved recurrence"; + return cost; + }; + auto invalid = optimizePipelineSchedule(graph, initial, 3, target, unproved, error); + TEST_CHECK(search->costs.empty() || (mlir::failed(invalid) && error.find("test unproved recurrence") != std::string::npos)); + if (!profile && !fixture) { + auto badGuidance = [&](const MergeScheduleResult &schedule) { + auto cost = evaluate(schedule); cost.processorTaskWorkNs.resize(target.processorCount + 1); + return cost; + }; + TEST_CHECK(mlir::failed(optimizePipelineSchedule(graph, initial, 3, target, badGuidance, error))); + TEST_CHECK(error.find("physical processor count") != std::string::npos); + TEST_CHECK(mlir::failed(optimizePipelineSchedule(graph, initial, 3, target, evaluate, error, 0))); + TEST_CHECK(mlir::failed(optimizePipelineSchedule(graph, initial, 3, target, evaluate, error, kPipelineQueueEvaluationBudget + 1))); + } +} + +int main() { + for (size_t profile = 0; profile < 2; ++profile) + for (size_t fixture = 0; fixture < 6; ++fixture) runReference(profile, fixture); + TEST_CHECK(sawJointMove && sawSuccessiveMoves); +} diff --git a/test/PIM/ProjectionMemoryCostTest.cpp b/test/PIM/ProjectionMemoryCostTest.cpp index d4fe0fd..9aa8041 100644 --- a/test/PIM/ProjectionMemoryCostTest.cpp +++ b/test/PIM/ProjectionMemoryCostTest.cpp @@ -74,7 +74,7 @@ static void checkProjection(ArrayRef shape, ArrayRef sizes, target.residentWeightCapacity = 1; target.interProcessorLatencyNs = {0}; function.setName("main_graph"); - auto schedules = prepareSpatialSchedules(*module, target, 1, false); + auto schedules = prepareSpatialSchedules(*module, target, 1); TEST_CHECK(succeeded(schedules) && schedules->candidates.size() == 1); IRMapping mapping; OwningOpRef scheduled = cast(module->getOperation()->clone(mapping)); diff --git a/test/PIM/SpatialSchedulingTargetTest.cpp b/test/PIM/SpatialSchedulingTargetTest.cpp index 39c5973..9500b55 100644 --- a/test/PIM/SpatialSchedulingTargetTest.cpp +++ b/test/PIM/SpatialSchedulingTargetTest.cpp @@ -578,9 +578,6 @@ static void testPipelineStageSpan() { assert(peer != core); assert(schedule.processorStages[core] == schedule.processorStages[peer]); } - for (size_t stage = 0; stage < 2; ++stage) - assert(stageWork[stage] <= 10 * static_cast( - llvm::count(schedule.processorStages, stage))); } } } @@ -1617,6 +1614,57 @@ static void testBaselineConstantMemory() { assert(graph.baselineMemory.serviceNs == 46); } +static void testOrderedPackCostCurve() { + // Heterogeneous weights, reuse, plateaus, capacity cuts, and excess cores. + const std::vector> weights{{0, 1}, {1}, {2}, {}, {0, 2}, {1}}; + const std::vector costs{7, 3, 11, 0, 5, 7}; + const std::vector tasks{0, 1, 2, 3, 4, 5}; + for (size_t capacity : {size_t{2}, size_t{3}}) { + Cost previous = std::numeric_limits::max(); + bool plateau = false; + for (size_t cores = 1; cores <= tasks.size() + 3; ++cores) { + // Enumerate all ordered partitions for an independent small reference. + Cost optimum = std::numeric_limits::max(); + for (size_t cuts = 0; cuts < (size_t{1} << (tasks.size() - 1)); ++cuts) { + size_t packs = 1; + Cost work = 0, maximum = 0; + llvm::SmallBitVector resident(3); + bool fits = true; + for (size_t task : tasks) { + if (task && (cuts & (size_t{1} << (task - 1)))) { + maximum = std::max(maximum, work); + work = 0; resident.reset(); ++packs; + } + work += costs[task]; + for (size_t weight : weights[task]) resident.set(weight); + fits &= resident.count() <= capacity; + } + if (fits && packs <= cores) optimum = std::min(optimum, std::max(maximum, work)); + } + std::vector assignment; + Cost cost = estimateOrderedPackCost(costs, weights, 3, tasks, capacity, cores, &assignment); + if (optimum == std::numeric_limits::max()) { + assert(assignment.empty()); + continue; + } + assert(cost == optimum && cost <= previous && assignment.size() == tasks.size()); + plateau |= cost == previous; + previous = cost; + std::vector loads(cores); + std::vector resident(cores, llvm::SmallBitVector(3)); + for (size_t task : tasks) { + assert(assignment[task] < cores); + loads[assignment[task]] += costs[task]; + for (size_t weight : weights[task]) resident[assignment[task]].set(weight); + } + for (size_t core = 0; core < cores; ++core) { + assert(loads[core] <= cost && resident[core].count() <= capacity); + } + } + assert(plateau && previous == 11); + } +} + static void testPipelineSharedMemory() { mlir::MLIRContext context(mlir::MLIRContext::Threading::DISABLED); auto fragment = mlir::RankedTensorType::get({33}, mlir::IntegerType::get(&context, 8)); @@ -1628,20 +1676,6 @@ static void testPipelineSharedMemory() { assert(shared != getHostPublicationIdentity(firstGeneration.getAsOpaquePointer(), 1, fragment, true)); assert(shared != getHostPublicationIdentity(nextGeneration.getAsOpaquePointer(), 0, fragment, true)); assert(!getHostPublicationIdentity(firstGeneration.getAsOpaquePointer(), 0, fragment, false)); - ScheduleCostSummary previous, candidate; - previous.stageSpanNs = {25000}; - previous.maximumCoreWorkNs = 15000; - previous.baselineMemory.serviceNs = 50000; - candidate.stageSpanNs = {32000}; - candidate.maximumCoreWorkNs = 27000; - candidate.baselineMemory.serviceNs = 40000; - // Reduced memory and stage demand must not hide concentrated core work. - assert(!isBetterPipelineSchedule(candidate, previous)); - candidate.maximumCoreWorkNs = 24000; - assert(isBetterPipelineSchedule(candidate, previous)); - candidate.stageSpanNs = {36000}; - assert(!isBetterPipelineSchedule(candidate, previous)); - assert(!isBetterPipelineSchedule(previous, previous)); ComputeGraph graph; graph.predecessors.resize(4); graph.successors.resize(4); @@ -1679,7 +1713,11 @@ static void testPipelineSharedMemory() { && assignment[task] <= assignment[task - 1] + 1); auto memory = getPipelineBoundaryMemoryDemand(graph, assignment, target); assert(memory.readRequests == 1 && memory.writeRequests == 1); - assert(memory.readBytes == (latency == 1 ? 16384 : 32)); + Cost expected = 0; + for (const auto &transfer : graph.publicationTransfers) + if (assignment[transfer.source] != assignment[transfer.target]) + expected += transfer.bytes * transfer.count; + assert(memory.readBytes == expected); assert(memory.writeBytes == memory.readBytes); assert(schedule.costs.boundaryMemory.serviceNs == memory.serviceNs); } @@ -1787,6 +1825,7 @@ static void testPipelineRestartOrder() { } int main() { + testOrderedPackCostCurve(); testStridedPublicationMemory(); testPipelineSharedMemory(); testBaselineConstantMemory(); diff --git a/test/PIM/test_PimPeriodicDiagnostic.py b/test/PIM/test_PimPeriodicDiagnostic.py index 963acd6..a380cf3 100644 --- a/test/PIM/test_PimPeriodicDiagnostic.py +++ b/test/PIM/test_PimPeriodicDiagnostic.py @@ -170,6 +170,11 @@ def main(): assert vector['per_core'][0]['resource_service_ns'][1] == vector_cost issued = run([[bw, record(10, length=65)]], issue=True) assert issued['per_core'][0]['serial_work_ns'] == vector_cost + 2 * hardware['period'] + omitted = run([[record(21, length=65), bw, record(21, length=65), + record(10, length=65), record(21, length=65)]], issue=True) + assert omitted['per_core'][0]['serial_work_ns'] == issued['per_core'][0]['serial_work_ns'] + assert omitted['per_core'][0]['resource_service_ns'] == issued['per_core'][0]['resource_service_ns'] + assert omitted['opcode_service_ns']['vsoftmax'] == 0 repeated = run([[bw] + [record(10, length=65)] * 10000]) assert repeated['per_core'][0]['serial_work_ns'] == 10000 * vector_cost assert repeated['events'] < 10 @@ -320,12 +325,42 @@ def main(): measured = float(re.search(r'average latency:\s+(\S+)', result.stdout)[1]) * 1e6 estimate = prediction['memory_controller']['interval_ns'] if controller else prediction['interval_estimate_ns'] assert (round(estimate), round(measured)) == expected + + # Omitting Softmax must leave timing, power, and energy unchanged. + baseline = [bw, record(1), load(64), store(64), record(1)] + omitted = [record(21, length=65)] + baseline[:2] + [record(21, length=65)] + baseline[2:] + [record(21, length=65)] + predictions, measurements = [], [] + + def metrics(instructions, omitted_softmax=False): + result = subprocess.run([args.pimsim, str(instructions), str(path), '--gui=false', '--fast'], + capture_output=True, text=True, timeout=30) + assert result.returncode == 0, result.stdout + result.stderr + assert result.stderr.count('Warning: vsoftmax has no timing model;') == int(omitted_softmax) + return tuple(float(re.search(rf'{key}:\s+(\S+)', result.stdout)[1]) + for key in ('average latency', 'average power', 'average energy')) + + for index, program in enumerate((baseline, omitted)): + prediction = run([program], issue=True, controller=True) + predictions.append(prediction['memory_controller']['interval_ns']) + measurements.append(metrics(root, omitted_softmax=bool(index))) + assert predictions[0] == predictions[1] + json_directory = root / 'softmax-json' + json_directory.mkdir(exist_ok=True) + json_program = json_directory / 'program.json' + scalar_json = {'op': 'sldi', 'rd': 0, 'imm': 0} + softmax_json = {'op': 'vsoftmax', 'rd': 0, 'rs1': 0, 'len': 65} + json_program.write_text(json.dumps({'config': json.loads((root / 'config.json').read_text()), + 'core0': [softmax_json, {'op': 'setbw', 'ibiw': 8, 'obiw': 8}, scalar_json, + softmax_json, {'op': 'ld', 'rd': 0, 'rs1': 0, 'size': 64}, + {'op': 'st', 'rd': 0, 'rs1': 0, 'size': 64}, scalar_json, softmax_json]})) + measurements.append(metrics(json_program, omitted_softmax=True)) + assert measurements[0] == measurements[1] == measurements[2] path.write_text(json.dumps(config)) network.write_text(json.dumps(dict(latency=latency))) run([[send(1), recv(1)], [send(0), recv(0)]], valid=False) run([[sync(1, 0)], [wait(0, 2)]], valid=False) - run([[record(21, length=1)]], valid=False) + run([[record(255, length=1)]], valid=False) run([[]], valid=False) unsupported = copy.deepcopy(config) unsupported['chip_config']['core_config']['rob_size'] = 2 diff --git a/test/PIM/test_PimSpatialSelection.py b/test/PIM/test_PimSpatialSelection.py index f2c8065..11926e5 100644 --- a/test/PIM/test_PimSpatialSelection.py +++ b/test/PIM/test_PimSpatialSelection.py @@ -81,7 +81,7 @@ def main(): assert selected["single_candidate_analysis_skipped"] and selected["lowered_candidates"] == 1 assert "spatial_cost" not in selected["candidates"][0] - # An unknown estimate cannot silently exclude any alternative. + # Softmax remains in the functional program but has no oracle timing work. model = onnx.load(vgg) output = model.graph.output[0].name internal = output + "_before_softmax" @@ -95,10 +95,17 @@ def main(): model.graph.node.append(helper.make_node("Softmax", [internal], [output], axis=-1)) source = root / "vgg-softmax.onnx" onnx.save(model, source) + report, program, _ = compile(source) + assert report["generated_candidates"] > 1 and report["status"] == "selected" + assert report["lowered_candidates"] == 1 and program + assert all(candidate["eligible"] for candidate in report["candidates"]) + + # An unknown estimate cannot silently exclude any alternative. + source = REPO / "validation/operations/matmul/yolo_attention/matmul_yolo_attention.onnx" report, program, diagnostics = compile(source, succeeds=False) - assert report["generated_candidates"] > 1 and report["status"] == "incomplete-costs" + assert report["generated_candidates"] >= 1 and report["status"] == "incomplete-costs" assert "cannot compare the complete candidate set" in diagnostics - assert "unsupported abstract operation" in diagnostics + assert "unavailable indirect" in diagnostics assert not program and "backend_attempts" not in report # Latency PEFT and earlier emission do not invoke candidate costing. report, program, _ = compile(vgg, pipeline=1) diff --git a/test/PIM/test_QueueSelection.py b/test/PIM/test_QueueSelection.py index 5f6161c..ea12582 100644 --- a/test/PIM/test_QueueSelection.py +++ b/test/PIM/test_QueueSelection.py @@ -5,6 +5,7 @@ import json import os from pathlib import Path import shutil +import struct import subprocess import tempfile @@ -19,6 +20,59 @@ def check_selected_program(out): return report +def check_synchronization_ablation(root, compiler, env): + vgg = REPO / "validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx" + out = root / "synchronization-ablation" + # Ablation keeps normal selection and costs omitted synchronization separately. + for arch, cores, crossbars in (("arch-a", 168, 96), ("arch-b", 138, 128)): + programs, evidence, schedules = [], [], [] + for disabled in (False, True): + out.mkdir() + command = [compiler, str(vgg), "-o", str(out / "model"), "--maccel=PIM", + "--EmitPimCodegen", "--pipeline=4", f"--core-count={cores}", + f"--crossbar-count={crossbars}", "--crossbar-size=128", + "--mlir-disable-threading", "--pim-detect-communication-deadlock", + f"--pim-target-config={REPO}/validation/pimsim_configs/pimcomp/{arch}/throughput_config_1000ms.json"] + if disabled: + command.append("--pim-disable-synchronization") + result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=90) + assert result.returncode == 0, result.stdout + result.stderr + report = check_selected_program(out) + assert report["generated_candidates"] > 1 + assert bool(report.get("synchronization_ablation")) == disabled + if disabled: + assert report["cost_model_program"] == "synchronized-reference" + assert "report zero-cost SYNC/WAIT separately" in result.stderr + ablation = report["backend_attempts"][0]["synchronization_ablation_cost"] + assert ablation["cost_model_program"] == "synchronization-omitted" + assert not ablation["used_for_selection"] and ablation["diagnostic_only"] + assert all(candidate["eligible"] for candidate in report["candidates"]) + evidence.append((report["selection"], [(c["interval_ns"], c["alternative_tie_interval_ns"], + c["eligible"], c["stage_core_budgets"], c["task_stage_runs"]) + for c in report["candidates"]])) + schedules.append((out / "reports/spatial3_scheduled_no_comm.txt").read_bytes()) + program = {} + sync_count = 0 + for path in (out / "pim").rglob("*"): + if not path.is_file(): + continue + data = path.read_bytes() + if path.suffix == ".pim": + magic, version, count = struct.unpack("<4sII", data[:12]) + assert magic == b"PIMB" and version == 1 and len(data) == 12 + 20 * count + records = [data[i:i + 20] for i in range(12, len(data), 20)] + sync_count += sum(record[0] in (31, 32) for record in records) + data = b"".join(record for record in records if record[0] not in (31, 32)) + program[str(path.relative_to(out / "pim"))] = hashlib.sha256(data).hexdigest() + assert (sync_count == 0) == disabled + programs.append(program) + shutil.rmtree(out) + assert evidence[0] == evidence[1], arch + assert schedules[0] == schedules[1], arch + assert programs[0] == programs[1], arch + + + def main(): parser = argparse.ArgumentParser() parser.add_argument("--compiler", required=True) @@ -87,7 +141,7 @@ def main(): result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=90) assert result.returncode == 0, result.stdout + result.stderr report = check_selected_program(out) - assert report["selection"] != 0 and bool((out / "dialects").exists()) == diagnostics + assert bool((out / "dialects").exists()) == diagnostics identity = {str(p.relative_to(out / "pim")): hashlib.sha256(p.read_bytes()).hexdigest() for p in (out / "pim").rglob("*") if p.is_file()} schedule = (out / "reports/spatial3_scheduled_no_comm.txt").read_bytes() @@ -114,22 +168,23 @@ def main(): "--core-count=168", "--crossbar-count=96", "--crossbar-size=128", "--mlir-disable-threading", f"--pim-target-config={hardware}"] result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=60) - assert result.returncode == 0, result.stdout + result.stderr + assert result.returncode != 0, result.stdout + result.stderr + assert "cannot compare the complete candidate set" in result.stderr report = json.loads((out / "reports/pipeline_selection.json").read_text()) - assert report["selection"] == 0 and report["single_candidate_analysis_skipped"] - assert report["generated_candidates"] == 1 and report["lowered_candidates"] == 1 - assert "no cost comparison required" in report["reason"] - assert report["temporary_disk_bytes"] == 0 and not list(root.glob("candidate*")) + assert report["status"] == "incomplete-costs" and "backend_attempts" not in report + assert "ROB capacity one" in report["reason"] shutil.rmtree(out) out.mkdir() - command[-1] = f"--pim-target-config={config_path}" - command += ["--pim-disable-synchronization"] - result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=60) - assert result.returncode == 0, result.stdout + result.stderr + result = subprocess.run([*command, "--pim-disable-synchronization"], + cwd=root, env=env, capture_output=True, text=True, timeout=60) + assert result.returncode != 0, result.stdout + result.stderr report = json.loads((out / "reports/pipeline_selection.json").read_text()) - assert report["selection"] == 0 and report["single_candidate_analysis_skipped"] - assert report["generated_candidates"] == 1 and report["lowered_candidates"] == 1 - assert "no cost comparison required" in report["reason"] + assert report["status"] == "incomplete-costs" and "backend_attempts" not in report + assert "ROB capacity one" in report["reason"] + shutil.rmtree(out) + + check_synchronization_ablation(root, compiler, env) + assert not (root / "executed").exists(), "production compiler executed a diagnostic or simulator" diff --git a/validation/README.md b/validation/README.md index 0f0a5f8..2ffbd45 100644 --- a/validation/README.md +++ b/validation/README.md @@ -82,9 +82,21 @@ for measurement windows, power, and energy interpretation. Functional compilation/output comparison determines PASS/FAIL. Performance failures show `ERROR` without changing a functional PASS; explicit unsupported -operations show `UNSUPPORTED`. Pimsim's `vsoftmax` is unsupported. +operations show `UNSUPPORTED`. Pimsim-nn omits `vsoftmax` timing and energy; +the functional simulator still executes Softmax. Skipped measurements remain visibly skipped. +Floating-point validation requires both checks for each output: + +- Every element satisfies `abs(actual - reference) <= threshold + relative_threshold * abs(reference)`. +- For a nonzero reference, `norm(actual - reference) / norm(reference) <= relative_threshold`, using the L2 norm. + +The second check prevents a large absolute tolerance from accepting a lost small output. +It does not replace or weaken the elementwise check. Set `--relative-threshold=0` for absolute-only floating-point comparison. +Zero references use the absolute check. Integer and Boolean outputs require exact agreement. +Nonfinite values and mismatched output shapes or counts fail validation. +Reports show maximum absolute differences and relative L2 errors. + Exit codes: `0` for all models passing or successful cleanup; `1` for validation failure, an invalid model directory, or no ONNX files; `2` for invalid arguments. diff --git a/validation/networks/pimcomp_models/README.md b/validation/networks/pimcomp_models/README.md index 3600223..f90115f 100644 --- a/validation/networks/pimcomp_models/README.md +++ b/validation/networks/pimcomp_models/README.md @@ -13,9 +13,9 @@ operations, standalone commands, configuration layout, and GA/OpenMP controls. |---|---|---| | `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. | | `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. | -| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12). | +| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12), with only its two LRN nodes bypassed. | | `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. | -| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Derived from `../yolo11n/depth_51/yolo11n_depth_51.onnx`. | +| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`; this directory retains the historical Softmax-free variant. | The paper's checkpoints were not published. The three Model Zoo networks produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is @@ -23,12 +23,25 @@ suitable for compiler/simulator comparison, not paper-accuracy reproduction. The paper's MNIST description for ResNet-18 differs from the released graph's ImageNet input shape. -Dedicated performance models change the original computation: +The current GoogLeNet comparison uses `googlenet/googlenet-12-no-lrn.onnx`. +Only its two LRN nodes are bypassed because Raptor cannot lower LRN to PIM. +Its terminal Softmax, output `prob_1`, weights, and other operations are retained. +The native reference uses this same graph. VGG8 and both ResNet graphs have no +removed operators; ResNet-34 fixes the symbolic batch size to one. + +Historical performance models change the original computation: - `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal Softmax, which Pimcomp does not schedule. -- `yolo11n/yolo11n-pimsim-nn.onnx` removes two Softmax nodes unsupported by - Pimsim. Use the original YOLO model for functional validation. +- `yolo11n/yolo11n-pimsim-nn.onnx` removes two internal Softmax nodes. It remains + available for reproducing historical measurements. + +The current YOLO comparison uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`, +including both Softmax operations. Its native reference uses the same full model. +The functional simulator executes Softmax. Pimsim-nn skips `vsoftmax` before +fetch, and Raptor's PIM queue estimator omits the same instruction cost. +Timing and energy exclude Softmax work; they do not model hardware Softmax support. +Comparison artifacts remain under this suite's `yolo11n/artifacts/` directory. Model SHA-256 checksums: @@ -36,6 +49,7 @@ Model SHA-256 checksums: 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx +c960c195953423388733cd0e60712cf425ca1f8917e264f5cf98e028752c61cf googlenet/googlenet-12-no-lrn.onnx a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx 229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx diff --git a/validation/networks/pimcomp_models/googlenet/googlenet-12-no-lrn.onnx b/validation/networks/pimcomp_models/googlenet/googlenet-12-no-lrn.onnx new file mode 100644 index 0000000..b7b25c2 Binary files /dev/null and b/validation/networks/pimcomp_models/googlenet/googlenet-12-no-lrn.onnx differ diff --git a/validation/raptor_validation/output_comparison.py b/validation/raptor_validation/output_comparison.py new file mode 100644 index 0000000..5b691cc --- /dev/null +++ b/validation/raptor_validation/output_comparison.py @@ -0,0 +1,44 @@ +from dataclasses import dataclass +import math + +import numpy as np + + +VALIDATION_POLICY = "pointwise-and-relative-l2-v1" + + +@dataclass(frozen=True) +class OutputComparison: + passed: bool + max_abs_diff: float + relative_l2_error: float | None + pointwise_passed: bool + + +def compare_output_arrays(actual, expected, *, atol, rtol): + """Require pointwise tolerance and relative L2 tolerance for nonzero outputs.""" + if any(not math.isfinite(value) or value < 0 for value in (atol, rtol)): + raise ValueError("output tolerances must be finite and non-negative") + actual, expected = np.asarray(actual), np.asarray(expected) + if actual.dtype.kind not in "biuf" or expected.dtype.kind not in "biuf": + raise ValueError("output comparison requires real numeric arrays") + if actual.shape != expected.shape: + raise ValueError(f"output shapes differ: {actual.shape} != {expected.shape}") + if not actual.size: + return OutputComparison(True, 0.0, None, True) + if not np.all(np.isfinite(actual)) or not np.all(np.isfinite(expected)): + return OutputComparison(False, math.inf, None, False) + if expected.dtype.kind in "biu": + difference = np.abs(actual.astype(object) - expected.astype(object)) + passed = bool(np.array_equal(actual, expected)) + return OutputComparison(passed, float(difference.max()), None, passed) + actual, expected = actual.astype(np.float64), expected.astype(np.float64) + with np.errstate(over="ignore", invalid="ignore", divide="ignore"): + difference = np.abs(actual - expected) + pointwise = bool(np.all(difference <= atol + rtol * np.abs(expected))) + scale = float(np.max(np.abs(expected))) + relative = (float(np.linalg.norm(difference / scale) / + np.linalg.norm(expected / scale)) if scale else None) + # A zero reference has no relative scale. rtol=0 explicitly requests absolute-only checks. + relative_passed = relative is None or rtol == 0 or relative <= rtol + return OutputComparison(pointwise and relative_passed, float(difference.max()), relative, pointwise) diff --git a/validation/raptor_validation/pimcomp_models.py b/validation/raptor_validation/pimcomp_models.py index 8dc7853..709ce18 100644 --- a/validation/raptor_validation/pimcomp_models.py +++ b/validation/raptor_validation/pimcomp_models.py @@ -10,11 +10,7 @@ MODELS = { "vgg8": SUITE / "vgg8/vgg8-mnist-reconstructed.onnx", "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx", "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx", - "googlenet": SUITE / "googlenet/googlenet-12-pimsim-nn.onnx", - "yolo11n": SUITE / "yolo11n/yolo11n-pimsim-nn.onnx", -} -FUNCTIONAL_MODELS = { - **MODELS, + "googlenet": SUITE / "googlenet/googlenet-12-no-lrn.onnx", "yolo11n": REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx", } MODEL_NAMES = tuple(MODELS) diff --git a/validation/raptor_validation/pimsim_nn.py b/validation/raptor_validation/pimsim_nn.py index 2e5a387..a1eaf91 100644 --- a/validation/raptor_validation/pimsim_nn.py +++ b/validation/raptor_validation/pimsim_nn.py @@ -27,6 +27,12 @@ def read_raptor_instruction_count(path): def parse_pimsim_nn_metrics(output): metrics = {"raw_output": output} + if "using legacy full-duration throughput result" in output: + metrics["fast_throughput_converged"] = False + metrics["throughput_measurement"] = "full_duration_fallback" + elif "Fast throughput converged; stopping early" in output: + metrics["fast_throughput_converged"] = True + metrics["throughput_measurement"] = "converged_windows" for name, pattern in _METRIC_PATTERNS.items(): match = re.search(pattern, output) if match: diff --git a/validation/raptor_validation/validate_one.py b/validation/raptor_validation/validate_one.py index dbd9c33..b3a2a3a 100644 --- a/validation/raptor_validation/validate_one.py +++ b/validation/raptor_validation/validate_one.py @@ -24,6 +24,7 @@ from .onnx_utils import ( from .raptor import compile_with_raptor from .pimsim_nn import export_raptor_pimsim_artifact, parse_pimsim_nn_metrics, read_raptor_instruction_count from .subprocess_utils import run_command_with_reporter +from .output_comparison import compare_output_arrays STAGE_TITLES = ( "Compile ONNX", @@ -465,34 +466,35 @@ def parse_pim_simulator_outputs(output_bin_path, outputs_descriptor): def validate_outputs(sim_arrays, runner_out_dir, outputs_descriptor, threshold, rtol, verbose): + if len(sim_arrays) != len(outputs_descriptor): + raise ValueError("simulator output count differs from the model output count") all_passed = True rows = [] - for sim_array, (oi, name, _, shape) in zip(sim_arrays, outputs_descriptor): + for sim_array, (oi, name, elem_type, shape) in zip(sim_arrays, outputs_descriptor): csv_name = f"output{oi}_{sanitize_output_name(name)}.csv" - runner_array = np.loadtxt(runner_out_dir / csv_name, delimiter=',', dtype=np.float32).reshape(shape) - sim_array64 = sim_array.astype(np.float64) - runner_array64 = runner_array.astype(np.float64) - abs_diff = np.abs(sim_array64 - runner_array64) - allowed_diff = threshold + rtol * np.abs(runner_array64) - max_diff = float(np.max(abs_diff)) - passed = bool(np.all(abs_diff <= allowed_diff)) - rows.append((name, f"{max_diff:.6e}", passed)) + runner_array = np.loadtxt(runner_out_dir / csv_name, delimiter=',', dtype=_ONNX_TO_NP[elem_type]).reshape(shape) + comparison = compare_output_arrays(sim_array, runner_array, atol=threshold, rtol=rtol) + passed = comparison.passed + relative = comparison.relative_l2_error + rows.append((name, f"{comparison.max_abs_diff:.6e}", + f"{relative:.6e}" if relative is not None else "N/A", passed)) if not passed: all_passed = False - name_width = max(len("Output"), *(len(name) for name, _, _ in rows)) - diff_width = max(len("Max diff"), *(len(diff) for _, diff, _ in rows)) + name_width = max(len("Output"), *(len(name) for name, _, _, _ in rows)) + diff_width = max(len("Max diff"), *(len(diff) for _, diff, _, _ in rows)) + relative_width = max(len("Relative L2"), *(len(relative) for _, _, relative, _ in rows)) result_width = len("Result") - separator = f" +-{'-' * name_width}-+-{'-' * diff_width}-+-{'-' * result_width}-+" + separator = f" +-{'-' * name_width}-+-{'-' * diff_width}-+-{'-' * relative_width}-+-{'-' * result_width}-+" if verbose or not all_passed: print(separator) - print(f" | {'Output'.ljust(name_width)} | {'Max diff'.ljust(diff_width)} | {'Result'} |") + print(f" | {'Output'.ljust(name_width)} | {'Max diff'.ljust(diff_width)} | {'Relative L2'.ljust(relative_width)} | {'Result'} |") print(separator) - for name, diff_text, passed in rows: + for name, diff_text, relative, passed in rows: status_text = ("PASS" if passed else "FAIL").ljust(result_width) status = Fore.GREEN + status_text + Style.RESET_ALL if passed else Fore.RED + status_text + Style.RESET_ALL - print(f" | {name.ljust(name_width)} | {diff_text.ljust(diff_width)} | {status} |") + print(f" | {name.ljust(name_width)} | {diff_text.ljust(diff_width)} | {relative.ljust(relative_width)} | {status} |") print(separator) return all_passed @@ -577,10 +579,9 @@ def validate_execution( f"Latency: {state['metrics']['latency_ms']:.2f} ms" if name == "latency" else f"Throughput: {state['metrics']['throughput']:.2f} samples/s") - energy_unit = "pJ" if name == "latency" else "pJ/it" print_info( - reporter, f"{metric}, Power: {state['metrics']['average_power_mw']:.2f} mW, " - f"Energy: {state['metrics']['average_energy_pj']:.2f} {energy_unit}") + reporter, f"{metric}, Average power: {state['metrics']['average_power_mw']:.2f} mW, " + f"Energy per inference: {state['metrics']['average_energy_pj']:.2f} pJ/inference") except PimSimUnsupportedError as exc: state["pimsim_status"] = PIMSIM_UNSUPPORTED print_info(reporter, str(exc)) diff --git a/validation/tests/test_output_comparison.py b/validation/tests/test_output_comparison.py new file mode 100644 index 0000000..4af5520 --- /dev/null +++ b/validation/tests/test_output_comparison.py @@ -0,0 +1,75 @@ +from pathlib import Path +import sys + +import numpy as np +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from raptor_validation.output_comparison import compare_output_arrays + + +def test_absolute_floor_does_not_accept_lost_small_output(): + result = compare_output_arrays(np.zeros(2), np.array([1e-4, -2e-4]), atol=1e-3, rtol=1e-4) + assert result.pointwise_passed + assert result.relative_l2_error == pytest.approx(1) + assert not result.passed + + +@pytest.mark.parametrize("scale", [1e-200, 1e-4, 1, 1e200]) +def test_small_relative_roundoff_is_independent_of_scale(scale): + expected = np.array([1, -2, 3], dtype=np.float64) * scale + result = compare_output_arrays(expected * (1 + 1e-6), expected, atol=1e-3, rtol=1e-4) + assert result.passed + assert result.relative_l2_error == pytest.approx(1e-6) + + +def test_norm_agreement_cannot_hide_an_incorrect_small_element(): + result = compare_output_arrays([1e6, .01], [1e6, 0.0], atol=1e-3, rtol=1e-4) + assert result.relative_l2_error < 1e-4 + assert not result.pointwise_passed + assert not result.passed + + +@pytest.mark.parametrize("actual,expected", [([np.inf], [1.0]), ([1.0], [np.inf]), + ([np.inf], [np.inf]), ([np.nan], [np.nan])]) +def test_nonfinite_outputs_never_pass(actual, expected): + assert not compare_output_arrays(actual, expected, atol=1e-3, rtol=1e-4).passed + + +@pytest.mark.parametrize("actual,expected", [([1.0], [[1.0]]), ([1.0], [])]) +def test_broadcasting_cannot_hide_output_shape_errors(actual, expected): + with pytest.raises(ValueError, match="shapes differ"): + compare_output_arrays(actual, expected, atol=1e-3, rtol=1e-4) + + +def test_zero_reference_retains_absolute_check(): + result = compare_output_arrays([1e-6], [0.0], atol=1e-3, rtol=1e-4) + assert result.passed and result.relative_l2_error is None + assert not compare_output_arrays([.01], [0.0], atol=1e-3, rtol=1e-4).passed + + +def test_zero_relative_tolerance_requests_absolute_only_check(): + result = compare_output_arrays([0.0], [1e-4], atol=1e-3, rtol=0) + assert result.passed + + +def test_integer_outputs_require_exact_agreement_without_float_rounding(): + expected = np.array([2**53], np.int64) + result = compare_output_arrays(expected + 1, expected, atol=1e-3, rtol=1e-4) + assert not result.passed and result.max_abs_diff == 1 + + +def test_empty_outputs_match(): + assert compare_output_arrays([], [], atol=1e-3, rtol=1e-4).passed + + +@pytest.mark.parametrize("atol,rtol", [(-1, 0), (0, -1), (np.inf, 0), (0, np.nan)]) +def test_invalid_tolerances_are_rejected(atol, rtol): + with pytest.raises(ValueError, match="tolerances"): + compare_output_arrays([0.0], [0.0], atol=atol, rtol=rtol) + + +def test_complex_outputs_are_not_silently_cast_to_real(): + with pytest.raises(ValueError, match="real numeric"): + compare_output_arrays([1 + 2j], [1 + 0j], atol=1e-3, rtol=1e-4) diff --git a/validation/tests/test_pim_experiments.py b/validation/tests/test_pim_experiments.py index a227f8b..489455a 100644 --- a/validation/tests/test_pim_experiments.py +++ b/validation/tests/test_pim_experiments.py @@ -8,6 +8,8 @@ import os from pathlib import Path import sys +import numpy as np +import onnx import pytest @@ -18,6 +20,93 @@ sys.path.insert(0, str(TOOLS / "pimcomp/compare")) import run_ablation as ablation import run_pimcomp_models as suite import compare_raptor_pimcomp_model as comparator +from raptor_validation.validate_one import validate_outputs + + +@pytest.mark.parametrize("mode", ["latency", "throughput"]) +def test_comparison_report_energy_is_per_inference(tmp_path, mode): + report = tmp_path / "comparison_report.md" + validation = comparator.CompareResult(True, {"output": 1e-7}, relative_l2_errors={"output": 1e-6}) + metrics = {"average_power_mw": 2, "average_energy_pj": 30, + "latency_ms": 4, "average_latency_ms": 4, + "throughput": 250, "output_count": 6} + comparator.write_report( + report, model_path=tmp_path / "model.onnx", hardware={}, steps=[], failures=[], + raptor_validation=validation, pimcomp_validation=validation, + raptor_perf=metrics, pimcomp_perf=metrics, raptor_instr={}, pimcomp_instr={}, + raptor_pass_timings={}, pimsim_mode=mode, pimcomp_pipeline="batch", + pimcomp_replication="weight") + text = report.read_text() + assert "Average power (mW) | Energy per inference (pJ/inference)" in text + assert "Avg latency" not in text + assert "Avg energy" not in text + assert metrics["average_energy_pj"] == 30 + assert "Raptor relative L2 | Pimcomp relative L2" in text + assert "1.000000e-06 | 1.000000e-06" in text + + +@pytest.mark.parametrize("zero_output", [False, True]) +def test_comparison_frontends_reject_missing_small_outputs(tmp_path, zero_output): + expected = np.array([[1e-4, -2e-4]], dtype=np.float32) + actual = np.zeros_like(expected) if zero_output else expected * np.float32(1 + 1e-6) + np.savetxt(tmp_path / "output0_output.csv", expected.reshape(-1), delimiter=",") + actual.tofile(tmp_path / "output.bin") + outputs = [(0, "output", 1, [1, 2])] + result = comparator.compare_simulator_outputs( + tmp_path / "output.bin", outputs, tmp_path, threshold=1e-3, rtol=1e-4) + assert result.passed == (not zero_output) + assert result.relative_l2_errors["output"] == pytest.approx(1 if zero_output else 1e-6, rel=.1) + assert validate_outputs([actual], tmp_path, outputs, 1e-3, 1e-4, False) == (not zero_output) + + +def test_comparison_preserves_double_reference_precision(tmp_path): + expected = np.array([1 + 2**-40], dtype=np.float64) + np.savetxt(tmp_path / "output0_output.csv", expected, delimiter=",") + expected.tofile(tmp_path / "output.bin") + outputs = [(0, "output", 11, [1])] + assert comparator.compare_simulator_outputs( + tmp_path / "output.bin", outputs, tmp_path, threshold=0, rtol=0).passed + assert validate_outputs([expected], tmp_path, outputs, 0, 0, False) + + +def test_relative_agreement_only_reports_possible_numerical_sensitivity(tmp_path): + expected = np.array([1e6, 0], dtype=np.float32) + actual = np.array([1e6, .01], dtype=np.float32) + np.savetxt(tmp_path / "output0_output.csv", expected, delimiter=",") + actual.tofile(tmp_path / "output.bin") + result = comparator.compare_simulator_outputs( + tmp_path / "output.bin", [(0, "output", 1, [2])], tmp_path, threshold=1e-3, rtol=1e-4) + assert not result.passed + assert result.relative_l2_errors["output"] < 1e-4 + assert "possible numerical sensitivity, not proof of correctness" in result.error + + +def test_comparison_rejects_missing_output_arrays(tmp_path, monkeypatch): + outputs = [(0, "output", 1, [1])] + monkeypatch.setattr(comparator, "parse_pim_simulator_outputs", lambda *args: []) + with pytest.raises(ValueError, match="output count"): + comparator.compare_simulator_outputs(tmp_path / "missing.bin", outputs, tmp_path, threshold=1e-3, rtol=1e-4) + with pytest.raises(ValueError, match="output count"): + validate_outputs([], tmp_path, outputs, 1e-3, 1e-4, False) + + +@pytest.mark.parametrize("raptor_pass,pimcomp_pass,winner", [ + (True, True, "pimcomp"), (True, False, "NA"), + (False, True, "NA"), (False, False, "NA"), +]) +def test_timing_winner_requires_both_functional_passes(tmp_path, raptor_pass, pimcomp_pass, winner): + report = suite.result_dir(tmp_path, "vgg8", "arch-a", "latency", 1) / "pimcomp/comparison_report.json" + report.parent.mkdir(parents=True) + data = {"validation_policy": suite.VALIDATION_POLICY} + for side, passed, latency in [("raptor", raptor_pass, 10), ("pimcomp", pimcomp_pass, 1)]: + data[side + "_validation"] = {"status": "done", "passed": passed} + data[side + "_performance"] = {"latency_ms": latency, "output_count": 1} + report.write_text(json.dumps(data)) + output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("latency", 1, "element"),)) + with output.open() as stream: + row = next(csv.DictReader(stream)) + assert row["better_compiler"] == winner + assert row["speedup"] == "10.00" # Diagnostic timing remains available. def test_ablation_results_stay_in_independent_output_roots(tmp_path): @@ -30,7 +119,7 @@ def test_ablation_results_stay_in_independent_output_roots(tmp_path): path.parent.mkdir(parents=True) row = {"arch": "arch-a", "model": "vgg8", "mode": "latency", "raptor_pipeline": "1", "raptor_functional_validation": "PASS"} - row.update({field: 100 if variant.name == "baseline" else value + row.update({field: value if variant.name == "baseline" else 100 for field, _ in ablation.PERCENTAGE_FIELDS}) with path.open("w", newline="") as stream: writer = csv.DictWriter(stream, fieldnames=list(row)) @@ -40,11 +129,124 @@ def test_ablation_results_stay_in_independent_output_roots(tmp_path): assert failures == [] assert output == root / "results_ablation.csv" outputs.append(output) - for output, change in zip(outputs, ("-20.00%", "-40.00%")): + for output, change in zip(outputs, ("+20.00%", "+40.00%")): with output.open(newline="") as stream: rows = list(csv.DictReader(stream)) - assert rows[0]["latency"] == "+0.00%" - assert rows[1]["latency"] == change + assert len(rows) == 1 + assert rows[0]["comparison"] == "plan" + assert rows[0]["baseline_variant"] == "no-plan" + assert rows[0]["latency_gain_percent"] == change + + +@pytest.mark.parametrize("options,expected", [ + ([], {("baseline", "latency", "1"), ("no-plan", "latency", "1"), + ("baseline", "throughput", "4"), ("no-plan", "throughput", "4"), + ("no-sync", "throughput", "4")}), + (["--mode", "latency"], {("baseline", "latency", "1"), ("no-plan", "latency", "1")}), + (["--pipeline=8"], {(variant, "throughput", "8") for variant in ("baseline", "no-plan", "no-sync")}), + (["--mode", "latency", "throughput", "--pipeline", "2"], + {(variant, "throughput", "2") for variant in ("baseline", "no-plan", "no-sync")}), +]) +def test_ablation_runs_only_required_variants(options, expected, capsys): + import shlex + + assert ablation.main(["--dry-run", *options]) == 0 + commands = [shlex.split(line) for line in capsys.readouterr().out.splitlines()] + actual = set() + for command in commands: + variant = command[command.index("--ablation-variant") + 1] + actual.add((variant, *ablation.case_key(command))) + assert "--raptor-only" in command + assert ("--raptor-extra-arg=--pim-disable-synchronization" in command) == (variant == "no-sync") + assert ("--raptor-extra-arg=--pim-disable-spatial-planning" in command) == (variant == "no-plan") + assert actual == expected + assert len(commands) == len(expected) + + +def test_ablation_rejects_combined_disabled_variant(): + with pytest.raises(SystemExit): + ablation.parse_args(["--variant", "no-sync-no-plan"]) + + +def test_ablation_reports_gains_against_each_disabled_feature(tmp_path): + variants = ablation.selected_variants(None) + for variant in variants: + path = ablation.variant_output_dir(tmp_path, variant) / "results_comparison.csv" + path.parent.mkdir(parents=True) + rows = [] + for mode, pipeline in (("latency", "1"), ("throughput", "4")): + if variant.name == "no-sync" and mode == "latency": + continue + rows.append({"arch": "arch-b", "model": "vgg8", "mode": mode, + "raptor_pipeline": pipeline, + "raptor_functional_validation": "FAIL" if variant.name == "no-sync" else "PASS", + "raptor_latency_ms": {"baseline": "8", "no-plan": "10", "no-sync": "6.4"}[variant.name], + "raptor_throughput_samples_s": {"baseline": "125", "no-plan": "100", "no-sync": "156.25"}[variant.name], + "raptor_power_mw": "100", "raptor_energy_pj": "100"}) + with path.open("w", newline="") as stream: + writer = csv.DictWriter(stream, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + output, failures = ablation.aggregate_results(tmp_path, variants) + assert not failures + with output.open(newline="") as stream: + rows = list(csv.DictReader(stream)) + assert len(rows) == 3 + latency, plan, sync = rows + assert (latency["comparison"], latency["latency_gain_percent"]) == ("plan", "+20.00%") + assert (plan["comparison"], plan["throughput_gain_percent"]) == ("plan", "+25.00%") + assert (sync["comparison"], sync["throughput_gain_percent"]) == ("sync", "-20.00%") + assert plan["baseline_variant"] == "no-plan" + assert sync["baseline_variant"] == "no-sync" + assert sync["baseline_functional_validation"] == "FAIL" + assert sync["functional_validation"] == "PASS" + + +@pytest.mark.parametrize("variant", [None, "baseline", "no-plan", "no-sync"]) +def test_unconverged_fast_throughput_is_available_in_comparison_and_ablation(tmp_path, variant): + raw = """Fast throughput convergence failed before sim_time; using legacy full-duration throughput result +Chip Simulation Result: + - output count: 22 samples + - throughput: 22 samples/s + - average latency: 45.45454545 ms + - average power: 300 mW + - average energy: 10000000000 pJ/it +""" + metrics = comparator.parse_pimsim_nn_metrics(raw) + assert metrics["fast_throughput_converged"] is False + assert metrics["throughput_measurement"] == "full_duration_fallback" + assert "unconverged" in comparator.perf_status(metrics) + report = suite.result_dir(tmp_path, "vgg8", "arch-b", "throughput", 4, variant) / "pimcomp/comparison_report.json" + report.parent.mkdir(parents=True) + data = {"validation_policy": suite.VALIDATION_POLICY} + for side in ("raptor", "pimcomp"): + data[side + "_validation"] = {"status": "done", "passed": True} + # Older reports can contain only the raw simulator output. + data[side + "_performance"] = {"raw_output": raw, "fast_throughput_converged": False} + report.write_text(json.dumps(data)) + output = suite.write_results_csv(tmp_path, "arch-b", ["vgg8"], + (("throughput", 4, "batch"),), ablation_variant=variant) + with output.open(newline="") as stream: + row = next(csv.DictReader(stream)) + for side in ("raptor", "pimcomp"): + assert float(row[side + "_throughput_samples_s"]) == 22 + assert float(row[side + "_latency_ms"]) == pytest.approx(1000 / 22) + if variant is not None: + enabled = ablation.variant_output_dir(tmp_path, ablation.VARIANT_BY_NAME["baseline"]) / "results_comparison.csv" + enabled.parent.mkdir(parents=True, exist_ok=True) + enabled.write_text(output.read_text()) + aggregate, failures = ablation.aggregate_results(tmp_path, ablation.selected_variants([variant])) + assert not failures + with aggregate.open(newline="") as stream: + for gain in csv.DictReader(stream): + assert gain["throughput_gain_percent"] == "+0.00%" + + +def test_fast_throughput_convergence_metadata(): + metrics = comparator.parse_pimsim_nn_metrics("Fast throughput converged; stopping early\nthroughput: 100 samples/s") + assert metrics["fast_throughput_converged"] is True + assert metrics["throughput_measurement"] == "converged_windows" + assert metrics["throughput"] == 100 @pytest.fixture @@ -56,7 +258,9 @@ def recorded_failure(tmp_path): pimcomp_config=config, pimsim_mode="throughput", pimsim_time_ms=1000, pimcomp_pipeline="batch", pimcomp_replication="GA", pimcomp_model_name=None, seed=0, batch_size=64, threshold=1e-3, rtol=1e-4, no_fast=False, - skip_pimsim_nn=False, + skip_pimsim_nn=False, pimcomp_dir=tmp_path / "pimcomp", + pimsim_nn_build_dir=tmp_path / "oracle", pim_simulator_dir=tmp_path / "functional", + raptor_path=tmp_path / "raptor", raptor_extra_arg=["--pipeline=2"], ) hardware = {"core_count": 168, "crossbar_size": 128} context = comparator.pimcomp_reuse_context(args, model, functional, hardware) @@ -66,7 +270,7 @@ def recorded_failure(tmp_path): "error": "Unsupported model"}, "pimcomp_performance": {"skipped": True, "reason": "Compilation failed"}, "pimcomp_instruction_summary": comparator.empty_instruction_summary("No artifact"), - "paths": {}, "steps": [], + "paths": {}, "steps": [], "artifact_signatures": {"pimcomp": {}}, "failures": [{"stage": "Compile Pimcomp frontend", "error": "Unsupported model"}], } path = tmp_path / "comparison_report.json" @@ -84,6 +288,117 @@ def test_failed_pimcomp_report_needs_no_simulation_artifacts(recorded_failure): assert failures == report["failures"] +@pytest.mark.parametrize("tool", ["compiler", "oracle", "exporter"]) +def test_reuse_checks_tool_content(recorded_failure, tool): + path, report, args, model, functional, hardware = recorded_failure + original = report["pimcomp_reuse_context"]["files"][tool]["path"] + tool_path = Path(original) + tool_path.parent.mkdir(parents=True, exist_ok=True) + tool_path.write_bytes(b"changed executable or exporter") + expected = comparator.pimcomp_reuse_context(args, model, functional, hardware) + with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"): + comparator.load_reused_pimcomp_report(path, expected) + + +@pytest.mark.parametrize("mutation", ["instructions", "weights", "missing"]) +def test_reuse_checks_artifact_content(recorded_failure, mutation): + path, report, *_ = recorded_failure + artifact = path.parent / "artifact" + artifact.mkdir() + (artifact / "core_0.json").write_text('[{"op":"nop"}]') + (artifact / "weights.bin").write_bytes(b"weights") + report["paths"]["pimcomp_exported_pim"] = str(artifact) + report["artifact_signatures"]["pimcomp"] = comparator.artifact_signatures(report, "pimcomp") + path.write_text(json.dumps(report)) + assert comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"]) + if mutation == "missing": + (artifact / "core_0.json").unlink() + elif mutation == "weights": + (artifact / "weights.bin").write_bytes(b"wrong weights") + else: + (artifact / "core_0.json").write_text('[{"op":"wait"}]') + with pytest.raises(ValueError, match="artifacts differ"): + comparator.load_reused_pimcomp_report(path, report["pimcomp_reuse_context"]) + + +def test_raptor_context_covers_pipeline_and_ignores_pimcomp_pipeline(recorded_failure): + _, _, args, model, functional, hardware = recorded_failure + expected = comparator.experiment_context(args, model, functional, hardware, "raptor") + args.pimcomp_pipeline = "element" + assert expected == comparator.experiment_context(args, model, functional, hardware, "raptor") + args.raptor_extra_arg = ["--pipeline=8"] + assert expected != comparator.experiment_context(args, model, functional, hardware, "raptor") + + +def test_reuse_rejects_previous_validation_policy(recorded_failure): + path, report, *_ = recorded_failure + context = copy.deepcopy(report["pimcomp_reuse_context"]) + del report["pimcomp_reuse_context"]["validation_policy"] + path.write_text(json.dumps(report)) + with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"): + comparator.load_reused_pimcomp_report(path, context) + + +def test_raptor_reuse_rejects_previous_validation_policy(recorded_failure, monkeypatch): + path, _, _, model, _, _ = recorded_failure + path.write_text('{"validation_policy": "legacy"}') + monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {}) + monkeypatch.setattr(comparator, "load_model_inputs", lambda *args: ([], [], [])) + monkeypatch.setattr(sys, "argv", [ + "compare_raptor_pimcomp_model.py", "--model", str(model), + "--out-dir", str(path.parent / "out"), "--pimcomp-config", str(path.parent / "config.json"), + "--reuse-raptor-report", str(path), + ]) + with pytest.raises(ValueError, match="validation policy differs"): + comparator.main() + + +@pytest.mark.parametrize("tolerances", [None, {"atol": 0.01, "rtol": 0.0001}, + {"atol": 0.001, "rtol": 0.01}]) +def test_raptor_reuse_checks_actual_tolerances(recorded_failure, monkeypatch, tolerances): + path, _, _, model, _, _ = recorded_failure + path.write_text(json.dumps({"validation_policy": comparator.VALIDATION_POLICY, + "validation_tolerances": tolerances})) + monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {}) + monkeypatch.setattr(comparator, "load_model_inputs", lambda *args: ([], [], [])) + monkeypatch.setattr(sys, "argv", [ + "compare_raptor_pimcomp_model.py", "--model", str(model), + "--out-dir", str(path.parent / "out"), "--pimcomp-config", str(path.parent / "config.json"), + "--reuse-raptor-report", str(path), + ]) + with pytest.raises(ValueError, match="validation tolerances differ"): + comparator.main() + + +def test_failed_raptor_compilation_is_not_reported_as_skipped(tmp_path, monkeypatch): + model = tmp_path / "model.onnx" + model.write_bytes(b"test model") + runner = tmp_path / "runner" + runner.touch() + monkeypatch.setattr(comparator, "load_effective_hardware", lambda args: {"core_count": 1}) + monkeypatch.setattr(comparator, "load_model_inputs", lambda *args: + ([(0, "input", 1, [1])], [(0, "output", 1, [1])], [np.zeros(1, np.float32)])) + monkeypatch.setattr(comparator, "ensure_reference_runner", lambda *args: runner) + monkeypatch.setattr(comparator, "generate_reference_outputs", lambda *args, **kwargs: tmp_path) + + def fail_compile(*args): + raise RuntimeError("unsupported test operation") + + monkeypatch.setattr(comparator, "compile_raptor_target", fail_compile) + out = tmp_path / "out" + monkeypatch.setattr(sys, "argv", [ + "compare_raptor_pimcomp_model.py", "--model", str(model), "--out-dir", str(out), + "--raptor-only", "--skip-pimsim-nn", "--fail-on-error", + ]) + with pytest.raises(SystemExit) as exit_code: + comparator.main() + assert exit_code.value.code == 1 + report = json.loads((out / "pimcomp/comparison_report.json").read_text()) + assert report["raptor_validation"]["status"] == "failed" + assert not report["raptor_validation"]["passed"] + assert "unsupported test operation" in report["failures"][0]["error"] + + @pytest.mark.parametrize("setting,value", [ ("pimsim_mode", "latency"), ("pimcomp_pipeline", "element"), ("pimsim_time_ms", 2000), ("pimcomp_replication", "balance"), @@ -177,6 +492,65 @@ def test_missing_pimcomp_report_is_not_reused(tmp_path): assert not suite.pimcomp_report_ready(tmp_path / "output", None) +@pytest.mark.parametrize("model", ["yolo11n", "googlenet"]) +def test_comparison_uses_same_model_for_reference_and_execution(tmp_path, monkeypatch, model): + commands = [] + monkeypatch.setattr(suite, "run", lambda command, **kwargs: commands.append(command) or 0) + monkeypatch.setattr(sys, "argv", [ + "run_pimcomp_models.py", "--out-dir", str(tmp_path), "--models", model, + "--archs", "arch-b", "--mode", "throughput", "--pipeline", "8", + "--raptor-only", "--dry-run", + ]) + assert suite.main() == 0 + reference, comparison = commands + expected = str(suite.MODELS[model]) + assert reference[reference.index("--model") + 1] == expected + assert comparison[comparison.index("--model") + 1] == expected + assert comparison[comparison.index("--functional-model") + 1] == expected + assert Path(comparison[comparison.index("--common-dir") + 1]).name == "common-full-model" + + +def test_yolo_uses_full_model_without_moving_comparison_artifacts(): + assert suite.MODELS["yolo11n"] == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx" + assert suite.model_dir(None, "yolo11n") == suite.SUITE / "yolo11n" + assert suite.common_dir(None, "yolo11n") == suite.SUITE / "yolo11n/artifacts/common-full-model" + + +def test_googlenet_removes_only_lrn(): + original = onnx.load(suite.SUITE / "googlenet/googlenet-12.onnx") + actual = onnx.load(suite.MODELS["googlenet"]) + removed = {node.output[0]: node.input[0] for node in original.graph.node if node.op_type == "LRN"} + assert len(removed) == 2 + retained = [] + for node in original.graph.node: + if node.op_type == "LRN": + continue + for i, value in enumerate(node.input): + while value in removed: + value = removed[value] + node.input[i] = value + retained.append(copy.deepcopy(node)) + del original.graph.node[:] + original.graph.node.extend(retained) + assert actual.SerializeToString() == original.SerializeToString() + assert sum(node.op_type == "Softmax" for node in actual.graph.node) == 1 + assert [output.name for output in actual.graph.output] == ["prob_1"] + onnx.checker.check_model(actual) + assert suite.common_dir(None, "googlenet").name == "common-full-model" + + +@pytest.mark.parametrize("model,counts", [ + ("yolo11n", {"Softmax": 2, "Split": 11, "Resize": 2}), + ("resnet18", {"BatchNormalization": 20, "Conv": 20}), + ("resnet34", {"BatchNormalization": 36, "Conv": 36}), + ("vgg8", {"Conv": 6, "Gemm": 2, "Relu": 7, "MaxPool": 3}), +]) +def test_comparison_models_retain_required_operations(model, counts): + graph = onnx.load(suite.MODELS[model]).graph + for operation, expected in counts.items(): + assert sum(node.op_type == operation for node in graph.node) == expected + + def test_missing_pimcomp_report_is_rejected_at_load(recorded_failure): path, report, *_ = recorded_failure with pytest.raises(ValueError, match="Cannot reuse Pimcomp report"): @@ -341,7 +715,8 @@ def test_zero_outputs_preserve_only_measured_average_power(): def test_functional_pass_survives_unavailable_performance(tmp_path): report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json" report.parent.mkdir(parents=True) - report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True}, + report.write_text(json.dumps({"validation_policy": suite.VALIDATION_POLICY, + "raptor_validation": {"status": "done", "passed": True}, "raptor_performance": {"error": "timeout", "throughput": 0}})) output = suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),)) with output.open(newline="") as stream: @@ -350,6 +725,15 @@ def test_functional_pass_survives_unavailable_performance(tmp_path): assert all(row[field] == "NA" for field, _ in ablation.PERCENTAGE_FIELDS) +def test_aggregate_rejects_previous_validation_policy(tmp_path): + report = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4) / "pimcomp/comparison_report.json" + report.parent.mkdir(parents=True) + report.write_text(json.dumps({"raptor_validation": {"status": "done", "passed": True}})) + with pytest.raises(ValueError, match="different validation policy"): + suite.write_results_csv(tmp_path, "arch-a", ["vgg8"], (("throughput", 4, "batch"),)) + assert not (tmp_path / "results_comparison.csv").exists() + + def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path): variants = ablation.selected_variants(["no-plan"]) for variant in variants: @@ -368,5 +752,7 @@ def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path): assert not failures and output == destination with output.open(newline="") as stream: rows = list(csv.DictReader(stream)) - assert rows[1]["functional_validation"] == "PASS" - assert all(rows[1][metric] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS) + assert len(rows) == 1 + assert rows[0]["functional_validation"] == "PASS" + assert rows[0]["baseline_functional_validation"] == "PASS" + assert all(rows[0][f"{metric}_gain_percent"] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS) diff --git a/validation/tests/test_pim_plots.py b/validation/tests/test_pim_plots.py index 954700d..76adf46 100644 --- a/validation/tests/test_pim_plots.py +++ b/validation/tests/test_pim_plots.py @@ -53,13 +53,167 @@ def test_comparison_plot_cli_input_paths(comparison, tmp_path, monkeypatch, cust raise InputCaptured monkeypatch.setattr(comparison, "load_results", capture) - monkeypatch.setattr(comparison, "configure_style", lambda: None) + monkeypatch.setattr(comparison, "configure_style", lambda **kwargs: None) monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--input", str(expected)] if custom else [])]) with pytest.raises(InputCaptured): comparison.main() assert seen == [expected] +@pytest.mark.parametrize("custom", [False, True]) +def test_comparison_plot_cli_output_paths(comparison, tmp_path, monkeypatch, custom): + expected = tmp_path / "custom" if custom else TOOLS / "pimcomp/compare/plots/default" + seen = [] + + class OutputCaptured(Exception): + pass + + def capture(rows, output): + seen.append(output) + raise OutputCaptured + + monkeypatch.setattr(comparison, "draw_validation", capture) + monkeypatch.setattr(sys, "argv", ["generate_plots.py", *(["--output-dir", str(expected)] if custom else [])]) + with pytest.raises(OutputCaptured): + comparison.main() + assert seen == [expected] + + +@pytest.mark.parametrize("flags,colors,anonymize", [([], True, False), + (["--colors", "--no-anonymize"], True, False), (["--no-colors"], False, False), + (["--anonymize"], True, True), (["--no-colors", "--anonymize"], False, True), + (["--no-colors", "--colors", "--anonymize", "--no-anonymize"], True, False)]) +def test_plot_display_flags(comparison, monkeypatch, flags, colors, anonymize): + captured = [] + + class StyleCaptured(Exception): + pass + + def capture(**options): + captured.append(options) + raise StyleCaptured + + monkeypatch.setattr(comparison, "configure_style", capture) + monkeypatch.setattr(sys, "argv", ["generate_plots.py", *flags]) + with pytest.raises(StyleCaptured): + comparison.main() + assert captured == [{"colors": colors, "anonymize": anonymize}] + + +@pytest.mark.parametrize("flags,borders", [([], True), (["--marker-borders"], True), + (["--no-marker-borders"], False), (["--no-marker-borders", "--marker-borders"], True)]) +def test_marker_border_cli_flags(comparison, monkeypatch, flags, borders): + captured = [] + + def capture(*args, marker_borders): + captured.append(marker_borders) + + for name in ("draw_validation", "draw_metric", "draw_absolute"): + monkeypatch.setattr(comparison, name, lambda *args, **kwargs: None) + for name in ("draw_compact_summary", "draw_pipeline_effect"): + monkeypatch.setattr(comparison, name, capture) + monkeypatch.setattr(sys, "argv", ["generate_plots.py", *flags]) + comparison.main() + assert captured == [borders, borders] + + +@pytest.mark.parametrize("borders", [True, False]) +def test_marker_borders_reach_data_legends_and_svg(comparison, tmp_path, monkeypatch, borders): + from xml.etree import ElementTree as ET + from matplotlib.lines import Line2D + from matplotlib.colors import to_rgba + + comparison.configure_style() + rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + save = comparison.save_figure + + def capture(fig, name, output): + try: + size = 6.5 if name == "performance_energy_compact" else 7 + for ax in fig.axes: + points = [line for line in ax.lines if line.get_marker() in ("o", "s", "^", "D")] + assert points + for point in points: + assert point.get_markersize() == size + assert (point.get_markeredgewidth() > 0) == borders + assert to_rgba(point.get_markerfacecolor())[3] == 1 + if not borders: + assert to_rgba(point.get_markerfacecolor()) == to_rgba(point.get_markeredgecolor()) + handles = [handle for handle in fig.legends[0].legend_handles if isinstance(handle, Line2D)] + assert handles + for handle in handles: + assert handle.get_markersize() == size + assert (handle.get_markeredgewidth() > 0) == borders + if not borders: + assert to_rgba(handle.get_markerfacecolor()) not in ((1, 1, 1, 1), (0, 0, 0, 0)) + assert to_rgba(handle.get_markerfacecolor()) == to_rgba(handle.get_markeredgecolor()) + save(fig, name, output) + finally: + comparison.plt.close(fig) + root = ET.parse(output / f"svg/{name}.svg") + paths = [node for node in root.findall('.//{http://www.w3.org/2000/svg}defs/{http://www.w3.org/2000/svg}path') + if node.get("id", "").startswith("m")] + assert paths + assert all(("stroke:" in node.get("style", "")) == borders for node in paths) + + monkeypatch.setattr(comparison, "save_figure", capture) + comparison.draw_compact_summary(rows, tmp_path, marker_borders=borders) + comparison.draw_pipeline_effect(rows, tmp_path, marker_borders=borders) + + +@pytest.mark.parametrize("colors", [True, False]) +@pytest.mark.parametrize("anonymize", [True, False]) +def test_plot_display_variant_exports(comparison, tmp_path, colors, anonymize): + import re + from PIL import Image, ImageChops + + try: + comparison.configure_style(colors=colors, anonymize=anonymize) + rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + comparison.draw_compact_summary(rows, tmp_path) + selected = comparison.select_compact_speedups([row for row in rows if row.model != "yolo11n"]) + comparison.draw_metric(selected, "speedup", "Speedup (x)", "speedup_compact_comparison", + tmp_path, compact=True) + for name in ("performance_energy_compact", "speedup_compact_comparison"): + svg = (tmp_path / f"svg/{name}.svg").read_text() + if name == "performance_energy_compact": + assert ("Anonimized better" in svg) == anonymize + assert ("Raptor better" in svg) == (not anonymize) + svg_colors = re.findall(r"(?:fill|stroke):\s*#([0-9a-fA-F]{6})", svg) + assert svg_colors + assert any(color[:2] != color[2:4] or color[2:4] != color[4:] for color in svg_colors) == colors + with Image.open(tmp_path / f"png/{name}.png") as picture: + red, green, blue = picture.convert("RGB").split() + chromatic = (ImageChops.difference(red, green).getbbox() is not None or + ImageChops.difference(green, blue).getbbox() is not None) + assert chromatic == colors + assert (tmp_path / f"pdf/{name}.pdf").read_bytes().startswith(b"%PDF-") + finally: + comparison.configure_style() + + +def test_anonymization_reaches_all_plot_labels(comparison, monkeypatch): + from matplotlib.text import Text + + captured = {} + + def capture(fig, name, output): + captured[name] = [text.get_text() for text in fig.findobj(Text)] + comparison.plt.close(fig) + + monkeypatch.setattr(comparison, "save_figure", capture) + monkeypatch.setattr(sys, "argv", ["generate_plots.py", "--anonymize"]) + try: + comparison.main() + assert len(captured) == 30 + assert all("Raptor" not in label for labels in captured.values() for label in labels) + assert "Anonimized pipeline stages" in captured["raptor_pipeline_throughput"] + assert "Anonimized better →" in captured["performance_energy_compact"] + assert "Anonimized\n2 stages" in captured["validation_status"] + finally: + comparison.configure_style() + + def test_exact_comparison_configuration_matrix(comparison, comparison_csv, tmp_path): fields, rows = comparison_csv results = comparison.load_results(write_csv(tmp_path / "results.csv", fields, rows)) @@ -185,7 +339,7 @@ def test_compact_validation_combines_pipeline_statuses(comparison, tmp_path, mon assert [label.get_text() for label in ax.get_yticklabels()] == ([ "VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet", "YOLO11n"] if index == 0 else []) assert len(ax.patches) == len(ax.texts) == 4 * len(comparison.MODELS) - assert [label.get_text() for label in ax.get_xticklabels()] == ["Latency", "Throughput"] + assert [label.get_text() for label in ax.get_xticklabels()] == ["Latency-mode", "Throughput-mode"] assert all(text.get_fontsize() == 12 and text.get_weight() == "normal" for text in ax.texts) assert all(text.get_color() == "#1D2530" for text in ax.texts) ax = fig.axes[0] @@ -211,15 +365,21 @@ def test_compact_speedups_select_best_pipeline(comparison, rates, stage): @pytest.mark.parametrize("diverging", [False, True]) @pytest.mark.parametrize("metric,ylabel", [("speedup", "Speedup (x)"), - ("energy", "Energy-per-inference advantage (x)"), ("power", "Average power advantage (x)")]) + ("energy", "Energy reduction (x)"), ("power", "Average power reduction (x)")]) def test_compact_metric_bars_and_units(comparison, tmp_path, monkeypatch, diverging, metric, ylabel): from matplotlib.colors import to_rgba template = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")[0] - rows = [replace(template, arch=arch, mode=mode, pipeline=stage, latency_speedup=0.5, - energy=0.5 if mode == "latency" else 3, power=0.5 if mode == "latency" else 3, - raptor_throughput=300, pimcomp_throughput=100, pimcomp_validation="FAIL") - for arch in comparison.ARCHITECTURES for mode, stage in (("latency", 1), ("throughput", 4))] + rows = [replace(template, model=model, arch=arch, mode=mode, pipeline=stage, + latency_speedup=0.5 if arch == "arch-a" else 0.25, + energy=(0.5 if arch == "arch-a" else 0.25) if mode == "latency" else + (3 if arch == "arch-a" else 4), + power=(0.5 if arch == "arch-a" else 0.25) if mode == "latency" else + (3 if arch == "arch-a" else 4), + raptor_throughput=300 if arch == "arch-a" else 400, + pimcomp_throughput=100, pimcomp_validation="FAIL") + for arch in reversed(comparison.ARCHITECTURES) for model in ("resnet18", "vgg8") + for mode, stage in (("throughput", 4), ("latency", 1))] captured = [] monkeypatch.setattr(comparison, "save_figure", lambda fig, name, path: captured.append(fig)) comparison.draw_metric(rows, metric, ylabel, "compact", tmp_path, @@ -227,30 +387,84 @@ def test_compact_metric_bars_and_units(comparison, tmp_path, monkeypatch, diverg fig = captured[0] try: assert fig.get_size_inches() == pytest.approx((8.5, 4.4)) - expected_legend = { - "speedup": ["Single-inference latency", "Best-case throughput"], - "energy": ["Single-inference energy", "Best-case throughput energy"], - "power": ["Single-inference power", "Best-case throughput power"], + expected_panels = { + "speedup": ["Latency-mode", "Throughput-mode"], + "energy": ["Latency-mode energy", "Throughput-mode energy"], + "power": ["Latency-mode power", "Throughput-mode power"], } - assert [text.get_text() for text in fig.legends[0].get_texts()] == expected_legend[metric] + assert [text.get_text() for text in fig.legends[0].get_texts()] == ["Arch-A", "Arch-B"] for index, ax in enumerate(fig.axes): - latency, throughput = (container.patches[0] for container in ax.containers) + bars = [container.patches[0] for container in ax.containers] + assert len(bars) == 4 + assert ax.get_xlabel() == expected_panels[metric][index] + assert [text.get_text() for text in ax.get_xticklabels()] == ["VGG-8", "ResNet-18"] + assert list(ax.get_xticks()) == [0.5, 3.5] lower, upper = ax.get_xlim() - assert latency.get_x() - lower == pytest.approx(0.4) - assert upper - throughput.get_x() - throughput.get_width() == pytest.approx(0.4) - assert latency.get_x() + latency.get_width() == throughput.get_x() - assert latency.get_facecolor() == to_rgba(comparison.METRIC_STYLES["latency"][0]) - assert throughput.get_facecolor() == to_rgba(comparison.METRIC_STYLES["throughput"][0]) - assert latency.get_edgecolor() == to_rgba(comparison.METRIC_STYLES["latency"][1]) - assert throughput.get_edgecolor() == to_rgba(comparison.METRIC_STYLES["throughput"][1]) - assert not latency.get_hatch() and throughput.get_hatch() == "//" - assert latency.get_height() == (-1 if diverging else 0.5) - assert throughput.get_height() == (2 if diverging else 3) + assert bars[0].get_x() - lower == pytest.approx(0.4) + assert upper - bars[-1].get_x() - bars[-1].get_width() == pytest.approx(0.4) + for offset in (0, 2): + arch_a, arch_b = bars[offset:offset + 2] + assert arch_a.get_x() + arch_a.get_width() == arch_b.get_x() + for bar, arch in zip((arch_a, arch_b), comparison.ARCHITECTURES): + fill, border, pattern = comparison.ARCH_STYLES[arch] + assert bar.get_facecolor() == to_rgba(fill) + assert bar.get_edgecolor() == to_rgba(border) + assert bar.get_hatch() == pattern + assert arch_a.get_height() == ((-1 if diverging else 0.5) if index == 0 else (2 if diverging else 3)) + assert arch_b.get_height() == ((-3 if diverging else 0.25) if index == 0 else (3 if diverging else 4)) + assert bars[1].get_x() + bars[1].get_width() < bars[2].get_x() assert ax.get_ylabel() == (ylabel if index == 0 else "") assert bool(ax.get_yticklabels()) == (index == 0) assert all("×" not in label.get_text() for label in ax.get_yticklabels()) - assert [text.get_text() for text in ax.texts] == [ - "2.00×" if diverging else "0.50×", "P. INVALID", "3.00×", "P. INVALID"] + values = (["2.00×", "4.00×"] if diverging else ["0.50×", "0.25×"]) if index == 0 else ["3.00×", "4.00×"] + assert [text.get_text() for text in ax.texts] == [value for _ in range(2) for number in values + for value in (number, "P. INVALID")] + finally: + comparison.plt.close(fig) + + +@pytest.mark.parametrize("minimum,expected", [(0.4, 0.25), (0.5, 0.5), (0.8, 0.5)]) +@pytest.mark.parametrize("compact", [False, True]) +def test_ratio_bar_minimum_keeps_values_and_labels_visible(comparison, tmp_path, monkeypatch, + minimum, expected, compact): + template = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv")[0] + rows = [replace(template, arch=arch, mode=mode, pipeline=stage, + power=minimum if arch == "arch-a" else 2 if mode == "latency" else math.nan) + for arch in comparison.ARCHITECTURES for mode, stage in (("latency", 1), ("throughput", 4))] + captured = [] + monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) + comparison.draw_metric(rows, "power", "Average power reduction (x)", "power", tmp_path, compact=compact) + fig = captured[0] + try: + fig.canvas.draw() + renderer = fig.canvas.get_renderer() + assert fig.axes[0].get_ylim() == fig.axes[1].get_ylim() + for index, ax in enumerate(fig.axes): + assert ax.get_ylim()[0] == expected + if index == 0: + assert f"{expected:g}" in [text.get_text() for text in ax.get_yticklabels(which="both")] + else: + assert not ax.get_yticklabels(which="both") + bounds = ax.get_window_extent(renderer) + for text in ax.texts: + extent = text.get_window_extent(renderer) + assert bounds.y0 <= extent.y0 <= extent.y1 <= bounds.y1 + assert any(text.get_text() == "N/A" for ax in fig.axes for text in ax.texts) + finally: + comparison.plt.close(fig) + + +def test_compact_summary_uses_lower_minimum_when_needed(comparison, tmp_path, monkeypatch): + rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + rows = [replace(row, energy=0.4) if row.model == "vgg8" and row.mode == "latency" else row for row in rows] + captured = [] + monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) + comparison.draw_compact_summary(rows, tmp_path) + fig = captured[0] + try: + fig.canvas.draw() + assert all(ax.get_xlim()[0] == 0.25 for ax in fig.axes) + assert "0.25" in [text.get_text() for text in fig.axes[0].get_xticklabels()] finally: comparison.plt.close(fig) @@ -270,7 +484,7 @@ def test_compact_metrics_use_speedup_pipeline_selection(comparison, monkeypatch, captured.append((selected, field, basename, diverging)) monkeypatch.setattr(comparison, "load_results", lambda path: rows) - for name in ("configure_style", "draw_validation", "draw_pipeline_effect", "draw_absolute"): + for name in ("configure_style", "draw_validation", "draw_pipeline_effect", "draw_absolute", "draw_compact_summary"): monkeypatch.setattr(comparison, name, lambda *args, **kwargs: None) monkeypatch.setattr(comparison, "draw_metric", capture) monkeypatch.setattr(sys, "argv", ["generate_plots.py"]) @@ -293,8 +507,8 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat assert comparison.COMPILER_STYLES == { "raptor": ("#B3D5CA", "#4B7C6A"), "pimcomp": ("#E1BC8B", "#856A44")} - assert comparison.METRIC_STYLES == { - "latency": ("#B3CCE3", "#456B90"), "throughput": ("#B3D5CA", "#4B7C6A")} + assert comparison.ARCH_STYLES == { + "arch-a": ("#B3CCE3", "#456B90", ""), "arch-b": ("#B3D5CA", "#4B7C6A", "//")} assert comparison.VALIDATION_STYLES["PASS"] == ("#B3D5CA", "#4B7C6A") assert comparison.TEXT_COLOR == "#222222" assert comparison.NETWORK_STYLES["yolo11n"] == ("#FF9494", "#C52020") @@ -304,7 +518,7 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat for row in rows if row.model == "vgg8" and row.mode == "latency"] captured = [] monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) - comparison.draw_metric(selected, "power", "Average power advantage (x)", "power", tmp_path, diverging=True) + comparison.draw_metric(selected, "power", "Average power reduction (x)", "power", tmp_path, diverging=True) comparison.draw_absolute(selected, "latency", "Single-inference latency (ms)", "latency", tmp_path) try: for ax, compiler in zip(captured[0].axes, ("raptor", "pimcomp")): @@ -319,6 +533,44 @@ def test_compiler_palette_and_grayscale_patterns(comparison, tmp_path, monkeypat comparison.plt.close(fig) +@pytest.mark.parametrize("hatch", ["//", "\\\\", ".."]) +@pytest.mark.parametrize("width,height", [(20, 130), (160, 15), (20, 1)]) +@pytest.mark.parametrize("clipped", [False, True]) +def test_hatch_geometry_stays_inside_bar(comparison, monkeypatch, hatch, width, height, clipped): + from matplotlib.backends.backend_agg import RendererAgg + from matplotlib.path import Path as PlotPath + from matplotlib.transforms import Affine2D, Bbox + from plot_style import CenteredHatch + + renderer = RendererAgg(400, 300, 100) + gc = renderer.new_gc() + gc.set_hatch(hatch) + gc.set_hatch_color((0, 0, 0, 1)) + path = PlotPath.unit_rectangle() + transform = Affine2D().scale(width, height).translate(50, 70) + bar = transform.transform_path(path).get_extents() + if clipped: + gc.set_clip_rectangle(Bbox.from_extents(55, 70.25, 180, 180)) + bar = Bbox.intersection(bar, gc.get_clip_rectangle()) + drawn = [] + + def capture(context, geometry, affine, face=None): + drawn.append((context.get_hatch(), affine.transform_path(geometry).get_extents())) + + def capture_markers(context, marker, marker_transform, centers, affine, face=None): + for x, y in affine.transform_path(centers).vertices: + capture(context, marker, marker_transform + Affine2D().translate(x, y), face) + + monkeypatch.setattr(renderer, "draw_path", capture) + monkeypatch.setattr(renderer, "draw_markers", capture_markers) + CenteredHatch().draw_path(renderer, gc, path, transform, (1, 1, 1, 1)) + assert len(drawn) > 1 + assert all(pattern is None for pattern, _ in drawn) + for _, bounds in drawn[1:]: + assert bar.x0 - 1e-8 <= bounds.x0 <= bounds.x1 <= bar.x1 + 1e-8 + assert bar.y0 - 1e-8 <= bounds.y0 <= bounds.y1 <= bar.y1 + 1e-8 + + def test_tick_rotation_changes_only_overlapping_labels(comparison): fig, ax = comparison.plt.subplots(figsize=(4, 3), layout="constrained") try: @@ -341,6 +593,14 @@ def test_all_plot_terminology(comparison, monkeypatch): def capture(fig, name, output): fig.canvas.draw() + if "compact" in name: + compact_text = "\n".join(text.get_text().lower() for text in fig.findobj(Text)) + assert "single-inference" not in compact_text + assert "best-case throughput" not in compact_text + if name == "performance_energy_compact": + labels.update(text.get_text() for text in fig.findobj(Text)) + plotter.plt.close(fig) + return if name != "raptor_pipeline_throughput": assert fig.axes[1].get_ylabel() == "" assert fig.axes[1].get_yticklabels() == [] @@ -393,13 +653,14 @@ def test_all_plot_terminology(comparison, monkeypatch): plotter.main() text = "\n".join(labels) for old in ("Pimcomp", "ARCHITECTURE", "VGG8", "RESNET", "GOOGLENET", "YOLO11N", - "Single inference", "Best case", "Latency, 1 stage", "Energy per inference advantage"): + "Single inference", "Best case", "Latency, 1 stage", "Energy per inference advantage", + "Energy-per-inference advantage", "Energy consumption reduction", "power advantage"): assert old not in text assert {"VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet"} <= labels assert "Arch-A" in text and "Arch-B" in text assert {"Raptor", "PIMCOMP", "YOLO11n", "Single-inference", "2 stages", "4 stages", "8 stages", "Single-inference latency (ms)", "Throughput (inferences/s)", "Energy per inference (mJ)", - "Energy-per-inference advantage (x)", "Derived as 1 / latency", + "Energy reduction (x)", "Average power reduction (x)", "Derived as 1 / latency", "Throughput measurement"} <= labels @@ -432,7 +693,7 @@ def test_advantage_throughput_groups_touch(comparison, tmp_path, monkeypatch, di selected = [row for row in rows if row.model in ("vgg8", "resnet18")] captured = [] monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) - comparison.draw_metric(selected, "energy", "Energy-per-inference advantage (x)", "energy", tmp_path, + comparison.draw_metric(selected, "energy", "Energy reduction (x)", "energy", tmp_path, diverging=diverging) fig = captured[0] try: @@ -466,8 +727,10 @@ def test_grouped_absolute_plot_scheduling_labels(comparison, tmp_path, monkeypat comparison.plt.close(fig) -def test_pipeline_plot_connects_derived_and_measured_rates(comparison, tmp_path, monkeypatch): +@pytest.mark.parametrize("status", ["PASS", "FAIL", "SKIPPED", "NA"]) +def test_pipeline_plot_connects_derived_and_measured_rates(comparison, tmp_path, monkeypatch, status): rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + rows = [replace(row, raptor_validation=status, pimcomp_validation=status) for row in rows] captured = [] monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) comparison.draw_pipeline_effect(rows, tmp_path) @@ -518,6 +781,7 @@ def test_plotter_exports_png_svg_and_pdf(comparison, tmp_path): for row in rows if row.model == "vgg8" and row.mode == "latency"] comparison.draw_absolute(selected, "latency", "Latency (ms)", "comparison-smoke", tmp_path) comparison.draw_validation(rows, tmp_path, compact=True) + comparison.draw_compact_summary(rows, tmp_path) compact_names = [] for metric in ("speedup", "energy", "power"): for suffix, diverging in (("comparison", False), ("diverging", True)): @@ -525,10 +789,182 @@ def test_plotter_exports_png_svg_and_pdf(comparison, tmp_path): compact_names.append(name) comparison.draw_metric(comparison.select_compact_speedups(rows), metric, "Advantage (x)", name, tmp_path, diverging=diverging, compact=True) - for name in ("comparison-smoke", "validation_status_compact", *compact_names): + for name in ("comparison-smoke", "validation_status_compact", "performance_energy_compact", *compact_names): assert (tmp_path / f"png/{name}.png").read_bytes().startswith(b"\x89PNG\r\n\x1a\n") svg = (tmp_path / f"svg/{name}.svg").read_text() assert " 0) == (key[1] == "arch-a") + children = [child for child in ax.texts if getattr(child, "xycoords", None) is text] + assert len(children) == 1 + for dagger in children: + assert dagger.get_text() == r"$\boldsymbol{\dagger}$" + assert dagger.get_color() == comparison.DAGGER_COLOR == "#AA1C1C" + assert dagger.get_fontsize() == 9 + assert dagger.get_fontsize() > text.get_fontsize() + assert dagger.get_alpha() == int(invalid) + groups.append([text, *children]) + for lane in (0, 2): + offsets = [text.get_window_extent(renderer).y1 - ax.transData.transform(text.xy)[1] + for text in labels if text.xy[1] == lane] + assert max(offsets) == pytest.approx(min(offsets)) + assert len(labels) == 8 + assert list(ax.get_yticks()) == [0, 2] + assert [text.get_text() for text in ax.get_yticklabels()] == ["Arch-A", "Arch-B"] + assert ax.get_title() == "" + assert ax.yaxis.label.get_rotation() == 90 + assert ax.yaxis.label._get_multialignment() == "center" + assert ax.yaxis.label.get_fontsize() == 8.5 + assert len(ax.get_ylabel().splitlines()) <= 2 + assert ax.yaxis.label.get_window_extent(renderer).x1 < ax.get_window_extent(renderer).x0 + assert bool(ax.get_xticklabels()) == (ax is fig.axes[0]) + if ax is fig.axes[0]: + assert "0.5" in [text.get_text() for text in ax.get_xticklabels()] + assert "0" not in [text.get_text() for text in ax.get_xticklabels()] + assert all(tick.label2.get_visible() and not tick.label1.get_visible() + for tick in ax.xaxis.get_major_ticks()) + assert not any(text.get_text() in ("Raptor / PIMCOMP", "PIMCOMP / Raptor") for text in ax.texts) + baseline = next(line for line in ax.lines if list(line.get_xdata()) == [1, 1]) + assert baseline.get_linestyle() == "--" + bounds = ax.get_window_extent(renderer) + paths = [] + for line in ax.lines: + box = line.get_window_extent(renderer).padded(1) + paths.append(PlotPath.unit_rectangle().transformed( + Affine2D().scale(box.width, box.height).translate(box.x0, box.y0))) + for group in groups: + text = group[0] + box = Bbox.union([part.get_window_extent(renderer) for part in group]) + assert bounds.x0 <= box.x0 <= box.x1 <= bounds.x1 + assert bounds.y0 <= box.y0 <= box.y1 <= bounds.y1 + path = PlotPath.unit_rectangle().transformed( + Affine2D().scale(box.width, box.height).translate(box.x0, box.y0)) + assert not any(path.intersects_path(other, filled=True) for other in paths), text.get_text() + paths.append(path) + assert "YOLO11n" not in [text.get_text() for text in ax.get_yticklabels()] + assert {text.get_position()[1] > 0 for ax in fig.axes for text in ax.texts + if getattr(text, "xycoords", None) == "data"} == {False, True} + assert all(ax.get_xlim() == fig.axes[0].get_xlim() for ax in fig.axes) + assert [ax.get_ylabel() for ax in fig.axes] == [ + "Latency-mode\nspeedup", "Throughput-mode\nspeedup", + "Latency-mode\nenergy reduction", "Throughput-mode\nenergy reduction"] + assert fig.axes[0].xaxis.get_label_position() == "top" + xlabel = fig.axes[0].xaxis.label.get_window_extent(renderer) + legend = fig.legends[0].get_window_extent(renderer) + assert fig.axes[0].get_window_extent(renderer).y1 < xlabel.y0 < xlabel.y1 < legend.y0 + tick_top = max(text.get_window_extent(renderer).y1 for text in fig.axes[0].get_xticklabels()) + assert 0 < xlabel.y0 - tick_top < renderer.points_to_pixels(12) + axis_bounds = fig.axes[0].get_window_extent(renderer) + assert (xlabel.x0 + xlabel.x1) / 2 == pytest.approx((axis_bounds.x0 + axis_bounds.x1) / 2) + assert fig.axes[0].get_xlabel() == "Ratio (x)" + assert {"← PIMCOMP better", "Raptor better →"} <= {text.get_text() for text in fig.axes[0].texts} + left, right = [text.get_window_extent(renderer) for text in fig.axes[0].texts if not hasattr(text, "xy")] + assert left.y0 == right.y0 == pytest.approx(xlabel.y0) + assert left.x1 < xlabel.x0 < xlabel.x1 < right.x0 + legend_labels = {text.get_text() for text in fig.legends[0].get_texts() if text.get_visible()} + assert legend_labels == {"VGG-8", "ResNet-18", "ResNet-34", "GoogLeNet", "PIMCOMP invalid"} + for handle, text in zip(fig.legends[0].legend_handles, fig.legends[0].get_texts()): + if text.get_text() in comparison.MODEL_NAMES.values(): + model = next(model for model, name in comparison.MODEL_NAMES.items() if name == text.get_text()) + assert handle.get_marker() == {"vgg8": "o", "resnet18": "s", "resnet34": "^", "googlenet": "D"}[model] + assert handle.get_markerfacecolor() == comparison.NETWORK_STYLES[model][0] + elif text.get_text() == "PIMCOMP invalid": + assert handle.get_text() == r"$\boldsymbol{\dagger}$" + assert handle.get_fontsize() == 9 + assert handle.get_math_fontfamily() == "cm" + assert handle.get_color() == comparison.DAGGER_COLOR + result_dagger = next(child for child in fig.axes[0].texts + if getattr(getattr(child, "xycoords", None), "xycoords", None) == "data" + and child.get_alpha() == 1) + result_box = result_dagger.get_window_extent(renderer) + legend_box = handle.get_window_extent(renderer) + assert legend_box.width == pytest.approx(result_box.height) + assert legend_box.height == pytest.approx(result_box.width) + ratios = [line.get_xdata()[0] for ax in fig.axes for line in ax.lines + if line.get_marker() in ("o", "s", "^", "D")] + lower, upper = fig.axes[0].get_xlim() + padding = (max(ratios) - min(ratios)) * 0.04 + assert lower == 0.5 + assert upper - max(ratios) == pytest.approx(padding) + finally: + comparison.plt.close(fig) + + +def test_compact_summary_rejects_missing_ratio(comparison, tmp_path): + rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + rows = [replace(row, energy=math.nan) if row.mode == "throughput" and row.model == "vgg8" + else row for row in rows] + with pytest.raises(ValueError, match="requires a positive energy ratio"): + comparison.draw_compact_summary(rows, tmp_path) + + +@pytest.mark.parametrize("compiler,status,label", [("pimcomp", "FAIL", r"$\boldsymbol{\dagger}$"), ("raptor", "FAIL", "R. INVALID"), + ("pimcomp", "SKIPPED", "NOT VALIDATED"), ("pimcomp", "NA", "NOT VALIDATED")]) +def test_compact_summary_validation_labels(comparison, tmp_path, monkeypatch, compiler, status, label): + rows = comparison.load_results(TOOLS / "pimcomp/compare/results_comparison.csv") + rows = [replace(row, **{f"{compiler}_validation": status}) + if row.model == "vgg8" and row.arch == "arch-a" and row.mode == "latency" else row for row in rows] + captured = [] + monkeypatch.setattr(comparison, "save_figure", lambda fig, *args: captured.append(fig)) + comparison.draw_compact_summary(rows, tmp_path) + fig = captured[0] + try: + ax = fig.axes[0] + if compiler == "pimcomp" and status == "FAIL": + assert any(label in text.get_text() and text.get_color() == comparison.DAGGER_COLOR for text in ax.texts) + else: + assert label in ax.texts[0].get_text() + assert ax.texts[0].get_color() == comparison.TEXT_COLOR + finally: + comparison.plt.close(fig) diff --git a/validation/tools/pim/ablation/README.md b/validation/tools/pim/ablation/README.md index 83451db..7172f99 100644 --- a/validation/tools/pim/ablation/README.md +++ b/validation/tools/pim/ablation/README.md @@ -12,10 +12,14 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`. | Variant | Raptor options | |---|---| -| `baseline` | None | -| `no-sync` | `--pim-disable-synchronization` | -| `no-plan` | `--pim-disable-spatial-planning` | -| `no-sync-no-plan` | Both options | +| `baseline` | None; planning and synchronization enabled | +| `no-sync` | `--pim-disable-synchronization`; throughput only | +| `no-plan` | `--pim-disable-spatial-planning`; synchronization remains enabled | + +Latency runs compare `baseline` with `no-plan`. Throughput runs compare +`baseline` with each disabled feature separately. No run disables both features. +The `baseline` artifact name identifies the fully enabled program, not the +denominator used for percentage gains. ## Options and outputs @@ -25,7 +29,9 @@ explicitly when needed. Only Raptor runs; Pimcomp validation is `SKIPPED`. Other [suite-runner options](../pimcomp/compare/README.md), including `--models`, `--archs`, and `--no-fast`, are forwarded. Supplying `--mode` or `--pipeline` -replaces the default latency/pipeline-4 case set. `--only` is rejected because +replaces the default latency/pipeline-4 case set. Mode-only selection runs all +suite pipelines for that mode. Pipeline-only selection uses its matching mode. +`--only` is rejected because the wrapper always runs Raptor only. Ctrl+C stops the active variant's workers. Artifacts use `//artifacts//[/pipelineN]/ablation//`; @@ -37,19 +43,44 @@ By default, the combined [results_ablation.csv](results_ablation.csv) is written beside this script and printed after the run. An explicit `--out-dir PATH` writes `PATH/results_ablation.csv`, keeping independent experiments separate. Default model artifacts and per-variant summaries remain in the locations above. -The aggregate records case identifiers, -`functional_validation`, and signed percentage changes in latency, throughput, -power, and energy against the matching baseline case. Positive means higher; -baseline values are zero. +The aggregate contains one row per comparison: + +| Mode | `comparison` | `baseline_variant` | Main gain | +|---|---|---|---| +| Latency | `plan` | `no-plan` | `latency_gain_percent` | +| Throughput | `plan` | `no-plan` | `throughput_gain_percent` | +| Throughput | `sync` | `no-sync` | `throughput_gain_percent` | + +Throughput gain is `100 × (enabled / disabled − 1)`. +Latency, power, and energy gains are reductions: `100 × (1 − enabled / disabled)`. +The corresponding columns use the suffix `_gain_percent`. Positive means improvement; +negative means a cost. Synchronization gain can be negative. No sign is forced. +Both programs' functional statuses are recorded as `functional_validation` +and `baseline_functional_validation`. The latter belongs to the disabled feature. Disabling synchronization removes host-buffer lifetime and pipeline restart guarantees, so those variants may fail numerical validation and exit nonzero. -Baseline and `no-plan` must pass. Percentages from failing variants are not -valid speedups; deadline averages do not prove steady-state convergence. +Baseline and `no-plan` must pass. Percentages involving failing programs remain +diagnostic comparisons, not valid speedups. + +When fast-mode convergence fails, positive fallback throughput remains available. +Its output interval uses the full simulation window. The per-case JSON report +records `throughput_measurement: full_duration_fallback` and +`fast_throughput_converged: false`. This result does not prove steady-state convergence. +Zero completed outputs still produce `NA` for per-inference metrics. + +The `no-sync` compiler option keeps the normal selected schedule and omits +SYNC/WAIT instructions during code generation. Selection costs describe the +`synchronized-reference` program. Each backend attempt also reports a separate +`synchronization_ablation_cost`, with zero SYNC/WAIT work and no synchronization +dependencies. This diagnostic estimate is not used for selection. Its complete-state +recurrence can remain unproved because cores can advance independently. +Numerical validation still checks the emitted program and can fail. ## Result data -The checked-in `results_ablation.csv` is an intentional result dataset for tabular -reporting. Ablation experiments and aggregation remain available through the +The checked-in `results_ablation.csv` is a historical dataset until the next run +regenerates it with the comparison schema above. Existing measurements are preserved. +Ablation experiments and aggregation remain available through the runner above. Percentages from failing variants are not valid speedups. See the [comparison metric definitions](../../../../docs/evaluation/methodology.md#metric-interpretation). diff --git a/validation/tools/pim/ablation/run_ablation.py b/validation/tools/pim/ablation/run_ablation.py index 29caa04..bd53ec2 100755 --- a/validation/tools/pim/ablation/run_ablation.py +++ b/validation/tools/pim/ablation/run_ablation.py @@ -37,10 +37,6 @@ VARIANTS = ( Variant("baseline", ()), Variant("no-sync", ("--pim-disable-synchronization",)), Variant("no-plan", ("--pim-disable-spatial-planning",)), - Variant( - "no-sync-no-plan", - ("--pim-disable-synchronization", "--pim-disable-spatial-planning"), - ), ) VARIANT_BY_NAME = {variant.name: variant for variant in VARIANTS} COMPARISON_RESULTS_FILENAME = "results_comparison.csv" @@ -53,17 +49,19 @@ PERCENTAGE_FIELDS = ( ("raptor_power_mw", "power"), ("raptor_energy_pj", "energy"), ) -RESULT_FIELDS = (*CASE_FIELDS, "functional_validation", *(target for _, target in PERCENTAGE_FIELDS)) +RESULT_FIELDS = ( + "comparison", *CASE_FIELDS, "baseline_variant", "baseline_functional_validation", + "functional_validation", *(f"{target}_gain_percent" for _, target in PERCENTAGE_FIELDS), +) DEFAULT_CASE_ARGUMENTS = ( ("latency", ("--mode", "latency")), ("throughput/pipeline4", ("--mode", "throughput", "--pipeline", "4")), ) -DEFAULT_CASE_KEYS = frozenset({("latency", "1"), ("throughput", "4")}) def parse_args(argv: list[str] | None = None) -> tuple[argparse.Namespace, list[str]]: parser = argparse.ArgumentParser( - description="Run the complete compiler ablation matrix on the Pimcomp model suite.", + description="Measure planning and synchronization gains on the Pimcomp model suite.", allow_abbrev=False, ) parser.add_argument( @@ -97,12 +95,30 @@ def has_option(arguments: list[str], option: str) -> bool: def runner_argument_sets(forwarded: list[str]) -> tuple[tuple[str, list[str]], ...]: - if has_option(forwarded, "--mode") or has_option(forwarded, "--pipeline"): - return (("requested", forwarded),) - return tuple( - (label, [*forwarded, *case_arguments]) - for label, case_arguments in DEFAULT_CASE_ARGUMENTS - ) + if not has_option(forwarded, "--mode") and not has_option(forwarded, "--pipeline"): + return tuple( + (label, [*forwarded, *case_arguments]) + for label, case_arguments in DEFAULT_CASE_ARGUMENTS + ) + parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False) + parser.add_argument("--mode", nargs="+", choices=("latency", "throughput")) + parser.add_argument("--pipeline", type=int, choices=(1, 2, 4, 8)) + args, remaining = parser.parse_known_args(forwarded) + cases = [(mode, pipeline) for mode, pipeline in + (("latency", 1), ("throughput", 2), ("throughput", 4), ("throughput", 8)) + if (args.mode is None or mode in args.mode) + and (args.pipeline is None or pipeline == args.pipeline)] + if not cases: + parser.error("--pipeline does not belong to the selected --mode") + return tuple((f"{mode}/pipeline{pipeline}", + [*remaining, "--mode", mode, "--pipeline", str(pipeline)]) + for mode, pipeline in cases) + + +def case_key(arguments: list[str]) -> tuple[str, str]: + mode = arguments[arguments.index("--mode") + 1] + pipeline = arguments[arguments.index("--pipeline") + 1] if "--pipeline" in arguments else "1" + return mode, pipeline def variant_output_dir(out_dir: Path, variant: Variant) -> Path: @@ -154,7 +170,9 @@ def terminate_process_group(process: subprocess.Popen[bytes]) -> None: process.wait() -def percentage_difference(value: str | None, reference: str | None) -> str: +def percentage_difference( + value: str | None, reference: str | None, *, lower_is_better: bool = False +) -> str: try: current = float(value) if value is not None else math.nan baseline = float(reference) if reference is not None else math.nan @@ -162,7 +180,8 @@ def percentage_difference(value: str | None, reference: str | None) -> str: return "NA" if not math.isfinite(current) or not math.isfinite(baseline) or current <= 0 or baseline <= 0: return "NA" - change = (current / baseline - 1) * 100 + ratio = current / baseline + change = (1 - ratio if lower_is_better else ratio - 1) * 100 return f"{change:+.2f}%" if math.isfinite(change) else "NA" @@ -202,25 +221,43 @@ def aggregate_results( selected_rows.append(row) if fields is None: return None, failures - baseline_rows = { + enabled_rows = { tuple(row.get(field, "") for field in CASE_FIELDS): row for row in rows_by_variant.get("baseline", []) } if "baseline" not in rows_by_variant: - failures.append("baseline: missing percentage reference results") + failures.append("baseline: missing fully enabled results") result_rows = [] - variant_order = {variant.name: index for index, variant in enumerate(VARIANTS)} for variant in variants: + if variant.name == "baseline": + continue + disabled_keys = { + tuple(row.get(field, "") for field in CASE_FIELDS) + for row in rows_by_variant.get(variant.name, []) + } + for key, row in enabled_rows.items(): + if variant.name == "no-sync" and row.get("mode") != "throughput": + continue + if key not in disabled_keys: + failures.append(f"{variant.name}: missing baseline case " + "/".join(key)) for row in rows_by_variant.get(variant.name, []): - reference = baseline_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS)) + if variant.name == "no-sync" and row.get("mode") != "throughput": + continue + enabled = enabled_rows.get(tuple(row.get(field, "") for field in CASE_FIELDS)) + if enabled is None: + failures.append(f"{variant.name}: missing fully enabled case " + + "/".join(row.get(field, "") for field in CASE_FIELDS)) result_rows.append( { - "variant": variant.name, + "comparison": "plan" if variant.name == "no-plan" else "sync", **{field: row.get(field, "") for field in CASE_FIELDS}, - "functional_validation": row.get("raptor_functional_validation", "NA"), + "baseline_variant": variant.name, + "baseline_functional_validation": row.get("raptor_functional_validation", "NA"), + "functional_validation": enabled.get("raptor_functional_validation", "NA") if enabled else "NA", **{ - target: percentage_difference( - row.get(source), reference.get(source) if reference else None + f"{target}_gain_percent": percentage_difference( + enabled.get(source) if enabled else None, row.get(source), + lower_is_better=target != "throughput", ) for source, target in PERCENTAGE_FIELDS }, @@ -231,14 +268,14 @@ def aggregate_results( row["arch"], row["model"], row["mode"], - variant_order[row["variant"]], int(row["raptor_pipeline"]), + row["comparison"], ) ) output = output if output is not None else out_dir / ABLATION_RESULTS_FILENAME output.parent.mkdir(parents=True, exist_ok=True) with output.open("w", newline="", encoding="utf-8") as stream: - writer = csv.DictWriter(stream, fieldnames=("variant", *RESULT_FIELDS), lineterminator="\n") + writer = csv.DictWriter(stream, fieldnames=RESULT_FIELDS, lineterminator="\n") writer.writeheader() writer.writerows(result_rows) return output, failures @@ -252,9 +289,13 @@ def main(argv: list[str] | None = None) -> int: variants = selected_variants(args.variants) out_dir = args.out_dir.resolve() if args.out_dir is not None else DEFAULT_OUT_DIR case_sets = runner_argument_sets(forwarded) + if not any(case_key(arguments)[0] == "throughput" for _, arguments in case_sets): + variants = [variant for variant in variants if variant.name != "no-sync"] if args.dry_run: for variant in variants: for _, case_forwarded in case_sets: + if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput": + continue print( shlex.join( runner_command( @@ -275,6 +316,8 @@ def main(argv: list[str] | None = None) -> int: try: for variant in variants: for case_label, case_forwarded in case_sets: + if variant.name == "no-sync" and case_key(case_forwarded)[0] != "throughput": + continue command = runner_command( out_dir, variant, args.models, case_forwarded, out_dir, False ) @@ -296,7 +339,7 @@ def main(argv: list[str] | None = None) -> int: return 130 remove_lock_files(out_dir) - selected_cases = DEFAULT_CASE_KEYS if len(case_sets) > 1 else None + selected_cases = frozenset(case_key(arguments) for _, arguments in case_sets) output, aggregation_failures = aggregate_results( out_dir, variants, diff --git a/validation/tools/pim/pimcomp/compare/README.md b/validation/tools/pim/pimcomp/compare/README.md index 8a67e51..f4341ee 100644 --- a/validation/tools/pim/pimcomp/compare/README.md +++ b/validation/tools/pim/pimcomp/compare/README.md @@ -39,13 +39,20 @@ beside this script. An explicit `--out-dir PATH` writes Default model artifacts remain under `validation/networks/pimcomp_models`. Artifacts use `//artifacts//[/pipelineN]/`, with shared references -under `artifacts/common/` (`common-functional/` for YOLO's canonical reference) +under `artifacts/common/` and `comparison_report.{md,json}` under each comparison's `pimcomp/`. +YOLO and GoogLeNet use `artifacts/common-full-model/` to avoid reusing historical Softmax-free references. Pimcomp runs once per model/architecture/mode; later pipelines reuse its newly written report, including failures without simulation artifacts. Missing or unchanged anchor reports trigger fresh attempts. The runner continues after failures and exits nonzero if a compiler, simulation, or functional comparison fails. +If fast-mode convergence fails, the reports and CSV retain valid full-duration +fallback throughput and output interval values. This is not a converged steady-state result. +Per-case JSON records `throughput_measurement: full_duration_fallback` and +`fast_throughput_converged: false`; the text report labels the fallback. +Zero completed outputs still give `NA` for per-inference metrics. + Reuse checks model/configuration paths and SHA-256 contents, effective hardware, mode, Pimcomp pipeline/replication, seed, batch, tolerances, and simulation flags. Malformed, incomplete, mismatched, or legacy reports without fingerprints fail @@ -56,6 +63,14 @@ exists. Raptor pipeline depth is deliberately excluded from Pimcomp identity. Tool/build revisions and manually edited reference artifacts are not hashed; rerun the relevant compiler after those change. +Functional validation uses the same ONNX model and Raptor program as the performance comparison. +YOLO uses the full model, including both internal Softmax operations. +The functional simulator executes Softmax; Pimsim-nn and the queue estimator omit its timing cost. +Reported timing and energy exclude Softmax work. Other instructions and transfers retain their costs. +GoogLeNet removes only its two LRN nodes and retains terminal Softmax. +VGG8 and both ResNet graphs have no removed operators. +The native reference uses the same model as each compiled program. + ## One-model comparator `compare_raptor_pimcomp_model.py` requires `--model PATH` and `--out-dir PATH`. @@ -63,6 +78,13 @@ Use `--help` for hardware and simulation options. The suite runner supplies these automatically. The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`; `validate.py` uses `1e-3`/`1e-5`. +Both use the [same output checks](../../../../README.md#performance-and-result-interpretation), +including a relative L2 limit for each nonzero reference output. +`--rtol=0` disables that additional limit in the comparator. +Reports from an older validation policy require regeneration before reuse. +An elementwise failure with relative L2 agreement remains `FAIL`. +Its report notes possible numerical sensitivity. That note does not establish correctness. +Every batch iteration must pass. Reported errors are maxima across iterations for each output. ## Comparison charts @@ -71,19 +93,43 @@ The comparator defaults to absolute/relative tolerances `1e-3`/`1e-4`; ``` Requires Matplotlib from the root `requirements.txt`. Defaults to the supplied -`results_comparison.csv` beside the script and writes `plots/png/` and -`plots/svg/` and `plots/pdf/` there. Use `--input PATH` for a runner-generated CSV and +`results_comparison.csv` beside the script and writes `plots/default/png/`, +`plots/default/svg/`, and `plots/default/pdf/` there. Use `--input PATH` for a runner-generated CSV and `--output-dir PATH` to override the destination. The plotter requires the full 40-case matrix: five models × Arch-A/B × latency pipeline 1 and throughput pipelines 2/4/8. Duplicate, missing, unexpected, or inconsistent shared Pimcomp configurations are rejected before plots are written. +Charts use the pastel palette and the Raptor name by default. +Use `--no-colors` for grayscale output; `--colors` explicitly enables colors. +Use `--anonymize` to replace displayed `Raptor` labels with `Anonimized`; +`--no-anonymize` explicitly keeps the Raptor name. The flags are independent. +Use `--no-marker-borders` for solid markers without outlines in the compact dot +plot and line chart, including their legends. These markers use the darker +border colors as solid fills. `--marker-borders` keeps the default outlines. +Line colors and bar patterns remain unchanged. +Input keys and output basenames remain stable. Use these output directories +to keep the grayscale and colored anonymized versions: + +```bash +.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \ + --no-colors --anonymize \ + --output-dir validation/tools/pim/pimcomp/compare/plots/gray_anonimized +.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \ + --anonymize \ + --output-dir validation/tools/pim/pimcomp/compare/plots/anonimized +.venv/bin/python validation/tools/pim/pimcomp/compare/generate_plots.py \ + --anonymize --no-marker-borders \ + --output-dir validation/tools/pim/pimcomp/compare/plots/no_border_anonimized +``` + | Plot family | Meaning | |---|---| | `validation_status` | Recorded functional status for each compiler and experiment. | | `validation_status_compact` | Compact functional status matrix. | -| `speedup_compact_*` | Touching single-inference latency and best-pipeline throughput bars for each network. | -| `energy_compact_*`, `power_compact_*` | Energy and power advantages for the same cases selected by the compact speedup plots. | +| `performance_energy_compact` | Four horizontal dot panels: latency-mode speedup, throughput-mode speedup, and separate energy panels for both modes. | +| `speedup_compact_*` | Latency panel on the left and best-pipeline throughput panel on the right, with touching Arch-A/B bars per network. | +| `energy_compact_*`, `power_compact_*` | Energy reduction and power reduction for the same cases selected by the compact speedup plots. | | `latency_*` | Single-inference latency, latency mode only. | | `throughput_*` | Inferences/s, throughput mode only. | | `longest_stage_latency_*` | Combined single-inference latency and throughput output interval; the latter is a stage-latency proxy. | @@ -95,17 +141,47 @@ Suffixes `_absolute` and `_yolo` show measurements (`_yolo` is Raptor-only); advantages in either direction. Ratios above 1 favor Raptor: Pimcomp/Raptor for latency, power, and energy; Raptor/Pimcomp for throughput. Diverging bars use winning factor minus one, with parity at zero. +Direct ratio axes start at 0.5, or 0.25 when a result is below 0.5. +Panels in each figure share the same minimum. The shared `plot_style.py` defines the manuscript palette and display names. Raptor uses sage green; PIMCOMP uses plain sand. Stage patterns, outlines, marker shapes, and status text preserve distinctions in grayscale. -Compact plots use blue for the single-inference case and hatched sage -green for the throughput case. +Compact bar plots use plain blue for Arch-A and hatched sage green for Arch-B. +The left panel contains latency-mode measurements. The right panel contains +throughput-mode measurements. Each network has touching Arch-A/B bars. +All compact figures use the display names `Latency-mode` and `Throughput-mode`. Each architecture/network pair uses the highest recorded throughput across pipelines. Energy and power use those same cases, without selecting a different pipeline for each metric. Validation labels remain visible for the selected cases. Units appear in axis titles and bar values; axis tick labels contain numbers only. +The `performance_energy_compact` figure includes the eight Arch-A/B comparisons +for VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It excludes YOLO11n. +All panels share a linear ratio axis from 0.5, or 0.25 when needed, through the largest ratio, +with a small right margin. A dashed line marks parity at 1. +The shared axis title and ticks appear at the top, below the legend. +The `Ratio (x)` title is centered over the axis. Panel titles appear to the left, +rotate 90° counterclockwise, and use centered text on at most two lines. +Titles start with `Latency-mode` or `Throughput-mode`, followed by the metric. +The PIMCOMP-better and Raptor-better indicators flank the shared axis title. +The figure has no mean markers. +Labels give ratios to two decimal places; marker positions use unrounded ratios. +Each panel has separate Arch-A and Arch-B lanes. The legend identifies models +through marker shapes and pastel colors. Result labels contain ratios and use +vertical text above or below their architecture line, without connectors. +Placement prefers above for Arch-A and below for Arch-B. +Lane spacing and label placement avoid other labels, markers, and reference lines. +Larger dark-red daggers mark invalid PIMCOMP comparisons, including every batch comparison, +even when the CSV records PASS. Marker borders retain the model colors; +ratio values use normal text colors. The legend dagger matches the result daggers. +Both use the same font and size. Labels without daggers reserve the same space, +so numeric values align within each lane. +These simulated ratios remain visible. +The two energy panels show latency-mode energy and throughput-selected energy. +The throughput energy uses the selected throughput pipeline, without a separate +energy optimization. + Throughput-stage bars touch within each network in both absolute and advantage plots. Bar-label widths are measured before export; dense figures expand only as needed to contain values and invalidity annotations. Legends use the available @@ -116,6 +192,7 @@ pipeline plot retains its own scale, with y-axis ticks on both architecture rows Only the bottom row shows the shared x-axis ticks. Network names appear above the top row; architecture names appear on the right. That plot trims unused vertical range and is intended to span both manuscript columns. +The line chart shows throughput only; validation status does not change its markers or colors. Failed validation is labeled `P. INVALID`; other non-PASS statuses keep their names, and missing measurements show `N/A`. Relative bars require both @@ -130,3 +207,15 @@ publication provenance. The checked-in dataset is intentional. See the See the [evaluation methodology](../../../../../docs/evaluation/methodology.md#metric-interpretation) for measurement windows, units, numerical validity, and result provenance. + +Report reuse checks model and configuration hashes, compiler and simulator +fingerprints, validation policy, actual tolerances, seed, batch size, and +simulation options. Artifact signatures check instructions, weights, and memory +images. Missing or changed artifacts require regeneration. Redundant Raptor +JSON instruction dumps do not affect signatures when native binaries exist. + +`better_compiler` is available only when both compilers pass functional +validation. `speedup` retains the diagnostic timing ratio when both timings +exist. A timing ratio for a failed implementation does not identify a better +correct implementation. Valid-only plot aggregates require both validations +to pass. diff --git a/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py b/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py index 1294000..fb510e7 100755 --- a/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py +++ b/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp_model.py @@ -18,7 +18,7 @@ import time import types from collections import Counter from concurrent.futures import ThreadPoolExecutor -from dataclasses import asdict, dataclass +from dataclasses import asdict, dataclass, field from pathlib import Path from tempfile import TemporaryDirectory from typing import Any @@ -45,6 +45,7 @@ from raptor_validation.onnx_utils import ( write_inputs_to_memory_bin, ) from raptor_validation.raptor import compile_with_raptor +from raptor_validation.output_comparison import VALIDATION_POLICY, compare_output_arrays from raptor_validation.pimsim_nn import ( export_raptor_pimsim_artifact, parse_pimsim_nn_metrics, @@ -75,6 +76,7 @@ class CompareResult: max_diffs: dict[str, float] status: str = "done" error: str | None = None + relative_l2_errors: dict[str, float | None] = field(default_factory=dict) def load_pimcomp_exporter(): @@ -264,20 +266,29 @@ def compare_simulator_outputs( channel_last: bool = False, ) -> CompareResult: sim_arrays = parse_pim_simulator_outputs(output_bin, outputs_desc) + if len(sim_arrays) != len(outputs_desc): + raise ValueError("simulator output count differs from the model output count") max_diffs: dict[str, float] = {} + relative_errors: dict[str, float | None] = {} passed = True - for sim_array, (idx, name, _, shape) in zip(sim_arrays, outputs_desc): + error = None + for sim_array, (idx, name, elem_type, shape) in zip(sim_arrays, outputs_desc): if channel_last and len(shape) == 4: n, c, h, w = shape sim_array = sim_array.reshape(n, h, w, c).transpose(0, 3, 1, 2) csv_name = reference_dir / f"output{idx}_{sanitize_output_name(name)}.csv" - ref = np.loadtxt(csv_name, delimiter=",", dtype=np.float32).reshape(shape) - diff = np.abs(sim_array.astype(np.float64) - ref.astype(np.float64)) - allowed = threshold + rtol * np.abs(ref.astype(np.float64)) - max_diffs[name] = float(np.max(diff)) - if not np.all(diff <= allowed): - passed = False - return CompareResult(passed=passed, max_diffs=max_diffs) + ref = np.loadtxt(csv_name, delimiter=",", dtype=_ONNX_TO_NP[elem_type]).reshape(shape) + comparison = compare_output_arrays(sim_array, ref, atol=threshold, rtol=rtol) + max_diffs[name] = comparison.max_abs_diff + relative_errors[name] = comparison.relative_l2_error + passed &= comparison.passed + if not comparison.passed and error is None: + if (comparison.relative_l2_error is not None + and comparison.relative_l2_error <= rtol and not comparison.pointwise_passed): + error = f"{name}: elementwise mismatch despite relative L2 agreement; possible numerical sensitivity, not proof of correctness" + else: + error = f"{name}: output tolerance exceeded" + return CompareResult(passed=passed, max_diffs=max_diffs, relative_l2_errors=relative_errors, error=error) def sanitize_output_name(name: str) -> str: @@ -652,7 +663,9 @@ def run_functional_validation( stage="Run functional simulation", ) max_diffs: dict[str, float] = {} + relative_errors: dict[str, float | None] = {} failed_iterations = [] + failure_detail = None for index, reference_dir in enumerate(reference_dirs): result = compare_simulator_outputs( batch_output_dir / f"output_{index:06d}.bin", @@ -664,12 +677,20 @@ def run_functional_validation( ) if not result.passed: failed_iterations.append(index) + if failure_detail is None: + failure_detail = result.error for name, diff in result.max_diffs.items(): max_diffs[name] = max(max_diffs.get(name, 0.0), diff) + for name, relative in result.relative_l2_errors.items(): + if relative is not None: + relative_errors[name] = max(relative_errors.get(name) or 0.0, relative) + else: + relative_errors.setdefault(name, None) return CompareResult( passed=not failed_iterations, max_diffs=max_diffs, - error=(f"batch iterations failed: {failed_iterations}" if failed_iterations else None), + error=(f"batch iterations failed: {failed_iterations}; {failure_detail}" if failed_iterations else None), + relative_l2_errors=relative_errors, ) @@ -1126,6 +1147,8 @@ def perf_status(perf: dict[str, Any]) -> str: return "SKIPPED" if perf.get("error"): return "FAILED" + if perf.get("throughput_measurement") == "full_duration_fallback": + return "DONE (full-duration fallback; unconverged)" return "DONE" @@ -1163,29 +1186,77 @@ def path_from_report(report: dict[str, Any], name: str) -> Path | None: return Path(value) if value else None -def pimcomp_reuse_context( - args: argparse.Namespace, - model: Path, - functional_model: Path, - hardware: dict[str, int], -) -> dict[str, Any]: - files = {"model": model, "functional_model": functional_model, - "config": args.pimcomp_config} - digests = {} - for path in set(files.values()): - with path.open("rb") as stream: - digests[path] = hashlib.file_digest(stream, "sha256").hexdigest() - return { - "version": 1, - "files": {name: {"path": str(path), "sha256": digests[path]} - for name, path in files.items()}, - "hardware": hardware, - "settings": {name: getattr(args, name) for name in ( - "pimsim_mode", "pimsim_time_ms", "pimcomp_pipeline", - "pimcomp_replication", "pimcomp_model_name", "seed", "batch_size", - "threshold", "rtol", "no_fast", "skip_pimsim_nn", - )}, - } +def fingerprint_files(files: dict[str, Path]) -> dict[str, Any]: + fingerprints = {} + for name, path in files.items(): + digest = None + if path.is_file(): + with path.open("rb") as stream: + digest = hashlib.file_digest(stream, "sha256").hexdigest() + fingerprints[name] = {"path": str(path), "sha256": digest} + return fingerprints + + +def experiment_context(args, model, functional_model, hardware, side): + files = {"model": model, "functional_model": functional_model, "config": args.pimcomp_config, + "oracle": args.pimsim_nn_build_dir / "ChipTest", + "comparison": Path(__file__), + "validation": VALIDATION_DIR / "raptor_validation/output_comparison.py", + "functional_simulator": args.pim_simulator_dir / "target/release/pim-simulator"} + if side == "raptor": + files["compiler"] = args.raptor_path + else: + files["compiler"] = args.pimcomp_dir / "build/PIMCOMP-NN" + files["exporter"] = args.pimcomp_dir / "verification/export_to_pim_simulator.py" + files.update({f"frontend/{p.relative_to(args.pimcomp_dir / 'frontend')}": p + for p in sorted((args.pimcomp_dir / "frontend").rglob("*.py"))}) + settings = {name: getattr(args, name) for name in ( + "pimsim_mode", "pimsim_time_ms", "seed", "batch_size", "threshold", "rtol", + "no_fast", "skip_pimsim_nn")} + settings.update({name: getattr(args, name) for name in ( + ("raptor_extra_arg",) if side == "raptor" else + ("pimcomp_pipeline", "pimcomp_replication", "pimcomp_model_name"))}) + return {"version": 2, "validation_policy": VALIDATION_POLICY, + "files": fingerprint_files(files), "hardware": hardware, "settings": settings} + + +def pimcomp_reuse_context(args, model, functional_model, hardware): + return experiment_context(args, model, functional_model, hardware, "pimcomp") + + +def artifact_signatures(report, side): + names = (("raptor_pim", "raptor_pimsim_nn", "reference_runner", "reference_library", + "reference_inputs", "batch_inputs") if side == "raptor" else + ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn")) + signatures = {} + content = {} + for name in names: + root = path_from_report(report, name) + if root is None: + continue + if not root.exists(): + signatures[name] = None + continue + paths = [root] if root.is_file() else sorted(p for p in root.rglob("*") if p.is_file()) + binary = root.is_dir() and any(root.glob("core_*.pim")) + digest = hashlib.sha256() + for path in paths: + # JSON instruction dumps are redundant when native binaries exist. + if binary and path.parent == root and path.match("core_*.json"): + continue + resolved = path.resolve() + if resolved not in content: + with path.open("rb") as stream: + content[resolved] = hashlib.file_digest(stream, "sha256").hexdigest() + digest.update(str(path.relative_to(root) if root.is_dir() else path.name).encode()) + digest.update(b"\0" + content[resolved].encode() + b"\0") + signatures[name] = digest.hexdigest() + return signatures + + +def verify_reused_artifacts(report, side): + if report.get("artifact_signatures", {}).get(side) != artifact_signatures(report, side): + raise ValueError(f"Reused {side} artifacts differ; regenerate the report") def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> dict[str, Any]: @@ -1212,6 +1283,7 @@ def load_reused_pimcomp_report(path: Path, context: dict[str, Any] | None) -> di restore_side_records(report, "pimcomp", [], []) for name in ("pimcomp_simulation_info", "pimcomp_exported_pim", "pimcomp_pimsim_nn", "pimsim_config"): path_from_report(report, name) + verify_reused_artifacts(report, "pimcomp") return report except (OSError, ValueError, TypeError, AttributeError, KeyError) as exc: raise ValueError(f"Cannot reuse Pimcomp report {path}: {exc}. Rerun Pimcomp to regenerate it.") from exc @@ -1334,14 +1406,19 @@ def write_report( if pimcomp_validation.error: lines.append(f"- Pimcomp validation note: `{pimcomp_validation.error.splitlines()[0]}`") - lines.extend(["", "### Max output differences", ""]) + lines.extend(["", "### Output errors", ""]) diff_names = sorted(set(raptor_validation.max_diffs) | set(pimcomp_validation.max_diffs)) if diff_names: - lines.extend(["| Output | Raptor max diff | Pimcomp max diff |", "|---|---:|---:|"]) + lines.extend(["| Output | Raptor max diff | Pimcomp max diff | Raptor relative L2 | Pimcomp relative L2 |", + "|---|---:|---:|---:|---:|"]) for name in diff_names: + relative = [result.relative_l2_errors.get(name) + for result in (raptor_validation, pimcomp_validation)] + relative_text = [f"{value:.6e}" if value is not None else "N/A" for value in relative] lines.append( f"| `{name}` | {raptor_validation.max_diffs.get(name, float('nan')):.6e} | " - f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} |" + f"{pimcomp_validation.max_diffs.get(name, float('nan')):.6e} | " + f"{relative_text[0]} | {relative_text[1]} |" ) else: lines.append("No output differences are available because validation did not run or failed before comparison.") @@ -1358,24 +1435,24 @@ def write_report( if pimsim_mode == "throughput": lines.extend( [ - "| Compiler | Status | Avg latency | Throughput | Avg power | Avg energy | Output count |", + "| Compiler | Status | Output interval (ms) | Throughput | Average power (mW) | Energy per inference (pJ/inference) | Output count |", "|---|---|---:|---:|---:|---:|---:|", f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'average_latency_ms', 'ms')} | {perf_value(raptor_perf, 'throughput', 'samples/s')} | " - f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(raptor_perf, 'output_count')} |", + f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(raptor_perf, 'output_count')} |", f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'average_latency_ms', 'ms')} | {perf_value(pimcomp_perf, 'throughput', 'samples/s')} | " - f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/it')} | {perf_value(pimcomp_perf, 'output_count')} |", + f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} | {perf_value(pimcomp_perf, 'output_count')} |", "", ] ) else: lines.extend( [ - "| Compiler | Status | Latency | Avg power | Avg energy |", + "| Compiler | Status | Latency (ms) | Average power (mW) | Energy per inference (pJ/inference) |", "|---|---|---:|---:|---:|", f"| Raptor | {perf_status(raptor_perf)} | {perf_value(raptor_perf, 'latency_ms', 'ms')} | " - f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ')} |", + f"{perf_value(raptor_perf, 'average_power_mw', 'mW')} | {perf_value(raptor_perf, 'average_energy_pj', 'pJ/inference')} |", f"| Pimcomp | {perf_status(pimcomp_perf)} | {perf_value(pimcomp_perf, 'latency_ms', 'ms')} | " - f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ')} |", + f"{perf_value(pimcomp_perf, 'average_power_mw', 'mW')} | {perf_value(pimcomp_perf, 'average_energy_pj', 'pJ/inference')} |", "", ] ) @@ -1471,8 +1548,10 @@ def main(): parser.add_argument("--pim-simulator-dir", default=REPO / "backend-simulators/pim/pim-simulator", type=Path) parser.add_argument("--pimsim-nn-build-dir", default=REPO / "backend-simulators/pim/pimsim-nn/build", type=Path) parser.add_argument("--seed", type=int, default=0) - parser.add_argument("--threshold", type=float, default=1e-3) - parser.add_argument("--rtol", type=float, default=1e-4) + parser.add_argument("--threshold", type=float, default=1e-3, + help="Absolute tolerance for per-element output comparison.") + parser.add_argument("--rtol", type=float, default=1e-4, + help="Relative tolerance for per-element and output L2 comparison; 0 disables the L2 check.") parser.add_argument( "--timeout-seconds", type=float, @@ -1671,6 +1750,14 @@ def main(): reuse_report_path = args.reuse_raptor_report.resolve() with open(reuse_report_path, "r", encoding="utf-8") as f: reused = json.load(f) + if reused.get("validation_policy") != VALIDATION_POLICY: + raise ValueError("Reused Raptor validation policy differs; regenerate the Raptor report") + if reused.get("validation_tolerances") != {"atol": args.threshold, "rtol": args.rtol}: + raise ValueError("Reused Raptor validation tolerances differ; regenerate the Raptor report") + expected = experiment_context(args, model_path, functional_model_path, hardware, "raptor") + if reused.get("raptor_reuse_context") != expected: + raise ValueError("Reused Raptor model, compiler, simulation settings, or inputs differ; regenerate the report") + verify_reused_artifacts(reused, "raptor") reused_hardware = reused["hardware"] if reused_hardware != hardware: raise ValueError(f"Reused Raptor hardware differs: {reused_hardware} != {hardware}") @@ -1848,13 +1935,13 @@ def main(): ) raptor_validation = validation if validation is not None else failed_validation("Raptor validation failed") elif not reference_dirs: - raptor_validation = skipped_validation("Reference outputs are not available") + raptor_validation = failed_validation("Reference outputs are not available") elif not outputs_desc: - raptor_validation = skipped_validation("Output descriptors are not available") + raptor_validation = failed_validation("Output descriptors are not available") else: - raptor_validation = skipped_validation("Raptor input materialization failed") + raptor_validation = failed_validation("Raptor input materialization failed") elif not reuse_raptor: - raptor_validation = skipped_validation("Raptor Pim compilation did not produce a Pim directory") + raptor_validation = failed_validation("Raptor Pim compilation did not produce a Pim directory") pimcomp_model_path = model_path @@ -2058,6 +2145,8 @@ def main(): ) json_report = { + "validation_policy": VALIDATION_POLICY, + "validation_tolerances": {"atol": args.threshold, "rtol": args.rtol}, "model": str(model_path), "functional_model": str(functional_model_path), "hardware": hardware, @@ -2066,7 +2155,9 @@ def main(): "pimcomp_pipeline": args.pimcomp_pipeline, "pimcomp_replication": args.pimcomp_replication, "pimcomp_model_source": "not_run" if args.raptor_only else "original_onnx", - "pimcomp_reuse_context": reuse_context, + "pimcomp_reuse_context": (reuse_context if reuse_pimcomp else + None if args.raptor_only else pimcomp_reuse_context(args, model_path, functional_model_path, hardware)), + "raptor_reuse_context": experiment_context(args, model_path, functional_model_path, hardware, "raptor"), "pimcomp_config": str(args.pimcomp_config), "raptor_extra_args": args.raptor_extra_arg, "reused_raptor_report": optional_path(args.reuse_raptor_report.resolve()) if reuse_raptor else None, @@ -2087,6 +2178,8 @@ def main(): "batch_inputs": optional_path(out_dir / "inputs.csv"), "batch_outputs": optional_path(out_dir / "simulation/out_iterations"), "reference_runner": optional_path(runner_path), + "reference_library": optional_path(common_dir / "runner" / f"{functional_model_path.stem}.so") + if (common_dir / "runner" / f"{functional_model_path.stem}.so").is_file() else None, "raptor_pim": optional_path(raptor_pim_dir), "raptor_functional_pim": optional_path(raptor_functional_pim_dir), "raptor_pimsim_nn": optional_path(raptor_pimsim_dir), @@ -2097,6 +2190,8 @@ def main(): "report_markdown": str(report_path), }, } + json_report["artifact_signatures"] = { + side: artifact_signatures(json_report, side) for side in ("raptor", "pimcomp")} json_path = out_dir / "pimcomp/comparison_report.json" with open(json_path, "w", encoding="utf-8") as f: json.dump(json_report, f, indent=2) diff --git a/validation/tools/pim/pimcomp/compare/generate_plots.py b/validation/tools/pim/pimcomp/compare/generate_plots.py index cac9938..983758e 100755 --- a/validation/tools/pim/pimcomp/compare/generate_plots.py +++ b/validation/tools/pim/pimcomp/compare/generate_plots.py @@ -11,6 +11,7 @@ beside it, with outputs grouped in png/, svg/, and pdf/ under the output directo YOLO models without PIMCOMP measurements get separate Raptor-only charts. Comparisons use vertical bars for ratios, diverging advantages, and absolute values, with compiler colors and patterns for the Raptor pipeline stages. +A compact dot-plot summary combines latency, throughput, and both energy cases. Throughput-mode latency is the output interval, a bottleneck-stage proxy, not end-to-end inference latency. Separate throughput plots use inferences/s. Energy is reported per inference; power is averaged over the simulation. @@ -36,11 +37,13 @@ matplotlib.use("Agg") import matplotlib.pyplot as plt from matplotlib.patches import Patch, Rectangle from matplotlib.lines import Line2D +from matplotlib.text import Text from matplotlib.ticker import FuncFormatter, MultipleLocator +from matplotlib.transforms import Bbox sys.path.insert(0, str(Path(__file__).resolve().parents[2])) -from plot_style import (ARCH_NAMES, BAND_COLOR, COMPILER_NAMES, COMPILER_STYLES, - ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR, METRIC_STYLES, +from plot_style import (ARCH_NAMES, ARCH_STYLES, BAND_COLOR, COMPACT_MODE_NAMES, COMPILER_NAMES, COMPILER_STYLES, + DAGGER_COLOR, ERROR_COLOR, LINE_COLOR, MATRIX_TEXT_COLOR, MODE_NAMES, MODEL_NAMES, NETWORK_STYLES, PALETTE, TEXT_COLOR, add_legend, configure_style, fit_value_labels, fit_bar_widths, fit_x_tick_labels, save_figure, value_label) @@ -276,7 +279,7 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal gap = 0.4 if index and mode != configurations[index - 1][0] else 0 positions.append(positions[-1] + 1 + gap if index else 0) configuration_x = dict(zip(configurations, positions)) - headers = [f"Raptor\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact + headers = [f"{COMPILER_NAMES['raptor']}\n{stage} stages" if mode == "throughput" and compiler == "raptor" and not compact else "PIMCOMP\nbatch" if mode == "throughput" and compiler == "pimcomp" and not compact else COMPILER_NAMES[compiler] for mode, stage, compiler in configurations] @@ -288,7 +291,8 @@ def draw_validation(results: list[Result], output_dir: Path, compact: bool = Fal for mode, group in groupby(zip(configurations, positions), key=lambda item: item[0][0]): group_positions = [x for _, x in group] centers.append((group_positions[0] + group_positions[-1]) / 2) - modes.append("Latency" if mode == "latency" else MODE_NAMES[mode]) + modes.append(COMPACT_MODE_NAMES[mode] if compact else + "Latency" if mode == "latency" else MODE_NAMES[mode]) ax.set_xticks(centers, modes, fontsize=10) ax.set_xticks(positions, headers, minor=True) ax.tick_params(axis="x", which="minor", top=True, labeltop=True, @@ -320,6 +324,140 @@ def select_compact_speedups(results: list[Result]) -> list[Result]: return selected +def draw_compact_summary(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None: + selected = select_compact_speedups([row for row in results if row.model != "yolo11n"]) + lookup = {(row.model, row.arch, row.mode): row for row in selected} + keys = [(model, arch) for model in MODELS if model != "yolo11n" for arch in ARCHITECTURES] + panels = ((f"{COMPACT_MODE_NAMES['latency']}\nspeedup", "latency", "latency_speedup"), + (f"{COMPACT_MODE_NAMES['throughput']}\nspeedup", "throughput", "throughput_speedup"), + (f"{COMPACT_MODE_NAMES['latency']}\nenergy reduction", "latency", "energy"), + (f"{COMPACT_MODE_NAMES['throughput']}\nenergy reduction", "throughput", "energy")) + measurements = [] + for _, mode, field in panels: + points = [] + for model, arch in keys: + row = lookup[(model, arch, mode)] + ratio = getattr(row, field) + if not math.isfinite(ratio) or ratio <= 0: + raise ValueError(f"Compact summary requires a positive {field} ratio for {arch}/{model}") + pimcomp_invalid = row.pimcomp_schedule == "batch" or row.pimcomp_validation == "FAIL" + status = ("P. INVALID" if pimcomp_invalid else + f"{COMPILER_NAMES['raptor'][0]}. INVALID" if row.raptor_validation == "FAIL" else + "NOT VALIDATED" if not row.comparison_passed else "") + points.append((row, ratio, status)) + measurements.append(points) + ratios = [ratio for points in measurements for _, ratio, _ in points] + padding = max(max(ratios) - min(ratios), 1) * 0.04 + limits = (0.25 if min(ratios) < 0.5 else 0.5, max(1, *ratios) + padding) + fig, axes = plt.subplots(4, 1, sharex=True, figsize=(6.8, 7.6), layout="constrained") + markers = dict(zip((model for model in MODELS if model != "yolo11n"), ("o", "s", "^", "D"))) + lanes = {arch: index * 2 for index, arch in enumerate(ARCHITECTURES)} + daggers = {} + for ax, (title, _, _), points in zip(axes, panels, measurements): + ax.set_xlim(*limits) + ax.set_ylim(4, -2) + ax.set_ylabel(title, rotation=90, ha="center", va="bottom", multialignment="center", + fontsize=8.5, labelpad=10) + ax.set_yticks(list(lanes.values()), [ARCH_NAMES[arch] for arch in lanes], fontsize=8) + ax.set_xticks([limits[0], *range(1, math.floor(limits[1]) + 1)]) + ax.xaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{value:g}")) + ax.tick_params(axis="both", length=0, pad=4, labelsize=8.5) + ax.tick_params(axis="x", bottom=False, labelbottom=False, + top=ax is axes[0], labeltop=ax is axes[0]) + ax.grid(axis="x", linewidth=0.6) + ax.axvline(1, color=TEXT_COLOR, linewidth=0.9, linestyle="--", zorder=1) + for lane in lanes.values(): + ax.axhline(lane, color=LINE_COLOR, linewidth=0.6) + ax.spines["left"].set_visible(False) + ax.spines["bottom"].set_visible(False) + ax.spines["top"].set_visible(ax is axes[0]) + for row, ratio, status in points: + fill, border = NETWORK_STYLES[row.model] + lane = lanes[row.arch] + ax.plot(ratio, lane, marker=markers[row.model], markersize=6.5, linestyle="none", + markerfacecolor=fill if marker_borders else border, markeredgecolor=border, + markeredgewidth=1.1 if marker_borders else 0) + suffix = f" [{status}]" if status and status != "P. INVALID" else "" + label = f"{ratio:.2f}×{suffix}" + annotation = ax.annotate(label, (ratio, lane), xytext=(0, -5), textcoords="offset points", + rotation=90, rotation_mode="anchor", ha="right", va="center", + fontsize=7.5, color=TEXT_COLOR, math_fontfamily="cm") + annotation.set_in_layout(False) + dagger = ax.annotate(r"$\boldsymbol{\dagger}$", (0.5, 1), xycoords=annotation, + xytext=(0, 1), textcoords="offset points", rotation=90, + rotation_mode="anchor", ha="left", va="center", fontsize=9, + color=DAGGER_COLOR, math_fontfamily="cm", + alpha=1 if status == "P. INVALID" else 0) + dagger.set_in_layout(False) + daggers[annotation] = dagger + handles = [Line2D([], [], marker=marker, color=NETWORK_STYLES[model][1], + markerfacecolor=NETWORK_STYLES[model][0 if marker_borders else 1], markersize=6.5, + markeredgewidth=1 if marker_borders else 0, + linestyle="none", label=MODEL_NAMES[model]) + for model, marker in markers.items()] + handles.append(Text(0, 0, r"$\boldsymbol{\dagger}$", color=DAGGER_COLOR, fontsize=9, + math_fontfamily="cm", label="PIMCOMP invalid")) + add_legend(fig, handles) + axes[0].xaxis.set_label_position("top") + axes[0].set_xlabel("Ratio (x)", fontsize=9, labelpad=7, va="bottom") + axes[0].xaxis.set_label_coords(0.5, 1.18) + axes[0].text(0, 1.18, "← PIMCOMP better", transform=axes[0].transAxes, ha="left", va="bottom", fontsize=8) + axes[0].text(1, 1.18, f"{COMPILER_NAMES['raptor']} better →", transform=axes[0].transAxes, + ha="right", va="bottom", fontsize=8) + fig.canvas.draw() + renderer = fig.canvas.get_renderer() + + def label_bounds(text): + return Bbox.union([text.get_window_extent(renderer), daggers[text].get_window_extent(renderer)]) + + def position_label(text, dx, dy): + text.set_ha("left" if dy > 0 else "right") + extra = (daggers[text].get_window_extent(renderer).height + renderer.points_to_pixels(1) + if dy < 0 else 0) + text.set_position((dx, dy - extra * 72 / fig.dpi)) + + labels_by_axis = [[text for text in ax.texts if getattr(text, "xycoords", None) == "data"] for ax in axes] + label_height = max(label_bounds(text).height for labels in labels_by_axis for text in labels) + lane_gap = min(abs(ax.transData.transform((0, 0))[1] - ax.transData.transform((0, 2))[1]) + for ax in axes) + height_scale = (label_height + renderer.points_to_pixels(5)) / lane_gap + if height_scale > 1: + fig.set_size_inches(fig.get_figwidth(), fig.get_figheight() * height_scale) + fig.canvas.draw() + renderer = fig.canvas.get_renderer() + for ax, labels in zip(axes, labels_by_axis): + placed = [] + for line in ax.lines: + placed.append(line.get_window_extent(renderer).padded(1)) + bounds = ax.get_window_extent(renderer) + choices = [] + for text in labels: + preferred_dy = 5 if text.xy[1] == lanes[ARCHITECTURES[0]] else -5 + candidates = [] + for dx in (0, -3, 3, -6, 6, -8, 8): + for dy in (-5, 5): + position_label(text, dx, dy) + box = label_bounds(text) + overlap = any(box.overlaps(other) for other in placed) + outside = not (bounds.x0 <= box.x0 <= box.x1 <= bounds.x1 and + bounds.y0 <= box.y0 <= box.y1 <= bounds.y1) + if not outside and not overlap: + candidates.append(((dy != preferred_dy, abs(dx) + abs(dy)), dx, dy, box)) + choices.append((text, candidates)) + layouts = [((0, 0), [])] + for text, candidates in sorted(choices, key=lambda choice: len(choice[1])): + layouts = sorted( + (((cost[0] + distance[0], cost[1] + distance[1]), assignments + [(text, dx, dy, box)]) + for cost, assignments in layouts for distance, dx, dy, box in candidates + if not any(box.overlaps(other[3]) for other in assignments)), + key=lambda layout: layout[0])[:32] + if not layouts: + raise ValueError("Dot labels do not fit without overlap; increase lane spacing or figure width") + for text, dx, dy, _ in layouts[0][1]: + position_label(text, dx, dy) + save_figure(fig, "performance_energy_compact", output_dir) + + def draw_metric(results: list[Result], field: str, ylabel: str, basename: str, output_dir: Path, diverging: bool = False, compact: bool = False) -> None: @@ -327,48 +465,52 @@ def draw_metric(results: list[Result], field: str, ylabel: str, for row in results] available = [v for v in data if math.isfinite(v)] upper = max(0, max(available, default=1.0)) - lower = min(0, min(available, default=0)) if diverging else 0 + lower = (min(0, min(available, default=0)) if diverging else + 0.25 if min(available, default=1) < 0.5 else 0.5) padding = max(upper - lower, 1.0) * 0.3 y_max = math.ceil((upper + padding) * 2) / 2 - y_min = math.floor((lower - padding) * 2) / 2 if diverging else 0 + y_min = math.floor((lower - padding) * 2) / 2 if diverging else lower y_interval = 1.0 if y_max - y_min > 4.5 else 0.5 fig, axes = plt.subplots(1, 2, sharey=True, figsize=(8.5 if compact else 12, 4.4), layout="constrained") if compact: - single_metric = "latency" if field == "speedup" else field - throughput_suffix = "" if field == "speedup" else f" {field}" - handles = [Patch(facecolor=METRIC_STYLES["latency"][0], - edgecolor=METRIC_STYLES["latency"][1], - label=f"Single-inference {single_metric}"), - Patch(facecolor=METRIC_STYLES["throughput"][0], - edgecolor=METRIC_STYLES["throughput"][1], hatch="//", - label=f"Best-case throughput{throughput_suffix}")] + metric_suffix = "" if field == "speedup" else f" {field}" + panel_labels = {mode: label + metric_suffix for mode, label in COMPACT_MODE_NAMES.items()} + handles = [Patch(facecolor=fill, edgecolor=border, hatch=pattern, label=ARCH_NAMES[arch]) + for arch, (fill, border, pattern) in ARCH_STYLES.items()] - for ax, arch in zip(axes, ARCHITECTURES): - selected = [row for row in results if row.arch == arch] + panels = ("latency", "throughput") if compact else ARCHITECTURES + for panel_index, (ax, panel) in enumerate(zip(axes, panels)): + selected = [row for row in results if getattr(row, "mode" if compact else "arch") == panel] + if compact: + selected.sort(key=lambda row: (MODELS.index(row.model), ARCHITECTURES.index(row.arch))) positions = [index + index // 2 for index in range(len(selected))] if compact else grouped_positions(selected) - ax.set_xlabel(ARCH_NAMES[arch], + ax.set_xlabel(panel_labels[panel] if compact else ARCH_NAMES[panel], fontsize=11, weight="bold", labelpad=8) ax.set_axisbelow(True) ax.set_ylim(y_min, y_max) ax.set_xlim(-0.9, positions[-1] + 0.9 if positions else 0.9) label_groups(ax, selected, positions) - ax.tick_params(axis="y", labelleft=arch == ARCHITECTURES[0], pad=5, length=0) + ax.tick_params(axis="y", labelleft=panel_index == 0, pad=5, length=0) ax.tick_params(axis="x", labelsize=9.5, length=0, pad=6) ax.yaxis.set_major_locator(MultipleLocator(y_interval)) if diverging: ax.yaxis.set_major_formatter(FuncFormatter(lambda value, _: f"{abs(value) + 1:g}")) + else: + ax.set_yticks([y_min], minor=True) + ax.yaxis.set_minor_formatter(FuncFormatter(lambda value, _: f"{value:g}")) + ax.tick_params(axis="y", which="minor", labelleft=panel_index == 0, pad=5, length=0) ax.grid(axis="y") ax.axhline(0 if diverging else 1, color=TEXT_COLOR, linewidth=1.0, linestyle="--", zorder=3) - if arch == ARCHITECTURES[0]: + if panel_index == 0: ax.set_ylabel(ylabel, fontsize=10.5, labelpad=6) for position, row in zip(positions, selected): ratio = getattr(row, field) if not math.isfinite(ratio): - ax.text(position, (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom", + ax.text(position, (0 if diverging else y_min) + (y_max - y_min) * 0.02, "N/A", ha="center", va="bottom", fontsize=9.5, color=TEXT_COLOR, fontstyle="italic") continue compiler = advantage_compiler(ratio) @@ -377,8 +519,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str, pattern = STAGE_PATTERNS[row.pipeline] color, border = COMPILER_STYLES[compiler] if compiler else ("white", TEXT_COLOR) if compact: - pattern = "//" if row.mode == "throughput" else "" - color, border = METRIC_STYLES[row.mode] + color, border, pattern = ARCH_STYLES[row.arch] measurement = diverging_ratio(ratio) if diverging else ratio ax.bar(position, measurement, width=1.0, color=color, @@ -395,7 +536,7 @@ def draw_metric(results: list[Result], field: str, ylabel: str, else: figure_legend(fig, results, field) fit_x_tick_labels(fig) - fit_value_labels(fig) + fit_value_labels(fig, lower_limit=None if diverging else y_min) save_figure(fig, basename, output_dir) @@ -463,7 +604,7 @@ def draw_absolute(results: list[Result], field: str, ylabel: str, save_figure(fig, basename, output_dir) -def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None: +def draw_pipeline_effect(results: list[Result], output_dir: Path, *, marker_borders: bool = True) -> None: rows = [row for row in results if row.mode == "throughput" or row.pipeline == 1] stages = sorted({row.pipeline for row in rows}) panels = [model for model in MODELS if any(row.model == model for row in rows)] @@ -479,15 +620,12 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None: ax = axes[index, column] keys = [(arch, model, stage) for stage in stages] values = [rates.get(key, math.nan) for key in keys] - valid = [key in lookup and lookup[key].raptor_validation == "PASS" for key in keys] ax.plot(stages, values, color=border, linewidth=1.8) - for stage, value, passed in zip(stages, values, valid): + for stage, value in zip(stages, values): if math.isfinite(value): - marker = ("s" if stage == 1 else "o") if passed else "x" - status = lookup[(arch, model, stage)].raptor_validation - ax.plot(stage, value, color=ERROR_COLOR if status == "FAIL" else border, - markerfacecolor=fill, marker=marker, - markersize=6, linestyle="none") + ax.plot(stage, value, color=border, + markerfacecolor=fill if marker_borders else border, marker="s" if stage == 1 else "o", + markeredgewidth=1 if marker_borders else 0, markersize=7, linestyle="none") ax.set_xticks(stages) if index == 0: ax.set_title(MODEL_NAMES[model], fontsize=10.5, weight="normal", pad=8) @@ -500,15 +638,13 @@ def draw_pipeline_effect(results: list[Result], output_dir: Path) -> None: ax.set_ylabel(ARCH_NAMES[arch], fontsize=10.5, weight="bold", rotation=270, labelpad=24) for ax in axes.flat: ax.margins(y=0.08) - handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white", - marker=marker, linestyle="none", label=label) + handles = [Line2D([], [], color=TEXT_COLOR, markerfacecolor="white" if marker_borders else TEXT_COLOR, + markeredgewidth=1 if marker_borders else 0, + marker=marker, markersize=7, linestyle="none", label=label) for marker, label in (("s", "Derived as 1 / latency"), ("o", "Throughput measurement"))] - if any(row.raptor_validation != "PASS" and math.isfinite(rates[key]) for key, row in lookup.items()): - handles.append(Line2D([], [], color=TEXT_COLOR, marker="x", linestyle="none", - label="Raptor validation not PASS")) add_legend(fig, handles) - fig.supxlabel("Raptor pipeline stages", fontsize=11) + fig.supxlabel(f"{COMPILER_NAMES['raptor']} pipeline stages", fontsize=11) fit_x_tick_labels(fig) save_figure(fig, "raptor_pipeline_throughput", output_dir) @@ -517,16 +653,22 @@ def main() -> None: parser = argparse.ArgumentParser(description=__doc__) script_dir = Path(__file__).resolve().parent parser.add_argument("--input", type=Path, default=script_dir / "results_comparison.csv") - parser.add_argument("--output-dir", type=Path, default=script_dir / "plots") + parser.add_argument("--output-dir", type=Path, default=script_dir / "plots/default") + parser.add_argument("--colors", action=argparse.BooleanOptionalAction, default=True, + help="Use the pastel palette (default); --no-colors exports grayscale charts") + parser.add_argument("--anonymize", action=argparse.BooleanOptionalAction, default=False, + help="Replace the displayed Raptor name with Anonimized") + parser.add_argument("--marker-borders", action=argparse.BooleanOptionalAction, default=True, + help="Outline markers (default); --no-marker-borders uses solid markers in dot and line charts") args = parser.parse_args() - configure_style() + configure_style(colors=args.colors, anonymize=args.anonymize) results = load_results(args.input) valid = sum(row.comparison_passed for row in results) print(f"Results: {len(results)} configurations, {valid} valid comparisons, " f"{len(results)-valid} comparisons without two PASS statuses") draw_validation(results, args.output_dir) draw_validation(results, args.output_dir, compact=True) - draw_pipeline_effect(results, args.output_dir) + draw_pipeline_effect(results, args.output_dir, marker_borders=args.marker_borders) yolo_only_models = { model for model in MODELS if model.startswith("yolo") and not any(math.isfinite(getattr(row, f"pimcomp_{field}")) @@ -536,9 +678,10 @@ def main() -> None: comparisons = [row for row in results if row.model not in yolo_only_models] yolo_only = [row for row in results if row.model in yolo_only_models] compact_speedups = select_compact_speedups(comparisons) + draw_compact_summary(comparisons, args.output_dir, marker_borders=args.marker_borders) for metric, label in (("speedup", "Speedup (x)"), - ("energy", "Energy-per-inference advantage (x)"), - ("power", "Average power advantage (x)")): + ("energy", "Energy reduction (x)"), + ("power", "Average power reduction (x)")): for suffix, diverging in (("comparison", False), ("diverging", True)): draw_metric(compact_speedups, metric, label, f"{metric}_compact_{suffix}", args.output_dir, diverging=diverging, compact=True) @@ -546,8 +689,8 @@ def main() -> None: ("latency_speedup", "longest_stage_latency", "Longest stage latency speedup (x)", None), ("latency_speedup", "latency", "Single-inference latency speedup (x)", "latency"), ("throughput_speedup", "throughput", "Throughput speedup (x)", "throughput"), - ("power", "power", "Average power advantage (x)", None), - ("energy", "energy", "Energy-per-inference advantage (x)", None), + ("power", "power", "Average power reduction (x)", None), + ("energy", "energy", "Energy reduction (x)", None), ]: selected = [row for row in comparisons if mode is None or row.mode == mode] draw_metric(selected, metric, label, diff --git a/validation/tools/pim/pimcomp/compare/results_comparison.csv b/validation/tools/pim/pimcomp/compare/results_comparison.csv index 51b38a5..1711033 100644 --- a/validation/tools/pim/pimcomp/compare/results_comparison.csv +++ b/validation/tools/pim/pimcomp/compare/results_comparison.csv @@ -1,41 +1,41 @@ arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup -arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.04,1597904071.12,raptor,5.26 -arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,429.31,482.60,407972136.20,541691691.00,raptor,1.14 -arch-a,vgg8,throughput,4,batch,PASS,PASS,0.78,1.04,1280.00,965.00,457.04,482.60,415761605.20,541691691.00,raptor,1.33 -arch-a,vgg8,throughput,8,batch,PASS,PASS,0.78,1.04,1290.00,965.00,464.07,482.60,396424007.70,541691691.00,raptor,1.34 -arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394478521.12,13868866808.12,raptor,4.15 -arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15 -arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.21,13.46,52.10,74.30,380.10,481.60,7957033431.00,6563451004.00,pimcomp,1.43 -arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.93,13.46,50.20,74.30,361.80,481.60,7763795670.00,6563451004.00,pimcomp,1.48 -arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24 -arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.27,23.58,43.00,42.40,471.72,515.68,13055037600.00,12283627410.00,raptor,1.01 -arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.47,23.58,36.40,42.40,422.22,515.68,14899055210.00,12283627410.00,pimcomp,1.16 -arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.84,23.58,25.10,42.40,352.28,515.68,17062658890.00,12283627410.00,pimcomp,1.69 -arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.87,231.19,5513262106.92,14547744283.24,raptor,6.44 -arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.42,18.26,119.00,54.80,610.75,435.11,5777456331.00,8164787803.00,raptor,2.17 -arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.73,18.26,93.20,54.80,528.03,435.11,6592892213.00,8164787803.00,raptor,1.70 -arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.12,18.26,98.80,54.80,523.81,435.11,6521737843.00,8164787803.00,raptor,1.80 -arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA -arch-a,yolo11n,throughput,2,batch,PASS,FAIL,102.34,NA,9.77,NA,337.88,NA,38308060150.00,NA,NA,NA -arch-a,yolo11n,throughput,4,batch,PASS,FAIL,87.91,NA,11.40,NA,358.20,NA,38729846260.00,NA,NA,NA -arch-a,yolo11n,throughput,8,batch,PASS,FAIL,83.68,NA,11.90,NA,365.64,NA,38623589740.00,NA,NA,NA -arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66 -arch-b,vgg8,throughput,2,batch,PASS,PASS,0.69,1.07,1460.00,932.00,456.04,409.47,403694238.30,439288835.80,raptor,1.57 -arch-b,vgg8,throughput,4,batch,PASS,PASS,0.67,1.07,1490.00,932.00,448.90,409.47,413526781.30,439288835.80,raptor,1.60 -arch-b,vgg8,throughput,8,batch,PASS,PASS,0.75,1.07,1330.00,932.00,425.74,409.47,394457860.80,439288835.80,raptor,1.43 -arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19 -arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.12,13.43,82.50,74.40,461.41,484.80,6295321671.00,6569655551.00,raptor,1.11 -arch-b,resnet18,throughput,4,batch,PASS,FAIL,12.68,13.43,78.90,74.40,451.19,484.80,7096979122.00,6569655551.00,raptor,1.06 -arch-b,resnet18,throughput,8,batch,PASS,FAIL,20.59,13.43,48.60,74.40,321.63,484.80,7162570047.00,6569655551.00,pimcomp,1.53 -arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83 -arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03 -arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.09,22.85,34.40,43.80,364.96,493.33,16328513230.00,11392215660.00,pimcomp,1.27 -arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.85,21.00,43.80,289.82,493.33,16346419570.00,11392215660.00,pimcomp,2.09 -arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422781257.92,9182062336.24,raptor,3.38 -arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21,21.39,109.00,46.80,545.49,417.87,5593397699.00,9014726090.00,raptor,2.33 -arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56,21.39,117.00,46.80,555.26,417.87,5863121048.00,9014726090.00,raptor,2.50 -arch-b,googlenet,throughput,8,batch,PASS,FAIL,16.19,21.39,61.80,46.80,355.70,417.87,6726458739.00,9014726090.00,raptor,1.32 -arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA -arch-b,yolo11n,throughput,2,batch,PASS,FAIL,104.49,NA,9.57,NA,302.12,NA,35801517850.00,NA,NA,NA -arch-b,yolo11n,throughput,4,batch,PASS,FAIL,108.25,NA,9.24,NA,297.48,NA,37629699620.00,NA,NA,NA -arch-b,yolo11n,throughput,8,batch,PASS,FAIL,100.13,NA,9.99,NA,308.83,NA,37180785770.00,NA,NA,NA +arch-a,vgg8,latency,1,element,PASS,PASS,1.51833,7.98507,NA,NA,321.046,200.111,4.87455e+08,1.5979e+09,raptor,5.26 +arch-a,vgg8,throughput,2,batch,PASS,PASS,0.906321,1.0364,1100,965,429.311,482.597,4.07972e+08,5.41692e+08,raptor,1.14 +arch-a,vgg8,throughput,4,batch,PASS,PASS,0.778242,1.0364,1280,965,457.039,482.597,4.15762e+08,5.41692e+08,raptor,1.33 +arch-a,vgg8,throughput,8,batch,PASS,PASS,0.778094,1.0364,1290,965,464.071,482.597,3.96424e+08,5.41692e+08,raptor,1.34 +arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,raptor,4.15 +arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.7314,13.4585,85.2,74.3,503.111,481.604,6.62807e+09,6.56345e+09,raptor,1.15 +arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.2072,13.4585,52.1,74.3,380.1,481.604,7.95703e+09,6.56345e+09,pimcomp,1.43 +arch-a,resnet18,throughput,8,batch,PASS,FAIL,14.1578,13.4585,70.6,74.3,453.416,481.604,6.77909e+09,6.56345e+09,pimcomp,1.05 +arch-a,resnet34,latency,1,element,PASS,FAIL,29.7186,66.7114,NA,NA,411.806,277.485,1.22383e+10,1.85114e+10,raptor,2.24 +arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.2705,23.5817,43,42.4,471.717,515.68,1.3055e+10,1.22836e+10,raptor,1.01 +arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.468,23.5817,36.4,42.4,422.223,515.68,1.48991e+10,1.22836e+10,pimcomp,1.16 +arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.8392,23.5817,25.1,42.4,352.28,515.68,1.70627e+10,1.22836e+10,pimcomp,1.69 +arch-a,googlenet,latency,1,element,PASS,PASS,9.77772,62.9248,NA,NA,563.864,231.193,5.5133e+09,1.45477e+10,raptor,6.44 +arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.41545,18.2643,119,54.8,610.746,435.107,5.77746e+09,8.16479e+09,raptor,2.17 +arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.7292,18.2643,93.2,54.8,528.026,435.107,6.59289e+09,8.16479e+09,raptor,1.70 +arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.1219,18.2643,98.8,54.8,523.808,435.107,6.52174e+09,8.16479e+09,raptor,1.80 +arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.174,NA,4.27977e+10,NA,NA,NA +arch-a,yolo11n,throughput,2,batch,SKIPPED,FAIL,102.337,NA,9.77,NA,337.882,NA,3.83081e+10,NA,NA,NA +arch-a,yolo11n,throughput,4,batch,SKIPPED,FAIL,87.9103,NA,11.4,NA,358.201,NA,3.87298e+10,NA,NA,NA +arch-a,yolo11n,throughput,8,batch,SKIPPED,FAIL,82.9508,NA,12.1,NA,366.725,NA,3.85347e+10,NA,NA,NA +arch-b,vgg8,latency,1,element,PASS,PASS,1.5333,7.15212,NA,NA,292.559,173.769,4.48582e+08,1.24281e+09,raptor,4.66 +arch-b,vgg8,throughput,2,batch,PASS,PASS,0.685556,1.07282,1460,932,456.044,409.47,4.03694e+08,4.39289e+08,raptor,1.57 +arch-b,vgg8,throughput,4,batch,PASS,PASS,0.673192,1.07282,1490,932,448.898,409.47,4.13527e+08,4.39289e+08,raptor,1.60 +arch-b,vgg8,throughput,8,batch,PASS,PASS,0.751106,1.07282,1330,932,425.742,409.47,3.94458e+08,4.39289e+08,raptor,1.43 +arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,raptor,4.19 +arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.1214,13.434,82.5,74.4,461.407,484.804,6.29532e+09,6.56966e+09,raptor,1.11 +arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3601,13.434,88,74.4,479.333,484.804,6.98745e+09,6.56966e+09,raptor,1.18 +arch-b,resnet18,throughput,8,batch,PASS,FAIL,13.1216,13.434,76.2,74.4,441.248,484.804,6.38271e+09,6.56966e+09,raptor,1.02 +arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,raptor,3.83 +arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.282,22.8454,44.9,43.8,438.238,493.332,1.42187e+10,1.13922e+10,raptor,1.03 +arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.0906,22.8454,34.4,43.8,364.956,493.332,1.63285e+10,1.13922e+10,pimcomp,1.27 +arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.8454,21,43.8,289.821,493.332,1.63464e+10,1.13922e+10,pimcomp,2.09 +arch-b,googlenet,latency,1,element,PASS,PASS,11.2212,37.9367,NA,NA,483.266,242.037,5.42282e+09,9.18206e+09,raptor,3.38 +arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21209,21.3902,109,46.8,545.488,417.866,5.5934e+09,9.01473e+09,raptor,2.33 +arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56194,21.3902,117,46.8,555.264,417.866,5.86312e+09,9.01473e+09,raptor,2.50 +arch-b,googlenet,throughput,8,batch,PASS,FAIL,10.8652,21.3902,92,46.8,483.477,417.866,6.12683e+09,9.01473e+09,raptor,1.97 +arch-b,yolo11n,latency,1,element,PASS,FAIL,168.144,NA,NA,NA,245.314,NA,4.1248e+10,NA,NA,NA +arch-b,yolo11n,throughput,2,batch,SKIPPED,FAIL,104.494,NA,9.57,NA,302.121,NA,3.58015e+10,NA,NA,NA +arch-b,yolo11n,throughput,4,batch,SKIPPED,FAIL,108.247,NA,9.24,NA,297.484,NA,3.76297e+10,NA,NA,NA +arch-b,yolo11n,throughput,8,batch,SKIPPED,FAIL,100.129,NA,9.99,NA,308.833,NA,3.71808e+10,NA,NA,NA diff --git a/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py b/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py index 123036b..1b2848a 100755 --- a/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py +++ b/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py @@ -23,13 +23,13 @@ REPO = Path(__file__).resolve().parents[5] sys.path.insert(0, str(REPO / "validation")) from raptor_validation.pimcomp_models import ( - FUNCTIONAL_MODELS, MODELS, SUITE, add_models_argument, ) from raptor_validation.artifacts import artifacts_dir, remove_lock_files from raptor_validation.pimsim_nn import parse_pimsim_nn_metrics +from raptor_validation.output_comparison import VALIDATION_POLICY from raptor_validation.validate_one import STAGE_COLORS PIMCOMP_SOURCE = REPO / "third_party/PIMCOMP-NN" @@ -63,7 +63,7 @@ class ComparisonSpec: def model_dir(root: Path | None, name: str) -> Path: - return root / name if root is not None else MODELS[name].parent + return (root or SUITE) / name def result_dir( @@ -83,9 +83,9 @@ def result_dir( def common_dir(root: Path | None, name: str, common_root: Path | None = None) -> Path: - suffix = "common" if FUNCTIONAL_MODELS[name] == MODELS[name] else "common-functional" base = common_root / name if common_root is not None else model_dir(root, name) - return artifacts_dir(base) / suffix + # Keep references that retain Softmax separate from historical outputs. + return artifacts_dir(base) / ("common-full-model" if name in ("yolo11n", "googlenet") else "common") def clean_artifacts( @@ -199,6 +199,8 @@ def write_results_csv( } if report_path.exists(): report = json.loads(report_path.read_text(encoding="utf-8")) + if report.get("validation_policy") != VALIDATION_POLICY: + raise ValueError(f"Report uses a different validation policy: {report_path}. Regenerate this comparison.") raptor_values = performance_values(report.get("raptor_performance") or {}) pimcomp_values = performance_values(report.get("pimcomp_performance") or {}) raptor_metric = raptor_values["throughput"] if comparison_mode == "throughput" else raptor_values["latency"] @@ -222,7 +224,11 @@ def write_results_csv( if raptor_metric is not None and pimcomp_metric is not None: min_metric = min(raptor_metric, pimcomp_metric) row.update( - better_compiler=comparison_winner(comparison_mode, raptor_metric, pimcomp_metric), + better_compiler=( + comparison_winner(comparison_mode, raptor_metric, pimcomp_metric) + if row["raptor_functional_validation"] == row["pimcomp_functional_validation"] == "PASS" + else "NA" + ), speedup=( f"{max(raptor_metric, pimcomp_metric) / min_metric:.2f}" if min_metric > 0 @@ -684,10 +690,9 @@ def main() -> int: ) missing = [ - str(path) + str(MODELS[name]) for name in args.models - for path in (MODELS[name], FUNCTIONAL_MODELS[name]) - if not path.exists() + if not MODELS[name].exists() ] if missing: parser.error(f"missing model(s): {', '.join(missing)}") @@ -733,7 +738,7 @@ def main() -> int: try: run( prepare_common_command( - FUNCTIONAL_MODELS[name], + MODELS[name], common_dir(out_dir, name, common_root), args.timeout_seconds, reference_batch_size, @@ -770,7 +775,7 @@ def main() -> int: ComparisonSpec( label=label, model=MODELS[name], - functional_model=FUNCTIONAL_MODELS[name], + functional_model=MODELS[name], output_dir=model_result_dir, common_dir=common_dir(out_dir, name, common_root), config=configs[mode], diff --git a/validation/tools/pim/pimcomp/correctness/README.md b/validation/tools/pim/pimcomp/correctness/README.md index 9967036..a012e95 100644 --- a/validation/tools/pim/pimcomp/correctness/README.md +++ b/validation/tools/pim/pimcomp/correctness/README.md @@ -49,6 +49,13 @@ loaded provider tensors have no store producer; prefill reduces maximum reference error from about `6.70705` to `4.05e-6`. This reproduces Pimcomp's verifier contract; it does not repair the standalone batch program. +The reconstructed VGG8 batch program also loads intermediate Pool and Flatten +buffers without corresponding store producers. Its output can remain zero. +The small reference output can fit inside an absolute tolerance despite this +missing computation. Comparison therefore requires a relative L2 check in +addition to the elementwise check. A zero output against a nonzero reference +has relative L2 error one and must fail the default relative tolerance. + ## ResNet BatchNorm gap Pimcomp's [frontend fusion](../../../../../third_party/PIMCOMP-NN/frontend/frontend.py#L535) diff --git a/validation/tools/pim/plot_style.py b/validation/tools/pim/plot_style.py index d97fc1b..18131b9 100644 --- a/validation/tools/pim/plot_style.py +++ b/validation/tools/pim/plot_style.py @@ -6,10 +6,14 @@ from math import ceil import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt +from matplotlib.colors import to_rgba +from matplotlib.legend_handler import HandlerBase +from matplotlib.lines import Line2D from matplotlib.patches import Patch from matplotlib.path import Path as PlotPath from matplotlib.patheffects import AbstractPathEffect -from matplotlib.transforms import Affine2D, IdentityTransform, TransformedPath +from matplotlib.text import Text +from matplotlib.transforms import Affine2D, Bbox, IdentityTransform, TransformedPath PALETTE = { @@ -25,11 +29,12 @@ PALETTE = { TEXT_COLOR = "#222222" MATRIX_TEXT_COLOR = "#1D2530" ERROR_COLOR = PALETTE["red"][1] +DAGGER_COLOR = "#AA1C1C" GRID_COLOR = "#E6E9EC" LINE_COLOR = "#C9CED3" BAND_COLOR = "#F7F9FA" COMPILER_STYLES = {"raptor": PALETTE["sage"], "pimcomp": PALETTE["sand"]} -METRIC_STYLES = {"latency": PALETTE["blue"], "throughput": PALETTE["sage"]} +ARCH_STYLES = {"arch-a": (*PALETTE["blue"], ""), "arch-b": (*PALETTE["sage"], "//")} NETWORK_STYLES = {"vgg8": PALETTE["blue"], "resnet18": PALETTE["sage"], "resnet34": PALETTE["sand"], "googlenet": PALETTE["purple"], "yolo11n": PALETTE["red"]} @@ -38,6 +43,8 @@ ARCH_NAMES = {"arch-a": "Arch-A", "arch-b": "Arch-B"} MODEL_NAMES = {"vgg8": "VGG-8", "resnet18": "ResNet-18", "resnet34": "ResNet-34", "googlenet": "GoogLeNet", "yolo11n": "YOLO11n"} MODE_NAMES = {"latency": "Single-inference", "throughput": "Throughput"} +COMPACT_MODE_NAMES = {"latency": "Latency-mode", "throughput": "Throughput-mode"} +_USE_COLORS = True class CenteredHatch(AbstractPathEffect): @@ -47,11 +54,14 @@ class CenteredHatch(AbstractPathEffect): hatch = gc.get_hatch() or "" base_gc = renderer.new_gc() base_gc.copy_properties(gc) - base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\")) + base_gc.set_hatch("".join(symbol for symbol in hatch if symbol not in "./\\") or None) renderer.draw_path(base_gc, tpath, affine, rgbFace) base_gc.restore() bounds = affine.transform_path(tpath).get_extents() + clip_bounds = bounds if gc.get_clip_rectangle() is None else Bbox.intersection(bounds, gc.get_clip_rectangle()) + if clip_bounds is None: + return center_x, center_y = (bounds.x0 + bounds.x1) / 2, (bounds.y0 + bounds.y1) / 2 vertices = [] diagonal = 2 ** -0.5 @@ -61,13 +71,16 @@ class CenteredHatch(AbstractPathEffect): continue spacing = renderer.points_to_pixels(12 / count) half_span = (abs(dy) * bounds.width + abs(dx) * bounds.height) / 2 - half_length = (abs(dx) * bounds.width + abs(dy) * bounds.height) / 2 extent = ceil(half_span / spacing) for step in range(-extent, extent + 1): x = center_x - dy * step * spacing y = center_y + dx * step * spacing - vertices.extend(((x - dx * half_length, y - dy * half_length), - (x + dx * half_length, y + dy * half_length))) + y_limits = sorted(((clip_bounds.y0 - y) / dy, (clip_bounds.y1 - y) / dy)) + start = max((clip_bounds.x0 - x) / dx, y_limits[0]) + end = min((clip_bounds.x1 - x) / dx, y_limits[1]) + if start < end: + vertices.extend(((x + dx * start, y + dy * start), + (x + dx * end, y + dy * end))) line_gc = renderer.new_gc() line_gc.copy_properties(gc) line_gc.set_hatch(None) @@ -81,17 +94,22 @@ class CenteredHatch(AbstractPathEffect): spacing = renderer.points_to_pixels(12 / hatch.count(".")) x_extent = ceil(bounds.width / (2 * spacing)) y_extent = ceil(bounds.height / (2 * spacing)) - centers = [(center_x + x * spacing, center_y + y * spacing) - for x in range(-x_extent, x_extent + 1) - for y in range(-y_extent, y_extent + 1)] + # Trim geometry too: SVG importers can discard clip paths. + dots = [PlotPath.unit_circle().transformed( + Affine2D().scale(renderer.points_to_pixels(0.9)).translate( + center_x + x * spacing, center_y + y * spacing)).clip_to_bbox(clip_bounds) + for x in range(-x_extent, x_extent + 1) + for y in range(-y_extent, y_extent + 1)] line_gc.set_linewidth(0) - renderer.draw_markers(line_gc, PlotPath.unit_circle(), - Affine2D().scale(renderer.points_to_pixels(0.9)), - PlotPath(centers), IdentityTransform(), gc.get_hatch_color()) + renderer.draw_path(line_gc, PlotPath.make_compound_path(*dots), + IdentityTransform(), gc.get_hatch_color()) line_gc.restore() -def configure_style() -> None: +def configure_style(*, colors: bool = True, anonymize: bool = False) -> None: + global _USE_COLORS + _USE_COLORS = colors + COMPILER_NAMES["raptor"] = "Anonimized" if anonymize else "Raptor" plt.rcParams.update({ "font.family": "DejaVu Sans", "axes.spines.top": False, @@ -109,6 +127,14 @@ def configure_style() -> None: }) +class TextLegendHandler(HandlerBase): + def create_artists(self, legend, original, xdescent, ydescent, width, height, fontsize, transform): + return [Text(width / 2 - xdescent, height / 2 - ydescent, original.get_text(), + color=original.get_color(), fontproperties=original.get_fontproperties(), + math_fontfamily=original.get_math_fontfamily(), ha="center", va="center", + transform=transform)] + + def add_legend(fig, handles) -> None: """Use the fewest balanced legend rows that fit the figure width.""" renderer = fig.canvas.get_renderer() @@ -123,6 +149,7 @@ def add_legend(fig, handles) -> None: ordered = [handle for column in zip(*grid) for handle in column] legend = fig.legend(handles=[handle if handle is not None else Patch(visible=False) for handle in ordered], labels=[handle.get_label() if handle is not None else "" for handle in ordered], + handler_map={Text: TextLegendHandler()}, loc="outside upper center", ncols=columns, frameon=False, fontsize=9.5, handlelength=1.7, columnspacing=1.5, handletextpad=0.6, labelspacing=0.4) for handle, patch, text in zip(ordered, legend.legend_handles, legend.get_texts()): @@ -197,7 +224,7 @@ def fit_bar_widths(fig) -> None: fig.set_size_inches(fig.get_figwidth() * scale, fig.get_figheight()) -def fit_value_labels(fig) -> None: +def fit_value_labels(fig, *, lower_limit: float | None = None) -> None: """Fit common vertical limits to the data and rendered value labels.""" for ax in fig.axes: for text in ax.texts: @@ -215,18 +242,39 @@ def fit_value_labels(fig) -> None: labels.append((y, max(0, anchor - extent.y0 + 3) / bounds.height, max(0, extent.y1 - anchor + 3) / bounds.height)) limits.append(ax.dataLim.intervaly if ax.has_data() else ax.get_ylim()) - lower = min(0, *(lower for lower, _ in limits)) + lower = min(0, *(lower for lower, _ in limits)) if lower_limit is None else lower_limit upper = max(0, *(upper for _, upper in limits)) if lower == upper: upper = lower + 1 for _ in range(12): - lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)]) + if lower_limit is None: + lower = min([lower, *((y - below * upper) / (1 - below) for y, below, _ in labels)]) upper = max([upper, *((y - above * lower) / (1 - above) for y, _, above in labels)]) for ax in fig.axes: ax.set_ylim(lower, upper) +def grayscale_figure(fig) -> None: + def gray(color): + red, green, blue, alpha = to_rgba(color) + level = 0.2126 * red + 0.7152 * green + 0.0722 * blue + return level, level, level, alpha + + for artist in fig.findobj(): + if isinstance(artist, Patch): + artist.set_facecolor(gray(artist.get_facecolor())) + artist.set_edgecolor(gray(artist.get_edgecolor())) + elif isinstance(artist, Line2D): + artist.set_color(gray(artist.get_color())) + artist.set_markerfacecolor(gray(artist.get_markerfacecolor())) + artist.set_markeredgecolor(gray(artist.get_markeredgecolor())) + elif isinstance(artist, Text): + artist.set_color(gray(artist.get_color())) + + def save_figure(fig, basename: str, output_dir: Path) -> None: + if not _USE_COLORS: + grayscale_figure(fig) for patch in fig.findobj(Patch): hatch = patch.get_hatch() or "" if any(symbol in hatch for symbol in "./\\"): diff --git a/validation/validate.py b/validation/validate.py index 48576d5..9dbd97c 100755 --- a/validation/validate.py +++ b/validation/validate.py @@ -225,7 +225,7 @@ def main(): ap.add_argument("--threshold", type=float, default=1e-3, help="Absolute tolerance for per-element output comparison.") ap.add_argument("--relative-threshold", type=float, default=1e-5, - help="Relative tolerance for per-element output comparison.") + help="Relative tolerance for per-element and output L2 comparison; 0 disables the L2 check.") ap.add_argument("--seed", type=int, default=0, help="RNG seed for generated validation inputs.") ap.add_argument("--crossbar-size", type=int, default=default_hardware["crossbar_rows"], help=f"Crossbar size (default: {default_hardware['crossbar_rows']}).") @@ -416,10 +416,10 @@ def main(): n_passed = sum(1 for result in results.values() if result.passed) n_total = len(results) latency_headers = ("Operation", "Arch", "Result", "Compile", "Host mem", "Cores mem", - "Cores", "Xbars", "Latency", "Power", "Energy") + "Cores", "Xbars", "Latency", "Average power", "Energy per inference") throughput_headers = ("Operation", "Arch", "Result", "Compile", "Host mem", "Cores mem", - "Cores", "Xbars", "Throughput", "Avg latency", "Avg power", - "Avg energy") + "Cores", "Xbars", "Throughput", "Output interval", "Average power", + "Energy per inference") csv_headers = ( "Operation", "Arch", "Result (l)", "Result (t)", "Compile (l)", "Host mem (l)", "Cores mem (l)", "Cores (l)", "Xbars (l)",