diff --git a/docs/compiler/scheduling.md b/docs/compiler/scheduling.md index af3fe02..fd034b6 100644 --- a/docs/compiler/scheduling.md +++ b/docs/compiler/scheduling.md @@ -63,8 +63,11 @@ lanes of an operation and computes one ordered partition with a prefix DP. The DP jointly assigns operation prefixes and at most sixteen units of spare core capacity. Each configured stage reserves one physical core. Quantization limits initialization only; refinement uses geometric core transfers, including one core. -Quantization can miss a feasible allocation. Initialization failure reports a -bounded-search limitation, not proved hardware infeasibility. +If the quality grid has no feasible state, a minimum-resource prefix DP checks +every physical core count for each segment. This preserves feasibility over +the segment assignments found by the constructive packers, independently of +resource quantization. Packing failure remains a bounded-search limitation, +not proved hardware infeasibility. The seed minimizes maximum stage packing/dependency priority over the legal active-stage counts within the configured depth. These priorities provide a starting point, not a timing guarantee. Refinement can remove stage boundaries. @@ -76,6 +79,13 @@ constructive ordered-packing curves and uses reuse-first resident-weight packing when order alone cannot realize a capacity bound. A feasible packing remains feasible with idle extra cores. All placements pass the existing topology-aware physical packing, operation-group, adjacency and capacity checks. +Ordered packing returns an unavailable cost when no ordered assignment fits. +Every finite feasibility cost has an ordered or reuse-first constructive assignment. +Initialization retains that assignment before adding idle spare cores. If topology-aware +greedy placement and reconstruction in physical task order fail, it uses the +retained assignment. +Task reordering cannot discard the initial partition's constructive capacity evidence. +This evidence does not prove final backend legality. Refinement uses a beam of two schedules ranked by the larger queue estimate. Its ordered neighborhood moves operation groups across adjacent stage boundaries, @@ -163,7 +173,9 @@ Unsupported or incomplete target timing produces an `unavailable` report. The analysis brackets the maximum cycle ratio within one nanosecond. It tests edge delays `w - II*d`, sweeping the distance-zero DAG between recurrence -relaxations. At most `R+1` sweeps detect a positive cycle, where `R` counts +relaxations. A positive predecessor cycle can terminate an infeasible probe early. +The final witness pass retains the full relaxation procedure and deterministic +witness selection. At most `R+1` sweeps detect a positive cycle, where `R` counts positive-distance edges. Bisection uses at most 64 tests. Timing outside the supported numeric range produces an explicit unavailable result. A predecessor cycle supplies a witness, including delay, iteration distance, cores, and links. @@ -337,6 +349,11 @@ Functional simulation still executes Softmax. Estimated timing and energy exclud The shared queue engine models controller arrival arbitration and cross-generation dependencies. The compiler does not execute candidate programs or invoke a simulator. +Spatial extraction permits at most one million operation visits per physical core. +The total visit count is diagnostic; adding cores does not reduce another core's analysis allowance. +Abstract queue construction permits two million non-work event visits per candidate. +The complete-state recurrence limits remain unchanged. + Both arbitration policies must establish complete-state recurrence within bounded analysis budgets. Their estimates are sensitivity checks, not formal bounds. The candidate with the lowest larger estimate must separate from the initialized schedule. @@ -362,6 +379,8 @@ If every candidate fails, one diagnostic lists all attempted candidates and caus `reports/pipeline_selection.json` records Spatial costs, eligibility, selection, and backend attempts. It also records search parents, compact task-stage runs, core budgets, recurrence contributions, termination, attempts, queue budget, and phase times. +Phase times separate neighborhood scoring, constructive packing, dependency-witness +analysis, and queue execution with its complete-state proof. `--enable-timing` adds per-pass lowering times and IR operation/value counts. Final instruction generation uses the normal compiler timing report. Explicit Spatial exports and IR dumps reconstruct pre-lowering artifacts for the selected schedule. @@ -424,7 +443,8 @@ fragment/consumer demands, and resource quantization `Q <= 16`. | Weight indexing | Up to `O(R W)` equality comparisons | | Weight lower bounds | `O(R + W)` / `O(W)` bits | | Constructive best fit | `O(T log T + P R + P W)` / `O(P W)` bits plus `O(T + P)`; bounded retries only on failure | -| Joint initialization DP | `O(G² Q Kpack + S G² Q²)` time / `O(G² Q + S G Q)` storage; `Kpack` is one ordered-pack cost query | +| Joint initialization DP | `O(G² Q Kpack + S G² Q²)` time / `O(G² Q + S G Q)` storage; `Kpack` is one constructive pack-cost query | +| Feasibility completion, only if the quality grid fails | `O(G² P Kpack + S G²)` time / `O(G² + S G)` additional storage | | Neighborhood memory service | Expected `O(T + F)` per partition / `O(T + F)` scratch | | Topology clustering / cached PEFT distances | `O(P²)` time | | Concat cost extraction | `O(A D)` for `A` inputs of rank `D`; constant auxiliary space | diff --git a/src/PIM/Compiler/PimPeriodicAnalysis.cpp b/src/PIM/Compiler/PimPeriodicAnalysis.cpp index a661fc0..44ba1bf 100644 --- a/src/PIM/Compiler/PimPeriodicAnalysis.cpp +++ b/src/PIM/Compiler/PimPeriodicAnalysis.cpp @@ -369,7 +369,7 @@ public: std::vector> memoryServices; std::array opcodeService{}, opcodeCounts{}; std::string error; - size_t abstractBudget = 1000000; + size_t abstractBudget = 2000000; bool readCore(int64_t core, llvm::StringRef data, llvm::ArrayRef groups); bool readExecution(int64_t core, const std::vector &program); @@ -845,13 +845,17 @@ bool Analysis::connect() { EvaluatedAnalysis Analysis::evaluate() { EvaluatedAnalysis result; + double dependencySeconds = 0; if (cores.empty()) result.error = "artifact has no active repeating instruction stream"; else if (!connect()) result.error = error; if (!result.error.empty()) { result.cost.reason = result.error; return result; } if (options.detailed || options.searchGuidance) { + auto start = std::chrono::steady_clock::now(); auto recurrence = analyzePeriodicDependencies(events.size(), edges, result.error); if (mlir::failed(recurrence)) { result.cost.reason = result.error; return result; } result.recurrence = std::move(*recurrence); + dependencySeconds = std::chrono::duration( + std::chrono::steady_clock::now() - start).count(); } if (!options.detailed || memoryQueue) { auto start = std::chrono::steady_clock::now(); @@ -885,6 +889,7 @@ EvaluatedAnalysis Analysis::evaluate() { += static_cast(edge.delayNs) / distance; } } + result.cost.timing.dependencySeconds = dependencySeconds; return result; } @@ -894,6 +899,8 @@ llvm::json::Object Analysis::report(const EvaluatedAnalysis &evaluation) const { result["status"] = "unavailable"; result["reason"] = evaluation.error; return result; } result["global_memory_service_ns"] = static_cast(memory.serviceNs); + if (options.measureTime) + result["dependency_analysis_ms"] = evaluation.cost.timing.dependencySeconds * 1000; result["global_read_bytes"] = static_cast(memory.readBytes); result["global_write_bytes"] = static_cast(memory.writeBytes); result["global_read_requests"] = static_cast(memory.readRequests); diff --git a/src/PIM/Compiler/PimPipelineSelection.cpp b/src/PIM/Compiler/PimPipelineSelection.cpp index 49985cd..2632943 100644 --- a/src/PIM/Compiler/PimPipelineSelection.cpp +++ b/src/PIM/Compiler/PimPipelineSelection.cpp @@ -113,7 +113,7 @@ struct PimPipelineSelectionPass final : PassWrapper &costs, llvm::json::Object &report) { llvm::json::Array summaries; - double realization = 0, extraction = 0, expansion = 0, queue = 0, evaluation = 0; + double realization = 0, extraction = 0, expansion = 0, queue = 0, evaluation = 0, dependency = 0; auto evaluate = [&](const MergeScheduleResult &schedule) { size_t index = summaries.size(); llvm::json::Object summary{{"index", static_cast(index)}}; @@ -131,6 +131,7 @@ struct PimPipelineSelectionPass final : PassWrapper(search->rounds); report["packing_attempts"] = static_cast(search->packingAttempts); + report["neighborhood_seconds"] = search->neighborhoodSeconds; + report["constructive_packing_seconds"] = search->packingSeconds; report["considered_assignments"] = static_cast(search->consideredAssignments); report["search_termination"] = search->termination; report["queue_evaluation_budget"] = static_cast(kPipelineQueueEvaluationBudget); diff --git a/src/PIM/Compiler/PimSpatialCost.cpp b/src/PIM/Compiler/PimSpatialCost.cpp index 1b6073f..60fb7ec 100644 --- a/src/PIM/Compiler/PimSpatialCost.cpp +++ b/src/PIM/Compiler/PimSpatialCost.cpp @@ -58,6 +58,10 @@ public: program(target.processorCount) {} bool build(ModuleOp module); private: + // Bound interpretation per physical core, independently of batch lane count. + static constexpr size_t maximumCoreVisits = 1000000; + size_t coreVisits = 0; + int64_t currentCore = -1; llvm::SmallDenseSet assignedCores; DenseMap residence; DenseMap publicationAccess; @@ -378,7 +382,12 @@ bool Extraction::operation(Operation &op, Sequence &sequence) { bool Extraction::block(Block &body, Sequence &sequence) { for (Operation &op : body) { - if (++visited > 1000000) { error = "abstract extraction budget exceeded"; return false; } + ++visited; + if (++coreVisits > maximumCoreVisits) { + error = "abstract extraction budget exceeded on physical core " + + std::to_string(currentCore); + return false; + } if (!operation(op, sequence)) return false; for (Value result : op.getResults()) if (auto outputs = directOutputs.find(result); outputs != directOutputs.end()) @@ -400,6 +409,7 @@ bool Extraction::core(Operation *op, int64_t physicalCore, unsigned lane) { loweringSupport = StaticLoweringSupport::Unsupported; error = "duplicate physical core assignment: " + std::to_string(physicalCore); return false; } + coreVisits = 0; currentCore = physicalCore; residence.clear(); publicationAccess.clear(); directOutputs.clear(); compactStorage.clear(); environment.bindings.clear(); ValueRange inputs, weights; diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.cpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.cpp index a583c03..3c1f37b 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.cpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.cpp @@ -57,7 +57,29 @@ mlir::FailureOr analyzePeriodicDependencies( const size_t absent = std::numeric_limits::max(); std::vector potential(nodeCount); std::vector predecessor(nodeCount, absent); - auto infeasible = [&](double interval) { + std::vector cycleSeen(nodeCount); + auto hasPositivePredecessorCycle = [&](double interval) { + std::fill(cycleSeen.begin(), cycleSeen.end(), absent); + for (size_t start = 0; start < nodeCount; ++start) { + size_t node = start; + while (cycleSeen[node] == absent && predecessor[node] != absent) { + cycleSeen[node] = start; + node = edges[predecessor[node]].source; + } + if (cycleSeen[node] != start || predecessor[node] == absent) continue; + const size_t first = node; + Cost delay = 0, distance = 0; + do { + const auto &edge = edges[predecessor[node]]; + delay = checkedAdd(delay, edge.delayNs); + distance = checkedAdd(distance, static_cast(edge.distance)); + node = edge.source; + } while (node != first); + if (static_cast(delay) - interval * distance > 1e-7) return true; + } + return false; + }; + auto infeasible = [&](double interval, bool earlyCycle = true) { std::fill(potential.begin(), potential.end(), 0); std::fill(predecessor.begin(), predecessor.end(), absent); auto relax = [&](size_t index) { @@ -85,6 +107,9 @@ mlir::FailureOr analyzePeriodicDependencies( changed |= relax(index); if (!changed) return false; + // A positive predecessor cycle already proves infeasibility. Keep the + // final witness pass unchanged so search guidance remains deterministic. + if (earlyCycle && hasPositivePredecessorCycle(interval)) return true; } return true; }; @@ -104,7 +129,7 @@ mlir::FailureOr analyzePeriodicDependencies( error = "periodic timing exceeds one-nanosecond floating-point precision"; return mlir::failure(); } - if (!infeasible(result.recurrenceLowerNs)) + if (!infeasible(result.recurrenceLowerNs, false)) return result; // Recover witness cycles from the last longest-path predecessor forest. diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp index 7d1d5e1..7f3c60a 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -71,6 +72,7 @@ struct PipelineGroup { struct PipelineStageAssignment { std::vector taskStages; std::vector stageSizes; + std::vector constructivePacks; }; static bool consumesPipelineInput(const ComputeGraphNode &node) { @@ -229,7 +231,7 @@ static FailureOr initialPipelinePartition( std::string &error) { size_t count = constraints.groups.size(), stages = layout.getStageCount(); if (!count) return PipelineStageAssignment{{}, - {layout.getStageSizes().begin(), layout.getStageSizes().end()}}; + {layout.getStageSizes().begin(), layout.getStageSizes().end()}, {}}; size_t extra = layout.getProcessorCount() - stages; // Resource quantization bounds seed construction only. Refinement uses // individual physical cores and final allocation uses every available core. @@ -259,9 +261,9 @@ static FailureOr initialPipelinePartition( for (size_t amount = 0; amount <= units; ++amount) { size_t cores = coreCount(amount); if (weights.count() > checkedMultiply(cores, residentWeightCapacity)) continue; - costs[start][end][amount] = checkedAdd(span - largest, - estimateOrderedPackCost(model.schedulingCosts, model.weights.taskWeightIds, - model.weights.weightCount, tasks, residentWeightCapacity, cores)); + if (auto packing = model.weights.estimatePackCost(tasks, + model.schedulingCosts, residentWeightCapacity, cores)) + costs[start][end][amount] = checkedAdd(span - largest, *packing); } } } @@ -290,8 +292,53 @@ static FailureOr initialPipelinePartition( && (!selected || best[stage][count][units].priority < best[selected][count][units].priority)) selected = stage; + // Quantization controls quality, not feasibility. Complete the same prefix + // DP with minimum constructive core counts if no quantized state survives. + std::vector> feasible; + if (!selected) { + feasible.resize(stages + 1, std::vector(count + 1)); + feasible[0][0].valid = true; + std::vector> minimum(count + 1, + std::vector(count + 1, std::numeric_limits::max())); + for (size_t start = 0; start < count; ++start) { + TaskList tasks; + for (size_t end = start + 1; end <= count; ++end) { + llvm::append_range(tasks, constraints.groups[constraints.groupOrder[end - 1]].tasks); + const size_t lower = std::max(1, + model.weights.getPackCountLowerBound(tasks, residentWeightCapacity)); + const size_t maximum = layout.getProcessorCount() - stages + 1; + for (size_t cores = lower; cores <= maximum; ++cores) + if (model.weights.estimatePackCost(tasks, model.schedulingCosts, + residentWeightCapacity, cores)) { + minimum[start][end] = cores; + break; + } + } + } + for (size_t stage = 0; stage < stages; ++stage) + for (size_t start = 0; start < count; ++start) { + if (!feasible[stage][start].valid) continue; + for (size_t end = start + 1; end <= count; ++end) { + if (stage == 0 && end < constraints.minimumStageZeroEnd) continue; + if (stage && constraints.hasCrossingEdge + && constraints.furthestSuccessorBefore[start] >= end) continue; + const size_t cores = minimum[start][end]; + if (cores > layout.getProcessorCount()) continue; + Cost used = checkedAdd(feasible[stage][start].priority, static_cast(cores)); + if (used > layout.getProcessorCount() - (stages - stage - 1)) continue; + auto ¤t = feasible[stage + 1][end]; + if (!current.valid || used < current.priority) + current = {used, start, cores, true}; + } + } + for (size_t stage = 1; stage <= stages; ++stage) + if (feasible[stage][count].valid + && (!selected || feasible[stage][count].priority < feasible[selected][count].priority)) + selected = stage; + } if (!selected && count) { - error = "pipeline initialization found no dependency-adjacent operation partition within its bounded core-budget search"; + error = "pipeline initialization found no constructively packable dependency-adjacent partition; " + "bounded packing failure does not prove physical infeasibility"; return failure(); } PipelineStageAssignment result; @@ -299,13 +346,31 @@ static FailureOr initialPipelinePartition( result.taskStages.resize(constraints.taskToGroup.size()); size_t end = count, remaining = units; for (size_t stage = selected; stage; --stage) { - const auto &cell = best[stage][end][remaining]; - result.stageSizes[stage - 1] = coreCount(cell.units); + const auto &cell = feasible.empty() ? best[stage][end][remaining] : feasible[stage][end]; + result.stageSizes[stage - 1] = feasible.empty() ? coreCount(cell.units) : cell.units; for (size_t position = cell.start; position < end; ++position) for (size_t task : constraints.groups[constraints.groupOrder[position]].tasks) result.taskStages[task] = stage - 1; end = cell.start; - remaining -= cell.units; + if (feasible.empty()) remaining -= cell.units; + } + // Preserve the constructive witness before adding spare cores. Reordering + // tasks for topology-aware placement must not lose DP feasibility evidence. + result.constructivePacks.resize(model.schedulingCosts.size()); + std::vector stageTasks(selected); + for (size_t group : constraints.groupOrder) + for (size_t task : constraints.groups[group].tasks) + stageTasks[result.taskStages[task]].push_back(task); + for (size_t stage = 0; stage < selected; ++stage) { + const auto &tasks = stageTasks[stage]; + TaskList packs; + if (!model.weights.estimatePackCost(tasks, model.schedulingCosts, + residentWeightCapacity, result.stageSizes[stage], &packs)) { + error = "pipeline initializer could not reconstruct its constructive packing witness"; + return failure(); + } + for (auto [index, task] : llvm::enumerate(tasks)) + result.constructivePacks[task] = packs[index]; } size_t assigned = std::accumulate(result.stageSizes.begin(), result.stageSizes.end(), size_t{0}); for (; assigned < layout.getProcessorCount(); ++assigned) { @@ -431,7 +496,7 @@ static LogicalResult packPipelineStages( std::vector &tasksByCpu, const PipelineCoreLayout &layout, size_t pipelineStages, size_t residentWeightCapacity, const SchedulingTarget &target, size_t &failedStage, - std::string &error) { + std::string &error, ArrayRef constructivePacks) { std::vector indegree(graph.nodes.size()); std::vector successors(graph.nodes.size()); for (size_t task = 0; task < graph.nodes.size(); ++task) @@ -493,13 +558,11 @@ static LogicalResult packPipelineStages( llvm::sort(tasks, [&](size_t lhs, size_t rhs) { return topologicalPosition[lhs] < topologicalPosition[rhs]; }); - estimateOrderedPackCost(model.schedulingCosts, model.weights.taskWeightIds, - model.weights.weightCount, tasks, residentWeightCapacity, range.size, &taskPacks); - if (taskPacks.size() != tasks.size()) { - auto assignment = model.weights.pack( - tasks, model.schedulingCosts, residentWeightCapacity, range.size); - if (assignment) - taskPacks = std::move(*assignment); + model.weights.estimatePackCost(tasks, model.schedulingCosts, + residentWeightCapacity, range.size, &taskPacks); + if (taskPacks.size() != tasks.size() && !constructivePacks.empty()) { + taskPacks.clear(); + for (size_t task : tasks) taskPacks.push_back(constructivePacks[task]); } packed = taskPacks.size() == tasks.size() && packPipelineStage( graph, model.schedulingCosts, tasksByCpu, @@ -508,9 +571,9 @@ static LogicalResult packPipelineStages( } if (!packed) { failedStage = stage; - error = "pipeline resident-weight heuristic search could not pack stage " + error = "pipeline constructive physical-placement search could not pack stage " + std::to_string(stage) - + " within its physical core budget"; + + " within its physical core budget; bounded search failure does not prove physical infeasibility"; return failure(); } } @@ -633,7 +696,7 @@ static LogicalResult packPipelineAssignment(const ComputeGraph &graph, if (succeeded(packPipelineStages( graph, taskModel, tasksByPhysicalCpu, candidateLayout, pipelineStages, physicalTarget.residentWeightCapacity, - physicalTarget, failedStage, packingError))) { + physicalTarget, failedStage, packingError, assignment.constructivePacks))) { packed = true; break; } @@ -651,6 +714,7 @@ static LogicalResult packPipelineAssignment(const ComputeGraph &graph, donor = stage; if (!donor) break; + assignment.constructivePacks.clear(); --assignment.stageSizes[*donor]; ++assignment.stageSizes[failedStage]; } @@ -748,24 +812,10 @@ static std::optional allocatePipelineStageCores( largest = std::max(largest, model.schedulingCosts[task]); } for (size_t cores = minimum; cores <= target.processorCount; ++cores) { - TaskList packs; Cost packedCost = std::max(largest, work[stage] / cores + (work[stage] % cores != 0)); if (constructPacks) - packedCost = estimateOrderedPackCost(model.schedulingCosts, - model.weights.taskWeightIds, model.weights.weightCount, tasks[stage], - target.residentWeightCapacity, cores, &packs); - if (constructPacks && packs.size() != tasks[stage].size()) { - auto constructive = model.weights.pack(tasks[stage], model.schedulingCosts, - target.residentWeightCapacity, cores); - packedCost = std::numeric_limits::max(); - if (constructive) { - TaskCosts loads(cores); - for (size_t i = 0; i < tasks[stage].size(); ++i) - loads[(*constructive)[i]] = checkedAdd(loads[(*constructive)[i]], - model.schedulingCosts[tasks[stage][i]]); - packedCost = *llvm::max_element(loads); - } - } + packedCost = model.weights.estimatePackCost(tasks[stage], model.schedulingCosts, + target.residentWeightCapacity, cores).value_or(std::numeric_limits::max()); // A feasible assignment remains feasible when extra cores stay idle. if (cores > minimum) packedCost = std::min(packedCost, costs[stage][cores - 1]); costs[stage][cores] = packedCost; @@ -1060,9 +1110,8 @@ static std::vector pipelineNeighborhood(const ComputeGraph &gr auto packingCost = [&](const TaskList &tasks, size_t cores, size_t stage) { auto *cached = tasks == originalView.tasks[stage] ? &packCosts[stage][cores] : nullptr; if (cached && *cached) return **cached; - Cost value = estimateOrderedPackCost(model.schedulingCosts, - model.weights.taskWeightIds, model.weights.weightCount, tasks, - target.residentWeightCapacity, cores); + Cost value = model.weights.estimatePackCost(tasks, model.schedulingCosts, + target.residentWeightCapacity, cores).value_or(std::numeric_limits::max()); if (cached) *cached = value; return value; }; @@ -1093,6 +1142,7 @@ static std::vector pipelineNeighborhood(const ComputeGraph &gr if (model.weights.getPackCountLowerBound(view.tasks[stage], target.residentWeightCapacity) > assignment.stageSizes[stage]) return; Cost work = packingCost(view.tasks[stage], assignment.stageSizes[stage], stage); + if (work == std::numeric_limits::max()) return; maximum = std::max(maximum, work); // A within-generation span is latency, not an initiation-interval bound. total = checkedAdd(total, view.spans[stage]); @@ -1174,7 +1224,10 @@ mlir::FailureOr optimizePipelineSchedule( std::set visited; std::set placements{placementIdentity(graph, initial)}; std::vector beam{0}; + auto neighborhoodStart = std::chrono::steady_clock::now(); auto proposals = pipelineNeighborhood(graph, executionModel, *groups, result, beam, visited, 2 * evaluationBudget); + result.neighborhoodSeconds = std::chrono::duration( + std::chrono::steady_clock::now() - neighborhoodStart).count(); if (proposals.empty() || evaluationBudget == 1) { result.termination = "one neighborhood schedule; queue comparison skipped"; return result; @@ -1197,7 +1250,10 @@ mlir::FailureOr optimizePipelineSchedule( if (!error.empty()) return failure(); constexpr size_t maximumPackingAttempts = 4 * kPipelineQueueEvaluationBudget; while (result.costs.size() < evaluationBudget && result.packingAttempts < maximumPackingAttempts) { + neighborhoodStart = std::chrono::steady_clock::now(); proposals = pipelineNeighborhood(graph, executionModel, *groups, result, beam, visited, 2 * evaluationBudget); + result.neighborhoodSeconds += std::chrono::duration( + std::chrono::steady_clock::now() - neighborhoodStart).count(); if (proposals.empty()) break; ++result.rounds; size_t evaluated = 0; @@ -1211,17 +1267,25 @@ mlir::FailureOr optimizePipelineSchedule( ++result.packingAttempts; visited.emplace(proposal.parent, proposal.assignment.taskStages, proposal.assignment.stageSizes, proposal.optimizeCores, proposal.adjustableStages); + auto packingStart = std::chrono::steady_clock::now(); auto assignment = proposal.assignment; if (proposal.optimizeCores) { auto allocated = allocatePipelineStageCores(graph, executionModel, assignment.taskStages, pipelineStages, target, assignment.stageSizes, proposal.adjustableStages); - if (!allocated) continue; + if (!allocated) { + result.packingSeconds += std::chrono::duration( + std::chrono::steady_clock::now() - packingStart).count(); + continue; + } assignment = std::move(*allocated); } MergeScheduleResult schedule; std::string packingError; - if (failed(packPipelineAssignment(graph, model, assignment, schedule, - pipelineStages, target, packingError))) continue; + auto packed = packPipelineAssignment(graph, model, assignment, schedule, + pipelineStages, target, packingError); + result.packingSeconds += std::chrono::duration( + std::chrono::steady_clock::now() - packingStart).count(); + if (failed(packed)) continue; if (!placements.insert(placementIdentity(graph, schedule)).second) continue; auto estimate = cost(schedule); if (!error.empty()) return failure(); @@ -1261,7 +1325,7 @@ mlir::LogicalResult packPipelineSchedule(const ComputeGraph &graph, } PipelineTaskModel model(graph, target); return packPipelineAssignment(graph, model, - {{taskStages.begin(), taskStages.end()}, {stageSizes.begin(), stageSizes.end()}}, + {{taskStages.begin(), taskStages.end()}, {stageSizes.begin(), stageSizes.end()}, {}}, schedule, stageSizes.size(), target, error); } diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp index 9948b06..cc4dcbe 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.hpp @@ -114,6 +114,7 @@ struct PipelineSearchResult { std::vector costs; std::vector parents; size_t rounds = 0, packingAttempts = 0, consideredAssignments = 0; + double neighborhoodSeconds = 0, packingSeconds = 0; std::string termination; }; diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp index d4578e9..d5b743b 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ResidentWeightPacking.hpp @@ -11,15 +11,18 @@ namespace onnx_mlir::spatial { // Greedy maximal prefixes minimize ordered pack count: both constraints are -// closed under taking subranges. The optimum cost cannot increase with packs. -inline Cost estimateOrderedPackCost( +// closed under taking subranges. Feasible optimum costs cannot increase with packs. +inline std::optional estimateOrderedPackCost( llvm::ArrayRef taskCosts, const std::vector> &taskWeightIds, size_t weightCount, llvm::ArrayRef tasks, size_t residentWeightCapacity, size_t maximumPacks, std::vector *taskPacks = nullptr) { + if (taskPacks) taskPacks->clear(); + if (!maximumPacks) return std::nullopt; Cost low = 0; Cost high = 0; for (size_t task : tasks) { + if (taskWeightIds[task].size() > residentWeightCapacity) return std::nullopt; low = std::max(low, taskCosts[task]); high = checkedAdd(high, taskCosts[task]); } @@ -65,6 +68,7 @@ inline Cost estimateOrderedPackCost( } return fits; }; + if (!pack(high, nullptr)) return std::nullopt; while (low < high) { Cost middle = low + (high - low) / 2; if (pack(middle, nullptr)) @@ -73,9 +77,7 @@ inline Cost estimateOrderedPackCost( low = middle + 1; } if (taskPacks) { - taskPacks->clear(); - if (!pack(low, taskPacks)) - taskPacks->clear(); + pack(low, taskPacks); } return low; } @@ -113,6 +115,22 @@ struct IndexedResidentWeights { return std::max(!tasks.empty(), count / capacity + (count % capacity != 0)); } + std::optional estimatePackCost(llvm::ArrayRef tasks, + llvm::ArrayRef costs, size_t capacity, size_t packCount, + std::vector *taskPacks = nullptr) const { + if (taskPacks) taskPacks->clear(); + if (!packCount) return std::nullopt; + if (auto ordered = estimateOrderedPackCost(costs, taskWeightIds, + weightCount, tasks, capacity, packCount, taskPacks)) return ordered; + auto assignment = pack(tasks, costs, capacity, packCount); + if (!assignment) return std::nullopt; + std::vector loads(packCount); + for (auto [index, task] : llvm::enumerate(tasks)) + loads[(*assignment)[index]] = checkedAdd(loads[(*assignment)[index]], costs[task]); + if (taskPacks) *taskPacks = std::move(*assignment); + return *llvm::max_element(loads); + } + // A constructive reuse-first best fit; failure is not a proof of infeasibility. std::optional> pack( llvm::ArrayRef tasks, llvm::ArrayRef costs, diff --git a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp index ac775ec..35e2775 100644 --- a/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp +++ b/src/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp @@ -67,6 +67,7 @@ struct WholeScheduleCost { std::vector processorTaskWorkNs, processorCriticalDelayNs; struct AnalysisTiming { double extractionSeconds = 0, expansionSeconds = 0, evaluationSeconds = 0; + double dependencySeconds = 0; } timing; std::optional convergenceProved() const; std::optional loweringSupported() const; diff --git a/test/PIM/CMakeLists.txt b/test/PIM/CMakeLists.txt index 855aa1c..5a12c1e 100644 --- a/test/PIM/CMakeLists.txt +++ b/test/PIM/CMakeLists.txt @@ -69,6 +69,7 @@ add_pim_unittest(SpatialSchedulingTargetTest add_pim_unittest(PeriodicDependenciesTest PeriodicDependenciesTest.cpp + PeriodicDependenciesReference.cpp LINK_LIBS PRIVATE OMPimCompilerUtils diff --git a/test/PIM/PeriodicDependenciesReference.cpp b/test/PIM/PeriodicDependenciesReference.cpp new file mode 100644 index 0000000..67214e1 --- /dev/null +++ b/test/PIM/PeriodicDependenciesReference.cpp @@ -0,0 +1,142 @@ +// Test-only reference from commit 3c7679594beb29ba9497aaf3e341cee74fa0be50. +#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.hpp" + +#include + +namespace onnx_mlir::spatial { + +mlir::FailureOr analyzeFrozenPeriodicDependencies( + size_t nodeCount, llvm::ArrayRef edges, + std::string &error) { + std::vector> outgoing(nodeCount); + std::vector indegree(nodeCount), recurrent; + Cost upper = 0; + for (auto [index, edge] : llvm::enumerate(edges)) { + if (edge.source >= nodeCount || edge.target >= nodeCount) { + error = "periodic dependency references an invalid event"; + return mlir::failure(); + } + if (edge.delayNs > std::numeric_limits::max() - upper) { + error = "periodic delay sum exceeds the timing range"; + return mlir::failure(); + } + upper += edge.delayNs; + if (edge.distance) { + recurrent.push_back(index); + } else { + outgoing[edge.source].push_back(index); + ++indegree[edge.target]; + } + } + std::vector order; + for (size_t node = 0; node < nodeCount; ++node) + if (!indegree[node]) + order.push_back(node); + for (size_t cursor = 0; cursor < order.size(); ++cursor) + for (size_t index : outgoing[order[cursor]]) + if (!--indegree[edges[index].target]) + order.push_back(edges[index].target); + if (order.size() != nodeCount) { + error = "periodic distance-zero dependencies contain a cycle"; + return mlir::failure(); + } + + PeriodicDependencyEstimate result; + std::vector span(nodeCount); + for (size_t node : order) { + result.withinGenerationSpanNs = + std::max(result.withinGenerationSpanNs, span[node]); + for (size_t index : outgoing[node]) { + const auto &edge = edges[index]; + span[edge.target] = std::max(span[edge.target], + checkedAdd(span[node], edge.delayNs)); + } + } + if (recurrent.empty() || !upper) + return result; + + const size_t absent = std::numeric_limits::max(); + std::vector potential(nodeCount); + std::vector predecessor(nodeCount, absent); + auto infeasible = [&](double interval) { + std::fill(potential.begin(), potential.end(), 0); + std::fill(predecessor.begin(), predecessor.end(), absent); + auto relax = [&](size_t index) { + const auto &edge = edges[index]; + double value = potential[edge.source] + edge.delayNs + - interval * edge.distance; + if (value <= potential[edge.target] + 1e-7) + return false; + potential[edge.target] = value; + predecessor[edge.target] = index; + return true; + }; + for (size_t node : order) + for (size_t index : outgoing[node]) + relax(index); + // Eliminate distance-zero paths by a DAG sweep. A simple residual path + // visits at most R recurrent edges, so R+1 rounds detect a positive cycle. + for (size_t round = 0; round <= recurrent.size(); ++round) { + ++result.relaxationRounds; + bool changed = false; + for (size_t index : recurrent) + changed |= relax(index); + for (size_t node : order) + for (size_t index : outgoing[node]) + changed |= relax(index); + if (!changed) + return false; + } + return true; + }; + result.recurrenceUpperNs = upper; + for (unsigned iteration = 0; + iteration < 64 && result.recurrenceUpperNs - result.recurrenceLowerNs > 1; + ++iteration) { + double middle = (result.recurrenceLowerNs + result.recurrenceUpperNs) / 2; + if (middle == result.recurrenceLowerNs || middle == result.recurrenceUpperNs) + break; + if (infeasible(middle)) + result.recurrenceLowerNs = middle; + else + result.recurrenceUpperNs = middle; + } + if (result.recurrenceUpperNs - result.recurrenceLowerNs > 1) { + error = "periodic timing exceeds one-nanosecond floating-point precision"; + return mlir::failure(); + } + if (!infeasible(result.recurrenceLowerNs)) + return result; + + // Recover witness cycles from the last longest-path predecessor forest. + std::vector seen(nodeCount, absent); + double best = 0; + for (size_t start = 0; start < nodeCount; ++start) { + size_t node = start; + while (seen[node] == absent && predecessor[node] != absent) { + seen[node] = start; + node = edges[predecessor[node]].source; + } + if (seen[node] != start || predecessor[node] == absent) + continue; + size_t first = node; + Cost delay = 0, distance = 0; + std::vector cycle; + do { + size_t index = predecessor[node]; + cycle.push_back(index); + delay = checkedAdd(delay, edges[index].delayNs); + distance = checkedAdd(distance, static_cast(edges[index].distance)); + node = edges[index].source; + } while (node != first); + double ratio = distance ? static_cast(delay) / distance : 0; + if (ratio > best) { + best = ratio; + std::reverse(cycle.begin(), cycle.end()); + result.criticalCycle = std::move(cycle); + } + } + return result; +} + +} // namespace onnx_mlir::spatial diff --git a/test/PIM/PeriodicDependenciesTest.cpp b/test/PIM/PeriodicDependenciesTest.cpp index a6e07d2..4096b68 100644 --- a/test/PIM/PeriodicDependenciesTest.cpp +++ b/test/PIM/PeriodicDependenciesTest.cpp @@ -5,11 +5,96 @@ #error "Periodic dependency tests require assertions" #endif #include +#include +#include #include #include using namespace onnx_mlir::spatial; +namespace onnx_mlir::spatial { +mlir::FailureOr analyzeFrozenPeriodicDependencies( + size_t, llvm::ArrayRef, std::string &); +} + +static size_t differentialCases = 0, acceleratedRounds = 0, frozenRounds = 0; +static double acceleratedSeconds = 0, frozenSeconds = 0; + +static void differential(size_t count, const std::vector &edges) { + std::string actualError, referenceError; + auto start = std::chrono::steady_clock::now(); + auto actual = analyzePeriodicDependencies(count, edges, actualError); + acceleratedSeconds += std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + start = std::chrono::steady_clock::now(); + auto reference = analyzeFrozenPeriodicDependencies(count, edges, referenceError); + frozenSeconds += std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + ++differentialCases; + assert(mlir::succeeded(actual) == mlir::succeeded(reference)); + assert(actualError == referenceError); + if (mlir::failed(actual)) return; + acceleratedRounds += actual->relaxationRounds; + frozenRounds += reference->relaxationRounds; + assert(actual->recurrenceLowerNs == reference->recurrenceLowerNs); + assert(actual->recurrenceUpperNs == reference->recurrenceUpperNs); + assert(actual->withinGenerationSpanNs == reference->withinGenerationSpanNs); + assert(actual->criticalCycle == reference->criticalCycle); +} + +static void testFrozenDifferential() { + std::mt19937 random(171221); + for (size_t sample = 0; sample < 800; ++sample) { + const size_t count = sample < 600 ? 2 + random() % 31 : 64 + random() % 193; + std::vector edges; + for (size_t source = 0; source < count; ++source) { + const size_t component = source / 32; + for (size_t link = 0; link < 4; ++link) { + size_t target = random() % count; + if (sample % 3 == 0 && target / 32 != component) continue; + unsigned distance = target > source && link % 2 == 0 ? 0 : 1 + random() % 7; + edges.push_back({source, target, random() % 100001, distance}); + } + } + differential(count, edges); + } + for (size_t sample = 0; sample < 100; ++sample) { + std::vector edges; + for (size_t node = 0; node < 32; ++node) { + edges.push_back({node, node, (Cost{1} << 40) + random() % 10001, + static_cast(1024 + random() % 7)}); + edges.push_back({node, (node + 1) % 32, random() % 101, 1}); + } + differential(32, edges); + } + differential(2, {{0,1,1,0}, {1,0,1000000000000ULL, std::numeric_limits::max()}}); + for (size_t count : {2u, 64u, 1024u}) { + std::vector edges; + for (size_t node = 0; node < count; ++node) { + edges.push_back({node, node, 1000000 + node % 2, 3}); + if (node + 1 < count) edges.push_back({node, node + 1, 1, 0}); + } + differential(count, edges); + } + std::vector large; + constexpr size_t componentSize = 2048, components = 32; + for (size_t component = 0; component < components; ++component) { + size_t first = component * componentSize; + for (size_t node = first; node + 1 < first + componentSize; ++node) + large.push_back({node, node + 1, 1, 0}); + large.push_back({first + componentSize - 1, first, 17 + component, + static_cast(1 + component % 7)}); + } + differential(componentSize * components, large); + differential(0, {}); + differential(2, {{0, 1, 0, 0}, {1, 0, 0, 0}}); + differential(2, {{0, 1, 10, 0}, {1, 0, 3, 0}}); + differential(1, {{0, 1, 1, 1}}); + differential(1, {{0, 0, Cost{1} << 60, 1}}); + differential(1, {{0, 0, std::numeric_limits::max(), 1}, {0, 0, 1, 1}}); + std::cout << "frozen_differential_cases=" << differentialCases + << " accelerated_rounds=" << acceleratedRounds << " frozen_rounds=" << frozenRounds + << " accelerated_seconds=" << acceleratedSeconds << " frozen_seconds=" << frozenSeconds << '\n'; +} + static double enumerateCycles(size_t count, const std::vector &edges) { double maximum = 0; @@ -59,11 +144,22 @@ static void check(size_t count, const std::vector &edges, } int main() { + testFrozenDifferential(); check(3, {{0, 1, 12, 0}, {1, 2, 7, 0}}, 0); check(3, {{0, 1, 12, 1}, {1, 2, 7, 1}}, 0); check(1, {{0, 0, 30, 2}}, 15); // Independent core periods remain parallel. check(2, {{0, 0, 10, 1}, {1, 1, 20, 1}}, 20); + // Positive-cycle checks must not change the optimum or the final witness. + std::vector independent; + for (size_t core = 0; core < 128; ++core) + independent.push_back({core, core, 1000 + core, 1}); + check(128, independent, 1127); + std::string performanceError; + auto accelerated = analyzePeriodicDependencies(128, independent, performanceError); + assert(mlir::succeeded(accelerated)); + assert(accelerated->criticalCycle == std::vector{127}); + assert(accelerated->relaxationRounds < 512); // Equal serial demand, different cross-core blocking recurrences. check(4, {{0, 1, 10, 0}, {2, 3, 10, 0}, {1, 0, 0, 1}, {3, 2, 0, 1}}, 10); diff --git a/test/PIM/PimAbstractExecutionTest.cpp b/test/PIM/PimAbstractExecutionTest.cpp index cfdf3e5..182fa59 100644 --- a/test/PIM/PimAbstractExecutionTest.cpp +++ b/test/PIM/PimAbstractExecutionTest.cpp @@ -243,6 +243,17 @@ static void testSharedEngine(PimInstructionTiming target) { actual = analyzePimExecution(abstract, target); TEST_CHECK(actual.getInteger("global_read_requests") == 2); TEST_CHECK(actual.getInteger("global_memory_service_ns") == 160); + // The construction limit applies to interpreted patterns, not just traffic. + // These zero-trip request bodies do not allocate a large event graph. + abstract[0] = {{E::Work, 4}, {E::Repeat, 1500000, 0, 0, 0, + {{E::Repeat, 0, 0, 0, 0, {{E::Load, 1}}}}}}; + auto largerPattern = evaluatePimExecution(abstract, target); + TEST_CHECK(largerPattern.cost.available()); + TEST_CHECK(largerPattern.report.getInteger("global_read_requests") == 0); + abstract[0] = {{E::Repeat, 2000001, 0, 0, 0, {{E::Load, 1}}}}; + auto aboveBudget = evaluatePimExecution(abstract, target); + TEST_CHECK(!aboveBudget.cost.available()); + TEST_CHECK(aboveBudget.cost.reason == "abstract repetition budget exceeded"); abstract[0] = {{E::Repeat, 1000000000000ull, 0, 0, 0, {{E::Load, 1}}}}; actual = analyzePimExecution(abstract, target); TEST_CHECK(actual.getString("status") == "unavailable"); @@ -433,6 +444,48 @@ static void testDependencyTokens() { TEST_CHECK(drained.empty() && drained.remaining().empty()); } +static void testSpatialExtractionBudget(PimInstructionTiming target) { + MLIRContext context; + context.loadDialect(); + auto module = ModuleOp::create(UnknownLoc::get(&context)); + OpBuilder builder(module.getBodyRegion()); + auto loc = module.getLoc(); + auto function = func::FuncOp::create(builder, loc, "test", builder.getFunctionType({}, {})); + builder.setInsertionPointToStart(function.addEntryBlock()); + auto batch = spatial::SpatScheduledComputeBatch::create(builder, loc, TypeRange{}, + builder.getI32IntegerAttr(2), ValueRange{}, ValueRange{}); + batch->setAttr("coreIds", builder.getDenseI32ArrayAttr({0, 1})); + batch.getBody().push_back(new Block); + batch.getBody().front().addArgument(builder.getIndexType(), loc); + builder.setInsertionPointToStart(&batch.getBody().front()); + Value zero = arith::ConstantIndexOp::create(builder, loc, 0); + Value upper = arith::ConstantIndexOp::create(builder, loc, 100000); + Value one = arith::ConstantIndexOp::create(builder, loc, 1); + auto type = RankedTensorType::get({16}, builder.getF32Type()); + Value local = tensor::EmptyOp::create(builder, loc, type.getShape(), type.getElementType()); + spatial::SpatReluOp::create(builder, loc, type, local); + auto loop = scf::ForOp::create(builder, loc, zero, upper, one); + builder.setInsertionPointToStart(loop.getBody()); + for (int64_t i = 0; i < 5; ++i) arith::ConstantIndexOp::create(builder, loc, i); + // A zero-count wait is runtime-free, but requires indexed interpretation. + spatial::SpatWaitOp::create(builder, loc, zero, zero); + builder.setInsertionPointAfter(loop); + spatial::SpatYieldOp::create(builder, loc, ValueRange{}); + builder.setInsertionPointAfter(batch); + func::ReturnOp::create(builder, loc); + TEST_CHECK(succeeded(verify(module))); + auto supported = evaluatePimSpatialSchedule(module, target); + TEST_CHECK(supported.report.getInteger("visited_operations") > 1000000); + TEST_CHECK(supported.cost.available()); + builder.setInsertionPointToStart(loop.getBody()); + for (int64_t i = 0; i < 5; ++i) arith::ConstantIndexOp::create(builder, loc, i); + auto unsupported = evaluatePimSpatialSchedule(module, target); + TEST_CHECK(!unsupported.cost.available()); + TEST_CHECK(unsupported.cost.reason == "abstract extraction budget exceeded on physical core 0"); + module.erase(); +} + static void testReadOnlySpatial(PimInstructionTiming target, bool batched) { MLIRContext context; context.loadDialect #include #include @@ -14,6 +16,76 @@ using namespace onnx_mlir::spatial; using E = PimExecutionEvent; static bool sawJointMove = false, sawSuccessiveMoves = false; +static void testPackingFeasibility() { + ComputeGraph graph; + for (size_t task = 0; task < 4; ++task) { + ResidentWeight weight; + weight.opaqueLane = task % 2; + graph.nodes.push_back({{nullptr, static_cast(task), 1}, 10, {weight}, task}); + } + IndexedResidentWeights weights(graph); + const std::vector tasks{0, 1, 2, 3}; + const std::vector work{10, 10, 10, 10}; + std::vector assignment{99}; + TEST_CHECK(!estimateOrderedPackCost(work, weights.taskWeightIds, weights.weightCount, + tasks, 1, 2, &assignment)); + TEST_CHECK(assignment.empty()); + TEST_CHECK(!estimateOrderedPackCost(work, weights.taskWeightIds, weights.weightCount, + tasks, 1, 0)); + TEST_CHECK(weights.estimatePackCost(tasks, work, 1, 2) == 20); + auto placement = weights.pack(tasks, work, 1, 2); + TEST_CHECK(placement && (*placement)[0] == (*placement)[2] + && (*placement)[1] == (*placement)[3] && (*placement)[0] != (*placement)[1]); + TEST_CHECK(!weights.estimatePackCost(tasks, work, 1, 1)); + for (size_t cores = 4; cores <= 8; ++cores) + TEST_CHECK(estimateOrderedPackCost(work, weights.taskWeightIds, weights.weightCount, + tasks, 1, cores) == 10); + weights.taskWeightIds[0] = {0, 1}; + TEST_CHECK(!estimateOrderedPackCost(work, weights.taskWeightIds, weights.weightCount, + tasks, 1, 4)); + weights.taskWeightIds = {{0, 1}, {2}, {0, 1}, {2}}; + weights.weightCount = 3; + TEST_CHECK(!estimateOrderedPackCost(work, weights.taskWeightIds, weights.weightCount, + tasks, 2, 2)); + TEST_CHECK(weights.estimatePackCost(tasks, work, 2, 2) == 20); +} + +static void testQuantizedFeasibility(uint32_t first) { + mlir::MLIRContext context; + context.loadDialect(); + auto module = mlir::ModuleOp::create(mlir::UnknownLoc::get(&context)); + mlir::OpBuilder builder(module.getBodyRegion()); + ComputeGraph graph; + graph.predecessors.resize(40); graph.successors.resize(40); + for (uint32_t group = 0; group < 2; ++group) { + const uint32_t lanes = group ? 40 - first : first; + auto batch = spatial::SpatGraphComputeBatch::create(builder, module.getLoc(), + mlir::TypeRange{}, builder.getI32IntegerAttr(lanes), mlir::ValueRange{}, mlir::ValueRange{}); + for (uint32_t lane = 0; lane < lanes; ++lane) { + size_t task = graph.nodes.size(); + ResidentWeight weight; weight.opaqueLane = task; + graph.nodes.push_back({{batch, lane, 1}, 10, {weight}, task}); + if (group) { + graph.predecessors[task].push_back({0, {0, 0}}); + graph.successors[0].push_back({task, {0, 0}}); + } + } + } + SchedulingTarget target; + target.processorCount = 40; target.residentWeightCapacity = 1; + target.interProcessorLatencyNs.assign(1600, 0); + MergeScheduleResult schedule; + std::string error; + TEST_CHECK(mlir::succeeded(runPipelineScheduler(graph, schedule, 2, target, error))); + std::vector budgets(2); + for (size_t stage : schedule.processorStages) ++budgets[stage]; + TEST_CHECK(budgets == std::vector({first, 40 - first})); + for (size_t task = 0; task < graph.nodes.size(); ++task) + TEST_CHECK(schedule.processorStages[schedule.computeToCpuMap.lookup(graph.nodes[task].instance)] + == (task < first ? 0 : 1)); + module.erase(); +} + // Test-only workloads use stage-boundary host transfers and local direct links. // The same queue // engine and complete-state evidence rank reference and searched schedules. @@ -112,6 +184,176 @@ static PimInstructionTiming hardware(size_t profile) { return target; } +static void testInitializerPackingWitness() { + mlir::MLIRContext context; + context.loadDialect(); + auto module = mlir::ModuleOp::create(mlir::UnknownLoc::get(&context)); + mlir::OpBuilder builder(module.getBodyRegion()); + std::vector operations; + for (size_t i = 0; i < 3; ++i) operations.push_back( + spatial::SpatGraphComputeBatch::create(builder, module.getLoc(), + mlir::TypeRange{}, builder.getI32IntegerAttr(3), mlir::ValueRange{}, mlir::ValueRange{})); + const std::vector> resident{{2,1,5},{2,4},{0,2},{0},{4,6},{1},{2},{2},{1,4}}; + const std::vector costs{4,1,5,4,1,2,5,2,2}; + ComputeGraph graph; + graph.predecessors.resize(9); graph.successors.resize(9); + for (uint32_t task = 0; task < 9; ++task) { + ResidentWeightSet weights; + for (size_t id : resident[task]) { + ResidentWeight w; w.opaqueLane = id; weights.push_back(w); + } + graph.nodes.push_back({{operations[task % 3], task / 3, 1}, costs[task], weights, task}); + } + IndexedResidentWeights indexed(graph); + TEST_CHECK(!indexed.estimatePackCost({1,2,4,5,7,8}, costs, 3, 2)); + std::vector witness; + TEST_CHECK(indexed.estimatePackCost({1,4,7,2,5,8}, costs, 3, 2, &witness)); + TEST_CHECK(witness.size() == 6); + auto target = hardware(1); + target.processorCount = 4; target.residentWeightCapacity = 3; + target.interProcessorLatencyNs.assign(16, 1); + MergeScheduleResult schedule; + std::string error; + TEST_CHECK(mlir::succeeded(runPipelineScheduler(graph, schedule, 2, target, error))); + std::vector actual(4); + for (const auto &node : graph.nodes) { + size_t core = schedule.computeToCpuMap.lookup(node.instance); + TEST_CHECK(core < 4); + insertResidentWeights(actual[core], node.residentWeights); + TEST_CHECK(actual[core].size() <= 3); + } + // Exhaust all core assignments under the same stage map. The initialized + // assignment must belong to this physical-capacity reference set. + size_t feasible = 0; + for (size_t encoded = 0; encoded < (size_t{1} << 18); ++encoded) { + size_t choices = encoded; + std::vector weights(4); + std::vector groupStages(3, 2); + bool valid = true; + for (size_t task = 0; task < 9 && valid; ++task) { + size_t core = choices % 4; choices /= 4; + size_t stage = schedule.processorStages[core], group = task % 3; + if (groupStages[group] == 2) groupStages[group] = stage; + valid &= groupStages[group] == stage; + insertResidentWeights(weights[core], graph.nodes[task].residentWeights); + valid &= weights[core].size() <= 3; + } + feasible += valid; + } + TEST_CHECK(feasible > 0); + MergeScheduleResult alternative; + TEST_CHECK(mlir::succeeded(packPipelineSchedule(graph, std::vector(9), + {3, 1}, target, alternative, error))); + std::cout << "initializer_physical_feasible_assignments=" << feasible << '\n'; + module.erase(); +} + +static void testPhysicalReference() { + auto target = hardware(1); + target.processorCount = 3; + target.sharedMemoryRequestLatencyNs.resize(3); + target.sharedMemoryResponseLatencyNs.resize(3); + target.interProcessorLatencyNs = {0, 2, 11, 2, 0, 7, 11, 7, 0}; + ComputeGraph graph; + graph.predecessors.resize(3); graph.successors.resize(3); + for (uint32_t task = 0; task < 3; ++task) { + ResidentWeight weight; weight.opaqueLane = task % 2; + graph.nodes.push_back({{nullptr, task, 1}, task == 1 ? 700u : 80u, {weight}, task}); + if (task) { + graph.predecessors[task].push_back({task - 1, {64, 1}}); + graph.successors[task - 1].push_back({task, {64, 1}}); + } + } + double optimum = std::numeric_limits::infinity(); + size_t evaluations = 0; + for (size_t mask = 1; mask < 7; ++mask) + for (size_t placement = 0; placement < 27; ++placement) { + MergeScheduleResult schedule; + for (size_t core = 0; core < 3; ++core) schedule.processorStages.push_back((mask >> core) & 1); + size_t encoded = placement; + std::vector stages, slots(3); + std::vector resident(3); + bool valid = true; + for (const auto &node : graph.nodes) { + size_t core = encoded % 3; encoded /= 3; + schedule.computeToCpuMap[node.instance] = core; + schedule.computeToCpuSlotMap[node.instance] = slots[core]++; + stages.push_back(schedule.processorStages[core]); + insertResidentWeights(resident[core], node.residentWeights); + valid &= resident[core].size() <= target.residentWeightCapacity; + } + if (!valid || stages[0] != 0 || stages[1] < stages[0] || stages[2] < stages[1]) continue; + auto cost = queueCost(graph, schedule, target); + optimum = std::min(optimum, std::max(*cost.primary, *cost.alternative)); + ++evaluations; + } + std::string error; + MergeScheduleResult initial; + TEST_CHECK(mlir::succeeded(runPipelineScheduler(graph, initial, 2, target, error))); + auto evaluate = [&](const MergeScheduleResult &schedule) { return queueCost(graph, schedule, target); }; + auto search = optimizePipelineSchedule(graph, initial, 2, target, evaluate, error); + TEST_CHECK(mlir::succeeded(search)); + size_t winner = search->costs.empty() ? 0 : selectPipelineCandidate(search->costs, error); + auto selected = evaluate(search->schedules[winner]); + double value = std::max(*selected.primary, *selected.alternative); + TEST_CHECK(value + 1e-6 >= optimum); + std::cout << "physical_reference_ns=" << optimum << " selected_ns=" << value + << " gap_percent=" << 100 * (value / optimum - 1) + << " physical_evaluations=" << evaluations + << " search_evaluations=" << search->costs.size() + << " packing_attempts=" << search->packingAttempts << '\n'; +} + +static void testWorseningIntermediate() { + // A controlled cost oracle isolates the search's non-monotone landscape. + // Hardware queue objectives are covered separately by runReference(). + auto target = hardware(0); + target.processorCount = 8; + target.interProcessorLatencyNs.assign(64, 0); + ComputeGraph graph; + graph.predecessors.resize(4); graph.successors.resize(4); + for (uint32_t task = 0; task < 4; ++task) + graph.nodes.push_back({{nullptr, task, 1}, 10, {}, task}); + auto evaluate = [&](const MergeScheduleResult &schedule) { + bool firstStage = llvm::all_of(graph.nodes, [&](const auto &node) { + return schedule.processorStages[schedule.computeToCpuMap.lookup(node.instance)] == 0; + }); + size_t cores = llvm::count(schedule.processorStages, 0); + double value = !firstStage ? 120 : cores == 1 ? 80 : cores >= 6 ? 100 : 110; + WholeScheduleCost cost; + cost.primary = cost.alternative = value; + cost.loweringSupport = StaticLoweringSupport::Supported; + cost.selectionMargin = kPipelineSelectionMargin; + for (auto &proof : cost.recurrence) + proof = {ScheduleRecurrenceEvidence::Proved, 1, value, {}}; + return cost; + }; + std::string error; + MergeScheduleResult initial, intermediate; + TEST_CHECK(mlir::succeeded(runPipelineScheduler(graph, initial, 2, target, error))); + TEST_CHECK(evaluate(initial).primary == 100); + const std::vector stages(4); + TEST_CHECK(mlir::succeeded(packPipelineSchedule(graph, stages, {3, 5}, target, intermediate, error))); + TEST_CHECK(evaluate(intermediate).primary == 110); + double reference = std::numeric_limits::infinity(); + for (size_t cores = 1; cores < 8; ++cores) { + MergeScheduleResult schedule; + TEST_CHECK(mlir::succeeded(packPipelineSchedule(graph, stages, {cores, 8 - cores}, target, schedule, error))); + reference = std::min(reference, *evaluate(schedule).primary); + } + TEST_CHECK(reference == 80); + auto search = optimizePipelineSchedule(graph, initial, 2, target, evaluate, error); + TEST_CHECK(mlir::succeeded(search)); + size_t winner = selectPipelineCandidate(search->costs, error); + double selected = *search->costs[winner].primary; + TEST_CHECK(selected >= reference && selected <= 100); + TEST_CHECK(search->costs.size() <= kPipelineQueueEvaluationBudget); + std::cout << "non_monotone_oracle_reference=" << reference << " selected=" << selected + << " gap_percent=" << 100 * (selected / reference - 1) + << " evaluations=" << search->costs.size() + << " packing_attempts=" << search->packingAttempts << '\n'; +} + static void runReference(size_t profile, size_t fixture) { auto target = hardware(profile); ComputeGraph graph; @@ -217,8 +459,10 @@ static void runReference(size_t profile, size_t fixture) { << " gap_percent=" << 100 * (cost / reference - 1) << " best_evaluated_ns=" << bestEvaluated << " best_evaluated_gap_percent=" << 100 * (bestEvaluated / reference - 1) + << " reference_scope=logical_with_deterministic_packing" << " reference_evaluations=" << referenceEvaluations - << " search_evaluations=" << search->costs.size() << " refinement_depth=" << depth + << " search_evaluations=" << search->costs.size() + << " packing_attempts=" << search->packingAttempts << " refinement_depth=" << depth << " search_seconds=" << searchSeconds << " reference_seconds=" << referenceSeconds << " total_seconds=" << std::chrono::duration(std::chrono::steady_clock::now() - begin).count() << '\n'; @@ -246,6 +490,11 @@ static void runReference(size_t profile, size_t fixture) { } int main() { + testInitializerPackingWitness(); + testPackingFeasibility(); + for (uint32_t first : {19u, 21u}) testQuantizedFeasibility(first); + testPhysicalReference(); + testWorseningIntermediate(); for (size_t profile = 0; profile < 2; ++profile) for (size_t fixture = 0; fixture < 6; ++fixture) runReference(profile, fixture); TEST_CHECK(sawJointMove && sawSuccessiveMoves); diff --git a/test/PIM/SpatialSchedulingTargetTest.cpp b/test/PIM/SpatialSchedulingTargetTest.cpp index 9500b55..9423299 100644 --- a/test/PIM/SpatialSchedulingTargetTest.cpp +++ b/test/PIM/SpatialSchedulingTargetTest.cpp @@ -1642,14 +1642,14 @@ static void testOrderedPackCostCurve() { if (fits && packs <= cores) optimum = std::min(optimum, std::max(maximum, work)); } std::vector assignment; - Cost cost = estimateOrderedPackCost(costs, weights, 3, tasks, capacity, cores, &assignment); + auto cost = estimateOrderedPackCost(costs, weights, 3, tasks, capacity, cores, &assignment); if (optimum == std::numeric_limits::max()) { - assert(assignment.empty()); + assert(!cost && assignment.empty()); continue; } - assert(cost == optimum && cost <= previous && assignment.size() == tasks.size()); - plateau |= cost == previous; - previous = cost; + assert(cost && *cost == optimum && *cost <= previous && assignment.size() == tasks.size()); + plateau |= *cost == previous; + previous = *cost; std::vector loads(cores); std::vector resident(cores, llvm::SmallBitVector(3)); for (size_t task : tasks) { @@ -1658,7 +1658,7 @@ static void testOrderedPackCostCurve() { for (size_t weight : weights[task]) resident[assignment[task]].set(weight); } for (size_t core = 0; core < cores; ++core) { - assert(loads[core] <= cost && resident[core].count() <= capacity); + assert(loads[core] <= *cost && resident[core].count() <= capacity); } } assert(plateau && previous == 11); diff --git a/test/PIM/test_QueueSelection.py b/test/PIM/test_QueueSelection.py index ea12582..78e7a84 100644 --- a/test/PIM/test_QueueSelection.py +++ b/test/PIM/test_QueueSelection.py @@ -78,7 +78,7 @@ def main(): parser.add_argument("--compiler", required=True) args = parser.parse_args() compiler = str(Path(args.compiler).resolve()) - model = REPO / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx" + model = REPO / "validation/networks/pimcomp_models/googlenet/googlenet-12-no-lrn.onnx" with tempfile.TemporaryDirectory(prefix="raptor-static-selection-") as temporary: root = Path(temporary) history = root / "validation/tools/pim/pimcomp/compare/results_comparison.csv" diff --git a/validation/networks/pimcomp_models/README.md b/validation/networks/pimcomp_models/README.md index f90115f..766fb41 100644 --- a/validation/networks/pimcomp_models/README.md +++ b/validation/networks/pimcomp_models/README.md @@ -11,11 +11,11 @@ operations, standalone commands, configuration layout, and GA/OpenMP controls. | Directory | Model / input | Source | |---|---|---| -| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. | +| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../../resnet18/depth_68/resnet18_depth_68.onnx`. | | `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. | | `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12), with only its two LRN nodes bypassed. | | `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. | -| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`; this directory retains the historical Softmax-free variant. | +| `yolo11n/` | YOLO11n detection, `1x3x640x640` | [Ultralytics YOLO11n](https://docs.ultralytics.com/models/yolo11/); symlink to `../../yolo11n/depth_51/yolo11n_depth_51.onnx`. | The paper's checkpoints were not published. The three Model Zoo networks produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is @@ -29,15 +29,9 @@ Its terminal Softmax, output `prob_1`, weights, and other operations are retaine The native reference uses this same graph. VGG8 and both ResNet graphs have no removed operators; ResNet-34 fixes the symbolic batch size to one. -Historical performance models change the original computation: - -- `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal - Softmax, which Pimcomp does not schedule. -- `yolo11n/yolo11n-pimsim-nn.onnx` removes two internal Softmax nodes. It remains - available for reproducing historical measurements. - -The current YOLO comparison uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`, +The current YOLO comparison uses `yolo11n/yolo11n.onnx`, including both Softmax operations. Its native reference uses the same full model. +The checked-in ONNX graph uses opset 18. The original exporter version and checkpoint hash are not recorded. The functional simulator executes Softmax. Pimsim-nn skips `vsoftmax` before fetch, and Raptor's PIM queue estimator omits the same instruction cost. Timing and energy exclude Softmax work; they do not model hardware Softmax support. @@ -50,9 +44,8 @@ Model SHA-256 checksums: c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx c960c195953423388733cd0e60712cf425ca1f8917e264f5cf98e028752c61cf googlenet/googlenet-12-no-lrn.onnx -a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx -229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx +5e7bd85dc60b252674f8d7e1ad24f2317d00dfb8c524826536698462de04620d yolo11n/yolo11n.onnx ``` ## Hardware profiles diff --git a/validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx b/validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx deleted file mode 100644 index 887efed..0000000 Binary files a/validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx and /dev/null differ diff --git a/validation/networks/pimcomp_models/yolo11n/yolo11n-pimsim-nn.onnx b/validation/networks/pimcomp_models/yolo11n/yolo11n-pimsim-nn.onnx deleted file mode 100644 index a043ea6..0000000 Binary files a/validation/networks/pimcomp_models/yolo11n/yolo11n-pimsim-nn.onnx and /dev/null differ diff --git a/validation/networks/pimcomp_models/yolo11n/yolo11n.onnx b/validation/networks/pimcomp_models/yolo11n/yolo11n.onnx new file mode 120000 index 0000000..5bfedae --- /dev/null +++ b/validation/networks/pimcomp_models/yolo11n/yolo11n.onnx @@ -0,0 +1 @@ +../../yolo11n/depth_51/yolo11n_depth_51.onnx \ No newline at end of file diff --git a/validation/raptor_validation/pimcomp_models.py b/validation/raptor_validation/pimcomp_models.py index 709ce18..96cca55 100644 --- a/validation/raptor_validation/pimcomp_models.py +++ b/validation/raptor_validation/pimcomp_models.py @@ -11,7 +11,7 @@ MODELS = { "resnet18": SUITE / "resnet18/resnet18-v1-7.onnx", "resnet34": SUITE / "resnet34/resnet34-v1-7.onnx", "googlenet": SUITE / "googlenet/googlenet-12-no-lrn.onnx", - "yolo11n": REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx", + "yolo11n": SUITE / "yolo11n/yolo11n.onnx", } MODEL_NAMES = tuple(MODELS) DEFAULT_MODELS = MODEL_NAMES diff --git a/validation/tests/test_pim_experiments.py b/validation/tests/test_pim_experiments.py index 489455a..80c0731 100644 --- a/validation/tests/test_pim_experiments.py +++ b/validation/tests/test_pim_experiments.py @@ -511,7 +511,9 @@ def test_comparison_uses_same_model_for_reference_and_execution(tmp_path, monkey def test_yolo_uses_full_model_without_moving_comparison_artifacts(): - assert suite.MODELS["yolo11n"] == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx" + assert suite.MODELS["yolo11n"] == suite.SUITE / "yolo11n/yolo11n.onnx" + assert suite.MODELS["yolo11n"].is_symlink() + assert suite.MODELS["yolo11n"].resolve() == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx" assert suite.model_dir(None, "yolo11n") == suite.SUITE / "yolo11n" assert suite.common_dir(None, "yolo11n") == suite.SUITE / "yolo11n/artifacts/common-full-model" @@ -756,3 +758,43 @@ def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path): assert rows[0]["functional_validation"] == "PASS" assert rows[0]["baseline_functional_validation"] == "PASS" assert all(rows[0][f"{metric}_gain_percent"] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS) + + +def test_completed_no_sync_ablation_preserves_numerical_failure(): + report = {"pimsim_mode": "throughput", "raptor_extra_args": ["--pim-disable-synchronization"], + "raptor_validation": {"status": "done", "passed": False, "max_diffs": {"output": 1}}, + "raptor_performance": {"throughput": 100, "output_count": 8}, + "steps": [{"name": "Functional validation Raptor", "status": "passed"}], + "failures": []} + assert suite.completed_synchronization_ablation(report) + assert not suite.comparison_passed(report) + assert suite.functional_validation_status(report["raptor_validation"]) == "FAIL" + for field, value in (("pimsim_mode", "latency"), ("raptor_extra_args", []), + ("raptor_validation", {"status": "failed", "passed": False}), + ("raptor_validation", {"status": "done", "passed": False}), + ("raptor_performance", {"error": "SIGSEGV", "throughput": 100}), + ("raptor_performance", {"throughput": 100, "output_count": 0}), + ("steps", [{"status": "failed"}]), + ("failures", [{"stage": "Compile Raptor Pim", "error": "failed"}])): + assert not suite.completed_synchronization_ablation({**report, field: value}) + + +@pytest.mark.parametrize("process_failed", [False, True]) +@pytest.mark.parametrize("variant,expected", [("no-sync", 0), ("no-plan", 1), ("baseline", 1)]) +def test_ablation_runner_distinguishes_invalid_outputs_from_failed_execution(tmp_path, monkeypatch, variant, expected, process_failed): + report = {"pimsim_mode": "throughput", "raptor_extra_args": ["--pim-disable-synchronization"], + "raptor_validation": {"status": "done", "passed": False, "max_diffs": {"output": 1}}, + "raptor_performance": {"throughput": 100, "output_count": 8}, + "steps": [{"status": "passed"}], "failures": []} + output = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4, variant) / "pimcomp/comparison_report.json" + output.parent.mkdir(parents=True) + output.write_text(json.dumps(report)) + csv = tmp_path / "results.csv" + csv.write_text("summary\n") + monkeypatch.setattr(suite, "run", lambda *a, **k: 0) + monkeypatch.setattr(suite, "run_comparison_jobs", lambda *a: (["arch-a/vgg8/throughput/pipeline4"] if process_failed else [], 1)) + monkeypatch.setattr(suite, "write_results_csv", lambda *a: csv) + monkeypatch.setattr(sys, "argv", ["run_pimcomp_models.py", "--out-dir", str(tmp_path), + "--models", "vgg8", "--archs", "arch-a", "--mode", "throughput", "--pipeline", "4", + "--raptor-only", "--ablation-variant", variant, "--jobs", "1"]) + assert suite.main() == (1 if process_failed else expected) diff --git a/validation/tools/pim/ablation/README.md b/validation/tools/pim/ablation/README.md index 7172f99..0024d93 100644 --- a/validation/tools/pim/ablation/README.md +++ b/validation/tools/pim/ablation/README.md @@ -59,8 +59,10 @@ Both programs' functional statuses are recorded as `functional_validation` and `baseline_functional_validation`. The latter belongs to the disabled feature. Disabling synchronization removes host-buffer lifetime and pipeline restart -guarantees, so those variants may fail numerical validation and exit nonzero. -Baseline and `no-plan` must pass. Percentages involving failing programs remain +guarantees, so those variants may fail numerical validation. A completed +`no-sync` measurement retains `FAIL` in the CSV and is reported as diagnostic +in the runner summary. Compile, execution, missing-output, and measurement +failures still make the run fail. Baseline and `no-plan` must pass. Percentages involving failing programs remain diagnostic comparisons, not valid speedups. When fast-mode convergence fails, positive fallback throughput remains available. diff --git a/validation/tools/pim/ablation/results_ablation.csv b/validation/tools/pim/ablation/results_ablation.csv index 3909e1c..a761bb4 100644 --- a/validation/tools/pim/ablation/results_ablation.csv +++ b/validation/tools/pim/ablation/results_ablation.csv @@ -1,65 +1,25 @@ -variant,arch,model,mode,raptor_pipeline,functional_validation,latency,throughput,power,energy -baseline,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00% -no-plan,arch-a,googlenet,latency,1,PASS,+11513.29%,NA,-69.42%,+3452.69% -no-sync-no-plan,arch-a,googlenet,latency,1,PASS,+11512.99%,NA,-69.42%,+3452.59% -baseline,arch-a,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-a,googlenet,throughput,4,FAIL,+26.17%,-20.74%,+9.97%,+18.50% -no-plan,arch-a,googlenet,throughput,4,PASS,-100.00%,-100.00%,-62.73%,-100.00% -no-sync-no-plan,arch-a,googlenet,throughput,4,FAIL,+4094.63%,-97.62%,-59.86%,+1311.01% -baseline,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24% -no-sync-no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24% -baseline,arch-a,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-a,resnet18,throughput,4,FAIL,+110.65%,-52.55%,+43.25%,+177.48% -no-plan,arch-a,resnet18,throughput,4,PASS,+1598.93%,-94.12%,-51.04%,+664.87% -no-sync-no-plan,arch-a,resnet18,throughput,4,FAIL,+1174.21%,-92.16%,-45.06%,+543.74% -baseline,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99% -no-sync-no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99% -baseline,arch-a,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-a,resnet34,throughput,4,FAIL,+33.65%,-25.21%,+31.76%,+15.56% -no-plan,arch-a,resnet34,throughput,4,PASS,+1726.82%,-94.52%,-54.50%,+455.98% -no-sync-no-plan,arch-a,resnet34,throughput,4,FAIL,+813.41%,-89.04%,-49.77%,+206.84% -baseline,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39% -no-sync-no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39% -baseline,arch-a,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-a,vgg8,throughput,4,PASS,+2.33%,-1.72%,+18.49%,-0.98% -no-plan,arch-a,vgg8,throughput,4,PASS,+1251.16%,-92.58%,-56.68%,+577.04% -no-sync-no-plan,arch-a,vgg8,throughput,4,PASS,+1260.47%,-92.63%,-53.36%,+467.83% -baseline,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00% -no-plan,arch-b,googlenet,latency,1,PASS,+9849.64%,NA,-70.51%,+2833.89% -no-sync-no-plan,arch-b,googlenet,latency,1,PASS,+9849.55%,NA,-70.51%,+2833.89% -baseline,arch-b,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-b,googlenet,throughput,4,FAIL,+11.45%,-10.29%,+21.82%,+19.27% -no-plan,arch-b,googlenet,throughput,4,PASS,-100.00%,-100.00%,-63.39%,-100.00% -no-sync-no-plan,arch-b,googlenet,throughput,4,FAIL,+3348.28%,-97.10%,-61.58%,+1026.09% -baseline,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,-0.00% -no-plan,arch-b,resnet18,latency,1,PASS,+2935.63%,NA,-63.12%,+1019.81% -no-sync-no-plan,arch-b,resnet18,latency,1,PASS,+2936.09%,NA,-63.12%,+1019.97% -baseline,arch-b,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-b,resnet18,throughput,4,FAIL,+8.03%,-7.44%,+47.59%,+40.61% -no-plan,arch-b,resnet18,throughput,4,PASS,+1511.85%,-93.80%,-52.93%,+560.80% -no-sync-no-plan,arch-b,resnet18,throughput,4,FAIL,+867.12%,-89.67%,-45.41%,+359.83% -baseline,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-plan,arch-b,resnet34,latency,1,PASS,+2551.16%,NA,-62.51%,+893.70% -no-sync-no-plan,arch-b,resnet34,latency,1,PASS,+2550.91%,NA,-62.51%,+893.62% -baseline,arch-b,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-b,resnet34,throughput,4,FAIL,+28.25%,-22.13%,+49.76%,+57.30% -no-plan,arch-b,resnet34,throughput,4,PASS,+2427.17%,-96.05%,-49.58%,+938.62% -no-sync-no-plan,arch-b,resnet34,throughput,4,FAIL,+531.79%,-84.19%,-42.32%,+197.08% -baseline,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-sync,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00% -no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73% -no-sync-no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73% -baseline,arch-b,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00% -no-sync,arch-b,vgg8,throughput,4,PASS,+23.91%,-19.82%,+25.67%,+23.93% -no-plan,arch-b,vgg8,throughput,4,PASS,+1158.70%,-92.07%,-58.95%,+408.33% -no-sync-no-plan,arch-b,vgg8,throughput,4,PASS,+1172.83%,-92.17%,-55.83%,+327.00% +comparison,arch,model,mode,raptor_pipeline,baseline_variant,baseline_functional_validation,functional_validation,latency_gain_percent,throughput_gain_percent,power_gain_percent,energy_gain_percent +plan,arch-a,googlenet,latency,1,no-plan,PASS,PASS,+99.14%,NA,-226.98%,+97.19% +plan,arch-a,googlenet,throughput,4,no-plan,PASS,PASS,NA,NA,-281.83%,NA +sync,arch-a,googlenet,throughput,4,no-sync,FAIL,PASS,-5.17%,-4.96%,+12.23%,-1.65% +plan,arch-a,resnet18,latency,1,no-plan,PASS,PASS,+96.96%,NA,-156.23%,+92.21% +plan,arch-a,resnet18,throughput,4,no-plan,PASS,PASS,+95.17%,+1970.00%,-157.48%,+85.39% +sync,arch-a,resnet18,throughput,4,no-sync,FAIL,PASS,+7.15%,+7.81%,+14.70%,+7.92% +plan,arch-a,resnet34,latency,1,no-plan,PASS,PASS,+95.94%,NA,-130.05%,+90.65% +plan,arch-a,resnet34,throughput,4,no-plan,PASS,PASS,+96.00%,+2400.00%,-189.18%,+88.43% +sync,arch-a,resnet34,throughput,4,no-sync,FAIL,PASS,-9.11%,-8.42%,+7.32%,-1.02% +plan,arch-a,vgg8,latency,1,no-plan,PASS,PASS,+94.75%,NA,-80.65%,+90.51% +plan,arch-a,vgg8,throughput,4,no-plan,PASS,PASS,+92.77%,+1279.31%,-170.46%,+82.49% +sync,arch-a,vgg8,throughput,4,no-sync,FAIL,PASS,+19.18%,+24.03%,+21.29%,+18.62% +plan,arch-b,googlenet,latency,1,no-plan,PASS,PASS,+98.99%,NA,-239.10%,+96.59% +plan,arch-b,googlenet,throughput,4,no-plan,PASS,PASS,NA,NA,-306.99%,NA +sync,arch-b,googlenet,throughput,4,no-sync,FAIL,PASS,-1.28%,-1.63%,+9.99%,-4.29% +plan,arch-b,resnet18,latency,1,no-plan,PASS,PASS,+96.71%,NA,-171.16%,+91.07% +plan,arch-b,resnet18,throughput,4,no-plan,PASS,PASS,+95.46%,+2102.50%,-200.52%,+83.26% +sync,arch-b,resnet18,throughput,4,no-sync,FAIL,PASS,+2.22%,+2.32%,+8.26%,-7.88% +plan,arch-b,resnet34,latency,1,no-plan,PASS,PASS,+96.23%,NA,-166.76%,+89.94% +plan,arch-b,resnet34,throughput,4,no-plan,PASS,PASS,+97.58%,+4030.00%,-168.71%,+91.79% +sync,arch-b,resnet34,throughput,4,no-sync,FAIL,PASS,-0.80%,-0.96%,+10.61%,-13.51% +plan,arch-b,vgg8,latency,1,no-plan,PASS,PASS,+94.68%,NA,-98.04%,+89.47% +plan,arch-b,vgg8,throughput,4,no-plan,PASS,PASS,+93.70%,+1484.16%,-205.54%,+81.37% +sync,arch-b,vgg8,throughput,4,no-sync,FAIL,PASS,+17.07%,+20.30%,+19.03%,+9.63% diff --git a/validation/tools/pim/pimcomp/compare/results_comparison.csv b/validation/tools/pim/pimcomp/compare/results_comparison.csv index 1711033..a2ec44a 100644 --- a/validation/tools/pim/pimcomp/compare/results_comparison.csv +++ b/validation/tools/pim/pimcomp/compare/results_comparison.csv @@ -1,41 +1,41 @@ arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup arch-a,vgg8,latency,1,element,PASS,PASS,1.51833,7.98507,NA,NA,321.046,200.111,4.87455e+08,1.5979e+09,raptor,5.26 -arch-a,vgg8,throughput,2,batch,PASS,PASS,0.906321,1.0364,1100,965,429.311,482.597,4.07972e+08,5.41692e+08,raptor,1.14 -arch-a,vgg8,throughput,4,batch,PASS,PASS,0.778242,1.0364,1280,965,457.039,482.597,4.15762e+08,5.41692e+08,raptor,1.33 -arch-a,vgg8,throughput,8,batch,PASS,PASS,0.778094,1.0364,1290,965,464.071,482.597,3.96424e+08,5.41692e+08,raptor,1.34 -arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,raptor,4.15 -arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.7314,13.4585,85.2,74.3,503.111,481.604,6.62807e+09,6.56345e+09,raptor,1.15 -arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.2072,13.4585,52.1,74.3,380.1,481.604,7.95703e+09,6.56345e+09,pimcomp,1.43 -arch-a,resnet18,throughput,8,batch,PASS,FAIL,14.1578,13.4585,70.6,74.3,453.416,481.604,6.77909e+09,6.56345e+09,pimcomp,1.05 -arch-a,resnet34,latency,1,element,PASS,FAIL,29.7186,66.7114,NA,NA,411.806,277.485,1.22383e+10,1.85114e+10,raptor,2.24 -arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.2705,23.5817,43,42.4,471.717,515.68,1.3055e+10,1.22836e+10,raptor,1.01 -arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.468,23.5817,36.4,42.4,422.223,515.68,1.48991e+10,1.22836e+10,pimcomp,1.16 -arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.8392,23.5817,25.1,42.4,352.28,515.68,1.70627e+10,1.22836e+10,pimcomp,1.69 -arch-a,googlenet,latency,1,element,PASS,PASS,9.77772,62.9248,NA,NA,563.864,231.193,5.5133e+09,1.45477e+10,raptor,6.44 -arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.41545,18.2643,119,54.8,610.746,435.107,5.77746e+09,8.16479e+09,raptor,2.17 -arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.7292,18.2643,93.2,54.8,528.026,435.107,6.59289e+09,8.16479e+09,raptor,1.70 -arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.1219,18.2643,98.8,54.8,523.808,435.107,6.52174e+09,8.16479e+09,raptor,1.80 -arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.174,NA,4.27977e+10,NA,NA,NA -arch-a,yolo11n,throughput,2,batch,SKIPPED,FAIL,102.337,NA,9.77,NA,337.882,NA,3.83081e+10,NA,NA,NA -arch-a,yolo11n,throughput,4,batch,SKIPPED,FAIL,87.9103,NA,11.4,NA,358.201,NA,3.87298e+10,NA,NA,NA -arch-a,yolo11n,throughput,8,batch,SKIPPED,FAIL,82.9508,NA,12.1,NA,366.725,NA,3.85347e+10,NA,NA,NA +arch-a,vgg8,throughput,2,batch,PASS,FAIL,0.625998,1.0364,1600,965,523.278,482.597,4.17419e+08,5.41692e+08,NA,1.66 +arch-a,vgg8,throughput,4,batch,PASS,FAIL,0.625998,1.0364,1600,965,523.278,482.597,4.17419e+08,5.41692e+08,NA,1.66 +arch-a,vgg8,throughput,8,batch,PASS,FAIL,0.62635,1.0364,1600,965,519.182,482.597,4.14242e+08,5.41692e+08,NA,1.66 +arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,NA,4.15 +arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.0904,13.4585,90.2,74.3,520.244,481.604,6.55167e+09,6.56345e+09,NA,1.21 +arch-a,resnet18,throughput,4,batch,PASS,FAIL,12.0822,13.4585,82.8,74.3,487.576,481.604,6.91662e+09,6.56345e+09,NA,1.11 +arch-a,resnet18,throughput,8,batch,PASS,FAIL,11.9279,13.4585,83.8,74.3,499.354,481.604,6.84729e+09,6.56345e+09,NA,1.13 +arch-a,resnet34,latency,1,element,PASS,FAIL,29.7189,66.7114,NA,NA,411.804,277.485,1.22384e+10,1.85114e+10,NA,2.24 +arch-a,resnet34,throughput,2,batch,PASS,FAIL,21.8196,23.5817,45.8,42.4,477.108,515.68,1.57899e+10,1.22836e+10,NA,1.08 +arch-a,resnet34,throughput,4,batch,PASS,FAIL,20,23.5817,50,42.4,548.372,515.68,1.09674e+10,1.22836e+10,NA,1.18 +arch-a,resnet34,throughput,8,batch,PASS,FAIL,20.6363,23.5817,48.5,42.4,504.854,515.68,1.23725e+10,1.22836e+10,NA,1.14 +arch-a,googlenet,latency,1,element,PASS,FAIL,9.77731,62.9248,NA,NA,563.88,231.193,5.51323e+09,1.45477e+10,NA,6.44 +arch-a,googlenet,throughput,2,batch,PASS,FAIL,7.34567,18.2643,136,54.8,679.351,435.107,5.5053e+09,8.16479e+09,NA,2.48 +arch-a,googlenet,throughput,4,batch,PASS,FAIL,7.43728,18.2643,134,54.8,674.025,435.107,5.60809e+09,8.16479e+09,NA,2.45 +arch-a,googlenet,throughput,8,batch,PASS,FAIL,9.01886,18.2643,111,54.8,595.582,435.107,5.71605e+09,8.16479e+09,NA,2.03 +arch-a,yolo11n,latency,1,element,PASS,FAIL,149.46,NA,NA,NA,288.579,NA,4.31311e+10,NA,NA,NA +arch-a,yolo11n,throughput,2,batch,PASS,FAIL,94.946,NA,10.5,NA,346.466,NA,3.75539e+10,NA,NA,NA +arch-a,yolo11n,throughput,4,batch,PASS,FAIL,80.1193,NA,12.5,NA,377.408,NA,3.8811e+10,NA,NA,NA +arch-a,yolo11n,throughput,8,batch,PASS,FAIL,99.5488,NA,10,NA,339.331,NA,4.27113e+10,NA,NA,NA arch-b,vgg8,latency,1,element,PASS,PASS,1.5333,7.15212,NA,NA,292.559,173.769,4.48582e+08,1.24281e+09,raptor,4.66 -arch-b,vgg8,throughput,2,batch,PASS,PASS,0.685556,1.07282,1460,932,456.044,409.47,4.03694e+08,4.39289e+08,raptor,1.57 -arch-b,vgg8,throughput,4,batch,PASS,PASS,0.673192,1.07282,1490,932,448.898,409.47,4.13527e+08,4.39289e+08,raptor,1.60 -arch-b,vgg8,throughput,8,batch,PASS,PASS,0.751106,1.07282,1330,932,425.742,409.47,3.94458e+08,4.39289e+08,raptor,1.43 -arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,raptor,4.19 -arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.1214,13.434,82.5,74.4,461.407,484.804,6.29532e+09,6.56966e+09,raptor,1.11 -arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3601,13.434,88,74.4,479.333,484.804,6.98745e+09,6.56966e+09,raptor,1.18 -arch-b,resnet18,throughput,8,batch,PASS,FAIL,13.1216,13.434,76.2,74.4,441.248,484.804,6.38271e+09,6.56966e+09,raptor,1.02 -arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,raptor,3.83 -arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.282,22.8454,44.9,43.8,438.238,493.332,1.42187e+10,1.13922e+10,raptor,1.03 -arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.0906,22.8454,34.4,43.8,364.956,493.332,1.63285e+10,1.13922e+10,pimcomp,1.27 -arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.8454,21,43.8,289.821,493.332,1.63464e+10,1.13922e+10,pimcomp,2.09 -arch-b,googlenet,latency,1,element,PASS,PASS,11.2212,37.9367,NA,NA,483.266,242.037,5.42282e+09,9.18206e+09,raptor,3.38 -arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21209,21.3902,109,46.8,545.488,417.866,5.5934e+09,9.01473e+09,raptor,2.33 -arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56194,21.3902,117,46.8,555.264,417.866,5.86312e+09,9.01473e+09,raptor,2.50 -arch-b,googlenet,throughput,8,batch,PASS,FAIL,10.8652,21.3902,92,46.8,483.477,417.866,6.12683e+09,9.01473e+09,raptor,1.97 -arch-b,yolo11n,latency,1,element,PASS,FAIL,168.144,NA,NA,NA,245.314,NA,4.1248e+10,NA,NA,NA -arch-b,yolo11n,throughput,2,batch,SKIPPED,FAIL,104.494,NA,9.57,NA,302.121,NA,3.58015e+10,NA,NA,NA -arch-b,yolo11n,throughput,4,batch,SKIPPED,FAIL,108.247,NA,9.24,NA,297.484,NA,3.76297e+10,NA,NA,NA -arch-b,yolo11n,throughput,8,batch,SKIPPED,FAIL,100.129,NA,9.99,NA,308.833,NA,3.71808e+10,NA,NA,NA +arch-b,vgg8,throughput,2,batch,PASS,FAIL,0.624729,1.07282,1600,932,489.899,409.47,3.93036e+08,4.39289e+08,NA,1.72 +arch-b,vgg8,throughput,4,batch,PASS,FAIL,0.62331,1.07282,1600,932,490.955,409.47,3.93418e+08,4.39289e+08,NA,1.72 +arch-b,vgg8,throughput,8,batch,PASS,FAIL,0.62331,1.07282,1600,932,490.955,409.47,3.93418e+08,4.39289e+08,NA,1.72 +arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,NA,4.19 +arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.9113,13.434,77.5,74.4,438.931,484.804,6.47794e+09,6.56966e+09,NA,1.04 +arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3505,13.434,88.1,74.4,478.606,484.804,6.66517e+09,6.56966e+09,NA,1.18 +arch-b,resnet18,throughput,8,batch,PASS,FAIL,11.3685,13.434,88,74.4,478.33,484.804,6.51513e+09,6.56966e+09,NA,1.18 +arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,NA,3.83 +arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.4967,22.8454,44.5,43.8,440.06,493.332,1.49623e+10,1.13922e+10,NA,1.02 +arch-b,resnet34,throughput,4,batch,PASS,FAIL,24.1838,22.8454,41.3,43.8,426.129,493.332,1.30195e+10,1.13922e+10,NA,1.06 +arch-b,resnet34,throughput,8,batch,PASS,FAIL,24.1516,22.8454,41.4,43.8,420.745,493.332,1.39184e+10,1.13922e+10,NA,1.06 +arch-b,googlenet,latency,1,element,PASS,FAIL,11.2206,37.9367,NA,NA,483.283,242.037,5.42274e+09,9.18206e+09,NA,3.38 +arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.08396,21.3902,110,46.8,550.292,417.866,5.70509e+09,9.01473e+09,NA,2.35 +arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.23423,21.3902,121,46.8,597.327,417.866,5.62683e+09,9.01473e+09,NA,2.59 +arch-b,googlenet,throughput,8,batch,PASS,FAIL,7.78944,21.3902,128,46.8,620.65,417.866,5.47039e+09,9.01473e+09,NA,2.74 +arch-b,yolo11n,latency,1,element,PASS,FAIL,171.598,NA,NA,NA,243.177,NA,4.17288e+10,NA,NA,NA +arch-b,yolo11n,throughput,2,batch,PASS,FAIL,99.5318,NA,10,NA,309.102,NA,3.5219e+10,NA,NA,NA +arch-b,yolo11n,throughput,4,batch,PASS,FAIL,85.3497,NA,11.7,NA,326.712,NA,3.58691e+10,NA,NA,NA +arch-b,yolo11n,throughput,8,batch,PASS,FAIL,97.9941,NA,10.2,NA,308.041,NA,3.5011e+10,NA,NA,NA diff --git a/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py b/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py index 1b2848a..4d84331 100755 --- a/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py +++ b/validation/tools/pim/pimcomp/compare/run_pimcomp_models.py @@ -282,6 +282,23 @@ def comparison_passed(report: dict) -> bool: return result.get("status") == "done" and bool(result.get("passed")) +def completed_synchronization_ablation(report: dict) -> bool: + """A completed no-sync measurement may have intentionally invalid outputs.""" + validation = report.get("raptor_validation") or {} + performance = report.get("raptor_performance") or {} + return ( + report.get("pimsim_mode") == "throughput" + and "--pim-disable-synchronization" in report.get("raptor_extra_args", []) + and validation.get("status") == "done" and validation.get("passed") is False + and bool(validation.get("max_diffs")) + and not report.get("failures") + and bool(report.get("steps")) + and all(step.get("status") == "passed" for step in report["steps"]) + and not performance.get("error") + and performance_values(performance)["throughput"] is not None + ) + + def functional_validation_status(result: dict | None) -> str: if result is None: return "NA" @@ -869,6 +886,7 @@ def main() -> int: assert results_path is not None print_stage(results_path.name, STAGE_COLORS["Compare outputs"]) print(results_path.read_text(encoding="utf-8"), end="") + diagnostic_failures = [] for arch in args.archs: for name in args.models: for mode, pipeline, _ in comparisons_by_arch[arch]: @@ -876,15 +894,19 @@ def main() -> int: report_path = result_dir( out_dir, name, arch, mode, pipeline, args.ablation_variant ) / "pimcomp/comparison_report.json" - if not report_path.exists() or not comparison_passed( - json.loads(report_path.read_text(encoding="utf-8")), - ): - if label not in failed: + report = json.loads(report_path.read_text(encoding="utf-8")) if report_path.exists() else {} + if not comparison_passed(report): + if (label not in failed and args.ablation_variant == "no-sync" + and completed_synchronization_ablation(report)): + diagnostic_failures.append(label) + elif label not in failed: failed.append(label) print(Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL) total_jobs = sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values()) - print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL) + print(Style.BRIGHT + f"Passed: {total_jobs - len(failed) - len(diagnostic_failures)}" + Style.RESET_ALL) print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL) + if diagnostic_failures: + print("No-sync numerical failures (diagnostic only): " + ", ".join(diagnostic_failures)) print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL) if failed: print( diff --git a/validation/tools/pim/pimcomp/correctness/README.md b/validation/tools/pim/pimcomp/correctness/README.md index a012e95..b1adce3 100644 --- a/validation/tools/pim/pimcomp/correctness/README.md +++ b/validation/tools/pim/pimcomp/correctness/README.md @@ -35,13 +35,15 @@ The corresponding scheduling call is disabled in Rust validation starts those buffers at zero, exposing the missing computation. ```bash -.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py +.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py \ + /path/to/comparison /path/to/matching-model.onnx ``` -The default uses GoogLeNet Arch-A throughput/pipeline2. Optional positional -arguments are comparison directory, model, work directory, and shared reference -directory; the default work directory is `correctness/prefill/` below the -comparison artifacts. +The comparison directory and its matching model are required. Optional positional +arguments are work directory and shared reference directory. The default work +directory is `correctness/prefill/` below the comparison artifacts. +Historical Softmax-free models can be recovered from Git for historical artifacts. +Current GoogLeNet retains Softmax; prefill alone does not correct omitted Softmax execution. The experiment compares the original memory image with one prefilled using [`batch_memory.py`](prefill/batch_memory.py). In the recorded case, 20 of 39 diff --git a/validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py b/validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py index ae81ccd..53fc57e 100755 --- a/validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py +++ b/validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py @@ -16,8 +16,6 @@ from batch_memory import prefill_batch_memory SCRIPT_DIR = Path(__file__).resolve().parent REPO_ROOT = SCRIPT_DIR.parents[5] -DEFAULT_COMPARISON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2" -DEFAULT_MODEL = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx" DEFAULT_COMMON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/artifacts/common" SIMULATOR_MANIFEST = REPO_ROOT / "backend-simulators/pim/pim-simulator/Cargo.toml" @@ -78,8 +76,8 @@ def compare_outputs( def main() -> int: parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("comparison_dir", nargs="?", type=Path, default=DEFAULT_COMPARISON_DIR) - parser.add_argument("model", nargs="?", type=Path, default=DEFAULT_MODEL) + parser.add_argument("comparison_dir", type=Path) + parser.add_argument("model", type=Path, help="ONNX model used to generate the comparison artifacts") parser.add_argument("work_dir", nargs="?", type=Path) parser.add_argument("common_dir", nargs="?", type=Path, default=DEFAULT_COMMON_DIR) args = parser.parse_args()