Raptor/tools/raptor_graph_explorer/ARCHITECTURE.md

Graph Explorer architecture and API

The README covers installation, input headers, commands, and tests. This reference describes the contracts maintained by the implementation.

Ingestion and identity

discovery.py recursively pairs reports by their full filename prefix, including ZIP inputs, and rejects unsafe archive paths. That prefix is report_id; unrelated reports never share endpoints.

ingest.py streams CSVs in batches of 2,000 rows through temporary SQLite staging tables. Node IDs, operation IDs, lanes, cores, and channels remain separate fields. Known node prefixes are gc, gcb, sc, scb; unknown ones retain unknown:<prefix>. Explicit operation and lane columns are authoritative. Blank operation IDs, and blank batch lanes, may use a parseable source ID with a warning. Explicit edge lanes override endpoint lanes, with contradictory values diagnosed. Missing numbers stay SQL NULL; malformed rows are diagnosed and omitted, never coerced to zero. Extra columns survive in attributes_json.

Duplicate node IDs retain the first source row and produce diagnostics. Bulk joins resolve edges within their report; unknown endpoints are omitted with bounded examples. Staging tables are dropped even on error. The ingestion connection closes before indexes, aggregates, motifs, and diagnostics are built. Secondary indexes are created after raw insertion. CSVs without extra columns reuse an empty JSON representation.

Raw edge IDs use report ID plus original CSV row number; aggregate IDs use report ID and grouping keys. Input filesystem paths and browser layout do not define identity. The manifest stores versions and counts, not duplicate raw rows or machine-specific input paths. Database schema compatibility is checked before serving; incompatible outputs must be rebuilt.

Aggregation and exact lane mappings

Operation nodes group by (report_id, op_id), retaining kind counts, lane/core cardinalities and ranges, and SSA summaries. Operation edges group by the source/target operation pair, retaining counts, weights, tensor types, channels, and exact mapping classification. Core aggregation has a distinct unassigned group for missing cores. Node-kind aggregation is a small diagnostic view.

mappings.py uses all normalized lane pairs, in the following precedence order:

Class	Meaning
scalar_to_scalar	Both lanes absent for every pair.
scalar_to_batch / batch_to_scalar	One side entirely absent, the other entirely present; range, cardinality, and contiguity recorded.
identity	One-to-one relation with equal lanes.
constant_offset	One-to-one with one nonzero target-minus-source offset.
permutation	Remaining one-to-one relation with equal cardinalities.
broadcast / fan_in	One source to many targets / many sources to one target.
stencil	Multiple sources share exactly the same target-offset set, containing 2–64 offsets.
irregular	Complete lane data matching none of the preceding forms.
unknown	Empty or partially missing relation outside the scalar/batch cases.

SQL computes degrees/cardinalities; stencil comparison streams one operation pair at a time. No sampling determines classification. Mapping rows remain paginated; matrices show exact sparse points up to the configured threshold, then bounded density bins.

Display projection and layout

projection.py rebuilds the entire display from the current expanded-operation set. Expanding replaces an operation aggregate with its raw nodes, including isolated nodes. An aggregate edge survives only while both endpoint operations are collapsed. Otherwise its raw edges replace it: each endpoint is raw if its operation is expanded and aggregate if collapsed. The complete set determines endpoints, so expansion order cannot leave stale endpoints or duplicate aggregate/raw representations. Raw view expands every operation, preserving each valid source edge once.

Operation ranks use the complete operation graph, including isolated nodes. Collapsed operations occupy one row; expanded lanes use compact rows in numeric order, followed by deterministic lane-less rows. Sparse lane numbers leave no empty rows, and nodes sharing a lane get centered deterministic horizontal offsets. Operation/raw views use stable server coordinates restored by Reset layout. Core/node-kind views use ELK. Views flow top-to-bottom.

The viewer first requests an operation aggregate before applying the report's view selection. Raw selections use raw details rather than aggregate-only endpoints. Operation labels use the first SSA name, falling back to source identity; label collision handling and bounded node sizes keep dense views usable. Dependencies are pinned in that file (Graphology, Sigma.js, ELK); first browser use requires CDN access.

Structural motifs

motifs.py loads only the operation aggregate into NetworkX. It condenses SCCs into a DAG, attaches synthetic source/sink nodes, and computes dominators and reverse post-dominators. Branching entries are paired with their immediate post-dominator; regions retain nodes reachable before that exit which can reach it, rejecting external entrances/exits. One shortest representative path per branch avoids enumerating all paths.

exact_diamond has two direct one-node branches; rhomboid has disjoint branch interiors; split_rejoin covers other valid single-entry/single-exit regions. Identical memberships are deduplicated; nested motifs remain. SCC members map back to operation IDs, with deterministic representative operations for multi-operation SCC endpoints/paths and contains_cycle markers. Screen geometry never determines motif membership.

Read-only API

All routes below use GET; see api.py. Responses include Cache-Control: no-store to avoid mixing viewer revisions.

Route	Result / relevant query parameters
/api/manifest	Tool/schema versions and report counts.
/api/reports	Available reports.
/api/reports/{report_id}/summary	Counts, mapping classes, diagnostics.
/api/reports/{report_id}/graph/{level}	Complete operation, core, or node_kind aggregate.
/api/reports/{report_id}/display-graph	Deterministic projection; view=operation or raw, repeated expanded_op_id, expand_all.
/api/reports/{report_id}/motifs	Structural motifs.
/api/aggregate-nodes/{aggregate_id}	Aggregate details; operation SSA names capped at 100.
/api/aggregate-edges/{aggregate_edge_id}	Aggregate edge details.
/api/aggregate-edges/{aggregate_edge_id}/mapping	Exact raw lane rows; limit, offset. Operation edges only.
/api/aggregate-edges/{aggregate_edge_id}/matrix	Exact sparse points or density bins. Operation edges only.
/api/reports/{report_id}/raw-nodes	Raw node page; limit, offset.
/api/reports/{report_id}/raw-edges	Raw edge page; limit, offset.
/api/reports/{report_id}/subgraph	aggregate_id, direction=in/out/both, depth, optional expand_lanes.

create_app defaults: page size 100 with a maximum of 500; expansion cap 5,000; exact matrix threshold 2,000 points; density grid 64×64. Subgraph depth defaults to 1 and caps at 5. Oversized raw/expanded projections return HTTP 400 without a partial graph. The projection cap counts raw nodes plus incident raw edges; collapsed aggregate nodes/edges are additional. Aggregate endpoints have no pagination or expansion cap, so very large operation graphs remain a scaling limit. Raw CSV graphs are never loaded wholesale into NetworkX.