Raptor/tools/raptor_graph_explorer/README.md

Raptor Graph Explorer

Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.

Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.

Architecture

CSV pairs / directories / ZIP archives
  -> recursive discovery and safe ZIP extraction
  -> streaming CSV validation and SQLite staging
  -> normalized raw_nodes and raw_edges
  -> operation, core, and node-kind aggregates
  -> exact lane mappings
  -> SCC condensation and structural split/rejoin motifs
  -> graph.sqlite + manifest.json
  -> read-only FastAPI API
  -> Sigma.js WebGL viewer with stable operation/lane layout

Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.

Installation

From the Raptor repository root:

python3.13 -m venv .venv
source .venv/bin/activate
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]

The installed command is raptor-graph-explorer. The equivalent module entry point is python3.13 -m raptor_graph_explorer.

Commands

Build a persistent output:

raptor-graph-explorer build \
  validation/networks/vgg16/depth_07/raptor/reports \
  --output /tmp/vgg_graph

Use repeated inputs or pass several paths after build. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit .nodes.csv / .edges.csv files. --strict rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. --overwrite replaces an existing output.

Serve an existing output:

raptor-graph-explorer serve /tmp/vgg_graph --port 8765

Build and immediately serve:

raptor-graph-explorer run \
  /mnt/data/conv_relu_conv.zip \
  --output /tmp/conv_relu_conv_graph \
  --overwrite

Print counts, mapping classes, SCCs, motifs, and diagnostics:

raptor-graph-explorer inspect /tmp/vgg_graph

The server defaults to 127.0.0.1:8765. It has no authentication and is intended for local read-only use.

Supported CSV reports

Files are paired using the complete prefix before .nodes.csv and .edges.csv. That prefix is the stable report_id; spatial2_scheduled_no_comm.nodes.csv therefore pairs only with spatial2_scheduled_no_comm.edges.csv.

Node columns:

Id,op_id,lane,core,ssa_name

Edge columns:

Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id

Known node prefixes are gc, gcb, sc, and scb. Unknown prefixes are retained as unknown:<prefix>. Explicit CSV op_id and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL NULL; malformed numeric values are never changed to zero. Additional CSV columns are preserved in attributes_json.

Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.

Output

OUTPUT_DIRECTORY/
├── graph.sqlite
└── manifest.json

graph.sqlite contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. manifest.json contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.

Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.

Aggregate views

Operation

Nodes group by (report_id, op_id) and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by (report_id, source_op_id, target_op_id) and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.

Core

Assigned nodes group by (report_id, core). Missing cores use a separate unassigned group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.

Node kind

Nodes group by (report_id, node_kind). This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.

Raw instances

Raw instances displays every raw_nodes row exactly once and every valid raw_edges row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial3 report starts in this view automatically; Operation, Core, and Node kind remain selectable.

Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.

Exact lane mappings

Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:

• scalar_to_scalar: every pair has two absent lanes.
• scalar_to_batch: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
• batch_to_scalar: the corresponding source-lane case.
• identity: a one-to-one relation with equal source and target lanes.
• constant_offset: a one-to-one relation with one nonzero target - source offset.
• permutation: a one-to-one relation that is neither identity nor constant offset.
• broadcast: one distinct source lane maps to multiple target lanes.
• fan_in: multiple source lanes map to one distinct target lane.
• stencil: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
• irregular: complete lane data matches none of the exact forms.
• unknown: lane data is partially missing outside the scalar/batch cases.

No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.

Structural motifs

Motifs are not inferred from rendered geometry. For each operation graph the tool:

1. computes strongly connected components and condenses them into a DAG;
2. attaches synthetic source and sink nodes;
3. computes dominators and reverse-graph post-dominators;
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
5. retains nodes reachable before the exit that can also reach the exit;
6. rejects unexpected entrances to, or exits from, internal nodes;
7. records one shortest representative path per branch without enumerating all paths;
8. maps condensed components back to operation aggregate IDs.

exact_diamond is a two-branch direct reconvergence with one internal condensed node per branch. rhomboid has clean disjoint branch interiors. split_rejoin is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks contains_cycle; cycles are never analyzed as if the raw operation graph were a DAG.

Viewer and API

Except for spatial3, the viewer initially fetches an aggregate operation graph unless the browser has a saved view choice. Sigma renders the graph with WebGL. Controls cover report/view/metric selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Mapping panels remain available for operation aggregate edges.

Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.

Operation ranks are calculated from the complete operation graph, including isolated operations. Collapsed nodes use stable operation anchors. Expanded nodes are sorted by lane and node ID; lane numbers form perpendicular rows, equal lanes align across adjacent operations, and lane-less nodes use distinct deterministic scalar rows. This same model is used by Rerun layout, so unrelated operations do not jump during expansion or collapse and disconnected nodes never pile up at (0, 0).

Browser dependencies are pinned in one place, static/index.html:

• Graphology 0.25.4;
• Sigma.js 2.4.0;
• ELK.js 0.11.1.

They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.

Read-only endpoints include:

/api/manifest
/api/reports
/api/reports/{report_id}/summary
/api/reports/{report_id}/graph/{level}
/api/reports/{report_id}/display-graph
/api/reports/{report_id}/motifs
/api/aggregate-nodes/{aggregate_id}
/api/aggregate-edges/{aggregate_edge_id}
/api/aggregate-edges/{aggregate_edge_id}/mapping
/api/aggregate-edges/{aggregate_edge_id}/matrix
/api/reports/{report_id}/raw-nodes
/api/reports/{report_id}/raw-edges
/api/reports/{report_id}/subgraph

Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured expansion_cap. These limits are configurable when embedding create_app.

Performance behavior

• CSV readers stream rows and insert them in bounded batches.
• Temporary SQLite staging and bulk joins resolve endpoints; ingestion performs no per-edge node query.
• Secondary indexes are created after raw insertion.
• Raw edges are never retained as a Python object graph or loaded into NetworkX.
• Only aggregate operation nodes/edges enter NetworkX.
• Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
• API raw pages, display projections, expansions, and matrices have hard limits.
• Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.

The normal suite contains a generated multi-lane smoke report. Set RAPTOR_GRAPH_SLOW=1 to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:

RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow

Current limitations

• The first browser load requires access to the pinned CDNs; offline vendoring is not included.
• Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
• Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
• The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.

Troubleshooting

• No complete report pairs: check that both files share the exact prefix and end in .nodes.csv / .edges.csv.
• Strict build failed: run without --strict, then inspect the diagnostics panel or diagnostics table for bounded examples.
• Output already exists: pass --overwrite only when replacement is intended.
• Unsupported schema: rebuild the output with the installed tool version; old databases are not silently reinterpreted.
• Raw/expanded display returns 400: use Operation, Core, or Node kind view, or embed the app with a larger expansion_cap when the local machine can safely handle it.
• Viewer is blank but APIs work: verify browser access to the three pinned CDN URLs and inspect the browser console.
• No motifs: this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.

No Raptor build or compiler modification is required.