add plot generation scripts adjust validation tools add up-to-date results
This commit is contained in:
@@ -0,0 +1,131 @@
|
||||
# Graph Explorer architecture and API
|
||||
|
||||
The [README](README.md) covers installation, input headers, commands, and tests.
|
||||
This reference describes the contracts maintained by the implementation.
|
||||
|
||||
## Ingestion and identity
|
||||
|
||||
[`discovery.py`](raptor_graph_explorer/discovery.py) recursively pairs reports
|
||||
by their full filename prefix, including ZIP inputs, and rejects unsafe archive
|
||||
paths. That prefix is `report_id`; unrelated reports never share endpoints.
|
||||
|
||||
[`ingest.py`](raptor_graph_explorer/ingest.py) streams CSVs in batches of 2,000
|
||||
rows through temporary SQLite staging tables. Node IDs, operation IDs, lanes,
|
||||
cores, and channels remain separate fields. Known node prefixes are `gc`,
|
||||
`gcb`, `sc`, `scb`; unknown ones retain `unknown:<prefix>`. Explicit operation
|
||||
and lane columns are authoritative. Blank operation IDs, and blank batch lanes,
|
||||
may use a parseable source ID with a warning. Explicit edge lanes override
|
||||
endpoint lanes, with contradictory values diagnosed. Missing numbers stay
|
||||
SQL NULL; malformed rows are diagnosed and omitted, never coerced to zero.
|
||||
Extra columns survive in `attributes_json`.
|
||||
|
||||
Duplicate node IDs retain the first source row and produce diagnostics. Bulk
|
||||
joins resolve edges within their report; unknown endpoints are omitted with
|
||||
bounded examples. Staging tables are dropped even on error. The ingestion
|
||||
connection closes before indexes, aggregates, motifs, and diagnostics are
|
||||
built. Secondary indexes are created after raw insertion. CSVs without extra
|
||||
columns reuse an empty JSON representation.
|
||||
|
||||
Raw edge IDs use report ID plus original CSV row number; aggregate IDs use
|
||||
report ID and grouping keys. Input filesystem paths and browser layout do not
|
||||
define identity. The manifest stores versions and counts, not duplicate raw
|
||||
rows or machine-specific input paths. Database schema compatibility is checked
|
||||
before serving; incompatible outputs must be rebuilt.
|
||||
|
||||
## Aggregation and exact lane mappings
|
||||
|
||||
Operation nodes group by `(report_id, op_id)`, retaining kind counts, lane/core
|
||||
cardinalities and ranges, and SSA summaries. Operation edges group by the
|
||||
source/target operation pair, retaining counts, weights, tensor types, channels,
|
||||
and exact mapping classification. Core aggregation has a distinct `unassigned`
|
||||
group for missing cores. Node-kind aggregation is a small diagnostic view.
|
||||
|
||||
[`mappings.py`](raptor_graph_explorer/mappings.py) uses all normalized lane
|
||||
pairs, in the following precedence order:
|
||||
|
||||
| Class | Meaning |
|
||||
|---|---|
|
||||
| `scalar_to_scalar` | Both lanes absent for every pair. |
|
||||
| `scalar_to_batch` / `batch_to_scalar` | One side entirely absent, the other entirely present; range, cardinality, and contiguity recorded. |
|
||||
| `identity` | One-to-one relation with equal lanes. |
|
||||
| `constant_offset` | One-to-one with one nonzero target-minus-source offset. |
|
||||
| `permutation` | Remaining one-to-one relation with equal cardinalities. |
|
||||
| `broadcast` / `fan_in` | One source to many targets / many sources to one target. |
|
||||
| `stencil` | Multiple sources share exactly the same target-offset set, containing 2–64 offsets. |
|
||||
| `irregular` | Complete lane data matching none of the preceding forms. |
|
||||
| `unknown` | Empty or partially missing relation outside the scalar/batch cases. |
|
||||
|
||||
SQL computes degrees/cardinalities; stencil comparison streams one operation
|
||||
pair at a time. No sampling determines classification. Mapping rows remain
|
||||
paginated; matrices show exact sparse points up to the configured threshold,
|
||||
then bounded density bins.
|
||||
|
||||
## Display projection and layout
|
||||
|
||||
[`projection.py`](raptor_graph_explorer/projection.py) rebuilds the entire
|
||||
display from the current expanded-operation set. Expanding replaces an
|
||||
operation aggregate with its raw nodes, including isolated nodes. An aggregate
|
||||
edge survives only while both endpoint operations are collapsed. Otherwise its
|
||||
raw edges replace it: each endpoint is raw if its operation is expanded and
|
||||
aggregate if collapsed. The complete set determines endpoints, so expansion
|
||||
order cannot leave stale endpoints or duplicate aggregate/raw representations.
|
||||
Raw view expands every operation, preserving each valid source edge once.
|
||||
|
||||
Operation ranks use the complete operation graph, including isolated nodes.
|
||||
Collapsed operations occupy one row; expanded lanes use compact rows in numeric
|
||||
order, followed by deterministic lane-less rows. Sparse lane numbers leave no
|
||||
empty rows, and nodes sharing a lane get centered deterministic horizontal
|
||||
offsets. Operation/raw views use stable server coordinates restored by Reset
|
||||
layout. Core/node-kind views use ELK. Views flow top-to-bottom.
|
||||
|
||||
The [viewer](raptor_graph_explorer/static/index.html) first requests an
|
||||
operation aggregate before applying the report's view selection. Raw selections
|
||||
use raw details rather than aggregate-only endpoints. Operation labels use the
|
||||
first SSA name, falling back to source identity; label collision handling and
|
||||
bounded node sizes keep dense views usable. Dependencies are pinned in that
|
||||
file (Graphology, Sigma.js, ELK); first browser use requires CDN access.
|
||||
|
||||
## Structural motifs
|
||||
|
||||
[`motifs.py`](raptor_graph_explorer/motifs.py) loads only the operation aggregate
|
||||
into NetworkX. It condenses SCCs into a DAG, attaches synthetic source/sink
|
||||
nodes, and computes dominators and reverse post-dominators. Branching entries
|
||||
are paired with their immediate post-dominator; regions retain nodes reachable
|
||||
before that exit which can reach it, rejecting external entrances/exits.
|
||||
One shortest representative path per branch avoids enumerating all paths.
|
||||
|
||||
`exact_diamond` has two direct one-node branches; `rhomboid` has disjoint
|
||||
branch interiors; `split_rejoin` covers other valid single-entry/single-exit
|
||||
regions. Identical memberships are deduplicated; nested motifs remain.
|
||||
SCC members map back to operation IDs, with deterministic representative
|
||||
operations for multi-operation SCC endpoints/paths and `contains_cycle` markers.
|
||||
Screen geometry never determines motif membership.
|
||||
|
||||
## Read-only API
|
||||
|
||||
All routes below use GET; see [`api.py`](raptor_graph_explorer/api.py).
|
||||
Responses include `Cache-Control: no-store` to avoid mixing viewer revisions.
|
||||
|
||||
| Route | Result / relevant query parameters |
|
||||
|---|---|
|
||||
| `/api/manifest` | Tool/schema versions and report counts. |
|
||||
| `/api/reports` | Available reports. |
|
||||
| `/api/reports/{report_id}/summary` | Counts, mapping classes, diagnostics. |
|
||||
| `/api/reports/{report_id}/graph/{level}` | Complete `operation`, `core`, or `node_kind` aggregate. |
|
||||
| `/api/reports/{report_id}/display-graph` | Deterministic projection; `view=operation` or `raw`, repeated `expanded_op_id`, `expand_all`. |
|
||||
| `/api/reports/{report_id}/motifs` | Structural motifs. |
|
||||
| `/api/aggregate-nodes/{aggregate_id}` | Aggregate details; operation SSA names capped at 100. |
|
||||
| `/api/aggregate-edges/{aggregate_edge_id}` | Aggregate edge details. |
|
||||
| `/api/aggregate-edges/{aggregate_edge_id}/mapping` | Exact raw lane rows; `limit`, `offset`. Operation edges only. |
|
||||
| `/api/aggregate-edges/{aggregate_edge_id}/matrix` | Exact sparse points or density bins. Operation edges only. |
|
||||
| `/api/reports/{report_id}/raw-nodes` | Raw node page; `limit`, `offset`. |
|
||||
| `/api/reports/{report_id}/raw-edges` | Raw edge page; `limit`, `offset`. |
|
||||
| `/api/reports/{report_id}/subgraph` | `aggregate_id`, `direction=in/out/both`, `depth`, optional `expand_lanes`. |
|
||||
|
||||
`create_app` defaults: page size 100 with a maximum of 500; expansion cap 5,000;
|
||||
exact matrix threshold 2,000 points; density grid 64×64. Subgraph depth defaults
|
||||
to 1 and caps at 5. Oversized raw/expanded projections return HTTP 400 without a
|
||||
partial graph. The projection cap counts raw nodes plus incident raw edges;
|
||||
collapsed aggregate nodes/edges are additional. Aggregate endpoints have no
|
||||
pagination or expansion cap, so very large operation graphs remain a scaling
|
||||
limit. Raw CSV graphs are never loaded wholesale into NetworkX.
|
||||
@@ -1,228 +1,117 @@
|
||||
# Raptor Graph Explorer
|
||||
|
||||
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
|
||||
Python 3.13+ tool for exploring Raptor Spatial dataflow CSVs in a browser.
|
||||
It streams reports into SQLite, aggregates operations/cores/node kinds, and
|
||||
provides bounded raw-lane inspection. Compiler output remains unchanged.
|
||||
|
||||
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
|
||||
## Install and run
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
CSV pairs / directories / ZIP archives
|
||||
-> recursive discovery and safe ZIP extraction
|
||||
-> streaming CSV validation and SQLite staging
|
||||
-> normalized raw_nodes and raw_edges
|
||||
-> operation, core, and node-kind aggregates
|
||||
-> exact lane mappings
|
||||
-> SCC condensation and structural split/rejoin motifs
|
||||
-> graph.sqlite + manifest.json
|
||||
-> read-only FastAPI API
|
||||
-> Sigma.js WebGL viewer with stable operation/lane layout
|
||||
```
|
||||
|
||||
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
|
||||
|
||||
## Installation
|
||||
|
||||
From the Raptor repository root:
|
||||
From the repository root, create a Python 3.13+ environment if `.venv` does not
|
||||
exist:
|
||||
|
||||
```bash
|
||||
python3.13 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
|
||||
```
|
||||
|
||||
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
|
||||
|
||||
## Commands
|
||||
|
||||
Build a persistent output:
|
||||
For a new or existing environment, verify the version before installing. Select
|
||||
a compatible environment if this check fails; do not reuse an older Python:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer build \
|
||||
validation/networks/vgg16/depth_07/raptor/reports \
|
||||
--output /tmp/vgg_graph
|
||||
.venv/bin/python -c 'import sys; assert sys.version_info >= (3, 13), "Python 3.13+ required"'
|
||||
.venv/bin/python -m pip install -e 'tools/raptor_graph_explorer[dev]'
|
||||
.venv/bin/raptor-graph-explorer build /path/to/raptor/reports --output /tmp/raptor_graph
|
||||
.venv/bin/raptor-graph-explorer serve /tmp/raptor_graph --port 8765
|
||||
```
|
||||
|
||||
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
|
||||
Open `http://127.0.0.1:8765`. The server is read-only, has no authentication,
|
||||
and is intended for local use. Browser dependencies load from pinned CDNs and
|
||||
require network access on first use; preprocessing and inspection work offline.
|
||||
|
||||
Serve an existing output:
|
||||
Other commands:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
|
||||
# Build and serve a directory, CSV pair, or ZIP archive.
|
||||
.venv/bin/raptor-graph-explorer run /path/to/reports.zip --output /tmp/raptor_graph
|
||||
# Print counts, mappings, motifs, and diagnostics.
|
||||
.venv/bin/raptor-graph-explorer inspect /tmp/raptor_graph
|
||||
```
|
||||
|
||||
Build and immediately serve:
|
||||
`build` and `run` accept multiple input paths and discover reports recursively.
|
||||
`--overwrite` replaces an existing output. `build --strict` rejects
|
||||
data-integrity errors; otherwise valid pairs and unambiguous rows are retained.
|
||||
Outputs are `graph.sqlite` and `manifest.json`. Rebuild outputs if the tool
|
||||
reports an unsupported schema version.
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer run \
|
||||
/mnt/data/conv_relu_conv.zip \
|
||||
--output /tmp/conv_relu_conv_graph \
|
||||
--overwrite
|
||||
```
|
||||
See [architecture and API guarantees](ARCHITECTURE.md) for identity, expansion,
|
||||
mapping/motif analysis, ingestion constraints, and the endpoint reference.
|
||||
|
||||
Print counts, mapping classes, SCCs, motifs, and diagnostics:
|
||||
## Input contract
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer inspect /tmp/vgg_graph
|
||||
```
|
||||
|
||||
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
|
||||
|
||||
## Supported CSV reports
|
||||
|
||||
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial3_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial3_scheduled_no_comm.edges.csv`.
|
||||
|
||||
Node columns:
|
||||
Pair files by the complete prefix before `.nodes.csv` and `.edges.csv`.
|
||||
Required headers:
|
||||
|
||||
```csv
|
||||
Id,op_id,lane,core,ssa_name
|
||||
```
|
||||
|
||||
Edge columns:
|
||||
|
||||
```csv
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
```
|
||||
|
||||
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
|
||||
Explicit operation IDs and lanes take precedence over parsed node IDs. Missing
|
||||
numeric values remain missing; malformed values produce diagnostics. Extra
|
||||
columns are preserved. Edges resolve only within their report; unknown
|
||||
endpoints are diagnosed and omitted. Unsafe ZIP paths are rejected.
|
||||
|
||||
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
|
||||
Graph lanes, physical fragment slots, scheduled lanes, cores, and channels are
|
||||
distinct identities. Spatial3 reports contain explicit communication only;
|
||||
isolated nodes are valid and no dependencies are inferred from SSA text.
|
||||
|
||||
## Output
|
||||
## Views and analysis
|
||||
|
||||
```text
|
||||
OUTPUT_DIRECTORY/
|
||||
├── graph.sqlite
|
||||
└── manifest.json
|
||||
```
|
||||
- **Operation:** aggregates by operation and supports lane expansion, exact
|
||||
mapping details, and structural motifs.
|
||||
- **Core / Node kind:** summarize placement and operation kinds.
|
||||
- **Raw instances:** displays source nodes and valid edges, including isolated
|
||||
nodes; spatial4 reports select this view automatically.
|
||||
|
||||
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
|
||||
The viewer first loads an operation aggregate. Search, filters, expansion,
|
||||
selection details, and layout controls support focused inspection. Raw and
|
||||
expanded graphs cap raw nodes plus incident raw edges at 5,000; exceeding it returns an
|
||||
error rather than a partial graph. Use an aggregate view or increase
|
||||
`create_app(..., expansion_cap=...)` when the machine can handle it.
|
||||
|
||||
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
|
||||
Lane mappings use all normalized pairs, without sampling. Classes include
|
||||
scalar/batch relations, identity, constant offset, permutation, broadcast,
|
||||
fan-in, stencil, irregular, and unknown. Small matrices show exact sparse
|
||||
points; large ones use a bounded 64×64 density view.
|
||||
|
||||
## Aggregate views
|
||||
Motifs identify structural single-entry/single-exit split/rejoin regions on the
|
||||
operation graph's SCC condensation. `exact_diamond`, `rhomboid`, and
|
||||
`split_rejoin` describe progressively broader forms; a diamond-shaped drawing
|
||||
alone is not evidence of a motif. Cycles remain marked.
|
||||
|
||||
### Operation
|
||||
## API and limits
|
||||
|
||||
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
|
||||
Read-only FastAPI endpoints expose manifests, report summaries, aggregate
|
||||
graphs/details, mappings, matrices, motifs, and raw pages. See
|
||||
[`api.py`](raptor_graph_explorer/api.py) for routes and `create_app` limits.
|
||||
Raw pages default to 100 rows and cap at 500; subgraph depth caps at five.
|
||||
Aggregate graph endpoints return the complete aggregate, so very large
|
||||
operation counts can still be expensive.
|
||||
|
||||
### Core
|
||||
|
||||
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
|
||||
|
||||
### Node kind
|
||||
|
||||
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
|
||||
|
||||
### Raw instances
|
||||
|
||||
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial4 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
|
||||
|
||||
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
|
||||
|
||||
## Exact lane mappings
|
||||
|
||||
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
|
||||
|
||||
- `scalar_to_scalar`: every pair has two absent lanes.
|
||||
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
|
||||
- `batch_to_scalar`: the corresponding source-lane case.
|
||||
- `identity`: a one-to-one relation with equal source and target lanes.
|
||||
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
|
||||
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
|
||||
- `broadcast`: one distinct source lane maps to multiple target lanes.
|
||||
- `fan_in`: multiple source lanes map to one distinct target lane.
|
||||
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
|
||||
- `irregular`: complete lane data matches none of the exact forms.
|
||||
- `unknown`: lane data is partially missing outside the scalar/batch cases.
|
||||
|
||||
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
|
||||
|
||||
## Structural motifs
|
||||
|
||||
Motifs are not inferred from rendered geometry. For each operation graph the tool:
|
||||
|
||||
1. computes strongly connected components and condenses them into a DAG;
|
||||
2. attaches synthetic source and sink nodes;
|
||||
3. computes dominators and reverse-graph post-dominators;
|
||||
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
|
||||
5. retains nodes reachable before the exit that can also reach the exit;
|
||||
6. rejects unexpected entrances to, or exits from, internal nodes;
|
||||
7. records one shortest representative path per branch without enumerating all paths;
|
||||
8. maps condensed components back to operation aggregate IDs.
|
||||
|
||||
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
|
||||
|
||||
## Viewer and API
|
||||
|
||||
The viewer initially fetches an aggregate operation graph, so a previously selected raw view cannot make a new report exceed the display safety cap before the first render. Browser responses disable caching so HTML and JavaScript from different tool revisions cannot be mixed. Sigma renders the graph with WebGL. Controls cover report/view selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Edges use a fixed width. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Nodes without an SSA name fall back to their source identity. Mapping panels remain available for operation aggregate edges.
|
||||
|
||||
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
|
||||
|
||||
Operation ranks are calculated from the complete operation graph, including isolated operations, and every view flows top-to-bottom. A collapsed operation reserves one displayed row. Expanded nodes use compact rows for distinct lanes in numeric order, followed by deterministic lane-less rows, so sparse lane numbers create no empty geometric space. Raw nodes sharing a lane receive centered deterministic horizontal offsets. Operation and raw views use stable server coordinates, and `Reset layout` restores them; core and node-kind views use ELK, and `Rerun layout` runs it again. Operation labels use the first SSA name; Sigma's collision grid thins normal labels, and a label is skipped when its measured screen rectangle intersects another visible node. Selected or hovered nodes keep the original white bubble with black text. Selecting a motif frames its member nodes on the first click. Aggregate-node sizes remain bounded.
|
||||
|
||||
Browser dependencies are pinned in one place, `static/index.html`:
|
||||
|
||||
- Graphology 0.25.4;
|
||||
- Sigma.js 2.4.0;
|
||||
- ELK.js 0.11.1.
|
||||
|
||||
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
|
||||
|
||||
Read-only endpoints include:
|
||||
|
||||
```text
|
||||
/api/manifest
|
||||
/api/reports
|
||||
/api/reports/{report_id}/summary
|
||||
/api/reports/{report_id}/graph/{level}
|
||||
/api/reports/{report_id}/display-graph
|
||||
/api/reports/{report_id}/motifs
|
||||
/api/aggregate-nodes/{aggregate_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}/mapping
|
||||
/api/aggregate-edges/{aggregate_edge_id}/matrix
|
||||
/api/reports/{report_id}/raw-nodes
|
||||
/api/reports/{report_id}/raw-edges
|
||||
/api/reports/{report_id}/subgraph
|
||||
```
|
||||
|
||||
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
|
||||
|
||||
## Performance behavior
|
||||
|
||||
- CSV readers stream rows and insert them in bounded batches.
|
||||
- Temporary SQLite staging tables are function-scoped and dropped immediately; bulk joins resolve endpoints without per-edge node queries.
|
||||
- CSVs without additional columns use a constant empty JSON representation.
|
||||
- Ingestion closes its write connection before derived indexes, aggregation, motifs, and diagnostics reopen the database.
|
||||
- Secondary indexes are created after raw insertion and focus on serving and aggregation queries.
|
||||
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
|
||||
- Only aggregate operation nodes/edges enter NetworkX.
|
||||
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
|
||||
- API raw pages, display projections, expansions, and matrices have hard limits.
|
||||
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
|
||||
|
||||
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
|
||||
## Tests and troubleshooting
|
||||
|
||||
```bash
|
||||
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
|
||||
.venv/bin/python -m pytest tools/raptor_graph_explorer/tests
|
||||
RAPTOR_GRAPH_SLOW=1 .venv/bin/python -m pytest \
|
||||
tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
|
||||
```
|
||||
|
||||
## Current limitations
|
||||
The opt-in test generates 100 operations, 1,000 lanes, and 100,000 raw edges.
|
||||
|
||||
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
|
||||
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
|
||||
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
|
||||
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
|
||||
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
|
||||
- **Output already exists:** pass `--overwrite` only when replacement is intended.
|
||||
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
|
||||
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
|
||||
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
|
||||
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
|
||||
|
||||
No Raptor build or compiler modification is required.
|
||||
- **No complete pairs:** check matching prefixes and both CSV suffixes.
|
||||
- **Strict build failed:** inspect diagnostics from a build without `--strict`.
|
||||
- **Raw/expanded display returns 400:** use aggregation or adjust the cap.
|
||||
- **Blank viewer with working APIs:** check browser CDN access and console.
|
||||
- **No motifs:** valid when no structural split/rejoin region exists.
|
||||
|
||||
Reference in New Issue
Block a user