File / module catalog
Important paths with role, receives, ships, interacts with,
and risks. Grouped by area. Invented nothing: if a consumer is unknown,
it is listed as a gap. Fresher local main may add files not yet on this
branch.
Serving
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
main.py | FastAPI query API | HTTP (via Worker / direct origin), Neo4j reader, corpus dir, Bedrock | JSON answers, reviewer artifacts, MCP mount | query_engine/*, mcp_server.py, reviewer_bot*.py, bedrock_client.py | Monolith; fail-open internal key; ensure_indexes on first request; ephemeral logs/counters (F-06) |
mcp_server.py | MCP ASGI + KV auth | Bearer tokens, KV REST, graph_ask | MCP tools / SSE | oauth-trial.js (same KV), mcp-proxy.js origin shield | Budget PUT may silently no-op if token is read-only (F-17) |
query_engine/router.py | NL slot router | Question text | Template id or decline | gates, templates, main | Divergence from candidate registry until promotion |
query_engine/linker.py | Entity resolution + index helper | Mentions / ids | Linked entities | Neo4j | ensure_indexes() CREATE on request path |
query_engine/gates.py | Pre-spend coverage | Template id, live counts | Block or allow | coverage.py, /graph/ask | Historically incomplete table (F-15; fixed at source on later trees) |
query_engine/templates.py | T1–T17 Cypher executors | Params, session | Rows / preview Cypher | main, MCP tools | Large file; keep in sync with REST handlers |
query_engine/text2cypher.py | Freeform Cypher | NL + schema snapshot | Validated Cypher | syntology_schema.py | Stale schema → silent unreachability of code lane (F-07 historically) |
query_engine/logging.py | Request/proposal logs | Pipeline traces | request_log.jsonl | Cost/pricing plans | Container FS — lost on deploy |
reviewer_bot.py / _grounding.py / _pdf.py | Reviewer product | PDF/text, Bedrock, graph | Review JSON/PDF | Stripe reviewer-billing path | Not fully public per llms.txt “not yet live” caveats |
Dockerfile.api | Serving image | COPY list, requirements-api | App Runner image | deploy/deploy_api_to_aws.sh | COPY drift → rollback (F-02) |
Edge / Cloudflare
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
worker.js | Single-domain router + security headers | All site traffic | Proxied/static responses | Every *.js module, public/ | Route vs Custom Domain; run_worker_first required |
wrangler.toml | Worker + KV + assets config | n/a | Deploy bindings | Wrangler CLI | Do not add [[routes]] without zone grant; never commit secret values |
graph-agent-proxy.js | Reverse proxy to App Runner | Browser/agent REST | Upstream /graph/* | Turnstile, internal key, App Runner | Origin must stay shielded |
mcp-proxy.js | Streaming /mcp proxy | MCP HTTP | SSE/chunked | mcp_server.py, origin-shield header | Buffering would break MCP |
oauth-trial.js | Trial tokens | Turnstile / session | KV token:* | MCP verifier, graph-agent-proxy | Budget accounting dual-path with MCP |
billing.js + billing-session.js | Stripe + signed cookie | Checkout/webhooks | Entitlement | KV BILLING, Stripe secrets | Session cookie HMAC; tax_code product requirement (README) |
graph-pulse.js (and slice/neighborhood/atlas-api/code-library) | Fixed Cypher over Aura HTTP | Public GETs | Aggregates / graph JSON | syntology_web | Parameterisation discipline; cache TTLs |
correction.js | User corrections | POST | KV pending records | Human review | Must never auto-write Neo4j |
agent-telemetry.js | Token/auth events | Worker events | Analytics Engine | AE binding | Read token for AE query may be unprovisioned |
Ingestion
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
05_incremental_paper_sweep.py | Discovery | OpenReview vs graph | New-papers JSON | 06 | Interactive token (F-09) |
06_process_new_papers.py | Chain orchestrator | Discovery JSON, staging env | Staging dirs, subprocess products | Stages 01–04c, methods, loader | Chain itself vs per-stage tests; exit 3/4 continue |
01_grobid_extract.py / 09_binned_parse.py | PDF→TEI | PDFs, GROBID URL | TEI trees; 09 may delete PDFs per bin | Docker GROBID | Local SPOF (F-19); bin state files |
02_parse_tei.py | TEI→JSON + review queue | TEI | parsed JSON, _needs_review.json | review_queue.py | Queue without consumer historically loaded anyway (R4 lesson) |
02b_backfill_from_s2.py | Header repair | parsed JSON, S2 | *_backfilled/ | S2 client | Rate limits / quoted keys |
04_resolve_citations.py / 04c_resolve_citations_s2ag.py | Citation resolve | backfilled JSON | *_cited* | S2, acceptance gates | Identity spaces / stubs |
03_openreview_sync.py | Reviews ingest | OpenReview | reviews_synced.json | venue registry gate | Large artifact; anonymity rules |
acquire_*.py | Venue acquisition | Venue sites | metadata + PDFs | corpus dirs | Site HTML churn |
run_methods_full_corpus.py + apply_name_shape_veto.py | Method/dataset/concept extract | cited JSON | extraction JSON | Bedrock llama | Own-client ratchet (F-04) |
pipeline_stages.py | Hand-written flow description | n/a | Docs/scan input | scan_corpus_* | Can lag 07–22 (F-10) |
Graph / ledger
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
load_papers_into_neo4j.py | Primary paper loader | cited JSON, methods, reviews | MERGE graph writes | WriteLedger, review_queue_gate | Dual identity stubs; must use writer + intent |
groundwork/19_structured_provenance/write_ledger.py | Write ledger + lock | intent, session | write_ledger.jsonl, lock file | All bulk writers | Lives in experiments dir (F-08); tests write prod (F-05) |
16_load_s2_reference_edges.py | CITES gap-fill | S2 refs | CITES edges | S2 caches | Stub identity |
stream_paper_loader.py | Continuous TEI→graph | TEI dir, state JSON | Papers | launchd | State file; writer creds |
arxiv_ids.py | arXiv id shape | strings | bool / normalized ids | Many consumers | Three identity spaces |
provenance.py | Stamp derived files | payloads, inputs | stamped JSON / sidecars | QC ratchet | Unstamped writers still exist |
review_queue.py | R4 queue | parse flags | queue files | 02, 02b, 04, loader | Queue must have a consumer |
Code harvest
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
stream_harvest_follower.py | Continuous CodeSample harvest | classified repos, GitHub | CodeSample, state JSON | launchd KeepAlive | Continuous writer |
probe_harvested_contracts.py / synthesize_* | Contract probes | samples | probe reports | contract loop | GPU/Lambda paths exist separately |
assign_roles_harvested.py / attach_harvested_to_methods.py | Role + method attach | harvest graph | IMPLEMENTS / HAS_HARVESTED_IMPL | graph_write_guard | Role downgrade guard |
groundwork/13_impl_generation_verification/ | V-ladder generation | papers, Bedrock | reports, HAS_REFERENCE_IMPL | loaders in 14 | Huge artifact tree in git |
groundwork/14_code_ontology_surface/graph_write_guard.py | Refuse regressions | proposed writes | allow/deny | loaders | Path coupling |
Frontend / Atlas
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
public/index.html et al. | Marketing site | ASSETS | HTML | graph-pulse widgets | Illustrative vs live confusion |
public/llms.txt | Agent-facing truth | n/a | Policy/API map | developers.html | Can lag live graph coverage |
public/app/ | Atlas map | atlas-api, local data | WebGL UI | atlas_build.py | data/ gitignored; large bundles |
atlas-api.js | Atlas Cypher | HTTP | JSON | Aura HTTP | Auth/rate at Worker |
Modular kernel (if present)
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
syntology/ | Additive kernel | env, graph, corpus | doctor/qc/parity/census | Re-exports ONE-way modules | Not production until promotion |
syntology/serving/ | Strangler routers | same HTTP | subset of routes | parity harness | Dual tables until main.py switched |
syntology/config.py | One env parser | .env path | redacted explain() | all future callers | Admin password naming |
Ops
| Path | Role | Receives | Ships | Interacts with | Risks |
|---|---|---|---|---|---|
deploy/deploy_api_to_aws.sh | App Runner deploy | Docker context | service update | ECR, secrets | IAM grants; ONNX staging |
joblog.sh | Job registry sidecar | wrapped command | logs/*.status | QC dropped-jobs | Wrapper pid vs child; JOB_DETACH |
backup_corpus_to_s3.sh | Nightly corpus sync | local dirs | S3 prefixes | launchd 03:00 | Must fail if live dirs absent (F-01 class) |
ensure_grobid.sh | Local GROBID | Docker | TEI service | parse stages | RestartPolicy historically none |
ops_dashboard/ | Local monitor | cloud APIs, logs | localhost:8790 | .env source | Read-only intent; broad env source |
qc_*.py | Executable checks | repo + optional graph | exit codes | session protocol | Ratchets can be red by design |
Root docs (trust ranking)
See Reading order. Do not treat CURRENT_STATUS.md as
the first-read for “what is still true.”