Syntology Docs

Ingestion — canebrake pipeline

Local Mac runtime. Corpus is outside git. Stages are scripts; orchestration for new-paper batches is 06_process_new_papers.py (subprocess per validated stage).

Core chain (new papers)

  1. Discover — 05_incremental_paper_sweep.py (OpenReview accepted pool vs graph). Needs a fresh OPENREVIEW_TOKEN via interactive get_openreview_token.py (F-09 / launchd friction).
  2. Orchestrate — 06_process_new_papers.py: download → GROBID → 02_parse_tei → 02b_backfill_from_s2 → citation resolve (04/04c) → methods extract → name-shape veto → load_papers_into_neo4j.py. Flags for dry-run load / skip-load; WriteLedger intent on load.
  3. Review gate — 02 writes _needs_review.json; loader review_queue_gate() refuses flagged papers (R4).

Bulk / venue acquisition

comprehensive_paper_acquisition.py (Phase 1 NeurIPS/ICML/ICLR), acquire_vision.py, acquire_language.py, acquire_aaai_ijcai.py, acquire_frontier_labs.py, acquire_landmark_papers.py, plus sweeps 07_arxiv_year_sweep, 10_arxiv_daily_delta, 11_frontier_lab_watch, 12–14 venue metadata/PDF manifests.

GROBID

Local Docker (ensure_grobid.sh). Bulk path 09_binned_parse.py (bin state in bins_done.json). Cloud GROBID App Runner historically paused; origin exposure addressed in the 2026-08-21/22 incident follow-up. Single local container is a throughput SPOF (F-19).

Shared helpers

provenance.py, review_queue.py, corpus_files.py (AppleDouble-safe), s2_api_key.py / s2_backoff.py, arxiv_ids.py, venue_review_platform_registry.py, bedrock_client.py, joblog.sh.

Scheduling (examples)

launchd labels under ai.syntology.*: arxiv-daily, frontier-lab-watch, venue-decision-watch, corpus-backup, qc-session, embeddings, incremental-sweep, pipeline-scan, mcp-discovery-check, stream-harvest, contract-loop — see architecture review §9 when available on your tree.

Gaps

  • Stage numbering after 06 is chronological, not topological (F-10).
  • pipeline_stages.py can lag the real chain; 06 / syntology.stages are the executable truths on fresher trees.
  • Discovery depends on manual OpenReview token refresh.