25:[["$","p",null,{"children":"Reading-first knowledge management system with semantic search, persistent anchoring, and LLM-augmented conversation. Built as the week 3 group project at Fractal with Nyan and Jahnik."}],"\n",["$","h2",null,{"id":"what-i-built","children":"what I built"}],"\n",["$","p",null,{"children":["A complete document management and semantic search system built as a production specification. You upload documents (PDF, EPUB, HTML), highlight text, and search across everything you've saved using AI-powered similarity search. The core innovation is ",["$","strong",null,{"children":"persistent anchoring"}],"—highlights use byte offsets into canonical text that survive document re-ingestion via fuzzy text matching, so your annotations never get lost when you update a file."]}],"\n",["$","p",null,{"children":["The system enforces privacy-first design: all data is indexed by user ID, search results respect ACLs, and every resource uses prefixed typed IDs (",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"doc_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"hl_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"conv_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"msg_"}}],", etc.) for type safety at the API boundary."]}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"What it does:"}]}],"\n",["$","ol",null,{"children":["\n",["$","li",null,{"children":"Upload documents → deterministic canonical text extraction (same file always produces identical UTF-8 output)"}],"\n",["$","li",null,{"children":"Highlight & annotate → anchors stored as byte offsets into canonical text that survive re-extraction via fuzzy text matching"}],"\n",["$","li",null,{"children":"Semantic search → hybrid BM25 + pgvector-powered similarity search with ACL enforcement"}],"\n",["$","li",null,{"children":"Chat with documents → LLM conversations with RAG pipeline that retrieves relevant chunks and injects context (V1.0 complete)"}],"\n"]}],"\n",["$","h2",null,{"id":"tech-stack","children":"tech stack"}],"\n",["$","ul",null,{"children":["\n",["$","li",null,{"children":[["$","strong",null,{"children":"Backend"}],": Python 3.11+, FastAPI, SQLAlchemy 2.0, Alembic migrations"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Frontend"}],": Next.js 14, React 18, TypeScript, TanStack Query"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Database"}],": PostgreSQL 15+ with pgvector"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Job Queue"}],": Redis + Celery for async document processing"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Auth"}],": Clerk (OIDC, JWTs)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Embeddings"}],": OpenAI ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"text-embedding-3-small"}}]]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Package Managers"}],": uv (Python), pnpm (JavaScript)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Infrastructure"}],": Docker Compose for local development"]}],"\n"]}],"\n",["$","h2",null,{"id":"current-state","children":"current state"}],"\n",["$","p",null,{"children":"$26"}],"\n",["$","h2",null,{"id":"architecture","children":"architecture"}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"Core Design Principles"}]}],"\n",["$","ol",null,{"children":["\n",["$","li",null,{"children":["\n",["$","p",null,{"children":[["$","strong",null,{"children":"Canonical Text"}]," - Deterministic UTF-8 transformation of PDFs, EPUBs, HTML into immutable byte arrays with stable offsets. Same file always produces identical text."]}],"\n"]}],"\n",["$","li",null,{"children":["\n",["$","p",null,{"children":[["$","strong",null,{"children":"Persistent Anchoring"}]," - Highlights store byte offsets into canonical text (not original file). When documents are re-ingested, the system detects hash changes and remaps anchors via fuzzy text matching (up to 10% edit distance)."]}],"\n"]}],"\n",["$","li",null,{"children":["\n",["$","p",null,{"children":[["$","strong",null,{"children":"Privacy Enforcement"}]," - All access control derives from ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"Visible(U, O)"}}]," predicate. Every database query filters by user_id. Unknown or forbidden resources return 404 (not 403) to prevent existence leaks."]}],"\n"]}],"\n",["$","li",null,{"children":["\n",["$","p",null,{"children":[["$","strong",null,{"children":"Typed IDs"}]," - All external API resources use prefixed IDs (",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"doc_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"hl_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"conv_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"msg_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"cnk_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"ann_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"rdr_"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"lnk_"}}],") for type safety."]}],"\n"]}],"\n",["$","li",null,{"children":["\n",["$","p",null,{"children":[["$","strong",null,{"children":"Async Ingestion"}]," - Redis + Celery handles background jobs: text extraction, chunking, embedding generation, anchor remapping on re-ingestion."]}],"\n"]}],"\n"]}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"Pipeline Stages"}]}],"\n",["$","p",null,{"children":"| Stage | Component | Status |\n|-------|-----------|--------|\n| 1. Ingest | Document upload, file storage, media detection | ✅ Complete |\n| 2. Extract | Canonical text extraction (PDF/EPUB/HTML) | ✅ Complete |\n| 3. Chunk | Paragraph-aware chunking with byte offsets | ✅ Complete |\n| 4. Embed | OpenAI embeddings + pgvector storage | ✅ Complete |\n| 5. Search | Hybrid BM25 + vector similarity | ✅ Complete |\n| 6. Retrieve | ACL-filtered chunk retrieval | ✅ Complete |\n| 7. Chat | LLM with RAG-assembled context | ✅ Complete |"}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"API Overview"}]}],"\n",["$","p",null,{"children":["All endpoints require Clerk JWT (except ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"/health"}}],"). Key endpoints:"]}],"\n",["$","ul",null,{"children":["\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"POST /documents"}}]," - Upload document (triggers async ingestion pipeline)"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"GET /documents"}}]," - List documents (paginated)"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"GET /documents/{id}/chunks"}}]," - List document chunks with embeddings"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"POST /highlights"}}]," - Create highlight (byte-offset anchor into canonical text)"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"POST /search"}}]," - Hybrid search (BM25 + vector, ACL-filtered)"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"GET /conversations"}}],", ",["$","code",null,{"dangerouslySetInnerHTML":{"__html":"POST /conversations/{id}/messages"}}]," - Chat threads with RAG context"]}],"\n",["$","li",null,{"children":[["$","code",null,{"dangerouslySetInnerHTML":{"__html":"GET /health"}}]," - Health check (no auth required)"]}],"\n"]}],"\n",["$","h2",null,{"id":"roadmap","children":"roadmap"}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"Phase 1 (V1.0 – Complete)"}]}],"\n",["$","p",null,{"children":"24 PRs across 8 stages with detailed spec references:"}],"\n",["$","ul",null,{"children":["\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 0"}],": Bootstrap (repo, CI, FastAPI skeleton)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 1"}],": Backend infrastructure (schema, typed IDs, error envelopes, pagination)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 2"}],": Users + Auth (Clerk OIDC, rate limiting, security headers)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 3"}],": Documents + Ingestion (upload, text extraction, chunking)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 4"}],": Frontend shell (React app, auth wiring, OpenAPI client)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 5"}],": Reading + Highlighting (reader sessions, highlights, annotations, document UI)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 6"}],": Retrieval + Embeddings (pgvector, BM25 indexing, hybrid search)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 7"}],": Conversations + LLM (chat threads, LLM integration, RAG pipeline, vertical slice)"]}],"\n",["$","li",null,{"children":[["$","strong",null,{"children":"Stage 8"}],": Links + Hardening (document linking, comprehensive tests, production readiness)"]}],"\n"]}],"\n",["$","p",null,{"children":["$","strong",null,{"children":"V1.1+ (Future)"}]}],"\n",["$","ul",null,{"children":["\n",["$","li",null,{"children":"Multi-tab document interface"}],"\n",["$","li",null,{"children":"Evergreen linked-notes panel"}],"\n",["$","li",null,{"children":"Global conversations (currently per-document)"}],"\n",["$","li",null,{"children":"Linked-object browser"}],"\n",["$","li",null,{"children":"Podcast/video support with transcripts"}],"\n",["$","li",null,{"children":"Typed links (v2; currently untyped/symmetric)"}],"\n",["$","li",null,{"children":"Mobile native apps (React Native)"}],"\n",["$","li",null,{"children":"Advanced summaries and integrations"}],"\n"]}]]