Building a RAG Investment Assistant

A venture fund’s edge is knowing things: who a founder is, what a company does, what was said in the last few meetings with them. That knowledge is scattered across PDFs, inboxes, call transcripts, and news feeds, and none of it is where you need it when a call starts in ten minutes. We built a system for a New York-based VC fund (we’ll call it the NY VC fund) that pulls all of that into one searchable knowledge base and puts LLM agents on top of it to prepare meeting briefings, research people and companies, and answer questions across everything the firm knows.

It shipped in two increments (Phase 1 around the middle of 2025, Phase 1.1 later in the year), ran in production for real users, and is now in the client’s hands to operate and grow. In this post, we’ll walk through how it’s built: the storage design that makes retrieval work, the seven-stage ingestion pipeline, the agent layer (and how we tore it up and rebuilt it between phases), the Airflow automations, and a production incident that taught us to take monitoring more seriously than we had.

Where it started, and what changed

This one started from the client’s side. The fund came to us with the RAG idea already formed and a fairly specific set of prerequisites for the stack: LlamaCloud, S3, SQL, Pinecone, Elastic, Neo4j, Temporal, Airflow, and a React frontend. We took those requirements and scoped them into a fixed-scope MVP, with the first stage framed explicitly as a feasibility check: prove out each step, find the complications early, and produce a roadmap and firmer estimates for the rest.

While scoping it, we flagged a call we’d end up leaning on hard: that “changing the whole pipeline to a working LLM agents network will be much more effective than using a single LLM for everything,” though we deferred that agent-network approach as too time-consuming for an MVP. We shelved it for Phase 1 and came back to it in Phase 1.1.
A few pieces of the stack the client had in mind didn’t survive contact with the actual work, which is what a feasibility phase is for. Temporal got dropped: Airflow turned out to handle the orchestration on its own, so a second orchestrator was dead weight. LlamaCloud got replaced with OpenRouter as the LLM gateway, for convenience. And the client’s React MVP gave way to a Streamlit app for the working product, with a separate design-led “Frontend Demo” carrying the longer-term UX vision.

The storage layer

The system is organized into five layers: a Streamlit web app (backed by PostgreSQL via SQLAlchemy, with Yoyo for migrations), a data-processing layer, an AI agents layer, an AI requests layer that routes every model call through OpenRouter and logs it in PromptLayer, and the storage layer. The storage layer is where the RAG design lives, because we don’t rely on any single index. Four systems run in parallel, each doing what it’s best at:

• S3 is the raw-document store and ground truth. Buckets separate content by type: raw PDFs, transcripts, HTML, plain text, markdown, and a combined RSS-and-email bucket.
• Pinecone holds the vector embeddings for semantic search. This is the part that lets a query about “financial problems” surface documents that talk about “losses,” “bankruptcy,” or “revenue decline” without those exact words appearing. The production index runs at dimension 1024 with cosine distance; at one point it held 29,754 vectors in a namespace called final.
• OpenSearch does exact keyword and metadata filtering (dates, companies, specific fields), which is what vector search is bad at. The two complement each other for hybrid retrieval.
• Neo4j stores an entity-relationship graph: companies, people, experts, themes, technologies, and markets, connected to the documents that mention them by MENTIONED_IN edges. That’s how you answer “who is connected to what.”

Redis caches expensive results (LLM inference, enrichment, repeated lookups), and PostgreSQL holds application data plus the deduplication bookkeeping we’ll get to in the automation section.

Turning documents into a knowledge base

Every document (PDF, Office file, markdown, HTML, email, RSS article, or audio) goes through a seven-stage pipeline before it’s queryable. File-type detection routes each input first: markdown and HTML through Unstructured.io, plain text raw, PDFs and Office formats through LlamaParse (LlamaCloud plus a vision model), and emails through a dedicated parser that strips “Re:” chains.

1. Parsing. PDFs and Office documents go through LlamaCloud with a vision model, Anthropic Claude Sonnet 3.5, where each page is sent as a screenshot. That’s what lets charts and graphs come out as structured HTML tables with generated summaries, while text keeps page-level metadata.
2. Table serialization. This was the standout win. Naïve chunking splits a large table across chunks and loses its headers and the numeric relationships that make the numbers mean anything. So we used GPT-4o-mini to convert table rows into self-contained text blocks, each carrying its headers, units and currencies, title and footnotes, and a window of surrounding context. The payoff showed up in the numbers: retrieval accuracy on table-related questions went from ~70% to ~94% (at least 4% above the industry standard), and for chart-related questions the serialized tables pushed answer accuracy to ~95%.
3. Chunking. A chunk_by_title strategy starts new chunks at section headers, with a target size, a hard cap, and some overlap between chunks so context doesn’t fall off a cliff between them.
4. Metadata extraction. Google Gemini 2.0 Flash Lite estimates publication dates, summarizes each document, and extracts entities (companies, experts, themes, technologies, markets), deduplicating and linking them and inferring relationships as it goes.
5. Vector store upload. Enriched chunks are embedded with llama-text-embed-v2 in batches and written to Pinecone with their metadata.
6. Knowledge graph upload. Entities and documents become nodes in Neo4j, joined by MENTIONED_IN edges.
7. Search index upload. Full text plus metadata lands in OpenSearch for Boolean and faceted keyword search.

Two more ingestion routes feed the same pipeline: an RSS scraper that polls feeds, pulls article text with BeautifulSoup, and drops JSON into S3; and audio transcription through the Deepgram API for recordings.

The agent layer, and rebuilding it mid-project

In Phase 1 we built the agent layer the obvious way: a class hierarchy. A base RAGAgent handled knowledge-graph access, LLM-provider abstraction, function calling, and conversation management, and specialized agents inherited from it. A Dossier Agent produced structured reports on companies, people, and themes. A Call Preparation Agent combined Neo4j, Outlook, and Perplexity to research external attendees and generate briefings, filtering out the fund’s own internal participants, since you don’t want to “research” your own colleagues. A Deep Research module scored each document for relevance from 0.0 to 1.0 with google/gemini-2.5-flash, with the heavy web research handled by perplexity/sonar-deep-research.

It worked, but it had a cost that showed up fast: every new agent type meant new Python code and a redeploy. Want to tweak how the company-research agent reasons? Edit code, ship it. That’s fine for engineers and a wall for everyone else. And the people who knew what a good briefing looked like were the fund’s own staff, while the code stayed with us.

So in Phase 1.1 we threw out the inheritance hierarchy and replaced it with a single universal PromptLayerAgent driven by configurable prompt templates. This is where that deferred “agents network” idea from the original proposal finally landed. Agent behavior is now defined entirely by a template (edited in the PromptLayer UI, no code changes, no redeploy), made of a system prompt (the role), a user prompt (the task, with {{variable}} placeholders), context calls (deterministic retrieval that runs before the model reasons), and available functions (the tools the model can call).

Execution runs in two phases. First, context preparation runs the configured knowledge-graph and document searches and caches the results before the LLM sees the query. Then conversational execution lets the model reason and call functions dynamically (a capped number of function calls per conversation) before it answers. The split matters: context calls are pre-configured and always run in the same order, while function calls are the model’s own runtime decisions. A variable-injection system threads it all together: input variables, context results (ctx_1_result_…), and function-call results (fc_1_result_…) are all available to the template.

The reworked layer exposes a library of handles: the functions an agent can call. They cover knowledge-base search (search_text_in_local_vectors and a filtered variant over Pinecone; search_entity_in_local_graph and a hybrid graph-plus-vector search over Neo4j and Pinecone; check_knowledge_graph_has_person / _has_company to skip redundant external research when we already know someone), Outlook access (get_emails, get_calendar_events, get_emails_for_attendees, and a non-AI statistical get_outlook_interactions_analysis), web research via Perplexity through OpenRouter, and agent orchestration: run_simple_agent for isolated calls, run_chain_of_agents for deterministic sequential pipelines where each step can reference {{agent{n}_output}}. The main handle, prepare_call_briefing, still runs the whole meeting-prep sequence and still refuses to research the fund’s own internal-domain addresses.

Automation: the Airflow DAGs

Everything runs on a schedule through Apache Airflow, each DAG pinned to a single active run to avoid overlap:

• s3_document_pipeline (on a short cycle, with automatic retries) discovers new files and pushes them through parsing to a parallel upload into Pinecone, Neo4j, and OpenSearch. Failures go into a failed_documents table so one bad file doesn’t stall the whole pipeline.
• email_polling_pipeline (on a regular cycle) pulls a batch of recent emails from each Microsoft 365 mailbox, using an “inbox pattern” (a PostgreSQL processed_emails table keyed on the RFC 5322 message id) for exactly-once processing.
• rss_content_scraper (on a short cycle) health-checks the feed server, scrapes a capped batch of the most recent entries, and stores them to S3.
• calendar_briefing_pipeline (on a short cycle) looks a short window ahead across the fund’s calendars, deduplicates events on their ical_uid (RFC 5545), generates a briefing with the Call Preparation Agent, and emails it over SMTP, but only to the fund’s own attendees.

Phase 1.1 added a Transcripts Processing DAG so meeting transcripts became a first-class source alongside documents, emails, and RSS. It scans the transcripts bucket for files newer than a mid-2025 cutoff, pulls the meeting title, Calendar Event ID, and participant emails from each transcript header, enriches them through the Microsoft Graph API, deduplicates with the same inbox pattern (keyed on the Graph iCalUId where available), and writes structured JSON back to S3 in the same shape as emails and RSS. Then a set of querying tools combines PostgreSQL metadata lookups with OpenSearch full-text search so agents can pull transcript context by event, participant, date, or subject.

Prompt experimentation and PromptLayer

The defining feature of Phase 1.1 was letting the fund’s own people design, test, and deploy prompts without an engineer in the loop. Two UI pages carry it: a Prompt Experiment page (admins only) that auto-detects variables in prompt text, lets you configure context calls, and builds test datasets from the Cartesian product of variable values; and a Prompt Execution page (open to everyone) that runs read-only production prompts from a final folder with variable injection and validation. Both persist session state through Redis, so a page reload doesn’t wipe your work.

Underneath is a bidirectional integration with PromptLayer. Uploading turns a local run into a reusable template (system and user text, detected variables, context-call configuration, model parameters, function definitions) that you then refine in the PromptLayer web UI. Downloading pulls a refined template straight into production with no code deploy, and every run is logged with its full conversation, function calls, variable values, and timing. Prompts live in folders (drafts, final, evaluations, datasets), and the documented workflow walks a user from a draft, through experimentation, to moving a finished prompt into final for execution. Templates are split into system-level parts and user-editable sections, so people get controlled customization without being able to break production behavior.

Two faces: WebApp and Frontend Demo

There are two interfaces on the same backend. The WebApp is the working Streamlit app used for real operations: a two-level login (Azure for the fund’s Microsoft accounts or Google for Serokell accounts, then app credentials), then role-gated tabs. Admins get Calendar, Deep Research, Vector Query, RSS Feeds, Change Password, User Management, Storage Manager, and an Audio transcriber; staff get the search-and-agent subset and see only their own email data. The Vector Query tab is where the hybrid retrieval shows through most plainly: a “soft” semantic query against Pinecone plus “strict” must-contain/exclude/filter windows enforced through OpenSearch, with wildcard support. One example returned 500 Pinecone hits, 492 of which OpenSearch filtered out, leaving 8 documents. Deep Research runs through the Perplexity API via OpenRouter, and a saved report takes up to 15 minutes to show up in the knowledge base; the Storage Manager caps uploads at 200 MB per file.

The Frontend Demo is the forward-looking product vision, rebuilt on a relational database to interlink its sections: a dashboard, decision points, three feeds (Activity, Signal, News), a calendar, research themes on a Kanban board, deals, portfolio tracking, industry value-chain graphs, and a network view. It’s honest about its seams: a lot of it runs on mock or partially-integrated v0 data, there’s no authentication yet (so bookmarking and the internal-vs-external contact distinction don’t work), the news-feed sentiment indicators are randomly assigned and non-functional, and several dashboard panels show placeholders instead of live data. An Admin Panel (append /admin) was built beyond the original scope to manage the standardized data model directly, but it performs raw, unvalidated SQL-level operations, so it’s flagged for cautious internal use only. Some things came out along the way based on client feedback, including the deal agents and the portfolio News/History/Agents subtabs.

What running it in production taught us

This wasn’t a demo. Real users at the fund were relying on it, which changes the stakes, and we found that out the uncomfortable way. At one point Airflow had been unhealthy for more than 20 days before anyone noticed: a worker restart from an out-of-memory condition left a DAG stuck, the scheduler never marked the task as failed, and the health endpoint we had didn’t report the problem. The DAG processor was hitting a known upstream Airflow bug, issue #55029.

The fix was both immediate and structural. We upgraded Airflow to 3.1.0, added StatsD-to-CloudWatch metric collection, set sane DAG run timeouts, and wired up real alerts: the Airflow health endpoint and Celery Flower workers polled frequently, with DAG failures pushed to an SNS-backed feed channel. Internally we reclassified this from “technical debt” to “crucial,” because the client was live on the app. Monitoring you can silently lose for three weeks isn’t monitoring.

Other production realities kept us honest: Microsoft/Outlook OAuth tokens and client secrets expiring and needing rotation; the Neo4j service occasionally going unavailable, with the agent falling back to “general knowledge” when it did; a client-reported bug where an S3 key was captured at the top level of a result dict but not inside the metadata written to Pinecone, so matched chunks couldn’t be traced back to their source document; and Terraform state drift because the client was editing RAG-related AWS resources directly. To keep production stable while we kept shipping, Phase 1.1 formally separated dev and live environments with independent deployments: CI/CD auto-deployed the main branch to dev, and the live instance was updated separately. The whole stack runs on AWS (managed OpenSearch and PostgreSQL, ECS for the apps and Airflow), with Airflow using git-sync to pull DAGs, and we put Airflow’s metadata database on an automated pruning schedule to cut storage cost and speed up the UI.

Honest limitations

A few things this version doesn’t do, stated plainly. Email history is processed at roughly 1,000 emails per mailbox (for busy inboxes that’s only the last three to four weeks), and multi-mailbox or group accounts aren’t supported (one mailbox equals one account). Email data is isolated per user (admins aside), so sharing another user’s email context means manually forwarding it. Some advanced agent handles (conditioning, group-by, for-every) aren’t implemented, though you can simulate them with multi-agent calls; handles that would write new data into the knowledge base were left out on security grounds. The Streamlit frontend was never meant to be production-scale, and it got more restrictive and resource-heavy the further we pushed it. A proper UI is on the list. And the Airflow DAGs consume too much RAM because they’re written in a straightforward, non-idiomatic style, observability is thinner than we’d like, and the Python code is untyped to move faster. All of that is on the table for later.

Where it goes next

The roadmap is a natural continuation of what’s built, and it leans into where AI engineering has moved lately: more agentic, tool-driven, and composable. The handles are already a tool-calling layer in all but name, so the obvious next step is to expose them as first-class, MCP-style tools that any agent (ours or the client’s) can discover and call over a standard protocol, instead of being wired in by hand. The template-driven PromptLayerAgent points the same direction: packaged, versioned prompt-and-tool bundles become reusable agent skills for recurring jobs like call prep, company diligence, or thematic research, and the two-phase execution model extends into more autonomous agentic loops that plan multi-step research across a whole list of entities and summarize hierarchically to stay within context.

The rest of the near-term list comes straight from the delivered system’s own roadmaps: real role management and access control down to the knowledge-graph and chunk level; graduating from the Streamlit prototype to a proper UI and a config-driven control panel that assembles the right tools automatically; leaner, idiomatic Airflow pipelines with real observability; data-driven agent experimentation, fine-tuning, and degradation monitoring for data drift; a signals-and-themes system with its own UI; richer graph visualization with “power node” highlighting; and new agents like an Email Drafter, Domain Expert Finder, Connection Analyzer, and Timeline Builder. The foundation (hybrid retrieval, a clean tool layer, and a full prompt lifecycle) is exactly the kind of base that gets stronger as the agentic tooling around it matures. We’re keeping at it.