Multimodal document RAG

What you'll build

Step-by-step build

1. 1

   Split the document by content type

   Use PyMuPDF or unstructured to walk each PDF page and emit typed blocks: text runs, table regions, and image regions with their bounding boxes and page numbers. Save cropped images to disk keyed by page. You now have a stream you can route by type instead of a flat wall of text.

2. 2

   Describe the visuals

   For each image and chart, prompt the vision-language model to describe exactly what it shows, including axis labels, trends, and any numbers it can read, and to avoid guessing. Store that description as the searchable text for that figure, keeping a link back to the crop and page.

3. 3

   Linearize the tables

   Convert each detected table into markdown with headers preserved so rows and columns stay aligned. A well-formatted markdown table is something the LLM reads reliably, unlike a jumble of stray cell text. Keep the page number with it.

4. 4

   Embed everything into one space

   Embed the text passages, the image descriptions, and the markdown tables with the same model and store each as { text, embedding, page, modality, cropPath }. Tagging modality lets you later see whether an answer came from prose, a table, or a figure.

5. 5

   Retrieve across modalities

   On a query, embed it and vector-search the whole collection so a chart description can outrank a paragraph when it is more relevant. Return the top chunks with their modality and page so the next step and the UI both know what kind of evidence each is.

6. 6

   Compose the grounded answer

   Send the retrieved mixed-modality chunks to Groq with a prompt that says to answer only from the provided evidence, cite the page, and note when the answer comes from a described figure. This keeps you honest about evidence that was itself model-generated at ingest.

7. 7

   Surface the evidence

   In the UI, render the answer and beneath it the cited chunks, showing the actual cropped image for figure citations and the markdown for table citations. Seeing the source visual next to the claim is what makes multimodal RAG trustworthy.

8. 8

   Test on a chart-heavy document

   Pick a report where key answers live only in charts and tables and verify the system gets them, then confirm a text-only baseline fails the same questions. That contrast is the whole point of the project and your best demo.

Done when

• ✓A question whose answer exists only in a chart is answered correctly and cites that figure's page.
• ✓A text-only RAG baseline fails several questions that your multimodal pipeline gets right.
• ✓Table-based numeric questions return the right value from the linearized table, not a hallucination.
• ✓Every answer names the page and the modality of its evidence, and figure citations show the real crop.
• ✓A brand-new chart-heavy PDF works end to end with no code changes.

Ship it

Dockerize the FastAPI service with the parser and vision model, and deploy to Hugging Face Spaces or AWS free tier with an Atlas M0 store. Ship a live URL and a README that shows one question answered from a chart with the cropped figure beside it, next to a text-only baseline getting it wrong. That single before-and-after sells the whole project.

What it proves: You can build RAG over the messy, visual documents that real companies actually have, routing content by type, using a vision model to make figures and tables searchable, and grounding answers in mixed-modality evidence with honest citations.

Paste this into Cursor, Claude, or ChatGPT and build it step by step.

You are my senior AI engineer pair. Help me build "Multimodal document RAG" step by step. RAG means retrieval-augmented generation: fetch relevant evidence and make the LLM answer only from it. Multimodal here means the evidence includes images, charts, and tables, not just text; a vision-language model (VLM) reads each visual and writes a text description we can index. Stack: FastAPI, PyMuPDF or unstructured for PDF layout parsing, a free vision-language model (Llama 3.2 Vision) for figure descriptions, all-MiniLM-L6-v2 embeddings, MongoDB Atlas Vector Search (free M0) with a modality tag, Groq (Llama 3.3 70B) for the grounded answer.

Requirements:
1. Parse each PDF into typed blocks: text, tables, images, each with a page number.
2. Describe every image/chart with the VLM and linearize tables to markdown.
3. Embed text, descriptions, and tables into one store as { text, embedding, page, modality, cropPath }.
4. Retrieve across all modalities and answer only from the evidence with page + modality citations.
5. UI that shows the cited crop for figure answers and the markdown for table answers.

Work in this order: PDF splitting by type, then visual description, then table linearization, then unified embedding, then cross-modal retrieval, then the grounded answer, then the evidence UI. Give me the commands and code for each step and STOP after each so I can test. Do not write the whole app at once.