Guide

HTML to Markdown for RAG

A RAG pipeline is only as clean as the source material it loads. Cleaning web pages into Markdown before chunking gives you a source that is easier to inspect, easier to debug, and less likely to carry browser chrome into retrieval.

Updated May 7, 2026

Convert a source page before ingestion Inspect Markdown chunks Back to guides

Markdown keeps structure without page clutterCleaner sources are easier to inspect before retrievalPlain text is useful, but not always enough

Short answer

Convert HTML to Markdown before RAG when page chrome would pollute chunks.

Clean HTML to Markdown before RAG when the source has browser chrome that would pollute chunks. Raw HTML often carries navigation, layout wrappers, repeated links, scripts, and UI text into the retrieval store. Cleaned Markdown keeps headings, lists, links, tables, and code easier to inspect before embedding.

Core idea

RAG pipelines usually want semantic structure, not front-end scaffolding.

Most public pages are built for browsers, not for retrieval systems. That means the useful article or docs body often arrives wrapped in navigation, layout containers, tracking hooks, UI chrome, and repeated calls to action. Markdown is often a better storage or inspection format because it preserves the content shape while dropping most of the extra page furniture.

Less layout noise

Raw HTML carries wrappers, classes, scripts, navigation, footers, and other page furniture that often adds tokens without adding retrieval value.

Structure still survives

Markdown keeps the pieces that usually matter for retrieval and QA: headings, lists, tables, code blocks, links, and readable section boundaries.

Cleaner downstream handling

A lighter intermediate format is easier to inspect, chunk, store, and reuse across agent pipelines, retrieval prep, and prompt assembly.

Ingestion path

Clean first, load second.

Paepae Stack fits before your loader, splitter, or embeddings layer. The point is not to replace the rest of the stack. The point is to improve the source shape before it becomes chunks you have to trust.

Suggested flow

public docs page or copied HTML
-> Paepae Stack cleanup
-> inspected Markdown
-> document loader
-> splitter or ingestion pipeline
-> embeddings or retrieval store

Format choice

HTML, Markdown, and plain text each solve a different problem.

The best format depends on what the next step needs. The mistake is assuming the raw browser page is automatically the best input for every downstream model or retrieval workflow.

Raw HTML

Best when you need the original DOM or want to preserve page-level implementation detail. Usually too noisy for direct RAG ingestion without cleanup.

Markdown

Usually the best middle ground when you want semantic structure without the bulk of page chrome and front-end scaffolding.

Plain text

Useful when only the words matter, but it often flattens headings, code, lists, and table boundaries that help both retrieval and human QA.

Source format	Retrieval impact	What can break	Recommended step
Raw HTML	High noise unless the page is already minimal.	Navigation, repeated links, scripts, wrappers, and hidden layout text become chunks.	Clean to Markdown before loading into a splitter or document store.
Cleaned Markdown	Often a practical balance of readable structure and compact context.	Dense pages can still produce oversized or context-light chunks.	Inspect chunk boundaries before embedding or exporting JSONL records.
Plain text	Compact, but less structured for QA and source attribution.	Headings, tables, code fences, and list boundaries can disappear.	Use only after checking that structure no longer helps retrieval.

Workflow

A practical cleanup path before chunking or embedding.

Treat cleanup as a staging step. The stored source is easier to debug when it is already compact, legible, and inspectable during QA.

Step 1

Fetch or paste the page content before it enters the retrieval pipeline.

Step 2

Strip layout chrome and non-content blocks so the stored material is mostly semantic signal.

Step 3

Preserve readable headings, lists, links, tables, and code blocks in Markdown.

Step 4

Chunk or embed the cleaned output only after the source is compact and structurally legible.

Example handoff

Keep metadata close to the cleaned source.

Source URL, title, and capture date help with QA later. They also make it easier to refresh stale pages and understand where a retrieval hit originally came from.

Example Markdown document

---
source_url: https://example.com/docs/widget-api
title: Widget API
captured_at: 2026-04-30
prepared_with: Paepae Stack HTML to Markdown for AI
---

# Widget API

## Authentication

Use API keys for server-side requests only.

## Rate limits

- 120 requests per minute
- Burst behavior may vary by plan

## Common errors

- 401 when the API key is missing
- 429 when the request limit is exceeded

Common mistakes

Most retrieval mess starts with source formatting, not with embeddings.

When retrieval output feels vague or cluttered, the root problem is often that the stored source material is bloated, flattened, or harder to inspect than it needs to be.

Embedding the whole page shell

If the stored text includes navigation, repeated CTAs, and boilerplate chrome, retrieval results become noisier before the model even answers anything.

Flattening everything to plain text too early

If headings and section breaks disappear, retrieval chunks become harder to label, inspect, and debug later.

Assuming every URL fetch is browser-perfect

Some sites hide the useful content behind client-side rendering, login walls, or anti-bot interstitials. The cleanup layer should make those limits visible.

What to test

Trust the pipeline only after you inspect the retrieval outcomes.

The goal is not just smaller input. The goal is source material that is easier to inspect before chunking, embedding, and retrieval QA.

Quality check

Do top retrieval hits quote the article body instead of navigation or footer text?

Quality check

Do chunks begin under meaningful headings instead of random wrappers?

Quality check

Did code blocks, tables, or ordered steps survive in a usable form?

Quality check

Can a human skim the cleaned Markdown and understand the source immediately?

Next steps

Keep the RAG path moving with the next useful page.

The strongest follow-up from here is either to run the cleanup, move into the developer-ingestion branch, or validate the output against examples before you scale the pipeline.

Run the cleanup before ingestion

Convert the public page or copied HTML before it becomes chunks, embeddings, or agent context.

Open HTML to Markdown for AI

Move into the developer ingestion branch

Use the LangChain and LlamaIndex guide when you want this same cleanup path framed around loaders, metadata, and early retrieval QA.

Read the ingestion guide

Inspect the chunks before embedding

Use the Markdown Chunk Inspector when the cleaned source needs token estimates, heading context checks, and JSONL-ready records.

Open chunk inspector

Compare chunking examples

Use the examples guide when you want to see heading, token-window, paragraph-safe, and code/table-safe chunking on the same Markdown source.

View chunking examples

Check cleanup examples

See before-and-after examples when you want a clearer sense of what survives cleanup across docs, support, blog, and wiki-style pages.

View cleanup examples

Decide how much structure to keep

Compare Markdown and plain text when you need to know whether the cleaned source should stay structured after retrieval prep.

Compare Markdown and plain text