RAG chunk staging
Paste cleaned Markdown and inspect how it splits before embedding, retrieval, or agent-context storage.
Input
Start chunks at Markdown headings, then split long sections conservatively.
Use Tool A first for public pages or raw HTML, then paste the cleaned Markdown here before embedding or storing retrieval context.
Output
Paste cleaned Markdown, choose a preset, then inspect heading context, token estimates, code/table boundaries, and JSONL-ready chunk records.
Short answer
Use Markdown Chunk Inspector after HTML cleanup and before embedding. It reviews chunk boundaries, heading context, token estimates, overlap previews, table and code safety, and export shape so retrieval records are easier to inspect before they enter a vector database or agent context store.
Preset proof
The same cleaned Markdown can need different splits depending on whether the source is structured around headings, dense sections, long paragraphs, code, or tables. The inspector makes that choice visible before ingestion.
| Preset | Best for | Proof signal | Watch for |
|---|---|---|---|
| Heading-based | Docs, guides, and support pages with meaningful H2/H3 structure. | Chunks retain a heading path, which makes retrieved records easier to cite. | Very long sections may still need a token-window pass. |
| Token-window | Dense documents where sections are too large or uneven. | Chunk sizes become more predictable for the target model context. | A window can split a concept if the source has weak paragraph boundaries. |
| Paragraph-safe | Narrative pages, policy pages, and articles where paragraph meaning matters. | Chunks avoid cutting through a paragraph just to hit the target size. | Long paragraphs can still create oversized chunks. |
| Code/table-safe | Reference docs, API guides, and pages with examples or comparison tables. | Code fences and table-like blocks are less likely to split into broken records. | Large tables may need manual cleanup before embedding. |
{"id":"chunk-001","headingPath":["Widget API","Authentication"],"tokens":142,"warnings":[],"text":"## Authentication\nUse API keys for server-side requests only."}
{"id":"chunk-002","headingPath":["Widget API","Rate limits"],"tokens":188,"warnings":["table-boundary"],"text":"## Rate limits\n| Plan | Requests | Burst |"}Workflow fit
Use this when a page is already clean enough to read, but not yet inspected enough to trust as retrieval context.
Tool A removes page chrome. The chunk inspector checks whether the cleaned Markdown still splits into retrieval-friendly sections.
The first pass flags missing heading context, tiny chunks, oversized chunks, Markdown tables, fenced code, and overlap previews.
Use Markdown for human QA, JSON for app pipelines, and JSONL when each chunk should become one retrieval or agent-context record.
| Warning | What it means | Best next fix |
|---|---|---|
| Missing heading context | A chunk may be technically valid but hard to interpret when retrieved alone. | Preserve the nearest useful H2/H3 path or split from a cleaner section boundary. |
| Oversized chunk | A dense section may exceed the target window before the retrieval layer sees it. | Use paragraph-safe or token-window chunking, then check that the split keeps meaning intact. |
| Code or table boundary risk | Splitting inside a table or fenced code block can damage the retrieved answer. | Use the code/table-safe preset and review the exported JSONL record before ingestion. |
FAQ
No. It only inspects and exports chunks so you can review them before using a vector database, RAG framework, or agent context store.
Use HTML to Markdown for AI first when the source is a public page or raw HTML. Then paste the cleaned Markdown here to inspect chunk boundaries.
No. They are deterministic estimates for planning and comparison. Always leave margin for the tokenizer used by your final model or retrieval stack.