AI Refinement Pipeline

ContextMD employs a multi-stage pipeline to transform noisy, multi-page documentation websites into a single, high-density Markdown file optimized specifically for AI Agents and LLMs. Rather than performing a simple "copy-paste" of web content, the system "chemically refines" the data to maximize information density and retrieval accuracy.

The Transformation Process

The pipeline consists of three distinct phases designed to strip away human-centric UI elements and prioritize machine-readable logic.

1. Heuristic HTML Cleaning

Before content is sent to the AI, ContextMD uses cheerio to perform a structural sweep of the raw HTML. It identifies and removes "noise" that consumes unnecessary tokens and confuses LLMs:

Navigation & Discovery: Sidebars, headers, and footers are purged.
Scripts & Styles: All <script>, <style>, and <iframe> tags are stripped.
Semantic Extraction: The engine prioritizes content within <main> or <article> tags to ensure the core documentation is preserved while UI boilerplate is discarded.

2. Markdown Normalization

The cleaned HTML is converted into standard Markdown using a specialized Turndown configuration. This ensures that technical structures like fenced code blocks and ATX-style headers are standardized before reaching the refinement stage.

3. Agentic Optimization (LLM Refinement)

This is the core of the pipeline. The normalized Markdown is passed to OpenAI's gpt-4o-mini model with a highly specialized system prompt. The model acts as a technical editor, rewriting the content to meet the following "Agentic" criteria:

High Density: Strips conversational filler (e.g., "In this section, we will explore...") in favor of direct technical facts.
Retrieval Optimization: Reorganizes content to ensure keywords and API signatures are easily indexable by RAG systems.
Technical Integrity: Strictly preserves all code blocks, environment variables, and technical constraints.
Structural Repair: Fixes any broken Markdown syntax or table formatting resulting from the initial scrape.

The Refinement Logic

The AI is instructed with the following system persona to ensure consistency across your context.md file:

You are an expert technical writer optimizing documentation for AI Agents.
Your task is to rewrite the provided documentation Markdown to be:
1. Extremely high-density and concise.
2. Optimized for retrieval (keywords, clear logic).
3. Stripped of conversational filler.
4. Strictly preserving ALL code blocks and technical constraints.
5. Formatted with clear headers.

Usage and Configuration

The refinement pipeline is triggered automatically during the contextmd execution. To use the AI-powered refinement, you must provide a valid OpenAI API key.

# Provide the key via flag
npx contextmd-cli https://docs.example.com --key YOUR_OPENAI_API_KEY

# Or via environment variable
export OPENAI_API_KEY='your-key-here'
npx contextmd-cli https://docs.example.com

Performance & Cost

By default, the pipeline uses GPT-4o-mini. This model was selected because it offers the optimal balance between:

Context Window: Large enough to handle long documentation pages.
Cost-Efficiency: Significantly cheaper than GPT-4o for processing high-volume documentation.
Speed: Near-instant processing of page batches, visualized through the CLI's real-time progress tracker.

If the OpenAI API call fails for any reason (e.g., rate limits or network issues), ContextMD gracefully falls back to the Raw Markdown version of the page, ensuring your crawl is never lost.