Crawl4AI Webpage → Clean Markdown/JSON

# Crawl4AI Extraction Strategy Playbook (v0.8.x)

## When to use Crawl4AI (vs requests/BeautifulSoup)
Use Crawl4AI when:
- You need clean, LLM-ready content from webpages (especially for research/RAG/reference files).
- The page is JavaScript-heavy and normal requests/BeautifulSoup scraping likely fails.
- You need structured JSON from repeated elements (e.g., product listings, docs sections, job posts, changelogs, directory pages).
- You want a cheaper/non-LLM extraction first, with a fallback to LLM only when needed.

## Reliability rule
- Always check `result.success` before trusting outputs.

## Prefer strategy order
1) Markdown extraction (for reference/RAG)
2) LLM-free structured JSON:
   - CSS/XPath schema extraction for repeated structured elements
   - RegexExtractionStrategy for simple fields (URLs/emails/dates/currency/phone/IDs/etc.)
3) LLMExtractionStrategy only when content is unstructured/semantic interpretation is required.
   - Note: LLM extraction can be slower and costlier than schema/regex.

## Minimal async crawler baseline (Markdown output)
```python
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    config = CrawlerRunConfig(markdown_generator=DefaultMarkdownGenerator())
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com", config=config)
        if result.success:
            print(result.markdown)
        else:
            print("Crawl failed:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())
```

## LLM-friendly Markdown vs filtered “fit_markdown”
- Crawl4AI can provide both:
  - `result.markdown` (raw/unfiltered markdown)
  - `fit_markdown` when content filters are used (filtered “fit” markdown)

(Use filtered markdown when you want fewer tokens for downstream LLM tasks; the LLM extraction docs explicitly support `input_format="fit_markdown"`.)

## LLM-Free JSON extraction (schema-based): CSS or XPath
Use when the page has repeated structured elements.

### JsonCssExtractionStrategy (CSS)
- Strategy uses:
  - `baseSelector`: container for each repeated item
  - `fields`: per-item fields with CSS selectors and `type`
- Example structure (from docs):
```python
schema = {
  "name": "Crypto Prices",
  "baseSelector": "div.crypto-row",
  "fields": [
    {"name": "coin_name", "selector": "h2.coin-name", "type": "text"},
    {"name": "price", "selector": "span.coin-price", "type": "text"},
  ],
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

config = CrawlerRunConfig(
  cache_mode=CacheMode.BYPASS,
  extraction_strategy=extraction_strategy,
)
```

### JsonXPathExtractionStrategy (XPath)
Same concept as CSS version but using XPath selectors.

### raw:// scheme (for local HTML testing)
- Crawl4AI supports passing dummy HTML directly via `raw://<html_string>`.
- This is shown in the XPath example.

## LLM-Free JSON extraction: RegexExtractionStrategy
Use when you only need fast pattern extraction (emails/phones/URLs/dates/currency/etc.).

### Built-in patterns (IntFlag)
- `RegexExtractionStrategy.Url`
- `RegexExtractionStrategy.Email`
- `RegexExtractionStrategy.PhoneUS`, `RegexExtractionStrategy.PhoneIntl`
- `RegexExtractionStrategy.DateIso`, `RegexExtractionStrategy.DateUS`
- `RegexExtractionStrategy.Currency`, `RegexExtractionStrategy.Number`, `RegexExtractionStrategy.Uuid`, etc.

### Simple example
```python
strategy = RegexExtractionStrategy(
    pattern = RegexExtractionStrategy.Email | RegexExtractionStrategy.Url
)
config = CrawlerRunConfig(extraction_strategy=strategy)

result = await crawler.arun(url="https://example.com", config=config)
if result.success:
    data = json.loads(result.extracted_content)
```

### Regex match output shape
Returns a JSON array of matches like:
```json
[
  {
    "url": "https://example.com",
    "label": "email",
    "value": "contact@example.com",
    "span": [145, 163]
  }
]
```

### LLM-assisted regex pattern generation (one-time)
- RegexExtractionStrategy includes `generate_pattern(label, html, query, llm_config)`.
- Workflow described: fetch sample HTML, generate pattern once via LLM, save to disk, then extract with regex only.

## LLM extraction: LLMExtractionStrategy
Use when:
- content is unstructured
- you need semantic interpretation / reorganization / knowledge-graph style extraction

### Key facts from docs
- Provider-agnostic via LiteLLM using `LLMConfig(provider="<provider>/<model>", api_token=..., base_url=optional)`.
- LLM-based extraction can be slower and costlier than schema-based.
- Strategy must be placed in `CrawlerRunConfig(extraction_strategy=...)` (not passed directly as a param to `arun()`).

### extraction_type
- `extraction_type="schema"`: model returns JSON conforming to a Pydantic-derived schema.
- `extraction_type="block"`: model returns freeform text or smaller JSON structures; library collects.

### input_format
Controls what the LLM receives:
- `"markdown"` (default)
- `"fit_markdown"`
- `"html"`

### chunking parameters
- `chunk_token_threshold` (max tokens per chunk)
- `overlap_rate` (e.g., 0.1 for 10% overlap)
- `apply_chunking` (bool)

### Required/typical parameters inside LLMExtractionStrategy
From the docs:
- `llm_config: LLMConfig`
- `schema: dict` (when `extraction_type="schema"`)
- `extraction_type: "schema" | "block"`
- `instruction: str`
- optional chunking fields (`chunk_token_threshold`, `overlap_rate`, `apply_chunking`)
- `input_format: "markdown" | "fit_markdown" | "html"`
- optional `extra_args`

### Minimal pattern
```python
import os, json
from pydantic import BaseModel
from crawl4ai import (
    AsyncWebCrawler, BrowserConfig, CrawlerRunConfig,
    CacheMode, LLMConfig, LLMExtractionStrategy
)

class Product(BaseModel):
    name: str
    price: str

async def main():
    llm_strategy = LLMExtractionStrategy(
        llm_config=LLMConfig(
            provider="openai/gpt-4o-mini",
            api_token=os.getenv("OPENAI_API_KEY"),
        ),
        schema=Product.model_json_schema(),
        extraction_type="schema",
        instruction="Extract all product objects with 'name' and 'price'.",
        apply_chunking=True,
        input_format="markdown",
        extra_args={"temperature": 0.0, "max_tokens": 800},
        verbose=True,
    )

    crawl_config = CrawlerRunConfig(
        extraction_strategy=llm_strategy,
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url="https://example.com/products", config=crawl_config)

    if result.success:
        data = json.loads(result.extracted_content)
        print(data)
        llm_strategy.show_usage()
    else:
        print("Error:", result.error_message)

import asyncio
if __name__ == "__main__":
    asyncio.run(main())
```

## LLM output handling
- When `extraction_type="schema"`, the docs recommend parsing/validating the JSON output.
- `llm_strategy.show_usage()` prints token usage when provided by the provider.

## Strategy placement constraint
- Docs explicitly state: strategy definitions belong inside `CrawlerRunConfig`, not as direct parameters to `arun()`.

## Sibling layout handling for schema extraction (CSS/XPath)
If an item’s data is split across sibling elements (not descendants of a base container), schema supports per-field `source` to navigate to a sibling before applying that field’s selector.

Syntax shown:
- `"source": "+ <selector>"`

Example from docs (Hacker News):
```json
{"name": "score", "selector": "span.score", "type": "text", "source": "+ tr"}
```

## JS-heavy pages
- Crawl4AI supports JS execution and dynamic content rendering via its browser integration (JS execution described in examples using `js_code` in `CrawlerRunConfig`).
- For structured extraction on JS-rendered sections, docs show passing `js_code` to click tabs and wait.

(See separate note in install/dynamic-pages file about how to use async + js_code; the code example shown in Crawl4AI README uses `run_config = CrawlerRunConfig(js_code=[...])`.)

# Crawl4AI Install & Dynamic Crawling (async + JS)

## Install (Python)
- Basic install:
  - `pip install -U crawl4ai`
- If you need pre-release:
  - `pip install crawl4ai --pre`
- After installing, run post-install setup:
  - `crawl4ai-setup`
- Verify installation:
  - `crawl4ai-doctor`

## Playwright dependencies
- Crawl4AI uses Playwright for web crawling (async version).
- If Playwright-related errors occur, install browsers manually:
  - `playwright install`
  - or: `python -m playwright install chromium`

## Minimal async crawl
```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://www.nbcnews.com/business")
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
```

## JS-heavy pages: use the browser and run custom JS
Crawl4AI supports executing JavaScript and waiting for dynamic content. Docs show using `CrawlerRunConfig(js_code=[...])`.

Example: click through tabs to reveal content, then extract structured data without LLMs:
```python
import asyncio, json
from crawl4ai import (
    AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode,
    JsonCssExtractionStrategy
)

async def main():
    schema = {
        "name": "KidoCode Courses",
        "baseSelector": "section.charge-methodology .w-tab-content > div",
        "fields": [
            {"name": "section_title", "selector": "h3.heading-50", "type": "text"},
            {"name": "section_description", "selector": ".charge-content", "type": "text"},
            {"name": "course_name", "selector": ".text-block-93", "type": "text"},
            {"name": "course_description", "selector": ".course-content-text", "type": "text"},
            {"name": "course_icon", "selector": ".image-92", "type": "attribute", "attribute": "src"},
        ],
    }

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    browser_config = BrowserConfig(headless=False, verbose=True)
    run_config = CrawlerRunConfig(
        extraction_strategy=extraction_strategy,
        js_code=["""
        (async () => {
          const tabs = document.querySelectorAll(
            "section.charge-methodology .tabs-menu-3 > div"
          );
          for (let tab of tabs) {
            tab.scrollIntoView();
            tab.click();
            await new Promise(r => setTimeout(r, 500));
          }
        })();
        """],
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.kidocode.com/degrees/technology",
            config=run_config,
        )

    companies = json.loads(result.extracted_content)
    print(f"Successfully extracted {len(companies)} companies")
    print(json.dumps(companies[0], indent=2))

if __name__ == "__main__":
    asyncio.run(main())
```

## Browser/session persistence (persistent user profile)
Docs show `BrowserConfig(user_data_dir=..., use_persistent_context=True)` with `magic=True` when crawling:
```python
import os
from pathlib import Path
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def test_news_crawl():
    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
    os.makedirs(user_data_dir, exist_ok=True)

    browser_config = BrowserConfig(
        verbose=True,
        headless=True,
        user_data_dir=user_data_dir,
        use_persistent_context=True,
    )

    run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

    async with AsyncWebCrawler(config=browser_config) as crawler:
        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
        result = await crawler.arun(url, config=run_config, magic=True)
        print(f"Content length: {len(result.markdown)}")

if __name__ == "__main__":
    asyncio.run(test_news_crawl())
```

## Docker quick test (self-hosted API)
The README describes a Docker image exposing endpoints like `/crawl` and `/task/{task_id}`.

Run container:
- `docker pull unclecode/crawl4ai:latest`
- `docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest`

Submit crawl job (POST `/crawl`):
```python
import requests

response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    if "results" in response.json():
        results = response.json()["results"]
        for result in results:
            print(result)
    else:
        task_id = response.json()["task_id"]
        result = requests.get(f"http://localhost:11235/task/{task_id}")
```

(For monitoring dashboard/playground URLs, the README states: `http://localhost:11235/dashboard` and `http://localhost:11235/playground`.)