Crawl4AI Webpage → Clean Markdown/JSON

# Crawl4AI Install & Dynamic Crawling (async + JS)

## Install (Python)
- Basic install:
  - `pip install -U crawl4ai`
- If you need pre-release:
  - `pip install crawl4ai --pre`
- After installing, run post-install setup:
  - `crawl4ai-setup`
- Verify installation:
  - `crawl4ai-doctor`

## Playwright dependencies
- Crawl4AI uses Playwright for web crawling (async version).
- If Playwright-related errors occur, install browsers manually:
  - `playwright install`
  - or: `python -m playwright install chromium`

## Minimal async crawl
```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://www.nbcnews.com/business")
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())
```

## JS-heavy pages: use the browser and run custom JS
Crawl4AI supports executing JavaScript and waiting for dynamic content. Docs show using `CrawlerRunConfig(js_code=[...])`.

Example: click through tabs to reveal content, then extract structured data without LLMs:
```python
import asyncio, json
from crawl4ai import (
    AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode,
    JsonCssExtractionStrategy
)

async def main():
    schema = {
        "name": "KidoCode Courses",
        "baseSelector": "section.charge-methodology .w-tab-content > div",
        "fields": [
            {"name": "section_title", "selector": "h3.heading-50", "type": "text"},
            {"name": "section_description", "selector": ".charge-content", "type": "text"},
            {"name": "course_name", "selector": ".text-block-93", "type": "text"},
            {"name": "course_description", "selector": ".course-content-text", "type": "text"},
            {"name": "course_icon", "selector": ".image-92", "type": "attribute", "attribute": "src"},
        ],
    }

    extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)

    browser_config = BrowserConfig(headless=False, verbose=True)
    run_config = CrawlerRunConfig(
        extraction_strategy=extraction_strategy,
        js_code=["""
        (async () => {
          const tabs = document.querySelectorAll(
            "section.charge-methodology .tabs-menu-3 > div"
          );
          for (let tab of tabs) {
            tab.scrollIntoView();
            tab.click();
            await new Promise(r => setTimeout(r, 500));
          }
        })();
        """],
        cache_mode=CacheMode.BYPASS,
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://www.kidocode.com/degrees/technology",
            config=run_config,
        )

    companies = json.loads(result.extracted_content)
    print(f"Successfully extracted {len(companies)} companies")
    print(json.dumps(companies[0], indent=2))

if __name__ == "__main__":
    asyncio.run(main())
```

## Browser/session persistence (persistent user profile)
Docs show `BrowserConfig(user_data_dir=..., use_persistent_context=True)` with `magic=True` when crawling:
```python
import os
from pathlib import Path
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def test_news_crawl():
    user_data_dir = os.path.join(Path.home(), ".crawl4ai", "browser_profile")
    os.makedirs(user_data_dir, exist_ok=True)

    browser_config = BrowserConfig(
        verbose=True,
        headless=True,
        user_data_dir=user_data_dir,
        use_persistent_context=True,
    )

    run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

    async with AsyncWebCrawler(config=browser_config) as crawler:
        url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"
        result = await crawler.arun(url, config=run_config, magic=True)
        print(f"Content length: {len(result.markdown)}")

if __name__ == "__main__":
    asyncio.run(test_news_crawl())
```

## Docker quick test (self-hosted API)
The README describes a Docker image exposing endpoints like `/crawl` and `/task/{task_id}`.

Run container:
- `docker pull unclecode/crawl4ai:latest`
- `docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest`

Submit crawl job (POST `/crawl`):
```python
import requests

response = requests.post(
    "http://localhost:11235/crawl",
    json={"urls": ["https://example.com"], "priority": 10}
)
if response.status_code == 200:
    if "results" in response.json():
        results = response.json()["results"]
        for result in results:
            print(result)
    else:
        task_id = response.json()["task_id"]
        result = requests.get(f"http://localhost:11235/task/{task_id}")
```

(For monitoring dashboard/playground URLs, the README states: `http://localhost:11235/dashboard` and `http://localhost:11235/playground`.)

---
name: crawl4ai-webpage-to-markdown-json
description: "Turns webpages into clean, LLM-ready Markdown or structured JSON using the Crawl4AI Python library. Use when the user asks to scrape/convert a URL into “clean markdown for RAG”, “reference file markdown”, “structured JSON for repeated items”, or when pages are “JavaScript-heavy” / “dynamic” and requests + BeautifulSoup won’t reliably capture content. Also use when the user asks for “async crawl”, “run Crawl4AI with js_code”, or “extract with CSS/XPath/regex/LLM strategy”. Trigger keywords: crawl4ai, AsyncWebCrawler, CrawlerRunConfig, js_code, DefaultMarkdownGenerator, fit_markdown, JsonCssExtractionStrategy, JsonXPathExtractionStrategy, RegexExtractionStrategy, LLMExtractionStrategy, extracted_content, result.success. Prefer Crawl4AI over requests/BS unless the page is trivially static and the user explicitly wants lightweight HTML parsing."
---
Crawl4AI guidance for converting webpages to clean Markdown or structured JSON.

Workflow
1) Decide the extraction approach.
   - If the goal is “LLM-ready Markdown / RAG / reference files”, plan to use DefaultMarkdownGenerator and return result.markdown or fit_markdown.
   - If the goal is “structured JSON for repeated items” (lists of cards/rows/sections), plan schema-based JSON first via CSS or XPath.
   - If you only need a few fast fields (emails/urls/phones/dates/currency/ids), plan RegexExtractionStrategy.
   - If the goal requires semantic interpretation/unstructured reformatting, plan LLMExtractionStrategy.
2) Prefer Crawl4AI over requests/BeautifulSoup when:
   - The page is JS-heavy/dynamic, content loads after interactions, or static HTML misses the needed text.
   - You need consistent cleaning into LLM-ready output.
   Use when the user requests “clean markdown” or “structured extraction” from a URL.
3) Install and verify before coding.
   - Run: pip install -U crawl4ai
   - Run: crawl4ai-setup
   - Run: crawl4ai-doctor
   - If Playwright issues: run playwright install (or python -m playwright install chromium).
   (Read [references/crawl4ai-install-and-dynamic-crawling.md](references/crawl4ai-install-and-dynamic-crawling.md) when setting up/installing or debugging Playwright/dynamic crawling.)
4) Use the async crawling baseline.
   - Use AsyncWebCrawler in an asyncio program.
   - Run crawler.arun(url, config=CrawlerRunConfig(...)).
   - Always check result.success before using result.markdown or result.extracted_content.
   (Read [references/crawl4ai-extraction-strategy-playbook.md](references/crawl4ai-extraction-strategy-playbook.md) when selecting Markdown vs JSON strategies, ordering strategies, and enforcing result.success checks.)
5) Generate LLM-ready Markdown for RAG/reference files.
   - Configure CrawlerRunConfig(markdown_generator=DefaultMarkdownGenerator()).
   - After arun: if result.success, return result.markdown (or use fit_markdown when token reduction/filtered content is required).
   - Prefer fit_markdown when downstream LLM context budget is tight.
6) Handle JavaScript-heavy or interaction-revealed content.
   - Use CrawlerRunConfig(js_code=[...]) to run browser-side JS (e.g., click tabs, expand accordions, scroll to trigger lazy loads).
   - If needed, use BrowserConfig(headless=..., user_data_dir=..., use_persistent_context=True) to persist sessions.
   (Read [references/crawl4ai-install-and-dynamic-crawling.md](references/crawl4ai-install-and-dynamic-crawling.md) when the page requires dynamic JS, js_code usage, or persistent browser profiles.)
7) Produce structured JSON without LLM first.
   - For repeated structured items: use JsonCssExtractionStrategy or JsonXPathExtractionStrategy with a schema defining baseSelector and fields.
   - Place extraction_strategy in CrawlerRunConfig(extraction_strategy=...).
   - After arun: if result.success, parse result.extracted_content as JSON.
8) Use regex only for simple patterns.
   - Use RegexExtractionStrategy with built-in patterns (Email/Url/Phone/Date/Currency/Number/Uuid, etc.).
   - Parse result.extracted_content as JSON when result.success.
9) Use LLMExtractionStrategy only when needed.
   - Use LLMExtractionStrategy(...), and place it in CrawlerRunConfig(extraction_strategy=...); do not pass it directly to arun().
   - Choose extraction_type:
     * “schema” when you need strict JSON conforming to a schema.
     * “block” when you want freeform text/smaller structures.
   - Choose input_format among markdown/fit_markdown/html.
   - Configure llm_config (provider/model) from your externally prepared auth environment.
   (Read [references/crawl4ai-extraction-strategy-playbook.md](references/crawl4ai-extraction-strategy-playbook.md) when deciding between CSS/XPath vs regex vs LLM, and when enforcing placement + reliability checks.)

Constraints / edge cases
- Never trust output until result.success is True. If false, return the error (result.error_message) and suggest retry with js_code or a different strategy.
- Strategy ordering rule: Markdown first for reference/RAG; schema JSON next (CSS/XPath); regex for simple fields; LLM last.
- If extraction fails on a JS page, add js_code to reveal the content before extracting.
- For repeated structured content, prefer CSS/XPath over LLM for speed/cost.

Concrete I/O expectations
- Markdown path: return a string from result.markdown (or fit_markdown if configured/used).
- JSON path: return parsed JSON from json.loads(result.extracted_content) after result.success.

Example output mapping
- User asks: “Convert this docs URL into clean markdown for my RAG reference.” → Use MarkdownGenerator; return result.markdown/fit_markdown.
- User asks: “Extract product listings (name, price, url) into JSON.” → Use JsonCssExtractionStrategy or JsonXPathExtractionStrategy; return JSON.
- User asks: “Find all emails/phones/urls on this page.” → Use RegexExtractionStrategy; return matches.
- User asks: “Summarize each section into a structured knowledge graph.” → Use LLMExtractionStrategy (schema if strict), with fit_markdown input.