配置缓存与 Markdown 过滤，用 CSS 规则提取结构化 JSON

第 1 步：显式配置浏览器和缓存

import asyncio
from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    CacheMode,
)

async def main():
    browser = BrowserConfig(headless=True)
    run = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
    async with AsyncWebCrawler(config=browser) as crawler:
        result = await crawler.arun("https://example.com", config=run)
        text = result.markdown.raw_markdown
        print(result.success, len(text))

asyncio.run(main())

对照结果：

True 953

说明：

• BrowserConfig(headless=True)：无头模式，服务器上默认就该这样。
• CacheMode.BYPASS：不走本地缓存，每次都重新抓。官方 Quick Start 也写了默认常是 BYPASS；要省流量或加快重复实验，可改成 CacheMode.ENABLED。
• 把配置对象分别传给 AsyncWebCrawler(config=...) 和 arun(..., config=...)，浏览器级与「这一次运行」级的选项就分开了。

第 2 步：抓官方文档页，对比 raw 与 fit Markdown

文档站导航多、侧栏多，正好适合看内容过滤前后的差别。新建 md_filter.py：

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilterLXML
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    md_gen = DefaultMarkdownGenerator(
        content_filter=PruningContentFilterLXML(
            threshold=0.4,
            threshold_type="fixed",
        )
    )
    run = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        markdown_generator=md_gen,
    )
    browser = BrowserConfig(headless=True)
    url = "https://docs.crawl4ai.com/core/quickstart/"
    async with AsyncWebCrawler(config=browser) as crawler:
        result = await crawler.arun(url, config=run)
        md = result.markdown
        raw = md.raw_markdown
        fit = md.fit_markdown or ""
        print("SUCCESS", result.success)
        print("RAW_LEN", len(raw))
        print("FIT_LEN", len(fit))
        print("--- RAW head ---")
        print("\n".join(raw.splitlines()[:6]))
        print("--- FIT head ---")
        print("\n".join(fit.splitlines()[:4]))

asyncio.run(main())

对照结果：

SUCCESS True
RAW_LEN 26166
FIT_LEN 18578
--- RAW head ---
[Crawl4AI Documentation (v0.9.x)](https://docs.crawl4ai.com/)
  * [ Home ](https://docs.crawl4ai.com/)
  * [ Growth ](https://docs.crawl4ai.com/stats/)
  * [ Ask AI ](https://docs.crawl4ai.com/core/ask-ai/)
  * [ Quick Start ](https://docs.crawl4ai.com/core/quickstart/)
  * [ Code Examples ](https://docs.crawl4ai.com/core/examples/)
--- FIT head ---
New good news for everyone who runs Crawl4AI
## Crawl4AI _Cloud_ is live.The same crawler, hosted.
No browsers to run, no proxies to buy, no cookies to babysit. One key with **1,000 free credits every month** , and your AI agent can **scrape, extract, search, read Google's results page (SERP) and answer questions** through the same engine you already use. Free: 1,000 credits every month, 10 a minute, 2 at a time. Paid: 120 a minute, 10 at a time, from $10 a month.
**Always open source, forever.*

raw_markdown 保留了顶栏导航链接；fit_markdown 把大量导航剪掉后，长度从约 26166 降到约 18578（大约七成），开头直接进入正文。喂给大模型或做 RAG 切片时，通常更想用 fit。阈值 0.4 可按页面再调；过滤会多花几十毫秒量级的处理时间，一般可忽略。

第 3 步：用 JsonCssExtractionStrategy 抽结构化 JSON

CSS 提取不调用大模型，适合列表页、卡片墙、价格表这类结构稳定的页面。官方还支持把一段 HTML 直接喂给爬虫：URL 写成 raw:// 加上 HTML 字符串。新建 css_extract.py：

import asyncio
import json
from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CrawlerRunConfig,
    CacheMode,
    JsonCssExtractionStrategy,
)

SCHEMA = {
    "name": "Cards",
    "baseSelector": "div.card",
    "fields": [
        {"name": "title", "selector": "h2.title", "type": "text"},
        {
            "name": "link",
            "selector": "a.link",
            "type": "attribute",
            "attribute": "href",
        },
        {"name": "price", "selector": "span.price", "type": "text"},
    ],
}

RAW_HTML = """
<html><body>
  <div class="card">
    <h2 class="title">Alpha Tool</h2>
    <a class="link" href="/a">Open</a>
    <span class="price">$9</span>
  </div>
  <div class="card">
    <h2 class="title">Beta Tool</h2>
    <a class="link" href="/b">Open</a>
    <span class="price">$19</span>
  </div>
</body></html>
"""

async def main():
    run = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(SCHEMA),
    )
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun("raw://" + RAW_HTML, config=run)
        data = json.loads(result.extracted_content)
        print(json.dumps(data, ensure_ascii=False, indent=2))

asyncio.run(main())

对照结果：

[
  {
    "title": "Alpha Tool",
    "link": "/a",
    "price": "$9"
  },
  {
    "title": "Beta Tool",
    "link": "/b",
    "price": "$19"
  }
]

schema 怎么读：

• baseSelector：每一条记录对应的外层节点（这里是每张 div.card）。
• fields：字段名、CSS 选择器、类型。type 常用 text（取文本）和 attribute（取属性，并写 attribute 名）。

真实网站把 raw://... 换成普通 https://... 即可，schema 按页面改选择器。官方还提供 JsonCssExtractionStrategy.generate_schema(...)，可让大模型根据一段 HTML 样例自动生成 schema（一次性花费，之后重复提取不再调模型）；本课没有接 API Key，这一段未实测。

第 4 步：对真实页面抽字段时要注意 DOM 变化

课时 1 抓过的 https://example.com ，现在页面结构已经不含 <h1>（本课实测 HTML 里搜不到 <h1），只有多语言段落。若仍按旧教程写 selector: "h1"，对应字段会是空的。可以改成只抽段落：

schema = {
    "name": "ExamplePage",
    "baseSelector": "body",
    "fields": [
        {"name": "paragraphs", "selector": "p", "type": "text"},
    ],
}

本课实测抽出的第一条英文段落是：

This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

结论：CSS 提取很快、便宜，但选择器绑死 DOM；页面改版后要重写 schema。结构不稳定或字段含义要靠语义判断时，再考虑下一课提到的 LLMExtractionStrategy。

小结

• 用 BrowserConfig 管浏览器，用 CrawlerRunConfig 管这一次运行；缓存开关看 CacheMode。
• raw_markdown 偏完整，fit_markdown 偏干净；RAG 场景优先试 fit。
• JsonCssExtractionStrategy 适合稳定列表；可用 raw:// HTML 先在本地验证 schema。
• 真实站点改版会导致选择器失效，提取前先看一眼 HTML 结构。