Requests + BeautifulSoup 抓取静态网页完整流程

静态网页爬虫是入门必经之路:发请求、拿 HTML、解析字段、落盘。本文用 requests + BeautifulSoup 走完一条可复用流水线。

环境准备

pip install requests beautifulsoup4 lxml

最小可运行示例

import csv
import time
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (compatible; StudyBot/1.0; +https://antpython.net)"
}

def fetch(url: str) -> str:
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    resp.encoding = resp.apparent_encoding
    return resp.text

def parse_list(html: str):
    soup = BeautifulSoup(html, "lxml")
    items = []
    for a in soup.select("h2 a"):  # 按真实页面改选择器
        items.append({
            "title": a.get_text(strip=True),
            "url": a.get("href"),
        })
    return items

def save_csv(rows, path="result.csv"):
    with open(path, "w", newline="", encoding="utf-8-sig") as f:
        writer = csv.DictWriter(f, fieldnames=["title", "url"])
        writer.writeheader()
        writer.writerows(rows)

if __name__ == "__main__":
    html = fetch("https://example.com")
    rows = parse_list(html)
    save_csv(rows)
    print(f"共抓取 {len(rows)} 条")

选择器怎么找

  1. 浏览器 F12 → Elements 定位目标节点
  2. 优先用稳定的 class / 语义标签,少用超长绝对路径
  3. 先 print(soup.select(...)) 验证,再批量跑

礼貌爬取

  • 加 time.sleep(1),别把对方服务器打挂
  • 遵守 robots.txt 与网站服务条款
  • 只抓学习/公开数据,注意版权与隐私

常见报错速查

  • 403:UA/Headers 被拒,或需要登录 Cookie
  • 乱码:设置 resp.encoding
  • 空结果:选择器写错,或内容是 JS 动态加载(该换 Selenium/Playwright)

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注