静态网页爬虫是入门必经之路:发请求、拿 HTML、解析字段、落盘。本文用 requests + BeautifulSoup 走完一条可复用流水线。
环境准备
pip install requests beautifulsoup4 lxml
最小可运行示例
import csv
import time
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; StudyBot/1.0; +https://antpython.net)"
}
def fetch(url: str) -> str:
resp = requests.get(url, headers=HEADERS, timeout=15)
resp.raise_for_status()
resp.encoding = resp.apparent_encoding
return resp.text
def parse_list(html: str):
soup = BeautifulSoup(html, "lxml")
items = []
for a in soup.select("h2 a"): # 按真实页面改选择器
items.append({
"title": a.get_text(strip=True),
"url": a.get("href"),
})
return items
def save_csv(rows, path="result.csv"):
with open(path, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=["title", "url"])
writer.writeheader()
writer.writerows(rows)
if __name__ == "__main__":
html = fetch("https://example.com")
rows = parse_list(html)
save_csv(rows)
print(f"共抓取 {len(rows)} 条")
选择器怎么找
- 浏览器 F12 → Elements 定位目标节点
- 优先用稳定的 class / 语义标签,少用超长绝对路径
- 先
print(soup.select(...))验证,再批量跑
礼貌爬取
- 加
time.sleep(1),别把对方服务器打挂 - 遵守 robots.txt 与网站服务条款
- 只抓学习/公开数据,注意版权与隐私
常见报错速查
403:UA/Headers 被拒,或需要登录 Cookie- 乱码:设置
resp.encoding - 空结果:选择器写错,或内容是 JS 动态加载(该换 Selenium/Playwright)