{"id":9,"date":"2026-11-08T13:15:26","date_gmt":"2026-11-08T13:15:26","guid":{"rendered":"http:\/\/blog.antpython.net\/?p=9"},"modified":"2026-11-08T13:15:26","modified_gmt":"2026-11-08T13:15:26","slug":"requests-beautifulsoup-%e6%8a%93%e5%8f%96%e9%9d%99%e6%80%81%e7%bd%91%e9%a1%b5%e5%ae%8c%e6%95%b4%e6%b5%81%e7%a8%8b","status":"publish","type":"post","link":"http:\/\/blog.antpython.net\/?p=9","title":{"rendered":"Requests + BeautifulSoup \u6293\u53d6\u9759\u6001\u7f51\u9875\u5b8c\u6574\u6d41\u7a0b"},"content":{"rendered":"<p>\u9759\u6001\u7f51\u9875\u722c\u866b\u662f\u5165\u95e8\u5fc5\u7ecf\u4e4b\u8def\uff1a\u53d1\u8bf7\u6c42\u3001\u62ff HTML\u3001\u89e3\u6790\u5b57\u6bb5\u3001\u843d\u76d8\u3002\u672c\u6587\u7528 <code>requests<\/code> + <code>BeautifulSoup<\/code> \u8d70\u5b8c\u4e00\u6761\u53ef\u590d\u7528\u6d41\u6c34\u7ebf\u3002<\/p>\n<h2>\u73af\u5883\u51c6\u5907<\/h2>\n<pre><code>pip install requests beautifulsoup4 lxml\n<\/code><\/pre>\n<h2>\u6700\u5c0f\u53ef\u8fd0\u884c\u793a\u4f8b<\/h2>\n<pre><code>import csv\nimport time\nimport requests\nfrom bs4 import BeautifulSoup\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla\/5.0 (compatible; StudyBot\/1.0; +https:\/\/antpython.net)\"\n}\n\ndef fetch(url: str) -&gt; str:\n    resp = requests.get(url, headers=HEADERS, timeout=15)\n    resp.raise_for_status()\n    resp.encoding = resp.apparent_encoding\n    return resp.text\n\ndef parse_list(html: str):\n    soup = BeautifulSoup(html, \"lxml\")\n    items = []\n    for a in soup.select(\"h2 a\"):  # \u6309\u771f\u5b9e\u9875\u9762\u6539\u9009\u62e9\u5668\n        items.append({\n            \"title\": a.get_text(strip=True),\n            \"url\": a.get(\"href\"),\n        })\n    return items\n\ndef save_csv(rows, path=\"result.csv\"):\n    with open(path, \"w\", newline=\"\", encoding=\"utf-8-sig\") as f:\n        writer = csv.DictWriter(f, fieldnames=[\"title\", \"url\"])\n        writer.writeheader()\n        writer.writerows(rows)\n\nif __name__ == \"__main__\":\n    html = fetch(\"https:\/\/example.com\")\n    rows = parse_list(html)\n    save_csv(rows)\n    print(f\"\u5171\u6293\u53d6 {len(rows)} \u6761\")\n<\/code><\/pre>\n<h2>\u9009\u62e9\u5668\u600e\u4e48\u627e<\/h2>\n<ol>\n<li>\u6d4f\u89c8\u5668 F12 \u2192 Elements \u5b9a\u4f4d\u76ee\u6807\u8282\u70b9<\/li>\n<li>\u4f18\u5148\u7528\u7a33\u5b9a\u7684 class \/ \u8bed\u4e49\u6807\u7b7e\uff0c\u5c11\u7528\u8d85\u957f\u7edd\u5bf9\u8def\u5f84<\/li>\n<li>\u5148 <code>print(soup.select(...))<\/code> \u9a8c\u8bc1\uff0c\u518d\u6279\u91cf\u8dd1<\/li>\n<\/ol>\n<h2>\u793c\u8c8c\u722c\u53d6<\/h2>\n<ul>\n<li>\u52a0 <code>time.sleep(1)<\/code>\uff0c\u522b\u628a\u5bf9\u65b9\u670d\u52a1\u5668\u6253\u6302<\/li>\n<li>\u9075\u5b88 robots.txt \u4e0e\u7f51\u7ad9\u670d\u52a1\u6761\u6b3e<\/li>\n<li>\u53ea\u6293\u5b66\u4e60\/\u516c\u5f00\u6570\u636e\uff0c\u6ce8\u610f\u7248\u6743\u4e0e\u9690\u79c1<\/li>\n<\/ul>\n<h2>\u5e38\u89c1\u62a5\u9519\u901f\u67e5<\/h2>\n<ul>\n<li><code>403<\/code>\uff1aUA\/Headers \u88ab\u62d2\uff0c\u6216\u9700\u8981\u767b\u5f55 Cookie<\/li>\n<li>\u4e71\u7801\uff1a\u8bbe\u7f6e <code>resp.encoding<\/code><\/li>\n<li>\u7a7a\u7ed3\u679c\uff1a\u9009\u62e9\u5668\u5199\u9519\uff0c\u6216\u5185\u5bb9\u662f JS \u52a8\u6001\u52a0\u8f7d\uff08\u8be5\u6362 Selenium\/Playwright\uff09<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>\u9759\u6001\u7f51\u9875\u722c\u866b\u662f\u5165\u95e8\u5fc5\u7ecf\u4e4b\u8def\uff1a\u53d1\u8bf7\u6c42\u3001\u62ff HTML\u3001\u89e3\u6790\u5b57\u6bb5\u3001\u843d\u76d8\u3002\u672c\u6587\u7528 requests + Beautif [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-9","post","type-post","status-publish","format-standard","hentry","category-web-scraping"],"_links":{"self":[{"href":"http:\/\/blog.antpython.net\/index.php?rest_route=\/wp\/v2\/posts\/9","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/blog.antpython.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.antpython.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.antpython.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.antpython.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9"}],"version-history":[{"count":0,"href":"http:\/\/blog.antpython.net\/index.php?rest_route=\/wp\/v2\/posts\/9\/revisions"}],"wp:attachment":[{"href":"http:\/\/blog.antpython.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.antpython.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.antpython.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}