我被爬蟲上了一課

我之前懂點(diǎn)python的語法，近期發(fā)現(xiàn)想寫一個(gè)簡(jiǎn)單的爬蟲爬取一下網(wǎng)站的圖片,在有chatgpt的加持下豈不是分分鐘搞定。

1. 在瀏覽器發(fā)起一個(gè)請(qǐng)求，生成curl

下面相當(dāng)于抓到了最原始的、最完整的請(qǐng)求參數(shù)

curl 'https://stock.xinpianchang.com/photo/list/p-1?sort=hotDownload&secondCategoryId=99&thirdCategoryId=105' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'Accept-Language: zh-CN,zh;q=0.9' \
  -H 'Cache-Control: max-age=0' \
  -H 'Connection: keep-alive' \
  -H 'Cookie: _csrf=B3BjIVKe81t7GRFoCSvNDjyl; Device_ID=2cjqm1j7szje; sid=s%3AZd22ifqmMtsu_gNagfEJkR0b-S21Tlzy.k8Lrsi6yKOmLaXMSalPZtRhQxZd8KGHdS4YkoYykIVg; sajssdk_2015_cross_new_user=1; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%7D; Hm_lvt_e62ca320e624d95f87f6991b10628c17=1727350360; HMACCOUNT=0B2C8CA6804CAD41; Authorization=AA906011B43DAE6A2B43DA4C16B43DA984FB43DAC25790C192A0; MEIQIA_TRACK_ID=2mbfnLDrRs4Vy2qG2PBFU9Iccjl; MEIQIA_VISIT_ID=2mbfnGBSBv3eniStWKaLPnvxhZi; acw_tc=0aef82e417273530771207622e00c47a8f906ea8d239b36b407c2537f5102d; Hm_lpvt_e62ca320e624d95f87f6991b10628c17=1727353129' \
  -H 'If-None-Match: "35217-1yX7tAQx7fOSSHyUeT8xnlL4Aoc"' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-Site: none' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
  -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"''

然后我在命令行執(zhí)行可以如期獲取到數(shù)據(jù)。

2.讓chatgpt生成python代碼

import requests
from bs4 import BeautifulSoup
import os

# 創(chuàng)建保存圖片的目錄
output_dir = 'images'
os.makedirs(output_dir, exist_ok=True)

# 設(shè)置請(qǐng)求頭，模擬瀏覽器

# curl_command = f"""curl 'https://stock.xinpianchang.com/photo/list/p-1?sort=hotDownload&secondCategoryId=99&thirdCategoryId=105' \
#   -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
#   -H 'Accept-Language: zh-CN,zh;q=0.9' \
#   -H 'Cache-Control: max-age=0' \
#   -H 'Connection: keep-alive' \
#   -H 'Cookie: _csrf=B3BjIVKe81t7GRFoCSvNDjyl; Device_ID=2cjqm1j7szje; sid=s%3AZd22ifqmMtsu_gNagfEJkR0b-S21Tlzy.k8Lrsi6yKOmLaXMSalPZtRhQxZd8KGHdS4YkoYykIVg; sajssdk_2015_cross_new_user=1; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%7D; Hm_lvt_e62ca320e624d95f87f6991b10628c17=1727350360; HMACCOUNT=0B2C8CA6804CAD41; Authorization=AA906011B43DAE6A2B43DA4C16B43DA984FB43DAC25790C192A0; MEIQIA_TRACK_ID=2mbfnLDrRs4Vy2qG2PBFU9Iccjl; MEIQIA_VISIT_ID=2mbfnGBSBv3eniStWKaLPnvxhZi; acw_tc=0aef82e417273530771207622e00c47a8f906ea8d239b36b407c2537f5102d; Hm_lpvt_e62ca320e624d95f87f6991b10628c17=1727353129' \
#   -H 'If-None-Match: "35217-1yX7tAQx7fOSSHyUeT8xnlL4Aoc"' \
#   -H 'Sec-Fetch-Dest: document' \
#   -H 'Sec-Fetch-Mode: navigate' \
#   -H 'Sec-Fetch-Site: none' \
#   -H 'Sec-Fetch-User: ?1' \
#   -H 'Upgrade-Insecure-Requests: 1' \
#   -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
#   -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
#   -H 'sec-ch-ua-mobile: ?0' \
#   -H 'sec-ch-ua-platform: "macOS"'"""
headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'Accept-Language': 'zh-CN,zh;q=0.9',
    'Cache-Control': 'max-age=0',
    'Connection': 'keep-alive',
    'Cookie': '_csrf=B3BjIVKe81t7GRFoCSvNDjyl; Device_ID=2cjqm1j7szje; sid=s%3AZd22ifqmMtsu_gNagfEJkR0b-S21Tlzy.k8Lrsi6yKOmLaXMSalPZtRhQxZd8KGHdS4YkoYykIVg; sajssdk_2015_cross_new_user=1; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%7D; Hm_lvt_e62ca320e624d95f87f6991b10628c17=1727350360; HMACCOUNT=0B2C8CA6804CAD41; Authorization=AA906011B43DAE6A2B43DA4C16B43DA984FB43DAC25790C192A0; MEIQIA_TRACK_ID=2mbfnLDrRs4Vy2qG2PBFU9Iccjl; MEIQIA_VISIT_ID=2mbfnGBSBv3eniStWKaLPnvxhZi; acw_tc=0aef82e417273530771207622e00c47a8f906ea8d239b36b407c2537f5102d; Hm_lpvt_e62ca320e624d95f87f6991b10628c17=1727353129',
    'If-None-Match': '"35217-1yX7tAQx7fOSSHyUeT8xnlL4Aoc"',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',
    'sec-ch-ua': '"Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"macOS"',
}

# 發(fā)起請(qǐng)求
url = "https://stock.xinpianchang.com/photo/list/p-1?sort=hotDownload&secondCategoryId=99&thirdCategoryId=105"
response = requests.get(url, headers=headers)
response.raise_for_status()  # 確保請(qǐng)求成功

我直接點(diǎn)擊執(zhí)行，心想百分百成功，竟然提示的是405

image.png

臥槽，這怎么可能，python怎么不能執(zhí)行，看了一下沒有參數(shù)，也沒有錯(cuò)誤呢，debug了一下發(fā)現(xiàn)request框架沒有加任何其他的參數(shù)呢，那服務(wù)器是怎么識(shí)別到是爬蟲的？http參數(shù)都在里面了不少呀，難道是Python的代碼有問題，我換一個(gè)語言吧。

3. go語言版本的可以成功


package main

import (
 "fmt"
 "io/ioutil"
 "net/http"
)

func main() {
 // Generated by curl-to-Go: https://mholt.github.io/curl-to-go

 // curl 'https://stock.xinpianchang.com/photo/list/p-1?sort=hotDownload&secondCategoryId=99&thirdCategoryId=105' \
 //   -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
 //   -H 'Accept-Language: zh-CN,zh;q=0.9' \
 //   -H 'Cache-Control: max-age=0' \
 //   -H 'Connection: keep-alive' \
 //   -H 'Cookie: _csrf=B3BjIVKe81t7GRFoCSvNDjyl; Device_ID=2cjqm1j7szje; sid=s%3AZd22ifqmMtsu_gNagfEJkR0b-S21Tlzy.k8Lrsi6yKOmLaXMSalPZtRhQxZd8KGHdS4YkoYykIVg; sajssdk_2015_cross_new_user=1; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%7D; Hm_lvt_e62ca320e624d95f87f6991b10628c17=1727350360; HMACCOUNT=0B2C8CA6804CAD41; Authorization=AA906011B43DAE6A2B43DA4C16B43DA984FB43DAC25790C192A0; MEIQIA_TRACK_ID=2mbfnLDrRs4Vy2qG2PBFU9Iccjl; MEIQIA_VISIT_ID=2mbfnGBSBv3eniStWKaLPnvxhZi; acw_tc=0aef82e417273530771207622e00c47a8f906ea8d239b36b407c2537f5102d; Hm_lpvt_e62ca320e624d95f87f6991b10628c17=1727353129' \
 //   -H 'If-None-Match: "35217-1yX7tAQx7fOSSHyUeT8xnlL4Aoc"' \
 //   -H 'Sec-Fetch-Dest: document' \
 //   -H 'Sec-Fetch-Mode: navigate' \
 //   -H 'Sec-Fetch-Site: none' \
 //   -H 'Sec-Fetch-User: ?1' \
 //   -H 'Upgrade-Insecure-Requests: 1' \
 //   -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
 //   -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
 //   -H 'sec-ch-ua-mobile: ?0' \
 //   -H 'sec-ch-ua-platform: "macOS"'

 req, err := http.NewRequest("GET", "https://stock.xinpianchang.com/photo/list/p-1?sort=hotDownload&secondCategoryId=99&thirdCategoryId=105", nil)
 if err != nil {
  // handle err
 }
 req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
 req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9")
 req.Header.Set("Cache-Control", "max-age=0")
 req.Header.Set("Connection", "keep-alive")
 req.Header.Set("Cookie", "_csrf=B3BjIVKe81t7GRFoCSvNDjyl; Device_ID=2cjqm1j7szje; sid=s%3AZd22ifqmMtsu_gNagfEJkR0b-S21Tlzy.k8Lrsi6yKOmLaXMSalPZtRhQxZd8KGHdS4YkoYykIVg; sajssdk_2015_cross_new_user=1; sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22%24device_id%22%3A%221922e1ac8652b49-0ce856e4e13ed3-16525637-1484784-1922e1ac8662153%22%7D; Hm_lvt_e62ca320e624d95f87f6991b10628c17=1727350360; HMACCOUNT=0B2C8CA6804CAD41; Authorization=AA906011B43DAE6A2B43DA4C16B43DA984FB43DAC25790C192A0; MEIQIA_TRACK_ID=2mbfnLDrRs4Vy2qG2PBFU9Iccjl; MEIQIA_VISIT_ID=2mbfnGBSBv3eniStWKaLPnvxhZi; acw_tc=0aef82e417273530771207622e00c47a8f906ea8d239b36b407c2537f5102d; Hm_lpvt_e62ca320e624d95f87f6991b10628c17=1727353129")
 req.Header.Set("If-None-Match", "\"35217-1yX7tAQx7fOSSHyUeT8xnlL4Aoc\"")
 req.Header.Set("Sec-Fetch-Dest", "document")
 req.Header.Set("Sec-Fetch-Mode", "navigate")
 req.Header.Set("Sec-Fetch-Site", "none")
 req.Header.Set("Sec-Fetch-User", "?1")
 req.Header.Set("Upgrade-Insecure-Requests", "1")
 req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36")
 req.Header.Set("Sec-Ch-Ua", "\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"")
 req.Header.Set("Sec-Ch-Ua-Mobile", "?0")
 req.Header.Set("Sec-Ch-Ua-Platform", "\"macOS\"")

 resp, err := http.DefaultClient.Do(req)
 if err != nil {
  // handle err
 }

 body, err := ioutil.ReadAll(resp.Body)
 if err != nil {
  // handle err
  fmt.Println("Error reading response body:", err)
  return
 }

 fmt.Println(string(body)) // 打印響應(yīng)內(nèi)容
 defer resp.Body.Close()
}

我驚訝的發(fā)現(xiàn)go語言可以的，我更加摸不到頭腦了，似乎世界遇到了bug，隨后我將python換了環(huán)境，版本都一樣的。這他媽是咋回事，見鬼了，再對(duì)比還是發(fā)現(xiàn)python代碼沒錯(cuò)誤呢。

4. 真相大白

問了其他人也不太懂，沒有得到答案，突然想到之前知乎推薦過一個(gè)問題：服務(wù)器能否拒絕非瀏覽器發(fā)起的HTTP請(qǐng)求？, 但是很多老哥冷嘲熱諷或者調(diào)侃到不能，因?yàn)榉?wù)器需要什么參數(shù)程序都可以模擬，這一點(diǎn)我也非常熟悉，但是記得一個(gè)老哥說有個(gè)什么指紋的，雖然有點(diǎn)印象但是自己不是專門做爬蟲的，所以就看了一眼就沒有多留意。一句話：服務(wù)器是通過TLS的指紋識(shí)別的，參考：https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967/，看評(píng)論區(qū)說這個(gè)專門針對(duì)python的，go語言本身沒事，具體我也不太懂，基本上應(yīng)該是通過這個(gè)思路實(shí)現(xiàn)的。具體的破解辦法，參考：https://zhuanlan.zhihu.com/p/601474166?utm_psn=1731683376740827136，或者用Selenium實(shí)現(xiàn)

還是要多學(xué)習(xí)呀，不能墨守成規(guī)，遇到問題刨根問底還是有收獲的，長(zhǎng)見識(shí)了。

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容