1、數(shù)據(jù)來源網(wǎng)站（手機(jī)端微博）

• https://m.weibo.cn/

• image

2、數(shù)據(jù)內(nèi)容

• 2.1 搜索關(guān)鍵字"#丁真#"

  • image

• 2.2 拿到瀏覽接口地址1

  • "#丁真#" 的搜索結(jié)果，接口地址

    • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall%23%E4%B8%81%E7%9C%9F%23

  • 同樣方法拿到其余四個(gè)地址

    • 丁真的世界#

      • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%E7%9A%84%E4%B8%96%E7%95%8C%23&page_type=searchall

    • "#丁真說不要再p了#

      • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall

    • 四川為了丁真有多努力#

    • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E5%9B%9B%E5%B7%9D%E4%B8%BA%E4%BA%86%E4%B8%81%E7%9C%9F%E6%9C%89%E5%A4%9A%E5%8A%AA%E5%8A%9B%23&page_type=searchall

    • "#丁真所在國企負(fù)責(zé)人回應(yīng)拒絕選秀#"

      • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall

• 2.3 以第一個(gè)接口為例（#丁真的世界#）

  • image

• 2.4 拿到具體的接口調(diào)用地址

  • https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%E7%9A%84%E4%B8%96%E7%95%8C%23&page_type=searchall

  • image

• 2.5 分析相關(guān)的返回結(jié)果參數(shù)

  • image

  • 為什么要這個(gè)id 呢，因?yàn)椴榭淳唧w的微博評(píng)論需要傳入這個(gè)id

    • 微博評(píng)論接口地址：https://m.weibo.cn/comments/hotflow?id=4575845312890917&mid=4575845312890917&max_id_type=0

    • image

  • 觀察評(píng)論接口返回?cái)?shù)據(jù)，找到點(diǎn)贊數(shù)和評(píng)論內(nèi)容的參數(shù)

    • image

    • 其他一些需要的參數(shù)也能找到喲

3、分析除了數(shù)據(jù)來源，剩下的就是通過計(jì)算機(jī)程序去自動(dòng)抓取數(shù)據(jù)了

• 程序可以模擬接口調(diào)用，這里用Python實(shí)現(xiàn)的

  • 3.1 設(shè)置header
    headers = {
    'Cookie': '_T_WM=53629218447; XSRF-TOKEN=db4d17; WEIBOCN_FROM=1110006030; MLOGIN=0; M_WEIBOCN_PARAMS=fid%3D100103type%253D1%2526q%253D%2525E4%2525B8%252581%2525E7%25259C%25259F%26uicode%3D10000011', 'Referer': 'https://m.weibo.cn/detail/4312409864846621', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66', 'X-Requested-With': 'XMLHttpRequest'}

    • image

  • 3.2 定義爬蟲的地址（這里是固定的五個(gè)和丁真相關(guān)的話題）
    urls =[]
    def getHostUrls():

    #丁真# urls.append("https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall")

    #丁真的世界# urls.append("https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%E7%9A%84%E4%B8%96%E7%95%8C%23&page_type=searchall")

    "#丁真說不要再p了# urls.append("https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall")

    #四川為了丁真有多努力# urls.append("https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E5%9B%9B%E5%B7%9D%E4%B8%BA%E4%BA%86%E4%B8%81%E7%9C%9F%E6%9C%89%E5%A4%9A%E5%8A%AA%E5%8A%9B%23&page_type=searchall")

    "#丁真所在國企負(fù)責(zé)人回應(yīng)拒絕選秀#" urls.append("https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%23%E4%B8%81%E7%9C%9F%23&page_type=searchall")

    • image

  • 3.3 詳情的爬蟲代碼 (參考的百度的解析response的代碼，自己不太想寫了)
    def spider(page_num,hostUrl):

    main_url為要爬取的主頁地址 if page_num:

    main_url = hostUrl + '&page=' + str(page_num)

    微博的分頁機(jī)制是每頁10條微博 try:

    r = requests.get(url=main_url, headers=headers)
    r.raise_for_status()
    except Exception as e:
    print("爬取失敗", e)
    return 0 result_json = json.loads(r.content.decode('utf-8'))
    info_list = []
    for card in result_json['data']['cards']:
    info_list_sub = []
    if card.get("mblog"):
    info_list_sub.append(card['mblog']['attitudes_count']) # 獲贊數(shù) info_list_sub.append(card['mblog']['comments_count']) # 評(píng)論數(shù) info_list_sub.append(card['mblog']['reposts_count']) # 轉(zhuǎn)發(fā)數(shù) if page_num == 1:
    info_list_sub.append(card['mblog']['created_at']) # 發(fā)博時(shí)間 elif '2018' not in card['mblog']['created_at']:
    info_list_sub.append(card['mblog']['created_at'])
    else:
    print("2019年微博爬取完畢")
    break info_list_sub.append(card['mblog']['weibo_position']) # 是否原創(chuàng) if card['mblog'].get('raw_text'):
    info_list_sub.append(card['mblog']['raw_text']) # 微博內(nèi)容 else:
    info_list_sub.append(card['mblog']['text'])

    if card['mblog']['source'] == '': # info_list_sub.append(None) # else: # info_list_sub.append(card['mblog']['source']) # time.sleep(random.randint(4, 6)) # 每爬取一條微博暫停4到6秒，防反爬 info_list.append(info_list_sub)

    else:
    continue return info_list

  • 3.4 最終保存到csv文件中
    def save_csv(infolist):
    with open('weibo.csv', 'a+', encoding='utf_8_sig', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(infolist)

    • image

  • 3.5 定義運(yùn)行的main方法
    def main(num):
    for hostUrl in urls:
    for i in range(1, num+1):
    information = spider(i,hostUrl)
    save_csv(information)
    print("第%s頁爬取完畢" % i)

    • image

  • 3.6 啟動(dòng)代碼
    print("### 開始爬取微博 ")

    1、封裝地址到urls中g(shù)etHostUrls()

    2、遍歷封裝好的urls，循環(huán)查詢接口，獲取評(píng)論數(shù)if name == 'main':

    main(10)

    • image

  • 3.7 補(bǔ)充

    • 運(yùn)行代碼的時(shí)候，需要在.py 的同級(jí)建立一個(gè)weibo.csv文件

    • 微博有反爬機(jī)制，可以設(shè)置線程休眠

      • 代碼中是注釋的版本

      • time.sleep(random.randint(4, 6)) # 每爬取一條微博暫停4到6秒，防反爬

4、git源碼地址

• https://github.com/xjdm/pythonWorkspace/blob/master/spider.py