前言

嗨嘍，大家好呀~這里是愛看美女的茜茜吶

又到了學(xué)Python時(shí)刻~

image

開發(fā)環(huán)境:

• python 3.8: 解釋器

• pycharm: 代碼編輯器

• requests: 發(fā)送請(qǐng)求

• parsel: 解析數(shù)據(jù) jieba pandas stylecloud

實(shí)現(xiàn)流程:

一. 思路分析

找到數(shù)據(jù)來源

network 記錄網(wǎng)頁(yè)數(shù)據(jù) 請(qǐng)求

二. 實(shí)現(xiàn)代碼

1. 發(fā)送請(qǐng)求

2. 獲取數(shù)據(jù)

3. 解析數(shù)據(jù)

4. 保存數(shù)據(jù)

代碼展示

導(dǎo)入模塊

import requests     # 需要額外安裝
import csv

半成品 (最基本的架構(gòu))

分布式

f = open('評(píng)論.csv', mode='a', encoding='utf-8-sig', newline='')
csv_writer = csv.writer(f)
csv_writer.writerow(['id', 'text_raw', 'source', 'like_counts', 'screen_name', 'followers_count'])

請(qǐng)求頭

headers = {
    # 用戶身份信息
    'cookie': 'SINAGLOBAL=2983116536906.6313.1675064186827; UOR=,,www.baidu.com; SUB=_2A25O3LQhDeRhGeNH6lIY-CjFyT-IHXVqPtxprDV8PUJbkNANLXXDkW1NSsgssEqW-GyChLyvK7V0MBeFYsy23ElC; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WWNoXNAv0FwwfrPTiDyBL285NHD95Qf1K271Knc1Kz0Ws4Dqcjci--4iK.Ri-zXi--fi-2Xi-24i--Xi-zRiKy2i--4iKnfiKy2i--fi-isi-8Fi--Xi-z4iK.7; XSRF-TOKEN=gSKn52W-3CqnU16SUOqIOY5r; _s_tentry=weibo.com; Apache=3694052218654.842.1675252819884; ULV=1675252819940:5:2:5:3694052218654.842.1675252819884:1675237842632; WBPSESS=Q4mocWB9j3toNvru27wa1agqFWvDlaxybkjDQlV26btT8dAjnZKLHT6p8NAwVhAzIbygYan56xOn7f0H4XAe_IymVa9GmASzrKUCqV6J_Ml2bYGTL_URVrA5YnjNm62kuNboZbV4Cpn1MZTfoLbEWg==',
    # 防盜鏈
    'referer': 'https://***.com/1223178222/MqQsvemFc',
    # 瀏覽器基本信息
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}
next = 'count=10'

1. 發(fā)送請(qǐng)求

while True:
    url = f'https://****.com/ajax/statuses/buildComments?is_reload=1&id=4863981833423714&is_show_bulletin=2&is_mix=0&{next}&uid=1223178222'
    response = requests.get(url=url, headers=headers)

2. 獲取數(shù)據(jù)

.text: 獲取文本數(shù)據(jù)

.json(): json字符串?dāng)?shù)據(jù)

.content: 獲取二進(jìn)制數(shù)據(jù)

2.1 出現(xiàn)亂碼

    json_data = response.json()

3. 解析數(shù)據(jù)

結(jié)構(gòu)化數(shù)據(jù) : json數(shù)據(jù){}包裹的格式 轉(zhuǎn)成字典, 使用字典取值 re

非結(jié)構(gòu)化數(shù)據(jù): 網(wǎng)頁(yè)源代碼 lxml, bs4, parsel, re css/xpath/re

    data_list = json_data['data']
    max_id = json_data['max_id']
    for data in data_list:
        text_raw = data['text_raw']
        try:
            source = data['source']
        except:
            source = "未知"
        id = data['id']
        like_counts = data['like_counts']
        screen_name = data['user']['screen_name']
        followers_count = data['user']['followers_count']
        print(id, text_raw, source, like_counts, screen_name, followers_count)

4. 保存數(shù)據(jù)

        csv_writer.writerow([id, text_raw, source, like_counts, screen_name, followers_count])
    next = 'max_id='+str(max_id)

尾語(yǔ)

感謝你觀看我的文章吶~本次航班到這里就結(jié)束啦 ??

希望本篇文章有對(duì)你帶來幫助 ??，有學(xué)習(xí)到一點(diǎn)知識(shí)~

躲起來的星星??也在努力發(fā)光，你也要努力加油（讓我們一起努力叭）。

image