一次性的爬完某位博主的文章，并把它全部集中在一個(gè)html頁(yè)面 ，怎么做呢？？
我們?cè)诰W(wǎng)頁(yè)中打開(kāi)微博，并搜索某位知名情感博主的博客
f12,加載頁(yè)面，在network中可以看到一大堆鏈接
先按f12再加載頁(yè)面，不然network不會(huì)出現(xiàn)之前的鏈接
仔細(xì)分析可以知道，有一條鏈接會(huì)返回一大堆json
可以看到里面有數(shù)字，可以猜測(cè)，里面的數(shù)字與當(dāng)前顯示博主發(fā)微博的條數(shù)一致，里面可能包含我們想要的數(shù)據(jù)，打開(kāi)。
果然?。。±锩嬗形覀円臄?shù)據(jù)。
不過(guò)，這只是第一頁(yè)的數(shù)據(jù)
把瀏覽器往下拉
網(wǎng)頁(yè)會(huì)加載第二頁(yè)的數(shù)據(jù) ,準(zhǔn)確的說(shuō)，這種加載方式叫做懶加載
第二頁(yè)的數(shù)據(jù)也是以json數(shù)據(jù)返回
比較這兩條鏈接
第一條：https://m.weibo.cn/api/container/getIndex?uid=1333608873&t=0&luicode=10000011&lfid=100103type%3D1%26q%3D%E7%BE%8A%E5%B8%88%E5%A4%AA&type=uid&value=1333608873&containerid=1076031333608873
第二條：https://m.weibo.cn/api/container/getIndex?uid=1333608873&t=0&luicode=10000011&lfid=100103type%3D1%26q%3D%E7%BE%8A%E5%B8%88%E5%A4%AA&type=uid&value=1333608873&containerid=1076031333608873&page=2
顯而易見(jiàn)，前面的參數(shù)都是相同的，只有后面的page不一樣
上代碼
目錄結(jié)構(gòu)：
test.py
utilsaveUtil
init.py
htmlUtil.py
templates.html
test.py
import requests
import json
from util.saveUtil.htmlUtil import *
myCards = []
def getWeb(url):
try:
header = {
'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1'
}
r = requests.get(url, timeout=(3, 6), headers=header)
r = json.loads(r.text)
if r.get("ok") == 1:
cards = r.get("data").get("cards")
print(len(cards))
for card in cards:
print(card)
created_at = card.get("mblog").get("created_at") # 創(chuàng)作日期
text = card.get("mblog").get("text") # 文章內(nèi)容
if text.find("全文") != -1:
print("全文")
url = 'https://m.weibo.cn/statuses/extend?id=' + str(text).split('status/')[1].split('"')[0]
r2 = requests.get(url, timeout=(3, 6), headers=header)
print(r2.text)
try:
r2 = json.loads(r2.text)
print(r2)
if r2.get("ok") == 1:
text = r2.get("data").get("longTextContent") # 文章內(nèi)容
except Exception as e:
pass
comments_count = card.get("mblog").get("comments_count") # 評(píng)論數(shù)
attitudes_count = card.get("mblog").get("attitudes_count") # 點(diǎn)贊數(shù)
reposts_count = card.get("mblog").get("reposts_count") # 分享數(shù)
images = []
try:
for pic in card.get("mblog").get("pics"):
print("image")
images.append(pic.get("url"))
except Exception as e:
print(e)
myCard = {"created_at": created_at, "text": text, "comments_count": comments_count,
"attitudes_count": attitudes_count, "reposts_count": reposts_count, "images": images}
myCards.append(myCard)
except Exception as e:
pass
i = 1
while True:
try:
url = ""
if i == 1:
url = 'https://m.weibo.cn/api/container/getIndex?uid=1333608873&t=0&luicode=10000011&lfid=100103type%3D1%26q%3D%E7%BE%8A%E5%B8%88%E5%A4%AA&type=uid&value=1333608873&containerid=1076031333608873'
else:
url = 'https://m.weibo.cn/api/container/getIndex?uid=1333608873&t=0&luicode=10000011&lfid=100103type%3D1%26q%3D%E7%BE%8A%E5%B8%88%E5%A4%AA&type=uid&value=1333608873&containerid=1076031333608873&page=' + str(
i)
getWeb(url)
i = i + 1
if i >= 1000:
print("結(jié)束")
readerHtml("羊師太", myCards)
break
print(i)
except Exception as e:
print(e)
break
htmlUtil.py
安裝jinja2模塊
pip install jinja2
from jinja2 import Environment, FileSystemLoader
def readerHtml(title,myCards):
env = Environment(loader=FileSystemLoader('./'))
template = env.get_template('util/saveUtil/template.html')
with open("util/saveUtil/result.html", 'wb+') as fout:
html_content = template.render(title=title,cards = myCards)
fout.write(str(html_content).encode("utf-8"))
templates.html
<!DOCTYPE html>
<meta http-equiv="Content-Type"content="text/html;charset=utf-8">
<html align='left'>
<h1>{{title}}</h1>
<body>
<ul >
<hr>
{% for card in cards %}
<li style="float: none;list-style:none">
<h6>創(chuàng)作時(shí)間：{{card.created_at}}</h6>
{{card.text}}
<br>
{%if card.images!=[]%}
<ul style="width: 1200px;height: 300px">
{% for image in card.images %}
<li style="list-style:none;float:left;margin: 5px">
<div style="width: 200px;height:300px;overflow:hidden;">
<img src={{image}} >
</div>
</li>
{% endfor %}
</ul>
{% endif %}
<br>
<p style="float: none">分享數(shù)：{{card.reposts_count}} 評(píng)論數(shù)：{{card.comments_count}} 點(diǎn)贊數(shù)：{{card.attitudes_count}}</p>
</li>
<hr>
{% endfor %}
</ul>
</body>
</html>
這個(gè)templates.html是我們生成網(wǎng)頁(yè)的模板，爬到的數(shù)據(jù)會(huì)加載到模板中，生成一個(gè)result.html
這里只顯示一部分
在瀏覽器中打開(kāi)，這里只截取了一部分
這是未登錄情況下的爬取，有些數(shù)據(jù)會(huì)爬取不到，登錄的情況還在研究中
