CBA聯(lián)賽正進(jìn)行得如火如荼，而20支CBA球隊(duì)各自的陣容深度究竟幾何？
Scrapy框架 + MongoDB，獲取CBA中國男籃所有球隊(duì)球員的基本信息，以便往后的數(shù)據(jù)分析使用。

開發(fā)環(huán)境

• python3.7
• Scrapy框架及其組件
• json模塊
• pymongo模塊

獲取分析：

1.獲取球隊(duì)鏈接
獲取球隊(duì)鏈接的網(wǎng)頁是通過Ajax技術(shù)異步加載得到。
通過抓包發(fā)現(xiàn)，所需的數(shù)據(jù)正是json格式，是理想中的情況。

#解析球隊(duì)鏈接：
def parse(self, response):    
        club0 = json.loads(response.text)
        clubs = club0['data']
        baseurl = "https://api-all.9h-sports.com/cba-data/api/cba/v1/league/player-history?clubId={}"
        for oneclub in clubs:
            clubname = oneclub['name']
            clubid = oneclub['club_id']
            cluburl = baseurl.format(clubid)
            print (clubname + cluburl)
            yield scrapy.Request(url=cluburl,callback=self.parsecluburl,dont_filter=True,meta={'clubname':clubname}) 

2.進(jìn)入各支球隊(duì)URL鏈接頁面，獲取球隊(duì)名單下所有球員的數(shù)據(jù)。
通過抓包發(fā)現(xiàn)，所需要的球員數(shù)據(jù)同樣是通過Ajax技術(shù)異步加載得到，且數(shù)據(jù)格式也和上一個頁面相同，還是json格式的數(shù)據(jù)。

#解析球員數(shù)據(jù)
def parsecluburl(self,response0):
        players0 = json.loads(response0.text)
        players = players0['data']
        for player in players:
...

                allitem = items.CbaplayerItem( 
                          playername = playername,
                          playernumber = playernumber,
                          playercountry = playercountry,
                          playerposition = playerposition,
                          playerheight = playerheight,
                          playerweight = playerweight,
                          playerbirth = playerbirth,
                          playerclub = playerclub
                          )
                yield allitem

最后將獲取到的所有球員的數(shù)據(jù)通過pipelines組件存入MongoDB數(shù)據(jù)庫。

#pipelines入庫
import pymongo
import json

class CbaplayerPipeline(object):
    def __init__(self):
        self.conn = pymongo.MongoClient(host='127.0.0.1',port=27017) #連接數(shù)據(jù)庫
        self.dbb = self.conn.cbaplayers #創(chuàng)建數(shù)據(jù)庫
        self.dbbcc = self.dbb.cbaplayer0 #創(chuàng)建集合

    def process_item(self, item, spider):
        item = dict(item)
        self.dbbcc.insert(item)
        return item

結(jié)果展示：

廣東集合！