結(jié)果

這次先說結(jié)果吧,截止到目前(2019-01-28)位置總共爬了網(wǎng)易公開課4296個訂閱號(有的訂閱號是沒有內(nèi)容的),409030條內(nèi)容(視頻或者文章),243413個視頻集合,內(nèi)容里面的視頻和視頻集合里的視頻是有重疊的.分別保存到open163_subscribe,open163_content,open163,3個collection的結(jié)構(gòu)如下

<img width="800" src="https://user-images.githubusercontent.com/9245002/51821939-b0dbf000-2315-11e9-82dd-d09ae240cf7e.png">

<img width="800" src="https://user-images.githubusercontent.com/9245002/51821942-b1748680-2315-11e9-9c20-138716f838d2.png">

<img width="800" alt="20190128144850" src="https://user-images.githubusercontent.com/9245002/51821940-b0dbf000-2315-11e9-8f1b-35bff5cbf355.png">

數(shù)據(jù)抓取過程

• 首先,網(wǎng)易公開課的內(nèi)容都是通過訂閱號發(fā)布的,所以我先抓取了所有訂閱號的信息
  <img width="500" src="https://user-images.githubusercontent.com/9245002/51828496-a9711280-2326-11e9-8b94-17fc151d0d3b.jpeg">

通過在瀏覽器里調(diào)試訂閱號的首頁,可以發(fā)現(xiàn)獲取訂閱號的接口:https://c.open.163.com/open/mob/subscribe/detail/info.do?subscribeId=
集合open163_subscribe保存的就是這個接口返回的信息.而且還發(fā)現(xiàn)訂閱號的唯一標(biāo)識subscribeId是數(shù)字的形式,所以只要從1開始自增就可以遍歷所有的訂閱號了(當(dāng)然有的subscribeId是沒有訂閱號的,有可能是下架了,跳過就好)

def crawler_all_subscribe():
    empty = 0
    subscribeId = 4080
    while empty < 100:
        subscribe = crawler_subscribe(subscribeId)
        if 'subscribeName' in subscribe.keys():
            print(subscribe['subscribeName'])
            insert_subscribe(subscribe)
            crawler_content(subscribeId)
            empty = 0
        else:
            empty = empty + 1
        subscribeId = subscribeId + 1

<img width="1369" alt="20190128203135" src="https://user-images.githubusercontent.com/9245002/51828561-cc032b80-2326-11e9-9a1a-aec8855228e0.png">

• 同時在訂閱號首頁還有一個列出推送內(nèi)容的接口:
  https://c.open.163.com/open/mob/subscribe/detail/list.do?subscribeId=3994&rtypes=2%2C3%2C4%2C5%2C6%2C8%2C9%2C10%2C11%2C12&cursor=&pagesize=10
  這里使用游標(biāo)cursor來分頁,最新一頁cursor為空,每次請求結(jié)果會返回下一頁的cursor,如不返回則為最后一頁.接口返回的data字段(數(shù)組)即為存入open163_content中的內(nèi)容.
  <img width="1200" alt="20190128203135" src="https://user-images.githubusercontent.com/9245002/51836606-cb28c480-233b-11e9-998b-dc5fd7523389.png">
  其中,如果內(nèi)容是視頻類型的話,還會有一個plid字段,這個plid即為視頻集合的唯一標(biāo)識,通過抓取所有訂閱號的所有內(nèi)容,就可以獲得所有視頻集合的plid.

def crawler_content(subscribeId):
    content = []
    cursor = ''
    result = crawler_content_page(subscribeId, cursor)
    while 'data' in result.keys() and 'cursor' in result.keys():
        content = content + result['data']
        cursor = result['cursor']
        result = crawler_content_page(subscribeId, cursor)
    if 'data' in result.keys():
        content = content + result['data']
    print(len(content))
    insert_contents(content)


def crawler_content_page(subscribeId, cursor=''):
    # print(subscribe_url_prefix)
    url = content_url_prefix + '&subscribeId=' + str(
        subscribeId) + '&cursor=' + cursor
    response = urllib.request.urlopen(url)
    result = response.read().decode('utf-8')
    content = json.loads(result)
    if content is None or 'data' not in content.keys():
        return {}
    return content

• 有了視頻集合的plid,就可以通過這個接口來獲取視頻信息了:https://c.open.163.com/mob/${plid}/getMoviesForAndroid.do
  <img width="1353" alt="20190128210823" src="https://user-images.githubusercontent.com/9245002/51838315-e34f1280-2340-11e9-9966-8f84fd1ed186.png">
  返回結(jié)果中的data即為存入open163中的內(nèi)容.

def insert_movies(plid):
    try:
        response = urllib.request.urlopen(movies_url_prefix + plid + movies_url_suffix)
        result = response.read().decode('utf-8')
        data = json.loads(result)
        if data['code'] == 200:
            try:
                collection.insert_one(data['data'])
            except pymongo.errors.DuplicateKeyError:
                pass
    except:
        print(plid)

分析

收集了這么多數(shù)據(jù),可以用來做一些分析了,比如,可以看一下那些訂閱號最受歡迎:

<img width="800" src="https://user-images.githubusercontent.com/9245002/51838939-b26fdd00-2342-11e9-8682-4063c841b3d9.png">

可以看出,訂閱數(shù)最多的都是一些subscribeId比較小的,這可能是因?yàn)樽铋_始推出的訂閱號都是一些競品,審核比較嚴(yán)格,還有就是時間的積累,訂閱用戶因此比較多.

然后,我們還可以看到有不少訂閱號是沒有發(fā)布內(nèi)容的:
<img width="800" src="https://user-images.githubusercontent.com/9245002/51839004-e5b26c00-2342-11e9-819e-e6132817194b.png">

可以統(tǒng)計出最火(瀏覽次數(shù)最多)的內(nèi)容,結(jié)果如下(截至2019-02-28)

關(guān)于播放次數(shù),這里給出的和APP里的是一致的,但是網(wǎng)頁版的播放次數(shù)以及評論數(shù)量都要比APP上高很多,這個具體是怎么計算的不得而知.