最近打算做個(gè)關(guān)于手機(jī)推薦的分析，以京東在售手機(jī)為樣本。話說(shuō)以前也爬過(guò)京東的數(shù)據(jù)，但是二級(jí)頁(yè)面選擇器類型都是簡(jiǎn)單的text，本次想要抓取二級(jí)頁(yè)面中的店鋪名稱、好評(píng)率和評(píng)價(jià)標(biāo)簽，頁(yè)面需要滾動(dòng)下拉才能顯示完全的數(shù)據(jù)，因此涉及到在二級(jí)頁(yè)面中element sroll dowm的使用。鏈接地址：【手機(jī)手機(jī)手機(jī)】?jī)r(jià)格_圖片_品牌_怎么樣-京東商城
一、分析網(wǎng)站規(guī)則
1、起始頁(yè)面的數(shù)據(jù)可以顯示完全
2、分頁(yè)時(shí)，網(wǎng)址不變化，需要click點(diǎn)擊翻頁(yè)
3、從起始頁(yè)面link進(jìn)入二級(jí)頁(yè)面后，需要滾動(dòng)下拉才能顯示完整數(shù)據(jù)
因此確定抓取數(shù)據(jù)的方法：element click+link+element sroll down+text
二、sitemap建立
從圖中可看出，我設(shè)定了list、link、sroll down三個(gè)選擇器為串聯(lián)關(guān)系，其中scroll down是為了滾動(dòng)下拉輔助顯示數(shù)據(jù)，其余子選擇器類型均為text，為真正抓取數(shù)據(jù)的子選擇器，抓取數(shù)據(jù)維度有手機(jī)名稱、價(jià)格、評(píng)價(jià)人數(shù)、店鋪名、好評(píng)率、評(píng)價(jià)標(biāo)簽6個(gè)方面的信息。
需要注意的是：scroll down中必須設(shè)置delay，推薦2000ms，我剛開(kāi)始這里沒(méi)有設(shè)置delay導(dǎo)致好評(píng)率和評(píng)價(jià)標(biāo)簽沒(méi)爬到，就跳轉(zhuǎn)到下個(gè)頁(yè)面了。
代碼如下：
{"startUrl":"https://www.jd.com/chanpin/127371.html","selectors":[{"parentSelectors":["_root"],"type":"SelectorElementClick","multiple":true,"id":"list","selector":"div.gl-i-wrap","delay":"2000","clickElementSelector":"a.pn-next em","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","discardInitialElements":false},{"parentSelectors":["list"],"type":"SelectorText","multiple":false,"id":"price","selector":"div.p-price","regex":"","delay":""},{"parentSelectors":["list"],"type":"SelectorText","multiple":false,"id":"pingjianum","selector":"div.p-commit","regex":"","delay":""},{"parentSelectors":["list"],"type":"SelectorLink","multiple":false,"id":"link","selector":"div.p-name a","delay":""},{"parentSelectors":["link"],"type":"SelectorElementScroll","multiple":false,"id":"scroll down","selector":"div#J-global-toolbar","delay":"2000"},{"parentSelectors":["link"],"type":"SelectorText","multiple":false,"id":"store","selector":"div.popbox-inner div.mt","regex":"","delay":""},{"parentSelectors":["link"],"type":"SelectorText","multiple":false,"id":"percent","selector":"div.comment-percent","regex":"","delay":""},{"parentSelectors":["link"],"type":"SelectorText","multiple":false,"id":"label","selector":"div.tag-list","regex":"","delay":""}],"_id":"shouji2"}
三、數(shù)據(jù)預(yù)覽
設(shè)定好參數(shù)后就可以坐等結(jié)果了，預(yù)覽如下：
手機(jī)分析過(guò)程請(qǐng)關(guān)注后續(xù)發(fā)布哦~~~
