Some critical information has been crawled from a website. The website is as below:

The information we need is "item title", "image", "review number", "price", and "star". The result is shown here:

The general process for the web crawling could be described as below (from the course website) :

1) The html file could be read (r) or write (w) from open() function. There are two ways:?

(1) file = open('absolute or relative file path','r'); ? ?print(file.read()); ? ?file.close()

(2) with open('absolute or relative file path','r') as file: ? print(file.read())

2) A special, unique label information (i.e., css path) should be identified in the html file. The relevant commands are: inspect and copy selector. ?

2) One example of the css path looks like:?

"body > div > div > div.col-md-9 > div > div > div > div.ratings > p:nth-of-type(2)"

? ? Note: "nth-child" should be changed for "nth-of-type(n)" in BeautifulSoap.?

3) The information, or css path, should be incorporated in soup.select('css path') to get the result list:

"stars = soup.select('body > div > div > div.col-md-9 > div > div > div > div.ratings > p:nth-of-type(2)')"

The "starts" is a list.?

4) In order to get a single result from the list, we could use zip() function and for "for" "in" structure, to iterate through the "zipped" lists:

"for title,image,review,price,star in zip(titles,images,reviews,prices,stars):"

5) Use get_text(), get('src'), or get("href") functions to retrieve the desired content from the tag.?

data = {

'title': title.get_text(), ? ? ? ? ? ? ? # 使用get_text()方法取出文本

'image': image.get('src'), ? ? ? ? # 使用get 方法取出帶有src的圖片鏈接

'review': review.get_text(),

'price': price.get_text(),

'star':len(star.find_all("span",class_='glyphicon glyphicon-star'))*'★' ? ? ? ? ??

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?# 使用find_all 統(tǒng)計(jì)有幾處是★的樣式 ? ? ? ?

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? # 由于find_all()返回的結(jié)果是列表,我們?cè)偈褂胠en()方法去計(jì)算列表中的元素個(gè)數(shù),也就是星星的數(shù)量

}