構(gòu)建抽象層，然后逐層往下細(xì)化。自頂向下。分優(yōu)先級(jí)。
爬蟲(chóng)應(yīng)用：
接受一個(gè)網(wǎng)頁(yè)，PARSE 內(nèi)容，根據(jù)里面的HTML生成新的任務(wù)。
如何去重任務(wù)？ 單機(jī)的話，數(shù)據(jù)量小可以用SET，大的話LOCAL set DB去重。多臺(tái)機(jī)器用URL的HASH去做PARTITION。
寫B(tài)FS code for a simple web crawler
multi thread version
to design a more practical system
瓶頸：
網(wǎng)絡(luò)帶寬
計(jì)算資源去PARSE網(wǎng)頁(yè)。
隊(duì)列SHARDED BY URL。 每個(gè)爬蟲(chóng)只去一個(gè)隊(duì)列取任務(wù)。
常規(guī)PATTERN
1.經(jīng)典方法去分發(fā)JOB給不同WORKER
shard with (hash, range)
batch - group thing together
2.hash table
3.push/pull model
4.handle failure
master/salve
replication
5.cache
write through(寫CACHE，DB在一起，都完成再返回寫完成）
write around （先寫DB，當(dāng)讀CACHE MISS的時(shí)候，LOAD INTO CACHE）
write back
- database
ACID
sharding
7.knwoledge
network -> disk -> in memory -> l2 cache -> l1 cache
gist.github.com/jboner/2841832/
synchronous/ Asynchronous call
blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/-
stateful vs stateless
stateless service is great because can be replicated as many as you want - best scenario for scaliablity - perfect for computational task
easy to handle client retrysharded stateful service is hard to bring up new instance. because you need to redistribute data/ internal states
highscalability.com
aosabook.org/en/distsys.html
