最近想借著調(diào)度系統(tǒng)這個(gè)項(xiàng)目來(lái)看看業(yè)界出色的調(diào)度系統(tǒng)的源碼，如Oozie，Airflow目錄如下，

Overview

Oozie是雅虎開(kāi)源出來(lái)的一個(gè)出色的工作流，支持很多jobType，spark，email等。主要分為3個(gè)角色，

• server，用于保持長(zhǎng)鏈接，監(jiān)聽(tīng)來(lái)自于client的jobSubmit，然后分發(fā)job到各個(gè)executor上去執(zhí)行，執(zhí)行結(jié)果展示在UI上
• client，與用戶打交道，用戶在部署了client的機(jī)器上直接運(yùn)行cmd，用來(lái)submit cmd，如kill，check，submit
• shardlib(adaptor, executor)：真正的執(zhí)行部件

Version

<groupId>org.apache.oozie</groupId>
<artifactId>oozie-main</artifactId>
<version>5.1.0-SNAPSHOT</version>

Latest Submission

Client

下面看看它的源碼入口，

• bash oozie -> org.apache.oozie.cli.OozieCLI -> run() -> processCommand()
• e.g.: processCommand() -> jobCommand() -> KILL_OPTION -> wc.kill() -> new JobAction() -> call() -> createURL() -> JobAction.call(HttpURLConnection conn) （2個(gè)不同的call function，拼接URL，然后發(fā)送到server）
• e.g.: oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe (這句的oozie對(duì)應(yīng)上句的oozie，即每次運(yùn)行CommandLineTool命令都是java -cp了OozieCLI，只是每次的OozieCLI啟動(dòng)參數(shù)不同而已)

super("PUT", RestConstants.JOB, notEmpty(jobId, "jobId"), prepareParams(RestConstants.ACTION_PARAM, action));

public ClientCallable(String method, String collection, String resource, Map<String, String> params) {
    this(method, null, collection, resource, params);
}

public ClientCallable(String method, Long protocolVersion, String collection, String resource, Map<String, String> params) {
    this.method = method;
    this.protocolVersion = protocolVersion;
    this.collection = collection;
    this.resource = resource;
    this.params = params;
}

URL url = createURL(protocolVersion, collection, resource, params);

sb.append(getBaseURLForVersion(protocolVersion));

CLI

Server

源碼入口，

• bash oozied.sh -> oozie-jetty-server.sh -> org.apache.oozie.server.EmbeddedOozieServer
• embeddedOozieServer.setup() -> oozieServletMapper.mapOozieServlets(); -> mapServlet(V0JobServlet.class, "/v0/job/*"); -> BaseJobServlet.doPut() -> embeddedOozieServer.join()
• server hold till shutdown hook
• e.g.: mapServlet(V0JobServlet.class, "/v0/job/*");, *號(hào)就是Client的jobId，v0就是protocolVersion，job是字符串常量"job"

URL Mapping

shardlib

源碼入口，

• servletHandler.addServlet(new ServletHolder(v1JobsServletName, new V1JobsServlet()));
• EmbeddedOozieServer -> ServletMapper -> V1JobsServlet.submitJob().submitHttpJob() -> dagEngine.submitJob() -> submit.call() -> start(jobId) -> new StartXCommand(jobId).call() -> ...

將自定義的xxx.wf xml翻譯成DAG，然后定時(shí)運(yùn)行。

Actions Supported by Oozie