To Be Continued...

Brief Intro to Airflow

Airflow is a platform to programmatically author, schedule and monitor workflows like Oozie which was more famous in workflow industry.Airflow is a incubating project which is very new .But the project has not been hidden for his very huge advantage.

Airflow Advantages:

1. airflow is developed by Python .it destined that airflow is well maintained and second-developed
2. airflow has nice UI for controlling,displaying,monitoring workflow
3. airflow has been running in the backend of Electron Project which is bigddata log analyser application in Youzu .the airflow has been proved to be stable and smooth

Installation

airflow needs a home, ~/airflow is the default,

but you can lay foundation somewhere else if you prefer

(optional)

export AIRFLOW_HOME=~/airflow

install from pypi using pip

pip install airflow
pip install airflow[mysql]

initialize the database

airflow initdb

start the web server, default port is 8080

airflow webserver -p 8080

install celery when you intend to use celery executor

pip install airflow[celery]

Airflow Case

alter two lines in airflow.cfg

executor = CeleryExecutor

store metadata using mysql

sql_alchemy_conn = mysql://username:password@ipaddress/dbname?charset=utf8

start airflow webserver,airflow celery worker

airflow webserver
airflow worker

write dag file in dag_folder which can be modify in airflow.cfg setting file

eg: $AIRFLOW_HOME/dags/example.py #example.py is dag file

submit dag file to airflow for generating airflow task

airflow trigger_dag example #example is #eg: $AIRFLOW_HOME/dags/example.py

you can see example dag task in webserver 0.0.0.0:8080/admin/dagrun/