現(xiàn)在數(shù)據(jù)倉(cāng)庫(kù)基本上采用Hadoop平臺(tái)了，那么數(shù)據(jù)倉(cāng)庫(kù)里面元數(shù)據(jù)的血緣分析的思路有哪些呢

基本上有下面這兩種思路：

1、解析hql腳本，通過(guò)正則表達(dá)式去匹配每一行字符串

2、采用Hadoop自帶的語(yǔ)法分析類解析

這里比較建議采用第二種，比較直接簡(jiǎn)單，因?yàn)榈谝环N方式比較復(fù)雜，需要考慮場(chǎng)景比較多，容易出現(xiàn)遺漏

Hadoop 自帶的類 org.apache.hadoop.hive.ql.tools.LineageInfo

將hql語(yǔ)句通過(guò)解析語(yǔ)法tree，獲取hive表的源表和目標(biāo)表，達(dá)到血緣分析的目的

但是這個(gè)類有一點(diǎn)缺陷就是對(duì)于create table xx as 這種hql語(yǔ)句無(wú)法解析

我們稍加修改代碼就可以解決了

代碼如下：

package com.neo.datamanager;

import org.apache.hadoop.hive.ql.lib.*;
import org.apache.hadoop.hive.ql.parse.*;

import java.io.IOException;
import java.util.*;

public class HiveLineageInfo implements NodeProcessor {

//    private static final Logger logger = LoggerFactory.getLogger(HiveLineageInfo.class);

    /**
     * Stores input tables in sql.
     */
    TreeSet inputTableList = new TreeSet();
    /**
     * Stores output tables in sql.
     */
    TreeSet OutputTableList = new TreeSet();

    /**
     * @return java.util.TreeSet
     */
    public TreeSet getInputTableList() {
        return inputTableList;
    }

    /**
     * @return java.util.TreeSet
     */
    public TreeSet getOutputTableList() {
        return OutputTableList;
    }

    /**
     * Implements the process method for the NodeProcessor interface.
     */
    public Object process(Node nd, Stack stack, NodeProcessorCtx procCtx,
                          Object... nodeOutputs) throws SemanticException {
        ASTNode pt = (ASTNode) nd;

        switch (pt.getToken().getType()) {

            case HiveParser.TOK_CREATETABLE:
                OutputTableList.add(BaseSemanticAnalyzer.getUnescapedName((ASTNode) pt.getChild(0)));
                break;
            case HiveParser.TOK_TAB:
                OutputTableList.add(BaseSemanticAnalyzer.getUnescapedName((ASTNode) pt.getChild(0)));
                break;

            case HiveParser.TOK_TABREF:
                ASTNode tabTree = (ASTNode) pt.getChild(0);
                String table_name = (tabTree.getChildCount() == 1) ?
                        BaseSemanticAnalyzer.getUnescapedName((ASTNode) tabTree.getChild(0)) :
                        BaseSemanticAnalyzer.getUnescapedName((ASTNode) tabTree.getChild(0)) + "." + tabTree.getChild(1);
                inputTableList.add(table_name);
                break;
        }
        return null;
    }

    /**
     * parses given query and gets the lineage info.
     *
     * @param query
     * @throws ParseException
     */
    public void getLineageInfo(String query) throws ParseException,
            SemanticException {

    /*
     * Get the AST tree
     */
        ParseDriver pd = new ParseDriver();
        ASTNode tree = pd.parse(query);

        while ((tree.getToken() == null) && (tree.getChildCount() > 0)) {
            tree = (ASTNode) tree.getChild(0);
        }

    /*
     * initialize Event Processor and dispatcher.
     */
        inputTableList.clear();
        OutputTableList.clear();

        // create a walker which walks the tree in a DFS manner while maintaining
        // the operator stack. The dispatcher
        // generates the plan from the operator tree
        Map<Rule, NodeProcessor> rules = new LinkedHashMap<Rule, NodeProcessor>();

        // The dispatcher fires the processor corresponding to the closest matching
        // rule and passes the context along
        Dispatcher disp = new DefaultRuleDispatcher(this, rules, null);
        GraphWalker ogw = new DefaultGraphWalker(disp);

        // Create a list of topop nodes
        ArrayList topNodes = new ArrayList();
        topNodes.add(tree);
        ogw.startWalking(topNodes, null);
    }

    public static void main(String[] args) throws IOException, ParseException, SemanticException {
        String query = "insert into table aa  select * from bb union all select * from cc";
        HiveLineageInfo lep = new HiveLineageInfo();
        lep.getLineageInfo(query);
        System.out.println("Input tables = " + lep.getInputTableList());
        System.out.println("Output tables = " + lep.getOutputTableList());
    }
}

運(yùn)行之后結(jié)果如下: