0.開發(fā)環(huán)境概述

客戶端環(huán)境

Windows 7 64位
Oracle JDK8 64位
Eclipse4.7

服務(wù)器偽分布式安裝部署Hadoop3

1.Windows平臺(tái)下Hadoop客戶端運(yùn)行環(huán)境搭建

-下載winutils

-解壓縮到任意文件夾下

-新建環(huán)境變量HADOOP_HOME

-在環(huán)境變量PATH中添加%HADOOP_HOME%\bin

2.建立客戶端工程

-新建Maven項(xiàng)目，POM文件如下:

<dependencies>
        <dependency>
            <groupId>jdk.tools</groupId>
            <artifactId>jdk.tools</artifactId>
            <version>1.8</version>
            <scope>system</scope>
            <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs-client</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.0.0</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

-拷貝服務(wù)器端上的core-site.xml和hdfs-site.xml到maven工程的src文件夾下，開始碼代碼；

-建立與服務(wù)器端Hadoop的連接

    private static Configuration conf = new Configuration();
    private FileSystem hdfs;
    private final static String HDFSUri = "hdfs://192.168.1.111:9001";
    public FileSystem initHDFS() throws URISyntaxException {
        URI uri = new URI(HDFSUri);
        conf.addResource("hdfs-site.xml");
        conf.addResource("core-site.xml");
        try {
            hdfs = FileSystem.get(uri,conf);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return hdfs;
    }

-檢查文件是否存在

    /**
     * 檢查文件是否存在
     * @throws IllegalArgumentException
     * @throws IOException
     */
    @Test
    public void testCheckFile() throws IllegalArgumentException, IOException {
        Path path = new Path("hdfs://192.168.1.111:9001/user/root/test.txt");
        boolean result = hdfsDao.exists(path);
        System.out.println(result?"文件存在":"文件不存在");
    }

-在HDFS上新建文件夾

    /**
     * 新建文件夾
     * @throws IOException
     */
    @Test
    public void testMkdir() throws IOException {
        Path folderPath = new Path("hdfs://192.168.1.133:9000/user/root/tiffData");
        boolean result = hdfsDao.mkdirs(folderPath);
        System.out.println(result?"添加文件夾成功":"添加文件夾失敗");
    }

-刪除文件夾/文件

    /**
     * 刪除文件夾/文件
     * @throws IOException
     */
    @Test
    public void testDeldir() throws IOException {
        Path folderPath = new Path("hdfs://192.168.1.133:9000/user/root/tiffData");
        boolean result = hdfsDao.delete(folderPath, false);
        System.out.println(result?"刪除文件夾成功":"刪除文件夾失敗");
    }

-獲取指定目錄下所有文件(忽略目錄)

/**
     * 獲取指定目錄下所有文件(忽略目錄)
     * @throws FileNotFoundException
     * @throws IllegalArgumentException
     * @throws IOException
     */
    @Test
    public void testGetAllFile() throws FileNotFoundException, IllegalArgumentException, IOException {
        RemoteIterator<LocatedFileStatus> listFiles = hdfsDao.listFiles(new Path("/"), true);
        SimpleDateFormat sdf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
        while (listFiles.hasNext()) {
            LocatedFileStatus fileStatus = (LocatedFileStatus) listFiles.next();
            //權(quán)限
            FsPermission permission = fileStatus.getPermission();
            //擁有者
            String owner = fileStatus.getOwner();
            //組
            String group = fileStatus.getGroup();
            //文件大小byte
            long len = fileStatus.getLen();
            long modificationTime = fileStatus.getModificationTime();
            Path path = fileStatus.getPath();
            System.out.println("-------------------------------");
            System.out.println("permission:"+permission);
            System.out.println("owner:"+owner);
            System.out.println("group:"+group);
            System.out.println("len:"+len);
            System.out.println("modificationTime:"+sdf.format(new Date(modificationTime)));
            System.out.println("path:"+path);
        }
    }

-從HDFS上拷貝文件到本地

    /**
     * 從HDFS上拷貝文件到本地
     */
    @Test
    public void testDownLoadFile() {
        Path src = new Path("hdfs://192.168.1.133:9000/user/root/test.txt");
        Path des = new Path("D:\\workData\\test.txt");
        try {
            hdfsDao.copyToLocalFile(src, des);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

-本地文件上傳

    @Test
    public void testFileUpload() {
        Path src = new Path("D:\\workData\\test1.txt");
        Path des = new Path("hdfs://192.168.1.133:9000/user/root/test1.txt");
        try {
            hdfsDao.copyFromLocalFile(src, des);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }