Hadoop 的版本是 3.1.1
1. 啟動(dòng) Hadoop 服務(wù)
$ start-all.sh
2. 新建 IDEA 的 Maven 項(xiàng)目
2.1 選中 Maven，Project SDK 選擇為 1.8，再點(diǎn)擊 Next
點(diǎn)擊 Next
2.2 填寫(xiě)好 GroupId，ArtifactId 后，點(diǎn)擊 Next
2.3 點(diǎn)擊 Finish
image.png
3. 修改 Target bytecode version
打開(kāi) Setting，選中 Build, Execution, Deployment -> Compiler -> java，將 Target bytecode version 改為 1.8 或 8。
Target bytecode version
確認(rèn)這幾個(gè)配置下的 jdk 版本都為 1.8
Project SDK
Module SDK
4. 導(dǎo)入需要的 jar 包
4.1 選中 Dependencies 后點(diǎn)擊下方的 + 號(hào)，選擇「JARs or directories」
添加 jar 包
JARs or directories
4.2 進(jìn)入 Hadoop 目錄下的 share/hadoop/ 中，把這幾個(gè)包都導(dǎo)進(jìn)去
share/hadoop/
選擇 OK
繼續(xù) OK
4.2 在 pom.xml 中添加如下依賴
<dependencies>
<!-- https://mvnrepository.com/artifact/junit/junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!--<!– https://mvnrepository.com/artifact/commons-logging/commons-logging –>-->
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.2</version>
</dependency>
<!--<!– https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common –>-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>3.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.1.1</version>
</dependency>
</dependencies>
5. 編寫(xiě) Hadoop 項(xiàng)目的 Java 代碼
5.1 新建 Java 類「Test.java」
image.png
5.2 編寫(xiě)代碼
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
public class Test {
// 在 HDFS 中新建一個(gè) test 文件夾
public static void main(String[] args) {
FileSystem fileSystem = null;
try {
fileSystem = FileSystem.get(new URI("hdfs://localhost:9000/"),new Configuration(),"binguner");
fileSystem.mkdirs(new Path("/test"));
fileSystem.close();
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (URISyntaxException e) {
e.printStackTrace();
}
}
}
5.3 運(yùn)行 Java 程序
image.png
6. 運(yùn)行結(jié)果
6.1 運(yùn)行前的 HDFS 目錄下沒(méi)有 test 文件夾
6.2 運(yùn)行后的 HDFS 目錄下多了 test 文件夾
7. FileSystem 常用接口
- 7.1
mkdirs
public boolean mkdirs(Path f) throws IOException {
return this.mkdirs(f, FsPermission.getDirDefault());
}
參數(shù)是新的文件夾的路徑，可以在文件夾里嵌套文件夾進(jìn)行創(chuàng)建。
- 7.2
create
public FSDataOutputStream create(Path f) throws IOException {
return this.create(f, true);
}
public FSDataOutputStream create(Path f, boolean overwrite) throws IOException {
return this.create(f, overwrite, this.getConf().getInt("io.file.buffer.size", 4096), this.getDefaultReplication(f), this.getDefaultBlockSize(f));
}
public FSDataOutputStream create(Path f, Progressable progress) throws IOException {
return this.create(f, true, this.getConf().getInt("io.file.buffer.size", 4096), this.getDefaultReplication(f), this.getDefaultBlockSize(f), progress);
}
public FSDataOutputStream create(Path f, short replication) throws IOException {
return this.create(f, true, this.getConf().getInt("io.file.buffer.size", 4096), replication, this.getDefaultBlockSize(f));
}
public FSDataOutputStream create(Path f, short replication, Progressable progress) throws IOException {
return this.create(f, true, this.getConf().getInt("io.file.buffer.size", 4096), replication, this.getDefaultBlockSize(f), progress);
}
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws IOException {
return this.create(f, overwrite, bufferSize, this.getDefaultReplication(f), this.getDefaultBlockSize(f));
}
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws IOException {
return this.create(f, overwrite, bufferSize, this.getDefaultReplication(f), this.getDefaultBlockSize(f), progress);
}
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws IOException {
return this.create(f, overwrite, bufferSize, replication, blockSize, (Progressable)null);
}
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException {
return this.create(f, FsCreateModes.applyUMask(FsPermission.getFileDefault(), FsPermission.getUMask(this.getConf())), overwrite, bufferSize, replication, blockSize, progress);
}
public abstract FSDataOutputStream create(Path var1, FsPermission var2, boolean var3, int var4, short var5, long var6, Progressable var8) throws IOException;
public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException {
return this.create(f, permission, flags, bufferSize, replication, blockSize, progress, (ChecksumOpt)null);
}
public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress, ChecksumOpt checksumOpt) throws IOException {
return this.create(f, permission, flags.contains(CreateFlag.OVERWRITE), bufferSize, replication, blockSize, progress);
}
create 有多個(gè)重載函數(shù)，它的參數(shù)可以指定是否覆蓋已有的文件、文件備份數(shù)量、寫(xiě)入文件緩沖區(qū)大小、文件塊大小以及文件權(quán)限。它的返回值是一個(gè) FSDataOutputStream，通過(guò)返回的 FSDataOutputStream 對(duì)象可以對(duì)文件進(jìn)行寫(xiě)入。
- 7.3
copyFromLocal
public void copyFromLocalFile(Path src, Path dst) throws IOException {
this.copyFromLocalFile(false, src, dst);
}
public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws IOException {
this.copyFromLocalFile(delSrc, true, src, dst);
}
public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst) throws IOException {
Configuration conf = this.getConf();
FileUtil.copy(getLocal(conf), srcs, this, dst, delSrc, overwrite, conf);
}
將本地文件拷貝到文件系統(tǒng)，參數(shù)可以指定上傳本地文件的路徑，上傳的多個(gè)路徑組成的 Path 數(shù)組，存放目標(biāo)對(duì)路徑，可以指定是否刪除本地本地的文件或者覆蓋 hdfs 上已經(jīng)創(chuàng)建的文件。
- 7.4
copyToLocalFile
public void copyToLocalFile(Path src, Path dst) throws IOException {
this.copyToLocalFile(false, src, dst);
}
public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws IOException {
this.copyToLocalFile(delSrc, src, dst, false);
}
將目標(biāo)文件復(fù)制到本地指定路徑，delSrc 參數(shù)指定移動(dòng)文件后是否要?jiǎng)h除源文件。
- 7.6
moveToLocalFile
public void moveToLocalFile(Path src, Path dst) throws IOException {
this.copyToLocalFile(true, src, dst);
}
將目標(biāo)文件移動(dòng)到指定路徑，函數(shù)內(nèi)部調(diào)用的是 copyToLocalFile。
- 7.6
exists
public boolean exists(Path f) throws IOException {
try {
return this.getFileStatus(f) != null;
} catch (FileNotFoundException var3) {
return false;
}
}
輸入一個(gè)路徑，檢查 HDFS 上是否存在這個(gè)路徑，存在返回 true，不存在返回 false。
- 7.7
delete
public abstract boolean delete(Path var1, boolean var2) throws IOException;
第一個(gè)參數(shù)是要?jiǎng)h除的路徑，第二個(gè)參數(shù)為 true 時(shí)，如果目標(biāo)文件夾內(nèi)有文件，會(huì)強(qiáng)制刪除。
