上一篇文章Hadoop之編寫WordCount我們在本地搭建的Hadoop運行環(huán)境，并在本地運行成功，這篇主要是在上篇的基礎(chǔ)上將編寫好的WordCount程序打成可執(zhí)行jar，并在集群上運行。如果你還沒有集群環(huán)境參考Hadoop集群環(huán)境搭建（三臺）搭建即可

主要內(nèi)容：

• 1.修改Job的數(shù)據(jù)輸入和輸出文件夾
• 2.打成可執(zhí)行jar
• 3.提交集群并運行

相關(guān)文章：
1.VM12安裝配置CentOS7
2.Hadoop集群環(huán)境搭建（三臺）
3.Hadoop之本地運行WordCount
4.Hadoop之集群運行WordCount
5.Log4j2+Flume+Hdfs日志采集

1.修改Job的數(shù)據(jù)輸入和輸出文件夾

由于前面是在本地運行，所以輸入文件和輸出文件夾都指定在本地

FileInputFormat.setInputPaths(job, "D:\\hadoop\\input");
FileOutputFormat.setOutputPath(job, new Path("D:\\hadoop\\output"));

現(xiàn)在修改為Hdfs上的路徑

FileInputFormat.setInputPaths(job, "/input/words.txt");
FileOutputFormat.setOutputPath(job, new Path("/output/wc"));

提前將words.txt上傳到hdfs上的input目錄下

2.將WordCount打成可執(zhí)行jar

用Maven打包，在pom.xml里添加如下：

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <excludes>
                    <!-- 過濾指定的文件 -->
                    <exclude>org/**</exclude>
                </excludes>
                <archive>
                    <manifest>
                        <addClasspath>true</addClasspath>
                        <classpathPrefix>lib/</classpathPrefix>
                        <!-- 指定運行的主類 -->
                        <mainClass>me.jinkun.mr.wc.RunWcJob</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>

如果使用idea開發(fā)，那么直接在右側(cè)雙擊package即可

image.png

這時在項目的target下會有名為mapreduce-wc-1.0.jar的jar包

image.png

3.將jar提交集群運行

運行如下命令：

hadoop jar mapreduce-wc-1.0.jar

運行結(jié)果如下：

[hadoop@hadoop1 soft-install]$ hadoop jar mapreduce-wc-1.0.jar
18/03/08 17:00:25 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.2.111:8032
18/03/08 17:00:26 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/03/08 17:00:27 INFO input.FileInputFormat: Total input paths to process : 1
18/03/08 17:00:28 INFO mapreduce.JobSubmitter: number of splits:1
18/03/08 17:00:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1520498386048_0001
18/03/08 17:00:29 INFO impl.YarnClientImpl: Submitted application application_1520498386048_0001
18/03/08 17:00:29 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1520498386048_0001/
18/03/08 17:00:29 INFO mapreduce.Job: Running job: job_1520498386048_0001
18/03/08 17:00:40 INFO mapreduce.Job: Job job_1520498386048_0001 running in uber mode : false
18/03/08 17:00:40 INFO mapreduce.Job:  map 0% reduce 0%
18/03/08 17:00:47 INFO mapreduce.Job:  map 100% reduce 0%
18/03/08 17:00:56 INFO mapreduce.Job:  map 100% reduce 100%
18/03/08 17:00:56 INFO mapreduce.Job: Job job_1520498386048_0001 completed successfully
18/03/08 17:00:56 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=100
                FILE: Number of bytes written=237705
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=146
                HDFS: Number of bytes written=39
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4342
                Total time spent by all reduces in occupied slots (ms)=6070
                Total time spent by all map tasks (ms)=4342
                Total time spent by all reduce tasks (ms)=6070
                Total vcore-milliseconds taken by all map tasks=4342
                Total vcore-milliseconds taken by all reduce tasks=6070
                Total megabyte-milliseconds taken by all map tasks=4446208
                Total megabyte-milliseconds taken by all reduce tasks=6215680
        Map-Reduce Framework
                Map input records=4
                Map output records=8
                Map output bytes=78
                Map output materialized bytes=100
                Input split bytes=100
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=100
                Reduce input records=8
                Reduce output records=5
                Spilled Records=16
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=206
                CPU time spent (ms)=1430
                Physical memory (bytes) snapshot=300036096
                Virtual memory (bytes) snapshot=4156841984
                Total committed heap usage (bytes)=141660160
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=46
        File Output Format Counters
                Bytes Written=39

查看結(jié)果：
在Hdfs的webui里可以看到如下結(jié)果

image.png

其中part-r-00000里就存放的計算結(jié)果。

到此我們介紹了2種運行mapreduce的方式，一種本地模式便于本地調(diào)試，一種集群模式用于生產(chǎn)環(huán)境。