搭建Hadoop偽分布式環(huán)境-作業(yè)001

搭建Linux環(huán)境

1 配置VM###

2 配置虛擬機磁盤

3 配置CenOS

4 配置磁盤，選擇YES

5 配置主機名

6 配置網(wǎng)絡

這里也可以添加DNS####

7 配置普通用戶

useradd beifeng
passwd beifeng

2016-07-11_104420.jpg

sudo權(quán)限設置###

visudo

2016-07-11_141357.jpg

配置DNS

vim /etc/sysconfig/network-scripts/ifcfg-eth0

2016-07-11_104756.jpg

關(guān)閉selinux###

vim /etc/sysconfig/selinux

2016-07-11_104937.jpg

修改主機名###

vim /etc/hosts
192.168.159.111 bigdata-senior01.beifeng.com 192.168.159.112 bigdata-senior02.beifeng.com 192.168.159.113 bigdata-senior03.beifeng.com

2016-07-11_105224.jpg

關(guān)閉iptables###

service iptables stop
chkconfig iptables off

2016-07-11_105447.jpg

配置目錄，上傳對應的包###

2016-07-11_105703.jpg

chown修改權(quán)限###

chown -R beifeng:beifeng /opt/*

2016-07-11_110004.jpg

卸載openJDK，安裝JDK和MAVEN###

rpm -qa | grep jdk
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64

2016-07-11_110716.jpg

刪除掉： /opt/modules/hadoop-2.5.0/lib/native的包并替換####

刪除掉：/opt/modules/hadoop-2.5.0/share/doc####

2016-07-11_111251.jpg

配置JDK和MAVEN環(huán)境變量###

vim /etc/profile

#JAVA_HOME
export JAVA_HOME=/opt/modules/jdk1.7.0_67
export PATH=$PATH:$JAVA_HOME/bin

#MAVEN_HOME
export MAVEN_HOME=/opt/modules/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

2016-07-11_111950.jpg

source /etc/profile

2016-07-11_112111.jpg

配置hadoop相關(guān)xml文件##

配置*-evn.sh文件export JAVA_HOME###

2016-07-11_112341.jpg

Hadoop-env.sh，yarn-env.sh，mapred-env.sh 這3個文件配置export JAVA_HOME

hadoop-env.sh:

2016-07-11_112715.jpg

yarn-env.sh:

2016-07-11_112834.jpg

mapred-env.sh:

2016-07-11_112929.jpg

配置自定義文件*-site.xml###

     配置yarn-site.xml
            <property>
             <name>yarn.nodemanager.aux-services</name>
              <value>mapreduce_shuffle</value>
            </property>
    
             <property>
 <name>yarn.resourcemanager.hostname</name>
        <value>bigdata-senior01.beifeng.com</value>
    </property>
    
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>106800</value>
    </property>

    配置salves：
    `bigdata-senior01.beifeng.com`

    配置：mapred-site.xml
    <property>
                                                           <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    
    配置：hdfs-site.xml
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
    
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>bigdata-senior01.beifeng.com:50090</value>
    </property>
    配置core-site.xml
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://bigdata-senior01.beifeng.com:8020</value>
    </property>
    
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/modules/hadoop-2.5.0/data/tmp</value>
    </property>
    
    <property>
        <name>hadoop.http.staticuser.user</name>
        <value>beifeng</value>
    </property>