日韩在线直播,欧美大陆成人中出

安裝前準(zhǔn)備

Homebrew

jdk安裝

java -version

java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

如果沒(méi)有安裝, 建議安裝Java8

brew cask install java # 安裝最新版本
# 安裝Java8
brew tap caskroom/versions
brew cask install java8

配置ssh

配置ssh就是為了實(shí)現(xiàn)免密登錄, 這樣方便遠(yuǎn)程管理Hadoop并無(wú)需登錄密碼在Hadoop集群上共享文件資源

如果你的機(jī)子沒(méi)有配置ssh, 在命令終端輸入ssh localhost是需要輸入你的電腦登錄密碼的.
配置好ssh后, 就無(wú)需輸入密碼了.

打開(kāi)設(shè)置 > 共享 > 打開(kāi)遠(yuǎn)程登陸

image.png
iterm(終端)執(zhí)行

ssh-keygen -t rsa # 然后yes
cat ~/.ssh/local.pub >> ~/.ssh/authorized_keys

現(xiàn)在, 在終端輸入 ssh localhost就OK了.

ssh localhost # ssh 登陸
# Last login: Fri Jan 18 14:44:36 2019
exit # 退出登陸
# Connection to localhost closed.

安裝hadoop

下載安裝

brew install hadoop (推薦), 安裝完成后你會(huì)看到安裝路徑在那里
官網(wǎng)下載壓縮包, 解壓到你指定的目錄, 然后安裝(不推薦)

配置hadoop

配置hadoop-env.sh

hadoop-env.sh位置:

/usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop

添加JAVA_HOME路徑

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_202.jdk/Contents/Home 
# Mac查看jdk 位置 /usr/libexec/java_home -V

配置core-site.xml

配置hdfs地址和端口

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:8020</value>
  </property>
  <!-- 以下配置可防止系統(tǒng)重啟導(dǎo)致NameNode 不能啟動(dòng)-->
  <!-- /Users/用戶名/data 這個(gè)路徑你可以隨便配置, hadoop必須有權(quán)限-->
  <property>  
    <name>hadoop.tmp.dir</name>  
    <value>/Users/用戶名/data/hadoop/tmp</value>  
    <description>A base for other temporary directories.</description>  
  </property> 
  <!-- DataNode存放塊數(shù)據(jù)的本地文件系統(tǒng)路徑 -->
  <property>  
    <name>dfs.name.dir</name>  
    <value>/Users/用戶名/data/hadoop/filesystem/name</value>  
    <description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description>  
  </property>  
  <property>  
    <name>dfs.data.dir</name>  
    <value>/Users/用戶名/data/hadoop/filesystem/data</value>  
    <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.</description>  
  </property>  
</configuration>

配置hdfs-site.xml

修改HDFS備份數(shù), 配置namenode和datanode

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

配置mapred-site.xml

配置mapreduce中jobtracker的地址和端口. 3.1.1版本下有這個(gè)文件, 可直接配置

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

配置yarn-site.xml

<configuration> 
  <property> 
    <name>yarn.nodemanager.aux-services</name> 
    <value>mapreduce_shuffle</value> 
  </property> 
  <property> 
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> 
  </property> 
</configuration>

格式化HDFS

# /usr/local/Cellar/hadoop/3.1.1/libexec
bin/hdfs namenode -format

運(yùn)行

sbin/start-all.sh: 啟動(dòng)全部
sbin/start-dfs.sh: 啟動(dòng)NameNode和DataNode, NameNode - http://localhost:9870 (2.7的版本端口為50070)
sbin/start-yarn.sh: ResourceManager - http://localhost:8088 (All Applications)

jps 可以查看進(jìn)程

jps
# 34214 NameNode
# 34313 DataNode
# 34732 NodeManager
# 34637 ResourceManager
# 34446 SecondaryNameNode
# 34799 Jps

安裝hive

下載安裝

brew install hive

配置Hive元數(shù)據(jù)庫(kù)

Hive默認(rèn)用derby作為元數(shù)據(jù)庫(kù)這, 我們這里換用大家熟悉的mysql來(lái)存儲(chǔ)元數(shù)據(jù)

# 進(jìn)入數(shù)據(jù)庫(kù)
mysql -uroot -p 
# 在數(shù)據(jù)庫(kù)執(zhí)行
CREATE DATABASE metastore;
# CREATE user 'hive'@'localhost' IDENTIFIED BY 'hive';
# Unable to load authentication plugin 'caching_sha2_password'.
ALTER USER 'hive'@'localhost' IDENTIFIED WITH mysql_native_password BY 'hive';
GRANT SELECT,INSERT,UPDATE,DELETE,ALTER,CREATE,INDEX,REFERENCES ON METASTORE.* TO 'hive'@'localhost';
FLUSH PRIVILEGES;

配置hive

配置mysql-connector jar包

下載地址: https://dev.mysql.com/downloads/connector/j/
將下載的文件解壓, 復(fù)制

cp mysql-connector-java-5.1.44-bin.jar /usr/local/Cellar/hive/3.1.1/libexec/lib/

配置hive-site.xml

修改以下部分

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/metastore</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive(配置Hive元數(shù)據(jù)庫(kù): mysql中創(chuàng)建的用戶名)</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>hive(配置Hive元數(shù)據(jù)庫(kù): mysql中創(chuàng)建的用戶密碼)</value>
</property>
<property>
  <name>hive.exec.local.scratchdir</name>
  <value>/Users/用戶名/data/hive</value>
</property>
<property>
  <name>hive.querylog.location</name>
  <value>/Users/用戶名/data/hive/querylog</value>
</property>
<property>
  <name>hive.downloaded.resources.dir</name>
  <value>/Users/用戶名/data/hive/download</value>
</property>
<property>
  <name>hive.server2.logging.operation.log.location</name>
  <value>/Users/用戶名/data/hive/log</value>
</property>

注意3210行可能會(huì)有?最好刪掉, 不然在初始化元數(shù)據(jù)庫(kù)會(huì)報(bào)錯(cuò)

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/3.1.1/libexec/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/usr/local/Cellar/hive/3.1.1/libexec/conf/hive-site.xml"]
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
    at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
    at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5098)
    at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
    at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/usr/local/Cellar/hive/3.1.1/libexec/conf/hive-site.xml"]
    at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
    at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
    at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
    at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
    at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
    at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
    at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
    at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
    at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
    at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
    ... 15 more

初始化元數(shù)據(jù)庫(kù)

schematool -initSchema -dbType mysql

現(xiàn)在進(jìn)入數(shù)據(jù)庫(kù)metastore, 可以看到相關(guān)表(此處只做部分表展示)

mysql> show tables;
+-------------------------------+
| Tables_in_metastore           |
+-------------------------------+
| AUX_TABLE                     |
| BUCKETING_COLS                |
| CDS                           |
| COLUMNS_V2                    |
| COMPACTION_QUEUE              |
| COMPLETED_COMPACTIONS         |
| COMPLETED_TXN_COMPONENTS      |
| CTLGS                         |
| DATABASE_PARAMS               |
| DB_PRIVS                      |

運(yùn)行Hive

安裝spark

brew install apache-spark

一般直接安裝就好了, 然后直接運(yùn)行spark-shell

實(shí)際上完成以上配置之后還是會(huì)有一些問(wèn)題, 大家可以評(píng)論一起討論

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av

Mac上Hadoop, HDFS, Hive, Spark環(huán)境的安裝和搭建

Mac上Hadoop, HDFS, Hive, Spark環(huán)境的安裝和搭建

安裝前準(zhǔn)備

Homebrew

jdk安裝

配置ssh

安裝hadoop

下載安裝

配置hadoop

配置hadoop-env.sh

配置core-site.xml

配置hdfs-site.xml

配置mapred-site.xml

配置yarn-site.xml

格式化HDFS

運(yùn)行

安裝hive

下載安裝

配置Hive元數(shù)據(jù)庫(kù)

配置hive

配置mysql-connector jar包

配置hive-site.xml

初始化元數(shù)據(jù)庫(kù)

運(yùn)行Hive

安裝spark

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九 欧美,1769亚洲,黄色成人av

Mac上Hadoop, HDFS, Hive, Spark環(huán)境的安裝和搭建

安裝前準(zhǔn)備

Homebrew

jdk安裝

配置ssh

安裝hadoop

下載安裝

配置hadoop

配置hadoop-env.sh

配置core-site.xml

配置hdfs-site.xml

配置mapred-site.xml

配置yarn-site.xml

格式化HDFS

運(yùn)行

安裝hive

下載安裝

配置Hive元數(shù)據(jù)庫(kù)

配置hive

配置mysql-connector jar包

配置hive-site.xml

初始化元數(shù)據(jù)庫(kù)

運(yùn)行Hive

安裝spark

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av