您好，登录后才能下订单哦！
密码登录
登录注册
点击 登录注册 即表示同意《亿速云用户服务条款》
因为需要部署spark环境，特意重新安装了一个测试的hadoop集群，现将相关步骤记录如下：
硬件环境：四台虚拟机，hadoop1~hadoop4，3G内存，60G硬盘，2核CPU
软件环境：CentOS6.5，hadoop-2.6.0-cdh6.8.2，JDK1.7
部署规划：
hadoop1（192.168.0.3）：namenode（active）、resourcemanager
hadoop2（192.168.0.4）：namenode（standby）、journalnode、datanode、nodemanager、historyserver
hadoop3（192.168.0.5）：journalnode、datanode、nodemanager
hadoop4（192.168.0.6）：journalnode、datanode、nodemanager
HDFS的HA采用QJM的方式（journalnode）：
一、系统准备
1、每台机关闭selinux
#vi /etc/selinux/config
SELINUX=disabled
2、每台机关闭防火墙（切记要关闭，否则格式化hdfs时会报错无法连接journalnode）
#chkconfig iptables off
#service iptables stop
3、每台机安装jdk1.7
#cd /software
#tar -zxf jdk-7u65-linux-x64.gz -C /opt/
#cd /opt
#ln -s jdk-7u65-linux-x64.gz java
#vi /etc/profile
export JAVA_HOME=/opt/java
export PATH=$PATH:$JAVA_HOME/bin
4、每台机建立hadoop相关用户，并建立互信
#useradd grid
#passwd grid
（建立互信步骤略）
5、每台机建立相关目录
#mkdir -p /hadoop_data/hdfs/name
#mkdir -p /hadoop_data/hdfs/data
#mkdir -p /hadoop_data/hdfs/journal
#mkdir -p /hadoop_data/yarn/local
#chown -R grid:grid /hadoop_data
二、hadoop部署
HDFS HA主要是指定nameservices（如果不做HDFS ferderation，就只会有一个ID），同时指定该nameserviceID下面的两个namenode及其地址。此处的nameservice名设置为hadoop-spark
1、每台机解压hadoop包
#cd /software
#tar -zxf hadoop-2.6.0-cdh6.8.2.tar.gz -C /opt/
#cd /opt
#chown -R grid:grid hadoop-2.6.0-cdh6.8.2
#ln -s hadoop-2.6.0-cdh6.8.2 hadoop
2、切换到grid用户继续操作
#su - grid
$cd /opt/hadoop/etc/hadoop
3、配置hadoop-env.sh（其实只配置JAVA_HOME）
$vi hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/opt/java
4、设置hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.nameservices</name> <value>hadoop-spark</value> <description> Comma-separated list of nameservices. </description> </property> <property> <name>dfs.ha.namenodes.hadoop-spark</name> <value>nn1,nn2</value> <description> The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). </description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-spark.nn1</name> <value>hadoop1:8020</value> <description> RPC address for nomenode1 of hadoop-spark </description> </property> <property> <name>dfs.namenode.rpc-address.hadoop-spark.nn2</name> <value>hadoop2:8020</value> <description> RPC address for nomenode2 of hadoop-spark </description> </property> <property> <name>dfs.namenode.http-address.hadoop-spark.nn1</name> <value>hadoop1:50070</value> <description> The address and the base port where the dfs namenode1 web ui will listen on. </description> </property> <property> <name>dfs.namenode.http-address.hadoop-spark.nn2</name> <value>hadoop2:50070</value> <description> The address and the base port where the dfs namenode2 web ui will listen on. </description> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///hadoop_data/hdfs/name</value> <description>Determines where on the local filesystem the DFS name node should store the name table(fsp_w_picpath). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop2:8485;hadoop3:8485;hadoop4:8485/hadoop-spark</value> <description>A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. </description> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///hadoop_data/hdfs/data</value> <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. </description> </property> <!-- 这个如果不设置，会造成无法直接通过nameservice名称来访问hdfs，只能直接写active的namenode地址 --> <property> <name>dfs.client.failover.proxy.provider.hadoop-spark</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>false</value> <description> Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. </description> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/hadoop_data/hdfs/journal</value> </property> </configuration>
5、配置core-site.xml（配置fs.defaultFS，使用HA的nameservices名称）
<property> <name>fs.defaultFS</name> <value>hdfs://hadoop-spark</value> <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> </property>
6、配置mapred-site.xml
<configuration> <!-- MR YARN Application properties --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. </description> </property> <!-- jobhistory properties --> <property> <name>mapreduce.jobhistory.address</name> <value>hadoop2:10020</value> <description>MapReduce JobHistory Server IPC host:port</description> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hadoop2:19888</value> <description>MapReduce JobHistory Server Web UI host:port</description> </property> </configuration>
7、配置yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- Resource Manager Configs -->
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<description>The address of the scheduler interface.</description>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<description>The http address of the RM web application.</description>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<description>The https adddress of the RM web application.</description>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<description>The address of the RM admin interface.</description>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
<description>The class to use as the resource scheduler.</description>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<description>fair-scheduler conf location</description>
<name>yarn.scheduler.fair.allocation.file</name>
<value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value>
</property>
<property>
<description>List of directories to store localized files in. An
application's localized file directory will be found in:
${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}.
Individual containers' work directories, called container_${contid}, will
be subdirectories of this.
</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop_data/yarn/local</value>
</property>
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>
<property>
<description>Amount of physical memory, in MB, that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>2048</value>
</property>
<property>
<description>Number of CPU cores that can be allocated
for containers.</description>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
<property>
<description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>8、配置slaves
hadoop2
hadoop3
hadoop4
9、配置fairscheduler.xml
<?xml version="1.0"?> <allocations> <queue name="common"> <minResources>0mb, 0 vcores </minResources> <maxResources>6144 mb, 6 vcores </maxResources> <maxRunningApps>50</maxRunningApps> <minSharePreemptionTimeout>300</minSharePreemptionTimeout> <weight>1.0</weight> <aclSubmitApps>grid</aclSubmitApps> </queue> </allocations>
10、同步配置文件到各个节点
$cd /opt/hadoop/etc
$scp -r hadoop hadoop2:/opt/hadoop/etc/
$scp -r hadoop hadoop3:/opt/hadoop/etc/
$scp -r hadoop hadoop4:/opt/hadoop/etc/
三、启动集群（格式化文件系统）
1、建立环境变量
$vi ~/.bash_profile
export HADOOP_HOME=/opt/hadoop
export YARN_HOME_DIR=/opt/hadoop
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export YARN_CONF_DIR=/opt/hadoop/etc/hadoop
2、启动HDFS
先启动journalnode，在hadoop2~hadoop4上：
$cd /opt/hadoop/
$sbin/hadoop-daemon.sh start journalnode
格式化HDFS，然后启动namenode。在hadoop1上：
$bin/hdfs namenode -format
$sbin/hadoop-daemon.sh start namenode
同步另一个namenode，并启动。在hadoop2上：
$bin/hdfs namenode -bootstrapStandby
$sbin/hadoop-daemon.sh start namenode
此时两个namenode都是standby状态，将hadoop1切换成active（hadoop1在hdfs-site.xml里对应的是nn1）：
$bin/hdfs haadmin -transitionToActive nn1
启动datanode。在hadoop1上（active的namenode）：
$sbin/hadoop-daemons.sh start datanode
注意事项：后续启动，只需使用sbin/start-dfs.sh即可。但由于没有配置zookeeper的failover，所以只能HA只能使用手工切换。所以每次启动HDFS，都要执行$bin/hdfs haadmin -transitionToActive nn1来使hadoop1的namenode变成active状态
2、启动yarn
在hadoop1上（resourcemanager）：
$sbin/start-yarn.sh
————————————————————————————————————————————
以上配置的HDFS HA并不是自动故障切换的，如果配置HDFS自动故障切换，需要添加以下步骤（先停掉集群）：
1、部署zookeeper，步骤省略。部署在hadoop2、hadoop3、hadoop4，并启动
2、在hdfs-site.xml中添加：
<property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
<property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/exampleuser/.ssh/id_rsa</value> </property>
解释详见官方文档。这样配置设定了fencing方法是通过ssh去关闭前一个活动节点的端口。前提前两个namenode能互相SSH。
还有另外一种配置方法：
<property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
<property> <name>dfs.ha.fencing.methods</name> <value>shell(/path/to/my/script.sh arg1 arg2 ...)</value> </property>
这样的配置实际上是使用shell来隔绝端口和程序。如果不想做实际的动作，dfs.ha.fencing.methods可配置成shell(/bin/true)
3、在core-site.xml中添加：
<property> <name>ha.zookeeper.quorum</name> <value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value> </property>
4、初始化zkfc（在namenode上执行）
bin/hdfs zkfc -formatZK
5、启动集群
___________________________________________________________________________________________________
zkfc：每个namenode上都运行，是zk的客户端，负责自动故障切换
zk：奇数个节点，维护一致性锁、负责选举活动节点
joural node：奇数个节点，用于active和standby节点之间数据同步。活动节点把数据写入这些节点，standby节点读取
————————————————————————————————————————————
更改成resourcemanager HA：
选择hadoop2作为另一个rm节点
1、设置hadoop2对其它节点作互信
2、编译yarn-site.xml并同步到其它机器
3、复制fairSheduler.xml到hadoop2
4、启动rm
5、启动另一个rm
免责声明：本站发布的内容（图片、视频和文字）以原创、转载和分享为主，文章观点不代表本网站立场，如果涉及侵权请联系站长邮箱：is@yisu.com进行举报，并提供相关证据，一经查实，将立刻删除涉嫌侵权内容。