在CentOS上配置Hadoop分布式文件系统(HDFS)的高可用性(HA)涉及多个步骤,包括安装和配置Hadoop、ZooKeeper、以及设置NameNode和JournalNode等。以下是一个基本的指南,帮助你在CentOS系统上搭建一个高可用的HDFS集群。
wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
cd apache-zookeeper-3.8.0
conf/zoo.cfg
文件,设置数据目录和客户端连接端口。dataDir /var/lib/zookeeper
clientPort 2181
myid
文件。echo 1 > /var/lib/zookeeper/myid
./bin/zkServer.sh start
hdfs-site.xml
文件:<configuration>
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>namenode1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>namenode2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>namenode1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>namenode2:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
</configuration>
core-site.xml
文件:<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
</configuration>
hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
hdfs namenode -bootstrapStandby
sbin/hadoop-daemon.sh start namenode
在每个节点上启动JournalNode进程:
sbin/hadoop-daemon.sh start journalnode
http://namenode1:50070
查看NameNode状态。通过以上步骤,你可以在CentOS上配置一个高可用的HDFS集群,确保在发生故障时能够快速切换并继续提供服务。