在Linux系统中配置HDFS高可用性(High Availability, HA)主要包括以下几个步骤:
安装和配置ZooKeeper:
wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
cd apache-zookeeper-3.8.0
conf/zoo.cfg
文件,设置数据目录和客户端连接端口:dataDir /var/lib/zookeeper
clientPort 2181
myid
文件:echo 1 > /var/lib/zookeeper/myid
./bin/zkServer.sh start
配置Hadoop:
hdfs-site.xml
文件:<configuration>
<property>
name dfs.nameservices / name value mycluster / value /
</property>
<property>
name dfs.ha.namenodes.mycluster / name value nn1,nn2 / value /
</property>
<property>
name dfs.namenode.rpc-address.mycluster.nn1 / name value namenode1:8020 / value /
</property>
<property>
name dfs.namenode.rpc-address.mycluster.nn2 / name value namenode2:8020 / value /
</property>
<property>
name dfs.namenode.http-address.mycluster.nn1 / name value namenode1:50070 / value /
</property>
<property>
name dfs.namenode.http-address.mycluster.nn2 / name value namenode2:50070 / value /
</property>
<property>
name dfs.namenode.shared.edits.dir / name value qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster / value /
</property>
<property>
name dfs.client.failover.proxy.provider.mycluster / name value org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider / value /
</property>
<property>
name dfs.ha.fencing.methods / name value sshfence / value /
</property>
<property>
name dfs.ha.fencing.ssh.private-key-files / name value /root/.ssh/id_rsa / value /
</property>
<property>
name dfs.ha.automatic-failover.enabled / name value true / value /
</property>
</configuration>
core-site.xml
文件:<configuration>
<property>
name fs.defaultFS / name value hdfs://mycluster / value /
</property>
</configuration>
格式化和启动NameNode:
hdfs namenode -format
sbin/hadoop-daemon.sh start namenode
hdfs namenode -bootstrapStandby
sbin/hadoop-daemon.sh start namenode
启动JournalNode:
sbin/hadoop-daemon.sh start journalnode
验证高可用性:
http://namenode1:50070
查看NameNode状态。通过以上步骤,可以在Linux系统中配置一个高可用的HDFS集群,确保在发生故障时能够快速切换并继续提供服务。