在CentOS上实现HDFS(Hadoop Distributed File System)的高可用性,通常涉及以下几个关键步骤:
core-site.xml
, hdfs-site.xml
, yarn-site.xml
等配置文件。hdfs-site.xml
:<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>namenode1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>namenode2:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>namenode1:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>namenode2:50070</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/path/to/ssh/key</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/mycluster</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/path/to/journalnode/data</value>
</property>
hdfs-site.xml
(已在上面配置)。hdfs --daemon start journalnode
hdfs namenode -format
hdfs namenode -initializeSharedEdits
hdfs-site.xml
:<property>
<name>dfs.zkfc.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.zkfc.port</name>
<value>8080</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/path/to/ssh/key</value>
</property>
zkServer.sh start
hdfs zkfc -formatZK
hdfs --daemon start zkfc
hdfs haadmin
命令检查HA状态:hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2
hdfs-site.xml
:<property>
<name>dfs.namenode.secondary.http-address</name>
<value>secondarynamenode:50090</value>
</property>
hdfs --daemon start secondarynamenode
通过以上步骤,你可以在CentOS上实现HDFS的高可用性。确保所有服务都正常运行,并定期检查日志文件以确保系统的稳定性。