在CentOS上安装和配置Hadoop的步骤如下:
安装JDK:
/etc/profile
文件,添加以下内容:export JAVA_HOME=/usr/java/jdk1.8.0_211
export CLASSPATH=.:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
source /etc/profile
java -version
下载和解压Hadoop:
/usr/local/hadoop
:tar -xzvf hadoop-3.3.1.tar.gz -C /usr/local/hadoop
配置Hadoop环境变量:
~/.bashrc
文件,添加以下内容:export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
source ~/.bashrc
配置Hadoop配置文件:
etc/hadoop
目录。core-site.xml
文件,设置默认文件系统:<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://namenode:9000</value>
</property>
</configuration>
hdfs-site.xml
文件,配置HDFS相关参数:<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/hdfs/datanode</value>
</property>
</configuration>
mapred-site.xml
文件,配置MapReduce框架:<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
文件,配置YARN相关参数:<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
格式化HDFS:
hdfs namenode -format
启动Hadoop服务:
start-dfs.sh
start-yarn.sh
验证安装:
, , ,