在Linux下高效配置Hadoop可以遵循以下步骤:
sudo apt update
sudo apt install openjdk-11-jdk
sudo apt update && sudo apt upgrade
/usr/local/hadoop
:sudo tar -zxvf hadoop-3.3.5.tar.gz -C /usr/local/hadoop
/.bashrc
文件,添加以下内容:export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source ~/.bashrc
core-site.xml
:<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
hdfs-site.xml
:<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/data/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/data/dfs/datanode</value>
</property>
</configuration>
mapred-site.xml
(可选,如果需要YARN支持):<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml
:<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-copy-id hadoop@node2
ssh-copy-id hadoop@node3
hdfs namenode -format
start-dfs.sh
start-yarn.sh
jps
命令查看进程。以上步骤涵盖了在Linux下高效配置Hadoop的基本流程,包括环境准备、安装与配置、SSH免密登录配置、启动Hadoop集群以及验证配置。根据具体需求和环境,可能还需要进行一些额外的配置和优化。