在Ubuntu上配置Hadoop分布式文件系统(HDFS)涉及多个步骤和参数设置。以下是详细的配置参数详解:
sudo apt update
sudo apt install openjdk-8-jdk
java -version
wget http://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
tar -xzf hadoop-3.3.4.tar.gz
mv hadoop-3.3.4 hadoop
编辑~/.bashrc
文件,添加以下内容:
export HADOOP_HOME=/path/to/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source ~/.bashrc
core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>
hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/path/to/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/path/to/hadoop/hdfs/datanode</value>
</property>
</configuration>
hdfs namenode -format
start-dfs.sh
http://localhost:50070
,如果看到HDFS的Web界面,说明配置成功。hdfs dfs -mkdir /user/test
hdfs dfs -put /path/to/localfile /user/remotefile.txt
hdfs dfs -ls /user
hdfs dfs -get /user/remotefile.txt localfile.txt
hdfs dfs -rm /user/remotefile.txt
以上步骤涵盖了在Ubuntu上配置HDFS的主要要点,确保每个环节正确配置是成功运行HDFS的基础。