CentOS下hadoop完全分布式的基础配置

配置完全分布式

1、修改静态IP和主机名称

①规划IP和主机名映射

②涉及文件

etc/hostname

etc/hosts

etc/resolve.conf

etc/sysconfig/network-script/ifcfg-ens3

2、配置ssh

①删除各节点的~/.ssh目录

②在各节点创建~/.ssh目录，同时指定权限

mkdir ~/.ssh -m 700

③在主节点生成公私密钥对

//生成成公私密钥对

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cp id_rsa.pub authorized_keys        //authorized_keys为644权限

//分发公钥

scp ~/.ssh/authorized_keys centos@s202:/home/centos/.ssh/

ssh-copy-id centos@s202

④分发密钥

scp ~/.ssh/authorized_keys centos@s212:/~/.ssh/

⑤测试ssh是否成功

ssh s212

3、修改配置文件并分发到个节点

①core-site.xml

<?xml version="1.0"?>

<!-- core-site.xml -->

<configuration>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://s201</value>

    </property>

</configuration>

②hdfs-site.xml

<?xml version="1.0"?>

<!-- hdfs-site.xml -->

<configuration>

    <property>

            <name>dfs.replication</name>

            <value>3</value>

        </property>

    <property>

            <name>dfs.namenode.name.dir</name>

            <value>/home/centos/hadoop/hdfs/name</value>

    </property>

    <property>

            <name>dfs.datanode.data.dir</name>

            <value>/home/centos/hadoop/hdfs/data</value>

    </property>

    <property>

            <name>dfs.namenode.checkpoint.dir</name>

            <value>/home/centos/hadoop/hdfs/namesecondary</value>

    </property>

</configuration>

③mapred-site.xml

<?xml version="1.0"?>

<!-- mapred-site.xml -->   

<configuration>

    <property>

            <name>mapreduce.framework.name</name>

            <value>yarn</value>

    </property>

</configuration>

④yarn-site.xml

<?xml version="1.0"?>

<!-- yarn-site.xml -->

<configuration>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>s201</value>

    </property>

    <property>

        <name>yarn.nodemanager.local-dirs</name>

        <value>/home/centos/hadoop/nm-local-dir</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

</configuration>

⑤分发配置文件

rsync etc centos@s212:/soft/hadoop/

4、NN和DN指定

slaves文件配置

5、克隆主机并修改各节点主机IP和名称

方法同步骤1

6、格式化文件系统

hdfs namenode -format

7、启动集群，观察进程

1、启动hdfs

start-dfs.sh

//涉及进程

namenode    50070    元数据

datanode    50075    数据

secondarynamenode    50090    

2、启动yarn

start-yarn.sh

//涉及进程

resourcemanager    

nodemanager

涉及脚本：

1、xcall.sh

#!/bin/bash

for host in `cat /soft/hadoop/etc/hadoop/slaves` ; do

echo ========== $host ===========

ssh $host $@ ;

done

2、xsync.sh

#!/bin/bash

param=$1

dir=`dirname $param`

fullpath=`pwd -P`

user=`whoami`

filename=`basename $param`

cd $dir

for host in `cat /soft/hadoop/etc/hadoop/slaves` ; do

echo =========== $host ============

rsync -lr $filename $user@$host:$fullpath ;

done