hadoop-006完全分布式问题有哪些

发布时间：2021-12-08 09:58:33 作者：小新
来源：亿速云阅读：174

小编给大家分享一下hadoop-006完全分布式问题有哪些，希望大家阅读完这篇文章之后都有所收获，下面让我们一起去探讨吧！

一、运行mapreduce 任务，但是在yarn里看不到任务。

bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /test /out1

我配置的resourceManager 为 http://192.168.31.136:8088

原因是在 mapred-site.xml 没有配置

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

二、配置好问题一后，运行mr任务提示The auxService:mapreduce_shuffle does not exist

原因是在yarn-site.xml 没有配置yarn.nodemanager.aux-services 节点

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

报错如下

16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED
Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------------------------------

配置文件记录

已提前搭建好三个节点的zookeeper 小集群，配置可以实现 HDFS　ＨＡ和　ＹＡＲＮ　ＨＡ

一、hadoop-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

二、yarn-env.sh 修改了

export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111

三、core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<configuration>

   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://mycluster</value>
   </property>
   
   <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/home/jxlgzwh/hadoop-2.7.2/data/jn</value>
   </property>

   
   <property>
       <name>hadoop.tmp.dir</name>
       <value>/home/jxlgzwh/hadoop-2.7.2/data/tmp</value>
   </property>
   <property>
       <name>ha.zookeeper.quorum</name>
       <value>master:2181,slave01:2181,slave02:2181</value>
   </property>
</configuration>

四、hdfs-site.xml

http://www.apache.org/licenses/LICENSE-2.0

<configuration>

   <property>
      <name>dfs.nameservices</name>
      <value>mycluster</value>
   </property>
   
   <property>
      <name>dfs.ha.namenodes.mycluster</name>
      <value>nn1,nn2</value>
   </property>

   
   <property>
      <name>dfs.namenode.rpc-address.mycluster.nn1</name>
      <value>master:8020</value>
   </property>
   <property>
      <name>dfs.namenode.rpc-address.mycluster.nn2</name>
      <value>slave01:8020</value>
   </property>

   
   <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>master:50070</value>
   </property>
   <property>
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>slave01:50070</value>
   </property>
   
   <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://master:8485;slave01:8485;slave02:8485/mycluster</value>
   </property>
   
   <property>
      <name>dfs.client.failover.proxy.provider.mycluster</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
   </property>
   
   <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>
   
    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/home/jxlgzwh/.ssh/id_dsa</value>
    </property>
   
   <property>
       <name>dfs.replication</name>
       <value>3</value>
   </property>
       
   <property>
       <name>dfs.permissions.enabled</name>
       <value>false</value>
   </property>

   <property>
       <name>dfs.ha.automatic-failover.enabled</name>
       <value>true</value>
   </property>

</configuration>

五、mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

六、slaves

192.168.31.136
192.168.31.130
192.168.31.229

七、yarn-site.xml

<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>slave01</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>slave01:8088</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>master:2181,slave01:2181,slave02:2181</value>
</property>

<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.zk-state-store.parent-path</name>
<value>/rmstore</value>
</property>

<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>

<property>
<name>yarn.nodemanager.recovery.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.address</name>
<value>0.0.0.0:45454</value>
</property>

<property>
<name>yarn.nodemanager.recovery.dir</name>
<value>/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

八、配置 /etc/hosts 文件并配置 ssh免密码登录

192.168.31.136 master.com master
192.168.31.130 slave01
192.168.31.229 slave02

看完了这篇文章，相信你对“hadoop-006完全分布式问题有哪些”有了一定的了解，如果想了解更多相关知识，欢迎关注亿速云行业资讯频道，感谢各位的阅读！

hadoop-006完全分布式问题有哪些

相关阅读