您好,登录后才能下订单哦!
密码登录
登录注册
点击 登录注册 即表示同意《亿速云用户服务条款》
原因:用户代理未生效。检查core-site.xml文件是否正确配置。
<property>
<name>hadoop.proxyuser.bigdata.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.bigdata.groups</name>
<value>*</value>
</property>
备注hadoop.proxyuser.XXX.hosts 与 hadoop.proxyuser.XXX.groups 中XXX为异常信息中User:* 中的用户名部分
<property>
<name>hadoop.proxyuser.bigdata.hosts</name>
<value>*</value>
<description>The superuser can connect only from host1 and host2 to impersonate a user</description>
</property>
<property>
<name>hadoop.proxyuser.bigdata.groups</name>
<value>*</value>
<description>Allow the superuser oozie to impersonate any members of the group group1 and group2</description>
</property>
增加以上配置后,无需重启集群,可以直接在namenode节点上使用管理员账号重新加载这两个属性值,命令为:
$ hdfs dfsadmin -refreshSuperUserGroupsConfiguration
Refresh super user groups configuration successful
$ yarn rmadmin -refreshSuperUserGroupsConfiguration
19/01/16 15:02:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8033
如果集群配置了HA,执行如下命令namenode节点全部重新加载:
# hadoop dfsadmin -fs hdfs://ns -refreshSuperUserGroupsConfiguration
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Refresh super user groups configuration successful for master/192.168.99.219:9000
Refresh super user groups configuration successful for node01/192.168.99.173:9000
现象:使用beeline、jdbc、python调用hiveserver2时,无法查询、建表等Hbase关联表,
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
<description>
Setting this property to true will have HiveServer2 execute
Hive operations as the user making the calls to it.
</description>
</property>
在hive创建Hbase关联表
# Hive中的表名test_tb
CREATE TABLE test_tb(key int, value string)
# 指定存储处理器
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
# 声明列族,列名
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
# hbase.table.name声明HBase表名,为可选属性默认与Hive的表名相同
# hbase.mapred.output.outputtable指定插入数据时写入的表,如果以后需要往该表插入数据就需要指定该值
TBLPROPERTIES ("hbase.table.name" = "test_tb", "hbase.mapred.output.outputtable" = "test_tb");
使用spark standalone模式执行任务,没提交一次任务,在每个节点work目录下都会生成一个文件夹,命名规则app-xxxxxxx-xxxx。该文件夹下是任务提交时,各节点从主节点下载的程序所需要的资源文件。 这些目录每次执行都会生成,且不会自动清理,执行任务过多会将内存撑爆。
export SPARK_WORKER_OPTS="
-Dspark.worker.cleanup.enabled=true # 是否开启自动清理
-Dspark.worker.cleanup.interval=1800 # 清理周期,每隔多长时间清理一次,单位秒
-Dspark.worker.cleanup.appDataTtl=3600" # 保留最近多长时间的数据
2019-01-25 03:26:41,627 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@211] - Too many connections from /172.17.0.1 - max is 60
hbase-site.xml
hbase.zookeeper.property.maxClientCnxns
hive.server2.thrift.min.worker.threads
hive.server2.thrift.max.worker.threads
hive.zookeeper.session.timeout
# Limits the number of concurrent connections (at the socket level) that a single client, identified by IP address
maxClientCnxns=200
# The minimum session timeout in milliseconds that the server will allow the client to negotiate
minSessionTimeout=1000
# The maximum session timeout in milliseconds that the server will allow the client to negotiate
maxSessionTimeout=60000
持续更新....
免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。