spark-submit提交任务时报错,Error initializing SparkContext

发布时间:2020-07-20 12:12:21 作者:北斗狼52cto
来源:网络 阅读:7976

16/03/04 00:21:09 WARN SparkContext: Using SPARK_MEM to set amount of memory to use per executor process is deprecated, please use spark.executor.memory instead.

16/03/04 00:21:09 ERROR SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Could not parse Master URL: 'at'

at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2554)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:489)

at com.bigdata.deal.scala.DomainLib$.main(DomainLib.scala:22)

at com.bigdata.deal.scala.DomainLib.main(DomainLib.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


配置conf时,务必要有 sparkHome  master地址

注意在spark-env.sh中要配置这些,就可以了

[root@mini-cp1 spark-1.4.0-bin-hadoop2.6]# cat conf/spark-env.sh

#!/usr/bin/env bash


SPARK_MASTER_IP=mini-cp1

#必须导入JAVA根目录路径

export JAVA_HOME=/usr/local/jdk1.7.0_65

export HADOOP_HOME=/usr/local/hadoop-2.6.0



#export SCALA_HOME=/opt/scala

export SPARK_WORKER_MEMORY=3g

export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop

#SPARK_MEM=${SPARK_MEM:-1g}

export SPARK_MEM=3g

export HADOOP_HOME=/usr/local/hadoop-2.6.0

export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop-2.6.0/lib/native

export HADOOP_OPTS="-Djava.library.path=/usr/local/hadoop-2.6.0/lib"




推荐阅读:
  1. Spark 实践 - 客户端使用spark-submit提交
  2. Spark-submit 测试任务提交

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

spark mi

上一篇:两条带头结点的升序重复合并成一个无重复的升序链表

下一篇:协警身上佩戴的竟然是:你真的了解摩托罗拉吗?

相关阅读

您好,登录后才能下订单哦!

密码登录
登录注册
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》