WIn7下用Idea远程操作Spark

发布时间：2020-06-28 03:23:43 作者：moviebat
来源：网络阅读：2168

WIn7下用Idea远程操作Spark

main.scala
org.apache.spark.SparkContext._
org.apache.spark.{SparkConfSparkContext}

SogouResult {
  (args:Array[]){
    (args.length==) {
      System..println()
      System.()
    }

    conf=SparkConf().setAppName().setMaster()
    sc=SparkContext(conf)

    rdd1=sc.textFile(args()).map(_.split()).filter(_.length==)
    rdd2=rdd1.map(x=>(x())).reduceByKey(_+_).map(x=>(x._2x._1)).sortByKey().map(x=>(x._2x._1))
    rdd2.saveAsTextFile(args())
    sc.stop


  }

}

fs.defaultFShdfs://192.168.0.3:9000

4.0.0HdfsTestHdfsTest1.0-SNAPSHOT12         13             org.apache.hadoop14             hadoop-common15             2.6.416         17         18             org.apache.hadoop19             hadoop-mapreduce-client-jobclient20             2.6.421         22         23             commons-cli24             commons-cli25             1.226         27     28
    29     30         ${project.artifactId}31

运行参数如下：

hdfs://192.168.0.3:9000/input/SogouQ1 hdfs://192.168.0.3:9000/output/sogou1

出错：

"C:\Program Files\Java\jdk1.7.0_79\bin\java" -Didea.launcher.port=7535 -Didea.launcher.bin.path=D:\Java\IntelliJ\bin -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.7.0_79\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jce.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jfxrt.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\resources.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\rt.jar;D:\scalasrc\HdfsTest\target\classes;D:\scalasrc\lib\datanucleus-core-3.2.10.jar;D:\scalasrc\lib\datanucleus-rdbms-3.2.9.jar;D:\scalasrc\lib\spark-1.5.0-yarn-shuffle.jar;D:\scalasrc\lib\datanucleus-api-jdo-3.2.6.jar;D:\scalasrc\lib\spark-assembly-1.5.0-hadoop2.6.0.jar;D:\scalasrc\lib\spark-examples-1.5.0-hadoop2.6.0.jar;D:\Java\scala210\lib\scala-actors-migration.jar;D:\Java\scala210\lib\scala-actors.jar;D:\Java\scala210\lib\scala-library.jar;D:\Java\scala210\lib\scala-reflect.jar;D:\Java\scala210\lib\scala-swing.jar;D:\Java\IntelliJ\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain main.scala.SogouResult hdfs://192.168.0.3:9000/input/SogouQ1 hdfs://192.168.0.3:9000/output/sogou1

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/D:/scalasrc/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/D:/scalasrc/lib/spark-examples-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/09/16 12:00:43 INFO SparkContext: Running Spark version 1.5.0

16/09/16 12:00:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/09/16 12:00:44 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)

at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)

at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)

at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)

at org.apache.hadoop.security.Groups.<init>(Groups.java:86)

at org.apache.hadoop.security.Groups.<init>(Groups.java:66)

at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271)

at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:248)

at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:763)

at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:748)

at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:621)

at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2084)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2084)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:310)

at main.scala.SogouResult$.main(SogouResult.scala:16)

at main.scala.SogouResult.main(SogouResult.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

16/09/16 12:00:44 INFO SecurityManager: Changing view acls to: danger

16/09/16 12:00:44 INFO SecurityManager: Changing modify acls to: danger

16/09/16 12:00:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(danger); users with modify permissions: Set(danger)

16/09/16 12:00:45 INFO Slf4jLogger: Slf4jLogger started

16/09/16 12:00:45 INFO Remoting: Starting remoting

16/09/16 12:00:45 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.2:55944]

16/09/16 12:00:45 INFO Utils: Successfully started service 'sparkDriver' on port 55944.

16/09/16 12:00:45 INFO SparkEnv: Registering MapOutputTracker

16/09/16 12:00:45 INFO SparkEnv: Registering BlockManagerMaster

16/09/16 12:00:45 INFO DiskBlockManager: Created local directory at C:\Users\danger\AppData\Local\Temp\blockmgr-281e23a9-a059-4670-a1b0-0511e63c55a3

16/09/16 12:00:45 INFO MemoryStore: MemoryStore started with capacity 481.1 MB

16/09/16 12:00:45 INFO HttpFileServer: HTTP File server directory is C:\Users\danger\AppData\Local\Temp\spark-84f74e01-9ea2-437c-b532-a5cfec898bc8\httpd-876c9027-ebb3-44c6-8256-bd4a555eaeaf

16/09/16 12:00:45 INFO HttpServer: Starting HTTP Server

16/09/16 12:00:46 INFO Utils: Successfully started service 'HTTP file server' on port 55945.

16/09/16 12:00:46 INFO SparkEnv: Registering OutputCommitCoordinator

16/09/16 12:00:46 INFO Utils: Successfully started service 'SparkUI' on port 4040.

16/09/16 12:00:46 INFO SparkUI: Started SparkUI at http://192.168.0.2:4040

16/09/16 12:00:46 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

16/09/16 12:00:46 INFO Executor: Starting executor ID driver on host localhost

16/09/16 12:00:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55964.

16/09/16 12:00:46 INFO NettyBlockTransferService: Server created on 55964

16/09/16 12:00:46 INFO BlockManagerMaster: Trying to register BlockManager

16/09/16 12:00:46 INFO BlockManagerMasterEndpoint: Registering block manager localhost:55964 with 481.1 MB RAM, BlockManagerId(driver, localhost, 55964)

16/09/16 12:00:46 INFO BlockManagerMaster: Registered BlockManager

16/09/16 12:00:47 INFO MemoryStore: ensureFreeSpace(157320) called with curMem=0, maxMem=504511856

16/09/16 12:00:47 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 153.6 KB, free 481.0 MB)

16/09/16 12:00:47 INFO MemoryStore: ensureFreeSpace(14301) called with curMem=157320, maxMem=504511856

16/09/16 12:00:47 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 14.0 KB, free 481.0 MB)

16/09/16 12:00:47 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55964 (size: 14.0 KB, free: 481.1 MB)

16/09/16 12:00:47 INFO SparkContext: Created broadcast 0 from textFile at SogouResult.scala:18

16/09/16 12:00:48 WARN : Your hostname, danger-PC resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:ac1b:2301%24, but we couldn't find any external IP address!

Exception in thread "main" java.net.ConnectException: Call From danger-PC/192.168.0.2 to 192.168.0.3:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)

at org.apache.hadoop.ipc.Client.call(Client.java:1472)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)

at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)

at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)

at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)

at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)

at org.apache.hadoop.fs.Globber.glob(Globber.java:252)

at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1644)

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

at scala.Option.getOrElse(Option.scala:120)

at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65)

at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:290)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)

at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:289)

at main.scala.SogouResult$.main(SogouResult.scala:19)

at main.scala.SogouResult.main(SogouResult.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Caused by: java.net.ConnectException: Connection refused: no further information

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)

at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)

at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)

at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)

at org.apache.hadoop.ipc.Client.call(Client.java:1438)

... 61 more

16/09/16 12:00:50 INFO SparkContext: Invoking stop() from shutdown hook

16/09/16 12:00:50 INFO SparkUI: Stopped Spark web UI at http://192.168.0.2:4040

16/09/16 12:00:50 INFO DAGScheduler: Stopping DAGScheduler

16/09/16 12:00:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

16/09/16 12:00:51 INFO MemoryStore: MemoryStore cleared

16/09/16 12:00:51 INFO BlockManager: BlockManager stopped

16/09/16 12:00:51 INFO BlockManagerMaster: BlockManagerMaster stopped

16/09/16 12:00:51 INFO SparkContext: Successfully stopped SparkContext

16/09/16 12:00:51 INFO ShutdownHookManager: Shutdown hook called

16/09/16 12:00:51 INFO ShutdownHookManager: Deleting directory C:\Users\danger\AppData\Local\Temp\spark-84f74e01-9ea2-437c-b532-a5cfec898bc8

Process finished with exit code 1

半天没搞定啊感觉应该是没有winutils.exe

果断从网上找个，放在hadoop/bin下，

执行，没有了，但依然无法连接

telnet 192.168.0.3 9000，显示无法连接

应该问题就在这

在hadoop的配置文件中，配置的是主机名

将所有的主机名改为IP地址

"C:\Program Files\Java\jdk1.7.0_79\bin\java" -Didea.launcher.port=7536 -Didea.launcher.bin.path=D:\Java\IntelliJ\bin -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.7.0_79\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jce.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jfxrt.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\resources.jar;C:\Program Files\Java\jdk1.7.0_79\jre\lib\rt.jar;D:\scalasrc\HdfsTest\target\classes;D:\scalasrc\lib\datanucleus-core-3.2.10.jar;D:\scalasrc\lib\datanucleus-rdbms-3.2.9.jar;D:\scalasrc\lib\spark-1.5.0-yarn-shuffle.jar;D:\scalasrc\lib\datanucleus-api-jdo-3.2.6.jar;D:\scalasrc\lib\spark-assembly-1.5.0-hadoop2.6.0.jar;D:\scalasrc\lib\spark-examples-1.5.0-hadoop2.6.0.jar;D:\Java\scala210\lib\scala-actors-migration.jar;D:\Java\scala210\lib\scala-actors.jar;D:\Java\scala210\lib\scala-library.jar;D:\Java\scala210\lib\scala-reflect.jar;D:\Java\scala210\lib\scala-swing.jar;D:\Java\IntelliJ\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain main.scala.SogouResult hdfs://192.168.0.3:9000/input/SogouQ1 hdfs://192.168.0.3:9000/output/sogou1

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/D:/scalasrc/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/D:/scalasrc/lib/spark-examples-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/09/16 14:04:45 INFO SparkContext: Running Spark version 1.5.0

16/09/16 14:04:46 INFO SecurityManager: Changing view acls to: danger

16/09/16 14:04:46 INFO SecurityManager: Changing modify acls to: danger

16/09/16 14:04:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(danger); users with modify permissions: Set(danger)

16/09/16 14:04:47 INFO Slf4jLogger: Slf4jLogger started

16/09/16 14:04:47 INFO Remoting: Starting remoting

16/09/16 14:04:47 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.2:51172]

16/09/16 14:04:47 INFO Utils: Successfully started service 'sparkDriver' on port 51172.

16/09/16 14:04:47 INFO SparkEnv: Registering MapOutputTracker

16/09/16 14:04:47 INFO SparkEnv: Registering BlockManagerMaster

16/09/16 14:04:47 INFO DiskBlockManager: Created local directory at C:\Users\danger\AppData\Local\Temp\blockmgr-087e9166-2258-4f45-b449-d184c92702a3

16/09/16 14:04:47 INFO MemoryStore: MemoryStore started with capacity 481.1 MB

16/09/16 14:04:47 INFO HttpFileServer: HTTP File server directory is C:\Users\danger\AppData\Local\Temp\spark-0d6662f5-0bfa-4e6f-a256-c97bc6ce5f47\httpd-a2355600-9a68-417d-bd52-2ccdcac7bb13

16/09/16 14:04:47 INFO HttpServer: Starting HTTP Server

16/09/16 14:04:48 INFO Utils: Successfully started service 'HTTP file server' on port 51173.

16/09/16 14:04:48 INFO SparkEnv: Registering OutputCommitCoordinator

16/09/16 14:04:48 INFO Utils: Successfully started service 'SparkUI' on port 4040.

16/09/16 14:04:48 INFO SparkUI: Started SparkUI at http://192.168.0.2:4040

16/09/16 14:04:48 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

16/09/16 14:04:48 INFO Executor: Starting executor ID driver on host localhost

16/09/16 14:04:48 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51192.

16/09/16 14:04:48 INFO NettyBlockTransferService: Server created on 51192

16/09/16 14:04:48 INFO BlockManagerMaster: Trying to register BlockManager

16/09/16 14:04:48 INFO BlockManagerMasterEndpoint: Registering block manager localhost:51192 with 481.1 MB RAM, BlockManagerId(driver, localhost, 51192)

16/09/16 14:04:48 INFO BlockManagerMaster: Registered BlockManager

16/09/16 14:04:49 INFO MemoryStore: ensureFreeSpace(157320) called with curMem=0, maxMem=504511856

16/09/16 14:04:49 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 153.6 KB, free 481.0 MB)

16/09/16 14:04:49 INFO MemoryStore: ensureFreeSpace(14301) called with curMem=157320, maxMem=504511856

16/09/16 14:04:49 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 14.0 KB, free 481.0 MB)

16/09/16 14:04:49 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:51192 (size: 14.0 KB, free: 481.1 MB)

16/09/16 14:04:49 INFO SparkContext: Created broadcast 0 from textFile at SogouResult.scala:18

16/09/16 14:04:50 WARN : Your hostname, danger-PC resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:ac1b:2301%24, but we couldn't find any external IP address!

16/09/16 14:04:52 INFO FileInputFormat: Total input paths to process : 1

16/09/16 14:04:52 INFO SparkContext: Starting job: sortByKey at SogouResult.scala:19

16/09/16 14:04:52 INFO DAGScheduler: Registering RDD 4 (map at SogouResult.scala:19)

16/09/16 14:04:52 INFO DAGScheduler: Got job 0 (sortByKey at SogouResult.scala:19) with 2 output partitions

16/09/16 14:04:52 INFO DAGScheduler: Final stage: ResultStage 1(sortByKey at SogouResult.scala:19)

16/09/16 14:04:52 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)

16/09/16 14:04:52 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)

16/09/16 14:04:52 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[4] at map at SogouResult.scala:19), which has no missing parents

16/09/16 14:04:52 INFO MemoryStore: ensureFreeSpace(4208) called with curMem=171621, maxMem=504511856

16/09/16 14:04:52 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.1 KB, free 481.0 MB)

16/09/16 14:04:52 INFO MemoryStore: ensureFreeSpace(2347) called with curMem=175829, maxMem=504511856

16/09/16 14:04:52 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.3 KB, free 481.0 MB)

16/09/16 14:04:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:51192 (size: 2.3 KB, free: 481.1 MB)

16/09/16 14:04:52 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861

16/09/16 14:04:52 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[4] at map at SogouResult.scala:19)

16/09/16 14:04:52 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

16/09/16 14:04:52 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, ANY, 2135 bytes)

16/09/16 14:04:53 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)

16/09/16 14:04:53 INFO HadoopRDD: Input split: hdfs://192.168.0.3:9000/input/SogouQ1:0+134217728

16/09/16 14:04:53 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id

16/09/16 14:04:53 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id

16/09/16 14:04:53 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap

16/09/16 14:04:53 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition

16/09/16 14:04:53 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id

16/09/16 14:04:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)

java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(IILjava/nio/ByteBuffer;ILjava/nio/ByteBuffer;IILjava/lang/String;JZ)V

at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSums(Native Method)

at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:59)

at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:301)

at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:216)

at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:146)

at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:693)

at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:749)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:806)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:847)

at java.io.DataInputStream.read(DataInputStream.java:100)

at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)

at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)

at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)

at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:248)

at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:216)

at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)

at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)

at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)

at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)

at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:203)

at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)

at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)

at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)

at org.apache.spark.scheduler.Task.run(Task.scala:88)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

16/09/16 14:04:54 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]