您好,登录后才能下订单哦!
ssh-keygen
cd .ssh
touch authorized_keys
cat id_rsa.pub > authorized_keys
chmod 600 authorized_keys
系统 urbuntu jdk 1.7.0_79
scala 2.10.4
hadoop 2.6.0
spark 1.6.2
IDEA + sbt1.2打包工具
需要预先安装scala插件,点击File ->Setting ->Plugins ->输入框输入scala->install
安装完成需要重启IDE
File -> New Project ->Scala -> SBT 选择相应版本 ->finish
build.sbt 添加spark相关依赖
name := "demoPro"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.2"
创建WordCount.scala,编写如下代码
import org.apache.spark.{SparkContext, SparkConf}
/**
* Created by Administrator on 2018/2/20.
*/
object WordCount {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("wordcount")
val sc = new SparkContext(conf)
val input = sc.textFile("/home/dell/helloSpark.txt")
val lines = input.flatMap(line => (line.split(" ")))
val count = lines.map(word => (word, 1)).reduceByKey { case (x, y) => x + y }
val output=count.saveAsTextFile("/home/dell/helloSparkRes")
}
}
File -> Project Structure -> Aritifacts -> 点击+号 ->jar -> 第二个 -> 指定Module和 MainClass -> JAR files from libraries 选择第二个 ->点击ok
主题栏点击Build -> Build Aritifacts - Build
在工程目下out目录中生成相应jar包即打包成功
#进入sbin目录
cd $Hadoop_HOME/sbin
#启动hadoop集群
start-all.sh
hadoop fs -put test.txt /test/test.txt
是同filelize 或者sftp 或者 rz -y命令上传程序jar
sudo ./start-master.sh
访问localhost:8080 获取spark://xxx:7077
sudo ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://dell:7077
sudo ./bin/spark-submit --master spark://dell:7077 --class WordCount /home/dell/demopro.jar
免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。