第十一章 Spark Core · Hadoop

Scala --> 基于JVM --> java改进版多核-》多进程分布式 Spark 应用场合：大数据-分布式-批处理 Spark平台的工具库 Core- RDD Spark SQL离线 Streaming 在线 GraphX-rdd高维pageRank MLlib机器学习 Spark是一个计算框架，可以基于JAVA语言，可以基于Scala语言 Spark算子： ~~~ List.reduce((x,y)=>x+y) 简化为： List.reduce(_+_) 去除重复 raw.flatMap(_.split("\\W+")).distinct.collect.take(10) ~~~ 案例一、单词统计： val filerdd = sc.textFile(file).flatMap(_.split("\\W+")).map(x=>(x,1)).reduceByKey(_+_).collect.take(100).foreach(println) 案例二、分组案例（找到每门课程前3名）：文件样本： ~~~ hadoop 100 hadoop 68 spark 90 spark 80 hadoop 95 spark 100 hadoop 60 spark 92 spark 93 ~~~ 代码 ~~~ val raw = sc.textFile("file:/home/hadoop/g1"); var frdd = raw.map(_.split(" ")).map(x=>(x(0),x(1).toDouble)) frdd.groupByKey.map{ x=> var str = x._2.toArray.sortWith(_>_).take(3).mkString(",") pirntln(x._1+" top 3="+str) }.collect ~~~ 运行结果： ~~~ spark top 3 = 100.0, 93.0, 92.0 hadoop top3=100.0, 95.0, 68.0 ~~~ 案例三、日志分析 ~~~ #Software: Microsoft Internet Information Services 5.0 #Version: 1.0 #Date: 2004-12-19 00:01:21 #Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /index.asp - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /all.css - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/head.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c1.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c3.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c4.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c7.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c12.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c13.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c14.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c18.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c15.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c20.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/index_r2_c22.jpg - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) 2004-12-19 00:01:21 172.16.52.16 - 211.66.184.35 80 GET /images/spacer.gif - 200 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705) ~~~ 代码 ~~~ def main(args: Array[String]): Unit = { //创建SparkConf对象 val conf = new SparkConf() //设置应用程序名称，在程序运行的监控界面可以看到名称 conf.setAppName("My First Spark App!") //设置local使程序在本地运行，不需要安装Spark集群 conf.setMaster("local") val sc = new SparkContext(conf) val lines = sc.textFile("file:///e:/大数据资料/weblog.txt",1) val lines2 = lines.filter(!_.startsWith("#")); val lines3 = lines2.map(_.split("\\s+")).map(x=>(x(0),x(7))); val lines4 = lines3.map{x=> val k = x._1+":"+x._2 (k,1) }.reduceByKey(_+_); lines4.foreach(println); } ~~ 运行结果 ~~~ (2004-12-19:/news/newshtml/schoolNews/20040618182152.asp,1) (2004-12-19:/images/index_r2_c14.jpg,269) (2004-12-19:/south/administer/images/BYTESKY-64.GIF,1) (2004-12-19:/south/administer/ryjs/images/calling+card/cjd.jpg,1) (2004-12-19:/south/administer/images/bytesky-69.gif,1) (2004-12-19:/news/newshtml/insideInform/chengjiao/fu2.htm,1) (2004-12-19:/images/zuotiao1.jpg,272) (2004-12-19:/news/img/2004-9-22d.jpg,1) ~~~ 案例四、正则表达式 4.1得到数字的数据 ~~~ val numPattern="[0-9]+".r for(matchString <- numPattern.findAllIn("99345 Scala,22298 Spark")) println(matchString) ~~~ ~~~ 4.2 得到非数字的数据 ~~~ //创建SparkConf对象 val conf = new SparkConf() //设置应用程序名称，在程序运行的监控界面可以看到名称 conf.setAppName("My First Spark App!") //设置local使程序在本地运行，不需要安装Spark集群 conf.setMaster("local") val sc = new SparkContext(conf) val lines = sc.textFile("file:///e:/大数据资料/weblog.txt",1) //非数字 val reg = """[^0-9]""".r val lines2 = lines.map(x=>reg.findAllIn(x).mkString) lines2.foreach(println); ~~~