Spark 累加器 Accumulator  
   
所属分类 spark
浏览量 923
累加器   driver端定义,driver端读取,Executor Task端累加操做
内置三种类型 
LongAccumulator
DoubleAccumulator
CollectionAccumulator
import org.apache.spark.{SparkConf, SparkContext}
object AccumulatorDemo extends App{
  val sparkConf = new SparkConf().setMaster("local[*]").setAppName("sparkAccumulatorDemo")
  var sc =  new SparkContext(sparkConf);
  val list = List(1,2,3,4,5,6,7,8,9)
  val rdd = sc.makeRDD(list,3)
  val longAccumulator = sc.longAccumulator("longAccumulator");
  val doubleAccumulator = sc.doubleAccumulator("doubleAccumulator");
  val collectionAccumulator = sc.collectionAccumulator[Int]("collectionAccumulator");
  val rdd2 = rdd.map(e => {
    longAccumulator.add(1);
    doubleAccumulator.add(1);
    collectionAccumulator.add(e);
    e
  })
  rdd2.count
  println(longAccumulator)
  println(doubleAccumulator)
  println(collectionAccumulator)
  sc.stop();
}
 上一篇  
   
 下一篇  
 为何要收集整理创作技术段子 
 scala 类继承实例 
 一站式大数据平台StreamX 
 scala future 实例 
 AKKA actor model 的一些思考 
 Scala 柯里化 Currying