The basic steps for Spark Streaming Code ?

Initialize a Spark Streamingcontext Object
Appaly Transformations and ouput operations to Dstreams
Start reciving data and processing it using streamingContext.start()
Wait for the processing to be stopped using streamingContext.awaitTermination().

Comments

Post a Comment

Popular posts from this blog

What is Accumulator in Spark ?

Accumulators are variables that are only “added” to through an associative operation and can therefore be efficiently supported in parallel. They can be used to implement counters (as in MapReduce) or sums. Spark naively supports accumulators of numeric types, and programmers can add support for new types. If accumulators are created with a name, they will be displayed in Spark’s UI. This can be useful for understanding the progress of running stages. An accumulator is created from an initial value  v  by calling  SparkContext.accumulator(v) . Tasks running on the cluster can then add to it using the  add  method or the  +=  operator (in Scala and Python). However, they cannot read its value. Only the driver program can read the accumulator’s value, using its  value  method. scala > val accum = sc . accumulator ( 0 , "My Accumulator" ) accum : spark.Accumulator [ Int ] = 0 scala > sc . parallelize ( Array ( 1 , 2 ,...
Read more

Why Scala for Spark ?

In Summary: 1) Apache Spark is written in Scala and because of its scalability on JVM 2) Scala programming retains a perfect balance between productivity and performance.  3) Organizations want to enjoy the expressive power of dynamic programming language without having to lose type safety 4) Scala is designed with parallelism and concurrency in mind for big data applications 5) Scala collaborates well within the MapReduce big data model because of its functional paradigm 6)Scala programming language provides the best path for building scalable big data applications in terms of data size and program complexity 7) Scala programming is comparatively less complex unlike Java. 8) Scala has well-designed libraries for scientific computing, linear algebra and random number generation. 9) Efficiency and speed play a vital role regardless of increasing processor speeds.  10) Other programming languages like Python or Java have lag in the API coverage.  1...
Read more

What is SparkContext in Apache Spark?

A SparkContext is a client of Spark’s execution environment and it acts as the master of the  Spark  application. SparkContext sets up internal services and establishes a connection to a Spark execution environment. You can  create RDDs , accumulators and broadcast variables, access Spark services and run jobs (until SparkContext stops) after the creation of SparkContext. Only one SparkContext may be active per JVM. You must stop() the active SparkContext before creating a new one. In Spark shell, a special interpreter-aware SparkContext is already created for the user, in the variable called sc. The first step of any Spark driver application is to create a SparkContext. The SparkContext allows the Spark driver application to access the cluster through a resource manager. The resource manager can be  YARN , or  Spark’s Cluster Manager . Few functionalities which SparkContext offers are:  1. We can get the current status of a Spark application like c...
Read more