Accumulators are variables that are only “added” to through an associative operation and can therefore be efficiently supported in parallel. They can be used to implement counters (as in MapReduce) or sums. Spark naively supports accumulators of numeric types, and programmers can add support for new types. If accumulators are created with a name, they will be displayed in Spark’s UI. This can be useful for understanding the progress of running stages. An accumulator is created from an initial value v by calling SparkContext.accumulator(v) . Tasks running on the cluster can then add to it using the add method or the += operator (in Scala and Python). However, they cannot read its value. Only the driver program can read the accumulator’s value, using its value method. scala > val accum = sc . accumulator ( 0 , "My Accumulator" ) accum : spark.Accumulator [ Int ] = 0 scala > sc . parallelize ( Array ( 1 , 2 ,...
A SparkContext is a client of Spark’s execution environment and it acts as the master of the Spark application. SparkContext sets up internal services and establishes a connection to a Spark execution environment. You can create RDDs , accumulators and broadcast variables, access Spark services and run jobs (until SparkContext stops) after the creation of SparkContext. Only one SparkContext may be active per JVM. You must stop() the active SparkContext before creating a new one. In Spark shell, a special interpreter-aware SparkContext is already created for the user, in the variable called sc. The first step of any Spark driver application is to create a SparkContext. The SparkContext allows the Spark driver application to access the cluster through a resource manager. The resource manager can be YARN , or Spark’s Cluster Manager . Few functionalities which SparkContext offers are: 1. We can get the current status of a Spark application like c...
We have a requirement to monitor spark applications. For monitoring spark applications we used monit tool. Monit: Monit - utility for monitoring services on a Unix system, it is a open source For reference about Monit, please visit the manual page of Monit: https://mmonit.com/monit/documentation/monit.html#NAME . For Apache Spark Monitoring we want to keep a monitoring on Consumer (com.apple.gbi.edw.platform.spark.monitoring.eventing.EventStreamToDatabaseClient). Environment : Hadoop Cluster: AppServer: my-host-name02.com. USER: myuser MONIT LOCATION: /opts/dev/myuser/MONIT/SPARK_MONITORING Created MONIT_HOME variable in .bash_profile Download the latest version of Monit: https://bitbucket.org/tildeslash/monit/downloads/?tab=downloads Downloaded version: monit-5.25.1-linux-x64.tar.gz
Comments
Post a Comment