In how many ways can we create RDDs in Apache Spark?

These are three methods to create the RDD.
1.The first method is used when data is already available with the external systems like local filesystem, HDFS, HBase
RDD can be created by calling a textFile method of SparkContext with path / URL as the argument.
scala> val data = sc.textFile("File1.txt")
sc is the object of SparkContext
You need to create a file File1.txt in Spark_Home directory
2.The second approach can be used with the existing collections
scala> val arr1 = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala> val rdd1 = sc.parallelize(arr1)
3.The third one is a way to create new RDD from the existing one.
scala> val newRDD = arr1.map(data => (data * 2))
Ref : http://data-flair.training/forums/topic/in-how-many-ways-can-we-create-rdds-in-apache-spark-explain

Comments

Post a Comment

Popular posts from this blog

Apache Spark Driver logs redirecting to directory in both cluster and client mode

Apache Spark driver class logs need to direct to directory in both cluster and client mode. Application users can log useful information in driver class.  We have two things: 1) Spark Application run in yarn-client mode 2) Spark Application run in yarn-cluster mode Spark Application run in yarn-client mode When running a job in yarn-client mode, the driver logs are spilled on the console, but this may not useful for longer run, because the terminal will be aborted. So it is always a good approach to log the driver information in to definite location. Following is the approach as discussed in the HDP blogspot for yarn-client mode:  https://community.hortonworks.com/articles/138849/how-to-capture-spark-driver-and-executor-logs-in-y.html. Here are the steps: 1.     Place a driver_log4j.properties file in a certain location (say /tmp) on the machine where you will be submitting the job in yarn-client mode Contents o...
Read more

eclipse oxygen is not starting

I found Eclipse Oxygen is not working. it is throwing an below error I modified  workbench.xmi to  workbenchOLD.xmi. using  cd  .metadata/.plugins/org.eclipse.e4.workbench mv workbench.xmi workbenchOLD.xmi It got work !!!!!!!! in <ECLIPSE_WORKSPACE> .metadata/.plugins/org.eclipse.e4.workbench !ENTRY org.eclipse.e4.ui.workbench.swt 4 2 2018-05-14 13:02:12.435 !MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.e4.ui.workbench.swt". !STACK 0 java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:204) at scala.tools.scalap.Classfile.<init>(Classfile.scala:17) at org.scalaide.core.internal.jdt.model.ScalaClassFileDescriber$.isScala(ScalaClassFileDescriber.scala:16) at org.scalaide.core.internal.jdt.model.ScalaClassFileDescriber.describe(ScalaClassFileDescriber.scala:38) at org.eclipse.core.internal.content.ContentTypeCatalo...
Read more

Application Localisation in YARN and its pitfalls

An application container before it is started goes through a phase called Application Localisation. Node Manager takes care of this activity only once for an application irrespective of number of application's containers it has been entrusted to execute. This step mostly includes -- Downloading the application jars, files and archives from HDFS -- Log Aggregation setup for application If you notice your container not starting ( check container logs) as soon as the Application Master has sent a request to Node Manager ( check node manager logs ), it indicates delays with localisation. Use the jars more judiciously for your application to manage the localisation overhead.
Read more