SBT console increase memory
export SBT_OPTS="-Xmx6G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=250M -Xss256M -Duser.timezone=GMT"
sbt -v console
Runtime.getRuntime().freeMemory()
Runtime.getRuntime().maxMemory()
Apache Spark Driver logs redirecting to directory in both cluster and client mode
Apache Spark driver class logs need to direct to directory in both cluster and client mode. Application users can log useful information in driver class. We have two things: 1) Spark Application run in yarn-client mode 2) Spark Application run in yarn-cluster mode Spark Application run in yarn-client mode When running a job in yarn-client mode, the driver logs are spilled on the console, but this may not useful for longer run, because the terminal will be aborted. So it is always a good approach to log the driver information in to definite location. Following is the approach as discussed in the HDP blogspot for yarn-client mode: https://community.hortonworks.com/articles/138849/how-to-capture-spark-driver-and-executor-logs-in-y.html. Here are the steps: 1. Place a driver_log4j.properties file in a certain location (say /tmp) on the machine where you will be submitting the job in yarn-client mode Contents o...
Comments
Post a Comment