Posts

Showing posts from June, 2018

Hive insert overwrite output data delimiter

User cannot specify the data delimiter in hive during insert overwrite. The default delimiter it generates is '\1'

Application Localisation in YARN and its pitfalls

An application container before it is started goes through a phase called Application Localisation. Node Manager takes care of this activity only once for an application irrespective of number of application's containers it has been entrusted to execute. This step mostly includes -- Downloading the application jars, files and archives from HDFS -- Log Aggregation setup for application If you notice your container not starting ( check container logs) as soon as the Application Master has sent a request to Node Manager ( check node manager logs ), it indicates delays with localisation. Use the jars more judiciously for your application to manage the localisation overhead.

How to give arguments to kill via pipe

kill $(ps -e | grep <YOUR_SEARCH_STRING> | awk '{print $1}')