Hadoop - Hadoop Parameter Cheat Sheet

Hadoop - Hadoop Parameter Cheat Sheet

Hadoop Parameter Cheat Sheet

Hadoop Parameter Cheat Sheet

HDP Kerberos

hadoop.security.authentication=kerberos

alpine.principal=alpine/chorus.alpinenow.local@ALPINENOW.LOCAL
alpine.keytab=/home/chorus/alpine.keytab

mapreduce.jobhistory.principal=jhs/_HOST@ALPINENOW.LOCAL
yarn.resourcemanager.principal=rm/_HOST@ALPINENOW.LOCAL
dfs.namenode.kerberos.principal=nn/_HOST@ALPINENOW.LOCAL
dfs.datanode.kerberos.principal=dn/_HOST@ALPINENOW.LOCAL

CDH Kerberos

hadoop.rpc.protection=privacy (CDH only)
hadoop.security.authentication=kerberos
alpine.principal=alpine/chorus.alpinenow.local@ALPINENOW.LOCAL
alpine.keytab=/home/chorus/keytab/alpine.keytab
mapreduce.jobhistory.principal=mapred/_HOST@ALPINENOW.LOCAL
yarn.resourcemanager.principal=yarn/_HOST@ALPINENOW.LOCAL

Hive

hive.metastore.client.connect.retry.delay=1
hive.metastore.client.socket.timeout=600

Add this property if the value is different than 'Hive Metastore Location'
hive.hiveserver2.uris=jdbc:hive2://hdp23.alpinenow.local:10000/default

Hive Kerberos (add to Hive params)

hive.metastore.kerberos.principal=hive/_HOST@ALPINENOW.LOCAL

Add this property if the value is different than hive.metastore.kerberos.principal
hive.server2.authentication.kerberos.principal=hive/_HOST@ALPINENOW.LOCAL

HA NN+RM

dfs.client.failover.proxy.provider.nameservice1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.automatic-failover.enabled.nameservice1=true
dfs.ha.namenodes.nameservice1=namenode44,namenode49

for each nn

dfs.namenode.http-address.nameservice1.namenode44=nn1.alpinenow.local:50070
dfs.namenode.http-address.nameservice1.namenode49=nn2.alpinenow.local:50070
dfs.namenode.https-address.nameservice1.namenode44=nn1.alpinenow.local:50470
dfs.namenode.https-address.nameservice1.namenode49=nn2.alpinenow.local:50470

dfs.namenode.rpc-address.nameservice1.namenode44=nn1.alpinenow.local:8020
dfs.namenode.rpc-address.nameservice1.namenode49=nn2.alpinenow.local:8020
dfs.namenode.servicerpc-address.nameservice1.namenode44=nn1.alpinenow.local:8022
dfs.namenode.servicerpc-address.nameservice1.namenode49=nn2.alpinenow.local:8022

ha.zookeeper.quorum=cm.alpinenow.local:2181,nn1.alpinenow.local:2181,nn2.alpinenow.local:2181
failover_resource_manager_hosts=nn1.alpinenow.local,nn2.alpinenow.local

Queues

To submit MapReduce jobs to a specific queue:

mapred.job.queue.name=[queue_name]

spark.yarn.queue=[queue_name]