Hadoop Ecosystem Operations Guide: HDFS, YARN, and MapReduce Operational Standards

• Introduction
• 1. HDFS Operational Standards
  • 1.1 Block Size Configuration
  • 1.2 Replication Factor Management
  • 1.3 NameNode Management
  • 1.4 Safe Mode Management
  • 1.5 HDFS Balancer Operations
• 2. HDFS Capacity Planning and Data Lifecycle
  • 2.1 HDFS Quota Management
  • 2.2 Data Compression Strategy
  • 2.3 Erasure Coding (EC)
  • 2.4 Tiered Storage
  • 2.5 Data Lifecycle Automation
• 3. YARN Queue Design and Resource Management
  • 3.1 Capacity Scheduler vs Fair Scheduler
  • 3.2 Capacity Scheduler Queue Design
  • 3.3 Preemption Configuration
  • 3.4 YARN Resource Configuration
• 4. MapReduce Job Tuning
  • 4.1 Mapper/Reducer Count Optimization
  • 4.2 Shuffle Optimization
  • 4.3 Data Skew Handling
  • 4.4 Speculative Execution
  • 4.5 JVM Reuse
• 5. Cluster Monitoring and Alerting
  • 5.1 Key Monitoring Metrics
  • 5.2 Metric Collection via JMX
  • 5.3 Prometheus + Grafana Integration
  • 5.4 Grafana Alert Rule Examples
  • 5.5 Ambari / Cloudera Manager
• 6. Incident Response Guide
  • 6.1 DataNode Failure
  • 6.2 NameNode Failover
  • 6.3 Disk Failure Response
  • 6.4 YARN ResourceManager Failure
• 7. Operational Checklists
  • 7.1 Daily Checklist
  • 7.2 Weekly Checklist
  • 7.3 Monthly Checklist
  • 7.4 Capacity Planning Spreadsheet
• 8. Conclusion
• Quiz

Introduction

Setting up a Hadoop cluster and operating it reliably are two entirely different challenges. There is no shortage of installation guides, but it is far harder to find documentation on practical operational questions: what to do when HDFS hits 90% capacity, how to configure preemption policies when YARN queues are saturated, or where to look when a MapReduce job has been running for 4 hours due to data skew.

This guide covers operational standards for production Hadoop ecosystems (HDFS, YARN, MapReduce), including capacity planning, queue design, job tuning, monitoring, and incident response. It is based on Hadoop 3.x and includes real configuration examples and commands in every section.

1. HDFS Operational Standards

1.1 Block Size Configuration

The default HDFS block size is 128MB. For environments handling large files, 256MB is the common recommendation.

<!-- hdfs-site.xml -->
<property>
  <name>dfs.blocksize</name>
  <value>268435456</value> <!-- 256MB -->
  <description>256MB recommended for large ETL files. Keep 128MB if small files dominate.</description>
</property>

Block Size Selection Criteria:

Operational Rule: Each block consumes approximately 150 bytes of NameNode heap memory. 100 million blocks require about 15GB of NameNode heap. Reducing block count directly improves NameNode stability.

1.2 Replication Factor Management

The default replication factor is 3. Lowering it saves storage but reduces fault tolerance.

# Check cluster-wide replication factor
hdfs dfsadmin -report | grep "Default Replication"

# Change replication factor for specific directory (cold data archiving)
hdfs dfs -setrep -w 2 /data/archive/2024/

# Check under-replicated blocks
hdfs fsck / -files -blocks -replicaDetails | grep "Under-replicated"

Replication Factor Policy Example:

• /data/raw/ (source data): replication factor 3
• /data/processed/ (processed data): replication factor 2
• /data/tmp/ (temporary data): replication factor 1
• Erasure Coding directories: N/A (separate policy)

1.3 NameNode Management

The NameNode is the single point of failure (SPOF) in HDFS. HA configuration is mandatory. The following items require regular inspection.

<!-- hdfs-site.xml: NameNode HA configuration -->
<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>
<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  <value>namenode1.example.com:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  <value>namenode2.example.com:8020</value>
</property>
<property>
  <name>dfs.namenode.handler.count</name>
  <value>128</value>
  <description>Client RPC handler thread count. 128-256 recommended for 100+ node clusters.</description>
</property>

# Check NameNode status
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2

# Monitor NameNode heap usage
jstat -gcutil $(jps | grep NameNode | awk '{print $1}') 5000

# Check EditLog size (excessive transaction logs cause issues)
hdfs dfsadmin -fetchImage /tmp/fsimage_check
ls -lh /tmp/fsimage_check

1.4 Safe Mode Management

When NameNode is in safe mode, write operations are blocked. It activates automatically until sufficient block reports are received.

# Check safe mode status
hdfs dfsadmin -safemode get

# Force leave safe mode (caution: only after verifying block integrity)
hdfs dfsadmin -safemode leave

# Check safe mode entry conditions
hdfs dfsadmin -report | grep -E "Safe mode|Missing blocks"

<!-- hdfs-site.xml: Safe mode threshold adjustment -->
<property>
  <name>dfs.namenode.safemode.threshold-pct</name>
  <value>0.999</value>
  <description>Exit safe mode when 99.9% of blocks are reported</description>
</property>
<property>
  <name>dfs.namenode.safemode.extension</name>
  <value>30000</value>
  <description>Additional wait time after safe mode exit (30 seconds)</description>
</property>

1.5 HDFS Balancer Operations

When disk utilization variance across DataNodes grows, hotspots form on certain nodes. Run the Balancer regularly.

# Run Balancer (default threshold 10%)
hdfs balancer -threshold 5

# Set bandwidth limit (required during production hours)
hdfs dfsadmin -setBalancerBandwidth 52428800  # 50MB/s

# Run Balancer in background (register with cron)
nohup hdfs balancer -threshold 5 -idleiterations 5 > /var/log/hadoop/balancer.log 2>&1 &

Balancer Operational Rules:

• Run via cron during off-hours (00:00-06:00)
• Limit bandwidth to 20MB/s or less during production hours
• Set threshold in the 5-10% range
• Always run after adding new DataNodes

2. HDFS Capacity Planning and Data Lifecycle

2.1 HDFS Quota Management

Directory-level quotas are essential to prevent storage runaway.

# Set name quota (file/directory count limit)
hdfs dfsadmin -setQuota 1000000 /data/team-a/

# Set space quota (capacity limit, includes replication factor)
hdfs dfsadmin -setSpaceQuota 10T /data/team-a/

# Check quota status
hdfs dfs -count -q -h /data/team-a/
# Output: QUOTA  REM_QUOTA  SPACE_QUOTA  REM_SPACE_QUOTA  DIR_COUNT  FILE_COUNT  CONTENT_SIZE  PATHNAME

# Clear quota
hdfs dfsadmin -clrSpaceQuota /data/team-a/

Team Quota Design Example:

2.2 Data Compression Strategy

Applying compression in HDFS saves both storage and network I/O.

<!-- core-site.xml: Compression codec registration -->
<property>
  <name>io.compression.codecs</name>
  <value>
    org.apache.hadoop.io.compress.GzipCodec,
    org.apache.hadoop.io.compress.SnappyCodec,
    org.apache.hadoop.io.compress.ZStandardCodec,
    org.apache.hadoop.io.compress.Lz4Codec
  </value>
</property>

<!-- mapred-site.xml: MapReduce intermediate output compression -->
<property>
  <name>mapreduce.map.output.compress</name>
  <value>true</value>
</property>
<property>
  <name>mapreduce.map.output.compress.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

2.3 Erasure Coding (EC)

Introduced in Hadoop 3.x, Erasure Coding saves approximately 50% storage compared to replication factor 3 while providing equivalent fault tolerance.

# List available EC policies
hdfs ec -listPolicies

# Enable RS-6-3 policy (6 data blocks + 3 parity blocks)
hdfs ec -enablePolicy -policy RS-6-3-1024k

# Apply EC policy to a directory
hdfs ec -setPolicy -path /data/archive -policy RS-6-3-1024k

# Check EC policy
hdfs ec -getPolicy -path /data/archive

EC Considerations:

• Requires at least 9 DataNodes (for RS-6-3)
• Random read performance is lower than replication (encoding/decoding overhead)
• Better suited for cold/warm data than hot data
• Cannot be applied to existing files (must rewrite or use distcp)

2.4 Tiered Storage

HDFS storage policies allow hierarchical management across SSD, HDD, and archive storage.

# List storage policies
hdfs storagepolicies -listPolicies

# Hot data -> SSD
hdfs storagepolicies -setStoragePolicy -path /data/hot -policy HOT

# Warm data -> 1 SSD + N HDD
hdfs storagepolicies -setStoragePolicy -path /data/warm -policy WARM

# Cold data -> Archive
hdfs storagepolicies -setStoragePolicy -path /data/cold -policy COLD

# Move data according to policy
hdfs mover -p /data/

<!-- hdfs-site.xml: DataNode storage type configuration -->
<property>
  <name>dfs.datanode.data.dir</name>
  <value>[SSD]/ssd/hdfs/data,[DISK]/hdd1/hdfs/data,[DISK]/hdd2/hdfs/data,[ARCHIVE]/archive/hdfs/data</value>
</property>

2.5 Data Lifecycle Automation

#!/bin/bash
# data_lifecycle.sh - Data lifecycle automation script

DATE=$(date +%Y-%m-%d)
LOG="/var/log/hadoop/lifecycle_${DATE}.log"

echo "[${DATE}] Data lifecycle management started" >> ${LOG}

# 1. Delete temporary data older than 90 days
echo "=== Temp data cleanup ===" >> ${LOG}
hdfs dfs -find /data/tmp -name "*" -atime +90 -print >> ${LOG}
hdfs dfs -rm -r -skipTrash $(hdfs dfs -find /data/tmp -name "*" -atime +90) 2>> ${LOG}

# 2. Reduce replication factor to 2 for processed data older than 30 days
echo "=== Replication factor adjustment ===" >> ${LOG}
for dir in $(hdfs dfs -ls /data/processed/ | awk '{print $8}' | tail -n +2); do
    mod_date=$(hdfs dfs -stat "%Y" ${dir})
    age_days=$(( ($(date +%s) - $(date -d "${mod_date}" +%s)) / 86400 ))
    if [ ${age_days} -gt 30 ]; then
        hdfs dfs -setrep 2 ${dir} >> ${LOG} 2>&1
    fi
done

# 3. Move data older than 180 days to Erasure Coding directory
echo "=== Archive migration ===" >> ${LOG}
hadoop distcp -skipcrccheck /data/processed/old/ /data/archive/ >> ${LOG} 2>&1

# 4. Capacity report
echo "=== Capacity status ===" >> ${LOG}
hdfs dfs -du -s -h /data/* >> ${LOG}

echo "[${DATE}] Data lifecycle management completed" >> ${LOG}

3. YARN Queue Design and Resource Management

3.1 Capacity Scheduler vs Fair Scheduler

Recommendation: Capacity Scheduler is the official default in Hadoop 3.x and is best suited for large multi-tenant environments.

3.2 Capacity Scheduler Queue Design

<!-- capacity-scheduler.xml -->
<configuration>

  <!-- Root queue children -->
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>production,development,system</value>
  </property>

  <!-- production queue: 60% of total resources -->
  <property>
    <name>yarn.scheduler.capacity.root.production.capacity</name>
    <value>60</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.maximum-capacity</name>
    <value>80</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.queues</name>
    <value>etl,realtime</value>
  </property>

  <!-- production > etl queue -->
  <property>
    <name>yarn.scheduler.capacity.root.production.etl.capacity</name>
    <value>70</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.etl.maximum-capacity</name>
    <value>90</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.etl.user-limit-factor</name>
    <value>2</value>
  </property>

  <!-- production > realtime queue -->
  <property>
    <name>yarn.scheduler.capacity.root.production.realtime.capacity</name>
    <value>30</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.production.realtime.maximum-capacity</name>
    <value>50</value>
  </property>

  <!-- development queue: 30% of total resources -->
  <property>
    <name>yarn.scheduler.capacity.root.development.capacity</name>
    <value>30</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.development.maximum-capacity</name>
    <value>50</value>
  </property>

  <!-- system queue: 10% of total resources (monitoring, maintenance) -->
  <property>
    <name>yarn.scheduler.capacity.root.system.capacity</name>
    <value>10</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.system.maximum-capacity</name>
    <value>20</value>
  </property>

</configuration>

Queue Structure Diagram:

root (100%)
├── production (60%, max 80%)
│   ├── etl (70% of production, max 90%)
│   └── realtime (30% of production, max 50%)
├── development (30%, max 50%)
└── system (10%, max 20%)

3.3 Preemption Configuration

Preemption allows queues to reclaim guaranteed resources by forcibly terminating containers from other queues.

<!-- yarn-site.xml -->
<property>
  <name>yarn.resourcemanager.scheduler.monitor.enable</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.monitor.policies</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value>
</property>

<!-- Preemption wait time: start after 15 seconds -->
<property>
  <name>yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval</name>
  <value>3000</value>
</property>
<property>
  <name>yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</name>
  <value>15000</value>
</property>

<!-- Maximum preemption ratio per round -->
<property>
  <name>yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round</name>
  <value>0.1</value>
</property>

3.4 YARN Resource Configuration

<!-- yarn-site.xml: NodeManager resource configuration -->
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>65536</value> <!-- 64GB (approximately 80% of total memory) -->
</property>
<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>24</value> <!-- 1.5-2x physical core count -->
</property>

<!-- Container memory range -->
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>32768</value>
</property>

<!-- Container vcore range -->
<property>
  <name>yarn.scheduler.minimum-allocation-vcores</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-vcores</name>
  <value>12</value>
</property>

# Check YARN queue status
yarn queue -status production

# List running applications
yarn application -list -appStates RUNNING

# Check queue resource usage
yarn queue -status production.etl

# Reload YARN scheduler configuration (no restart required)
yarn rmadmin -refreshQueues

4. MapReduce Job Tuning

4.1 Mapper/Reducer Count Optimization

<!-- mapred-site.xml -->

<!-- Mapper memory: default 1GB, 2-4GB recommended for large datasets -->
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>2048</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1638m</value> <!-- 80% of memory.mb -->
</property>

<!-- Reducer memory: set larger than mapper (more aggregation work) -->
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx3276m</value> <!-- 80% of memory.mb -->
</property>

<!-- Explicit reducer count (adjust based on data size) -->
<property>
  <name>mapreduce.job.reduces</name>
  <value>100</value>
</property>

Mapper/Reducer Count Formulas:

Mapper count ≈ Input data size / Block size
  Example: 1TB input, 256MB blocks -> ~4,000 mappers

Reducer count ≈ 0.95 x (total reduce slots in cluster)
  Or tune empirically so each reducer processes 256MB-1GB

4.2 Shuffle Optimization

The shuffle phase is the most expensive stage of MapReduce, concentrating both network and disk I/O.

<!-- mapred-site.xml: Map-side sort/spill configuration -->
<property>
  <name>mapreduce.task.io.sort.mb</name>
  <value>512</value>
  <description>Sort buffer size. Increasing from default 100MB to 512MB reduces spill count.</description>
</property>
<property>
  <name>mapreduce.task.io.sort.factor</name>
  <value>100</value>
  <description>Number of streams to merge simultaneously. Increase from default 10 to 100.</description>
</property>
<property>
  <name>mapreduce.map.sort.spill.percent</name>
  <value>0.80</value>
  <description>Begin spilling when sort buffer reaches 80%</description>
</property>

<!-- Reduce-side shuffle configuration -->
<property>
  <name>mapreduce.reduce.shuffle.parallelcopies</name>
  <value>20</value>
  <description>Parallel threads for fetching mapper output</description>
</property>
<property>
  <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
  <value>0.70</value>
  <description>Heap fraction allocated for shuffle input</description>
</property>
<property>
  <name>mapreduce.reduce.shuffle.merge.percent</name>
  <value>0.66</value>
  <description>In-memory merge threshold</description>
</property>

4.3 Data Skew Handling

Data skew occurs when certain keys have disproportionate amounts of data, overloading specific reducers.

Diagnosing Skew:

# Check per-reducer input record variance from job counters
yarn logs -applicationId application_1234567890_0001 | grep "Reduce input records"

# Check per-task execution time from job history server
mapred job -history /path/to/job_history_file

Skew Resolution Methods:

1. Key Salting: Add random prefixes to distribute keys evenly

// Add salt to key in mapper
int salt = random.nextInt(10);
outputKey.set(salt + "_" + originalKey);

// First MR: partial aggregation with salted keys
// Second MR: remove salt and perform final aggregation

2. Combiner: Pre-aggregate mapper output locally

job.setCombinerClass(MyCombiner.class);

3. Custom Partitioner: Partition based on data distribution

public class SkewAwarePartitioner extends Partitioner<Text, IntWritable> {
    @Override
    public int getPartition(Text key, IntWritable value, int numPartitions) {
        String k = key.toString();
        if (k.equals("HOT_KEY")) {
            // Distribute hot keys across multiple reducers
            return (k.hashCode() + random.nextInt(10)) % numPartitions;
        }
        return (k.hashCode() & Integer.MAX_VALUE) % numPartitions;
    }
}

4.4 Speculative Execution

When a slow task (straggler) is detected, the same task is launched on another node in parallel. The result from whichever finishes first is used.

<!-- mapred-site.xml -->
<property>
  <name>mapreduce.map.speculative</name>
  <value>true</value>
</property>
<property>
  <name>mapreduce.reduce.speculative</name>
  <value>false</value>
  <description>Disable for reducers since they consume significant data. Recommended off.</description>
</property>

Warning: Speculative execution must be disabled for tasks that write to external systems (DB inserts, API calls, etc.) to prevent duplicate writes.

4.5 JVM Reuse

Creating a new JVM for every task adds significant overhead. This is especially effective when there are many small tasks.

<!-- mapred-site.xml -->
<property>
  <name>mapreduce.job.jvm.numtasks</name>
  <value>10</value>
  <description>Run up to 10 tasks per JVM. Set to -1 for unlimited.</description>
</property>

5. Cluster Monitoring and Alerting

5.1 Key Monitoring Metrics

HDFS Core Metrics:

YARN Core Metrics:

5.2 Metric Collection via JMX

# Query NameNode JMX metrics
curl -s http://namenode:9870/jmx | python3 -m json.tool | grep -A5 "CapacityUsed"

# Query ResourceManager JMX metrics
curl -s http://resourcemanager:8088/jmx | python3 -m json.tool | grep -A5 "AllocatedMB"

# Query DataNode JMX metrics
curl -s http://datanode:9864/jmx?qry=Hadoop:service=DataNode,name=FSDatasetState

5.3 Prometheus + Grafana Integration

# prometheus.yml - Hadoop JMX Exporter configuration
scrape_configs:
  - job_name: 'hadoop-namenode'
    scrape_interval: 30s
    static_configs:
      - targets: ['namenode1:7001', 'namenode2:7001']
        labels:
          cluster: 'production'
          component: 'namenode'

  - job_name: 'hadoop-datanode'
    scrape_interval: 30s
    static_configs:
      - targets: ['datanode1:7002', 'datanode2:7002', 'datanode3:7002']
        labels:
          cluster: 'production'
          component: 'datanode'

  - job_name: 'hadoop-resourcemanager'
    scrape_interval: 30s
    static_configs:
      - targets: ['resourcemanager:7003']
        labels:
          cluster: 'production'
          component: 'resourcemanager'

  - job_name: 'hadoop-nodemanager'
    scrape_interval: 30s
    static_configs:
      - targets: ['datanode1:7004', 'datanode2:7004', 'datanode3:7004']
        labels:
          cluster: 'production'
          component: 'nodemanager'

JMX Exporter Launch Configuration (hadoop-env.sh):

# hadoop-env.sh
export HDFS_NAMENODE_OPTS="$HDFS_NAMENODE_OPTS -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7001:/opt/jmx_exporter/namenode.yml"
export HDFS_DATANODE_OPTS="$HDFS_DATANODE_OPTS -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7002:/opt/jmx_exporter/datanode.yml"
export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7003:/opt/jmx_exporter/resourcemanager.yml"
export YARN_NODEMANAGER_OPTS="$YARN_NODEMANAGER_OPTS -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7004:/opt/jmx_exporter/nodemanager.yml"

5.4 Grafana Alert Rule Examples

# Grafana Alert Rules
groups:
  - name: hadoop_alerts
    rules:
      - alert: HDFSCapacityCritical
        expr: hadoop_namenode_capacity_used_gb / hadoop_namenode_capacity_total_gb > 0.9
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: 'HDFS capacity exceeds 90%'
          description: 'HDFS utilization is {{ $value | humanizePercentage }}. Immediate data cleanup or node addition required.'

      - alert: HDFSMissingBlocks
        expr: hadoop_namenode_missing_blocks > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'HDFS missing blocks detected'
          description: '{{ $value }} blocks are missing.'

      - alert: YARNUnhealthyNodes
        expr: hadoop_resourcemanager_unhealthy_nodes > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'YARN unhealthy nodes detected'
          description: '{{ $value }} nodes are in unhealthy state.'

      - alert: YARNMemoryPressure
        expr: hadoop_resourcemanager_allocated_mb / hadoop_resourcemanager_available_mb > 0.95
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: 'YARN memory resource shortage'
          description: 'Cluster memory utilization has exceeded 95%.'

5.5 Ambari / Cloudera Manager

Dedicated management tools provide integrated configuration management, monitoring, and alerting.

6. Incident Response Guide

6.1 DataNode Failure

Symptoms: DataNode process down, network disconnection, disk failure

# Step 1: Check dead DataNodes
hdfs dfsadmin -report | grep -A3 "Dead datanodes"

# Step 2: Check DataNode logs
tail -500 /var/log/hadoop/hdfs/hadoop-hdfs-datanode-*.log | grep -E "ERROR|WARN|FATAL"

# Step 3: Check disk status
df -h  # Disk capacity
smartctl -a /dev/sda  # Disk health
dmesg | grep -i "error\|fail\|i/o"  # Check kernel logs for disk errors

# Step 4: Restart or remove DataNode
# If restartable:
hdfs --daemon start datanode

# If decommission is required:
# 1) Add node to dfs.hosts.exclude
# 2) Refresh configuration
hdfs dfsadmin -refreshNodes
# 3) Monitor data migration progress
hdfs dfsadmin -report | grep "Decommission Status"

DataNode Decommission Configuration:

<!-- hdfs-site.xml -->
<property>
  <name>dfs.hosts.exclude</name>
  <value>/etc/hadoop/conf/dfs.exclude</value>
</property>

# /etc/hadoop/conf/dfs.exclude
datanode-bad-01.example.com
datanode-bad-02.example.com

6.2 NameNode Failover

In HA environments, automatic failover should activate when the Active NameNode fails.

# Check NameNode status
hdfs haadmin -getAllServiceState
# Output example:
# nn1    active
# nn2    standby

# Manual failover (emergency)
hdfs haadmin -failover nn1 nn2

# Check ZKFC process (core of automatic failover)
jps | grep DFSZKFailoverController

# Check ZooKeeper session state
echo "stat" | nc zookeeper1 2181

# When failover is not working: restart ZKFC
hdfs --daemon stop zkfc
hdfs --daemon start zkfc

Automatic Failover Configuration:

<!-- hdfs-site.xml -->
<property>
  <name>dfs.ha.automatic-failover.enabled</name>
  <value>true</value>
</property>

<!-- core-site.xml -->
<property>
  <name>ha.zookeeper.quorum</name>
  <value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181</value>
</property>

6.3 Disk Failure Response

When a specific disk fails on a DataNode, you can exclude just that disk without taking down the entire DataNode.

<!-- hdfs-site.xml: Disk failure tolerance -->
<property>
  <name>dfs.datanode.failed.volumes.tolerated</name>
  <value>2</value>
  <description>Allow up to 2 volume failures before DataNode shutdown. Keep below 1/3 of total volumes.</description>
</property>

# Disk replacement procedure
# 1. Identify failed disk
hdfs dfsadmin -getDatanodeInfo datanode01:9866

# 2. Remove the volume from dfs.datanode.data.dir
# 3. Reconfigure DataNode (Hadoop 3.x supports live reconfig)
hdfs dfsadmin -reconfig datanode datanode01:9866 start

# 4. Check reconfig status
hdfs dfsadmin -reconfig datanode datanode01:9866 status

# 5. Mount new disk, add to data.dir, reconfig again

6.4 YARN ResourceManager Failure

# Check RM HA status
yarn rmadmin -getAllServiceState

# Manual RM transition
yarn rmadmin -transitionToActive rm2

# When NodeManager cannot connect to RM
# Check NM logs
tail -200 /var/log/hadoop/yarn/hadoop-yarn-nodemanager-*.log | grep "ERROR"

# Restart NM
yarn --daemon stop nodemanager
yarn --daemon start nodemanager

7. Operational Checklists

7.1 Daily Checklist

#!/bin/bash
# daily_health_check.sh

echo "========== HDFS Daily Check =========="
# HDFS status summary
hdfs dfsadmin -report | head -20

# Check missing/under-replicated blocks
hdfs fsck / -list-corruptfileblocks

# Check NameNode safe mode
hdfs dfsadmin -safemode get

echo "========== YARN Daily Check =========="
# YARN node status
yarn node -list -all | head -20

# Failed applications (last 24 hours)
yarn application -list -appStates FAILED

# Queue usage summary
yarn queue -status root

echo "========== System Check =========="
# Disk usage
df -h | grep -E "/data|/hdfs"

# Memory/CPU
free -g
uptime

7.2 Weekly Checklist

7.3 Monthly Checklist

7.4 Capacity Planning Spreadsheet

Current cluster state:
- Total DataNodes: N
- Disks per node: 12 x 4TB HDD = 48TB
- Total raw capacity: N x 48TB
- HDFS usable capacity (replication factor 3): N x 48TB / 3 = N x 16TB
- Current HDFS usage: X TB
- Utilization: X / (N x 16) x 100 = Y%

Monthly growth: Z TB/month

Node addition timeline:
- Safety threshold: 80%
- Remaining capacity: (N x 16 x 0.8) - X = W TB
- At current growth rate: W / Z = M months until 80%
- Account for node procurement lead time -> order M - 2 months ahead

8. Conclusion

Operating a Hadoop cluster requires significantly more effort in ongoing management and tuning than in initial deployment. Here is a summary of the key points covered in this guide.

HDFS: Configure block size and replication factor based on data characteristics. Establish NameNode heap management and Balancer runs as regular routines. Leverage Erasure Coding and Tiered Storage to optimize storage costs.

YARN: Design hierarchical queues with Capacity Scheduler and enable preemption policies to ensure resource fairness. Periodically rebalance queue capacity based on actual usage patterns.

MapReduce: Shuffle optimization (sort buffer, spill threshold) and data skew handling are critical to job performance. Configure JVM reuse and speculative execution appropriately for each scenario.

Monitoring: Build a JMX + Prometheus + Grafana pipeline and set up alerts for HDFS capacity/block health and YARN resource/queue status.

Incident Response: Document and rehearse NameNode HA failover, DataNode decommission, and disk hot-swap procedures in advance.

Running operational checklists on daily/weekly/monthly cadences and building capacity plans based on data-driven projections will proactively prevent most operational issues.

Quiz