HBase Data Modeling Playbook: Hotspot Avoidance and Performance Stabilization

1. Introduction
2. HBase Data Model Fundamentals
3. RowKey Design Patterns
4. Hotspot Causes and Detection
5. Hotspot Avoidance Strategies
6. Schema Design Patterns in Practice
7. Performance Stabilization Operations
8. Practical Checklist and Anti-Patterns
9. Conclusion
Quiz

1. Introduction

HBase is a distributed Column-Family NoSQL database designed based on the Google Bigtable paper. Running on top of HDFS, it can handle billions of rows and millions of columns with millisecond-level latency. However, this performance is only achievable when data modeling is done correctly. A single poorly designed RowKey can funnel all traffic to one RegionServer out of dozens, creating a hotspot that effectively reduces the entire cluster's throughput to that of a single server.

This article is not a general HBase architecture overview but a hands-on playbook focused on data modeling and hotspot avoidance. It systematically covers RowKey design patterns, hotspot detection methods, Region distribution strategies, schema design patterns, and operational techniques for performance stabilization.

2. HBase Data Model Fundamentals

Core Components

HBase's data model is fundamentally different from RDBMS. The following five elements constitute a single Cell.

Component	Description	Example
Row (RowKey)	A byte array that uniquely identifies a row. Lexicographically sorted	`user001_20260308`
Column Family	Physical grouping of columns. Defined at table creation	`cf`, `info`, `stats`
Column Qualifier	Individual column name within a Column Family. Dynamically addable	`name`, `email`, `count`
Timestamp	Version of a cell in milliseconds	`1709856000000`
Cell (Value)	The actual data value identified by the four coordinates above	`"John Doe"`

Physical Storage Structure

HBase data appears as tables logically but is physically stored separately per Column Family. This is the most important constraint in schema design.

Logical View:
┌──────────────────────────────────────────────────────────────┐
│ RowKey         │ CF:info            │ CF:metrics             │
│                │ name    │ email    │ cpu_avg  │ mem_used    │
├──────────────────────────────────────────────────────────────┤
│ server_web01   │ Web-01  │ ...      │ 72.5     │ 8192       │
│ server_db01    │ DB-01   │ ...      │ 45.2     │ 16384      │
└──────────────────────────────────────────────────────────────┘

Physical Storage (separate HFiles per Column Family):
[HFile: info]
(server_db01, info:email, t1) → "admin@example.com"
(server_db01, info:name, t1)  → "DB-01"
(server_web01, info:email, t1) → "web@example.com"
(server_web01, info:name, t1)  → "Web-01"

[HFile: metrics]
(server_db01, metrics:cpu_avg, t2)   → 45.2
(server_db01, metrics:mem_used, t2)  → 16384
(server_web01, metrics:cpu_avg, t2)  → 72.5
(server_web01, metrics:mem_used, t2) → 8192

RowKey Sorting and Region Distribution

An HBase table is sorted by RowKey in lexicographic order, and contiguous RowKey ranges form a Region. Regions are assigned to RegionServers that handle actual reads and writes.

Entire RowKey space:
[aaa...] ─────────── [mmm...] ─────────── [zzz...]

Region partitioning:
Region 1: [aaa ~ fff]  → RegionServer A
Region 2: [fff ~ mmm]  → RegionServer B
Region 3: [mmm ~ sss]  → RegionServer C
Region 4: [sss ~ zzz]  → RegionServer D

The critical insight here is that RowKey distribution directly determines load distribution. If writes are concentrated in a specific RowKey range, the single RegionServer hosting that Region becomes overloaded.

Column Family Design Principles

Column Families must be defined at table creation time and serve as the unit for physical storage and configuration (compression, TTL, version count, etc.). Follow these principles:

Keep CF count to 2-3 or fewer: Flushes cascade across CFs, so many CFs increase unnecessary I/O.
Separate data with different access patterns: Placing frequently read metadata and large binaries in the same CF degrades BlockCache efficiency.
Keep CF names short: CF names are stored repeatedly in every KeyValue, so i is more storage-efficient than information.

# Column Family design example
create 'metrics', \
  {NAME => 'd', VERSIONS => 1, COMPRESSION => 'SNAPPY', BLOOMFILTER => 'ROW', TTL => 7776000}, \
  {NAME => 'm', VERSIONS => 1, COMPRESSION => 'SNAPPY', BLOOMFILTER => 'ROW', TTL => 31536000}

# d: raw data (90-day TTL)
# m: aggregated metadata (1-year TTL)

3. RowKey Design Patterns

RowKey design determines 80% of HBase performance. You must consider read/write patterns, data distribution, and scan ranges.

3.1 Salting (Prefix Distribution)

Salting prepends a hash-based prefix (salt) to the RowKey to distribute data evenly across multiple Regions.

public class SaltedRowKeyGenerator {

    private static final int NUM_BUCKETS = 16;  // Aligned with Region count

    /**
     * Adds a salt prefix to the original RowKey for distribution.
     * Example: "20260308_sensor001" → "0a_20260308_sensor001"
     */
    public static byte[] generateSaltedKey(String originalKey) {
        int bucket = Math.abs(originalKey.hashCode() % NUM_BUCKETS);
        String saltPrefix = String.format("%02x", bucket);
        String saltedKey = saltPrefix + "_" + originalKey;
        return Bytes.toBytes(saltedKey);
    }

    /**
     * Reverse-computes the salt for a specific original key (for Get requests).
     */
    public static byte[] getSaltedKey(String originalKey) {
        return generateSaltedKey(originalKey);  // Same hash result
    }

    /**
     * Full range Scan: must execute parallel scans across all salt buckets.
     */
    public static List<Scan> createParallelScans(String startKey, String endKey) {
        List<Scan> scans = new ArrayList<>();
        for (int i = 0; i < NUM_BUCKETS; i++) {
            String prefix = String.format("%02x", i);
            Scan scan = new Scan();
            scan.withStartRow(Bytes.toBytes(prefix + "_" + startKey));
            scan.withStopRow(Bytes.toBytes(prefix + "_" + endKey));
            scans.add(scan);
        }
        return scans;
    }
}

Best for: Maximizing write throughput with infrequent range scans (log ingestion, event collection).

Trade-off: Range scans require parallel scans across all salt buckets, increasing scan cost.

3.2 Hashing (Hash Prefix)

Transforms the RowKey entirely or partially through a hash function for even distribution. Similar to salting but uses the hash result itself as a prefix for a wider distribution range.

import org.apache.commons.codec.digest.DigestUtils;

public class HashedRowKeyGenerator {

    /**
     * Uses the first 4 characters of an MD5 hash as prefix.
     * Provides 65,536 distribution buckets.
     */
    public static byte[] createHashedKey(String userId, long timestamp) {
        String baseKey = userId + "_" + timestamp;
        String hashPrefix = DigestUtils.md5Hex(baseKey).substring(0, 4);
        String rowKey = hashPrefix + "_" + userId + "_" + timestamp;
        return Bytes.toBytes(rowKey);
    }

    /**
     * Querying a specific user's data also requires the same hash computation.
     */
    public static byte[] getHashedKey(String userId, long timestamp) {
        return createHashedKey(userId, timestamp);
    }
}

// Usage example
// Original: "user001_1709856000000"
// Hashed: "a3f2_user001_1709856000000"

Best for: Point Get-oriented access patterns where the specific RowKey is known.

Warning: Hashing destroys original key ordering, making range scans effectively impossible.

3.3 Key Reversing

Reverses keys like domain names or timestamps where the prefix is similar but the suffix varies, achieving distribution.

public class ReversedKeyExamples {

    /**
     * Domain reversal: distributes domains under the same TLD.
     * "www.google.com" → "moc.elgoog.www"
     */
    public static String reverseDomain(String domain) {
        return new StringBuilder(domain).reverse().toString();
    }

    /**
     * Reverse Timestamp: pattern for scanning most recent data first.
     * Since HBase sorts RowKeys in ascending order, subtracting from
     * Long.MAX_VALUE places the latest timestamp at the top (smallest value).
     */
    public static long reverseTimestamp(long timestamp) {
        return Long.MAX_VALUE - timestamp;
    }

    /**
     * RowKey design for fast lookup of a user's most recent activity.
     */
    public static byte[] createUserActivityKey(String userId, long timestamp) {
        long reversedTs = Long.MAX_VALUE - timestamp;
        String rowKey = userId + "_" + String.format("%019d", reversedTs);
        return Bytes.toBytes(rowKey);
        // Result: "user001_9223370449055775807"
        // Scan(startRow=user001_, stopRow=user002) → returns in newest-first order
    }
}

Best for: Patterns like "most recent N entries for a specific user" where newest data within a prefix should be retrieved first.

3.4 Composite Key

Combines multiple dimensions of data into a single RowKey. Uses _ or null byte (\x00) as delimiters.

public class CompositeKeyDesign {

    /**
     * Composite key design for IoT sensor data.
     * Structure: {region_code}_{device_id}_{reverse_timestamp}
     *
     * - region_code: geographic distribution (2 bytes)
     * - device_id: device identification (variable)
     * - reverse_timestamp: newest-first ordering (8 bytes)
     */
    public static byte[] createIoTRowKey(String regionCode, String deviceId, long timestamp) {
        long reversedTs = Long.MAX_VALUE - timestamp;
        String rowKey = String.format("%s_%s_%019d", regionCode, deviceId, reversedTs);
        return Bytes.toBytes(rowKey);
    }

    /**
     * Composite key for messaging systems.
     * Structure: {chat_room_id}_{reverse_timestamp}_{message_id}
     * → Efficiently scan latest messages in a specific chat room
     */
    public static byte[] createMessageRowKey(String roomId, long timestamp, String msgId) {
        long reversedTs = Long.MAX_VALUE - timestamp;
        return Bytes.toBytes(roomId + "_" + String.format("%019d", reversedTs) + "_" + msgId);
    }
}

3.5 RowKey Design for Time-Series Data

Time-series data is the most common HBase workload and the type most prone to hotspots. Using sequential timestamps directly as RowKeys always concentrates writes on the last Region.

[Bad Design] Timestamp at the beginning of RowKey
RowKey: 20260308120000_sensor001
RowKey: 20260308120001_sensor001
RowKey: 20260308120002_sensor001
→ All writes concentrated on the last Region (hotspot!)

[Good Design] salt + deviceID + reverse timestamp
RowKey: 0a_sensor001_9223370449055775807
RowKey: 03_sensor002_9223370449055775807
RowKey: 0f_sensor003_9223370449055775807
→ Evenly distributed across 16 Regions

Recommended RowKey pattern for time-series data:

/**
 * RowKey generator for time-series metric collection.
 *
 * Pattern: {salt}_{metric_name}_{device_id}_{reverse_timestamp}
 *
 * Advantages:
 * 1. Salt for Region distribution
 * 2. metric_name + device_id to narrow Scan range
 * 3. reverse_timestamp for newest-first queries
 */
public class TimeSeriesRowKeyGenerator {

    private static final int SALT_BUCKETS = 32;

    public static byte[] generate(String metricName, String deviceId, long timestamp) {
        String baseKey = metricName + "_" + deviceId;
        int salt = Math.abs(baseKey.hashCode() % SALT_BUCKETS);
        long reversedTs = Long.MAX_VALUE - timestamp;

        String rowKey = String.format("%02x_%s_%s_%019d",
            salt, metricName, deviceId, reversedTs);
        return Bytes.toBytes(rowKey);
    }

    /**
     * Time-range Scan for a specific device's specific metric.
     * Since the salt is known, efficient single-Region scan is possible.
     */
    public static Scan createRangeScan(
            String metricName, String deviceId,
            long startTime, long endTime) {

        String baseKey = metricName + "_" + deviceId;
        int salt = Math.abs(baseKey.hashCode() % SALT_BUCKETS);
        String prefix = String.format("%02x", salt);

        // Since using reverse timestamp, start/end are inverted
        long reversedEnd = Long.MAX_VALUE - startTime;
        long reversedStart = Long.MAX_VALUE - endTime;

        Scan scan = new Scan();
        scan.withStartRow(Bytes.toBytes(
            prefix + "_" + metricName + "_" + deviceId + "_"
            + String.format("%019d", reversedStart)));
        scan.withStopRow(Bytes.toBytes(
            prefix + "_" + metricName + "_" + deviceId + "_"
            + String.format("%019d", reversedEnd)));
        scan.setCaching(500);
        return scan;
    }
}

RowKey Design Decision Guide

Access Pattern	Recommended RowKey Strategy	Rationale
Random Point Get	Hashing	Even distribution, order not needed
Latest N entries for an entity	entity_id + reverse timestamp	Prefix Scan for newest-first queries
High-volume sequential writes (logs)	Salting + composite key	Write distribution + minimal Scan support
Time-series range queries	salt + metric + device + reverse ts	Supports both distribution and range queries
Full-text search alternative	reverse domain + path	Subdomain grouping

4. Hotspot Causes and Detection

What Is a Hotspot

A hotspot occurs when read or write requests are abnormally concentrated on a specific Region. While HBase is designed for horizontal scaling, a hotspot turns a single RegionServer's processing limit into the bottleneck for the entire cluster.

Normal distribution:
RS-1: ████████  (25%)
RS-2: ████████  (25%)
RS-3: ████████  (25%)
RS-4: ████████  (25%)

Hotspot:
RS-1: ██  (5%)
RS-2: █   (2%)
RS-3: █   (3%)
RS-4: ████████████████████████████████████████  (90%)  ← Overloaded!

Primary Causes of Hotspots

1. Sequential RowKey

Using monotonically increasing values like timestamps or auto-increment IDs as RowKeys causes all new data to be written to the end of the key space (the last Region).

# Bad example: Timestamp-based RowKey
2026030812000001 → Region [2026030811~ 2026030812]  ← All writes here
2026030812000002 → Region [2026030811~ 2026030812]
2026030812000003 → Region [2026030811~ 2026030812]

2. Skewed Key Distribution

When data with a specific prefix vastly outnumbers other prefixes. For example, if 90% of data starts with user_ and only 10% uses other prefixes, Regions covering the user_ range become overloaded.

3. Popular Key (Hot Key)

When reads/writes concentrate on a few specific RowKeys. Celebrity profiles, popular product pages, etc.

Hotspot Detection Methods

Checking per-Region request counts with HBase Shell:

# Check Region distribution for a table
hbase shell <<'EOF'
status 'detailed'
EOF

# Check per-Region request counts for a specific table
hbase shell <<'EOF'
list_regions 'metrics_table'
EOF

# Example output:
# REGION                  | START_KEY  | END_KEY    | SIZE   | REQ    | LOCALITY
# metrics_table,,1709...  | (empty)    | 08_       | 2.1GB  | 1204   | 0.95
# metrics_table,08_,17... | 08_        | 10_       | 2.3GB  | 982341 | 0.92  ← Hotspot!
# metrics_table,10_,17... | 10_        | 18_       | 1.8GB  | 1102   | 0.97

Checking per-RegionServer load via JMX metrics:

# Compare request counts across RegionServers
curl -s "http://regionserver1:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
for bean in data['beans']:
    print(f\"ReadRequests: {bean.get('readRequestCount', 'N/A')}\")
    print(f\"WriteRequests: {bean.get('writeRequestCount', 'N/A')}\")
    print(f\"TotalRequests: {bean.get('totalRequestCount', 'N/A')}\")
"

# If a specific server's requests exceed 3x the average,
# suspect a hotspot

Automated hotspot detection script using HBase metrics:

#!/bin/bash
# hotspot_detector.sh - Detect per-Region request deviation

TABLE="metrics_table"
THRESHOLD=3.0  # Alert if 3x above average

echo "=== Hotspot Detection for $TABLE ==="

# Extract per-Region request counts
REGIONS=$(echo "list_regions '$TABLE'" | hbase shell 2>/dev/null | grep -E "^\s+\{" | \
  awk -F',' '{for(i=1;i<=NF;i++) if($i ~ /REQ/) print $i}' | \
  grep -oP '\d+')

if [ -z "$REGIONS" ]; then
    echo "No region data found."
    exit 1
fi

# Calculate average
TOTAL=0
COUNT=0
for req in $REGIONS; do
    TOTAL=$((TOTAL + req))
    COUNT=$((COUNT + 1))
done
AVG=$((TOTAL / COUNT))

echo "Average requests per region: $AVG"
echo "Threshold (${THRESHOLD}x average): $(echo "$AVG * $THRESHOLD" | bc | cut -d. -f1)"
echo ""

# Identify hotspot Regions
IDX=0
for req in $REGIONS; do
    RATIO=$(echo "scale=2; $req / $AVG" | bc)
    if (( $(echo "$RATIO > $THRESHOLD" | bc -l) )); then
        echo "[HOTSPOT] Region $IDX: $req requests (${RATIO}x average)"
    fi
    IDX=$((IDX + 1))
done

5. Hotspot Avoidance Strategies

5.1 Pre-splitting

Pre-splitting Regions at table creation time prevents load concentration on a single Region during initial data loading.

# Method 1: Uniform split (hex-based)
# Suitable when RowKeys start with hash prefixes
create 'events', 'data', SPLITS => [
  '10', '20', '30', '40', '50', '60', '70', '80', '90',
  'a0', 'b0', 'c0', 'd0', 'e0', 'f0'
]

# Method 2: HexStringSplit utility
# Evenly splits the hex key space into the specified number of Regions
create 'events', 'data', {NUMREGIONS => 16, SPLITALGO => 'HexStringSplit'}

# Method 3: UniformSplit (binary key uniform split)
create 'events', 'data', {NUMREGIONS => 32, SPLITALGO => 'UniformSplit'}

# Method 4: Custom split points file
# List split points in a file, one per line
create 'events', 'data', SPLITS_FILE => '/path/to/splits.txt'

Determining the initial Region count:

Recommended formula:
Initial Region count = Number of RegionServers × Target Regions per server (10-30)

Example:
- 10 RegionServer cluster
- Target 20 Regions per server
- Initial Region count = 10 × 20 = 200

Verify that the RowKey distribution actually results in even partitioning.

5.2 Key Distribution (Bucketing)

Prepends a fixed number of bucket numbers as prefixes to RowKeys, distributing writes across multiple Regions.

/**
 * Bucket-based RowKey distributor.
 * Distributes writes across N buckets while allowing reads
 * to directly access the correct Region by computing the bucket number.
 */
public class BucketedKeyStrategy {

    private final int numBuckets;

    public BucketedKeyStrategy(int numBuckets) {
        this.numBuckets = numBuckets;
    }

    /**
     * Bucket number is determined by the RowKey's hash,
     * so the same original key always maps to the same bucket.
     */
    public byte[] createKey(String entityId, long timestamp) {
        int bucket = Math.abs(entityId.hashCode() % numBuckets);
        String key = String.format("%04d_%s_%d", bucket, entityId, timestamp);
        return Bytes.toBytes(key);
    }

    /**
     * To query all data for a specific entity:
     * compute the bucket number for an efficient single Scan.
     */
    public Scan createEntityScan(String entityId) {
        int bucket = Math.abs(entityId.hashCode() % numBuckets);
        String prefix = String.format("%04d_%s_", bucket, entityId);

        Scan scan = new Scan();
        scan.withStartRow(Bytes.toBytes(prefix));
        scan.withStopRow(Bytes.toBytes(prefix + "~"));  // ~ is high in ASCII
        return scan;
    }

    /**
     * For scanning all data:
     * execute parallel Scans across all buckets.
     */
    public List<Scan> createFullScans() {
        List<Scan> scans = new ArrayList<>();
        for (int i = 0; i < numBuckets; i++) {
            String startPrefix = String.format("%04d_", i);
            String endPrefix = String.format("%04d_~", i);
            Scan scan = new Scan();
            scan.withStartRow(Bytes.toBytes(startPrefix));
            scan.withStopRow(Bytes.toBytes(endPrefix));
            scans.add(scan);
        }
        return scans;
    }
}

5.3 TTL-Based Data Partitioning

Automatically expiring old time-series data maintains consistent Region sizes and reduces Compaction load.

# Set TTL on Column Family (in seconds)
# Auto-delete after 90 days
alter 'sensor_data', {NAME => 'raw', TTL => 7776000}

# Keep aggregated data for 1 year
alter 'sensor_data', {NAME => 'agg', TTL => 31536000}

<!-- hbase-site.xml: Configuration linked with table-level TTL policies -->
<configuration>
    <!-- Clean up expired data during Major Compaction after MemStore flush -->
    <property>
        <name>hbase.hstore.compaction.min</name>
        <value>3</value>
    </property>

    <!-- Shorten Major Compaction interval for TTL-enabled tables -->
    <!-- to clean up expired data quickly (default 7 days → 1 day) -->
    <property>
        <name>hbase.hregion.majorcompaction</name>
        <value>86400000</value>
    </property>
</configuration>

5.4 Time-Based Table Partitioning

Storing data in separate tables per time period allows dropping entire old tables for fast cleanup without Compaction.

/**
 * Daily/monthly table partitioning strategy.
 * Includes dates in table names for lifecycle management.
 */
public class TimePartitionedTableStrategy {

    private final Connection connection;
    private final Admin admin;

    public TimePartitionedTableStrategy(Connection connection) throws IOException {
        this.connection = connection;
        this.admin = connection.getAdmin();
    }

    /**
     * Auto-create monthly tables.
     * Example: logs_202603, logs_202604
     */
    public Table getOrCreateMonthlyTable(String baseName, LocalDate date) throws IOException {
        String tableName = baseName + "_" + date.format(DateTimeFormatter.ofPattern("yyyyMM"));
        TableName tn = TableName.valueOf(tableName);

        if (!admin.tableExists(tn)) {
            TableDescriptorBuilder builder = TableDescriptorBuilder.newBuilder(tn);
            ColumnFamilyDescriptor cf = ColumnFamilyDescriptorBuilder
                .newBuilder(Bytes.toBytes("d"))
                .setCompressingAlgo(Algorithm.SNAPPY)
                .setBloomFilterType(BloomType.ROW)
                .setMaxVersions(1)
                .build();
            builder.setColumnFamily(cf);

            // Pre-split into 16 Regions
            byte[][] splits = RegionSplitter.HexStringSplit
                .split(16);
            admin.createTable(builder.build(), splits);
        }

        return connection.getTable(tn);
    }

    /**
     * Delete tables older than 90 days.
     * DROP TABLE completes instantly without Compaction, more efficient than TTL.
     */
    public void purgeOldTables(String baseName, int retentionDays) throws IOException {
        LocalDate cutoff = LocalDate.now().minusDays(retentionDays);
        String cutoffStr = cutoff.format(DateTimeFormatter.ofPattern("yyyyMM"));

        for (TableDescriptor td : admin.listTableDescriptors()) {
            String name = td.getTableName().getNameAsString();
            if (name.startsWith(baseName + "_")) {
                String datePart = name.substring(baseName.length() + 1);
                if (datePart.compareTo(cutoffStr) < 0) {
                    admin.disableTable(td.getTableName());
                    admin.deleteTable(td.getTableName());
                    System.out.println("Purged table: " + name);
                }
            }
        }
    }
}

6. Schema Design Patterns in Practice

6.1 Tall-Narrow vs Flat-Wide

This is the most important structural choice in HBase table design.

Tall-Narrow

Many rows with few columns each. Each event or measurement is stored as a separate row.

RowKey                              | d:value | d:type
────────────────────────────────────┼─────────┼────────
sensor001_9223370449055775807       | 72.5    | cpu
sensor001_9223370449055775808       | 71.3    | cpu
sensor001_9223370449055775809       | 73.1    | cpu

Flat-Wide

Few rows with a very large number of columns. All time-series data for one entity is encoded in Column Qualifiers.

RowKey       | d:20260308120000 | d:20260308120100 | d:20260308120200 | ...
─────────────┼──────────────────┼──────────────────┼──────────────────┤
sensor001    | 72.5             | 71.3             | 73.1             | (thousands of columns)

Comparison and selection criteria:

Criterion	Tall-Narrow	Flat-Wide
Atomic row operations	Atomic per row only	Entire time series is atomic
Scan efficiency	Easy time-range Scan	Full retrieval with a single Get
Row size limits	No limit	Large column counts cause Region Split issues
Delete efficiency	Row-level deletion	Column-level deletion required
Recommended use	Most time-series	Small time-series, single-Get patterns

Tall-Narrow is generally recommended. HBase Scan performance scales with data size rather than row count, and Flat-Wide makes Region Split point determination difficult when individual rows grow too large.

6.2 Reverse Index (Secondary Index) Pattern

HBase only provides a built-in index on the RowKey. To search by other columns, you must maintain separate index tables.

/**
 * Multi-dimensional search support using reverse index tables.
 *
 * Main table: users (RowKey = user_id)
 * Index table: users_by_email (RowKey = email, value = user_id)
 * Index table: users_by_region (RowKey = region_user_id, value = "")
 */
public class SecondaryIndexManager {

    private final Table mainTable;
    private final Table emailIndex;
    private final Table regionIndex;

    /**
     * Update both main and index tables on data insertion.
     */
    public void putWithIndex(String userId, String email, String region,
                             Map<String, String> attributes) throws IOException {
        // 1. Insert data into main table
        Put mainPut = new Put(Bytes.toBytes(userId));
        mainPut.addColumn(Bytes.toBytes("info"), Bytes.toBytes("email"),
                          Bytes.toBytes(email));
        mainPut.addColumn(Bytes.toBytes("info"), Bytes.toBytes("region"),
                          Bytes.toBytes(region));
        for (Map.Entry<String, String> attr : attributes.entrySet()) {
            mainPut.addColumn(Bytes.toBytes("info"),
                              Bytes.toBytes(attr.getKey()),
                              Bytes.toBytes(attr.getValue()));
        }
        mainTable.put(mainPut);

        // 2. Update email index
        Put emailPut = new Put(Bytes.toBytes(email));
        emailPut.addColumn(Bytes.toBytes("idx"), Bytes.toBytes("uid"),
                           Bytes.toBytes(userId));
        emailIndex.put(emailPut);

        // 3. Update region index (composite key: region_userId)
        Put regionPut = new Put(Bytes.toBytes(region + "_" + userId));
        regionPut.addColumn(Bytes.toBytes("idx"), Bytes.toBytes(""),
                            Bytes.toBytes(""));
        regionIndex.put(regionPut);
    }

    /**
     * Lookup user by email: 2-hop lookup via index → main table.
     */
    public Result getUserByEmail(String email) throws IOException {
        Get indexGet = new Get(Bytes.toBytes(email));
        Result indexResult = emailIndex.get(indexGet);
        byte[] userId = indexResult.getValue(Bytes.toBytes("idx"),
                                             Bytes.toBytes("uid"));
        if (userId == null) return null;

        Get mainGet = new Get(userId);
        return mainTable.get(mainGet);
    }

    /**
     * Query all users in a specific region: Prefix Scan on index table.
     */
    public List<String> getUsersByRegion(String region) throws IOException {
        Scan scan = new Scan();
        scan.withStartRow(Bytes.toBytes(region + "_"));
        scan.withStopRow(Bytes.toBytes(region + "_~"));

        List<String> userIds = new ArrayList<>();
        try (ResultScanner scanner = regionIndex.getScanner(scan)) {
            for (Result r : scanner) {
                String rowKey = Bytes.toString(r.getRow());
                String userId = rowKey.substring(region.length() + 1);
                userIds.add(userId);
            }
        }
        return userIds;
    }
}

Operational considerations for reverse indexes:

Consistency between main and index tables must be guaranteed by the application. HBase does not support cross-table transactions.
Index entries must be cleaned up when data is deleted or updated. Otherwise, orphan indexes accumulate.
Phoenix or HBase Coprocessors can automate index management.

6.3 Secondary Index with Phoenix

Apache Phoenix provides a SQL layer on top of HBase and automatically manages Secondary Indexes.

-- Table creation and index usage with Phoenix

-- Create table
CREATE TABLE IF NOT EXISTS users (
    user_id VARCHAR NOT NULL PRIMARY KEY,
    email VARCHAR,
    region VARCHAR,
    created_at TIMESTAMP,
    login_count BIGINT
) SALT_BUCKETS=16, COMPRESSION='SNAPPY';

-- Covered Index: includes additional columns to avoid main table lookups
CREATE INDEX idx_users_email ON users (email)
    INCLUDE (region, login_count);

-- Index is automatically used when querying by email
SELECT user_id, email, region, login_count
FROM users
WHERE email = 'user@example.com';

-- Query users by region
CREATE INDEX idx_users_region ON users (region, created_at DESC)
    INCLUDE (email);

SELECT user_id, email, created_at
FROM users
WHERE region = 'ap-northeast-2'
ORDER BY created_at DESC
LIMIT 100;

7. Performance Stabilization Operations

7.1 Compaction Strategy

Compaction is the background operation with the greatest impact on HBase performance. Poor management leads to write stalls, read performance degradation, and I/O storms.

Minor Compaction: Merges small HFiles. Retains delete markers (tombstones). Runs automatically and frequently.

Major Compaction: Merges all HFiles into one. Removes delete markers and expired data. Generates heavy I/O and must be managed carefully.

<!-- hbase-site.xml: Production Compaction settings -->
<configuration>
    <!-- Disable automatic Major Compaction -->
    <property>
        <name>hbase.hregion.majorcompaction</name>
        <value>0</value>
        <description>Set to 0 to disable automatic Major Compaction.
        Run manually via cron during off-peak hours.</description>
    </property>

    <!-- Minor Compaction trigger: runs when 3+ HFiles exist -->
    <property>
        <name>hbase.hstore.compactionThreshold</name>
        <value>3</value>
    </property>

    <!-- Maximum HFiles included in Minor Compaction -->
    <property>
        <name>hbase.hstore.compaction.max</name>
        <value>10</value>
    </property>

    <!-- Minimum HFile size for Compaction -->
    <property>
        <name>hbase.hstore.compaction.min.size</name>
        <value>134217728</value> <!-- 128MB -->
    </property>

    <!-- Compaction throttle: limit I/O to minimize service impact -->
    <property>
        <name>hbase.regionserver.throughput.controller</name>
        <value>org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController</value>
    </property>
    <property>
        <name>hbase.hstore.compaction.throughput.lower.bound</name>
        <value>52428800</value> <!-- 50MB/s lower bound -->
    </property>
    <property>
        <name>hbase.hstore.compaction.throughput.higher.bound</name>
        <value>104857600</value> <!-- 100MB/s upper bound -->
    </property>
</configuration>

Running Major Compaction during off-peak hours (cron):

#!/bin/bash
# major_compaction_scheduler.sh
# crontab: 0 3 * * 0 /opt/hbase/scripts/major_compaction_scheduler.sh

TABLES=("metrics_table" "events_table" "logs_table")
LOG_FILE="/var/log/hbase/major_compaction_$(date +%Y%m%d).log"

echo "=== Major Compaction Start: $(date) ===" >> "$LOG_FILE"

for table in "${TABLES[@]}"; do
    echo "Compacting $table..." >> "$LOG_FILE"
    echo "major_compact '$table'" | hbase shell >> "$LOG_FILE" 2>&1

    # 1-hour interval between tables to distribute I/O load
    sleep 3600
done

echo "=== Major Compaction End: $(date) ===" >> "$LOG_FILE"

7.2 Region Split and Merge Management

Automatic Split policy tuning:

<configuration>
    <!-- Maximum Region size: auto-Split when this size is reached -->
    <property>
        <name>hbase.hregion.max.filesize</name>
        <value>10737418240</value> <!-- 10GB -->
    </property>

    <!-- Split policy selection -->
    <property>
        <name>hbase.regionserver.region.split.policy</name>
        <value>org.apache.hadoop.hbase.regionserver.SteppingSplitPolicy</value>
        <description>
        SteppingSplitPolicy: splits quickly when Region count is low,
        waits until max.filesize when Region count is sufficient.
        More stable than IncreasingToUpperBoundRegionSplitPolicy.
        </description>
    </property>
</configuration>

Manual Split and Merge:

# Manually split a hotspot Region
# First check the hotspot Region's encoded name and appropriate split key
hbase shell <<'EOF'
list_regions 'metrics_table'
EOF

# Split Region at a specific key
hbase shell <<'EOF'
split 'metrics_table', '08_sensor500'
EOF

# Region Merge: merge over-split small Regions
# In HBase 2.x, use the merge_region command in hbase shell
hbase shell <<'EOF'
merge_region 'ENCODED_REGION_NAME_1', 'ENCODED_REGION_NAME_2', true
EOF

7.3 BlockCache and BucketCache Configuration

Cache hit ratio is the key to read performance. BlockCache caches HFile data blocks in memory.

<configuration>
    <!-- On-heap BlockCache ratio (40% of heap) -->
    <property>
        <name>hfile.block.cache.size</name>
        <value>0.4</value>
    </property>

    <!-- Enable BucketCache: extend cache with off-heap memory -->
    <property>
        <name>hbase.bucketcache.ioengine</name>
        <value>offheap</value>
        <description>Choose from offheap, file:/path/to/cache, mmap:/path/to/cache</description>
    </property>

    <!-- BucketCache size (MB) -->
    <property>
        <name>hbase.bucketcache.size</name>
        <value>8192</value> <!-- 8GB Off-heap -->
    </property>

    <!-- CombinedBlockCache: On-heap(index/meta) + Off-heap(data) -->
    <property>
        <name>hbase.bucketcache.combinedcache.enabled</name>
        <value>true</value>
    </property>
</configuration>

# RegionServer JVM settings (hbase-env.sh)
# On-heap 32GB + Off-heap(BucketCache) 8GB
export HBASE_REGIONSERVER_OPTS="
  -Xmx32g -Xms32g
  -XX:MaxDirectMemorySize=10g
  -XX:+UseG1GC
  -XX:MaxGCPauseMillis=100
  -XX:G1HeapRegionSize=16m
  -XX:InitiatingHeapOccupancyPercent=65
"

Monitoring cache hit ratio:

# Check BlockCache hit ratio (JMX)
curl -s "http://regionserver:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server" \
  | python3 -c "
import json, sys
data = json.load(sys.stdin)
for bean in data['beans']:
    hit = bean.get('blockCacheHitCount', 0)
    miss = bean.get('blockCacheMissCount', 0)
    total = hit + miss
    ratio = (hit / total * 100) if total > 0 else 0
    print(f'BlockCache Hit Ratio: {ratio:.1f}%')
    print(f'  Hits: {hit:,}')
    print(f'  Misses: {miss:,}')
    print(f'  Evictions: {bean.get(\"blockCacheEvictionCount\", 0):,}')
"
# Target: maintain 95%+ hit ratio

7.4 MemStore Tuning

MemStore is a write buffer that flushes to HFile when full. MemStore size and flush frequency directly affect write performance.

<configuration>
    <!-- Individual MemStore flush size -->
    <property>
        <name>hbase.hregion.memstore.flush.size</name>
        <value>134217728</value> <!-- 128MB -->
    </property>

    <!-- RegionServer total MemStore upper limit (40% of heap) -->
    <property>
        <name>hbase.regionserver.global.memstore.size</name>
        <value>0.4</value>
    </property>

    <!-- Force flush starts when total MemStore exceeds this ratio -->
    <property>
        <name>hbase.regionserver.global.memstore.size.lower.limit</name>
        <value>0.95</value>
    </property>

    <!-- Ensure MemStore + BlockCache does not exceed 80% of heap -->
    <!-- global.memstore.size(0.4) + block.cache.size(0.4) = 0.8 -->
</configuration>

8. Practical Checklist and Anti-Patterns

Design Phase Checklist

RowKey design verified: Is the RowKey designed based on the most frequent read/write patterns?
Hotspot simulation: Have you generated RowKeys with expected data and simulated Region distribution?
Pre-split plan: Is the initial Region count determined based on cluster size?
Column Family minimized: Is the CF count 3 or fewer? Do CFs have clearly different access patterns?
TTL/VERSIONS configured: Are TTL and version counts set to prevent unlimited data accumulation?
Bloom Filter configured: ROW for Get-heavy, ROWCOL for Get+Column-heavy workloads?
Compression enabled: Is SNAPPY or LZ4 compression activated?
Secondary index strategy: If non-RowKey column searches are needed, is an index strategy established?

Operations Phase Checklist

Major Compaction schedule: Is auto-execution disabled with manual runs during off-peak hours?
Region distribution monitoring: Are per-Region request counts periodically checked for hotspot detection?
BlockCache hit ratio: Is 95%+ maintained?
GC pause monitoring: Are STW (Stop-the-World) GC pauses of 5+ seconds absent?
MemStore flush frequency: Are abnormally frequent flushes avoided?
Compaction queue: Is the queue size not persistently above 10?
HDFS disk utilization: Is it maintained below 80%?

Anti-Pattern Collection

Anti-Pattern 1: Timestamp at the beginning of RowKey

# Bad example
RowKey: 20260308120000_event_click
# → All latest writes concentrate on the last Region

# Correct alternative
RowKey: 0a_click_20260308120000  (salt + event_type + timestamp)

Anti-Pattern 2: Excessive Column Family count

# Bad example: 10 CFs
create 'user_profile', 'basic', 'contact', 'preference', 'history',
       'security', 'billing', 'social', 'activity', 'settings', 'cache'
# → Cascading flushes across CFs cause I/O storms, MemStore memory waste

# Correct alternative: consolidate to 2-3 CFs
create 'user_profile', \
  {NAME => 'i', VERSIONS => 1},   \  # info: basic + contact + settings
  {NAME => 'a', VERSIONS => 1, TTL => 7776000}  # activity: activity logs (90 days)

Anti-Pattern 3: Variable-length RowKey sorting issues

# Bad example: storing numbers as strings (beware lexicographic sorting)
"1", "10", "100", "2", "20", "3"  ← Lexicographic order: 1 < 10 < 100 < 2

# Correct alternative: fixed-length padding
"001", "002", "003", "010", "020", "100"  ← Correct order

Anti-Pattern 4: Sensitive information in RowKey

# Bad example: using email directly as RowKey
RowKey: user@example.com_20260308
# → RowKey is exposed in plaintext in WAL, HFile, and meta table

# Correct alternative: hash it
RowKey: sha256(user@example.com)_20260308

Anti-Pattern 5: Single Row too large

# Bad example: tens of thousands of columns in one Row (extreme Flat-Wide)
# → Cannot split the Region at this Row during Region Split
# → Single Row may exceed MemStore flush size

# Correct alternative: switch to Tall-Narrow or split Row by time
RowKey: entity001_20260308_00  (split Row by time units)
RowKey: entity001_20260308_01

Anti-Pattern 6: Scan without range specification

// Bad example: full table Scan
Scan scan = new Scan();
// → Traverses billions of rows, overloading RegionServer

// Correct alternative: explicitly restrict the range
Scan scan = new Scan();
scan.withStartRow(Bytes.toBytes("sensor001_"));
scan.withStopRow(Bytes.toBytes("sensor001_~"));
scan.setCaching(500);      // Rows returned per RPC
scan.setMaxResultSize(5 * 1024 * 1024);  // 5MB limit
scan.addColumn(Bytes.toBytes("d"), Bytes.toBytes("value"));  // Only needed columns

9. Conclusion

HBase data modeling is, without exaggeration, all about RowKey design. Here is a summary of the key takeaways:

RowKey determines load distribution: Always avoid Sequential Keys; use Salting/Hashing/Bucketing for even distribution.
Read patterns determine RowKey: Design RowKeys so the most frequent queries can be efficiently handled via Prefix Scan or Point Get.
Default to Tall-Narrow: More stable than Flat-Wide for most workloads.
Prevention is the best hotspot strategy: Block hotspots before they occur with Pre-split, Key distribution, and monitoring.
Control Compaction: Disable automatic Major Compaction, run manually during off-peak hours, and apply I/O throttling.
Defend cache hit ratio: Properly configure BlockCache + BucketCache and target 95%+ hit ratio.
Fewer Column Families, shorter RowKeys: Maintain brevity for both storage efficiency and performance.

Proper data modeling delivers performance improvements greater than doubling your cluster size. Invest sufficient time in designing a single RowKey.

Quiz

Q1: What is the main topic covered in "HBase Data Modeling Playbook: Hotspot Avoidance and Performance Stabilization"?

A practical playbook for HBase RowKey design patterns, hotspot detection and avoidance, Region distribution optimization, and performance stabilization in large-scale deployments.

Q2: What is HBase Data Model Fundamentals?

Core Components HBase's data model is fundamentally different from RDBMS. The following five elements constitute a single Cell. Physical Storage Structure HBase data appears as tables logically but is physically stored separately per Column Family.

Q3: Describe the RowKey Design Patterns.

RowKey design determines 80% of HBase performance. You must consider read/write patterns, data distribution, and scan ranges. 3.1 Salting (Prefix Distribution) Salting prepends a hash-based prefix (salt) to the RowKey to distribute data evenly across multiple Regions.

Q4: What are the key aspects of Hotspot Causes and Detection?

What Is a Hotspot A hotspot occurs when read or write requests are abnormally concentrated on a specific Region. While HBase is designed for horizontal scaling, a hotspot turns a single RegionServer's processing limit into the bottleneck for the entire cluster.

Q5: How does Hotspot Avoidance Strategies work?

5.1 Pre-splitting Pre-splitting Regions at table creation time prevents load concentration on a single Region during initial data loading.

HBase Data Modeling Playbook: Hotspot Avoidance and Performance Stabilization

1. Introduction

2. HBase Data Model Fundamentals

Core Components

Physical Storage Structure

RowKey Sorting and Region Distribution

Column Family Design Principles

3. RowKey Design Patterns

3.1 Salting (Prefix Distribution)

3.2 Hashing (Hash Prefix)

3.3 Key Reversing

3.4 Composite Key

3.5 RowKey Design for Time-Series Data

RowKey Design Decision Guide

4. Hotspot Causes and Detection

What Is a Hotspot

Primary Causes of Hotspots

Hotspot Detection Methods

5. Hotspot Avoidance Strategies

5.1 Pre-splitting

5.2 Key Distribution (Bucketing)

5.3 TTL-Based Data Partitioning

5.4 Time-Based Table Partitioning

6. Schema Design Patterns in Practice

6.1 Tall-Narrow vs Flat-Wide

6.2 Reverse Index (Secondary Index) Pattern

6.3 Secondary Index with Phoenix

7. Performance Stabilization Operations

7.1 Compaction Strategy

7.2 Region Split and Merge Management

7.3 BlockCache and BucketCache Configuration

7.4 MemStore Tuning

8. Practical Checklist and Anti-Patterns

Design Phase Checklist

Operations Phase Checklist

Anti-Pattern Collection

9. Conclusion

Quiz

Comments