What is the Hadoop Ecosystem?

Hadoop is an on-premise cluster framework built on Google's original MapReduce concept (2004 paper). Its philosophy: everything is a file, and many apps share their internal state.

Hadoop Ecosystem Components

• YARN – Cluster resource management
• MapReduce / Spark – Data processing engines
• Pig – Scripting platform
• Flink – Stream processing
• HBase – Columnar NoSQL database
• Ambari – Cluster management and monitoring
• Presto – Distributed SQL engine
• Oozie – Workflow scheduler
• Zookeeper – Coordination service
• Avro – Data serialization (similar to JSON)
• HCatalog – Metadata management
• Mahout / Spark ML – Machine learning libraries
• Flume – Log aggregation and transport
• Sqoop – Import/export of relational data

HDFS (Hadoop Distributed File System)

HDFS is the storage backbone of Hadoop and acts as the data lake. It is not POSIX-compliant and uses large block sizes (default: 64MB).

Hadoop workloads can be lifted to the cloud by persisting data in GCS (Google Cloud Storage). This allows querying from BigQuery using federated queries (EXTERNAL_QUERY).

HDFS Limitations in Cloud:

• Large block size (e.g., 64MB)
• Data locality requirements
• Built-in replication

hdfs:// can be replaced with gs:// in cloud-native environments.

Note: Cloud Storage doesn't support directories in the same way HDFS does (no real directory hierarchy).

Common Commands:

$ hadoop fs -lsr /
# hadoop-fuse-dfs dfs://hadoop-hdfs /mnt
$ ls /mnt/
# mount -t nfs -o vers=3,proto=tcp,nolock host:/ /mnt
$ ls /mnt/
    

MapReduce

MapReduce is a batch-oriented processing framework designed for large-scale data. It brings computation to the data and follows a highly parallelizable model.

• Batch processing (long-running jobs)
• Moves computation to data
• Simple but rigid programming model
• Based on (Key, Value) record pairs

Logical Phases of MapReduce

1. Map

Processes data by filtering, projecting, and transforming.

• Input: (Key1, Value1)
• Output: List(Key2, Value2)

2. Shuffle

Groups values by keys across partitions, typically sorted.

• Input: List(Key2, Value2)
• Output: Sort(Partition(List(Key2, List(Value2))))
• Custom shuffling options supported in Hadoop

3. Reduce

Aggregates all values for a single key.

• Input: List(Key2, List(Value2))
• Output: List(Key3, Value3)
• Typical use: aggregations

All keys and values must be serializable for network transmission.