Building an AWS Pipeline Ch 01

ETL

Metadata System

Defined as 'information that describes various aspects of a specific information asset to improve its usefulness throughout the information lifecycle.'

Emergence of the Data Lake

The emergence of Google File System. Open-source tools like Spark, Hive, and ODS appeared. Pipelines for structured data from databases plus semi-structured data were created. Presto, which allows using data from both DBMS and Hadoop together, also emerged.

Prediction or classification problems.

Hadoop System

Between 2016 and 2020, a tremendous number of open-source projects emerged.

Spark is leading the industry. Parallel processing. It provides extremely fast execution by caching data in memory. It supports various programming languages such as Java, Scala, Python, and R. It supports various workloads and provides fault tolerance for data nodes.

Data Storage Architecture

GCP has BigQuery, and more recently there is Snowflake. The trend of processing structured, semi-structured, and unstructured data from operational systems and storing it in a Data Lake is growing. A storage system that stores data well is needed.

End-to-end environments that make it easy for analysts to perform analysis are provided. Data Lakehouses use cloud-based analytical environments. We will explain which elements to focus on.

TCO (Total Cost of Ownership): Total Cost of Ownership A total cost of ownership analysis (acquisition and operating costs) comparing infrastructure environments running on-premises versus on AWS.

Proper capacity planning
Elastic architecture
Purchase options
Storage classes (S3 has several tiers)

Costs must be continuously monitored.

When building data pipelines, the most emphasized principle is decoupling. Decoupling Compute and Storage architecture is essential. Data from various sources at multiple locations can be built into a flexible, storable platform, organized into logically or virtually optimized environments as an easy and effective distributed platform.

Using idle EC2 resources through auctions per Availability Zone. Big data analytics, batch jobs, stateless web services... massively parallel computing.

Spot Fleet

Advantages of using S3 (Object Storage) as HDFS: Compute nodes and data nodes can be managed separately. Even after shutting down a cluster and recreating it, existing data can still be read. No need to worry about HDFS scaling.

Nodes can be added or removed even during analysis.

Data Lakes are inexpensive. Data Warehouses are proprietary.

Data Pipeline Architecture

Rapid response to data requirements within the company
Must be continuous and error-free
Must be flexible and scalable for system-level issues
Scale up and scale out must be freely available
Must be able to handle event-driven data loads: marketing events, push notifications, service launches
Easy-to-analyze data formats

Lambda Architecture

API Gateway: connects external and internal data
Kinesis: queue
Kinesis: method for consuming data from the queue
Pinpoint: marketing information delivery
DMS: migrates data in bulk from operational systems
Spark Streaming, Kinesis Analytics (Flink)
Elasticsearch, DynamoDB, Redshift

Quiz

Q1: What is the main topic covered in "Building an AWS Pipeline Ch 01"?

Building an AWS Pipeline Ch01

Q2: What is ETL?

Metadata System Defined as 'information that describes various aspects of a specific information asset to improve its usefulness throughout the information lifecycle.' Emergence of the Data Lake The emergence of Google File System.

Q3: Describe the Data Pipeline Architecture.

Rapid response to data requirements within the company Must be continuous and error-free Must be flexible and scalable for system-level issues Scale up and scale out must be freely available Must be able to handle event-driven data loads: marketing events, push notifications, ser...

Q4: Describe the Lambda Architecture.

API Gateway: connects external and internal data Kinesis: queue Kinesis: method for consuming data from the queue Pinpoint: marketing information delivery DMS: migrates data in bulk from operational systems Spark Streaming, Kinesis Analytics (Flink) Elasticsearch, DynamoDB, Redsh...