AWS Cloud Projects for Data & AI Engineers: 5 Projects

Master AWS Data Engineering: Build Production-Ready Data Lakehouse Projects

This 100% hands-on course empowers you to build portfolio-worthy AWS cloud projects that closely replicate the challenges and workflows of real-world data teams. Gain practical experience designing, deploying, and managing a complete Data Lakehouse on AWS, mirroring how modern organizations handle data.

You'll dive deep into essential AWS services, including:

• S3: For scalable and cost-effective data storage.
• Glue (PySpark): For data ETL (Extract, Transform, Load) processes.
• Athena: For interactive SQL queries on your data lake.
• Lake Formation: To enforce robust data governance and security.
• Glue Catalog: For metadata management and data discovery.
• Apache Iceberg: A modern table format enabling schema evolution and data versioning.
• Redshift Serverless: For fast and scalable data warehousing.
• IAM: For secure access management.
• Lambda: For serverless compute.
• DynamoDB: For NoSQL data storage.
• CloudWatch/CloudTrail: For monitoring and auditing.

This course focuses on cost optimization, reliability, and auditability – critical aspects of production data pipelines.

What You'll Build (5 Connected Projects):

1. Lakehouse on AWS: Establish a robust data lakehouse architecture using S3 and Apache Iceberg. You'll learn to land raw data in S3, transform it using Glue, publish data in Iceberg (bronze/silver layers), implement partitioning and schema evolution, and control data access with quality checks.

2. Data Governance with Lake Formation: Implement comprehensive data governance using Lake Formation. Learn to enforce tag-based policies (LF-Tags), apply column masking and row-level filters (Data Cell Filters), and demonstrate access control in Athena for different user roles (Analyst vs. Scientist). Add lightweight auditing capabilities.

3. Data Warehouse on Redshift Serverless (External + SCD2): Expose data stored in Iceberg via external tables in Redshift Serverless. Build a star schema with fact and dimension tables. Implement SCD2 (Slowly Changing Dimensions) using the MERGE statement for accurate historical data. Optimize performance and cost by configuring sort and distribution keys, and utilizing Workload Management (WLM) and workgroups.

4. A Day in the Life of a Data Engineer (Batch Ops Simulation): Simulate real-world batch operations. You'll orchestrate data ingestion, data quality checks, data publishing, and handle scenarios involving schema changes and late data. Learn to safely rerun failed jobs, perform data backfills, and document incidents with a clear postmortem.

5. AI/ML Readiness & Serving: Prepare your data for AI/ML workloads. Curate ML-friendly tables, ensure reproducible training sets using Iceberg snapshots and time travel, and (optionally) integrate SageMaker and Athena for streamlined model development and deployment workflows.