Data Engineering Vol2 AWS : Data Processing - Spark & Kafka
Learn Spark (PySpark) & Kafka on AWS for powerful batch & stream data processing. Master EMR & Databricks for scalable data pipelines.
Course Content
Data Engineering Vol2 AWS : Data Processing - Spark & Kafka is an all-levels Udemy course taught by Soumyadeep Dey. It runs 60h 41m, is taught in English, and includes a certificate of completion. Learners rate it 5.0 out of 5 from 5 reviews, with 16 students enrolled. It is a paid course, last updated October 2025.
Dive deeper into Data Engineering with Volume 2, mastering essential open-source data processing technologies! This course focuses on Spark and Kafka, the industry-leading frameworks for both batch processing and stream processing.
You'll embark on a comprehensive learning journey with Spark, progressing from beginner (Level 100) to advanced (Level 400) through practical, hands-on exercises and real-world projects. Gain expertise in:
- Spark Fundamentals: Build a strong foundation in Spark concepts.
- Spark Batch Processing: Learn to efficiently process large datasets in batch mode.
- Spark Stream Processing: Master real-time data processing with Spark Streaming.
- Spark Performance Tuning: Optimize your Spark applications for maximum performance.
- Streaming Ingestion: Implement robust data ingestion pipelines for continuous data streams.
- Window Functions: Analyze data over time windows for valuable insights.
- ACID Transactions on Iceberg: Ensure data consistency and reliability with ACID transactions.
Explore modern Data Lake architectures on AWS, including:
- Data Lake on AWS (S3): Understand and utilize Amazon S3 for scalable data storage.
- Data Lakehouse with Apache Iceberg: Leverage Apache Iceberg to build a modern data lakehouse.
This course utilizes AWS as the primary hosting platform and covers key AWS services:
- EMR (Elastic MapReduce): Manage and run Spark clusters on AWS.
- S3 (Simple Storage Service): Store and access your data securely.
- MSK (Managed Streaming for Kafka): Set up and manage Kafka clusters.
- Databricks: Explore Databricks as a managed Spark hosting platform.
Furthermore, you'll learn how to integrate Spark with other critical services:
- AWS RDS (MySQL or PostgreSQL): Connect Spark to relational databases.
- Redshift: Utilize Redshift for data warehousing and analytics.
Hands-on Experience:
Get ready to work with large datasets (100 GB - 300 GB+), gaining practical experience through:
- 10 Projects with diverse datasets.
- 250 GB+ total dataset size for realistic scenarios.
Additional Technologies Covered:
- EC2 (Elastic Compute Cloud)
- EBS (Elastic Block Storage)
- VPC (Virtual Private Cloud)
- IAM (Identity and Access Management)
Bonus Resources:
- Optional Python videos to enhance your data processing skills.
- Optional AWS and SQL Essentials videos for foundational knowledge.
This course is Volume 2 of a comprehensive Data Engineering curriculum. Volume 3 will cover:
- Flink
- Apache Airflow
- Apache Pinot
- AWS Kinesis
Your feedback is welcome! Suggest any additional topics you'd like to see covered in future volumes.
Want AI in your product? Hire the team behind TutorialSearch.io
LLM features, RAG, agents and automation wired into a real product — built by the devs who shipped this site.