Udemyallen

Data Engineering Vol2 AWS : Data Processing - Spark & Kafka

Learn Spark (PySpark) & Kafka on AWS for powerful batch & stream data processing. Master EMR & Databricks for scalable data pipelines.

Related:Spark SQL
5.0(5)
16students
60h 41m
Certificate
Soumyadeep Dey
Soumyadeep DeySr. Solutions Architect - Data Engineering & ML

Course Content

Data Engineering Vol2 AWS : Data Processing - Spark & Kafka is an all-levels Udemy course taught by Soumyadeep Dey. It runs 60h 41m, is taught in English, and includes a certificate of completion. Learners rate it 5.0 out of 5 from 5 reviews, with 16 students enrolled. It is a paid course, last updated October 2025.

Dive deeper into Data Engineering with Volume 2, mastering essential open-source data processing technologies! This course focuses on Spark and Kafka, the industry-leading frameworks for both batch processing and stream processing.

You'll embark on a comprehensive learning journey with Spark, progressing from beginner (Level 100) to advanced (Level 400) through practical, hands-on exercises and real-world projects. Gain expertise in:

  • Spark Fundamentals: Build a strong foundation in Spark concepts.
  • Spark Batch Processing: Learn to efficiently process large datasets in batch mode.
  • Spark Stream Processing: Master real-time data processing with Spark Streaming.
  • Spark Performance Tuning: Optimize your Spark applications for maximum performance.
  • Streaming Ingestion: Implement robust data ingestion pipelines for continuous data streams.
  • Window Functions: Analyze data over time windows for valuable insights.
  • ACID Transactions on Iceberg: Ensure data consistency and reliability with ACID transactions.

Explore modern Data Lake architectures on AWS, including:

  • Data Lake on AWS (S3): Understand and utilize Amazon S3 for scalable data storage.
  • Data Lakehouse with Apache Iceberg: Leverage Apache Iceberg to build a modern data lakehouse.

This course utilizes AWS as the primary hosting platform and covers key AWS services:

  • EMR (Elastic MapReduce): Manage and run Spark clusters on AWS.
  • S3 (Simple Storage Service): Store and access your data securely.
  • MSK (Managed Streaming for Kafka): Set up and manage Kafka clusters.
  • Databricks: Explore Databricks as a managed Spark hosting platform.

Furthermore, you'll learn how to integrate Spark with other critical services:

  • AWS RDS (MySQL or PostgreSQL): Connect Spark to relational databases.
  • Redshift: Utilize Redshift for data warehousing and analytics.

Hands-on Experience:

Get ready to work with large datasets (100 GB - 300 GB+), gaining practical experience through:

  • 10 Projects with diverse datasets.
  • 250 GB+ total dataset size for realistic scenarios.

Additional Technologies Covered:

  • EC2 (Elastic Compute Cloud)
  • EBS (Elastic Block Storage)
  • VPC (Virtual Private Cloud)
  • IAM (Identity and Access Management)

Bonus Resources:

  • Optional Python videos to enhance your data processing skills.
  • Optional AWS and SQL Essentials videos for foundational knowledge.

This course is Volume 2 of a comprehensive Data Engineering curriculum. Volume 3 will cover:

  • Flink
  • Apache Airflow
  • Apache Pinot
  • AWS Kinesis

Your feedback is welcome! Suggest any additional topics you'd like to see covered in future volumes.

$ whoami — the devs behind tutorialsearch.io

Want AI in your product? Hire the team behind TutorialSearch.io

LLM features, RAG, agents and automation wired into a real product — built by the devs who shipped this site.

5.0
53m
beginner
pluralsight
5.0
1h 29m
beginner
pluralsight
5.0(766)
2h 58m
intermediate
skillshare