Udemyintermediateen

Apache Airflow 2.0 : Complete Distributed Configuration

Learn to configure a highly available Airflow with multiple schedulers and Celery workers for robust workflow management.

4.2(31)
209students
2h 18m
Certificate
Ganesh Dhareshwar
Ganesh DhareshwarSenior Data Engineer at Delivery Hero

Course Content

Apache Airflow 2.0 : Complete Distributed Configuration is an intermediate-level Udemy course taught by Ganesh Dhareshwar. It runs 2h 18m, is taught in English, and includes a certificate of completion. Learners rate it 4.2 out of 5 from 31 reviews, with 209 students enrolled. It is a paid course, last updated April 2021.

Unlock the Power of Apache Airflow with a Comprehensive Distributed Setup!

This course dives deep into Airflow, the leading open-source platform for programmatically authoring, scheduling, and monitoring workflows. Discover why Airflow is a popular choice for its simplicity, scalability, and extensibility – and learn how to leverage its capabilities to handle demanding workloads.

Mastering Distributed Airflow with Celery Executor

The primary focus is building a robust Airflow distributed setup using the Celery Executor. You'll gain hands-on experience running over 100 jobs or DAGs in parallel on a single instance, significantly boosting your workflow processing power. We'll explore different executor options, including:

  • Sequential Executor: Understand the basics of task execution.
  • Local Executor: Learn to run Airflow on a single machine.
  • Celery Executor: Unlock parallel processing for high-performance workflows.

Hands-on with AWS & Executor Configuration

We'll acquire and configure EC2 instances from AWS to create a scalable and reliable Airflow environment. You'll learn the practical steps to set up and manage these executors effectively.

Embrace Airflow 2.0: The Next Generation

Stay ahead of the curve with Airflow 2.0! This course covers the latest advancements, including:

  • HA Scheduler: Achieve high availability for your Airflow deployments.
  • Massive Performance Improvements: Experience significant speed boosts in scheduler and Celery worker performance.
  • New Module: Explore the exciting features introduced in Airflow 2.0.

Deployment & Configuration

You'll gain practical skills in:

  • Installing key components: Webserver, Scheduler, Celery workers, and Flower.
  • Configuring multiple schedulers for enhanced reliability and scalability.
  • Implementing essential features: Login, Email alerting, and Logs management.

Outcome:

By the end of this course, you'll have a fully functional distributed Airflow setup ready to be shared across multiple teams in your organization. Gain the expertise to manage complex workflows, optimize performance, and build a scalable data pipeline with Apache Airflow. Learn from a seasoned professional who has successfully migrated from Airflow 1.x to Airflow 2.0 and is passionate about its capabilities.

$ whoami — the devs behind tutorialsearch.io

Hire the developers behind TutorialSearch.io

We built this site — full-stack web apps, AI integration & automation. Tell us what you need and we'll build yours.