Apache Airflow 2.0 : Complete Distributed Configuration
Learn to configure a highly available Airflow with multiple schedulers and Celery workers for robust workflow management.
Course Content
Apache Airflow 2.0 : Complete Distributed Configuration is an intermediate-level Udemy course taught by Ganesh Dhareshwar. It runs 2h 18m, is taught in English, and includes a certificate of completion. Learners rate it 4.2 out of 5 from 31 reviews, with 209 students enrolled. It is a paid course, last updated April 2021.
Unlock the Power of Apache Airflow with a Comprehensive Distributed Setup!
This course dives deep into Airflow, the leading open-source platform for programmatically authoring, scheduling, and monitoring workflows. Discover why Airflow is a popular choice for its simplicity, scalability, and extensibility – and learn how to leverage its capabilities to handle demanding workloads.
Mastering Distributed Airflow with Celery Executor
The primary focus is building a robust Airflow distributed setup using the Celery Executor. You'll gain hands-on experience running over 100 jobs or DAGs in parallel on a single instance, significantly boosting your workflow processing power. We'll explore different executor options, including:
- Sequential Executor: Understand the basics of task execution.
- Local Executor: Learn to run Airflow on a single machine.
- Celery Executor: Unlock parallel processing for high-performance workflows.
Hands-on with AWS & Executor Configuration
We'll acquire and configure EC2 instances from AWS to create a scalable and reliable Airflow environment. You'll learn the practical steps to set up and manage these executors effectively.
Embrace Airflow 2.0: The Next Generation
Stay ahead of the curve with Airflow 2.0! This course covers the latest advancements, including:
- HA Scheduler: Achieve high availability for your Airflow deployments.
- Massive Performance Improvements: Experience significant speed boosts in scheduler and Celery worker performance.
- New Module: Explore the exciting features introduced in Airflow 2.0.
Deployment & Configuration
You'll gain practical skills in:
- Installing key components: Webserver, Scheduler, Celery workers, and Flower.
- Configuring multiple schedulers for enhanced reliability and scalability.
- Implementing essential features: Login, Email alerting, and Logs management.
Outcome:
By the end of this course, you'll have a fully functional distributed Airflow setup ready to be shared across multiple teams in your organization. Gain the expertise to manage complex workflows, optimize performance, and build a scalable data pipeline with Apache Airflow. Learn from a seasoned professional who has successfully migrated from Airflow 1.x to Airflow 2.0 and is passionate about its capabilities.
Hire the developers behind TutorialSearch.io
We built this site — full-stack web apps, AI integration & automation. Tell us what you need and we'll build yours.