Conceptualizing the Processing Model for Apache Spark Structured Streaming
Learn Apache Spark 2 for real-time data analytics. Process streaming data from sensors & more with micro-batch & continuous processing.
Course Content
Conceptualizing the Processing Model for Apache Spark Structured Streaming is an intermediate-level Pluralsight course taught by Janani Ravi. It runs 2h 57m, is taught in English, and includes a certificate of completion. Learners rate it 4.8 out of 5 from 22 reviews. It is a paid course, first published September 2020.
Learn Apache Spark Structured Streaming in Spark 2 and unlock the power of real-time data processing. This course provides a comprehensive understanding of Spark's unified approach to batch and stream processing.
Key Concepts Covered:
- Structured Streaming Processing Model: Discover how Spark treats streaming data as a natural extension of batch processing, simplifying development and management. Understand the benefits of this unified model.
- Spark Streaming vs. Structured Streaming: Gain clarity on the key differences between Spark Streaming and the newer, more efficient Structured Streaming in Spark 2.
- Spark Architecture: Explore the core components of Spark – drivers, workers, executors, and tasks – and how they interact to process data.
- Streaming DataFrames: Work with streaming DataFrames using the DataFrame API and Spark SQL to query streaming sources.
- Data Sources & Sinks: Learn to execute queries on streaming data from sources like sockets and file systems, and write results to various data sinks.
- Output Modes: Master the use of append, complete, and update modes for writing data to sinks, ensuring data reliability.
- Scheduling & Checkpointing: Understand how Spark schedules streaming jobs and utilizes checkpointing for fault tolerance and recovery.
- Micro-batch vs. Continuous Processing: Compare and contrast the micro-batch and experimental continuous processing modes in Spark Structured Streaming.
- Tungsten Engine Optimizations: Explore the performance enhancements introduced by the Tungsten engine in Spark 2, leading to faster processing speeds.
- Catalyst Optimizer: Delve into the stages of optimization within the Catalyst optimizer, which enhances the performance of SQL queries.
By the end of this course, you will be able to:
- Build and execute streaming queries on various data inputs.
- Write processed data to reliable storage using different output modes.
- Implement checkpointing to ensure fault tolerance and data recovery in your streaming applications.
- Effectively leverage the power of Apache Spark Structured Streaming for real-time data analytics.
Want AI in your product? Hire the team behind TutorialSearch.io
LLM features, RAG, agents and automation wired into a real product — built by the devs who shipped this site.