Udemyintermediateen

Learning PySpark

Learn to build & deploy scalable data applications with Python and Apache Spark. Master big data processing and analysis.

Related:Spark SQL
4.1(189)
651students
2h 28m
Certificate
Packt Publishing
Packt PublishingTech Knowledge in Motion

Course Content

Learning PySpark is an intermediate-level Udemy course taught by Packt Publishing. It runs 2h 28m, is taught in English, and includes a certificate of completion. Learners rate it 4.1 out of 5 from 189 reviews, with 651 students enrolled. It is a paid course, first published April 2018.

Apache Spark: A Comprehensive Guide to Data Processing with Python

Unlock the power of big data with this tutorial on Apache Spark, an open-source distributed processing engine. Learn how to efficiently query and process large datasets using Python within the Spark ecosystem. This guide provides practical, time-saving techniques for leveraging Spark's capabilities.

What You'll Learn:

  • Apache Spark Architecture: Gain a solid understanding of Spark's core components and how they work together.
  • Python Environment Setup: Learn to configure a Python environment optimized for Spark development.
  • Data Collection Techniques: Explore various methods for gathering data into Spark.
  • Data Processing Techniques: Understand and differentiate between techniques for transforming and manipulating data within Spark.
  • RDDs vs. DataFrames: Delve into the differences between Resilient Distributed Datasets (RDDs) and DataFrames, and when to use each.
  • Data Input: Learn how to efficiently read data from various sources, including files and HDFS.
  • Schema Specification: Master schema definition using both reflection and programmatic approaches (specifically for DataFrames).
  • Lazy Execution: Understand the concept of lazy evaluation and its impact on Spark performance.
  • Transformations and Actions: Explore a wide range of transformations and actions available for both RDDs and DataFrames.
  • SQL Interaction: Utilize SQL to query and analyze data stored in Spark DataFrames.

Data Processing with Spark DataFrames: This tutorial focuses on using Spark DataFrames for efficient data manipulation and analysis. You'll gain proficiency in distributed data processing techniques to handle large datasets effectively.

About the Author:

Tomasz Drabas is a Data Scientist at Microsoft with over 12 years of international experience in data analytics and data science. His expertise spans diverse industries including technology, airlines, telecommunications, and finance. Tomasz holds a Master's degree in strategy management and a doctoral degree in operations research. He has a strong background in data analysis, machine learning, and big data technologies. He is a published author, conference speaker, and reviewer for scientific journals. Based in the Seattle area, Tomasz specializes in solving complex problems in high-dimensional feature spaces.

$ whoami — the devs behind tutorialsearch.io

Want AI in your product? Hire the team behind TutorialSearch.io

LLM features, RAG, agents and automation wired into a real product — built by the devs who shipped this site.