Modern Web Scraping with Python using Scrapy Splash Selenium
Learn web scraping & crawling with Python 3, Scrapy, Splash & Selenium. Master data extraction & automation. 2nd Edition (2021).
Course Content
Modern Web Scraping with Python using Scrapy Splash Selenium is an all-levels Udemy course taught by Ahmed Rafik. It runs 8h 37m, is taught in English, and includes a certificate of completion. Learners rate it 4.4 out of 5 from 3,815 reviews, with 25,449 students enrolled. It is a paid course, last updated May 2021.
Unlock the Power of Web Scraping with Python & Scrapy
Tired of limited, expensive web scraping tools? This comprehensive course empowers you to build your own powerful web scraping tools (spiders) using Scrapy, the leading Python framework. Go beyond consumer tools and gain complete control over your data extraction.
What You'll Learn:
- Web Scraping Fundamentals: Master the core concepts of web scraping and data extraction.
- Scrapy Spider Development: Learn to build complete, functional spiders to extract data from websites.
- XPath & CSS Selectors: Become proficient in locating specific content and nodes within the Document Object Model (DOM) using XPath and CSS selectors.
- Data Storage: Effectively store extracted data in various formats, including JSON, CSV, and external databases like MongoDB and SQLite3.
- Custom Pipelines: Develop your own custom pipelines to process and clean your scraped data.
- Splash & Selenium for JavaScript Scraping: Conquer dynamic websites that rely heavily on JavaScript. Learn to scrape JavaScript-heavy sites using Scrapy Splash and Selenium – a technique rarely covered in detail.
- Crawling Behavior: Understand how web crawlers work and build robust CrawlSpiders for comprehensive site exploration.
- Avoiding Blocks & Rate Limiting: Implement strategies to avoid getting your IP address banned while scraping websites.
- Custom Middleware: Develop custom middleware to enhance your spider's functionality and control.
- Web Scraping Best Practices: Adhere to industry best practices for ethical and effective web scraping.
- API Scraping: Learn techniques for extracting data from APIs.
- Request Cookies: Handle website cookies to maintain sessions and access restricted content.
- Infinite Scroll Handling: Scrape websites with infinite scroll features.
- Hosting Spiders: Deploy your spiders for free on Heroku.
- Scheduled Scraping: Run your spiders automatically with custom scripts for continuous data collection.
- Data Duplication Prevention: Implement methods to prevent storing duplicate data.
- Splash Deployment: Deploy Splash to Heroku (Exclusive).
- Excel Data Export: Write scraped data directly to Excel files.
- Website Login: Automate website logins using Scrapy.
- Image & File Downloading: Download files and images from scraped websites using Scrapy.
- Proxy Management: Utilize proxies with Scrapy spiders and CrawlSpiders for enhanced anonymity and reliability.
- Advanced Proxy Solutions: Leverage Crawlera with Scrapy & Splash for robust proxy management.
Why Choose This Course?
- Up-to-Date Content: This course features the latest versions of Python 3.7, Scrapy 1.6, and Splash 3.0.
- In-Depth, Step-by-Step Guidance: Receive a comprehensive, hands-on guide to becoming a professional web scraper.
- Exclusive Splash Expertise: Gain unparalleled knowledge of using Splash and Selenium for JavaScript scraping – techniques you won't find in other courses.
- Free Hosting: Learn to host your spiders and Splash instances on Heroku (Exclusive).
- Automated Scraping: Create custom scripts for scheduled, unattended spider execution.
- 30-Day Money-Back Guarantee: Enroll with confidence knowing you're protected.
Who is this course for?
Whether you're a data analyst looking to expand your skillset or someone seeking to extract structured data from unstructured HTML, this course is for you. Learn to transform raw web data into valuable insights.
What Students Are Saying:
"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! " - Hiran Kumar
"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed." - Raylyson Estanista
"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses." - Richard Blackmon
"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever)." - I S
"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." - Robert Nolte
*"That course is
Need a web app built? Hire the team behind TutorialSearch.io
We ship fast, production-grade full-stack apps (Next.js, React, edge infra) — the same stack that powers this site.