GoTrained

Scrapy Masterclass

Scrapy vs Other Web Scraping Libraries

Scrapy Installation

	Mac Scrapy Installation
	Windows Scrapy Installation
	Scrapy Installation Instructions
	Python Editor: Sublime Text
	Linux Scrapy Installation

Building Basic Spider with Scrapy

Xpath Syntax

	Using XPath with Scrapy
	Tools to Easily Get XPath

Q&A

	Scrapy Basics
	XPath Syntax

Building More Advanced Spider with Scrapy

– Web Scraping Best Practices

Deploying&Scheduling Scrapy Spider on ScrapingHub

Logging into website Using Scrapy

Scraoy as a Standalonr Script

Building Web Crawler with Scrapy

Scrapy With Selenium

	Why/When We Should Use Selenium
	Selenium WebDriver + Scrapy Selector to Extract URLs
	Selenium Loading Next for Data Extraction (usable even with JavaScript pages)
	Getting Data

Scrapy Spider-Books Store

	Grabbing URLs
	Data Extraction

More about Scrapy

Export Output to Files

	Scrapy Feed Exports to CSV, JSON, or XML
	Export Output to Excel
	Downloading Images with Scrapy Pipelines (NEW)
	Renaming Images with Scrapy Pipelines (NEW)

Scrapy Project1 Scraping Craigslist Eng Jobs in NY

	Craigslist Scraper - Overview
	Creating Scrapy Craigslist Spider
	Craigslist Scrapy Spider #1 – Titles
	Craigslist Scrapy Spider #2 – One Page
	Craigslist Scrapy Spider #3 – Multiple Pages
	Craigslist Scrapy Spider #4 – Job Descriptions
	Editing Scrapy settings.py (e.g. throttling, user agent, etc.)
	Final Scrapy Tutorial, Craigslist Spider Code

Extracting Data to Databases MysQL&MongoDB

	Installing MySQL
	MySQL Installation and Usage
	Writing Data to MySQL
	Installing MongoDB
	MongoDB Installation and Usage
	Writing Data to MongoDB

Scrapy Project 2 Web Scraping Class-Centralcom

	Scraping Class-Central - Part 1: Subjects
	Scraping Class-Central - Part 2: Courses
	Class-Central Code Update

Scrapy Advanced Topics (NEW)

Scrapy Project 3 Web Scraping Dynamic Website eplanningie (NEW)

	ePlanning Scraping Project Overview (NEW)
	ePlanning: Extracting Initial URLs (NEW)
	ePlanning: Crawling Internal Pages (NEW)
	ePlanning: Scrapy Form Requests (NEW)
	ePlanning: Scraping Data (NEW)
	ePlanning: Checking Data Existence (NEW)
	ePlanning: Scraping Data from Table (NEW

Project 4 Web Scraping LinkedIncom (NEW)

	LinkedIn Scraping Project: Overview & Requirements (NEW)
	LinkedIn Logging in (NEW)
	Finding LinkedIn Profiles: Part 1 (NEW
	Finding LinkedIn Profiles: Part 2 (NEW)
	Scraping Data Points from LinkedIn Profiles: Part 1 (NEW)
	Scraping Data Points from LinkedIn Profiles: Part 2 (NEW)
	Connecting to LinkedIn Profiles (NEW)

Solved Web Scraping Exercises

	Yield Data Items from 2 Functions
	How to Order Exported Data
	Xpath contains() and starts-with() functions
	Extracting Hidden Phone Number with Selenium
	Extracting Hidden Phone Number without Selenium
	Using Multiple Proxies with Crawlera (Optional)

Bonus Data Extraction with APIs