Scraping by Example - Handling JSON data

Today’s post will cover scraping sites where the pages are dynamically generated from JSON data. Compared to static pages, scraping pages rendered from JSON is often easier: simply load the JSON string and iterate through each object, extracting the relevent key/value pairs as you go. Continue reading

Scraping by Example - Iterating through Select Items With Mechanize

A common scraping task is to get all of the results returned for every option in a select menu on a given form. Continue reading

Form Handling With Mechanize And Beautifulsoup

Python Mechanize is a module that provides an API for programmatically browsing web pages and manipulating HTML forms. BeautifulSoup is a library for parsing and extracting data from HTML. Together they form a powerful combination of tools for web scraping. Continue reading