Module 3: Web Scraping with Scrapy

Now your toolkit is ready. Let’s learn Scrapy - the most powerful free web scraping tool available. We’ll scrape content from the Scrapy documentation itself, plus Zapier’s help forum.

Why Scrapy?

Why Zapier Needs Scraping

Zapier doesn’t publish downloadable manuals. They rely on:

• Help forum at community.zapier.com
• Blog posts at zapier.com/blog
• Help articles at help.zapier.com

This is common for many SaaS tools. If you want a specialist chatbot for any service, you often need to scrape their help content yourself.

Your First Scrapy Project

Create the Project

scrapy startproject myscraper
cd myscraper

Create a Spider

scrapy genspider zapier_help community.zapier.com

Edit the Spider

Open myspider/spiders/zapier_help.py:

import scrapy

class ZapierHelpSpider(scrapy.Spider):
    name = "zapier_help"
    allowed_domains = ["community.zapier.com"]
    start_urls = [
        "https://community.zapier.com/get-help-50"
    ]

    def parse(self, response):
        # Extract all article links
        for link in response.css("a::attr(href)").getall():
            if link and "/troubleshooting" in link:
                yield response.follow(link, self.parse_article)

    def parse_article(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "content": " ".join(response.css("p::text").getall()),
            "url": response.url
        }

Run the Spider

scrapy crawl zapier_help -o output.json

Batch Processing Strategy

Process sites in batches of 10 or fewer to:

• Limit failures to small batches
• Manage token usage when uploading to chatbots
• Avoid overwhelming target websites

Batch Folder Structure

zapier-content/
├── batch-01/
│   ├── article-01.md
│   ├── article-02.md
│   └── ...
├── batch-02/
│   ├── article-11.md
│   └── ...

Converting to Markdown

Scrapy outputs JSON by default. Convert to markdown:

# In your spider
def parse_article(self, response):
    title = response.css("h1::text").get()
    paragraphs = response.css("p::text").getall()
    
    markdown_content = f"""# {title}

{"".join(f"{p}\n\n" for p in paragraphs)}

*Source: {response.url}*
"""
    
    yield {
        "title": title,
        "markdown": markdown_content,
        "url": response.url
    }

Error Handling

Always handle failures gracefully:

def parse_article(self, response):
    try:
        title = response.css("h1::text").get()
        if not title:
            self.logger.warning(f"No title found: {response.url}")
            return
        # ... rest of parsing
    except Exception as e:
        self.logger.error(f"Failed to parse {response.url}: {e}")

Scraping the Scrapy Docs

Since docs.scrapy.org offers markdown downloads, you can:

1. Download directly if available
2. Or scrape key pages:

scrapy genspider scrapy_docs docs.scrapy.org

# scrapy_docs.py
class ScrapyDocsSpider(scrapy.Spider):
    name = "scrapy_docs"
    allowed_domains = ["docs.scrapy.org"]
    start_urls = [
        "https://docs.scrapy.org/en/latest/intro/overview.html",
        "https://docs.scrapy.org/en/latest/intro/install.html",
        "https://docs.scrapy.org/en/latest/intro/tutorial.html",
    ]
    
    def parse(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "content": "\n".join(response.css("article p::text").getall()),
            "url": response.url
        }

Practical Exercise

1. Create a Scrapy project
2. Scrape 10 pages from the Scrapy documentation
3. Save as markdown files in scrapy-docs/ folder
4. Try scraping a few Zapier help articles

Goal: Build a folder of 10 markdown files ready for chatbot training.

Key Takeaways

1. Scrapy is powerful and free - Best for batch scraping
2. Batch processing - Max 10 sites at a time
3. Error handling - Log failures, continue processing
4. Zapier needs scraping - No manual available, only help forum

Next Steps

Time to build your first chatbot! Continue to Module 4: Chatbot #1 - Scraping Assistant