ScrapyRT documentation

ScrapyRT is an HTTP server that provides an API for scheduling Scrapy spiders and making requests with spiders.

• Installation
• Scrapyrt HTTP API
  • GET
  • POST
  • Response
• Tweaking spiders for realtime
• Command line arguments
• Configuration
  • Available settings
  • Spider settings
• Logging
• Changes
  • ScrapyRT 0.17.0 (2025-07-28)
  • ScrapyRT 0.16 (2023-02-14)
  • ScrapyRT 0.12 (2021-03-08)