1.

What Does The Response Status Code 999 Means?

Answer»

999 is a custom response STATUS code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a DOWNLOAD delay of 2 (or higher) in your spider:

class MySpider(CrawlSpider):
NAME = 'myspider'
download_delay = 2
# [ ... rest of the spider code ... ]

Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.

999 is a custom response status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider:

class MySpider(CrawlSpider):
name = 'myspider'
download_delay = 2
# [ ... rest of the spider code ... ]

Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.


Discussion

No Comment Found

Related InterviewSolutions