Explore topic-wise InterviewSolutions in .

Both spider arguments and settings can be used to configure your spider. There is no strict RULE that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the START url of a spider).

To illustrate with an example, assuming you have a spider that needs to log into a site to SCRAPE data, and you only want to scrape data from a certain SECTION of the site (which varies each time). In that case, the CREDENTIALS to log in would be settings, while the url of the section to scrape would be a spider argument.

Both spider arguments and settings can be used to configure your spider. There is no strict rule that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider).

To illustrate with an example, assuming you have a spider that needs to log into a site to scrape data, and you only want to scrape data from a certain section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument.

RAISE the CloseSpider EXCEPTION from a CALLBACK. 

Raise the CloseSpider exception from a callback. 

ENABLE the COOKIES_DEBUG SETTING.

Enable the COOKIES_DEBUG setting.

YES, Scrapy receives and keeps track of cookies SENT by SERVERS, and sends them BACK on subsequent requests, like any REGULAR web browser does.

Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does.

Parsing big feeds with XPath SELECTORS can be problematic SINCE they need to build the DOM of the entire feed in memory, and this can be QUITE slow and consume a lot of memory.

In ORDER to avoid parsing all the entire feed at once in memory, you can use the functions xmliterand csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders use under the cover.

Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory.

In order to avoid parsing all the entire feed at once in memory, you can use the functions xmliterand csviter from scrapy.utils.iterators module. In fact, this is what the feed spiders use under the cover.

The __VIEWSTATE PARAMETER is USED in SITES BUILT with ASP.NET/VB.NET. 

The __VIEWSTATE parameter is used in sites built with ASP.NET/VB.NET. 

To dump into a JSON FILE:

scrapy crawl myspider -o items.json

To dump into a CSV file:

scrapy crawl myspider -o items.csv

To dump into a XML file:

scrapy crawl myspider -o items.xml

To dump into a JSON file:

scrapy crawl myspider -o items.json

To dump into a CSV file:

scrapy crawl myspider -o items.csv

To dump into a XML file:

scrapy crawl myspider -o items.xml

YES, but you can also use the Scrapy SHELL which allows you to quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than PLAIN old pdb.set_trace().

Yes, but you can also use the Scrapy shell which allows you to quickly analyze (and even modify) the response being processed by your spider, which is, quite often, more useful than plain old pdb.set_trace().

999 is a custom response STATUS code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a DOWNLOAD delay of 2 (or higher) in your spider:

class MySpider(CrawlSpider):
NAME = 'myspider'
download_delay = 2
# [ ... rest of the spider code ... ]

Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.

999 is a custom response status code used by Yahoo sites to throttle requests. Try slowing down the crawling speed by using a download delay of 2 (or higher) in your spider:

class MySpider(CrawlSpider):
name = 'myspider'
download_delay = 2
# [ ... rest of the spider code ... ]

Or by setting a global download delay in your project with the DOWNLOAD_DELAY setting.

Some signals SUPPORT RETURNING DEFERREDS from their HANDLERS, others don’t. 

Some signals support returning deferreds from their handlers, others don’t. 

It’ll DEPEND on how LARGE your OUTPUT is. 

It’ll depend on how large your output is. 

Those MESSAGES (logged with DEBUG level) don’t necessarily mean there is a PROBLEM, so you may not need to fix them.

Those messages are THROWN by the OFFSITE Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.

Those messages (logged with DEBUG level) don’t necessarily mean there is a problem, so you may not need to fix them.

Those messages are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.

YES. You can use the runspider command. For example, if you have a spider WRITTEN in a my_spider.py FILE you can run it with:

SCRAPY runspider my_spider.py

Yes. You can use the runspider command. For example, if you have a spider written in a my_spider.py file you can run it with:

scrapy runspider my_spider.py

TRY CHANGING the DEFAULT Accept-Language request HEADER by overriding the DEFAULT_REQUEST_HEADERSsetting.

Try changing the default Accept-Language request header by overriding the DEFAULT_REQUEST_HEADERSsetting.

Yes.

Yes.

By default, Scrapy uses a LIFO QUEUE for storing pending requests, which basically means that it CRAWLS in DFO ORDER. This order is more convenient in most cases. If you do want to CRAWL in true BFO order, you can do it by setting the following settings:

DEPTH_PRIORITY = 1

SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'

SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'

By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings:

DEPTH_PRIORITY = 1

SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'

SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'

Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP PROXY downloader MIDDLEWARE. SEE HttpProxyMiddleware.

Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP Proxy downloader middleware. See HttpProxyMiddleware.

PROBABLY, but we don’t like that word. We THINK Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.

We BELIEVE that, if something is already done well, there’s no NEED to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them PROPERLY, and focus on the real problems we need to solve.

Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.

We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.

Scrapy is SUPPORTED under Python 2.7 and Python 3.3+. Python 2.6 SUPPORT was DROPPED starting at Scrapy 0.20. Python 3 support was added in Scrapy 1.1.

Scrapy is supported under Python 2.7 and Python 3.3+. Python 2.6 support was dropped starting at Scrapy 0.20. Python 3 support was added in Scrapy 1.1.

Yes, you can. As mentioned above, BEAUTIFULSOUP can be used for parsing HTML responses in Scrapy CALLBACKS. You just have to feed the response’s body into a BeautifulSoup object and extract whatever DATA you need from it.

Here’s an example spider using BeautifulSoup API, with lxml as the HTML parser:

from bs4 import BeautifulSoup

import scrapy

class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = (
'http://www.example.com/',
)

def parse(self, response):
# USE lxml to get decent HTML parsing SPEED
soup = BeautifulSoup(response.text, 'lxml')
yield {
"url": response.url,
"title": soup.h1.string
}

Yes, you can. As mentioned above, BeautifulSoup can be used for parsing HTML responses in Scrapy callbacks. You just have to feed the response’s body into a BeautifulSoup object and extract whatever data you need from it.

Here’s an example spider using BeautifulSoup API, with lxml as the HTML parser:

from bs4 import BeautifulSoup

import scrapy

class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = (
'http://www.example.com/',
)

def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, 'lxml')
yield {
"url": response.url,
"title": soup.h1.string
}

BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an APPLICATION framework for writing web spiders that crawl web sites and extract DATA from them.

Scrapy provides a built-in MECHANISM for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any PYTHON code.

In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing JINJA2 to Django.

BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.

Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code.

In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.