Scale up as you grow — whether you're running one virtual machine or ten thousand.

From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.

This is my code snippet
import scrapy
class QuotesSpider(scrapy.Spider):
name = 'quotes'
custom_settings = {
'CONCURRENT_REQUESTS': 25,
'CONCURRENT_REQUESTS_PER_DOMAIN': 100,
'DOWNLOAD_DELAY': 0
}
f = open("list.txt")
start_urls = [url.strip() for url in f.readlines()]
f.close()
def parse(self, response):
for quote in response.xpath("//div[@class='border block']"):
urlgem = quote.xpath(".//div[@class='col-md-4 pull-right']/a/@href").extract()
if urlgem:
yield {
'text': urlgem,
}
Using Terminal I execute above code using command
scrapy runspider quotes_spider.py -o quotes.json
list.txt contains 50 URL of same domain separated by lines
As per my understanding the code should put 25 requests to the domain for 25 URL (from a list of 50 URL) and should complete in 2-3 seconds time span and generate file name quotes.json
The output in file quotes.json is coming out as expected but SCRAPY is not performing the task concurrently instead it is fetching URL one by one and takes approximately 55-60 seconds to complete.
Please help!!!
6a2e133a62fe4859920f9233303d8c
PowerAdSpy
phosphide
lking
santoshpatil
Rory McEwan
crisdan
kdcarter17
garantieheld
7c31733a76c942f69d6186090a3787
36d56192d3114513857618b22686be