使用 Scrapy 和 Splash 爬取 JS 页面的错误

不过，我在 shell 中一直遇到这个问题。

 2018-09-13 14:50:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
 2018-09-13 14:50:36 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6028
 2018-09-13 14:50:37 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
 2018-09-13 14:50:38 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://localhost:8050/robots.txt> (referer: None)
 2018-09-13 14:51:10 [scrapy.downloadermiddlewares.retry] DEBUG: 正在重试 <GET http://quotes.toscrape.com/js/ via http://localhost:8050/render.html> (失败 1 次): 504 Gateway Time-out
 2018-09-13 14:51:36 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)
 2018-09-13 14:51:40 [scrapy.downloadermiddlewares.retry] DEBUG: 正在重试 <GET http://quotes.toscrape.com/js/ via http://localhost:8050/render.html> (失败 2 次): 504 Gateway Time-out
 2018-09-13 14:52:00 [scrapy.downloadermiddlewares.retry] DEBUG: 放弃重试 <GET http://quotes.toscrape.com/js/ via http://localhost:8050/render.html> (失败 3 次): 502 Bad Gateway
 2018-09-13 14:52:00 [scrapy.core.engine] DEBUG: Crawled (502) <GET http://quotes.toscrape.com/js/ via http://localhost:8050/render.html> (referer: None)
 2018-09-13 14:52:00 [scrapy.spidermiddlewares.httperror] INFO: 忽略响应 <502 http://quotes.toscrape.com/js/>: HTTP 状态码未被处理或未被允许

这是我的代码：

import scrapy
from scrapy_splash import SplashRequest
class MySpider(scrapy.Spider):
    name = "jsscraper"

    start_urls = ["http://quotes.toscrape.com/js/"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url=url, callback=self.parse, endpoint='render.html')

    def parse(self, response):
        for quote in response.css("div.quote"):
        scraped_info={
         'authorname':quote.css('small.author::text').extract_first(),
         'quote':quote.css('span.text::text').extract_first(),}
          yield scraped_info

我已经安装了 scrapy-splash，并在 settings.py 中添加了这些命令。同时，我的 Splash 服务器正在运行在 http://localhost:8050/。

另外，当我尝试在 Splash 服务器上渲染任何 URL 时，我会遇到另一个错误：

HTTP Error 400 (Bad Request) Type: ScriptError -> LUA_ERROR Error happened while executing Lua script

Lua error: [string "function main(splash, args) ..."]:2: network3

使用的版本如下：

• Splash 版本：3.2

• Lua 5.2