Scrapy splash 无法登录,lua 错误:尝试调用方法 'send_text'(一个空值)

我想登录网站并从中爬取数据。尝试了很多次,仍然不起作用。

该网站包含 JavaScript,因此我使用了 splash 进行渲染。Splash 工作正常,但只是代码部分我无法解决。我们的工作环境不能考虑使用 selenium。

class MySpider(scrapy.Spider):
    name ="test"
    start_urls = ['https://steamcommunity.com/login/home/']
    req = 10
    series = {}

    def start_requests(self):
        script = """

        function main(splash)
            splash.private_mode_enabled = false
            local url = splash.args.url
            assert(splash:go(url))
            assert(splash:wait(10))

            splash:set_viewport_full()

            local search_input = splash:select_all('#steamAccountName')
            search_input:send_text("xxxxxxx")
            local search_input = splash:select_all('#steamPassword')
            search_input:send_text("xxxxxxx")
            assert(splash:wait(5))

            local submit_button = splash:select_all('#login_btn_signin')
            submit_button.click()

            assert(splash:wait(10))

            return {
                html = splash:html(),
                png = splash:png(),
            }
        end
        """
        yield SplashRequest(
            'https://steamcommunity.com',
            callback=self.after_login,
            endpoint='execute',
            args={'lua_source': script}
                            )

    def after_login(self, response):
        open_in_browser(response)

以下是错误消息:

2019-10-09 15:18:32 [scrapy.core.engine] INFO: Spider opened
2019-10-09 15:18:32 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items(at 0 items/min)
2019-10-09 15:18:32 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-10-09 15:18:43 [scrapy_splash.middleware] WARNING: Bad request to Splash:{'type': 'ScriptError', 'info': {'line_number': 12, 'type'
: 'LUA_ERROR', 'error': "attempt to call method 'send_text' (a nil value)", 'source': '[string "..."]', 'message': 'Lua error: [string ".
.."]:12: attempt to call method \'send_text\' (a nil value)'}, 'error': 400, 'description': 'Error happened while executing Lua script'}
2019-10-09 15:18:43 [scrapy.core.engine] DEBUG: Crawled (400) <GET https://steamcommunity.com via http://localhost:8050/execute> (referer
: None)
2019-10-09 15:18:43 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://steamcommunity.com>: HTTP status code is no
t handled or not allowed
2019-10-09 15:18:43 [scrapy.core.engine] INFO: Closing spider (finished)

非常感谢任何帮助!谢谢。

点赞