Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python - Scrapy Login form
#1
Hello guyes....i was messing with my script to login in this website but seems doesnt work....i'll like to login first to loggin_url and then request the rest urls to get data

# -*- coding: utf-8 -*-
import scrapy


class StrongbotSpider(scrapy.Spider):
name = 'StrongBot'
login_url = 'https://www.strongflex.eu/en/login?back=my-account'
start_urls = ['https://www.strongflex.eu/en/a3-96-03-8l-fwd/298-221077b-front-wishbone-rear-bush-5902553512151.html?search_query=221077&results=5']

def parse(self, response):
data = {
'email': '[email protected]',
'passwd': 'example',
}
yield scrapy.FormRequest(url=self.login_url, formdata=data, callback=self.after_login)


def after_login(self, response):
for item in response.css('div.pb-right-column.col-xs-12.col-sm-4.col-md-3'):
yield {
'Price' : item.css('#our_price_display::text').extract()
}
Reply
#2
Your spider is doing a few things wrong:
  1. It starts by opening search url (from start_urls) before login is even attempted
  2. Submits the form request with some missing data (SubmitLogin is required)
  3. Tries to parse the resulting page instead of the search page

I'd suggest using an InitSpider and doing it like this:
# -*- coding: utf-8 -*-
import scrapy
from scrapy.spiders.init import InitSpider


class StrongbotSpider(InitSpider):
    name = 'StrongBot'
    login_url = 'https://www.strongflex.eu/en/login'
    start_urls = [
        'https://www.strongflex.eu/en/a3-96-03-8l-fwd/'
        '298-221077b-front-wishbone-rear-bush-5902553512151.html'
        '?search_query=221077&results=5'
    ]

    def init_request(self):
        return scrapy.Request(
            url=self.login_url,
            callback=self.login,
        )

    def login(self, response):
        yield scrapy.FormRequest.from_response(
            response=response,
            formid='login_form',
            formdata={
                'email': '[email protected]',
                'passwd': 'gorosucks',
            },
            callback=self.initialized,
        )

    def parse(self, response):
        for item in response.css('div.pb-right-column.col-xs-12.col-sm-4.col-md-3'):
            yield {
                'Price': item.css('#our_price_display::text').extract()
            }
Reply
#3
im getting the price in zl, we can use request for euro?
Reply
#4
Sure, just submit the form for currency selection (the dropdown on the top of the page).
That should set a cookie, and you should get the desired result.
Reply
#5
i was trying to do the same thing with different website, but seems doesn't work as i wanted to....

# -*- coding: utf-8 -*-
import scrapy


class TabotSpider(scrapy.Spider):
    name = 'TaBot'
    login_url = 'https://www.tatechnix.de/tatechnix/gx/?language=en'
    start_urls = ['https://www.tatechnix.de/tatechnix/gx/product_info.php?info=p44235_ta-technix-sport-suspension-kit-opel-astra-h-caravan-2-0t-1-7-1-9cdti--without-level-control-type-a-h-30-30mm.html',]

    def init_request(self):
        return scrapy.Request(
            url=self.login_url,
            callback=self.login,
        )

    def login(self, response):
        yield scrapy.FormRequest.from_response(
            response=response,
            formid='loginbox',
            formdata={
                'text': 'example',
                'password': 'example',
            },
            callback=self.initialized,
        )

    def parse(self, response):
        for content in response.css('#gm_attr_calc_price'):
            yield {
                'Price' : content.css('span[itemprop="price"]::Text').extract()
            }
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Using python within an html form t4keheart 5 5,228 Aug-17-2020, 12:28 PM
Last Post: t4keheart
  Python Scrapy Date Extraction Issue tr8585 1 3,236 Aug-05-2020, 04:32 AM
Last Post: tr8585
  Python Scrapy tr8585 2 2,303 Aug-04-2020, 04:11 AM
Last Post: tr8585
  Python Webscraping with a Login Website warriordazza 0 2,571 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  How to perform a successful login(signin) through Requests in Python Kalet 1 2,304 Apr-24-2020, 01:44 AM
Last Post: Larz60+
  Automatic login hidden form Andra111 0 1,600 Mar-26-2020, 08:06 AM
Last Post: Andra111
  Posting value from excel to Form (Python+Selenium) revanth 0 1,764 Feb-05-2020, 10:44 AM
Last Post: revanth
  Python-selenium script for automated web-login does not work hectorKJ 2 3,983 Sep-10-2019, 01:29 PM
Last Post: buran
  HOWTO? Login DSL Modem with Python Requests: need Click "Apply" Button Webtest 4 8,406 Aug-20-2019, 04:03 PM
Last Post: johnmina
  Python - Scrapy Baggelhsk95 0 2,261 Apr-24-2019, 01:07 PM
Last Post: Baggelhsk95

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020