Python Forum

Python Forum > Python Coding > Web Scraping & Web Development > Requests login failure
Full Version: Requests login failure
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2

test

Sep-09-2018, 06:34 AM
Hello, i'm trying to learn to use the requests module. I have written the following code, but it doesnt login to my account and returns an error.

import requests

url = 'https://python-forum.io/member.php?action=login'
 
username	= 'test'
password 	= '*******'

with requests.Session() as session:
	session.get(url)
	csrftoken = session.cookies['csrftoken']
	login_data = dict(csrfmiddlewaretoken=csrftoken, username=username, password =password, next='/' )
	session.post(url, data=login_data, headers={'Referer' : "https://python-forum.io"})
	page = session.get('https://python-forum.io/usercp.php')
Im getting the following error:
Traceback (most recent call last):
File "python-io.py", line 12, in <module>
csrftoken = session.cookies['csrftoken']
File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 328, in __getitem__
return self._find_no_duplicates(name)
File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 399, in _find_no_duplicates
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='csrftoken', domain=None, path=None"

Kindly advice.
My project involves logging in to a website and searching for specific content which will then be downloaded and parsed to be presented in a clean format. I started learning python about 2 weeks back to achieve this. But i am stuck. Can you please tell me where i am going wrong and what i can do about it?

test

Sep-09-2018, 03:35 PM
Inspecting the page again, it looks like there is no csrf_token parameter being passed from the form. So, removed that from the code and checked. Still doesnt seem to work.

#test again

import requests
from bs4 import BeautifulSoup

url = 'https://python-forum.io/member.php?action=login'
 
payload = {
	'username'	: 'test',
	'password'	: '*******'
}

with requests.Session() as c:
	c.get(url)
	c.post(url, data=payload)
	r = c.get('https://python-forum.io/usercp.php')
	print ('Logout' in r.content)
The output is false, with no errors.
Can someone please point me in the right direction?
I understand that this might be a silly doubt, but it has actually taken up my entire day /o\
Any help will be greatly appreciated

test

Sep-09-2018, 05:05 PM
Trying to use selenium now..
#Trying selenium

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen

browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
username = browser.find_element_by_name('username')
username.send_keys("test")
password = browser.find_element_by_name('password')
password.send_keys("******")
time.sleep(5)
submit = browser.find_elements_by_class_name('button')[1]
submit.click()
time.sleep(5)
  
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl)
Error:
Traceback (most recent call last):
  File "sel.py", line 17, in <module>
    submit.click()
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element <input class="button" name="submit" type="submit"> could not be scrolled into view
Kindly advice
Cleaned up the code a little, different error now...
#Trying selenium

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen

browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("******")
time.sleep(5)
browser.find_elements_by_name('submit').click(0)
time.sleep(5)
  
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl)
Error:
Error:
Traceback (most recent call last):
  File "sel.py", line 14, in <module>
    browser.find_elements_by_name('submit').click(0)
AttributeError: 'list' object has no attribute 'click'
okay, so found out what was causing the trouble.
Replaced
browser.find_elements_by_name('submit').click(0)
with:
browser.find_element_by_name('submit').click(0)
Now back to the older error message:
Error:
Traceback (most recent call last):
  File "sel.py", line 14, in <module>
    browser.find_element_by_name('submit').click()
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element <input class="button" name="submit" type="submit"> could not be scrolled into view
I'm probably the most annoying user :D
Still, please do help :)
okay, trying ActionChains also is returning an error
#Trying selenium

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains

browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("******")
time.sleep(5)
button = browser.find_element_by_name('submit')
actions = ActionChains(browser)
actions.move_to_element(button).perform()
time.sleep(5)
  
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl)
Error:
Traceback (most recent call last):
  File "sel.py", line 17, in <module>
    actions.move_to_element(button).perform()
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/common/action_chains.py", line 80, in perform
    self.w3c_actions.perform()
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/common/actions/action_builder.py", line 76, in perform
    self.driver.execute(Command.W3C_ACTIONS, enc)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: TypeError: rect is undefined
_o\
I would have preferred getting this done by requests, but at this point i'd be happy even if selenium works /o\

test

Sep-10-2018, 02:36 AM
Hello again!
I was messing with the code again and this one seems to work
#Trying selenium

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains

browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("*******")
time.sleep(5)
button = browser.find_element_by_class_name('button submit')
button.send_keys(Keys.PAGE_DOWN)
button.click(1)
time.sleep(5)
  
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl)
:D
Any inputs on making it work with requests are very welcome as i think it will be faster that way. In the meanwhile i will try to write the rest of the code. Thank you :)
PS: Also, can you please explain why this started working? Where was i going wrong?

test

Sep-10-2018, 03:52 PM
Hello, i had put the wrong code as the working one..
here's the code that works..

#Trying selenium

from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains

browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("*********")
time.sleep(5)
button = browser.find_elements_by_class_name('button')[2]
# button.send_keys(Keys.PAGE_DOWN)
button.click()
time.sleep(5)
  
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl)
Inputs welcome, especially to make this work with requests

snippsat

Sep-10-2018, 06:01 PM
(Sep-10-2018, 03:52 PM)test Wrote: [ -> ]Inputs welcome, especially to make this work with requests
This should do it with Requests.
import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}

params = {
    "username": "your_username",
    "password": "xxxxxxx",
    "remember": "yes",
    "submit": "Login",
    "action": "do_login",
}

with requests.Session() as s:
    s.post('https://python-forum.io/member.php?action=login', headers=headers, params=params)
    # logged in! session cookies saved for future requests
    response = s.get('https://python-forum.io/index.php')
    # cookies sent automatically!
    soup = BeautifulSoup(response.content, 'lxml')
    welcome = soup.find('span', class_="welcome").text
    print(welcome)
Output:
Welcome back, snippsat. You last visited: Today, 05:44 PM Log Out

test

Sep-11-2018, 01:43 AM
Thanks a ton, sir Big Grin
i was desperately searching for ways to make it work, and i came across several of your posts where you kindly helped people like me and was just hoping my thread gets your attention.
Sir, can you please post the exact steps to follow to determine what parameters are required to make requests work with any website?
I had tried passing all headers as parameters, but then i read that requests handles it by itself for the most part. The i tried passing in several parameters too. It just wouldnt work. Then i was convinced that the reason it is not working is because i am not passing the csrf token. looking for a way to do that took my entire sunday!
Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed.
Thank you very much Smile

test

Sep-13-2018, 02:20 AM
(Sep-11-2018, 01:43 AM)test Wrote: [ -> ]Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed.
Thank you very much Smile

For example, the following code just does not work again...
#wreqt.py

import requests
from bs4 import BeautifulSoup
from lxml import html

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}


params = {	'acountName'	: '*****',
			'password'		: '*****',
			'useSrp'		: 'false',
			'publicA'		: '',
			'clientEvidenceM1'	:	""	}
					
					
with requests.Session() as session:
	session.post('https://us.battle.net/login/en/', headers = headers, params = params)
	response = session.get('https://us.battle.net/account/management/')
	tree = html.fromstring(response.content)
	balance = tree.xpath('//*[@id="primary-balance"]')
	print(balance)
Running this code outputs an empty list, while it should be the available balance for a user. As I am trying to learn the implementation for requests in general, and not just a specific case, can someone please explain how i can figure out what has to be passed to make it work everytime?

test

Sep-13-2018, 03:35 AM
I am using the InspectElement feature in FF and working out the field names. I do not know what to do with the button which says submit though.
Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist. When i try to modify the params dictionary based on the request params from the dev tools, it still doesnt work.
I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all?

snippsat

Sep-13-2018, 07:58 AM
Quote:Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist.
I inspect was send on network(Chrome Dev-tools or FF dev-tools) it will be different for other sites,this is how myBB do it.

(Sep-13-2018, 03:35 AM)test Wrote: [ -> ]I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all?
There is no magic method that work for all web-sites.
There a lot of different way,from more standard ways to homemade/tweaked bye developer.
Requests doc Authentication has some about basic ones,eg Digest, OAuth 1 and 2,OpenId ect...
There are site that has extra login verification,
like verify that human to avoid bots,CAPTCHA, reCAPTCHA and stuff like this make it hard to login.
Pages: 1 2
Python Forum > Python Coding > Web Scraping & Web Development > Requests login failure