Requests login failure - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: Requests login failure (/thread-12712.html) Pages:
1
2
|
Requests login failure - test - Sep-09-2018 Hello, i'm trying to learn to use the requests module. I have written the following code, but it doesnt login to my account and returns an error. import requests url = 'https://python-forum.io/member.php?action=login' username = 'test' password = '*******' with requests.Session() as session: session.get(url) csrftoken = session.cookies['csrftoken'] login_data = dict(csrfmiddlewaretoken=csrftoken, username=username, password =password, next='/' ) session.post(url, data=login_data, headers={'Referer' : "https://python-forum.io"}) page = session.get('https://python-forum.io/usercp.php')Im getting the following error: Traceback (most recent call last): File "python-io.py", line 12, in <module> csrftoken = session.cookies['csrftoken'] File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 328, in __getitem__ return self._find_no_duplicates(name) File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 399, in _find_no_duplicates raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path)) KeyError: "name='csrftoken', domain=None, path=None" Kindly advice. My project involves logging in to a website and searching for specific content which will then be downloaded and parsed to be presented in a clean format. I started learning python about 2 weeks back to achieve this. But i am stuck. Can you please tell me where i am going wrong and what i can do about it? RE: Requests login failure - test - Sep-09-2018 Inspecting the page again, it looks like there is no csrf_token parameter being passed from the form. So, removed that from the code and checked. Still doesnt seem to work. #test again import requests from bs4 import BeautifulSoup url = 'https://python-forum.io/member.php?action=login' payload = { 'username' : 'test', 'password' : '*******' } with requests.Session() as c: c.get(url) c.post(url, data=payload) r = c.get('https://python-forum.io/usercp.php') print ('Logout' in r.content)The output is false, with no errors. Can someone please point me in the right direction? I understand that this might be a silly doubt, but it has actually taken up my entire day /o\ Any help will be greatly appreciated RE: Requests login failure - test - Sep-09-2018 Trying to use selenium now.. #Trying selenium from selenium import webdriver from bs4 import BeautifulSoup import time from urllib.request import urlopen browser = webdriver.Firefox() url = 'https://python-forum.io/member.php?action=login' browser.get(url) username = browser.find_element_by_name('username') username.send_keys("test") password = browser.find_element_by_name('password') password.send_keys("******") time.sleep(5) submit = browser.find_elements_by_class_name('button')[1] submit.click() time.sleep(5) goUrl="https://python-forum.io/usercp.php" browser.get(goUrl) Kindly adviceCleaned up the code a little, different error now... #Trying selenium from selenium import webdriver from bs4 import BeautifulSoup import time from urllib.request import urlopen browser = webdriver.Firefox() url = 'https://python-forum.io/member.php?action=login' browser.get(url) browser.find_element_by_name('username').send_keys("test") browser.find_element_by_name('password').send_keys("******") time.sleep(5) browser.find_elements_by_name('submit').click(0) time.sleep(5) goUrl="https://python-forum.io/usercp.php" browser.get(goUrl)Error:
okay, so found out what was causing the trouble. Replaced browser.find_elements_by_name('submit').click(0)with: browser.find_element_by_name('submit').click(0)Now back to the older error message: I'm probably the most annoying user :DStill, please do help :) okay, trying ActionChains also is returning an error #Trying selenium from selenium import webdriver from bs4 import BeautifulSoup import time from urllib.request import urlopen from selenium.webdriver.common.action_chains import ActionChains browser = webdriver.Firefox() url = 'https://python-forum.io/member.php?action=login' browser.get(url) browser.find_element_by_name('username').send_keys("test") browser.find_element_by_name('password').send_keys("******") time.sleep(5) button = browser.find_element_by_name('submit') actions = ActionChains(browser) actions.move_to_element(button).perform() time.sleep(5) goUrl="https://python-forum.io/usercp.php" browser.get(goUrl) _o\I would have preferred getting this done by requests, but at this point i'd be happy even if selenium works /o\ RE: Requests login failure - test - Sep-10-2018 Hello again! I was messing with the code again and this one seems to work #Trying selenium from selenium import webdriver from bs4 import BeautifulSoup import time from urllib.request import urlopen from selenium.webdriver.common.action_chains import ActionChains browser = webdriver.Firefox() url = 'https://python-forum.io/member.php?action=login' browser.get(url) browser.find_element_by_name('username').send_keys("test") browser.find_element_by_name('password').send_keys("*******") time.sleep(5) button = browser.find_element_by_class_name('button submit') button.send_keys(Keys.PAGE_DOWN) button.click(1) time.sleep(5) goUrl="https://python-forum.io/usercp.php" browser.get(goUrl):D Any inputs on making it work with requests are very welcome as i think it will be faster that way. In the meanwhile i will try to write the rest of the code. Thank you :) PS: Also, can you please explain why this started working? Where was i going wrong? RE: Requests login failure - test - Sep-10-2018 Hello, i had put the wrong code as the working one.. here's the code that works.. #Trying selenium from selenium import webdriver from bs4 import BeautifulSoup import time from urllib.request import urlopen from selenium.webdriver.common.action_chains import ActionChains browser = webdriver.Firefox() url = 'https://python-forum.io/member.php?action=login' browser.get(url) browser.find_element_by_name('username').send_keys("test") browser.find_element_by_name('password').send_keys("*********") time.sleep(5) button = browser.find_elements_by_class_name('button')[2] # button.send_keys(Keys.PAGE_DOWN) button.click() time.sleep(5) goUrl="https://python-forum.io/usercp.php" browser.get(goUrl)Inputs welcome, especially to make this work with requests RE: Requests login failure - snippsat - Sep-10-2018 (Sep-10-2018, 03:52 PM)test Wrote: Inputs welcome, especially to make this work with requestsThis should do it with Requests. import requests from bs4 import BeautifulSoup headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' } params = { "username": "your_username", "password": "xxxxxxx", "remember": "yes", "submit": "Login", "action": "do_login", } with requests.Session() as s: s.post('https://python-forum.io/member.php?action=login', headers=headers, params=params) # logged in! session cookies saved for future requests response = s.get('https://python-forum.io/index.php') # cookies sent automatically! soup = BeautifulSoup(response.content, 'lxml') welcome = soup.find('span', class_="welcome").text print(welcome)
RE: Requests login failure - test - Sep-11-2018 Thanks a ton, sir i was desperately searching for ways to make it work, and i came across several of your posts where you kindly helped people like me and was just hoping my thread gets your attention. Sir, can you please post the exact steps to follow to determine what parameters are required to make requests work with any website? I had tried passing all headers as parameters, but then i read that requests handles it by itself for the most part. The i tried passing in several parameters too. It just wouldnt work. Then i was convinced that the reason it is not working is because i am not passing the csrf token. looking for a way to do that took my entire sunday! Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed. Thank you very much RE: Requests login failure - test - Sep-13-2018 (Sep-11-2018, 01:43 AM)test Wrote: Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed. For example, the following code just does not work again... #wreqt.py import requests from bs4 import BeautifulSoup from lxml import html headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'} params = { 'acountName' : '*****', 'password' : '*****', 'useSrp' : 'false', 'publicA' : '', 'clientEvidenceM1' : "" } with requests.Session() as session: session.post('https://us.battle.net/login/en/', headers = headers, params = params) response = session.get('https://us.battle.net/account/management/') tree = html.fromstring(response.content) balance = tree.xpath('//*[@id="primary-balance"]') print(balance)Running this code outputs an empty list, while it should be the available balance for a user. As I am trying to learn the implementation for requests in general, and not just a specific case, can someone please explain how i can figure out what has to be passed to make it work everytime? RE: Requests login failure - test - Sep-13-2018 I am using the InspectElement feature in FF and working out the field names. I do not know what to do with the button which says submit though. Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist. When i try to modify the params dictionary based on the request params from the dev tools, it still doesnt work. I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all? RE: Requests login failure - snippsat - Sep-13-2018 Quote:Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist.I inspect was send on network(Chrome Dev-tools or FF dev-tools) it will be different for other sites,this is how myBB do it. (Sep-13-2018, 03:35 AM)test Wrote: I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all?There is no magic method that work for all web-sites. There a lot of different way,from more standard ways to homemade/tweaked bye developer. Requests doc Authentication has some about basic ones,eg Digest, OAuth 1 and 2,OpenId ect... There are site that has extra login verification, like verify that human to avoid bots,CAPTCHA, reCAPTCHA and stuff like this make it hard to login. |