Posts: 53
Threads: 19
Joined: Sep 2018
Sep-09-2018, 06:34 AM
(This post was last modified: Sep-09-2018, 07:16 AM by test.)
Hello, i'm trying to learn to use the requests module. I have written the following code, but it doesnt login to my account and returns an error.
import requests
url = 'https://python-forum.io/member.php?action=login'
username = 'test'
password = '*******'
with requests.Session() as session:
session.get(url)
csrftoken = session.cookies['csrftoken']
login_data = dict(csrfmiddlewaretoken=csrftoken, username=username, password =password, next='/' )
session.post(url, data=login_data, headers={'Referer' : "https://python-forum.io"})
page = session.get('https://python-forum.io/usercp.php') Im getting the following error:
Traceback (most recent call last):
File "python-io.py", line 12, in <module>
csrftoken = session.cookies['csrftoken']
File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 328, in __getitem__
return self._find_no_duplicates(name)
File "/home/bunni/.local/lib/python2.7/site-packages/requests/cookies.py", line 399, in _find_no_duplicates
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='csrftoken', domain=None, path=None"
Kindly advice.
My project involves logging in to a website and searching for specific content which will then be downloaded and parsed to be presented in a clean format. I started learning python about 2 weeks back to achieve this. But i am stuck. Can you please tell me where i am going wrong and what i can do about it?
Posts: 53
Threads: 19
Joined: Sep 2018
Sep-09-2018, 03:35 PM
(This post was last modified: Sep-09-2018, 03:44 PM by test.)
Inspecting the page again, it looks like there is no csrf_token parameter being passed from the form. So, removed that from the code and checked. Still doesnt seem to work.
#test again
import requests
from bs4 import BeautifulSoup
url = 'https://python-forum.io/member.php?action=login'
payload = {
'username' : 'test',
'password' : '*******'
}
with requests.Session() as c:
c.get(url)
c.post(url, data=payload)
r = c.get('https://python-forum.io/usercp.php')
print ('Logout' in r.content) The output is false, with no errors.
Can someone please point me in the right direction?
I understand that this might be a silly doubt, but it has actually taken up my entire day /o\
Any help will be greatly appreciated
Posts: 53
Threads: 19
Joined: Sep 2018
Sep-09-2018, 05:05 PM
(This post was last modified: Sep-09-2018, 06:03 PM by test.)
Trying to use selenium now..
#Trying selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
username = browser.find_element_by_name('username')
username.send_keys("test")
password = browser.find_element_by_name('password')
password.send_keys("******")
time.sleep(5)
submit = browser.find_elements_by_class_name('button')[1]
submit.click()
time.sleep(5)
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl) Error: Traceback (most recent call last):
File "sel.py", line 17, in <module>
submit.click()
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element <input class="button" name="submit" type="submit"> could not be scrolled into view
Kindly advice
Cleaned up the code a little, different error now...
#Trying selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("******")
time.sleep(5)
browser.find_elements_by_name('submit').click(0)
time.sleep(5)
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl) Error:
Error: Traceback (most recent call last):
File "sel.py", line 14, in <module>
browser.find_elements_by_name('submit').click(0)
AttributeError: 'list' object has no attribute 'click'
okay, so found out what was causing the trouble.
Replaced browser.find_elements_by_name('submit').click(0) with: browser.find_element_by_name('submit').click(0) Now back to the older error message: Error: Traceback (most recent call last):
File "sel.py", line 14, in <module>
browser.find_element_by_name('submit').click()
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element <input class="button" name="submit" type="submit"> could not be scrolled into view
I'm probably the most annoying user :D
Still, please do help :)
okay, trying ActionChains also is returning an error #Trying selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains
browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("******")
time.sleep(5)
button = browser.find_element_by_name('submit')
actions = ActionChains(browser)
actions.move_to_element(button).perform()
time.sleep(5)
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl) Error: Traceback (most recent call last):
File "sel.py", line 17, in <module>
actions.move_to_element(button).perform()
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/common/action_chains.py", line 80, in perform
self.w3c_actions.perform()
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/common/actions/action_builder.py", line 76, in perform
self.driver.execute(Command.W3C_ACTIONS, enc)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/home/bunni/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: TypeError: rect is undefined
_o\
I would have preferred getting this done by requests, but at this point i'd be happy even if selenium works /o\
Posts: 53
Threads: 19
Joined: Sep 2018
Sep-10-2018, 02:36 AM
(This post was last modified: Sep-10-2018, 02:36 AM by test.)
Hello again!
I was messing with the code again and this one seems to work
#Trying selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains
browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("*******")
time.sleep(5)
button = browser.find_element_by_class_name('button submit')
button.send_keys(Keys.PAGE_DOWN)
button.click(1)
time.sleep(5)
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl) :D
Any inputs on making it work with requests are very welcome as i think it will be faster that way. In the meanwhile i will try to write the rest of the code. Thank you :)
PS: Also, can you please explain why this started working? Where was i going wrong?
Posts: 53
Threads: 19
Joined: Sep 2018
Hello, i had put the wrong code as the working one..
here's the code that works..
#Trying selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time
from urllib.request import urlopen
from selenium.webdriver.common.action_chains import ActionChains
browser = webdriver.Firefox()
url = 'https://python-forum.io/member.php?action=login'
browser.get(url)
browser.find_element_by_name('username').send_keys("test")
browser.find_element_by_name('password').send_keys("*********")
time.sleep(5)
button = browser.find_elements_by_class_name('button')[2]
# button.send_keys(Keys.PAGE_DOWN)
button.click()
time.sleep(5)
goUrl="https://python-forum.io/usercp.php"
browser.get(goUrl) Inputs welcome, especially to make this work with requests
Posts: 7,320
Threads: 123
Joined: Sep 2016
(Sep-10-2018, 03:52 PM)test Wrote: Inputs welcome, especially to make this work with requests This should do it with Requests.
import requests
from bs4 import BeautifulSoup
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
params = {
"username": "your_username",
"password": "xxxxxxx",
"remember": "yes",
"submit": "Login",
"action": "do_login",
}
with requests.Session() as s:
s.post('https://python-forum.io/member.php?action=login', headers=headers, params=params)
# logged in! session cookies saved for future requests
response = s.get('https://python-forum.io/index.php')
# cookies sent automatically!
soup = BeautifulSoup(response.content, 'lxml')
welcome = soup.find('span', class_="welcome").text
print(welcome) Output: Welcome back, snippsat. You last visited: Today, 05:44 PM Log Out
Posts: 53
Threads: 19
Joined: Sep 2018
Thanks a ton, sir
i was desperately searching for ways to make it work, and i came across several of your posts where you kindly helped people like me and was just hoping my thread gets your attention.
Sir, can you please post the exact steps to follow to determine what parameters are required to make requests work with any website?
I had tried passing all headers as parameters, but then i read that requests handles it by itself for the most part. The i tried passing in several parameters too. It just wouldnt work. Then i was convinced that the reason it is not working is because i am not passing the csrf token. looking for a way to do that took my entire sunday!
Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed.
Thank you very much
Posts: 53
Threads: 19
Joined: Sep 2018
(Sep-11-2018, 01:43 AM)test Wrote: Sir, it would be extremely helpful (not just for me, i think) if you could post your general thought process in figuring out what exactly has to be passed.
Thank you very much 
For example, the following code just does not work again...
#wreqt.py
import requests
from bs4 import BeautifulSoup
from lxml import html
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
params = { 'acountName' : '*****',
'password' : '*****',
'useSrp' : 'false',
'publicA' : '',
'clientEvidenceM1' : "" }
with requests.Session() as session:
session.post('https://us.battle.net/login/en/', headers = headers, params = params)
response = session.get('https://us.battle.net/account/management/')
tree = html.fromstring(response.content)
balance = tree.xpath('//*[@id="primary-balance"]')
print(balance) Running this code outputs an empty list, while it should be the available balance for a user. As I am trying to learn the implementation for requests in general, and not just a specific case, can someone please explain how i can figure out what has to be passed to make it work everytime?
Posts: 53
Threads: 19
Joined: Sep 2018
I am using the InspectElement feature in FF and working out the field names. I do not know what to do with the button which says submit though.
Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist. When i try to modify the params dictionary based on the request params from the dev tools, it still doesnt work.
I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all?
Posts: 7,320
Threads: 123
Joined: Sep 2016
Sep-13-2018, 07:58 AM
(This post was last modified: Sep-13-2018, 08:03 AM by snippsat.)
Quote:Snippsat had included the button_name:value element in the params dictionary and then 'action':'do_login'. Inspecting pages of other websites, these parameters do not exist.
I inspect was send on network(Chrome Dev-tools or FF dev-tools) it will be different for other sites,this is how myBB do it.
(Sep-13-2018, 03:35 AM)test Wrote: I am desperately trying to learn how to do this reliable for any website. Can someone please share a link to any webpage/tutorial which explains this in detail, or, maybe, if its not too much to as, explain it here themselves for once and for all? There is no magic method that work for all web-sites.
There a lot of different way,from more standard ways to homemade/tweaked bye developer.
Requests doc Authentication has some about basic ones,eg Digest, OAuth 1 and 2,OpenId ect...
There are site that has extra login verification,
like verify that human to avoid bots,CAPTCHA, reCAPTCHA and stuff like this make it hard to login.
|