+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: submit element from a list into a post request (/thread-9162.html)
submit element from a list into a post request - Godzilla - Mar-24-2018
Hello everybody,
I'm a Python beginner and I have no idea on how to solve this problem
I create a variable called list_id with several customer codes inside; then I should use every element of this list separately in a post request (this post request performs a search in a form), scrape the result of the search, make some cleaning and export to a .csv file.
I succeeded in doing it once, but I can't figure out how to create a loop.
This is the code I created so far:
list_id = [ ] #create an empty list where I append all the customer codes I need
for td in soup.find_all('td', {'data-review-id':True}):
list_id.append(td['data-review-id'].replace('"', ''). replace(r'\/', '/').replace('\\', '').encode("latin-1")) # I extract the customer codes from a web page and append them to my list_id
#print list_id
# I set up the data for my post request:
data2 = {
'reviews_review_search_presenter[ownClientOnlyYN]': 'false',
'reviews_review_search_presenter[userID]': '5228',
'utf8': '\u2713',
'authenticity_token': form3["authenticity_token"], #it takes the authenticity token from elsewhere
'reviews_review_search_presenter[timeOffset]': '0',
'reviews_review_search_presenter[reserveYN]': 'F',
'reviews_review_search_presenter[aktiv]': 'alle' ,
'reviews_review_search_presenter[meinungstyp]': 'R' ,
'reviews_review_search_presenter[typus]': '' ,
'reviews_review_search_presenter[minNote]': '1' ,
'reviews_review_search_presenter[maxNote]': '5' ,
'reviews_review_search_presenter[fromDate]': '' ,
'reviews_review_search_presenter[fromTime]': '' ,
'reviews_review_search_presenter[toDate]': '' ,
'reviews_review_search_presenter[toTime]': '' ,
'reviews_review_search_presenter[fromUpdateDate]': '' ,
'reviews_review_search_presenter[fromUpdateTime]': '' ,
'reviews_review_search_presenter[toUpdateDate]': '' ,
'reviews_review_search_presenter[toUpdateTime]': '' ,
'reviews_review_search_presenter[artikelNummer]': '' ,
'reviews_review_search_presenter[masterStyleID]': '' ,
'reviews_review_search_presenter[digiStyleID]': '' ,
'reviews_review_search_presenter[id]': list_id[0], #I take only the first customer code in my list_id
'reviews_review_search_presenter[text]': '' ,
# 'reviews_review_search_presenter[ownClientOnlyYN]': 'false' ,
'reviews_review_search_presenter[accountNumber]': '' ,
'reviews_review_search_presenter[name]': '' ,
'reviews_review_search_presenter[firstName]': '' ,
'reviews_review_search_presenter[email]': ''
}
response3 = s.post('http://mywebsite', data = data2)
#print(response3.status_code)
page3 = response3.text
#print page
soup2 = BeautifulSoup(page3, 'html.parser')
for table in soup2.find_all(["table", {"class": "customer-info-table"}], limit=1):
string = table.get_text().replace("\\n", "").replace("\/th", "").replace("\/span", "").replace("\/i", "").replace("\/p", "").replace("\/a", "").replace("\/tr", "").replace("\/div", "").replace("\/td", "").replace("<\/tbody", "").replace("\/table", "").replace("\/textarea", "").replace("\/b","").replace("\/select", "").replace("<\/option", "").replace("<\/label", "").replace("ead", "").replace("<>", "").encode("utf-8")
ext = "Letzte"
fileNameOnly = string[:string.find(ext) + len(ext)]
print fileNameOnly
reviews = []
reviews.append(fileNameOnly)
print reviews
file = open("out.csv", "wb")
file.write(fileNameOnly)For the moment it works using only the first element from my list_id, but I would like to set up a loop that takes each element from my list_id, make a different post request, append it to the variable "reviews", so that I can download the final .csv file with all the reviews I need together.Is it possible (and, if yes, how?). Any idea or suggestion would be really appreciated.
Thank you in advance for your help!