Python Forum
How do i loop through list of data from CSV file and post requests in aspx dynamics w
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How do i loop through list of data from CSV file and post requests in aspx dynamics w
#1
This is my another post related to python POST METHOD. i am trying to web scrape the website which is :https://maharerait.mahaonline.gov.in/SearchList/Search
I edited my this post after doing research and reached till here but still unable to loop through list
please select the registered agents first and then you can send 4 digits number

In the website it requires to send 4 digits number to get the data which starts from a500.

now i have an csv file starting from first cell and the range is from a500 to a599.

After sending 4 digits numbers some has data and some doesn't have data. i have written a code but it is not printing anything in CSV file, can anyone tell me where i am doing mistake.

import requests
from bs4 import BeautifulSoup
import csv

final_data = []
file = []
url = "https://maharerait.mahaonline.gov.in/SearchList/Search"

response = requests.get(url)
data = response.text
soup = BeautifulSoup(data, "html.parser")

RequestVerificationToken = soup.find(attrs={"name":"__RequestVerificationToken"})['value']

filename = "Agents.csv"
f = open(filename, "r")
file_data = f.read()
datas = file_data.split()
file.append(datas)
print(file)
#f.close()

for title in range(len(file)):#it doesnt loops all data
    search_item = title

    headers = {"user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
               "content-type":"application/x-www-form-urlencoded",
               "accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"}


    formfields= {"__RequestVerificationToken":RequestVerificationToken,
                 "Type":"Agent",
                 "ID":"0",
                 "pageTraverse":"1",
                 "Project":'',
                 "hdnProject":'',
                 "Promoter":'',
                 "hdnPromoter":'',
                 "CertiNo":search_item,
                 "hdnCertiNo":search_item,
                 "Division":'',
                 "hdnDivision":'',
                 "hdnDistrict":'',
                 "hdnProject":'',
                 "hdnDTaluka":'',
                 "hdnVillage":'',
                 "District":'',
                 "Taluka":'',
                 "Village":'',
                 "CompletionDate_From":'',
                 "hdnfromdate":'',
                 "CompletionDate_To":'',
                 "hdntodate":'',
                 "PType":'',
                 "hdnPType":'',
                 "btnSearch":"Search"}
             #"TotalRecords":"",
             #"CurrentPage":"1",
             #"TotalPages":"1"}
    r = requests.post(url, data=formfields, headers=headers)
    data = r.text
#print(data)


    soup = BeautifulSoup(data, "html.parser")
    get_details =soup.find_all(class_="grid-wrap")
    for details in get_details:
        text = details.find_all("tr")[1:]
        count = 0
        for tds in text:
            td = tds.find_all("td")[1]
            #rera = td.find_all("span")
            rnumber = ""
            for num in td:
                rnumber = num.replace("\n","")
                sublist = []
                sublist.append(rnumber)
            name = tds.find_all("td")[2]
            #prj_name = name.find_all("span")
            prj = ""
            for prjname in name:
                prj = prjname.replace("\n","")
                sublist.append(prj)
            final_data.append(sublist)
    
   
#get_details()
filename = "maharera_agents5.csv"
with open("./"+filename, "w") as csvfile:
    csvfile = csv.writer(csvfile, delimiter=",")
    csvfile.writerow("")
    for i in range(0, len(final_data)):
        csvfile.writerow(final_data[i])
i have removed functions and made it simple and i was getting confused with functions

attached is my csv file

Attached Files

.csv   Agents.csv (Size: 600 bytes / Downloads: 645)
Reply
#2
I think your first loop is wrong.

filename = "Agents.csv"
f = open(filename, "r")
datas = f.read().split()  #datas is already a list
f.close()
#print(datas)

for search_item in datas:
    print(search_item)
    ....
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  POST requests - different requests return the same response Default_001 3 1,901 Mar-10-2022, 11:26 PM
Last Post: Default_001
  Post HTML Form Data to API Endpoints Dexty 0 1,382 Nov-11-2021, 10:51 PM
Last Post: Dexty
  requests.post() does work Alto 1 1,993 Aug-13-2021, 07:58 AM
Last Post: ndc85430
  Scraping .aspx page Larz60+ 21 50,850 Mar-18-2021, 10:16 AM
Last Post: Larz60+
  How to get registeration data from a website that uses .aspx? Help me brothers. humble_coder 1 2,420 Feb-18-2021, 06:03 PM
Last Post: Larz60+
  POST request with form data issue web scraping hoff1022 1 2,649 Aug-14-2020, 10:25 AM
Last Post: kashcode
  How to POST html data to be handled by a route endpoint nikos 1 2,347 Mar-07-2020, 03:14 PM
Last Post: nikos
  requests issue with post on dot_net api Heinrich 1 2,437 Jan-23-2020, 04:28 AM
Last Post: Larz60+
  Making several POST requests RayeEThompson507 1 2,553 Nov-25-2019, 08:50 PM
Last Post: micseydel
  requests post/get to HTML form mrdominikku 1 2,300 Nov-03-2019, 07:12 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020