Python Forum
Add http to a list on a text file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Add http to a list on a text file
#1
I scraped a list of links of topo maps of mi.
The list is only half the link.
The first part of the link is "http://www.dnr.state.mi.us" is not on the list
This have to be added to each link.
The list looks like this:

links

/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf
/spatialdatalibrary/pdf_maps/topomaps/Adams_Park.pdf
/spatialdatalibrary/pdf_maps/topomaps/Adams_Point.pdf
/spatialdatalibrary/pdf_maps/topomaps/Adamsville.pdf
/spatialdatalibrary/pdf_maps/topomaps/Addis_Creek.pdf
/spatialdatalibrary/pdf_maps/topomaps/Addison.pdf
/spatialdatalibrary/pdf_maps/topomaps/Adrian.pdf

I need each link to look like this
(____this part is added___) to the list.
http://www.dnr.state.mi.us/spatialdatali.../Adair.pdf

I am using windows 7 and python 3.7
I hope their a way of doing this.
Thank you
Renny
Reply
#2
Hello,

I've forgotten how to readlines in a txt but if you copy and paste to a csv and then do:

import csv
from csv import reader # import the modules

with open('myfile.csv','r') as data #open the first as a read only so you don't lost any data if it goes wrong
    data_read = reader(data) #prep the object function
    for row in data_read: #for as many times as there are lines
        list1.append(row) #save to a variable list
This will save all of the lines to a list, which you can then write to another csv or output them in the shell - I'll put both below:

Adding the pre-fix:

for x in range(len(list1)):
    list2.append("http://www.dnr.state.mi.us" + list1[x]) #puts the prefix before every item in the first list and saves to a second list
To another csv:
with open('secondcsv.csv',"w",newline='') as csv2: #Creates the second CSV 
    csvWriter = csv.writer(csv2,delimiter=',') #Tells it to put each row on a new line/cell
    for x in range(len(list2)):  #will run as many times as there are items in the list
        csvWriter.writerow(list2[x]) #write to cell
To output in shell
for x in range(len(list1)):
    print("http://www.dnr.state.mi.us"+list1[x])

#or you can just do the below if you created the second list
for x in range(len(list2)):
    print(x)
while dad_has_cigs == True:
    happiness = True
    if dad_has_cigs == False:
    print("Dad come home!")
    happiness = not happiness
    break
Reply
#3
Do you not have some command line tools like sed that can help you do this easily, or at least a text editor that can?
Reply
#4
You should giver a try Blue Dog that's how it's work here,it's not hard task.

Good effort jamesaarr,but a some unnecessary stuff and look at Never use "for i in range(len(sequence))
Can do it like this,so can iterate over directly file object and the use f-string for string concatenation.
url = 'http://www.dnr.state.mi.us'
with open('url_refs.txt') as f:
    for line in f:
        print(f'{url}{line.strip()}')
Output:
http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adams_Park.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adams_Point.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adamsville.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Addis_Creek.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Addison.pdf http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adrian.pdf
Reply
#5
snippsat, that worked perfect. I looked up " line.strip()" and I still don't know how it works, I am going have to play around with it. Thank you very much. Think you two jamesaarr and ndc85430 for your help. Big Grin
Reply
#6
(Sep-23-2021, 05:47 PM)Blue Dog Wrote: snippsat, that worked perfect. I looked up " line.strip()" and I still don't know how it works
One thing that very useful in Python is take stuff out and test it interactively.
>>> # So when reading from file a new line(\n) is added
>>> url = 'http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf\n'
>>> url
'http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf\n'
>>> print(url)
http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf

>>> # strip() take away \n
>>> url = url.strip()
>>> url
'http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf'
>>> print(url)
http://www.dnr.state.mi.us/spatialdatalibrary/pdf_maps/topomaps/Adair.pdf
>>> 
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Thumbs Up Need to compare the Excel file name with a directory text file. veeran1991 1 1,071 Dec-15-2022, 04:32 PM
Last Post: Larz60+
  read a text file, find all integers, append to list oldtrafford 12 3,372 Aug-11-2022, 08:23 AM
Last Post: Pedroski55
  find some word in text list file and a bit change to them RolanRoll 3 1,482 Jun-27-2022, 01:36 AM
Last Post: RolanRoll
  Modify values in XML file by data from text file (without parsing) Paqqno 2 1,580 Apr-13-2022, 06:02 AM
Last Post: Paqqno
  Converted Pipe Delimited text file to CSV file atomxkai 4 6,843 Feb-11-2022, 12:38 AM
Last Post: atomxkai
  [split] How to convert the CSV text file into a txt file Pinto94 5 3,257 Dec-23-2020, 08:04 AM
Last Post: ndc85430
  Code not reading http link from .txt file (Beginner level) plarrip 3 2,354 Dec-17-2020, 11:33 PM
Last Post: bowlofred
  Regex text file to store data in list TheSithSiggi 1 1,501 Dec-03-2020, 04:46 PM
Last Post: bowlofred
  requests_futures.sessions retry list of HTTP 429 Johanoosterwaal 0 1,534 Nov-20-2020, 09:23 PM
Last Post: Johanoosterwaal
  Saving text file with a click: valueerror i/o operation on closed file vizier87 5 4,330 Nov-16-2020, 07:56 AM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020