Python Forum
The python script is continuously running
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
The python script is continuously running
#2
The repetition is in the Scanner.crawl() method. This method is recursively calling itself in line 56. When the first page contains a link to itself, the crawl will start again from the begin. Obviously this does not need to be the first page, this behaviour would always occur.
I see the __init__() method initializes self.target_links = [] which is not used. I would suggest to make target_links a set instead of a list and use it to filter links already visited. (Because sets can easily be filtered, for example: return urls - self.target_links.)
Reply


Messages In This Thread
RE: The python script is continuously running - by ibreeden - Jan-12-2020, 10:29 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Safely running a web scraping script londonhdi 1 1,892 Feb-17-2020, 08:08 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020