Make a Web crawler without using the recursion method.

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Thread Modes

Make a Web crawler without using the recursion method.

hibritusta
Unladen Swallow

Posts: 3

Threads: 1

Joined: Jul 2019

Reputation: 0

Jul-06-2019, 09:38 PM

(Jul-06-2019, 09:15 PM)ichabod801 Wrote: Have a list of links to crawl, initialized as [url]. As long as there is something in that list, pop a link out, scrape the links off that page, and add the valid ones to the list of links to crawl.

This does not exactly solve the problem, but it shifts a recursion limit problem to a memory problem, which may be sufficient.

I am sorry.I couldn't fully understand.
Can you change my code as you say?

Find

Messages In This Thread

Make a Web crawler without using the recursion method. - by hibritusta - Jul-06-2019, 08:16 PM

RE: Make a Web crawler without using the recursion method. - by ichabod801 - Jul-06-2019, 09:15 PM

RE: Make a Web crawler without using the recursion method. - by hibritusta - Jul-06-2019, 09:38 PM

RE: Make a Web crawler without using the recursion method. - by Gribouillis - Jul-06-2019, 09:47 PM

RE: Make a Web crawler without using the recursion method. - by hibritusta - Jul-06-2019, 10:09 PM

RE: Make a Web crawler without using the recursion method. - by noisefloor - Jul-07-2019, 11:46 AM

RE: Make a Web crawler without using the recursion method. - by rootVIII - Jul-08-2019, 11:39 PM

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Web Crawler help	Mr_Mafia	2	1,959	Apr-04-2020, 07:20 PM Last Post: Mr_Mafia
	Web Crawler help	takaa	39	27,788	Apr-26-2019, 12:14 PM Last Post: stateitreal

Users browsing this thread:

View a Printable Version

Make a Web crawler without using the recursion method.

User Panel Messages

Announcements