im completely stuck (webcrawler)

[Larz60+]

Try placing some print statements at critical points in your software, followed by

input()

Make sure what you expect, is actually what you get!

If you use an IDE with a debugger, use it, it really does speed things up