Jun-21-2018, 04:55 AM
I'd like to start this off by saying I know nothing at all about Python. If what I want to do is possible, I'll need to find someone to help me to do it.
I have a list of a few million URLs that I gathered using ScrapeBox. I would like to scrape all of those pages for external domains. My end goal is to find available domains that can be registered. I'd like to find a more cost effective method of scraping external URLs. ScrapeBox uses Windows hosting and to scale it to what I want isn't cost effective.
How difficult would this be to do in Python? Also, would it be costly to churn through millions of URLs?
I have a list of a few million URLs that I gathered using ScrapeBox. I would like to scrape all of those pages for external domains. My end goal is to find available domains that can be registered. I'd like to find a more cost effective method of scraping external URLs. ScrapeBox uses Windows hosting and to scale it to what I want isn't cost effective.
How difficult would this be to do in Python? Also, would it be costly to churn through millions of URLs?