Does anyone know how to scrape/find this data? - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: Does anyone know how to scrape/find this data? (/thread-15516.html) |
Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019 No one seems to know how to do this. Maybe it's not possible. I need to find out how quickly particular expired domains get registered after their release. I know of two ways to do it but both are unviable: Through the API of a registrar. Just make availability/registration calls but the API calls you get are way too few to be able to determine the speed. Through paid whois requests: https://jsonwhois.com/pricing but it's way too expensive. I'd need to spend something like hundreds of dollars per domain to do this. Absolutely non-practical. If anyone has an alternative idea, I would appreciate it a lot. RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019 You have not mentioned anything other that the pay site, no target site, Is this click bait? RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019 (Jan-20-2019, 04:18 PM)Larz60+ Wrote: You have not mentioned anything other that the pay site, no target site, Is this click bait? What do you mean? That's one of the two methods I know of. I am wondering if anyone knows another way to find this data. No one online seems to know so maybe it's impossible. RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019 so is json.whois the site you want to scrape? RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019 (Jan-20-2019, 05:45 PM)Larz60+ Wrote: so is json.whois the site you want to scrape? Nonono. The site can be anything. I simply want to be able to scrape when any given domain (which I know will drop in a 3 hour window) is released, but I estimate it needs to be to at least 0.01 second accuracy. Maybe even 0.001 but I doubt it. The site I would use for that IS the question. So json.whois is too expensive for the accuracy I'd need. Scraping through an API of a registrar is too slow because they can allocate you 1 to 10 calls per second (so 1 second or 0.1 second accuracy). I don't know of a third method and it seems that no one does. RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019 could you provide an example? RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019 (Jan-20-2019, 05:53 PM)Larz60+ Wrote: could you provide an example? Of a domain name? pma.io It will drop on 2019-01-21 01:45PST - 4:45 PST. It will be caught quickly after it drops. I'm trying to find out how quickly. My estimate is about 0.01 seconds but the exact information is purposefully held back. RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019 Ok, now I understand, I don't know how you would know without polling. RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019 I guess it's impossible to find out in another way since no one knows the answer. I've also tried finding out with a little bit of social engineering but no luck. The polling site I linked actually doesn't even allow that many calls. I'd need to make more accounts and spend $four digits to find out. Oh well. |