Python Forum
Does anyone know how to scrape/find this data? - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Does anyone know how to scrape/find this data? (/thread-15516.html)



Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019

No one seems to know how to do this. Maybe it's not possible.

I need to find out how quickly particular expired domains get registered after their release.

I know of two ways to do it but both are unviable:

Through the API of a registrar. Just make availability/registration calls but the API calls you get are way too few to be able to determine the speed.

Through paid whois requests: https://jsonwhois.com/pricing

but it's way too expensive. I'd need to spend something like hundreds of dollars per domain to do this. Absolutely non-practical.

If anyone has an alternative idea, I would appreciate it a lot.


RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019

You have not mentioned anything other that the pay site, no target site, Is this click bait?


RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019

(Jan-20-2019, 04:18 PM)Larz60+ Wrote: You have not mentioned anything other that the pay site, no target site, Is this click bait?

What do you mean? That's one of the two methods I know of. I am wondering if anyone knows another way to find this data.
No one online seems to know so maybe it's impossible.


RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019

so is json.whois the site you want to scrape?


RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019

(Jan-20-2019, 05:45 PM)Larz60+ Wrote: so is json.whois the site you want to scrape?

Nonono.

The site can be anything. I simply want to be able to scrape when any given domain (which I know will drop in a 3 hour window) is released, but I estimate it needs to be to at least 0.01 second accuracy. Maybe even 0.001 but I doubt it.

The site I would use for that IS the question. So json.whois is too expensive for the accuracy I'd need. Scraping through an API of a registrar is too slow because they can allocate you 1 to 10 calls per second (so 1 second or 0.1 second accuracy).

I don't know of a third method and it seems that no one does.


RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019

could you provide an example?


RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019

(Jan-20-2019, 05:53 PM)Larz60+ Wrote: could you provide an example?

Of a domain name?

pma.io

It will drop on 2019-01-21 01:45PST - 4:45 PST.

It will be caught quickly after it drops. I'm trying to find out how quickly. My estimate is about 0.01 seconds but the exact information is purposefully held back.


RE: Does anyone know how to scrape/find this data? - Larz60+ - Jan-20-2019

Ok, now I understand, I don't know how you would know without polling.


RE: Does anyone know how to scrape/find this data? - randomguy - Jan-20-2019

I guess it's impossible to find out in another way since no one knows the answer.

I've also tried finding out with a little bit of social engineering but no luck. The polling site I linked actually doesn't even allow that many calls. I'd need to make more accounts and spend $four digits to find out. Oh well.