Public or inexpensive whois lookup service

December 22, 2015 12:18 PM

I have a several-thousand word set. I would like to check each word to see if a .com domain (or other TLD) name is available. The command-line whois binary throttles my scripted requests after about 10 searches. Is there a reasonably comprehensive whois service that is free or inexpensive, which I can query in scripted manner, via a command-line session?
posted by a lungful of dragon to Computers & Internet (14 answers total)

Bulk Whois API starts at $10/month for 3000 requests.
posted by jedicus at 12:29 PM on December 22, 2015

Best answer: If all you're checking is availability, you can easily check that by doing a DNS query.

For example, on a Mac or Linux system, you can use the "host" command from the command-line:

$ host -t soa metafilter.com
metafilter.com has SOA record dns0.easydns.com. zone.easydns.com. 1407434301 43200 10800 604800 900
$ host -t soa mutafilter.com
Host mutafilter.com not found: 3(NXDOMAIN)

posted by 1970s Antihero at 12:32 PM on December 22, 2015

What are you trying to accomplish with this? Maybe there's another approach that would accomplish the same ends more expeditiously. The throttling is probably (partly) to try to prevent people just fishing for available real-world domain names so they can buy them up to resell them (a.k.a. domain name squatting - sort of legal but not really ethical/the intended use of the public service that is DNS); if you're trying to _find_ domain squatters, though, maybe there's a separate service that will help you with that.
posted by amtho at 12:37 PM on December 22, 2015

Response by poster: I'm trying to build a list of words that could be turned into domain names, for personal purposes.

If I do several thousands of DNS queries with host from a home IP, will I end up breaking name lookups (and effectively access to the network) for the rest of the networked devices in our home?

The Bulk Whois API is kind of what I'm looking for, except it is expensive for the number of words I want to filter.
posted by a lungful of dragon at 12:46 PM on December 22, 2015

Best answer: "If I do several thousands of DNS queries with host from a home IP, will I end up breaking name lookups (and effectively access to the network) for the rest of the networked devices in our home?"

No, this is what the DNS system is made for. Query away!

Also, whois data is _terrible_ (take it from someone in the industry). Every system has its own format, its own different set of fields - doing any kind of programmatic parsing is going to be a royal pain. Unless you need to extract owner information, querying for a SOA is the way to go.
posted by revertTS at 12:52 PM on December 22, 2015 [1 favorite]

Response by poster: No, this is what the DNS system is made for. Query away!

Just wanted to make sure I don't get all the DNS queries from our home network throttled, in the same way that whois lookups were. Thanks!
posted by a lungful of dragon at 12:55 PM on December 22, 2015

The short answer is that if you want to do thousands of WHOIS requests, you're going to build in rate limiting to your script and do it slowly, or pay for the privilege.

Most WHOIS service are rate limited to ensure that it's available for transfers, and to prevent people from mining WHOIS and selling the data.

DNS queries should not be rate limited. I'm not sure how the host command would work if a domain is registered but has no nameservers.

You could also look at: http://domainr.build/
posted by Sleddog_Afterburn at 12:57 PM on December 22, 2015 [1 favorite]

Best answer: DNS cannot be used to verify availability of a domain. I can have a domain registered, but not put it in DNS. Even if it's in DNS, I don't have to have the base domain (e.g. example.com) resolve. So you should do a whois on any domains that fail DNS lookup to eliminate your false positives.
posted by kindall at 1:03 PM on December 22, 2015 [4 favorites]

Best answer: What kindall says is not true. If you get back a NXDOMAIN response, then you can interpret that definitively that the name is not registered (at the time you made the query). And while it's true that a domain might be registered but not in DNS, every domain that is in DNS must have an SOA record, which is what I suggested you query for. If it's in DNS, the SOA record will be there for the base name. Furthermore, if the name is registered but not in DNS, you will be able to tell that because the query will time out, or possibly some other error besides NXDOMAIN, like SERVFAIL.
posted by 1970s Antihero at 1:29 PM on December 22, 2015

Response by poster: Great, thanks. Using host reports a non-zero error code for not found results, which is quite handy:

$ host -t soa "abjudge.com"
Host abjudge.com not found: 1(FORMERR)
$ echo $?
1

$ host -t soa "metafilter.com"
metafilter.com has SOA record dns0.easydns.com. zone.easydns.com. 1407434301 43200 10800 604800 900
$ echo $?
0

It seems to run a great deal faster than whois, too:

$ time awk 'length($0) >= 5 && length($0) <= 8' /usr/share/dict/words | head -200 | while read word; do host -t soa "$word.com" > /dev/null; if [[ $? -ne 0 ]]; then echo $word; fi; done > candidates.txt

real 0m22.288s
user 0m0.822s
sys 0m0.803s

A similar search with whois takes a couple hours and gets progressively slower. Thanks to all for your help.
posted by a lungful of dragon at 2:29 PM on December 22, 2015 [2 favorites]

Best answer: Furthermore, if the name is registered but not in DNS, you will be able to tell that because the query will time out, or possibly some other error besides NXDOMAIN, like SERVFAIL.

You can get these errors for, you know, server failures, not just unregistered domains.

I still recommend following up with a whois.
posted by kindall at 4:50 PM on December 22, 2015 [1 favorite]

Response by poster: Kindall is indeed correct. I'm doing spot checks and there are a lot of false positives. Still, I'm in a better position to explore than I was yesterday.
posted by a lungful of dragon at 11:10 PM on December 22, 2015

Were I to script this, I would run a DNS query against the .com servers themselves and ask for the NS record. Every registered domain must have an NS record held in the .com zone. SOA records are served from the authoritative name servers for a domain and so you're asking for a record that relies on the delegated name server being online and functioning. Asking the .com zone for the NS records it knows about would be better since the chance of every server that holds .com going down at the same time is...minuscule.

.com is held at [a-m].gtld-servers.net. I can't get host to do what I want, but dig does, where 192.12.94.30 is the address for e.gtld-servers.net:

user@host:/home/user# dig +norecurse ns metafilter.com @192.12.94.30

; <<>> DiG 9.9.5-3ubuntu0.6-Ubuntu <<>> +norecurse ns metafilter.com @192.12.94.30
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29249
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 4

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;metafilter.com. IN NS

;; AUTHORITY SECTION:
metafilter.com. 172800 IN NS dns3.easydns.org.
metafilter.com. 172800 IN NS dns1.easydns.com.
metafilter.com. 172800 IN NS dns2.easydns.net.
metafilter.com. 172800 IN NS dns4.easydns.info.

;; ADDITIONAL SECTION:
dns1.easydns.com. 172800 IN AAAA 2001:1838:f001::10
dns1.easydns.com. 172800 IN A 64.68.192.210
dns2.easydns.net. 172800 IN A 198.41.222.254

;; Query time: 192 msec
;; SERVER: 192.12.94.30#53(192.12.94.30)
;; WHEN: Tue Dec 22 23:50:35 PST 2015
;; MSG SIZE rcvd: 221

I threw in +norecurse so dig wouldn't ask the .com server to keep going (which it won't do).
posted by fireoyster at 11:52 PM on December 22, 2015

Response by poster: Looks interesting. What result should I expect to see from dig in the case of a non-existent domain? I ran a couple tests, and I'm not certain how to tell the results apart to make a "registered/not-registered" call.
posted by a lungful of dragon at 10:48 AM on December 23, 2015