The ds9 WHOIS Proxy is a component of the ds9 Farming package.
It supports requests to WHOIS servers to verify the ownership of crawled URLs.
When following links to URLs on unknown servers it is often mandatory to understand who is responsible for the content of the website to build trust.
Since many websites do not contain clear ownership information or it is hard to find such information automatically, one source to retrieve ownership
of a server is the domain's WHOIS record.
WHOIS the Owner of this Server?
The WHOIS registrar with who the owner registered the domain provides a database with ownership information. This information can be queried with
the ds9 WHOIS filter type, which in turn contacts the ds9 WHOIS Proxy.
The ds9 WHOIS Proxy maintains a database of domain records across all domain registrars that have been accredited by the Internet Assigned Numbers Authority (IANA).
IANA provides a database with more than 1.500 top level domains (TLDs) like .com or .net.
For each of these TLDs, the IANA WHOIS server provides the name of the responsible WHOIS server to which serving WHOIS data for domains is assigned.
Querying one of these TLD operators will either return the queried WHOIS data for the domain or will lead to another operator who has
himself been assigned operator for the domain. At the end of the chain, the WHOIS data will be served.
The ds9 WHOIS Proxy maintains records for all WHOIS servers maintained by the domain operators and executes WHOIS queries against these servers,
if information for a domain is requested by the WHOIS filter type.
The WHOIS Filter queries the ds9 WHOIS Proxy. If the WHOIS Proxy knows the domain, it immediately delivers the WHOIS data for the domain.
If the domain is unknown, the WHOIS Proxy queries the registrar of the TLD of the domain to retrieve the data.
If no WHOIS data can be retrieved, because the WHOIS server refuses to provide data, the WHOIS Filter builds a fall back record for the
location based on the TLD, in case the TLD is country specific or allows the determination of the country based on the TLD.
WHOIS servers have individual query restrictions which apply to the number of concurrent connections, the request frequency and an upper limit for
requests per day. These constraints limit the throughput of the ds9 WHOIS Proxy.