Netcraft has developed a technique to identify site operators who control large numbers of internet web sites, but host the sites with external companies rather than develop their own datacenters. These site operators may include web site development companies, shared hosting systems, domain registrars, managed service companies, and private label hosting resellers.
Because of their control over large numbers of sites, and their ability to move from one infrastructure provider to another, these site operators often make attractive prospects for hosting service providers, and hosting automation software companies.
Netcraft identifies such companies by comparing the results of a DNS lookup on each site found by the Web Server Survey with the results of a DNS lookup on its IP address. Typically, the name server in control of the hostname belongs to the site operator, and the name server in control of the IP address belongs to the hosting company.
Sites where the hosting provider name server is authoritative for both the IP address and domain name are tagged so that they can be removed from view - this is the conventional situation where a hosting company runs DNS for its customers, and there is likely to be no third party involved in the relationship.
The dataset covers all 55,000 site operators we find worldwide hosted at some 20,000 hosting locations. We preserve the mappings between each, such that there is a clear identification of the relationships between infrastructure providers and the site operators present on their networks. An analysis by operating system platform is available and similar datasets can be produced on demand for individual continents / countries.
- Only sites found by the Web Server Survey will be included. The number of hosts found worldwide running internet web sites by the Web Server Survey is large (over 1.2 billion hostnames [November 2020] ), but is not exhaustive.
- Attributing a site to a hosting company requires that the hoster provides a reverse DNS server for the network. In some cases where no reverse DNS server is configured, the hoster for a site will be shown as unknown.
- Due to Excel limits, hosting company / name server provider pairs that have below a set threshold of active sites are removed from the dataset. The thresholds are set per country.
- Name server providers, which are used by a large number hosting companies, are removed from the dataset. We have found that companies providing a DNS service to sites situated at many different hosting locations are more likely to be specialist DNS providers than web site developers or hosting services resellers.
The dataset is updated on a monthly basis and is available on a company license basis.