How does Netcraft count active sites?
Netcraft’s active-sites measure estimates the number of distinct sites by comparing the HTML structure of homepages served from the same IP address. It helps distinguish sites with distinct content from automatically generated pages that share a common template. Here, “active” refers to this content-based measure, rather than visitor traffic or how recently a site was updated.
Why count active sites?
The Netcraft Web Server Survey has run since August 1995, exploring the internet to find new websites. Every month, an HTTP request is sent to each site to identify the web server software used to support it.
In the early days of the web, hostnames were a good indication of actively managed content providing information and services to the internet community. Today, many hostnames serve automatically generated pages created by domain registrars, hosting companies, advertising providers or speculative domain registrants. Counting hostnames alone therefore does not reliably indicate the number of distinct websites.
IP addresses alone cannot reliably indicate the number of distinct websites. Multiple domains can point to the same content, while shared hosting allows many different websites to use a single IP address.
Netcraft’s active-sites survey addresses this by retrieving homepages and comparing their HTML tag structures. This helps distinguish sites with distinct content from groups of pages generated from the same template.
Methodology
The front page is taken from each hostname on an IP address and then compared with the front page of other hostnames on the same IP address. Only sites with distinct content will be counted, such that unique content is counted once no matter how many domains and hostnames point at the site.
Where an IP address serves a large number of websites, it is infeasible to sample all sites on the IP address, both from a logistical viewpoint and from an acceptability viewpoint. We therefore apply a sampling technique and extrapolate the results across all sites hosted on that IP.
Accordingly, Netcraft do not visit all sites on an IP but instead take a random sample of sites. For an IP address with N sites, Netcraft retrieves: 625 * N / (N + 624) sites.
This means that for IPs with a small number of sites (<100), almost all the sites are visited; and for IPs with huge numbers of sites (>100000), around 625 are visited. Keeping the number of requests to be under 700 even in the most pathological case ensures that the survey robots are not banned from sites, and that the results remain respectable and accurate.
Determining Activeness
The comparison is made based on the structure of the page, rather than the text. The typical situation for domains is that a domain registration or domain advertising company will use the same template tag structure for each domain, but vary some text on each site to reflect the domain name, the customer name, or the date registered, etc., which makes a strict comparison of the HTML body unsafe. Free online service providers, like blogging providers, will do something similar, with the variation that they usually provide their accounts hostnames under a common domain name, instead of separate domains; the domain under which these accounts occur often uses Wildcard DNS, and in some cases any hostname under the domain will return valid but computer-generated content, with the hostname taken to be as an account name and interpolated into the content. Hence it is important for the active sites methodology to discard the actual words in the page and focus instead on the page structure, as represented by the HTML tag structure.
Some companies like to use framesets for presentation, displaying more than one HTML document on the same page. When the front page contains a frameset tag, we follow the references to request each framed document within the frameset, and compare the tag structure of those in the same way as the initial front pages. Following these links is necessary otherwise there would be a risk that all front pages consisting of framesets would be incorrectly assessed as inactive.
If the site returns content (i.e., a page or frameset, as opposed to an error response or connection timeout), the HTML tag structure is extracted and an MD5 hash is computed, allowing the content to be quickly compared with other content on the same IP address.
Example
before hashing, and:
would be reduced down to:
afterwards.
The reason for doing this is that most inactive sites are automatically generated from a template, and whilst the page contents may differ between sites, the page structure remains constant.
Explore more Netcraft research
View the latest Web Server Survey for monthly website counts and web server market share.
For more detailed analysis of hosting providers, cloud platforms and internet infrastructure, explore Netcraft’s Internet Data and Research.