stillhq.com : Mikal, a geek from Canberra living in Silicon Valley

The web is probably parkier than it seems

Wed, 02 Apr 2008 13:08:00 GMT

I've been reading academic papers again (it tends to happen in batches) -- this time I've been focusing on the papers from the Usenix Internet Measurement Conference (IMC) last year. One of the more immediately interesting papers presented was The Web is Smaller than it Seems (bibtex).

The paper discusses a measurement of the size of "the web" based on a scan of domain names listed in either the DMOZ open directory or the .com and .net TLD zone files. You'll note that is a very similar technique to one of those that I use to acquire domain names for my survey of Internet mail servers, which is what originally interested me in this paper. The domains had their www hostname looked up, and then the number of domain names per IP address was used to create an estimate of the total number of web servers present on the Internet. It is of course a little bit more complicated than that, but you can read the paper for more details if you really want.

The paper's findings are interesting:

We find that as much as 60% of the Web servers are co-hosted with 10, 000 or more other Web servers, indicating that the Internet contains many small co-hosted Web servers. Likewise, more than 95% of Web servers share their AS with 1000 or more other Web servers. We additionally find that heavily co-hosted Web servers contribute much less traffic than Web servers that are not co-hosted, confirming that popular servers are not co-located, while less popular servers co-locate more frequently. When considering block lists, we find the vast majority of blocked Web servers are hosted on IPs hosting 100 Web servers or more. This indicates there may be a great deal of collateral damage with IP blocking. Finally, when looking at authoritative DNS servers, we see a high degree of co-location on a very small number of DNS servers, which may result in the Web being fragile from a DNS perspective.

That's a pretty interesting result. Unfortunately, I think that the researchers missed an opportunity here. While they determined that a small number of IP addresses host a large number of web sites, they didn't attempt to determine how many of those domains are just parked content. Now that would have been something interesting to know. Specifically, I've poked around a little with the parking behaviour of domains via the result of MX record look ups, which leads me to suspect that a large number of those heavily co-located domain names are simply parked, and not adding any interesting content to the Internet.

Tags for this post: research()

Comment on this post

The Internet is a strange place

Mon, 24 Mar 2008 07:14:00 GMT

As mentioned previously, I've been downloading HTTP pages as part of my survey of Internet mail servers in order to detect domain parking behaviour. I should have thought a bit harder about that code though, because the implementation is a bit naive. Specifically, the code downloads the source of the web page (to RAM), and then base64 encodes it (to RAM), and finally writes it to the log file. That means that there is a little bit more than two copies of a given page's source in RAM before the operation is complete. However, it hadn't occurred to me that sites such as http://sixela.com/ would exist. That URL results in an endless stream of the word "blah". It took me three worker deaths before I had figured out what the problem was, mainly because when workers use to much RAM their slice is killed, and often the log files are lost.

So the moral of this tale? Don't trust the Internets.

Tags for this post: research()

Comment on this post

Normalising mail server package names

Sat, 22 Mar 2008 11:37:00 GMT

While starting to look at mail server deployment trends, it came to my attention that I needed to normalise the names used for various mail servers across the mail server surveys for which I have data. In some cases the other guys' name for a given mail server was more accurate than mine, so you might notice over the next couple of days that mail server names are a bit variable in the results I have online.

Tags for this post: research() smtp() survey()

Comment on this post

Announcing early results of my survey of SMTP servers

Thu, 20 Mar 2008 01:54:00 GMT

Since June 2007 I have been building as close to an exhaustive survey of SMTP servers connected to the Internet as possible. This has involved coming up with a method for finding IP addresses to probe, probing those IP addresses, and generating results from the data collected. That code has been "finished" for a while now, and I am now ready to make it available to the public.

The current data set includes 46,135,101 IP addresses, with 1,942,603 successfully identified servers. The results for the survey are online, as well as status information for the machines running the measurement system (a different view of that data is available as well). You can even lookup your favourite domain name to see what software its running.

This is the most recent open survey of SMTP servers that I am aware of. There have been other surveys, but they are either quite old or don't make their data publically accessible. Its quite possible there are bugs in the web site which displays the data, so please let me know if you find one. Apart from that, I hope this data is useful to others.

Tags for this post: research() smtp() survey()

Comment on this post

What is the definition of publication?

Wed, 19 Mar 2008 02:11:00 GMT

Here's my quandary for the day -- I have an online interface for my SMTP survey data that allows users to do things like see current overall results, investigate the state of the worker nodes used to perform the measurements, and perform lookups against the data. However, I'm worried that I can't put this interface online because that might count as prior publication, and therefore preclude me from presenting information derived from the survey at academic conferences. I have a similar question about blog posts as well -- does blogging about something I am investigating mean that I can't publish it later?

This must be a common question. How do other researchers handle these sort of publication issues?

Tags for this post: research()

Comment on this post

Compendium of TLD domain access agreements

Mon, 17 Mar 2008 03:12:00 GMT

One of the things I need to further my SMTP survey is lists of domain names. Lots of domain names. I'm sure I'm not the only one who is interested in such things, so I figured I'd take notes on what the process was to get access to that sort of information for research. This post is a "living document", and I'll update it as I find details of new registrars, or actually experience their service.

Note that almost all of these TLDs use basically identical access agreements. Access is free, but you agree to not hammer the registrar's servers, and not to do something like spamming with the data. Additionally, the domain names can only be redistributed as part of a value-added product, and without the ability to dump all of the data the registrar provided in one pass. I need to think about that bit more, because it probably restricts the way I allow use of my dataset.

Access	TLD	Registrar	Procedure	Thanks to
Yes	.com .net .arpa .root	Verisign	To get access to the list of domains in these four TLDs, you need to sign the domain access agreement and fax the contract to Verisign. If accepted (they accepted me as an individual with no real problems), they say they will fax login back to you. However, they snail mailed mine to my contact address, for reasons I can't explain. Access is provided by FTP, and you must connect from a static IP which is defined in the access agreement.
No	.edu	eduCause	Access is not available to the list of registered domains, as per US Department of Commerce requirements. The registrar is seeking to be allowed to provide this data in the future, so it might be worth using the contact form to check if things have changed.
Partial	.info .asia	Afilias	Afilias is an outsourced TLD registrar. They have different access agreement forms for each of the TLDs they manage: .info: Print out the domain access agreement and fax it to Afilias at +1 215 706 5701. My request for access was not responded to. .asia: Print out the domain access agreement and fax it to Afilias at +1 215 706 5701. Access was granted via an email, the data is exposed over FTP. My request was responded to after a several week delay.
Yes	.mobi	dotMobi	Print out the domain access agreement, fill it out, and then scan / email it to operations@mtld.mobi. They replied within a week or so via email, providing a FTP username and password. Note that they only allow you to download the zone files once a day.
	.org		Application form is at http://www.pir.org/PDFs/zone_file_access_agreement.pdf	Andy Warner
	.biz		Application form is at https://www.neulevel.biz/zonefile/	Andy Warner

Tags for this post: research()

Comment on this post

Mikal, tell something I didn't know about SMTP servers on the Internet

Sat, 15 Mar 2008 13:14:00 GMT

As part of my survey of SMTP servers on the Internet (a graphical representation of the results from that post are here), I need to find SMTP servers to survey. One of the ways that I've been doing that is I've been performing large numbers of DNS Mail eXchanger (MX) lookups and then probing the SMTP servers identified by those lookups. I haven't been able to perform those lookups on every domain registered, because not all registrars make their zone files available to researchers. I have a compendium of what I've learnt about zone file access agreements online if you're interested.

Specifically, I performed the following lookups:

Zone		Number of lookups
.arpa		5
.asia		9,044
.com		72,529,657
.mobi		819,849
.net		10,734,157
.root		281
		84,092,993

For each of these domains a DNS MX record lookup was performed using around 100 machines, and the results stored in a series of sharded tables in a MySQL database.

In aggregate, the results look like this:

Total (IP, domain) tuples:	72,863,506
Total unique IPs:	2,136,511
Total unique domains:	46,993,011

There are some interesting things to be found in the MX record data. For example, only 55.8% of the domains I scanned have an MX record at all. That might seem a bit counter intuitive, but when you take into account that a lot of domain names are unused or used simply for a web site, I guess its not that surprising. I would like to spend some more time verifying that this isn't a bug in my survey code, but I haven't gotten around to doing that yet.

Another interesting fact is that GoDaddy appears to be hosting a very large number of domains. Specifically, I found 12,105,590 domains which had one of just two IP addresses owned by GoDaddy as their MX record. That's 25.76% of all of my results. This means that's GoDaddy's domain hosting business is massive -- certainly much larger than I realized previously.

The IP addresses in question are 64.202.166.11 and 64.202.166.12. Some detail:

IP	DNS Reverse
64.202.166.11	mailstore1.secureserver.net
64.202.166.12	smtp.secureserver.net

secureserver.net is a domain registered to "Wild West Domains, Inc.", who appear to be part of the GoDaddy family (according to this GoDaddy help page, secureserver.net is used for GoDaddy DNS servers among other things). To determine how many of these domains are parked, I fired off some download jobs to download the top level page of each domain. At the moment, 1,087,885 of those downloads are complete.

Domains parked with GoDaddy HTTP 302 redirect from the top level page to a URL which is the domain name followed by a short identifier. For example, rastegarenterprises.net 302 redirects to rastegarenterprises.net/?bdb1d640 -- which is a page displaying advertising. Of the sites I have tested so far, 714,455 are parked in this manner.

That means GoDaddy currently has approximately 7,950,196 domains parked. That's around 9.4% of all the domains I have scanned!

Based on looking at IPs serving as MX for an unusual number of domains, the only other immediately obvious entry is that 184,213 domains point to 127.0.0.1. That seems a little bit odd to me.

I'm sure there is other interesting information in this MX data, but I think I'll leave it here for now.

Tags for this post: research() smtp()

Comment on this post

Initial SMTP survey poster results in a pie chart

Fri, 07 Dec 2007 10:31:00 GMT

Graph generated with Google Chart API, which was announced today.

Tags for this post: research() smtp() survey()

Comment on this post

Interesting paper: "YouTube Traffic Characterization: A View From the Edge"

Thu, 06 Dec 2007 06:46:00 GMT

YouTube Traffic Characterization: A View From the Edge" from IMC 2007 is quite interesting. It tracks use of YouTube from the University of Calgary. Interesting random quote:

In total we recorded 23,250,438 valid (i.e., non-failed) HTTP transactions (i.e., request/response pairs). These transactions account for approximately 6.54 TB of data transfer. Only 3% of the HTTP requests were for video files; however, the corresponding HTTP responses accounted for 99% of the total bytes transferred.

Tags for this post: research()

Comment on this post

Microsoft Exchange the most popular SMTP server on the Internet?

Sun, 02 Dec 2007 09:27:00 GMT

Eric McCreath from the Department of Computer Science at the Australian National University and I presented a poster entitled "Inferring Relative Popularity of SMTP Servers" at USENIX LISA 2007. This blog post is a brief discussion of the content of the poster, as well as a landing page for the paper version of the poster as well as the the PDF of the actual poster. For more detail into the measurement techniques used, please check out the complete paper.

We conducted this research because there is little data on the relative popularity of the various available SMTP server implementations. This data is of interest because it aids the development of systems which interact with these servers. For example, a potential DDoS protection system should be tested with the most common SMTP servers, as these are the ones that it is most likely to encounter in everyday use.

Many businesses rely on email of some form for their day to day operation. This is especially true for product support organisations, who are largely unable to perform their role in the company if their in-boxes are unavailable. Allman in "Spam, Spam, Spam, Spam, Spam, the FTC, and Spam" states that Nuclear Research studies estimate that spam costs US businesses $87 billion a year. It seems reasonable to assume that if a low level attack is costing that much, then a complete outage would impose an even greater burden on an enterprise.

There has been little research conducted into the current state of SMTP servers on the Internet, perhaps because this area of research has not been particularly fashionable in comparison to the HTTP metrics which are commonly collected. This is an important area of research however given the level of traffic served by these systems has been growing for years. Barracuda Networks cite Radicati research which indicates that in 2009 228 billion emails will be sent per day, with the vast majority being spam (see Barracuda's site for more details). Afergan and Beverly in "The state of the email address" evaluate the state of email servers in an attempt to determine how SMTP servers are coping with the growth in traffic. Their approach involved sending out probe emails to a variety of domains. The email was crafted to have a strong assurance of bouncing because of not being addressed to a valid address. The authors then monitored the bounce traffic. They concluded that corporate SMTP servers are under surprising levels of strain and do not bounce undeliverable emails in a predictable manner.

We have therefore started to undertake research into SMTP servers as they appear on the Internet, with our first study being a simple survey of which SMTP implementations are most commonly deployed. Our poster discussed the current state of that survey, and provide some early results.

The challenge with determining the popularity of various SMTP server implementations is twofold -- firstly, not all of the SMTP servers which interact with the Internet are able to be probed from the public Internet (for example SMTP routers which route email that came from the Internet, but are not themselves accessible from the Internet); and secondly the sheer number of SMTP servers connected to the network. We have therefore used both passive and active measurements to survey these servers. Each of these measurement techniques is described below.

Bearing in mind that our survey is quite new, and that only 34.6 million IP addresses have been probed so far, the initial results are quite interesting.

You can see from the graph that the most popular SMTP server in our dataset is Microsoft Exchange, followed by Postfix and then Sendmail.

Additional analysis of our existing data, as well as further development of the email parser will improve the accuracy of our survey, which will also increase the number of machines included in the survey. The survey also needs a wider set of inputs for possible IP addresses to probe -- one example of another possible source of probable SMTP servers is MX records for registered domain names. The distributed probing system needs further development to handle the scale of the proving required for a large number of SMTP servers to be included in the survey, and improvements to the reliability of the central server are also required.

This SMTP survey is in its early stages, and there is much work still to do. However, research of this nature is likely to produce results which are of interest to both the research community, as well as software developers and systems administrators. So far a small dataset has been analysed, which has resulted in a reasonably robust distributed probing system being constructed. Further work on the survey will continue in the future, with updated results being published from time to time.

Tags for this post: research() smtp() survey()

Comment on this post