stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Measuring the popularity of SMTP server implementations on the Internet

Mon, 08 Feb 2010 21:29:00 GMT

I'm interested in measuring the performance of SMTP servers connected to the Internet. Before I can poke around inside a SMTP implementation, I want to ensure that I am using one which lots of people use. To that end I have been running a series of SMTP server surveys for the last several years. This work has been alluded to in the past, but I haven't published any results. This has mainly been because while I have written a number of papers on the topic, I am yet to have one accepted by an academic conference. I've been hesitant to comment about my results because of the requirement for academic publications not be previously published work.

I've decided to change that policy. I'm going to reserve a lot of the deeper analysis for academic publication (if I can make such a thing happen), but I am going to start talking about the work I am doing more in public. To start that off, I should mention what I've been doing...

There have been a number of previous surveys of SMTP servers connected to the Internet, with each survey using a different methodology. So although these results are not directly comparable, a comparison still provides some insight into how the server landscape has changed over the last 12 years. A comparison of published surveys is presented in the table below. Each survey in this table shows the: sample size, which is the number of IP addresses surveyed; sample approach, which is the methodology used to determine which IP addresses to sample and adds bias into the sampling; and the number of responses, which is the number of SMTP servers that responded. The majority of these surveys have relied on random sampling of the IP address space, perhaps with a selection algorithm to limit the results selected. Few of the more recent surveys provide complete information on their probing implementation or the rules they used to identify specific implementations from their observations. It should be noted that non-response from a surveyed IP generally indicates that it is not in fact running a SMTP server accessible from the Internet.

Date	Surveyor	Sample size	Sample method	Responses
27 Nov 1996	Bernstein	500,000	Selective random	25,121
214 Aug 1997	Bernstein	200,000	Selective random	8,056
211 May 1998	Bernstein	20,310	MX walk	17,592
22 Apr 2000	Bernstein	12,595	Selective random	10,087
25 Oct 2000	Bernstein	25,777	Random	859
227 Sep 2001	Bernstein	39,206	Random	937
21 Dec 2002	Credentia	4,096	Random	1,837
21 Jan 2003	Credentia	30,000	Random	17,540
21 Apr 2003	Credentia	37,563	Random	20,410
21 May 2007	MailChannels	400,000	Corporate domain names	254,400

Whereas the surveys that I have been running with the assistance of my ever patient PhD supervisor Dr Eric McCreath have been quite a bit larger. Note that larger isn't necessarily better with these sorts of surveys, but my methodology attempts to aim for completeness, and the relative power of PlanetLab makes these computations surprisingly cheap. Details of my surveys so far:

Date	Surveyor	Sample size	Sample method	Responses
January 2008	Still / McCreath	46,136,113	Exhaustive	1,973,748
April 2008	Still / McCreath	92,286,998	Exhaustive	1,609,111
July 2008	Still / McCreath	97,545,668	Exhaustive	1,579,507
October 2008	Still / McCreath	109,661,889	Exhaustive	1,801,081
January 2009	Still / McCreath	110,397,428	Exhaustive	1,916,719
April 2009	Still / McCreath	110,706,130	Exhaustive	1,925,760
October 2009	Still / McCreath	111,209,212	Exhaustive	1,800,573

Our survey is implemented by attempting to identify the MTA software running on an SMTP server using the SMTP connection banner. In other words, a collection of IP addresses are connected to on the SMTP port (TCP 25), and an attempt is made from the early stages of the SMTP protocol interaction to determine what SMTP server software is running on that host. The SMTP protocol will often reply to the connection with a status 220 line, referred to as the SMTP banner, this tells the connecting client that the server is ready. The SMTP banner also frequently states what software the server is running. Even if the software in use isn't explicitly named, it is often a string which is unique to a given SMTP implementation. This technique simply connects on the SMTP port, and logs any lines starting with 220. The connection is then closed, with no attempt to transfer an email occurring.

So what results have I found so far? I'm trying to keep these blog posts to less than 1,000 words each, so that's too big a question to answer here. I've found some quite unexpected things along the way, such as an accurate technique for measuring the occurrence of domain parking on the Internet, and I'll discuss those in future posts. Instead, let me leave you with this short graphical summary of the results so far:

This is the history of the currently five most popular implementations over time. You can see that Sendmail has fallen from a position of market dominance, and Exim is currently the most popular SMTP server implementation.

I have a lot more to say about all this work, but as I mentioned earlier I want to keep the length of these posts down. I'll say more in future posts.

Tags for this post: research()
Related posts: Initial SMTP survey poster results in a pie chart; Interesting paper: "YouTube Traffic Characterization: A View From the Edge"; RemoteWorker v74; Microsoft Exchange the most popular SMTP server on the Internet?; I think I've worked out the problem with the hotel network; Mikal, the massive domain squatter; Internet traffic; Domain name lookup helper for python?; Normalising mail server package names; Satellite internet at Walmart; Long time not much write; RemoteWorker v70; Announcing early results of my survey of SMTP servers; Noticed that smtpsurvey.stillhq.com is down?; Mikal, tell something I didn't know about SMTP servers on the Internet; The witty worm with Vern Paxson; Why does every man and his dog put man pages online?; Sensis Australian search

Comment

Books read in January 2010

Mon, 08 Feb 2010 17:41:00 GMT

Tags for this post: book() read()

Comment

Home power measurement

Sun, 07 Feb 2010 23:55:00 GMT

I've been spending some quality time with a Current Cost CC128 and my existing home sensor network. So far I've discovered that I use quite a bit of power, and that I can remotely monitor how many times a day my wife makes a cup of tea. Some example data:

You can see that it was relatively cool compared with days a few weeks ago today. That's more obvious in the graph showing the last two weeks though:

However, it was quite humid today:

Which is why we didn't have the evaporative cooler on, just the fan. That doesn't seem to really affect our power usage, which really needs more analysis:

The 500 watt minimum power draw makes me unhappy. You can see over a week it never goes away:

Tags for this post: blog()
Related posts: Extreme Machines: Eirik Raude; More on burial methods; It seems to me; The environmental friendliness of shrimp trawling?

Comment

Dogs of War

Sun, 07 Feb 2010 02:25:00 GMT

ISBN: 0446610895
Aspect (2002), Mass Market Paperback, 320 pages
LibraryThing

Another combat anthology, this time edited by David Drake.

Or Battle's Sound (Harry Harrison): already read in Battlefields Beyond Tomorrow and Body Armor: 2000.
Liberty Port (David Drake): already read in The Complete Hammer's Slammer's Volume 1.
Straw (Gene Wolfe): steam punk mercenaries in a future which is more like the past. A good read.
Tomb Tapper (James Blish): this one has a nice twist at the end, although I found the underlying premise of the story hard to believe.
A Relic of War (Keith Laumer): already read in The Compleat Bolo.
Basic Training (Mark L Van Name): this one is a bit sad. Well, all of the stories in this book are sad, but I find this one a bit harder than others because the main character is about the age of my kids.
Witch War (Richard Matheson): an interesting little tale which doesn't end up at all like you expect.
Transstar (Raymond Banks): I didn't think this story was all that good when I was reading it the other night, but its the one I kept thinking about for the next day. There are lots of interesting issues raised here -- arrogance, response, the cost / benefit of aggression.
Time Piece (Joe Haldeman): this story is very much like a small summary of The Forever War, which makes it disappointing in a way... The book is better, and this story glosses over many of the issues.
Clash by Night (Henry Kuttner and C.L. Moore): this story is interesting because it prompted a sub-genre all of its own about mercenaries in the future. However, this story is entertaining by not earth shatteringly great.

Tags for this post: book() Anthology()
Related posts: Battlefields Beyond Tomorrow; Body Armor: 2000; Bolos 1: Honor of the Regiment; The Complete Hammer's Slammers Volume 1; Bolos 4: Last Stand; The Complete Hammer's Slammers Volume 2; The Complete Hammer's Slammers Volume 3; Bolos 2: The Unconquerable; Bolo!; Bolos 3: The Triumphant; The Compleat Bolo; Bolos 5: Old Guard; Bill the Galactic Hero Series; Galactic Dreams; Bolo Strike; Bill the Galactic Hero on the Planet of Bottled Brains; Forever Peace; The Stars Must Wait; Bill The Galactic Hero; Bolo Brigade; Bolos 6: Cold Steel

Comment

Body Armor: 2000

Thu, 04 Feb 2010 22:06:00 GMT

ISBN: 0441069762
Ace (1986), Paperback
LibraryThing

I love a good anthology, although I'd read a few of these stories before:

Contact! (David Drake): this story is quite unlike the other stuff of his that I have read, mostly because its set in the Vietnam War, not the far future. A good story with an excellent twist.
The Warbots (Larry S. Todd): this one wasn't for me. I'm not sure what point the author was trying to make, if any. Its just a long history of imaginary weapons.
The Scapegoat (C. J. Cherryh): this is long, being a novella, and a little hard to read in places because the story jumps around. Overall, a good read though. My first Cherryh, although I have a lot queued up on the shelf.
The Last Crusade (George H. Smith): this was a good story, and would have fit well in Battlefields Beyond Tomorrow as it has a similar war-is-hopeless tone.
Hired Man (Richard C. Meredith): there is a nice twist at the end of this one, although its obvious a few pages before it happens.
Early Model (Robert Sheckley): I was surprised by this one, its the first Sheckley story I've read that I didn't hate. Its not fantastic, but its way better than Bill the Galactic Hero on the Planet of Bottled Brains or his story in Foundation's Friends.
In the Bone (Gordon R. Dickson): this story is odd, and I'm not 100% sure what I think of it. It was entertaining, but also a little hard to believe.
The Chemically Pure Warriors (Allen Kim Lang): this was a good story, being quite reminiscent of Tilley's Amtrak Wars series.
Right To Life (Thomas A. Easton): this story is in the same vein as Make Room, Make Room! and Logan's Run, whilst still being distinct and interesting.
Or Battle's Sound (Harry Harrison): appeared in Battlefields Beyond Tomorrow.
Hero (Joe Haldeman): appeared in Battlefields Beyond Tomorrow, and was turned into The Forever War.

Tags for this post: book() Anthology()
Related posts: Battlefields Beyond Tomorrow; Dogs of War; Bolos 1: Honor of the Regiment; Foundation's Friends; The Complete Hammer's Slammers Volume 1; The Complete Hammer's Slammers Volume 2; The Complete Hammer's Slammers Volume 3; Bill the Galactic Hero on the Planet of Bottled Brains; Bolos 2: The Unconquerable; Bill the Galactic Hero Series; First Family; Galactic Dreams; Death Bringer; Without Warning; Blood River; Bolos 4: Last Stand; Forever Peace; The Stars Must Wait; Bill The Galactic Hero; Iron Master; Cloud Warrior

Comment

Lyonesse

Tue, 02 Feb 2010 21:25:00 GMT

ISBN: 0441505309
Ace (1987), Paperback
LibraryThing

This is another book I read as a child, except in this case I didn't really remember much of it -- the only bit I remembered was the punishment of Madouc's mother, but that might have been because I was a teenaged boy at the time. Overall this is a very good book. It took me a while to read because of being distracted with other projects, but the ongoing oppression of Princess Suldrun didn't really help either -- it was interesting at first, but got depressing after a while. Its also disturbing how many times sexual assault is used as a plot element in this book...

Tags for this post: book() Jack_Vance()

Comment

Building a hygrometer with a HS1101

Fri, 22 Jan 2010 02:17:00 GMT

The next sensor I wanted to add to my home was a set of hygrometers. Specifically I wanted an exterior one, and a matching interior one. This would be useful as we have evaporative cooling, and if the humidity level outside is already high, then it doesn't make a lot of sense to put extra water into the air. Worse than that, it can also damage my books and make the house really clammy. So, adding some sensors was the first step in some form of alerting.

I picked up two HS1101s from ebay quite cheaply (about $4 each IIRC). These devices are capacitors whose capacitance varies proportionally with relative humidity. You also need to provide a temperature at the sensor to correct the value, although the correction is pretty minor so I guess you could skip this if you really wanted to cut costs. Given I have plenty of code for Dallas 1820s now, I just dropped one of those onto the board too.

I just used the circuit from the data sheet for my design, with a few simple tweaks (like the DS1820). Here's my surprisingly unprofessional circuit diagram:

The DS1820 stuff that's not on the data sheet is in red. When built, it looks like this (note the crazy amount of jumper wire):

This gives us temperature on a 1-Wire pin, and an oscillator on another pin which relates to the current humidity. You'll notice that my circuit has some extra wires, that's because I power down the 555 / HS1101 when I'm not taking a sample. I do this because Peter H. Anderson suggested that noise would be a problem otherwise. This circuit was actually quite hard to build and get working. There are a few reasons for that:

The large number of jumpers on the prototype PCB.
The lack of documentation from other arduino hackers (with the notable exception of the rather good Peter H. Anderson page).
The HS1101 data sheet forgets to mention that connecting pins 1 and 8 on the 555 is assumed knowledge.
The values for R1 and R2 vary depending on what model 555 you are using, and are crazily specific. For the LMC555 that I used, R1 is 1238K and R2 is 562K. I got close to these values, but not exact and it did seem to affect accuracy.
You must use a CMOS 555. That's buried in a six word sentence in the middle of a page on the data sheet, and I didn't notice it for a while. With a NMOS 555, you get effectively random numbers out of the circuit. Worse than that, CMOS 555s are actually a little hard to find, and I had to get mine from Farnell.
I attempted to calibrate with the government weather data from the next suburb over. Unfortunately, as best as I can tell, that data is wrong. It claims that its currently as humid here as it is in Cairns in the wet season, which I deny. Calibration is an ongoing issue for me, although I have some ideas on how to progress there. It might also not matter, as I am building an identical sensor for inside the house and as long as they are both equally wrong I can still detect the "turn off the water to the evap" state that I want to.

Here's a comparison between my data for today and the weather service's:

The code to run the HS1101 is relatively simple:

// Temperature and humidity sensor node. Based on the FridgeControlWeb project of mine
// as well as http://www.phanderson.com/picaxe/relhum_count.html

#include <enc28j60.h>
#include <etherShield.h>
#include <ip_arp_udp_tcp.h>
#include <ip_config.h>
#include <net.h>
#include <websrv_help_functions.h>
#include <avr/io.h>
#include <math.h>

#include <OneWire.h>
#include <DallasTemperature.h>

#define ONEWIRE 3
#define HS1101DATA 5
#define HS1101POWER 7

int count_transitions(int ms);

// How long between measurement cycles
#define SLEEP_SEC 10

OneWire oneWire(ONEWIRE);
DallasTemperature sensors(&oneWire);

unsigned long last_checked = 0, this_check = 0;

// Web server setup
#define MYWWWPORT 80
#define BUFFER_SIZE 550
#define ERROR_500 "HTTP/1.0 500 Error\r\nContent-Type: text/html\r\n\r\n<h1>500 Error</h1>"

static uint8_t mymac[6] = {0x54, 0x55, 0x58, 0x10, 0x00, 0x25}; 
static uint8_t myip[4] = {192, 168, 1, 252};
static uint8_t buf[BUFFER_SIZE + 1];
char data[BUFFER_SIZE + 1];

// The ethernet shield
EtherShield es = EtherShield();

uint16_t http200ok(void)
{
  return(es.ES_fill_tcp_data_p(buf, 0, PSTR("HTTP/1.0 200 OK\r\nContent-Type: text/html\r\n"
                                            "Pragma: no-cache\r\n\r\n")));
}

// prepare the webpage by writing the data to the tcp send buffer
uint16_t print_webpage(uint8_t *buf)
{
  uint16_t plen;
  plen = http200ok();
  plen = es.ES_fill_tcp_data_p(buf, plen, PSTR("<html><head><title>Temperature sensor</title>"
                                               "</head><body><pre>"));
  plen = es.ES_fill_tcp_data(buf, plen, data);
  plen = es.ES_fill_tcp_data_p(buf, plen, PSTR("</pre></body></html>"));

  return(plen);
}

// Float support is hard on arduinos
// (http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1164927646) with tweaks
char *ftoa(char *a, double f, int precision)
{
  long p[] = {0,10,100,1000,10000,100000,1000000,10000000,100000000};
  char *ret = a;
  long heiltal = (long)f;
  
  itoa(heiltal, a, 10);
  while (*a != '\0') a++;
  *a++ = '.';
  long desimal = abs((long)((f - heiltal) * p[precision]));
  itoa(desimal, a, 10);
  return ret;
}

#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))

int count_transitions(int ms)
{
     // configure Counter 1
     cbi(TCCR1A, WGM11);
     cbi(TCCR1A, WGM10);

     cbi(TCCR1B, WGM12);  // WGM12::WGM10 000 - Normal mode

     sbi(TCCR1B, CS12);   // CS12::CS10 111 - External clock, count on rising edge.
     sbi(TCCR1B, CS11);
     sbi(TCCR1B, CS10);

     TCNT1 = 0x0000;      // note that TCNT1 is 16-bits
     delay(ms);
     // not sure if should turn off the counter
     return(TCNT1);
}

void setup()   {                
  // initialize the digital pin as an output:
  Serial.begin(9600);
  sensors.begin();
  
  es.ES_enc28j60Init(mymac);
  es.ES_init_ip_arp_udp_tcp(mymac, myip, MYWWWPORT);
  
  pinMode(HS1101POWER, OUTPUT);
}

void loop()                     
{
  int i, j, data_inset;
  char float_conv[10];
  float t;
  DeviceAddress addr;
  uint16_t plen, dat_p;
  float relhum_raw, relhum_corrected;
  int relhum_count;

  // Read temperatures, we dump the state to a buffer so we can serve it
  this_check = millis();
  if(this_check > last_checked + SLEEP_SEC * 1000)
  {
    data_inset = 0;
    sensors.requestTemperatures();
    sprintf(data + data_inset, "Sensor count: %d\n",
            sensors.getDeviceCount());
    data_inset = strlen(data);
 
    for(i = 0; i < sensors.getDeviceCount(); i++)
    {
      t = sensors.getTempCByIndex(i);
      sensors.getAddress(addr, i);
      
      data[data_inset++] = 't';
      for (j = 0; j < 8; j++)
      {
        sprintf(data + data_inset, "%02x", addr[j]);
        data_inset += 2;
      }

      sprintf(data + data_inset, ": %s\n", ftoa(float_conv, t, 2));
      data_inset = strlen(data);
    }
      
    // Power up the 555 / HS1101, and take a measurement. Power it down again afterwards.
    digitalWrite(HS1101POWER, HIGH);
    delay(500);
    relhum_count = count_transitions(1000);
    digitalWrite(HS1101POWER, LOW);
    sprintf(data + data_inset, "HS1101 cycles: %d\n", relhum_count);
    data_inset = strlen(data);
    
    relhum_raw = 557.7 - 0.0759 * relhum_count;
    sprintf(data + data_inset, "Raw humidity: %s\n", ftoa(float_conv, relhum_raw, 2));
    data_inset = strlen(data);
    
    relhum_corrected = (1.0 + 0.001 * (t - 25.00)) * relhum_raw;
    sprintf(data + data_inset, "Corrected humidity: %s\n",
            ftoa(float_conv, relhum_corrected, 2));
    
    Serial.println(data);
    
    last_checked = this_check;
  }
  
  // Handle network packets
  dat_p = es.ES_packetloop_icmp_tcp(buf, es.ES_enc28j60PacketReceive(BUFFER_SIZE, buf));
  if(dat_p != 0)
  {
    if (strncmp("GET ", (char *)&(buf[dat_p]), 4) != 0){
      // head, post and other methods:
      dat_p = http200ok();
      dat_p = es.ES_fill_tcp_data_p(buf, dat_p, PSTR("<h1>200 OK</h1>"));
    }
    
    // just one web page in the "root directory" of the web server
    else if (strncmp("/ ", (char *)&(buf[dat_p+4]), 2) == 0){
      dat_p = print_webpage(buf);
      Serial.println("Served temperature web page");
    }
    
    else{
      dat_p = es.ES_fill_tcp_data_p(buf, 0, PSTR(ERROR_500));
    }
    
    es.ES_www_server_reply(buf, dat_p);
  }
}

As with previous circuits, I'm going to have to thank Doug for hints and tips along the way, as well as letting me steal his entire collection of 8 pin DIP sockets.

Tags for this post: arduino()
Related posts: Beer fridge controller 0.1; The Beer Fridge saga continues; Beer fridge controller 0.3; Beer fridge controller 0.2; Arduino with the kids: Cricket Noise Door Bell; Phoenix for business; Thinking about arduino as a prototyping platform

Comment

The Man in the Rubber Mask

Sat, 16 Jan 2010 15:58:00 GMT

ISBN: 0140235752
LibraryThing

I'd been looking for this book for ages, as it is quite rare, so it was exciting to find it at Gould's the other day. This is the memoir of the dude inside the rubber mask that Kryten wears in Red Dwarf. The book is an easy read, and entertaining, although I wouldn't call it funny. Most of the book focuses on how terribly horrible it is to be encased in rubber day after day while shooting a comedy in terrible locations. Oh, and Robert is slightly insecure which doesn't help.

Overall I'm glad I found this book, and glad I read it again.

Tags for this post: book() Robert_Llewellyn()
Related posts: Better Than Life; Red Dwarf; Last Human; Colony; Bill the Galactic Hero Series; Life, the Universe and Everything; The Restaurant at the End of the Universe; So Long, and Thanks For All the Fish; Bill the Galactic Hero on the Planet of Bottled Brains; Mostly Harmless; Bill The Galactic Hero; The Hitchhikers Guide to the Galaxy; Bill the Galactic Hero: The Planet of the Robot Slaves

Comment

The Renegades of Pern

Fri, 15 Jan 2010 19:50:00 GMT

ISBN: 0345369335
Del Rey (1990), Mass Market Paperback, 352 pages
LibraryThing

This book starts off in quite a disjointed manner, with the introduction of a variety of seemingly unrelated characters. The only thing that they all have in common is that they're holdless. However, as the book progresses these characters are all weaved together into a relatively cohesive story line. I say relatively because there are gaps in the story telling, which can be a little jarring.

Interestingly, this book also clarifies some of the events of the others in the series. Most satisfyingly it includes more detail of the buried settlement at Landing than The White Dragon did, which ties in nicely with the introduction provided in Dragonsdawn. This gives me hope that later books will take the science fiction track I've been wanting them to for a while.

Tags for this post: book() Anne_McCaffrey()
Related posts: The White Dragon; Dragonsdawn; Nerilka's Story; Dragonsinger; Dragondrums; Dragonquest; The Dragonlover's Guide to Pern; Dragonsong; Dragonflight; Moreta: Dragonlady of Pern

Comment

Asimov's Mirage

Thu, 07 Jan 2010 01:16:00 GMT

ISBN: 0743475232
I Books (2003), Mass Market Paperback, 464 pages
LibraryThing

If I was to name one flaw with the Robot City and Robots and Aliens series, it would have to be that they're not very good. They're lackluster, have difficult to believe plots, very simple structure, and are overall poorly thought through. Its a similar sensation to that I feel when I read the tie-in books written after Harrison's Bill the Galactic Hero series. I feel a little sorry for the writers in later books in these series, because I suspect their hands were tied by the poor decisions of previous authors (similarly to the mess that Bear's Foundation and Chaos had to dig that series out after Benford's tragically terrible Foundation's Fear).

Robot City and Robots and Aliens were disappointments because I read Roger MacBride Allen's Caliban series before them, and Caliban is ok. Not awesome, but ok.

I say all of this as an introduction to Mirage. I guess what I'm saying is that I've been wading through Asimov robot tie-ins from other authors for a while now, and some of them are not very good. That's why finding Mirage was such a delight. Its well written, has a similar style as Asimov's own writing, reuses characters and plot elements from previous tie-in books sufficiently to acknowledge their existence without getting bogged down by the poor decisions of those previous series. Its an engaging read, and I'm glad I stuck through these various series long enough to find it.

My only complaint with this book is that the epilogue is confusing and doesn't align with my understanding of the end of the story.

Tags for this post: book() Mark_W_Tiedemann()
Related posts: Isaac Asimov's Robot City: Robots and Aliens: Humanity; Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Robots and Aliens: Alliance; The Robot City, Robots and Aliens Series; Isaac Asimov's Robot City: Robots and Aliens: Intruder; Isaac Asimov's Robot City: Robots and Aliens: Changeling; Isaac Asimov's Robot City: Robots and Aliens: Renegade; Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Robot City: Perihelion; Caliban Series; Isaac Asimov's Utopia; Isaac Asimov's Caliban; Foundation's Triumph; Isaac Asimov's Robot City: Prodigy; Isaac Asimov's Robot City: Odyssey; Isaac Asimov's Robot City: Refuge; Isaac Asimov's Robot City: Suspicion; The Robot City Series; Isaac Asimov's Inferno; Isaac Asimov's Robot Short Stories; Isaac Asimov's Foundation Series

Comment

The Beer Fridge saga continues

Fri, 01 Jan 2010 14:06:00 GMT

Since the last update on the beer fridge, we've had to do some murdering of the original PCB to get it to fit in a case. In addition, we'd failed to take into account the startup power draw for the compressor (5 amps), and had to upgrade the relay we're using from a 3 amp solid state relay. Luckily Doug had the beefier relay just sitting around in his shed. That means we've lost our opto-isolation because the new relay is a simple mechanical one, but we have a relay coil doing the same thing now. A new PCB will make the world a lot neater, which will be nice. Oh, we've also been mentioned on Hack a Day, which has generated some interesting comments on their site.

I've observed that sometimes the relay doesn't turn on, even though the arduino thinks it has done so. An example can be seen here:

(Note that graph is a Google chart server image, generated by a simple visualization program I wrote in python. If you're that way inclined, the visualization software is in my public SVN repository).

At first we thought this was a software problem with comparing ints to floats, but it continued to happen even after we fixed that bug. You can see in the graph above that during testing the compressor was "turned on" (i.e. the arduino thinks it has activated the relay), but the temperature continued to rise. A visual inspection showed that in fact the relay hadn't turned the compressor on, and power cycling the controller (and therefore the relay) didn't help. However, the simple expedient of hitting the relay with a screw driver handle fixed the problem, which made me think that Doug had given me a bodgy relay. This got me thinking about if it would be possible to unstick a relay in software though -- its pretty easy to detect this case (temperature continues to rise despite the compressor being). I'm thinking that perhaps if you cycled the inputs over and over you might be able to get the relay to unstick.

Detecting this case in software might be generally useful anyway though... Ignoring stuck relays, it would also tell you when someone had left the door open for example. It would be nice to detect this case and then give up on the compressor for a few minutes before trying again. I wrote a quick implementation of this "door open detection" and gave it a try, but I've now given up on it as being too fiddly. I found problems such as when the outside world is heating up rapidly (summer mornings for example), the compressor can't make a big enough difference to the internal temperature of the fridge, but you still really want the compressor on because something is better than nothing.

In the end it turned out the problem with the relay was that the arduino wasn't providing enough current to pull the relay contact in reliably. This was fixed with the addition of a transistor to the board, which is yet another thing to add to the next PCB.

One of the big advantages of a microprocessor based custom thermostat is that I can implement new features that wouldn't normally be present in a fridge. Some of these features might turn out to be bad ideas, but I view the beer fridge as a bit of an experimentation platform.

Since my extra Dallas 1820s have arrived from ebay, I also added a sensor inside the freezer. This is interesting because the freezer isn't getting anywhere near the FDA recommended temperature (-17.7 Celsius). I tweaked the code yet again to turn on the compressor if either compartment is over temperature, but this resulted in the fridge running much too cold -- it was close to freezing point in fact. I think this is probably a hardware problem with the freezer, and given its a beer fridge I've just written "this is not the freezer you are looking for" on the door and gone back to only monitoring the temperature of the fridge compartment. Here's a graph of my experiments with the freezer yesterday:

Another recent software tweak is a start up delay for the compressor in order to reduce the risk of back pressure damage to the compressor. Thanks Murray! This comes with yet another LED which indicates that the compressor is currently disabled.

I think this project is nearly ready to start showing people the code. The code is also getting big enough that cutting and pasting it into blog posts is starting to get annoying. So, here are a few links:

The code for the arduino. This should be self explanatory, except for saying that we're using a nuelectronics.com ethernet shield, not the standard one.
The visualization software. You want temperature.py to scrape the web server on the arduino and push stuff into a MySQL database, and then server.py is a simple python web server that provides the UI for visualization. The UI isn't very good at the moment, but I shall improve it soon.

Tags for this post: arduino()
Related posts: Beer fridge controller 0.3; Beer fridge controller 0.2; Beer fridge controller 0.1; Building a hygrometer with a HS1101; Arduino with the kids: Cricket Noise Door Bell; Coffee and beer in one glass; Phoenix for business; Thinking about arduino as a prototyping platform; An awesome weekend; Surprise for the day: Australian beer is cheaper in the US than in Australia

Comment

Buy Jupiter Short Stories

Fri, 01 Jan 2010 13:45:00 GMT

ISBN: 0575041994
Gollancz (1988), Edition: New Ed, Paperback, 224 pages
LibraryThing

This is another Asimov short story collection. The following stories appear in the book, although I have already read a couple as part of either the Robot short stories or the Nightfall collection of short stories.

To be honest these stories aren't Asimov's strongest. They entertaining, but they're not as amazing as some of his other stuff. I guess its hard to be a genius all the time.

The following stories appear in this collection:

	Buy Jupiter 1975	The Complete Robot 1982	Robot Dreams 1986
Darwinian Pool Room
Day of the Hunters
Shah Guido G.
Button, Button
The Monkey's Finger
Everest
The Pause
Let's Not
Each an Explorer
Blank!
Does a Bee Care?
Silly Asses
Buy Jupiter
A Statue for Father
Rain, Rain, Go Away
Founding Father
Exile to Hell
Key Item
The Proper Study
2430 A.D.
The Greatest Asset
Take a Match
Thiotimoline to the Stars
Light Verse

Tags for this post: book() Isaac_Asimov()
Related posts: Isaac Asimov's Robot Short Stories; Robot Visions; Isaac Asimov's Nightfall Short Stories; The Complete Robot; Nightfall (short stories); I, Robot; The Rest of the Robots; Robot Dreams; Foundation's Friends; Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Foundation Series; Caves of Steel; Isaac Asimov's Robot City: Perihelion; Galactic Dreams; Caliban Series; Isaac Asimov's Utopia; For Christmas; The Complete Hammer's Slammers Volume 1; Body Armor: 2000; Foundation and Empire; Isaac Asimov's Caliban

Comment

Books read in December 2009

Thu, 31 Dec 2009 21:59:00 GMT

Tags for this post: book() read()

Comment

Thinking about arduino as a prototyping platform

Tue, 29 Dec 2009 03:18:00 GMT

So, I've mentioned in earlier posts about arduino projects that I consider the arduino to be a prototyping platform, and a damn good one at that. Hack a day seems to think of it in similar terms. It was really Doug who got be thinking in this direction with the initial PCB design for the fridge controller project, which includes space for an Atmega CPU right on the PCB, thus eliminating the need for a relatively expensive arduino board to be permanently consumed.

So I started to wonder how hard it would be to build a simple arduino replacement board. It wouldn't need the complicated USB hardware, as you could program the Atmega on a full arduino board before installation. It would just need a time source, perhaps a LED, and a voltage regulator.

You can see in the picture above a version I quickly whipped up on a breadboard to prove this is possible. I didn't bother with a voltage regulator in this version, and the wires off on the right go off to a power source. This quickly turned into a PCB prototype board design, which has a voltage regulator, and exposes all of the arduino digital pins except for pin 5 (which the 16MHz crystal gets in the way of). Pin 5 could be made to happen pretty easily though...

I think this was about $10 worth of parts, including the Atmega CPU, which makes it a pretty attractive option compared with the real arduino development boards. I expect to be doing a lot of my future development on a full arduino board, and then moving the finished products off onto boards like this. I expect Matt's door bell will be exiled to one of these soon in order to free up a board. Doug and I have talked about doing a custom PCB layout which is similar, but that is yet to happen.

The best bit of this is I am really surprised by how easy it was. I'm not a hardware person, and it took only an hour or so with a schematic to come up with a working version. I'm much pleased.

Tags for this post: arduino()
Related posts: Beer fridge controller 0.1; Arduino with the kids: Cricket Noise Door Bell; The Beer Fridge saga continues; Beer fridge controller 0.3; Beer fridge controller 0.2; Building a hygrometer with a HS1101

Comment

Colony

Mon, 28 Dec 2009 22:35:00 GMT

ISBN: 0140289755
Penguin Books Ltd (2001), Paperback, 304 pages
LibraryThing

The Times is wrong about this book. Its not "cruel, cynical and very funny", its cruel and cynical for sure, but it lacks hope and is overall just depressing. I certainly didn't feel it was funny. Its strange, I loved the Red Dwarf series, and this book is very similar. I think the problem is that this book lacks all the hope and charm of the Red Dwarf books and TV show. Its a book entirely comprised of Rimmers, and that's hard to take.

Tags for this post: book() Rob_Grant()
Related posts: Better Than Life; Bill the Galactic Hero Series; Life, the Universe and Everything; The Restaurant at the End of the Universe; Body Armor: 2000; So Long, and Thanks For All the Fish; The Man in the Rubber Mask; Red Dwarf; Dragonsdawn; Last Human; Bill the Galactic Hero on the Planet of Bottled Brains; Mostly Harmless; Bill The Galactic Hero; Saturn's Children; The Hitchhikers Guide to the Galaxy; Bill the Galactic Hero: The Planet of the Robot Slaves

Comment

Arduino with the kids: Cricket Noise Door Bell

Mon, 28 Dec 2009 14:23:00 GMT

When I was a child, I had a doorbell at my bedroom door to ward off uninvited guests. My six year old, Matthew, has always been pretty enthused about building things, and so he wanted to give an electronics project a try. I thought this would be a good project to start out with for the kids, because its relatively simple, and there is a tangible result at the end (you press a button and something happens). Matt liked the idea. Because this project involved a fair bit of soldering, it turns out that Matt spent most of his time taking photos of the work, although we talked about what was happening at each step. I need to think harder about how to get him involved in the construction process -- I think that will be easier once the bread boarding stuff from ebay arrives.

The design is relatively simple. I took the sample debounce circuit (a button, 10k resistor) and software and ran that first. Then we put a peizo buzzer across pin 13 and ground. That meant that with the sample software we had both a light and a noise when you pressed the button. Unfortunately, the pin 13 LED also turns on when the arduino is booting, which means we got two beeps per boot, which was annoying. The peizo buzzer therefore got exiled to pin 12.

Finally, the screech from the peizo buzzer was getting a bit much, so I implemented a simple on-off cycle instead of it staying completely on. This produces a noise a bit more like a cricket's chirp, which is much less annoying. Finally, we put the whole thing in a case, and I think it looks pretty good. At the same time as putting in the case, we also added a battery power supply and power switch, as Matthew is now keen to take his door bell to school for show and tell.

The pictures in this post were mostly taken by Matthew. The source code (which includes a list of the wiring needed) is in my source repository.

Tags for this post: arduino()
Related posts: Beer fridge controller 0.1; The Beer Fridge saga continues; Beer fridge controller 0.3; Beer fridge controller 0.2; Thinking about arduino as a prototyping platform; Building a hygrometer with a HS1101

Comment

The Robot City, Robots and Aliens Series

Sat, 26 Dec 2009 14:19:00 GMT

This series follows on from the Robot City series set in Asimov's Foundation Universe but written by other authors. Overall that first series was weak, and I think the same is true for this series as well. There are a few here that are better than the others, but I'd only recommend this series for those who are obsessed with Foundation universe completeness.

1989: Isaac Asimov's Robot City: Robots and Aliens: Changeling by Stephen Leigh
1989: Isaac Asimov's Robot City: Robots and Aliens: Renegade by Cordell Scotten
1990: Isaac Asimov's Robot City: Robots and Aliens: Intruder by Robert Thurston
1990: Isaac Asimov's Robot City: Robots and Aliens: Alliance by Jerry Oltion
1990: Isaac Asimov's Robot City: Robots and Aliens: Maverick by Bruce Bethke
1990: Isaac Asimov's Robot City: Robots and Aliens: Humanity by Jerry Oltion

Tags for this post: book() Isaac_Asimov()
Related posts: Isaac Asimov's Robot City: Robots and Aliens: Humanity; Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Robots and Aliens: Alliance; Isaac Asimov's Robot City: Robots and Aliens: Intruder; Isaac Asimov's Robot City: Robots and Aliens: Changeling; Isaac Asimov's Robot City: Robots and Aliens: Renegade; Asimov's Mirage; Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Robot City: Perihelion; Isaac Asimov's Robot City: Prodigy; Isaac Asimov's Robot City: Odyssey; Isaac Asimov's Robot City: Refuge; Isaac Asimov's Robot City: Suspicion; The Robot City Series; Isaac Asimov's Robot Short Stories; Isaac Asimov's Foundation Series; Caves of Steel; Robot Visions; Isaac Asimov's Nightfall Short Stories; Caliban Series; Isaac Asimov's Utopia

Comment

Isaac Asimov's Robot City: Robots and Aliens: Humanity

Sat, 26 Dec 2009 14:14:00 GMT

ISBN: 0441373860
Ace (1990), Paperback, 189 pages
LibraryThing

This book is actually the best one of the two robot city series (Robot City and Robots and Aliens). Unfortunately I had to wade through 12 not very good books to find it, and its still not stellar. I'd recommend giving both these series a miss unless you're obsessed with completeness in Asimov's Robot universe.

Tags for this post: book() Jerry_Oltion()
Related posts: Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Robots and Aliens: Alliance; The Robot City, Robots and Aliens Series; Isaac Asimov's Robot City: Robots and Aliens: Intruder; Isaac Asimov's Robot City: Robots and Aliens: Changeling; Isaac Asimov's Robot City: Robots and Aliens: Renegade; Asimov's Mirage; Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Robot City: Perihelion; Isaac Asimov's Robot City: Prodigy; Isaac Asimov's Robot City: Odyssey; Isaac Asimov's Robot City: Refuge; Isaac Asimov's Robot City: Suspicion; The Robot City Series; Isaac Asimov's Robot Short Stories; Isaac Asimov's Foundation Series; Caves of Steel; Robot Visions; Isaac Asimov's Nightfall Short Stories; Caliban Series; Isaac Asimov's Utopia

Comment

Caliban Series

Thu, 24 Dec 2009 20:40:00 GMT

These are books written in Isaac Asimov's Foundation Universe by Roger MacBride Allen. These books are actually better than the Robot City Series, in that they at least explore issues that Asimov himself touched upon.

1997: Caliban
1998: Inferno
1999: Utopia

Tags for this post: book() Isaac_Asimov()
Related posts: Isaac Asimov's Utopia; Isaac Asimov's Caliban; Foundation's Triumph; Asimov's Mirage; Isaac Asimov's Inferno; Isaac Asimov's Robot Short Stories; Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Foundation Series; Caves of Steel; Isaac Asimov's Robot City: Perihelion; Robot Visions; Isaac Asimov's Nightfall Short Stories; For Christmas; The Complete Robot; Foundation and Empire; Nightfall (short stories); I, Robot; Isaac Asimov's Robot City: Robots and Aliens: Humanity; Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Prodigy; Robots and Empire

Comment

The Robot City Series

Thu, 24 Dec 2009 20:24:00 GMT

These are books written in Isaac Asimov's Foundation Universe by other authors. They don't progress the overall plot of the Foundation series, they just use some of the concepts and characters to tell similar stories. The books in the series are:

1987: Isaac Asimov's Robot City: Odyssey by Michael P Kube-McDowell
1987: Isaac Asimov's Robot City: Suspicion by Mike McQuay
1987: Isaac Asimov's Robot City: Cyborg by William F. Wu
1988: Isaac Asimov's Robot City: Prodigy by Arthur Byron Cover
1988: Isaac Asimov's Robot City: Refuge by Rob Chilson
1988: Isaac Asimov's Robot City: Perihelion by William F. Wu<

To be honest these books aren't fantastic and I wouldn't recommend them unless you're trying to get exhaustive coverage of Asimov's Foundation universe.

Tags for this post: book() Isaac_Asimov()
Related posts: Isaac Asimov's Robot City: Cyborg; Isaac Asimov's Robot City: Perihelion; Isaac Asimov's Robot City: Robots and Aliens: Humanity; Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Prodigy; Isaac Asimov's Robot City: Odyssey; Asimov's Mirage; Isaac Asimov's Robot City: Refuge; Isaac Asimov's Robot City: Robots and Aliens: Alliance; Isaac Asimov's Robot City: Suspicion; The Robot City, Robots and Aliens Series; Isaac Asimov's Robot City: Robots and Aliens: Intruder; Isaac Asimov's Robot City: Robots and Aliens: Changeling; Isaac Asimov's Robot City: Robots and Aliens: Renegade; Isaac Asimov's Robot Short Stories; Isaac Asimov's Foundation Series; Caves of Steel; Robot Visions; Isaac Asimov's Nightfall Short Stories; Caliban Series; Isaac Asimov's Utopia

Comment