stillhq.com : Mikal, a geek from Canberra living in Silicon Valley

Python effective TLD library bug fix

Tue, 24 Nov 2009 13:57:00 PST

Some cool people commented on bugs in the etld library in the previous post about it. I've taken the opportunity to fix the bug, and a new release is now available at http://www.stillhq.com/python/etld/etld.py. If you've got specific examples of domains which either didn't work previously, or don't work now, let me know. I want to add unit tests to this code ASAP.

Tags for this post: python() etld()
Related posts: Python effective TLD library update; Python effective TLD library Comment

Python effective TLD library update

Sun, 01 Nov 2009 09:45:00 PST

The effective TLD library is now being used for a couple of projects of mine, but I've had some troubles with it being almost unusable slow. I ended up waking up this morning with the revelation that the problem is that I use regexps to match domain names, but the failure of a match occurs at the end of a string. That means that the FSA has to scan the entire string before it gets to decide that it isn't a match. That's expensive.

I ran some tests on tweaks to try and fix this. Without any changes, scanning 1,000 semi-random domain names took 6.941666 seconds. I then tweaked the implementation to reverse the strings it was scanning, and that halved the run time of the test to 3.212203 seconds. That's a big improvement, but still way too slow. The next thing I tried was then adding buckets of rules on top of those reverse matches.... In other words, the code now assumes that anything after the last dot is some for of TLD approximation, and only executes rules which also have that string after the last dot. This was a massive improvement, with 1,000 domains taking only 0.026120 seconds.

I've updated the code at http://www.stillhq.com/python/etld/etld.py.

Tags for this post: python() etld()
Related posts: Python effective TLD library bug fix; Python effective TLD library; Apple's Safari javascript implementation; Thinkpad x41 tablet PCMCIA IO; MySQL Tech Talks; Is there any way to access the match text in MySQL rlike selects? Comment

Python effective TLD library

Mon, 26 Oct 2009 06:42:00 PST

I had a need recently for a library which would take a host name and return the domain-specific portion of the name, and the effective TLD being used. "Effective TLD" is a term coined by the Mozilla project for something which acts like a TLD. For example, .com is a TLD and has domains allocated under it. However, .au is a TLD with no domains under it. The effective TLDs for the .au domain are things like .com.au and .edu.au. Whilst there are libraries for other languages, I couldn't find anything for python.

I therefore wrote one. Its very simple, and not optimal. For example, I could do most of the processing with a single regexp if python supported more than 100 match groups in a regexp, but it doesn't. I'm sure I'll end up revisiting this code sometime in the future. Additionally, the code ended up being much easier to write than I expected, mainly because the Mozilla project has gone to the trouble of building a list of rules to determine the effective TLD of a host name. This is awesome, because it saved me heaps and heaps of work.

The code is at http://www.stillhq.com/python/etld/etld.py if you're interested.

Tags for this post: python() etld()
Related posts: Python effective TLD library bug fix; Python effective TLD library update Comment

Calculating a SSH host key with paramiko

Mon, 05 Jan 2009 16:28:00 PST

I needed to compare a host key from something other than a known_hosts file with what paramiko reports as part of the SSH connection today. If you must know, the host keys for these machines are retrieved a XMLRPC API... It turned out to be a lot easier than I thought. Here's how I produced the host key entry as it appears in that API (as well as in the known_hosts file):

#!/usr/bin/python

# A host key calculation example for Paramiko.
# Args:
#   1: hostname

import base64
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
key = t.get_remote_server_key()

print '%s %s' %(key.get_name(),
                base64.encodestring(key.__str__()).replace('\n', ''))

t.close()
sock.close()

Note that I could also have constructed a paramiko key object based on the output of the XMLRPC API and then compared those two objects, but I prefer the human readable strings.

Tags for this post: python() paramiko()
Related posts: Dear Lazyweb: how do I check SSL keys for vulnerability? Comment

Killing a blocking thread in python?

Wed, 10 Dec 2008 14:03:00 PST

It seems that there is no way of killing a blocking thread in python? The standard way of implementing thread death seems to be to implement an exit() method on the class which is the thread, and then call that when you want the thread to die. However, if the run() method of the thread class is blocking when you call exit(), then the thread doesn't get killed. I can't find a way of killing these threads cleanly on Linux -- does anyone have any hints?

Tags for this post: python() Comment

Packet capture in python

Tue, 25 Nov 2008 10:22:00 PST

I'm home sick with a cold today and got bored. I wanted to play with packet capture in python, and the documentation for pcapy is a little sparse. I therefore wrote this simple little sample script:

#!/usr/bin/python

# A simple example of how to use pcapy. This needs to be run as root.

import datetime
import gflags
import pcapy
import sys

FLAGS = gflags.FLAGS
gflags.DEFINE_string('i', 'eth1',
                     'The name of the interface to monitor')


def main(argv):
  # Parse flags
  try:
    argv = FLAGS(argv)
  except gflags.FlagsError, e:
    print FLAGS

  print 'Opening %s' % FLAGS.i

  # Arguments here are:
  #   device
  #   snaplen (maximum number of bytes to capture _per_packet_)
  #   promiscious mode (1 for true)
  #   timeout (in milliseconds)
  cap = pcapy.open_live(FLAGS.i, 100, 1, 0)

  # Read packets -- header contains information about the data from pcap,
  # payload is the actual packet as a string
  (header, payload) = cap.next()
  while header:
    print ('%s: captured %d bytes, truncated to %d bytes'
           %(datetime.datetime.now(), header.getlen(), header.getcaplen()))

    (header, payload) = cap.next()


if __name__ == "__main__":
  main(sys.argv)

Which outputs something like this:

2008-11-25 10:09:53.308310: captured 98 bytes, truncated to 98 bytes
2008-11-25 10:09:53.308336: captured 66 bytes, truncated to 66 bytes
2008-11-25 10:09:53.315028: captured 66 bytes, truncated to 66 bytes
2008-11-25 10:09:53.316520: captured 130 bytes, truncated to 100 bytes
2008-11-25 10:09:53.317030: captured 450 bytes, truncated to 100 bytes
2008-11-25 10:09:53.324414: captured 124 bytes, truncated to 100 bytes
2008-11-25 10:09:53.327770: captured 114 bytes, truncated to 100 bytes
2008-11-25 10:09:53.328001: captured 210 bytes, truncated to 100 bytes

Next step, decode me some headers!

Tags for this post: python() pcapy()
Related posts: Dear lazy web: writing to the win32 event log in Python Comment

Finding locking deadlocks in python

Tue, 11 Nov 2008 15:46:00 PST

I re-factored some code today, and in the process managed to create a lock deadlock for myself. In the end it turned out to be an exception was being thrown when a lock was held, and adding a try / finally resolved the real underlying problem. However, in the process I ended up writing this little helper that I am sure will be useful in the future.

import gflags
import thread
import threading
import traceback
import logging

...

FLAGS = gflags.FLAGS
gflags.DEFINE_boolean('dumplocks', False,
                      'If true, dumps information about lock activity')
...

class LockHelper(object):
  """A wrapper which makes it easier to see what locks are doing."""

  lock = thread.allocate_lock()

  def acquire(self):
    if FLAGS.dumplocks:
      logging.info('%s acquiring lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.acquire()

  def release(self):
    if FLAGS.dumplocks:
      logging.info('%s releasing lock' % threading.currentThread().getName())
      for s in traceback.extract_stack():
        logging.info('  Trace %s:%s [%s] %s' % s)
    self.lock.release()

Now I can just use this helper in the place of thread.allocate_lock() when I want to see what is happening with locking. It saved me a lot of staring at random code today.

Tags for this post: python()
Related posts: Reducing the MySQL query lock timeout?; Interesting technique for finding leaks in code Comment

paramiko exec_command timeout

Sun, 05 Oct 2008 12:20:00 PST

I have a paramiko program which sshs to a large number of machines, and sometimes it hits a machine where Channel.exec_command() doesn't return. I know this is a problem with the remote machine, because the same thing happens when I try to ssh to the machine from the command line. However, I don't have any way of determining which machines are broken beforehand.

Paramiko doesn't support a timeout for exec_command(), so I am looking for a generic way of running a function call with a timeout. I can see sample code which does this using threads, but that's pretty ugly. I can't use SIGALARM because I am not running on the main thread.

Can anyone think of a better way of doing this?

Tags for this post: python() paramiko()
Related posts: Reducing the MySQL query lock timeout? Comment

Weird paramiko problem

Tue, 16 Sep 2008 11:41:00 PST

I had a strange paramiko problem the other day. Sometimes executing a command through a channel (via the exec_command() call) would result in an exit code being returned, but no stdout or stderr. This was for a command I was absolutely sure always returns output, and it wasn't consistent -- I'd run batches of commands and about 10% of them would fail, but not always on the same machine and not always at the same time. I spent ages looking at my code, and the code for the command running at the other end of the channel.

Then it occurred to me that this seemed a lot like a race condition. I started looking at the code for the paramiko Channel class, and ended up deciding that the answer was to check that the eof_received member variable was true before trying to close the channel.

It turns out this just works. I've my code running commands for a couple of days now and have had zero more instances of the "no output, but did exit" error. So, there you go. Its a shame that member variable doesn't have accessors and isn't documented though. I guess that makes my code a little more fragile than I would be happy with.

Tags for this post: python() paramiko()
Related posts: PNGtools 0.4; Why Debian?; Samba and MacOS X 10.4 (Tiger); Bad blog, bad bad blog; Mont 24 hour race; This is why I went to MythTV; ImageMagick bug?; All racehorses descended from 28 horses Comment

Executing a command with paramiko

Wed, 03 Sep 2008 15:11:00 PST

I wanted to provide a simple example of how to execute a command with paramiko as well. This is quite similar to the scp example, but is nicer than executing a command in a shell because there isn't any requirement to do parsing to determine when the command has finished executing.

#!/usr/bin/python

# A simple command example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: command to run

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a cmd channel
cmd_channel = t.open_session()
cmd_channel.exec_command(sys.argv[3])

data = cmd_channel.recv(1024)
while data:
  sys.stdout.write(data)
  data = cmd_channel.recv(1024)

# Cleanup
cmd_channel.close()
t.close()
sock.close()

Tags for this post: python() paramiko() Comment

Implementing SCP with paramiko

Wed, 03 Sep 2008 13:28:00 PST

Regular readers will note that I've been interested in how scp works and paramiko for the last couple of days. There are previous examples of how to do scp with paramiko out there, but the code isn't all on one page, you have to read through the mail thread and work it out from there. I figured I might save someone some time (possibly me!) and note a complete example of scp with paramiko...

#!/usr/bin/python

# A simple scp example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: local filename
#   4: remote filename

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a scp channel
scp_channel = t.open_session()
          
f = file(sys.argv[3], 'rb')
scp_channel.exec_command('scp -v -t %s\n'
                         % '/'.join(sys.argv[4].split('/')[:-1]))
scp_channel.send('C%s %d %s\n'
                 %(oct(os.stat(sys.argv[3]).st_mode)[-4:],
                   os.stat(sys.argv[3])[6],
                   sys.argv[4].split('/')[-1]))
scp_channel.sendall(f.read())

# Cleanup
f.close()
scp_channel.close()
t.close()
sock.close()

Tags for this post: python() paramiko() Comment

SSL, X509, ASN.1 and certificate validity dates

Tue, 05 Aug 2008 15:53:00 PST

I was curious about how SSL certificates store validity information (for example when a certificate expires), so I ended up reading the X509 specification (excitingly called "Internet X.509 Public Key Infrastructure Certificate and CRL Profile"), as well as the ASN.1 information for UTCTimes. This is all new to me, but I am sure lots of other people understand this.

In the end it wasn't too hard, and now I have hacked support for displaying certificate validity into Python's TLSlite. The point of this post is mainly so I can find that documentation again if I need it, although I'll put the TLSlite patch online as soon as I have had a chance to test it a little better.

Tags for this post: python() tlslite()
Related posts: Dear Lazyweb: how do I check SSL keys for vulnerability?; Twisted Python and Jabber SSL; Getting Google Talk working with PyXMPP Comment

Dealing with remote HTTP servers with buggy chunking implementations

Thu, 10 Jul 2008 22:27:00 PST

HTTP 1.1 implements chunking as a way of servers telling clients how much content is left for a given request, which enables you to send more than one piece of content in a given HTTP connection. Unfortunately for me, the site I was trying to access has a buggy chunking implementation, and that causes the somewhat fragile python urllib2 code to throw an exception:

Traceback (most recent call last):
  File "./mythingie.py", line 55, in ?
    xml = remote.readlines()
  File "/usr/lib/python2.4/socket.py", line 382, in readlines
    line = self.readline()
  File "/usr/lib/python2.4/socket.py", line 332, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.4/httplib.py", line 460, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.4/httplib.py", line 499, in _read_chunked
    chunk_left = int(line, 16)
ValueError: invalid literal for int():

I muttered about this earlier today, including finding the bug tracking the problem in pythonistan. However, finding the will not fix bug wasn't satisfying enough...

It turns out you can just have urllib2 lie to the server about what HTTP version it talks, and therefore turn off chunking. Here's my sample code for how to do that:

import httplib
import urllib2

class HTTP10Connection(httplib.HTTPConnection):
  """HTTP10Connection -- a HTTP connection which is forced to ask for HTTP
     1.0
  """

  _http_vsn_str = 'HTTP/1.0'


class HTTP10Handler(urllib2.HTTPHandler):
  """HTTP10Handler -- don't use HTTP 1.1"""

  def http_open(self, req):
    return self.do_open(HTTP10Connection, req)

// ...

  request = urllib2.Request(feed)
  request.add_header('User-Agent', 'mythingie')
  opener = urllib2.build_opener(HTTP10Handler())
  
  remote = opener.open(request)
  content = remote.readlines()
  remote.close()

I hereby declare myself Michael Still, bringer of the gross python hacks.

Tags for this post: python() Comment

Universal Feedparser and XML namespaces

Wed, 09 Jul 2008 05:22:00 PST

I've always found python's Universal Feedparser to be a bit hard to work with when using feeds with XML namespaces. Specifically, if you don't care about the stuff in the namespaces then you're fine, but if you want that data it gets a lot harder.

In the past I've had to do some gross hacks. For example this gem is from the MythNetTV code:

  # Modify the XML to work around namespace handling bugs in FeedParser
  lines = []
  re_mediacontent = re.compile('(.*)<media:content([^>]*)/ *>(.*)')

  for line in xmllines:
    m = re_mediacontent.match(line)
    count = 1
    while m:
      line = '%s<media:wannabe%d>%s</media:wannabe%d>%s' %(m.group(1), count,
                                                         m.group(2),
                                                         count, m.group(3))
      m = re_mediacontent.match(line)
      count = count + 1

    lines.append(line)

  # Parse the modified XML
  xml = ''.join(lines)
  parser = feedparser.parse(xml)

Which is horrible, but works. This time around the problem is that I am having trouble getting to the gr:annotation tags in my Google reader shared items feed. How annoying.

In the case of the Google reader feed, the problem seems to be that the annotation is presented like this:

<gr:annotation><content type="html">Awesome. Canberra has needed
something better than buses between the towncenters for a while, and light rail 
seems like a great way to do it. I much prefer trains to buses, and catch a 
light rail service to work every day when I am in Mountain View.
</content><author gr:user-id="09387883873401903052" 
gr:profile-id="114835605728492647856"><name>mikal</name>
</author></gr:annotation>

Feedparser can only handle simple elements (not elements that contain other elements). Therefore, this gross hack is required to get this to parse correctly:

  simplify_re = re.compile('(.*)<gr:annotation>'
                           '<content type="html">(.*)</content>'
                           '<author .*><name>.*</name></author>'
                           '</gr:annotation>(.*)')

  new_lines = []
  for line in lines:
    m = simplify_re.match(line)
    if m:
      new_lines.append('%s<gr:annotation>%s</gr:annotation>%s'
                       %(m.group(1), m.group(2), m.group(3)))
    else:
      new_lines.append(line)

  d = feedparser.parse(''.join(new_lines))

Gross, and fragile, but working. This is cool, because it now means that I can apply more logic in the shared links that end up in my blather feed. I'm thinking of something along the lines of only shared links with an annotation will end up in that feed, and the blather entry will include the annotation. Or something like that.

Tags for this post: python() feedparser()
Related posts: CVS digital cameras and handy cams Comment

Domain name lookup helper for python?

Tue, 01 May 2007 21:00:00 PST

Hi. I have a list of the domain portion of URLs which looks a bit like this:

Whois lookup for fycnds.digitalpoimt.com
Whois lookup for wvgpzdea.digitalpoimt.com
Whois lookup for zhnsht.digitalpoimt.com
Whois lookup for frigo25.php5.cz
Whois lookup for handrovina.php5.cz
Whois lookup for blabota.php5.cz
Whois lookup for pctuzing.php5.cz
Whois lookup for viagraviagra.php5.cz
Whois lookup for poiu.php5.cz
Whois lookup for flasa.php5.cz
Whois lookup for yoy4.digitalpoimt.com
Whois lookup for hskly.digitalpoimt.com
Whois lookup for 2i0wjwbc.digitalpoimt.com
Whois lookup for harnhjc.digitalpoimt.com
Whois lookup for gqru.digitalpoimt.com

I need some code which determines which portion of these hostnames is a whois-able domain name. My problem is this doesn't seem all that simple to do -- some countries have a second layer of TLDs, and some do not.

Does anyone know of a python library, or failing that simple algorithm, which will do this for me?

(For those left wondering, I am trying to do some analysis of the spam I get on this blog, and for that I want to know if the whois information for a domain that left a suspect comment indicates anything suspicious.)

Tags for this post: python()
Related posts: I think I've worked out the problem with the hotel network; Mikal, the massive domain squatter; Internet traffic; Measuring the popularity of SMTP server implementations on the Internet; Satellite internet at Walmart; Interesting paper: "YouTube Traffic Characterization: A View From the Edge"; The witty worm with Vern Paxson; Why does every man and his dog put man pages online?; Sensis Australian search; Methodology for my SMTP survey Comment

Dear lazy web: writing to the win32 event log in Python

Thu, 14 Dec 2006 23:01:00 PST

Dear Lazy Web,

I have a need to be able to write to the MS Windows event log in Python. I must admit I don't know a lot about Python on Windows. Does anyone have a good short sample they would like to share?

Hugs and kisses,
Mikal

Tags for this post: python()
Related posts: HP iPaq GPS FA256A; A side by side comparison of MythTV and Windows Media Center ; Gloat; Getting ASP.NET working on Windows XP Tablet PC edition; Nice touch; Packet capture in python; I feel a little vindicated; On freely available guide data; SQL Server is incompatible with Windows Vista?; Leon, get with the program; Windows Vista, now with nagging; Ok, where does one buy PCs in the US? Comment

Twisted conch

Mon, 08 May 2006 09:55:00 PST

It seems to me that every time I go to write some networking code in Python, the twisted guys have got there before me. Today's adventures are involving twisted conch, which seems very cool. The documentation is a bit patchy though.

Tags for this post: python() twisted()
Related posts: A ssh quickie; clusterssh Comment

Twisted Python and Jabber SSL

Tue, 18 Apr 2006 22:07:00 PST

Ok, so I thought it would be cool to be able to send Google Talk messages to my MythTV box. Can't be too hard to write a twisted python jabber client can it? Well, after an hour of surfing, I give up. I have the simple jabber client example, but it totally doesn't work with the Google servers, I suspect because it doesn't do SSL. I can see one of the twisted.words maintainers filing bugs against the xish stuff too, which I suspect means it's going to be a while.

A little bit disappointing me thinks.

Tags for this post: python() twisted()
Related posts: gtalkbot 1.1; mbot: new hotness in Google Talk bots; Getting Google Talk working with PyXMPP; gtalkbot 1.2; mbot: new hotness in Google Talk bots; Renaming mbot to gtalkbot; Blather, an open source Twitter work-alike for Blosxom and Google Talk; Worst timing evar!; A MythTV Jabber bot; MythTV talk at Google; MySQL Tech Talks; Seth Godin at Google; Slack talk at SLUG; Alternate queries on results pages making it easier for future evilness?; Cool people I have met at work; SSL, X509, ASN.1 and certificate validity dates; Seth Godin; Sydney Australia in Google Maps; MySQL Camp; Solar panel reflection effects in satellite imagery; My first keynote presentation Comment

Python DNS modules

Fri, 30 Dec 2005 17:07:00 PST

My first python script involves doing some DNS lookups (for TXT records if that matters), and I am currently working through using the pydns module for this. Is this really the best DNS module to use for python though? For a start, it was last released in May 2002, and the documentation is somewhat sparse...

Tags for this post: python()
Related posts: Compendium of TLD domain access agreements; Parked domains; Talk about a support life cycle...; What's happening with frozenchicken.com? Comment

Example 2.1 from Dive Into Python

Mon, 28 Nov 2005 11:16:00 PST

I've just started working through Dive Into Python, so I don't really have an opinion of the book yet. I did notice that Example 2.1 produces different output on my machine than from the example...

The example says I should get:

server=mpilgrim;uid=sa;database=master;pwd=secret

I get:

pwd=secret;database=master;uid=sa;server=mpilgrim

It's interesting that this is exactly the reverse of what the book says I should get. I have no idea why, as I can't read Python yet, but there you go.

Tags for this post: python() diveintopython() Comment