See recent comments. RSS feed of all comments. Raw dump of all comments for research purposes.
ImageMagick book
MythTV book
|
 |
|
 |
|
Mon, 05 Jan 2009
|
|
|
|
|
 |
|
 |
|
I needed to compare a host key from something other than a known_hosts file with what paramiko reports as part of the SSH connection today. If you must know, the host keys for these machines are retrieved a XMLRPC API... It turned out to be a lot easier than I thought. Here's how I produced the host key entry as it appears in that API (as well as in the known_hosts file):
#!/usr/bin/python
# A host key calculation example for Paramiko.
# Args:
# 1: hostname
import base64
import os
import paramiko
import socket
import sys
# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))
# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
key = t.get_remote_server_key()
print '%s %s' %(key.get_name(),
base64.encodestring(key.__str__()).replace('\n', ''))
t.close()
sock.close()
Note that I could also have constructed a paramiko key object based on the output of the XMLRPC API and then compared those two objects, but I prefer the human readable strings.
Tags for this post: python( ) paramiko( )
posted at: 16:28 | path: /python/paramiko | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Wed, 10 Dec 2008
|
|
|
|
|
 |
|
 |
|
It seems that there is no way of killing a blocking thread in python? The standard way of implementing thread death seems to be to implement an exit() method on the class which is the thread, and then call that when you want the thread to die. However, if the run() method of the thread class is blocking when you call exit(), then the thread doesn't get killed. I can't find a way of killing these threads cleanly on Linux -- does anyone have any hints?
Tags for this post: python( )
posted at: 14:03 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Tue, 25 Nov 2008
|
|
|
|
|
 |
|
 |
|
I'm home sick with a cold today and got bored. I wanted to play with packet capture in python, and the documentation for pcapy is a little sparse. I therefore wrote this simple little sample script:
#!/usr/bin/python
# A simple example of how to use pcapy. This needs to be run as root.
import datetime
import gflags
import pcapy
import sys
FLAGS = gflags.FLAGS
gflags.DEFINE_string('i', 'eth1',
'The name of the interface to monitor')
def main(argv):
# Parse flags
try:
argv = FLAGS(argv)
except gflags.FlagsError, e:
print FLAGS
print 'Opening %s' % FLAGS.i
# Arguments here are:
# device
# snaplen (maximum number of bytes to capture _per_packet_)
# promiscious mode (1 for true)
# timeout (in milliseconds)
cap = pcapy.open_live(FLAGS.i, 100, 1, 0)
# Read packets -- header contains information about the data from pcap,
# payload is the actual packet as a string
(header, payload) = cap.next()
while header:
print ('%s: captured %d bytes, truncated to %d bytes'
%(datetime.datetime.now(), header.getlen(), header.getcaplen()))
(header, payload) = cap.next()
if __name__ == "__main__":
main(sys.argv)
Which outputs something like this:
2008-11-25 10:09:53.308310: captured 98 bytes, truncated to 98 bytes
2008-11-25 10:09:53.308336: captured 66 bytes, truncated to 66 bytes
2008-11-25 10:09:53.315028: captured 66 bytes, truncated to 66 bytes
2008-11-25 10:09:53.316520: captured 130 bytes, truncated to 100 bytes
2008-11-25 10:09:53.317030: captured 450 bytes, truncated to 100 bytes
2008-11-25 10:09:53.324414: captured 124 bytes, truncated to 100 bytes
2008-11-25 10:09:53.327770: captured 114 bytes, truncated to 100 bytes
2008-11-25 10:09:53.328001: captured 210 bytes, truncated to 100 bytes
Next step, decode me some headers!
Tags for this post: python( ) pcapy( )
posted at: 10:22 | path: /python/pcapy | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Tue, 11 Nov 2008
|
|
|
|
|
 |
|
 |
|
I re-factored some code today, and in the process managed to create a lock deadlock for myself. In the end it turned out to be an exception was being thrown when a lock was held, and adding a try / finally resolved the real underlying problem. However, in the process I ended up writing this little helper that I am sure will be useful in the future.
import gflags
import thread
import threading
import traceback
import logging
...
FLAGS = gflags.FLAGS
gflags.DEFINE_boolean('dumplocks', False,
'If true, dumps information about lock activity')
...
class LockHelper(object):
"""A wrapper which makes it easier to see what locks are doing."""
lock = thread.allocate_lock()
def acquire(self):
if FLAGS.dumplocks:
logging.info('%s acquiring lock' % threading.currentThread().getName())
for s in traceback.extract_stack():
logging.info(' Trace %s:%s [%s] %s' % s)
self.lock.acquire()
def release(self):
if FLAGS.dumplocks:
logging.info('%s releasing lock' % threading.currentThread().getName())
for s in traceback.extract_stack():
logging.info(' Trace %s:%s [%s] %s' % s)
self.lock.release()
Now I can just use this helper in the place of thread.allocate_lock() when I want to see what is happening with locking. It saved me a lot of staring at random code today.
Tags for this post: python( )
posted at: 15:46 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Sun, 05 Oct 2008
|
|
|
|
|
 |
|
 |
|
Tue, 16 Sep 2008
|
|
|
|
|
 |
|
 |
|
Wed, 03 Sep 2008
|
|
|
|
|
 |
|
 |
|
Tue, 05 Aug 2008
|
|
|
|
|
 |
|
 |
|
Thu, 10 Jul 2008
|
|
|
|
|
 |
|
 |
|
HTTP 1.1 implements chunking as a way of servers telling clients how much content is left for a given request, which enables you to send more than one piece of content in a given HTTP connection. Unfortunately for me, the site I was trying to access has a buggy chunking implementation, and that causes the somewhat fragile python urllib2 code to throw an exception:
Traceback (most recent call last):
File "./mythingie.py", line 55, in ?
xml = remote.readlines()
File "/usr/lib/python2.4/socket.py", line 382, in readlines
line = self.readline()
File "/usr/lib/python2.4/socket.py", line 332, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib/python2.4/httplib.py", line 460, in read
return self._read_chunked(amt)
File "/usr/lib/python2.4/httplib.py", line 499, in _read_chunked
chunk_left = int(line, 16)
ValueError: invalid literal for int():
I muttered about this earlier today, including finding the bug tracking the problem in pythonistan. However, finding the will not fix bug wasn't satisfying enough...
It turns out you can just have urllib2 lie to the server about what HTTP version it talks, and therefore turn off chunking. Here's my sample code for how to do that:
import httplib
import urllib2
class HTTP10Connection(httplib.HTTPConnection):
"""HTTP10Connection -- a HTTP connection which is forced to ask for HTTP
1.0
"""
_http_vsn_str = 'HTTP/1.0'
class HTTP10Handler(urllib2.HTTPHandler):
"""HTTP10Handler -- don't use HTTP 1.1"""
def http_open(self, req):
return self.do_open(HTTP10Connection, req)
// ...
request = urllib2.Request(feed)
request.add_header('User-Agent', 'mythingie')
opener = urllib2.build_opener(HTTP10Handler())
remote = opener.open(request)
content = remote.readlines()
remote.close()
I hereby declare myself Michael Still, bringer of the gross python hacks.
Tags for this post: python( )
posted at: 22:27 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Wed, 09 Jul 2008
|
|
|
|
|
 |
|
 |
|
I've always found python's Universal Feedparser to be a bit hard to work with when using feeds with XML namespaces. Specifically, if you don't care about the stuff in the namespaces then you're fine, but if you want that data it gets a lot harder.
In the past I've had to do some gross hacks. For example this gem is from the MythNetTV code:
# Modify the XML to work around namespace handling bugs in FeedParser
lines = []
re_mediacontent = re.compile('(.*)<media:content([^>]*)/ *>(.*)')
for line in xmllines:
m = re_mediacontent.match(line)
count = 1
while m:
line = '%s<media:wannabe%d>%s</media:wannabe%d>%s' %(m.group(1), count,
m.group(2),
count, m.group(3))
m = re_mediacontent.match(line)
count = count + 1
lines.append(line)
# Parse the modified XML
xml = ''.join(lines)
parser = feedparser.parse(xml)
Which is horrible, but works. This time around the problem is that I am having trouble getting to the gr:annotation tags in my Google reader shared items feed. How annoying.
In the case of the Google reader feed, the problem seems to be that the annotation is presented like this:
<gr:annotation><content type="html">Awesome. Canberra has needed
something better than buses between the towncenters for a while, and light rail
seems like a great way to do it. I much prefer trains to buses, and catch a
light rail service to work every day when I am in Mountain View.
</content><author gr:user-id="09387883873401903052"
gr:profile-id="114835605728492647856"><name>mikal</name>
</author></gr:annotation>
Feedparser can only handle simple elements (not elements that contain other elements). Therefore, this gross hack is required to get this to parse correctly:
simplify_re = re.compile('(.*)<gr:annotation>'
'<content type="html">(.*)</content>'
'<author .*><name>.*</name></author>'
'</gr:annotation>(.*)')
new_lines = []
for line in lines:
m = simplify_re.match(line)
if m:
new_lines.append('%s<gr:annotation>%s</gr:annotation>%s'
%(m.group(1), m.group(2), m.group(3)))
else:
new_lines.append(line)
d = feedparser.parse(''.join(new_lines))
Gross, and fragile, but working. This is cool, because it now means that I can apply more logic in the shared links that end up in my blather feed. I'm thinking of something along the lines of only shared links with an annotation will end up in that feed, and the blather entry will include the annotation. Or something like that.
Tags for this post: python( ) feedparser( )
posted at: 05:22 | path: /python/feedparser | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Tue, 01 May 2007
|
|
|
|
|
 |
|
 |
|
Hi. I have a list of the domain portion of URLs which looks a bit like this:
Whois lookup for fycnds.digitalpoimt.com
Whois lookup for wvgpzdea.digitalpoimt.com
Whois lookup for zhnsht.digitalpoimt.com
Whois lookup for frigo25.php5.cz
Whois lookup for handrovina.php5.cz
Whois lookup for blabota.php5.cz
Whois lookup for pctuzing.php5.cz
Whois lookup for viagraviagra.php5.cz
Whois lookup for poiu.php5.cz
Whois lookup for flasa.php5.cz
Whois lookup for yoy4.digitalpoimt.com
Whois lookup for hskly.digitalpoimt.com
Whois lookup for 2i0wjwbc.digitalpoimt.com
Whois lookup for harnhjc.digitalpoimt.com
Whois lookup for gqru.digitalpoimt.com
I need some code which determines which portion of these hostnames is a whois-able domain name. My problem is this doesn't seem all that simple to do -- some countries have a second layer of TLDs, and some do not.
Does anyone know of a python library, or failing that simple algorithm, which will do this for me?
(For those left wondering, I am trying to do some analysis of the spam I get on this blog, and for that I want to know if the whois information for a domain that left a suspect comment indicates anything suspicious.)
Tags for this post: python( )
posted at: 21:00 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Thu, 14 Dec 2006
|
|
|
|
|
 |
|
 |
|
Dear Lazy Web,
I have a need to be able to write to the MS Windows event log in Python. I must admit I don't know a lot about Python on Windows. Does anyone have a good short sample they would like to share?
Hugs and kisses,
Mikal
Tags for this post: python( )
posted at: 23:01 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Mon, 08 May 2006
|
|
|
|
|
 |
|
 |
|
Tue, 18 Apr 2006
|
|
|
|
|
 |
|
 |
|
Fri, 30 Dec 2005
|
|
|
|
|
 |
|
 |
|
My first python script involves doing some DNS lookups (for TXT records if that matters), and I am currently working through using the pydns module for this. Is this really the best DNS module to use for python though? For a start, it was last released in May 2002, and the documentation is somewhat sparse...
Tags for this post: python( )
posted at: 17:07 | path: /python | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Mon, 28 Nov 2005
|
|
|
|
|
 |
|
 |
|
I've just started working through Dive Into Python, so I don't really have an opinion of the book yet. I did notice that Example 2.1 produces different output on my machine than from the example...
The example says I should get:
server=mpilgrim;uid=sa;database=master;pwd=secret
I get:
pwd=secret;database=master;uid=sa;server=mpilgrim
It's interesting that this is exactly the reverse of what the book says I should get. I have no idea why, as I can't read Python yet, but there you go.
Tags for this post: python( ) diveintopython( )
posted at: 11:16 | path: /python/diveintopython | permanent link to this entry
|
|
|
|
|
 |
|
 |
|
Wed, 23 Nov 2005
|
|
|
|
|
 |
|
 |
|
Due to a personality crisis I've now decided it's the time in my life to learn Python. So, does anyone have good pointers to tutorials and so forth?
Tags for this post: python( )
posted at: 14:37 | path: /python | permanent link to this entry
|
|
|
|
|
|