stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

Executing a command with paramiko

Wed, 03 Sep 2008 15:11:00 GMT

I wanted to provide a simple example of how to execute a command with paramiko as well. This is quite similar to the scp example, but is nicer than executing a command in a shell because there isn't any requirement to do parsing to determine when the command has finished executing.

#!/usr/bin/python

# A simple command example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: command to run

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a cmd channel
cmd_channel = t.open_session()
cmd_channel.exec_command(sys.argv[3])

data = cmd_channel.recv(1024)
while data:
  sys.stdout.write(data)
  data = cmd_channel.recv(1024)

# Cleanup
cmd_channel.close()
t.close()
sock.close()

Tags for this post: python() paramiko()

Comment

Implementing SCP with paramiko

Wed, 03 Sep 2008 13:28:00 GMT

Regular readers will note that I've been interested in how scp works and paramiko for the last couple of days. There are previous examples of how to do scp with paramiko out there, but the code isn't all on one page, you have to read through the mail thread and work it out from there. I figured I might save someone some time (possibly me!) and note a complete example of scp with paramiko...

#!/usr/bin/python

# A simple scp example for Paramiko.
# Args:
#   1: hostname
#   2: username
#   3: local filename
#   4: remote filename

import getpass
import os
import paramiko
import socket
import sys

# Socket connection to remote host
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((sys.argv[1], 22))

# Build a SSH transport
t = paramiko.Transport(sock)
t.start_client()
t.auth_password(sys.argv[2], getpass.getpass('Password: '))

# Start a scp channel
scp_channel = t.open_session()
          
f = file(sys.argv[3], 'rb')
scp_channel.exec_command('scp -v -t %s\n'
                         % '/'.join(sys.argv[4].split('/')[:-1]))
scp_channel.send('C%s %d %s\n'
                 %(oct(os.stat(sys.argv[3]).st_mode)[-4:],
                   os.stat(sys.argv[3])[6],
                   sys.argv[4].split('/')[-1]))
scp_channel.sendall(f.read())

# Cleanup
f.close()
scp_channel.close()
t.close()
sock.close()

Tags for this post: python() paramiko()

Comment

Robot Visions

Wed, 03 Sep 2008 13:07:00 GMT

This was a pretty short read -- in fact I read it on the bus into work this morning. That's mainly because there are only three short stories in this book which aren't covered in one of Asimov's other robot short story collections. The three stories were good, but I am not sure they were worth owning the entire book for.

LibraryThing link for ISBN 0451450647

Tags for this post: book() Isaac_Asimov()

Comment

The Belgariad

Tue, 02 Sep 2008 19:36:00 GMT

The Belgariad is a fantasy series by David Eddings. There are five books in the series, although there is also a follow series called the Mallorean, as well as three tie in books. The series follows a small farm boy as he grows up, discovering along the way that his relatives are famous, he's important to the history of the world, and that he has to defeat an ancient evil. The five books in the main series are:

Year	Title
1982	Pawn of Prophecy
1982	Queen of Sorcery
1983	Magician's Gambit
1984	Castle of Wizardry
1984	Enchanters' End Game

Tags for this post: book() David_Eddings()

Comment

Enchanters End Game

Tue, 02 Sep 2008 19:29:00 GMT

This is the final book in the Belgariad. Its a good one, although knowing there is another series involving these characters makes it feel a little less final to me. I enjoyed it though.

LibraryThing link for ISBN 0345338715

Tags for this post: book() David_Eddings()

Comment

Chrome user agent

Tue, 02 Sep 2008 13:24:00 GMT

I am sure this will come in handy... The chrome user agent looks like this:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13

Tags for this post: chrome()

Comment

Books read in August 2008

Sun, 31 Aug 2008 11:38:00 GMT

Tags for this post: book() read()

Comment

TCP_DENIED/403 for cache accesses using squid

Sun, 31 Aug 2008 10:21:00 GMT

I just spent some time debugging why squid wouldn't work for localhost, but seemed to work for other clients on my network. To cut a long story short, Linux for some reason wasn't using 127.0.0.1 for traffic to localhost. It was instead using the public IP address for the machine, which didn't match either the localhost ACL or the local network ACL. I am sure there is some fancy reason that this is the case, but if you see this problem, then consider checking your localhost ACL.

Tags for this post: linux() squid()

Comment

MythNetTV release 5

Sat, 30 Aug 2008 15:11:00 GMT

New things in this release:

There is now a users mailing list at http://lists.stillhq.com/listinfo.cgi/mythnettv-stillhq.com
Moved to a public SVN server at http://www.stillhq.com/mythtv/mythnettv/svn/
Added the 'justone' syntax to the download command
Another try at using gflags. This means that all the command lines have changed slightly.
Moved the core of MythTV out of the user interface file.
Started writing unit tests
Changed user output code so that it doesn't insist on writing to stdout. You can now write to other file descriptors, which makes things like unit tests much easier to write.
Added video/msvideo to the enclosure whitelist
Added HTTP download progress information
Added a flag which turns off the prompts for markread (--noprompt)
Patches from Thomas Mashos
- Search ~/.mythtv/mysql.txt, /usr/share/mythtv/mysql.txt and /etc/mythtv/mysql.txt in that order for MySQL connection information
- A manpage
- setup.py
video.py now has a simple command line interface to let you query it
Fix update of inactive programs bug per http://ubuntuforums.org/showpost.php?p=5580005&postcount=4
Better DB error handling
Included a COPYING file with the right version of the GPL (it was missing before)
Fixed a bug where programs would be downloaded more than once (found with a unit test!)
Started raising exceptions instead of just sys.exit(1). This should make life easier for user interfaces in the future
Default to using storage groups for storing recordings before falling back to the RecordFilePrefix. This makes the behaviour: use a storage group named "MythNetTV" if it exists; use the default storage group if it exists; use the value of RecordFilePrefix.
Transcode avc1 videos, because some need it
Force ASCII encoding of title, subtitle, and all fields in the database to get around feeds which use unicode which python / MySQL can't store correctly
If there is only one attachment to an item, and its not in our whitelist of video formats, then warn the user that you're assuming its a video file and then add it to the todo list
Slight tweak to the signature of video.MythNetTvVideo.Transcode()
Fix buf in RepairMissingDates which caused it to consistently crash
Fix typo in date warning code
Better handling of videos where the length of the video cannot be determined by mplayer

Release 5 is by far the best tested release of MythNetTV yet, with both unit tests and several users working quite closely with me to resolve problems found in the wild. You can grab your copy here.

Tags for this post: mythtv() mythnettv()

Comment

Castle Of Wizardry

Fri, 29 Aug 2008 21:55:00 GMT

This is book four in the Belgariad, and was as good as the others. This one only took a day to read, but that was helped a lot by the two hours I spent commuting to and from San Francisco today. This book is a little different than the others because it starts just as the quest for the Orb ends. Yet it turns out that the overall prophecy that the Belgariad describes is still incomplete, so the story continues.

LibraryThing link for ISBN 0345300807

Tags for this post: book() David_Eddings()

Comment

Magician's Gambit

Thu, 28 Aug 2008 16:40:00 GMT

This is the third book in the Belgariad (Book 1 and Book 2). This book like the others was an enjoyable quick and easy read. I am starting to rethink my comments about these books being good for young readers -- it just occurred to me that a lot of people die in these books. They're all bad guys, and the violence isn't all that graphic, but I guess it might worry some parents.

LibraryThing link for ISBN 0345335457

Tags for this post: book() David_Eddings()

Comment

Queen of Sorcery

Tue, 26 Aug 2008 19:10:00 GMT

This is book two of the Belgariad (following on from Pawn of Prophecy). This book was a good quick read, and I think they'd be a good mid level reading book for a child. I liked it.

LibraryThing link for ISBN 0345335651

Tags for this post: book() David_Eddings()

Comment

Open Source video creation

Tue, 26 Aug 2008 10:53:00 GMT

For the upcoming MythNetTV release I am toying with the idea of asking the user if its ok to include a default subscription to an announcement video blog. This blog could be used to inform MythNetTV users of things like new releases, and important bug fixes if such things happen.

This raises the question -- if I wanted to mix creative commons licensed music with some still images (the announcements), what tool is the best to do that? Specifically open source tools please.

Tags for this post: blog()

Comment

First Family

Mon, 25 Aug 2008 09:59:00 GMT

I finished this book on the bus into work this morning (I had a pretty distracted weekend, and didn't get much reading done). This is the second book in the Amtrak Wars series, and takes place immediately after Cloud Warrior. The book feels like it is only half a book -- there is plot development such as learning more about Roz, meeting the First Family, and the Iron Masters, but there isn't much action. Basically I look back on the book and wonder what happened in it -- it would have been better to combine this with the next book and have a complete story in one.

LibraryThing link for ISBN 0671655671

Tags for this post: book() Patrick_Tilley()

Comment

Please help me test trunk

Sun, 24 Aug 2008 14:42:00 GMT

I'm preparing a new release of MythNetTV, and would like some help testing the code, as I have re-factored how the user interface works and there is some risk that I have broken things in the process. You can get the code like this:

svn co http://www.stillhq.com/mythtv/mythnettv/svn

That will create a directory called mythnettv, with a subdirectory named trunk, which is the latest development version of the code. You should probably create that directory somewhere where you don't mind a new directory being created.

The biggest change is that the command line syntax has changed slightly -- the dashes have been removed from the commands. Therefore, to update your list of feeds, you now use:

  mythnettv update

Instead of:

  mythnettv --update

And so on. This was done so that I could add "real" flags, which are used to change default values like where the database configuration is read from, as well as what the default location for the temporary data directory is.

"Real" flags which are currently supported are:

  --datadirdefault:  The default location of the data directory
    (default: 'data')

  --db_host:  The name of the host the MySQL database is on,
              don't define if you want to parse
              ~/.mythtv/mysql.txt instead
    (default: '')
  --db_name:  The name of the database which MythNetTV uses,
              don't define if you want to parse 
              ~/.mythtv/mysql.txt instead
    (default: '')
  --db_password:  The password for the database user, don't
              define if you want to parse 
              ~/.mythtv/mysql.txt instead
    (default: '')
  --db_user:  The name of the user to connect to the database
              with, don't define if you want to parse 
              ~/.mythtv/mysql.txt instead
    (default: '')

  --[no]commflag:  Run the mythcommflag command on new videos
    (default: 'true')

(These are the result of adding the gflags module back into the implementation).

I am hoping to release this version in the next few days, so if you find any bugs please send email to the mailing list.

Tags for this post: mythtv() mythnettv()

Comment

The Spy Within

Thu, 21 Aug 2008 21:43:00 GMT

LibraryThing, of which I am a member runs a program where members are shipped early copies of books for free, with the preference for them writing a review when they're done reading the book. The books are shipped by the publisher directly to the reviewers. This isn't that uncommon in the publishing industry -- both of my books have experienced a similar process, although less formal.

(In fact, any very early review of a book on a site light amazon.com should be viewed with a little bit of caution I suppose. These people probably got their review copies for free from the publisher.)

LibraryThing's implementation is a little different though, mainly because of the scale at which they hand out books, and the fact that the publishers don't appear to get any direct say in who gets the books. That means that there is less incentive to write a positive review, and that more people get access to early copies of new books. You can see a list of the books LibraryThing is currently handing out here.

The Spy Within is the true story of a senior CIA agent who turned out to also be a Chinese spy. Its the first book I've received through the early review program, so I am still learning the ropes and have sat on this book for a few weeks before actually reading it.

As I said earlier, this is the "true story" a senior Chinese spy within the CIA. However, it should be noted that large portions of the book are pure speculation -- inserted simply to make the story more readable. In addition, as with all such works, the book is based on a limited number of interviews, and is subject to the biases of those who provide source material.

This kind of book isn't really my thing, and I would read less one one non-fiction contemporary history book a year. However, I found this to be an engaging read, especially because the books manuscript flows much like a novel. However, the story simply wasn't that gripping (so, Larry Chin was a dick, I get it). Its hard for non-fiction to compete with fantasy for story lines I suppose.

LibraryThing link for ISBN 1586421484

Tags for this post: book() Tod_Hoffman()

Comment

SIGCOMM 2008

Thu, 21 Aug 2008 15:48:00 GMT

SIGCOMM 2008 had about 600 people attend, and felt a little like one of the early AUUG conferences. One of the things I am learning as I get older is that I am having increasing trouble sitting in hard hotel meeting chairs all day -- my back just wont let me. Surprisingly, I find most university lecture theatre seats more comfortable.

Below is a summary of the talks I thought are particularly good.

Don Towsley keynote

Don Towsley is the winner of the 2008 ACM SIGCOMM Award. His talk wasn't really a paper, he instead spoke about the need for models when performing Internet measurement research. If you don't have a theoretical model, then you don't have a way of verifying that your samples from the Internet are valid or not. This is an interesting point I hadn't considered, and which affects my own research work. My new problem is I'm not really sure how to produce models which have meaning to my current SMTP survey project. Additionally, Don asserts that PhD candidates shouldn't attempt to implement new applications. Instead they should work on enabling new applications. This observation is based on many years of supervising PhD candidates and their relative success.

A Case for Adapting Channel Width in Wireless Networks

Microsoft Research -- WiFi cards use a fixed bandwidth of 20 MHz. Using this quite large bandwidth for idle connections consumes a lot of power. This research proposes varying the size of the channel depending on what needs to be transferred -- 5MHz for an idle connection, up to 40 MHz for an active transfer. This does require a protocol for both sides of the connection to agree on a channel width. Sample implementation using the Zune peer to peer song sharing protocol.

Spamming Botnets: Signatures and Characteristics

Microsoft Research -- using URL extraction and a regular expression generator to find spam emails from botnets. Extract URLs from emails, track by domain over time and note bursty arrivals of such URLs. Send the bursty ones to a regular expression generator, which is then generalized to exclude victim specific IDs as well as domains, and then filter based on that. An interesting talk.

To Filter or to Authorize: Network-Layer DoS Defense Against Multimillion-node Botnets

University of California, Irvine -- DoS flooding attacks are a serious problem. The number of sources can be huge, as well as the packet count and bandwidth consumed. There are currently two schools of though on DoS protections -- filtering (anyone can send, and then filters are added when an attack occurs), and capability based systems (senders request permission before sending, and then use proof of that permission in each packet they send). This paper compares the two approaches.

BitTorrent is an Auction: Analyzing and Improving BitTorrent�s Incentives

University of Maryland -- the amount of research that is being conducted into peer to peer protocols, especially BitTorrent, is really interesting. This paper presented an alternative algorithm for how to select which blocks to offer for upload in return for the highest possible download rates. Specifically, it reframes BitTorrent as an auction system, in which leechers should be bidding the lowest possible in order to be selected as a download partner. This is implemented in BitTyrant (http://bittyrant.cs.washington.edu/). The rest of the talk focuses on strategies for gaming BitTorrent based on this observation. Very interesting. Implemented in a client at http://www.cs.umd.edu/projects/propshare/.

Network Discovery from Passive Measurements

UW Madison -- the traditional approach to mapping the Internet is to use active measurements. This paper proposes a solution using passive measurements. The underlying problem is that you have data in the form (source IP, destination IP, number of hops) and from that you need to determine which hops are in common for any given pair of readings.

Taming the Torrent: A Practical Approach to Reducing Cross-ISP Traffic in Peer-to-Peer Systems

Northwestern University -- this paper proposes piggy backing on CDN networks in order to determine which peer to peer clients are nearly to a p2p leecher. The network routing overlays produced by these networks can be used to select peers which can provide downloads more efficiently.

Tags for this post: conference() sigcomm2008()

Comment

Pawn of Prophecy

Tue, 19 Aug 2008 17:34:00 GMT

The Belgariad is a pretty formula fantasy epic, with striking similarities to things like stories of King Arthur. The series is written in a style which is very accessible to younger readers, which might explain why I loved these books as a kid. Its also a pretty "safe" story, in that nothing truly bad is allowed to happen to the main character, who is a child who grows up during the books. That might be why I loved these books so much as a child, and read them many times.

As an adult, this is a pretty easy read, and quite entertaining. I imagine its a lot like the Harry Potter series in its accessibility, although I haven't actually read Harry Potter as its not my kind of thing.

I quite like this book, and would recommend it to young readers interested in fantasy books.

LibraryThing link for ISBN 0552148075

Tags for this post: book() David_Eddings()

Comment

Hotel Max

Tue, 19 Aug 2008 16:25:00 GMT

When I was looking for a hotel to stay at for SIGCOMM 2008, I had trouble finding one in downtown Seattle which wasn't insanely expensive. In the end I picked Hotel Max because it was only moderately expensive, instead of insane like the Grant Hyatt. The hotel is interesting because apparently it was quite run down until a few years ago when it was done up. Now every room has its own unique art, and the halls and lobby are filled with different bits of art as well. Very hip.

The room itself is quite small by American standards, which means its about the same size as the room that I stayed in while staying in London a few years ago. The shower and bathroom are literally cupboards off a corridor, but the bed is a full size queen. I'm not surprised about that given the yelp.com reviews.

Given all I do in hotels is sleep and work on my laptop, I like this place. If I had the kids with me I would go insane however. Also be careful to get a room that faces Stewart Street. Mine faces an alley and I can hear the binging noise from the streetcar until about 10pm, and the air conditioning plant from the building next door for a while after that. I still slept ok though, so I guess people turned all that stuff off at some point during the night.

Update: I was wrong. The binging isn't the street car, its instead the beeping thingie that all underground carparks seem to have here to warn pedestrians that there is a car about to enter the sidewalk. Its very annoying.

Tags for this post: travel() usa() washington() seattle()

Comment

Cloud Warrior

Mon, 18 Aug 2008 17:47:00 GMT

I read this post-nuclear holocaust book as a kid and liked it, so I thought I'd give it a try again. Given most second hand science fiction books are $2, the barrier to entry is admittedly quite low as well. This book revolves around two groups -- the Amtrak Federation (a military society formed from survivors of those who crewed MX missile trains, a real system of roaming trains carrying nuclear missiles from the late 1980s and early 1990s -- the idea was that something which moved was harder for the Russians to find) and the "Mutes" (the civilan survivors of the holocaust, who are now mutated by radiation). Its a good book, although very different from the Asimov I've been reading recently. That's a good thing, because I think I need a bit of an Asimov break to be honest.

LibraryThing link for ISBN 0671559729

Tags for this post: book() Patrick_Tilley()

Comment