stillhq.com : Mikal, a geek from Canberra living in Silicon Valley

Managing MySQL the Slack Way: How Google Deploys New MySQL Servers

Sat, 31 Mar 2007 04:47:00 GMT

I'll be presenting about Slack (the open sourced tool kit we use for deployment software configuration) at the MySQL user's conference in Santa Clara in late April. The talk will focus on the interesting aspects of Slack as it relates to MySQL and should be fun. A DBA mate of mine is gonna present with me, so it should be a barrel of laughs.

Tags for this post: mysql() conference() mysqluc2007() mysql() presentation()

Comment on this post

MySQL scaling: query snipers

Wed, 07 Mar 2007 05:03:00 GMT

I've been wanting a query sniper for a while for MySQL (some people seem to call them "query killers"). They're basically programs which inspect the MySQL "show processlist" command and decide to kill some processes if they violate specified rules. I have a prototype of one in python, and couldn't find any other implementations until tonight when I came across Querybane, which is what Wikipedia use to do exactly the same thing.

Now, I need to think further about if querybane does what I want, especially given I can't find the source code. Perhaps it's hidden in the media wiki code somewhere? The only code for a project named "servmon" (it's parent project) that I can find is four years old and in PHP.

(Oh, by the way, the Wikipedia MySQL docs are really interesting, for example this page has some nice practical advice about replication log corruption. I wish I had found this before now for two reasons: it's nice to see how other people do this stuff; and there is some stuff I can totally steal from here.)

So, does anyone know where I can find recent querybane code? Alternatively, what do other people use as a query sniper? Is it worth open sourcing mine?

Update: Apparently what I needed to do was sleep on it, and use Google Code Search. The querybane code appears to be at http://svn.wikimedia.org/svnroot/mediawiki/trunk/servmon/. In the end it's not going to work for me, as it seems to be really based around a world in which you use IRC to keep a track of the state of your servers, which isn't something I want to do.

Tags for this post: mysql() query() killer() sniper() admin()

Comment on this post

They all use MySQL...

Sun, 31 Dec 2006 16:09:00 GMT

I was walking down Mountain View's Castro Street this afternoon, and noticed that meebo is advertising for developers and system admins. Interestingly, they seem to match the design pattern used by pretty much every web 2.0 company I have seen around here (except MySpace): linux, MySQL, and Ajax. So, there you go.

Tags for this post: mysql() web() 2.0()

Comment on this post

MySQL Camp

Thu, 23 Nov 2006 10:43:00 GMT

Kynan and I came along to the MySQL camp, and thru a quirk of fate pretty much ended up running it (the person who was meant to be running it got injured on the first day and had to go off to hospital). In return we wrote the Google Code blog post about the event. Pretty cool, huh?

Tags for this post: mysql() user() camp() google() code() blog()

Comment on this post

Greg likes MySQL cluster, oh and Stewart's talk

Thu, 11 May 2006 15:01:00 GMT

Greg Linden (Google watcher, search engine dude, ex-Amazon) saw Stewart Smith's talk on Google Video about MySQL cluster, and likes the idea. That's cool.

Time for more content on Google Video?

Tags for this post: mysql() google() video() cluster()

Comment on this post

MySQL Tech Talks

Sat, 06 May 2006 03:06:00 GMT

Three intrepid MySQLers came to Google after the user conference to give internal tech talks. They were kind enough to agree to us hosting them for other people to see. The first two are up, so I'll mention those now, and put a link to the last one when it's available...

Click on the thumbnail to be taken to the video.

	Jay Pipes is a co-author of the recently published Pro MySQL (Apress, 2005), which covers all of the newest MySQL 5 features, as well as in-depth discussion and analysis of the MySQL server architecture, storage engines, transaction processing, benchmarking, and advanced SQL scenarios. You can also see his name on articles appearing in Linux Magazine and can read more articles about MySQL at his website. Jay Pipes is MySQL's Community Relations Manager for North America. Learn where to best focus your attention when tuning the performance of your applications and database servers, and how to effectively find the "low hanging fruit" on the tree of bottlenecks. It's not rocket science, but with a bit of acquired skill and experience, and of course good habits, you too can do this magic!
	Timour Katchaounov The first part of this talk describes the main principles behind MySQL's query optimiser and execution engine, how the optimiser transforms queries into executable query plans, what these plans look like, and how they are executed. The second part of the talk describes the major improvements in the query engine of MySQL 5.0, and how these improvements can benefit the users of MySQL 5.0. The "greedy" optimiser reduces compilation time for big queries with orders of magnitude. The "index merge" access method provides a way to use more than one index for the same query. For faster plan execution and to allow better join orders, the 5.0 optimiser transforms most outer joins into inner joins. The outer joins that cannot be transformed into inner ones are executed in a pipeline manner, so that no intermediate results need to be materialised. Finally, some GROUP BY and DISTINCT queries can be executed much faster thanks to "loose index scan" technique that reads only a fraction of an index. The talk concludes with the near-future plans for new features coming in the next versions of MySQL.
	Stewart Smith works for MySQL AB as a software engineer working on MySQL Cluster. He is an active member of the free and open source software community, especially in Australia. Although Australian, he does not dress like Steve Irwin—although if he wrestled crocodiles he probably would. He is a fan of great coffee, great beer, and is currently 39,000 feet above sea level. part 1 - Introduction to MySQL Cluster The NDB storage engine (MySQL Cluster) is a high-availability storage engine for MySQL. It provides synchronous replication between storage nodes and many mysql servers having a consistent view of the database. In 4.1 and 5.0 it's a main memory database, but in 5.1 non-indexed attributes can be stored on disk. NDB also provides a lot of determinism in system resource usage. I'll talk a bit about that. part 2 - new features in 5.1 including cluster to cluster replication, disk based data and a bunch of other things. anybody that is attending the mysql users conference may find this eerily familiar. I can also talk about latest-and-totally-greatest developments and future stuff we're working on. i can also take questions and constructive abuse :)

You can see a complete list of the MySQL tech talks at Google here.

Update: added Stewart's talk now that it is online.

Tags for this post: mysql() google() tech() talk() performance() optimiser() sql()

Comment on this post

Thoughts on the first day of the MySQL user's conference

Wed, 26 Apr 2006 07:45:00 GMT

So, I attended the first day of the MySQL user's conference yesterday, which was the tutorial day. Overall I was fairly impressed. Registration was easy, the actual rooms presentations are given in are comfortable, the PA system seemed to work after some initial problems in the morning tutorial I attended.

The conference center seems to be big on retirees hanging around, which I thought was weird. Each room comes with a little old lady, whose job appears to be to read a fiction novel at the door. I really have no idea what else they were achieving. They seemed to be having fun though. I did find it a bit odd that the only drinks provided by the catering staff during the day were acidic, and most of them caffinated. For example, we had choices between coffee, tea, soda water, coke, diet coke, pepsi and diet pepsi. Some fruit juice or even plain water would have been a nice change by the end of the day.

The food was good, unless you're a vegan like Stewart at which time the catering staff looked confused and had to go off and get him something special (which didn't look all that special to me when it came ten minutes later). If that happens again today, then I might try to talk him into just going to a restaurant for lunch with me.

The first tutorial I attended was about MySQL cluster, and to be honest I kinda lost interest at the point where I found out that the database currently has to store all of it's data in RAM. That basically renders it unusable for anyone with a large-ish database. The speaker did a good job in adverse circumstances such as the PA system initially not working, and trying to speak in a language which is not his native tongue in a room which was very wide. I do think that he needed to clamp down harder on the irrelevant, repetitive and self serving questions though. It seemed to me that there were a fair few people in the room who viewed the conference as their chance to get someone else to fix their problems at work. Whilst it's fair to go to a conference to ask advice on problems you're having, I do think that some research prior to attending (or even using the wireless during the conference) is a good idea, and that you shouldn't interrupt the flow of a talk to ask your tangential questions.

The second tutorial was by a Yahoo, and was on MySQL replication. It was excellent -- well prepared, relevant to what I am caring about at the moment, coherent, and flowed well. Jeremy and Eric did a great job (slides from the talk). I hung around at the end to try to offer them beer, but they were swamped by fans who didn't appear to want to leave. I'll have to chase them down today sometime to make the offer.

Overall, not a perfect start to the conference, but I thought the day was very useful. A good first day.

Tags for this post: mysql() mysqluc() conference() open() source()

Comment on this post

Thoughts on the first day of the MySQL user's conference

Wed, 26 Apr 2006 07:45:00 GMT

So, I attended the first day of the MySQL user's conference yesterday, which was the tutorial day. Overall I was fairly impressed. Registration was easy, the actual rooms presentations are given in are comfortable, the PA system seemed to work after some initial problems in the morning tutorial I attended.

The conference center seems to be big on retirees hanging around, which I thought was weird. Each room comes with a little old lady, whose job appears to be to read a fiction novel at the door. I really have no idea what else they were achieving. They seemed to be having fun though. I did find it a bit odd that the only drinks provided by the catering staff during the day were acidic, and most of them caffinated. For example, we had choices between coffee, tea, soda water, coke, diet coke, pepsi and diet pepsi. Some fruit juice or even plain water would have been a nice change by the end of the day.

The food was good, unless you're a vegan like Stewart at which time the catering staff looked confused and had to go off and get him something special (which didn't look all that special to me when it came ten minutes later). If that happens again today, then I might try to talk him into just going to a restaurant for lunch with me.

The first tutorial I attended was about MySQL cluster, and to be honest I kinda lost interest at the point where I found out that the database currently has to store all of it's data in RAM. That basically renders it unusable for anyone with a large-ish database. The speaker did a good job in adverse circumstances such as the PA system initially not working, and trying to speak in a language which is not his native tongue in a room which was very wide. I do think that he needed to clamp down harder on the irrelevant, repetitive and self serving questions though. It seemed to me that there were a fair few people in the room who viewed the conference as their chance to get someone else to fix their problems at work. Whilst it's fair to go to a conference to ask advice on problems you're having, I do think that some research prior to attending (or even using the wireless during the conference) is a good idea, and that you shouldn't interrupt the flow of a talk to ask your tangential questions.

The second tutorial was by a Yahoo, and was on MySQL replication. It was excellent -- well prepared, relevant to what I am caring about at the moment, coherent, and flowed well. Jeremy and Eric did a great job (slides from the talk). I hung around at the end to try to offer them beer, but they were swamped by fans who didn't appear to want to leave. I'll have to chase them down today sometime to make the offer.

Overall, not a perfect start to the conference, but I thought the day was very useful. A good first day.

Tags for this post: mysql() conference() mysqluc2006() conference() open() source()

Comment on this post

Links from Rasmus' PHP talk

Wed, 26 Apr 2006 07:09:00 GMT

The talk, and the demo are both very cool.

Tags for this post: mysql() conference() mysqluc2006() php() open() source() conference()

Comment on this post

Links from Rasmus' PHP talk

Wed, 26 Apr 2006 07:09:00 GMT

The talk, and the demo are both very cool.

Tags for this post: mysql() mysqluc() php() open() source() conference()

Comment on this post

I won a radio shark and headphones!

Wed, 26 Apr 2006 03:38:00 GMT

All I had to do was fill in a speaker evaluation form from yesterday, and now I've won a radio shark and headphones. Nice.

Tags for this post: mysql() conference() mysqluc2006() conference() open() source() massive() winnings()

Comment on this post

I won a radio shark and headphones!

Wed, 26 Apr 2006 03:38:00 GMT

All I had to do was fill in a speaker evaluation form from yesterday, and now I've won a radio shark and headphones. Nice.

Tags for this post: mysql() mysqluc() conference() open() source() massive() winnings()

Comment on this post

MySQL Workbench

Wed, 26 Apr 2006 02:11:00 GMT

Was just introduced in the keynote and looks kinda cool. It isn't packaged in Ubuntu yet, which is a shame...

Tags for this post: mysql() mysqluc() workbench() sql() ubuntu() linux()

Comment on this post

MySQL Workbench

Wed, 26 Apr 2006 02:11:00 GMT

Was just introduced in the keynote and looks kinda cool. It isn't packaged in Ubuntu yet, which is a shame...

Tags for this post: mysql() conference() mysqluc2006() workbench() sql() ubuntu() linux()

Comment on this post

MySQL cluster stores in RAM!

Tue, 25 Apr 2006 02:03:00 GMT

Argh! MySQL cluster currently stores in RAM only. That means if I have 100 machines each with 4 gigabytes of RAM, then I can store 200 gigabytes of data! Hurrah! That's at least what I learnt in the tutorial, feel free to correct me if I am wrong.

Tags for this post: mysql() conference() mysqluc2006() cluster() conference() tutorial()

Comment on this post

MySQL cluster stores in RAM!

Tue, 25 Apr 2006 02:03:00 GMT

Argh! MySQL cluster currently stores in RAM only. That means if I have 100 machines each with 4 gigabytes of RAM, then I can store 200 gigabytes of data! Hurrah! That's at least what I learnt in the tutorial, feel free to correct me if I am wrong.

Tags for this post: mysql() mysqluc() cluster() conference() tutorial()

Comment on this post

Conference Wireless not working yet?

Tue, 25 Apr 2006 01:28:00 GMT

So, there are two access points with the ESSID "MySQL-Hyatt" that I can see from my coffee drinking spot here in the foyer. iwlist tells me that they have encryption off, which matches what the conference handbook says should be the case. No DHCP leases are being offered though. I guess that means that the conference network hasn't been commissioned yet. Oh well.

Update: Oh, it works in the conference rooms, which have a different ESSID.

Tags for this post: mysql() conference() mysqluc2006() conference() open() source() wireless() network()

Comment on this post

Conference Wireless not working yet?

Tue, 25 Apr 2006 01:28:00 GMT

So, there are two access points with the ESSID "MySQL-Hyatt" that I can see from my coffee drinking spot here in the foyer. iwlist tells me that they have encryption off, which matches what the conference handbook says should be the case. No DHCP leases are being offered though. I guess that means that the conference network hasn't been commissioned yet. Oh well.

Update: Oh, it works in the conference rooms, which have a different ESSID.

Tags for this post: mysql() mysqluc() conference() open() source() wireless() network()

Comment on this post

Off to the MySQL tutorials

Mon, 24 Apr 2006 23:40:00 GMT

Up at 5:50am, dressed by 6:30am, and leaving the house by 7:10am to get on the VTA light rail by 7:30am. All because the tutorials start at 8:30am, and I want to catch the train instead of using up fuel...

Tags for this post: mysql() mysqluc() train()

Comment on this post

Off to the MySQL tutorials

Mon, 24 Apr 2006 23:40:00 GMT