Bimport Throtz

Nine hosts in %SystemRoot%\System32\Drivers\etc\hosts

2011-12-22T07:30:00.000-08:00

If you are running a web server locally for development on your Windows XP/Vista/7 system and you have set up (say) local.yet-another-site.com locally as a development site for www.yet-another-site.com for the 9th time, you might scratch your head and wonder why DNS resolution isn't working.

I've not found Google forth-coming on this matter, but the problem is that a maximum of nine hosts can be listed next to an IP address in %SystemRoot%\System32\Drivers\etc\hosts, and that means that with the following:

127.0.0.1    localhost h1 h2 h3 h4 h5 h6 h7 h8 h9 h10

...you are able to PING localhost, h1, h2, .. h8 but h9 and h10 do not resolve.

Don't go soul-searching to identify your top 10 and shuffle your list of development hosts, though. Happily you can have this:

127.0.0.1    localhost h1 h2 h3 h4 h5 h6 h7 h8 
127.0.0.1    h9 h10

Yes, it is a bit pathetic, and the Darwinistas of the Mac world are laughing at Windows users once again... but this works.

Overcoming a Network Filter limit on a Windows 7 DEV box

2011-11-02T04:52:00.000-07:00

In the past, my development set-up has typically consisted of a main development machine with my usual development tools and a Linux puppy sitting beside it. The puppy is there to act as a server for my code. However, the might of my DEV box now is such that I really ought to be making better use out of virtualisation and use some of the redundant CPU power of my DEV box to run the server too.

I find Oracle's VM VirtualBox nice to work with. Microsoft's Virtual PC isn't great for Linux, and you can even run a 64 bit server in VirtualBox.

I run server software in my puppy. I want to run clients from the host PC, but I'd like to be able to run other clients on my LAN too. To achieve that, I need to run bridged networking, with a static IP address assigned to my virtual server; it makes no sense to run DCHP and then have to mess around with HOSTS files in the various clients. I can write my service code in Windows, have it run in a Linux server, and then test it with various devices on my LAN.

I was surprised to run into a "Filters currently installed on the system have reached the limit" error when trying to install the bridged adapter filter. I was not alone. I needed to install the VirtualBox Bridged Networking driver, but I'd hit a limit, reached because of the necessity of running a Cisco VPN client amongst other bits and pieces for development. Fortunately there is a soft limit set to 8 for MaxNumfilters in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network, but this can be increased to the hard limit, which is 14 (or 0xe in hexadecimal) in Windows 7 via RegEdit. With MaxNumfilters set to its 14 hard limit, I can power down my old puppy and virtualise.

Boost 1.47 on Ubuntu Oneiric

2011-10-23T09:08:00.000-07:00

Why install 1.47?

[K]Ubuntu Oneiric comes with Boost 1.46 rather than 1.47. If you want to work with boost_asio, that's a nuisance, because the release documentation assumes you are already on 1.47. I avoid changing my set-up from the norm, if I can help it, but this upgrade/installation is a necessary evil.

What's the recommended quick upgrade approach?

This is a bum steer. Don't start typing.

The Personal Package Archives (PPAs) include a repository called PurpleKarrot, which you can add to your repositories as follows:

sudo add-apt-repository ppa:purplekarrot/ppa
sudo apt-get update

To fetch the latest packages you ought to be able to do the following:

sudo apt-get upgrade

However, on Kubuntu you see the message 'The following packages have been kept back', and sure enough none of the 1.47 packages are installed. I've not tries this with Ubuntu. It is conceivable that KDE depends on boost, and that's the cause of the conflict.

Don't be naive, like me, and assume that ~~to upgrade your Boost libraries you need to do a dist-upgrade:~~

sudo apt-get dist-upgrade

~~This upgrades all packages which have newer ones available and satisfies~~
~~dependencies.~~

You'll get a ton of dependency errors, as indicated in a small print comment here. Alarming though that is, removing the repository from /etc/apt/sources.d is quite painless, and an apt update shows the system is healthy.

So what's the actual approach we need to make?

It looks like we need to use the old Icelandic supermarket Bjam to build the libraries with a 1.47 Boost download, and keep the installation separate from the system. Bearing in mind the system has 1.46 installed, we need to be sure to have a complete installation to avoid pulling in 1.46 artifacts from the system.

The download takes a while and the build takes a while. Best to make tea during the download and do some gardening during the build. I created ~/lib/boost_1_47_0 in my home directory, which left me needing to set up CMakeLists.txt as follows:

set (BOOST_DIR "$ENV{HOME}/lib/boost_1_47_0")
set (BOOST_LIBDIR "${BOOST_DIR}/stage/lib")
include_directories("${BOOST_DIR}")

link_directories("${BOOST_LIBDIR}")set (EXTRA_LIBS ${EXTRA_LIBS} boost_system boost_filesystem boost_thread pthread)
add_executable(XXXXX main.cpp)
target_link_libraries(XXXXX ${EXTRA_LIBS})

This works nicely for me, allowing my project to use 1.47 and doesn't conflict with the 1.46 system libraries.

KDevelop Impressions

2011-10-23T07:30:00.000-07:00

Where am I coming from?

I occasionally try shaking things up in my development environment, to check whether I'm not missing out on things, clinging onto Luddite command-line ways. In that spirit I decided that it was time to look at KDE's KDevelop, which is now mature-looking in KDevelop4, and see what it feels like creating a new C++ project in that environment.

What I wanted to do was to set up a project that allows me to breath life back into a >10 year old C/C++ project, which would benefit from having C++0x goodness applied to it, and being recoded from the ground up. My existing project consists of two Makefiles (GNU/UN*X and Win32), source code, assets and no project files, because it wasn't built in an IDE. I was attracted to KDevelop, because it embraces cross-platform compilation, and thought that maybe I'd benefit from whetever we are supposed to call intellisense in the Open Source world.

However, one of things that puts me off IDEs, is that you often find the tail wagging the dog, and sure enough there were some new practices forced on me that I wouldn't have chosen otherwise.

Version Control

The first of these was version control.

I was pleased to see that CVS was available and a version control option, and I thought that I could use my familiar repository for the experiment, CVS preview version 1.12.9. After some floundering, I found that KDevelop wasn't going to play ball with my repository; it uses fully-qualified client-side paths, which chokes my version of CVS.

I didn't really want to look into upgrading/downgrading CVS on the server and also didn't want to dig too deep into KDevelop to see if I could get it to use relative paths. I am a great believer in going with the flow, and I use Subversion and Git for my main client, so I thought there would be safety in numbers setting up Subversion locally.

Setting up Subversion on a Linux box is of course a snap, and initialising a respository locally even more so. This was familiar territory. Making it play ball with the KDevelop IDE was less smooth.

My first surprise was that creating a new project with Subversion version control, expects the repository module to exist already. Using the console is comfortable for me, but I expected to be able to create a repository module, when creating the project itself in KDevelop. It looks like you were able to do this with KDevelop3, but KDevelop4 (version 4.7.1) only appears to let you create a project by checking out an existing module. So I created a directory at the command line and imported it into Subversion and checked out that module to get me started.

CMake

I've been aware of CMake for some time, but hadn't considered it for my own projects.

I was interested to see that CMakeLists.txt (CMake's equivalent of a Makefile) sits at the heart of the KDevelop project. Along with the minimal MyProject.kdev4 project file, I was pleased to see that appears to be all that KDevelop needs for the project. Human-readable text files, which are brief. Very nice.

It made sense to get to grips with CMake outside the IDE, so that I could focus on CMake concerns without stumbling on IDE issues, and that's what I did following a basic tutorial.

I am very pleased with CMake. I wish I'd started using it a long time ago.

Linkage

This isn't really relevant to KDevelop, but when you try something new other new things often get dragged in, unexpected.

To get to work with CMake, I knew that I would need to link the static libraries, so I pulled in some basic experimental threads code for boost_asio, and saw loads of linkage errors. I had recently run a distribution upgrade on my laptop to Oneiric, and thought the distribution libraries must be out of kilter.

I was wrong. The library versions set up by Aptitude were fine and CMake had been fine. My only error in CMake was that I needed to add boost_system to the libraries, which had not been the case in the previous Boost libraries.

However, being wary of the unfamiliar, I dropped out of CMake and down into a command-line to invoke the following:

g++ -I. -lboost_system -lboost_filesystem -lboost_thread main.cpp

The order is an old bad habit, which I have been getting away with. There was nothing wrong with the library versions I was pulling in, my command line error was linking them before the main.o file, which has the external requirements, which they needed to satisfy.

For linkage to work I needed:

g++ -I. main.cpp -lboost_system -lboost_filesystem -lboost_thread

The linkage order in gcc 4.6.1 matters more than its predecessors. It makes sense that having main.o linked first makes it quicker to pull the dependencies out of the subsequent libraries.

My CMakeLists.txt only needed boost_system to be added, just like my Makefiles needed it, but CMake correctly specified the link order, which I need to correct in my old project's Makefiles.

+1 to CMake, and therefore KDevelop. It is nice to have concerns like that taken care of for you.

Adding files to the project

My next move was to add a class to the project.

The jury is out whether it is a disappointment that KDevelop doesn't have a class wizard for creating the class. I confess I've never really liked those wizards in Microsoft's Visual C++ and Borland (yeah, I know I'm old), however I was unimpressed that the IDE assumed that I'd want to add a file in my home directory, but directories are sticky so that's no big deal. I think I like the balance, which KDevelop strikes here. You can navigate around classes in a class browser, but you create them as files. It is hard to see wizards working well with meta-programming, so the compromise is sensible.

What puzzled me, though, was how the new file stands with respect to version control. When you add a file from the IDE, it automatically adds it to Subversion. I like that. What was less elegant was the fact that it did not seem to realise that the file had been added, and having written code, I found that the right-click menu offers you the option to add the file (and then tells you you have already done that sucker!) and doesn't give you the option of committing it.

Stop grumbling. This is FOSS. I guess that fix is something I could contribute to the KDevelop project, if I get familiar enough with it to dig into its source, and someone else doesn't beat me to it.

Reloading the project lets you commit new code, and once you are dealing with code that's already been committed, it works OK, with only minor quibbles (like the fact that the cursor is in the diff box when you commit, when you'd expect it to be in the comments.

Intellisense

I was disappointed to find intellisense attempting code completion when I was adding comments to a README.txt file. That doesn't bode well. I may simply switch it off, if it doesn't help productivity. Hand-on-heart, I've not started coding sufficiently in anger in this environment to have a view on code completion yet, but I hate it when I find unexpected text deposited into my source.

Last thoughts

These are only initial impressions. KDevelop looks like quite a polished environment, but you can tell that it hasn't benefited from commercial QA. I am going to hang in with it and test it in anger by seeing through the porting of my old code. If I find that I really am too old school to move to this IDE or that its warts are too many, I'll thank it for bringing CMake to my attention.

Amazon Silk

2011-10-03T05:12:00.000-07:00

Perhaps I'm turning into an Amazon fanboi, but I find myself very much liking the approach under-pinning the up-coming Silk browser. They say it is going to kick off initially as an exclusive for Kindle Fire, which makes sense. Silk will give that tablet a real advantage over other mobile devices.

So the browser becomes split between EC2 and an "app" that runs on your mobile device. Most of the beef is in EC2; the "app" is a light-weight user of bandwidth. Neat. Watch the video, if you haven't already done so:

This approach is so right that it is bound to reach other devices, including the desktop. The assembly processes running in EC2 are bound to be shared, which I guess has some security implications, but I was wondering what this means to web application developers like me.

For starters, this is another nail in the coffin for voting applications that use IP addresses. That approach was always a bit flimsy, but we are going to see an increasing number of clients appearing with the same REMOTE_ADDR, where the IP address corresponds to some server in EC2. Applications should of course be using X-Forwarded-For, but the value of that IP address is likely to be diminished too.

A good approach for Caching becomes all the more relevant. EC2 will presumably become something of a CDN for clients, accessing common resources, and the more proxy-friendly we can make web applications, the better we can make it for all concerned. More reason to get expiry headers right.

sshfs to the rescue

2011-07-05T10:32:00.000-07:00

I'm building a new AMI in Amazon and am getting to the bottom of a problem with log4j. I always seem to have problems with log4j with JBOSS, but that's another story.

I want to run a quick directory comparison on two locally mounted directories to see what's different. The natural thing to do in EC2 is to snapshot and mount, so that both EBS volumes are accessible from the same EC2 host, and that wouldn't be a bad idea, but a quicker approach is simply to mount a remote directory on a host using sshfs.
Here's the drill:

Install sshfs via the package manager: sudo apt-get install sshfs
Put my working user into the privileged fuse group: sudo gpasswd -a $USER fuse
Create a directory to use as the remote mount: mkdir ~/remote
Mount the directory: sshfs -o idmap=user -o uid=1000 -o gid=1000 $USER@MY-REMOTE-IP-ADDRESS:/MY-REMOTE-DIRECTORY ~/remote (the remote user ID and group ID os 1000:1000 and this maps the local user ID to them)
Then use diff recursively for the directory comparison.
Clean up afterwards: fusermount -u ~/remote (this unmounts)

This is an example of FUSE, which I find really liberating. It is something that you expect on the KDE or Gnome desktop, where is runs behind dialogue boxes in much the same way, but you don't necessarily think of using it in the cloud... or maybe that's just me with my head in the wrong clouds.

Struts legacy and jQuery web services

2011-06-22T13:06:00.000-07:00

I have a new application that needs to mash up with an old application. The old application is built using an unfashionable design paradigm: Java Struts2, Dojo and working on my own. The new application is PHP, JSONP, jQuery and working with company. My colleagues are comfortable with the new technology set. Making the right decision with respect to the new application is much easier than the old, but today's soul-searching regards the old app. How do I move forwards with it? I need to add some flesh to what's there, but really don't want to add to its legacy.

Struts2 emerged from the WebWork Framework and thankfully is not much like the original Struts. Struts2/WebWork tags work nicely enough with view JSPs Integrating Spring for MVC wiring is natural for Struts2 and you find that its tags work very nicely for the MVC paradigm. My only real problem with Struts2 is that I don't spend enough time with it to feel like I can churn out code without giving it much thought. I couldn't ever really get on with Dojo (the preferred Struts2 JavaScript framework) and wound up favouring server side processing and/or using Prototype.js/Script.aculo.us for drag 'n' drop, but Prototype.js - like Dojo - is more technology I want to consign to history.

I don't really want to keep building old technology. I didn't ever really get much value out of Dojo, and my Prototype.js is looking unloved now. jQuery is great to work with, because it has become a Lingua Franca in web development.

How should I proceed with building on the model in the old application to support features that belong in the old application and are needed for a mash-up with the new application?

Web services

I was really tempted to keep on truckin' with Struts2, but it makes sense to build new technology in jQuery+JSONP, and present everything else that I need in the old application as a JSONP web service, with the JSP code.

Bye-bye, Struts2/WebWork tags. My view pages are pretty much static HTML, which flesh out content via JSONP. Bye-bye, complicated controllers. My model/controllers concern themselves with building light-wight JSON objects rather than setting up beans for views. The beauty is that most of the work has always been in views, but now my JSP views are not much different from PHP pages which I can use to demonstrate functionality. Out with Java beans and in with JavaScript objects.

The jury is out whether jQuery is going to feel different from the old technologies in the new pages. I expect it will. but perhaps they ought to feel new and different anyhow.

Memcache responsibility

2011-06-17T03:13:00.000-07:00

I shared a link with colleagues a while back for MongoDB: MongoDB is Web Scale. The line that tickled one of my colleagues most was: You turn it on and it scales right up. The terrible truth is that there are so many great technologies out there (MongoDB included), that we often don't give ourselves enough time to understand them properly.

Memcache is something that I realised I needed a lower-level understanding of this week.

This was my initial understanding:

Memcache allows you to make use of spare RAM in your production cluster, and avail it to your applications for caching. The more you scale out, the more benefit you get, because the RAM is shared as one great big pool.
Installation is very simple. You start a daemon on each node in the cluster, and that's it. You turn it on and it scales right up. Typically, you run one instance per node, and conventionally have it listen for TCP/IP connections on port 11211.
In the world of PHP you connect(!), and set data using a key with a TTL and and optional compression flag and get it simply with the key.
I appreciated that they key used for get/set, needed to be contrived in a clever manner to make sure that it was unique within the application and unique for all applications. A simple approach for caching SQL results was to generate a key from a MD5 of the concatenation of the application name and the SQL query used for the results - i.e. $key = md5('MyApplication'.$sql), adding servername to the concatenation, if there are multiple discrete versions of the application.

What I couldn't reconcile myself to was how the Memcache instances knew about each other to achieve the scale-out, and make the data available across the cluster. When the memcached daemons are instantiated, they aren't told about each other, so how do they swap information with each other.

I ran some experiments, connecting to different Memcache instances in the cluster and setting and getting, and found to my horror that they don't know about each other. Set a key on one node and the other nodes don't know about it!

My knee-jerk reaction was to fire an e-mail the long-suffering System Administrators to complain that the cluster wasn't set up properly, and my next foolish digression was to look at repcached's replication patch as a way to replicate data across the cluster. Fool that I am.

What I had failed to recognise was the essential design philosophy of Memcache, summarised in this paragraph:

Servers are Disconnected From Each Other

Memcached servers are generally unaware of each other. There is no crosstalk, no syncronization, no broadcasting. The lack of interconnections means adding more servers will usually add more capacity as you expect. There might be exceptions to this rule, but they are exceptions and carefully regarded.

What I'd failed to appreciate is that the responsibility for scale-out is on the client, and that the client has the responsibility to connect to the right Memcache node to set/get data.

In your standard high level Perl or PHP client, this is achieved by:

Requiring a list of Memcache instances to be added to the client in a consistent order.
Connect to required instances as and when needs be.
Determining which instance to connect to typically by a simple modulus on the CRC of the key used to reference the data, as a crude load-spreading technique.

Where I'd been doing wrong was explicitly to be connecting to Memcache instances, when I should have left it to the client library to select the right instance in a list to connect to as and when required.

The responsibilities are as follows:

The server (memcached) is responsible for maintaining a fast hash with a simple TCP/IP protocol for getting and setting data referenced by keys.
Client library is responsible for determining which server to connect to, and handles zlib compression, if your data size warrants it. It presents a simple API for you to work with, handling the sockets interface, and spreading the data across the Memcache nodes, typically using the key CRC modulus trick to select the node.
Your implementation is responsible for setting up a consistent list of Memcache instances for the client library to work with, and you need to be responsible for generating unique keys in whet is most likely a shared resource, appropriate TTLs and sensible judgments about compression.

SSL Certificate Revocation - Need to clean up our act

2011-03-26T06:22:00.000-07:00

With Comodo issuing a batch of dodgy SSL certificates, Certificate Revocation came into focus. Certificate Revocation is the way a certificate is withdrawn from trust; you cant force a signed certificate to be removed from a web site, but you can black-list it centrally.

Certificate Revocation Lists (CRLs) used to be the way certificate revocation has handled, with Distribution Points publicised in the Certificate Authorities' (CA) certificates. "Browser, go to this Distribution Point for this CA, and you'll see which current certificates are black-listed." This doesn't scale well however, and you'll find an empty list of CRL Distribution Points in Firefox (and Operating Systems for the other browsers). Nowadays OCSP is used to distribute certificate revocations rather like the way DNS works, responders acting for certificate revocation like local DNSs act for domain name look-ups.

Firefox and Internet Explorer 7+ support OCSP, the latter only via the Windows Vista or Windows 7 operating system. On Macs, I understand that Safari users need to activate OCSP in their KeyChain preferences in OSX (Do it!). Windows XP users should stick with Firefox.

Alas, having a browser that supports OCSP doesn't mean that you know that it is supported by the CA. Remarkably, Extended Verification certificates are not necessarily protected by OCSP. So that Green Bar on your browser doesn't tell you that the site you are looking at has a certificate that hasn't been revoked! So you don't have a visual indication that your site certificate is valid; you benefit only from seeing that some are invalid. That is not good enough.

Firefox has an extension called Calomel SSL Validation, which only green-flags sites which are OCSP-verified and strong ciphers. It flags interesting things like the fact that NatWest online bank is on a weak cipher and key, which seems very sloppy to me. If you accept anything less than a green-flagged site, it isn't explicit whether the CA provides OCSP or whether the failure was only due to a weak cipher. So, if you take the Draconian approach that only sites green-flagged by the Calomel Validation should be trsuted, you are likely to lose a few sites, but less likely to be subjected to fraud. I'd like to see OCSP mandated as a main-stream requirement for secure sites, but in the interim I'll stick to Firefox+Calomel for online security.

Windows 7 and PHP development

2011-03-16T02:58:00.000-07:00

I am told that I ought to be using Aptana for PHP develpoment to benefit from source code indexing, but I'm a creature of habit and was wondering if I could stick with my usual text editors and make better use of Windows explorer for search.

I haven't invested any time in the Windows search beyond looking at my Office documents and wondered why it was that when I looked for an HTML ID in a PHP project, I'd see relevant stuff in JavaScript, CSS and funily enough subversion entries, but nothing in the PHP source.

Time to look at Windows Explorer's Indexing Options, which in Windows 7 means hitting the start button an entering Indexing Options.Clicking the advanced button and tabbing to File Types shows how the various files are being indexed. Having some Lucene experience, it should have occurred to me that Windows assigns different indexing filters to different file types, with HTML and CSS getting a special HTML filter and JavaScript .txt etc being handled by a plain text filter. Office files of course get a special filter of their own. In a default set-up, PHP files get the same treatment as images and anything else which is deemed to be uninteresting for internal search... and that is a search on File properties only.

Well since I'm earning a crust working with PHP, it makes sense to assign the Plain Text filter to *.php so I can have Windows index them too. Doing that is simply a matter of hitting a radio button to Index Properties and File Contents. Job done.

Handling fall-through in a rewrite map

2011-03-11T04:17:00.000-08:00

If you have to deal with a lot of URLRewrites, you want to use a map, but what about the URLs that do not get matched by the map? Let's say there are a load of entries in your /2011/March/ folder in your blog, which you need to map over to some external domain.

Under normal circumstances you can use something like this in your virtual host configuration in Apache:

RewriteEngine on

RewriteMap march-2011-map txt:/somepath/march-2011-map.txt

RewriteRule ^/2011/March/(.+) ${march-2011-map:$1|/2011/March/$1} 

The march-2011-map.txt file (in directory /somepath) contains lines like this:

sausages http://example.com/sausages

Hits on anything in the /2011/March/ directory get the map treatment. A hit on /2011/March/sausages gets redirected to http://example.com/sausages and a hit on (say) /2011/March/eggs, if eggs is not in the map, gets handled locally.

This works because of the parameter encoded after the '|' in the RewriteRule, which replaces non-matches in the look-up.

The problem with this approach though, is that maving rewritten the lookup in such a way as to do nothing to the non-match, the URL is handled there and then and any other rewrite rules etc do not get a look in.

There are situations where you want to redirect URLs matched by the map, but to continue to handle other URLs. The solution is to use a RewriteCond, which is a look-ahead to make the Rewrite only apply to map matches.

RewriteEngine on

RewriteMap march-2011-map txt:/somepath/march-2011-map.txt

RewriteCond ${march-2011-map:$1} >""

RewriteRule ^/2011/March/(.+) ${march-2011-map:$1} 

We have lost the '|' in the RewriteRule now, which means that keys not matched return as an empty string. The RewriteCond, which precedes this is a look-ahead, which is unintuitive, if you've not played with RewriteCond before. It looks at the lookup in the subsequent RewriteRule and tests that it is something greater than the empty string. That means that RewriteRule handles /2011/March/sausages because it is a match and therefore returns something greater than "", but the RewriteRule is not applied to /2011/March/eggs, because that look-up returned "".

You can then go on to handle /2011/March/eggs with subsequent tests and rules in your configuration.

PHP and Binary Network Protocol

2011-02-07T03:28:00.000-08:00

A friend of mine consults to a company that has a Fingerprint identification module. This let people into and uot of some premises. He asked me if I'd be able to write some software to controll the device, given an SDK for C++ and C# and a specification document. What I knew that he really wanted was to be able to do this work himself in a language which he is comfortable with: PHP. What he wanted from me was a leg-up to be able to do this.

I flirted with the idea of digging through the C++ and/or C# source, but it soon became clear that most of what was there was GUI-related, and understanding the applications required understanding invocations to library files, and the invocations were undocumented and poorly commented. By contrast, the specification document described with great clarity the TCP/IP protocol to interrogate and control the module.

With some bravado, I said I'd knock up a PHP interface to the module, recalling a project 6 years ago, where I worked with socket client in a PHP management page, which controlled a proprietary mail transfer agent, for Email Systems; PHP has a relatively little-used low-level sockets library, which was just the ticket for customer management pages that needed to maniulate the MTA.

The Fingerprint device presented some challenges that hadn't occurred to me in my moment of hubris. The protocol has all the hallmarks of something derived from a serial protocol: fixed lengths, signature bytes and checksums - application checksums are superfluous in TCP, which has a checksum in its header. This sort of thing is all well and good when you are wirting assembler or C code, but it is a culture shock in PHP.

Fortunately, I've had to deal with a similar binary protocol (also with its roots in serial) for tests in Perl, and at the time I muddled through with pack/unpack, taking advantage of the fact that strings are binary safe. Pack/unpack are also available in PHP - I expect used even less than the sockets functions - and its strings are also binary safe.

To a C/C++ programmer it seems very alien, but initialising a variable with $my_variable = '' in PHP is equivalent to creating an empty buffer, and to the chagrin of UTF-8 using strlen() on that variable tells you how many bytes there are in the buffer. You can use substr() to extract portions of the buffer, as you would with memcpy() in C/C++. Using these core functions along with pack/unpack, we can pick through binary data packets.

It doesn't all smell of roses, though.

Like Perl, PHP's pack/unpack functions are quirky. The module is - we have to guess - little endian; the protocol is wirtten for MS Windows clients and there is no mention of network byte order and a little-endian implementation works. I'd like to think that we could have a PHP library that works on any architecture, including pre-Intel Macs, which are big endian. However, that would require extra work, because the 16-bit "short" format specifiers for pack/unpack are as follows:

s signed short (always 16 bit, machine byte order)
S unsigned short (always 16 bit, machine byte order)
n unsigned short (always 16 bit, big endian byte order)
v unsigned short (always 16 bit, little endian byte order)

If you know the source is a signed 16-bit integer on a little endian system and you don't want to have to know the endianism of the server your PHP is running on it seems silly that you have to work with unsigned shorts, because signed shorts are assumed to have local endianism.

The PHP unpack function differs from its Perl counterpart in that it returns data in an associative array. When you've been broght up with C/C++, it feels very inefficient working with hashes in a protocol handler, but that's OK for my friends application, because it isn't something that has to scale. The device will only ever have one client at a time, and indeed is designed only to allow one at a time.

So what was the outcome of my evening's fiddling? I think and hope I've given him a PHP class that he can use as the basis of his controller. It seems to work OK with a simulator that I knocked up in C++. I quite liked the quirkness of building something that can go into a PHP page (or CLI) that can do something you'd expect to have to use a lower-level language for.

MySQL laziness with UTF-8

2011-02-02T04:12:00.000-08:00

You can get away with putting UTF-8 into a Latin1 table in MySQL, if your client connects without using SET NAMES UTF-8 (i.e. you leave the client connecting for Latin1).

Certainly you will expect some things to go wrong, if you get too clever. You wouldn't expect the MySQL CHAR_LENGTH() function to play ball with multi-byte characters, and likewise functions like SUBSTR() will get their character boundaries wrong, but a bog standard CRUD application seldom uses string functions that are likely to trip up portability. So you can get away with not telling the MySQL server or its client code that it is dealing with UTF-8, and other than the odd hiccup with collation in sorted results you can slap on your XML and otherwise remember to put UTF-8 in and expect to get UTF-8 out, and have a happy lazy coding experience.

I have an odd application running that uses a MySQL database as a temporary file on steroids. I have a heap of UTF-8 data coming in from a daily feed that needs relationships to be set up and what better than an RDBMS for this, and what could be easier than MySQL or PostgreSQL for the relationships. I'm amongst LAMP flag-waving company so MySQL is what I'm using. My application imports this data using MySQL's excellent LOAD DATA INFILE syntax, which sets up the relational database in short order. Somehow, I'd forgotten to set up the database for UTF-8 and yet was stuffing UTF-8 data into it without a care. It worked for my purposes. My client code - a C++ application that is responsible for static publishing - likewise forgot to send a SET NAMES UTF-8query, so both ends were blithely handling the UTF-8 data as if it was ISO-8859-1 (Latin1) with no penalty. Where order was being used in my client application, it didn't matter that it was erroneously working on UTF-8 data with Latin1 collation.

Something went wrong in the data feed (invalid UTF-8 characters coming from the source) and I took a another look at the application and spotted the Latin1 encoding.

Time for some floundering (confession time):

I changed the database to UTF-8 encoding, so that my Navicat MySQL client would show nice UTF-8 data in my temporary database. My diacritics now looked good with the exception of the invalid UTF-8 data, which created the fuss in the first place. because it was invalid. However, this shot me in the foot in my MySQL client, which was now rendering the diacritics as ISO-8859-1 (Latin1) and slapping on its XML. My client was connecting to the UTF-8 database with a default Latin1 connection, and the MySQL drivers in good faith converted it to Latin1. Things were made worse.
Having made my error in an environment which unexpectedly was being used for production purposes (oops) and having discovered that an awful lot of XML was no longer well-formed because of ISO-8859-1 characters posing as UTF-8, I needed to run a hasty find . -type f | xargs perl -i.old -p -e 's/UTF-8/ISO-8859-1/' on my published XML files, and then to edit my C++ sources to declare ISO-8859-1 as the encoding the the XML declaration for future publishes.

Panic over, it is time to take stock.

I now have a UTF-8 database with UTF-8 data in it, which improves its appearance in Navicat (big deal) and also has UTF-8 collation (well OK there may be a subtle benefit from this).
My client code is now ISO-8859-1 (Latin1), and says as much in its XML declaration.

Unless we start seeing characters in the feed that cannot be converted to ISO-8859-1, we are in good shape, but it would be good to go the extra yard and have the client handle UTF-8 properly.

So, what's needed?

My C++ client, needs to connect to the database for the "utf8" charset. If it is going to render in UTF-8.

Hre's how:

void init()
{
    mysql_init(&mysql);
    const char *charset = "utf8";
    if (mysql_options(&mysql,MYSQL_SET_CHARSET_NAME,charset) != 0)
        cerr << "Warning: Unable to set CHARSET '" << charset << "' " << mysql_errno(&mysql) << ": " << mysql_error(&mysql) << endl;

    if ((con = mysql_real_connect(&mysql,host,user,password,db,0,0,0)) == 0) {
        cerr << "Error: Unable to connect code " << mysql_errno(&mysql) << ": " << mysql_error(&mysql) << endl;
        exit(1);
    }
    atexit(done);
}

C++ strings are more robust than C's ASCIIZ strings, so need to worry about '\0' characters if we treat the multibyte strings no differently from regular ASCII 8-bit characters.

Some incremental updates are applied with a Perl DBI client, which has yet to be made UTF-8 aware. Since that writes to the database, I need to clean up its act:

It need to read its update data in utf8 mode:

open FILE, $filename" or die "Couldn't open $filename: $!";
binmode FILE, ":utf8";
my $content = ;

Strings inserted, need to be UTF8-ised:

utf8::encode $mystring;

The database needs to be connected to for UTF-8 mode:

our $dbh = DBI->connect("DBI:mysql:$mydatabase;host=$myhost",$myuser, $mypassword, { RaiseError => 1 });
$dbh->{'mysql_enable_utf8'} = 1;
$dbh->do('SET NAMES utf8');

Sometimes laziness is king.

DTD go local

2010-11-16T04:53:00.000-08:00

I meant to look into this a long time ago, and regret I didn't. Here is an excellent walk through for the use of an XML catalogue file to allow DTDs to be used in the absence of Internet connectivity. This will stop my XSLT scripts from grinding to an embarrassing halt, when the resolver isn't resolving iin customer sites.

Elastic Load Balancing for HTTPS

2010-11-09T03:56:00.000-08:00

Amazon Web Services has added support for HTTPS termination to its Elastic Load Balancer, according to their blog, which means that the final piece for scalable E-commerce solutions is in place. The trick is putting the certificate onto the load balancer, and leaing it to the ELB to handle the SSL handshake.

Like vanilla HTTP, there is support for sticky sessions over the HTTPS Elastic Load Balancer.

Is it enough to trust vanilla HTTP between the ELB and your application instances? I guess it depends on the application, but if you do not you can always forward to port 443 on the application server and use a second encrypted channel for that hop. If the AWS security white paper cuts the mustard, settle for the promised X-Forwarded-Proto header to indicate that the communication arrived via a secure channel upstream, and you have adequate encryption for most privacy purposes.

I think this opens up all sorts of possibilities.

Going static in AWS

2010-08-06T04:14:00.000-07:00

Amazon has announced default root objects in Cloudfront, which makes me think of static publishing again. I really like the idea of throwing everything at S3 and hence Cloudfront and reducing my application to something that works with JSONP to get around cross-domain scripting issues with JSON. The default object is the final piece needed for this, because now I can have a home page and use www or default DNS entry for the CDN and something less catchy for the dynamic stuff that goes through JSONP.

No parentheses in PHP language constructs

2010-01-16T04:45:00.000-08:00

I list PHP amongst the languages, which I'm not great at, but which I occasionally earn a crust from.

PHP appears to be a simple language, because it spurns Perl's proclivity for hieroglyphics and is somewhat less "write-only" than Perl. Notably arrays and hashes are unified in PHP. PHP occasionally falls over itself, attempting to improve upon Perl with cleaner syntax - e.g. for regular expressions - but mostly it does a good job at keeping its syntax consistent.

I am therefore puzzled to see that the architects of PHP found the need to have language constructs which are distinguished from functions in that they do not need to have parentheses around their arguments. They are listed amongst the PHP keywords. Typically, you only come upon them, when you see that (say) an echo in a PHP page "forgot" to have its parentheses, but... seems to work anyhow.

Once discovered, the PHP coder delights in having two less characters to type, just like the Perl coder makes a point of avoiding superfluous brackets to remind himself that he's not writing C code. Poor code reviewer. Lucky coder, eh?

Database of choice - reacting to Oracle

2010-01-10T07:28:00.000-08:00

I have some loyalty to PostgreSQL, because at a time when I couldn't fathom the MySQL AB license, it provided an excellent affordable RDBMS. In a project where some messy Oracle code needed to be ported, I was pleased with the ease of porting into PostgreSQL. It would have been tougher with MySQL. I would love to say that I was guided by the principals of FOSS, but hand on heart Free = $0 was my main motive.

Times have moved on, and MySQL has been a choice of convenience in a lot of my recent projects, because it is by far the market leader and it is very good at what it does. People know it. Like all software developers, I've become wiser about licenses, and I'm OK with what I get from the package managers.

An article in Slashdot alerted me to MariaDB, which is a FOSS branch of MySQL 5.1 by Monty Widenius, the original MySQL creator. Monty expresses the opinion that PostgreSQL is dominated by closed source Enterprise DB, which has exclusively many of the features that you come to expect like replication (Slony). He also says that Oracle's buy-out of Sun is likely to be a way to retrieve the perceived sales loss of $1 billion per annum caused by MySQL. They are going to want to make some money out of it, and are unlikely to depend on Sun's preference community building etc, which is a bewildering way to make money anyhow. If Oracle are also likely to buy out talent from PostgreSQL, all FOSS RDBMS look threatened by the buy-out.

Should we care and do selfish individuals like me need to help? I guess we should sign the Save MySQL bill.

I guess MariaDB means going back to compiling from source for a while, but I expect it won't be long before we see it in our package managers, if Oracle does the dirty on MySQL. There are good reasons to doubt whether Oracle could buy out the top 20 PostgreSQL developers, though, bearing in mind that they are all typically employed and valued by other companies. Probably the best thing to do is to stick to standards and avoid vendor lock-in. Don't get too comfortable with the idiosyncrasies of your RDBMS, because you may need to change.

E-mail the old killer app

2010-01-04T05:03:00.000-08:00

E-mail is the old killer app of the Internet. In the early-mid '90s you were more likely to subscribe to ritual humiliation by ISPs because of e-mail than WWW; nobody could find anything in those days on the Web anyhow.

Familiarity with an e-mail client remains a good reason to avoid changing operating systems nowadays. A Microsoft user is likely to spend more time in Outlook than Word. So it ought to be a well-oiled machine, oughtn't it? But it isn't.

I spent a chunk of time working for a company that provided managed e-mail services, but I struggle with day-to-day e-mail issues, caused by the paradoxes of

wanting to make myself accessible and yet remaining impervious to spam
being intolerant of false-positive spam filtering in my mailbox and yet preferring my family not to be subjected to the freakish correspondence that's become the norm in unsolicited mail
attaching low value to something that's a life-line in business

Increasingly, I've worked with people whose e-mail is so broken, that they've come to depend on social networking or centralised project management applications like Basecamp or bug tracking software like Mantis/Bugzilla to handle correspondence. These all need message recipients to be alerted via IM, text messages, telephone calls or - yes - e-mail to alert message recipients, if they are not in the habit of polling the system, so none of them really cut the mustard for general messaging. What we all really want is e-mail that works.

My current imperfect solution is as follows:

I don't publicise my e-mail address much, but it is out there, which means that not much effort with Google will locate my address. This is security by obfuscation, which is ultimately doomed as the spammers put more an more effort into harvesting e-mail addresses, but it may prevent some nutters from sending unsolicited mail.
I use SpamAssassin for filtering my mail on a low-end miniserver, where I have my MX server. I use Postfix as my inbound and outbound MTA. I use Procmail as my MDA, putting mail judged to be spam into a spam folder in the Maildir, as specified in /home/{user}/.procmailrc. I use POP3S over Dovecot to fetch my mail, which I manage on my desktop PC with Outlook, because I like to be able to search through several years of e-mail, and Outlook's PSTs seemed to work best. The rest of the family/small-business use IMAP over Dovecot, because they move around more and expect e-mail to be accessible from any client.
I use Outlook as a second line of defense against spam, because I've found that it identifies a lot of spam that my SpamAssassin setup overlooks.

This isn't perfect for various reasons.

The worst reason is that it is over-engineered. I run a local certificate authority for the TLS, to save money, but that means installing the CA as a trusted authority on all PCs, which use POP3S or IMAP/TLS to fetch mail. I just works for a few years and then I need to refamiliarise myself with the server set up for an upgrade.

I upgraded the miniserver from Debian Etch to Lenny to take advantage - at long last - of improvements to SpamAssassin, which need Lenny's package manager. The upgrade looked smooth enough, but mail delivery stopped dead, and I realised that I needed to revisit the server set-up, because I'd forgotten too much and the log files didn't say enough to identify what was going wrong.

I knew it was going somewhere, but where could it be? I little it of fiddling with du showed me that /var/spool/postfix/hold was holding onto everything that ought to have been going through the MTA. A lot of head-scratching and floundering with Google, identified the following as the cause of the problem in /etc/postfix/main.cf:

header_checks = regexp:/etc/postfix/header_checks

It looks like everything was getting identified by the header check regex and getting thrown into the hold queue. The regex?

root@mini:/var/spool# cat /etc/postfix/header_checks
/^Received:/ HOLD

Hmmm. So it looks like everything was getting stamped with a Received: HOLD header.

I simply commented out the header_checks from Postfix's main.cf, and now I'm back in business with somewhat improved spam-filtering. If I was putting more energy into my e-mail handling, I might get some value out of using a hold queue, but life is too short to fix this properly and to get to the bottom of why this Received header was getting set and or detected.

My New Year's resolution ought to be to value e-mail more highly and get a better solution for all this, but it is tempting to close the book on this for a few more months and "make do".

Obsoleting deprecated detritus in Lucene 3

2009-12-12T03:38:00.000-08:00

Lucene went up a major version number in November, purging some of the deprecated detritus, which expediency has left littered in a lot of my code.

This is what I needed to address to get with it:

Nowadays Lucene refers to fields as analyzed rather than tokenized. This is much clearer, and its terminology which I've pushed all the way back to my query interface. I haven't obsoleted "tokenize" in my query interface - I have too many JavaScript, PHP and Perl clients scattered about to do that, but I have deprecated it, favouring "analyze" instead. My server code now compiles against Lucene 3 with "analyze".
Shock horror: Lucene has made compressed fields obsolete. My index writers now ignore requests to compress, requiring instead that data is requested to be zipped with a specified compression, and clients wishing to retrieve this content are required to indicate explicitly that they are fetching zipped binary data, which my reader should decompress. With some help from java-user@lucene.apache.org, I found that I had been abusing field compression anyhow by applying it to fields of less than 1K. It may wind up being inappropriate to zip the synopsis fields which I had been compressing, because upgrading indexes with compressed fields and decompressing the compressed fields resulted in an unexpected decrease in the index size. I should get a good performance improvement avoiding the spurious compression in my indexes, and it looks as though I shan't see in increase in index sizes in my application. It took the upgrade to force me to do the right thing.
Hits are out and it is time for me to embrace the more efficient TopDocs mechanism for getting hits. Changing over to TopDocs wasn't very painful in the event, my server code was sufficiently modular to make it one edit and - yes - I can see a performance improvement in my average query times.
I noticed that scoring is more discerning now, which is a good thing for relevancy sort order. I am seeing fewer score='1.0' results, which I believe is a consequence of better handling of proximity with stop words. There may be some repercussions from this in relevancy orders in the change-over to Lucene 3.0, where my Lucene 2.3 indexes have completely ignored stop words, and I need to deal with an evil script, I have come to depend on which uses absolute scores rather than percentage scores, which would always have been more reliable.

I was initially tempted to upgrade to 2.9.1 and leave all of my deprecated foibles in production, but I'm glad that I went the whole hog and implemented my software with Lucene 3; it forced me to address issues which I've sat on for too long.

Wrong Content-type with an HTTP/404

2009-12-01T09:01:00.000-08:00

This really threw me:

2009-12-01 16:07:22,875 INFO  [STDOUT] 16:07:22,874 ERROR [Rss2ListIndexHandler]
Unable to add http://SOMEPATHTOANXML.xml: Server returned HTTP response code: 
503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

My crawler was handling RSS 2.0 XML published on a blog site. Some of the RSS feeds did not exist, and the server returned the sites's standard HTTP/404 ErrorDocument, which was of course Content-type text/html, served up with the following DOCTYPE:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

My XML DOM handler naturally assumed it was getting XML, because it didn't spot the unexpected Content-type, and attempted to fetch http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd to validate it. W3 returned an HTTP/503 to reduce its bandwidth, because it reasonably enough viewed my fetch as abusive.

BeanShell for a sanity-check

2009-11-19T02:15:00.000-08:00

There are some development tools that come to be real friends in a crisis. One of these for me is BeanShell for quick Java tests and experiments.

I have a process in production that scans the published XML Sitemaps for various domains owned by a client. When changes have been made to pages, or new pages come into existence or pages are added, it needs to make changes to corresponding Lucene indexes. To stop it from working on pages unnecessarily, it looks at the lastmod elements, which are ISO 8601 date+time stamps. My utility is only interested in the date part of the lastmod, and it parses is with the Java SimpleDateFormat formatter DateFormat fmt = new SimpleDateFormat("yyyy-MM-dd");.

In one of the sites an area had become marooned from local linkage, and it was missing from the index. It was only referenced from another domain, and its XML sitemap generation used a quick'n'dirty approach with a point-it-at-the-root sitemap generation tool. These generators are fine, if you cannot publish your own sitemap with your content management application, with caveats:

Server-side scripts will typically always appear to have new lastmod values. Typically PHP developers do not fix the Last-modified header to reflect the last file edit, which the generators depend upon, because it is not always appropriate; the dynamic content may have changes for other reasons (e.g. data or an include).
Content accessible only via search or JavaScript won't appear in one of these sitemaps.

To add to the confusion, a sitemap generator had been producing files in the form "2009-11-13 22:23:38", but had changed or been replaced with another that generated the more ISO 8601-ish "2009-11-13T22:23:38+00:00". I needed a quick sanity-check that a format change wasn't responsible for the missing references in the index.

Enter BeanShell, with the following shell script:

import java.text.SimpleDateFormat;
import java.text.DateFormat;
import java.util.Date;

DateFormat fmt = new SimpleDateFormat("yyyy-MM-dd");
String[] ss = "2009-11-13,2009-11-13 22:23:38,2009-11-13T22:23:38+00:00".split(",");
for (String s : ss) {
 Date date = fmt.parse(s);
 print(s+": "+fmt.format(date)+"\n");
}

Sure enough, by SimpleDateFormat was OK for my purposes. It could cope with all three lastmod formats.

What I really like about this quick test, is that I didn't have to rub sticks together to boot-strap Eclipse to perform a simple Java test. I have the following simple Win32 bash.cmd file in my path on my Windows work-horse:

@echo off
setlocal
set classpath=%classpath%;/lib/BeanShell/bsh-2.0b4.jar
java bsh.Interpreter %*

All I have to do to run the test from my command prompt, is type bsh test.bsh.

I have a similar BASH script on my Linux laptop, but that wasn't at my finger-tips at the time.

In the event, we realised we needed to fix the sitemap generation process and pasted a chunk into the generated sitemap to index the marooned content as a stop-gap. Likewise Google will benefit from that sitemap.

Querying Freebase

2009-11-18T02:42:00.000-08:00

Freebase is a wealthy resource of free (Creative Commons) information in a structured form, which you can access using a sophisticated query language, called the Metaweb Query Language (MQL), which you can express in JSON.

Pulling information from Freebase in a server-side or desktop application, realistically requires you to use one of the client libraries or working with a data dump, if you don't have time to familiarise yourself with MQL.

I've been playing with the data dumps and MQL and I'm finding that it makes sense to work with MQL, because it models the data nicely, though I do find myself missing the functionality of Lucene for querying, especially fuzzy matching and full text search. But that's another story.

It is best to start playing with MQL using the MQL Query Editor application, which is a superb bit of JavaScript, allowing you to construct queries using your tab key to explore the database schema. This gets to up to speed with constructing empty maps with {}, empty arrays with [] and empty values with null in your JSON query, using key/value pairs for the selection criteria.

Initially, I assumed that the fill-the-blanks JSON approach would restrict MQL to exact matches, but special characters in the keys allow you to achieve range queries and regular expression matching. MQL is very powerful.

So buoyed up by my experimentation, I needed to bite the bullet with implementation in a project. My project has a lot of client-side JavaScript in it, using the prototype.js and script.aculo.us frameworks. I haven't made the move to JQuery, because my existing combination works very nicely and I do not like to change things for just for the sake of it. Neon signs in Freebase point you towards using Mjt (pronounced midget), which gets you into elegant JavaScript templating, but I have a nasty feeling that mixing this with prototype.js, is going to lead to debugging headaches. Also Basecamp polite milestone reminders keep telling me that my deadline is overdue.

So I needed to get MQL working with my JavaScript toolset, and leave Mjt for a rainy day, and that means getting a JSONP implementation working with prototype.js, which unlike JQuery, doesn't appear to support JSONP out of the box.

With JSONP, a calling parameter specifies the name of the callback function. The callback function is invoked with a JavaScript object, using JSON from a script loaded into the HEAD element dynamically loaded into the document. The calling parameter is JSON encoded into a single line as a query string parameter.

I've implemented something in the same vein as JSONP before. So I wanted to implement something similar as a prototype.js Class.

Here's what I wound up with for a base class for JSONP:

// Base class for JSONP - odd name, because of its similarity
// to my previous Uxml class .
var Ujson = Class.create({

 // Ctor
 initialize: function(url, queryObject) {
  this.reponse = null;
  this.transport = new Element('script',{
   type: 'text/javascript',
   src: url+(typeof queryObject === 'object' ?
    Object.toJSON(queryObject) : queryObject),
  });
  this.head.appendChild(this.transport);
 },

 head: document.getElementsByTagName('head')[0],

 // Handle the response - expect to override this
 handleResponse: function() {
  alert('Object response is: ' + 
   Object.toJSON(this.response));
 },

 callback: function(obj) {

  // It is already a bona fide object, passed by JSON.
  // No need to parse it.
  this.response = obj;

  if (this.transport && Object.isElement(this.transport)) {
   var transport = this.transport;
   setTimeout(function() {
    if (transport) transport.remove();
   }, 20);
  }

  // Handle the response with an override
  this.handleResponse();
 },
});

You can see that the script src="" attribute gets the concatenation of url plus the stringified JavaScript object, so the url parameter is for Freebase http://www.freebase.com/api/service/mqlread?callback=freebase_callback&query=, where freebase_callback is the name of the callback function and the stringified JS object is passed to the query parameter. There is probably a prettier way of doing this in the base class, but this looks OK to me.

I didn't initially use prototype.js's Object.toJSON(queryObject) to stringify the JS object, preferring instead the more general JSON.stringify:

// This technique appears to get clobbered by prototype.js
JSON.stringify(queryObject, function(key, value) { // Replacer
 if (typeof value === 'number' && !isFinite(value)) {
  return String(value);
 }
 return value;
})

However, I found that I was getting quotes URI entity-encoded with a spurious leading '\' escape, and my MQL keys were made illegible. Presumably it converted it into a string and then URI entity-encoded the string, complete with the '\' escape characters for double quotes.

I believe that this was a conflict between prototype.js and the JSON object, but the correct solution was to use prototype.js's static method Object.toJSON(obj). It took me a long time to figure that out, because of the distractions of modern life, paucity of debugging in JavaScript and a good dollop of stupidity on my part!

So here's an inherited class for Freebase, getting a list of stars in movies and images:

var UjsonFreebase = Class.create(Ujson, {

 // Ctor
 initialize: function($super, name) {
  $super(
   'http://www.freebase.com/api/service/mqlread?callback=freebase_callback&query=',
   {"query":
// Query built at http://www.freebase.com/app/queryeditor/ - using the tab key :-)
[{
  "type": "/film/film",
  "name": name, // Here's the movie name variable
  "guid": null,
  "id":   null,
  "starring": [{
    "actor":     null,
    "character": null
  }],
  "/common/topic/image": [{
    "id":       null,
    "optional": true,
    "limit":    3
  }],
  "genre": [{
    "name": null
  }],
  "initial_release_date" : null
}]
   }
  );
 },

 handleResponse: function() {
  // Response object passed to a dialogue
  // which does aplication-specific stuff 
  // with the data
  new MyDialog(this.response);
 },
});

For this to work, I needed a function called freebase_callback in the global namespace. I tried using an object, but the web service didn't seem to like to use that as the prefix for the JSONP callback.

So, assuming the UjsonFreebase object was created as window.freebase = new UjsonFreebase("Gone With The Wind"), we need a global freebase_callback function as follows:

function freebase_callback(obj) {
 if (window.freebase)
  window.freebase.callback(obj);
 else
  alert("No freebase object");
}

It works.

JSON for cross-domain scripting with XML

2009-11-12T07:56:00.000-08:00

There is a hack to allow cross domain scripting using a dynamic JavaScript file, which typically returns data as JSON. The idea is that you use JavaScript to generate a SCRIPT tag dynamically, which invokes a server-side script that generates JavaScript in response to the query string which depends on JavaScript variables in the client. It makes sense to have this server-side script return a JavaScript associative array, which is described by JSON, but it could be a bunch of variable assignments, if that was easier to work with.

I hate the fact that this is necessary, but can understand why. This approach is more efficient than using a proxy for cross-domain scripting, because it avoids the extra network hop. What I don't like about the approach, though, is that it tempts you to put client-side business logic into the web services.

I have a web service for querying a Lucene indexes. Queries are very expressive, encapsulating Lucene's query objects, and XML allows that expression nicely. Results are also expressive.

It is tempting to move the logic responsible for setting up the query into my web service and add JSPs for each client requirement, but that means making my web service know more about its clients than I want and delocalising my concerns. I don't like that.

So how about this approach? I generate the query XML and encodeURIComponent(xml) it to make it a parameter passed to my JSP. My JSP, which is local to the web service, simply URIDecodes the query parameter and passes the query on to my usual web service. This I invoke in much the same way as JSON, via a dynamic SCRIPT tag from the client. My JSP outputs one line: xml = decodeURIComponent('XXX');, where the 'XXX' is an URIEncoded XML response from my web service.

It should work, provided my query string doesn't get too long. There ought to be a way to avoid the extra server-side processing, but this is the best I can come up with for now.

It works

My servlet decodes the query string and gets XML

// Read the query from the query string
 String xmlQuery = java.net.URLDecoder.decode(
  // This is the XML document encoded by http://www.w3schools.com/jsref/jsref_encodeuricomponent.asp
  request.getQueryString() == null ? "" : request.getQueryString()
  ,"UTF-8"
 );

It returns the XML in URLEncoded form to a JavaScript object in the calling code:

String xmlResponseURI = java.net.URLEncoder.encode(xmlResponse, "ISO-8859-1");
 ServletOutputStream outStream = response.getOutputStream();
 outStream.println("if (window.uxml) window.uxml.callback('"+xmlResponseURI.replace('+', ' ')+"');");

The JavaScript object needs to have a callback function:

// prototype.js example

var Uxml = Class.create({

 // Ctor
 initialize: function(query) {
  this.reponse = null;
  this.transport = new Element('script',{
   type: 'text/javascript',
   src: 'http://example.com/servlet?'+encodeURIComponent(query),
  });
  this.head.appendChild(this.transport);
 },

 head: document.getElementsByTagName('head')[0],

 // Handle the response - expect to override this
 handleResponse: function() {
  alert(this.response);
 },

 callback: function(encoded_response) {
  this.response = decodeURIComponent(encoded_response);
  if (this.transport && Object.isElement(this.transport)) {
   var transport = this.transport;
   setTimeout(function() {transport.remove();}, 20);
  }

  // Handle the response with an override
  this.handleResponse();
 },
});

// Object instance must be called uxml for the callback invocation
window.uxml = new Uxml(xmlQuery);

The use of the object is similar to the use of a callback function in JSONP, as disucced in Wikipedia, in which a calling parameter specifies the name of the callback function. The callback function is invoked with a JavaScript object, using JSON. If you look at my subsequent Freebase post, you'll see a proper JSONP implementation with prototype.js.

Incidentally, since implementing the XML solution, I found that for the sake of i18n, it made better sense to use UTF-8 encoding in the XML response and therefore the URI entity encoding.

String xmlResponseURI = java.net.URLEncoder.encode(xmlResponse, "UTF-8");
 ServletOutputStream outStream = response.getOutputStream();
 outStream.println("if (window.uxml) window.uxml.callback('"+xmlResponseURI.replace('+', ' ')+"');");

OpenID Identity Providers

2009-11-10T02:14:00.000-08:00

Many moons ago I did some work on a system that accepted federated login from Shibboleth Identity Providers.Setting up that system as a Service Provider for the Shibboleth trust network was quite painful. It involved a bit of dependency hell, partly because the system administrator had a preference for Slackware. Shibboleth's approach for networks of trust is excellent for institutions, which need to trust institutions, but I've been finding increasingly that federated login for individuals is what's needed for my projects. Rather than having a network of trust, I have a list of trusted individuals and some knowledge of identity providers. My facility needs to trusts the individual and identity provider used by that individual.

The whole point of using OpenID for me is to avoid administering passwords, so I don't want to host my own identity provider. My individuals can manage their own identity providers. So what providers should I be looking at. A provider already used by the individual is a good starting point. You probably already have an OpenID if you use any of a list of familiar web services. Well no: those providers may be good enough for the individual, but not necessarily the application that I want to use.

I've been playing with my neglected Plaxo account, which accepts OpenIDs for federated login. I wanted to see what it feels like using the different IDs.

My starting point was VeriSign's Personal Identity Portal (PIP), which I've been using for services like the Basecamp collaboration platform, which is favoured by a US client. Putting this into Plaxo reminded me about the irritation of using VeriSign PIP in Basecamp after closing the browser: you get sent to a page with no link asking you to log into PIP first. This prevents phishing and is a feature VeriSign call OpenID Sign In Security. The lack of link is annoying, but VeriSign appears to be secure - provided you don't leave your browser open and computer unattended in an insecure place. I bolstered my personal security by installing a certificate on my PC for 2-way SSL during authentication, but who's to know if other PIP identities are protected by Strong Authentication? PIP has phishing-resistant authentication policy or Provider Authentication Policy Extension (PAPE), which has to be a good thing. The VeriSign PIP OpenID URL is natural-looking https://username.pip.verisignlabs.com/, which I like. What I'm not sure about, is how PIP deals with browser certificate expiry etc; there is nothing about this in the FAQ.
The next obvious choice for me was Google Accounts. The Google Account OpenID URL is difficult to find out and then pretty complicate looking when you do find it. You can't find it out from your account pages at Google, you have to get it by OAuth. Rather than implementing this myself, I used Plaxo to identify it and then read the URL from the OpenID list in Plaxo. It looks like this: https://www.google.com/accounts/o8/id?id=XXXXXXXXXXXX and is definitely a cut, copy and paste job with a jumble of letters at the end. Google don't seem to subscribe to the idea of making their OpenID entirely open. So perhaps, we shouldn't be publicising it? I'm wary about security through obfuscation. I would like to see 2-way SSL working with Google Accounts and I'd like to see evidence of anti-phishing policy extensions.
I had a go with myOpenID, which is also a free identity provider, giving you an Open ID URL of the form https://username.myopenid.com/. There is a nice way to manage personas with myOpenID. However, I found the SSL certificate installation didn't work for me, so I can't vouch for its security. Also it doesn't report any policy extensions like anti-phishing, which is a disappointment. Verisign PIP seems like a better option.
My other experiments really came back to Google Accounts again. Being the owner of http://bimport.blogspot.com, I can use that same URL as an OpenID URL and use by Google Account to authenticate. Likewise, some bright spark has put a wrapper/manager onto Google Accounts and has therefore handled OAuth and you can use a natural looking but 2-week maximum OpenID URL http://openid-provider.appspot.com/emailaddress generated by the OpenID Provider application at appspot. If anything, these providers must lessen the Google Accounts security and can only really be seen as a low security convenience.

OpenID identity providers are not born equal and I need to get more familiar with Verisign PIP's browser certificate handling before I get too dependent on it. I use a VeriSign digital ID for e-mail signing and seldom encryption for which I pay $20 per annum for renewals; I was disappointed not to be able to use this for OpenID login. Initially, I didn't want to go the whole hog and carry around VIP hardware for authentication, but it makes sense of you live in more than one browser, if you access security-sensitive applications.

Implementing a PHP site that uses OpenID federated authentication is a snap using the OpenID library for PHP. I've yet to bite the bullet with Java, but I expect it is a similar deal.

Hidden DIVs do trick Google

2009-10-31T09:42:00.000-07:00

I thought Google was going to great efforts not to index content on hidden DIVs. Indeed, I was told that you are liable to be put into Purgatory by Google if hidden DIVs appeared in your page.

The thinking was presumably that purveyors of adult entertainment, who now get irrelevant keywords via (say) <meta content="meaning, life" name="keywords"/> ignored, should not start doing things like this:

<div style="visibility: hidden">
Meaning of life 
</div>

...and thus mislead innocent searchers.

So I was surprised to find a client whose competitor was putting my client's name into a hidden DIV and benefiting from searches on that name. The trick was simply to use an external CSS rather than inlining the style.

This is what a C++ person would call depending on undefined behaviour. I wonder if Google will one day take that competitor to task, or if my expectations are unrealistic? S-E-oh-ho-ho-ho.

Windows Server Core desktop equals MS-DOS

2009-10-20T11:44:00.000-07:00

I think I've found a desktop I'm comfortable with at last. I thought I was going to wait for Sam Mitchell to iron out the bugs in TSE for Linux and get comfortable with runlevel 3 with TSE, but perhaps I should hang in with TSE 4.40a and cmd.exe on the default Server Core UI for Longhorn.

Gnome doesn't float my boat. KDE has never really felt right. OSX brings out the inverted snob in me. Perhaps I never really grew out of MS-DOS and QEdit. I'm using Vista most of the time now but with a cmd.exe box open most of the time and TSE with everything that's important apart from drafting invoices, which doesn't seem right without the latest incarnation of MS Word.

I'll probably sign up for Windows 7 next week, for the sake of the invoices, but I'd be better off with Longhorn and TSE. I'm a Luddite, aren't I?

If something is timing out in a CGI script, something is wrong

2009-10-19T10:20:00.000-07:00

A colleague has been setting up FastCGI in MovableType to improve performance in a production site and there have been a number of problems to overcome.

The first one was related to leaving the FastCGI script idle for a long period of time. I've discussed there problem here already.

The problem we were left with related to publishing relatively complicated index templates in a blog. We had FastCGI set up thus:

FastCgiIpcDir /tmp/fcgi_ipc/
AddHandler fastcgi-script .fcgi
FastCGIConfig -singleThreshold 4 -idle-timeout 30 -killInterval 300 -maxProcesses 50 -appConnTimeout 0

We were finding that publishing the index files was taking longer than 30 seconds required by the idle-timeout parameter, and our FastCGI script was being unceremoniously killed before it had finished its job.

If you're wondering, the singleThreshold parameter handles the MySQL "gone away problem" I reported previously.

So, increase idle-timeout, duh?

Well no. 30 seconds is a reasonable timeout for an HTTP request, if a request takes longer than that, there is something wrong. Increase this timeout and you'll move the problem elsewhere; it could be a proxy server returning an HTTP/408 or the browser retporting an error or the user's patience snapping (hitting the F5 refresh in anger).

In this case the problem was that MT by default uses static publishing, which is synonymous with manual publishing (I think). With static publishing, the publishing process is part of the CGI request and the CGI request returns HTML to the browser, when the static content has been rendered to the files. This is convenient in that you know immediately when the publishing is complete, but it is inconvenient in that you have to wait for the publishing to complete and - indeed - that process could take more than 30 seconds.

In MT 4, background publishing was introduced. When you click your publish button, a request goes into TheSchwartz job queue to be serviced by a background process launched by cron on a Publisher. The Publisher runs run-periodic-tasks every couple of minutes. This way the CGI script does not have to hand around while the heavy-lifting of publishing is processed.

By making our complex index templates publish via the Publish Queue, FastCGI manages just fine and the user experience is as it ought to be. Desktop user interfaces are very comfortable with UI threads, making windowing systems responsive. We need to do the same thing with our CGIs with or without AJAX.

Aliases with parameters in C-shell

2009-10-12T05:16:00.000-07:00

A colleague was doing battle with an alias on his Mac. It was a relatively complicated alias, which needed a command line parameter.

As a C-shell ignoramus, I proposed a solution using a BASH function. The Bourne Again mantra is that aliases are out and shell functions are in.

Unfortunately, Mac users expect to use tcsh (a flavour of C-shell) as their shell and therefore do not have support for shell functions. I wanted to tell him to use a script, but that seemed like defeat, so I dug around with Google for a bit and came across this post. Lo and behold C-shell aliases do have support for command line parameters. The first parameter is: \!^. Obvious, huh? I don't think so.

I'm going to stick to BASH. C-shell looks like a can of worms for the uninitiated.

Exceptions in IOStreams

2009-10-12T04:10:00.000-07:00

A former colleague and good friend was using the following "fast copy" in C++:

#include <algorithm>
#include <fstream>
#include <iterator>
#include <iostream>

int main(int argc, char* argv[]) {

    std::ifstream fin(argv[1]);
    if (!fin) 
        return 1;

    std::ofstream fout(argv[2]);
    if (!fout) 
        return 2;

    std::istreambuf_iterator<char> iitr(fin), end;
    std::ostreambuf_iterator<char> oitr(fout);

    std::copy(iitr, end, oitr);
}

He was using this to copy a file to a memory stick and was irked that when the stick was full, the copy loop carried on failing to copy right the way to the end of the input; it continues blithely along.

It is only when you think of the likely copy algorithm implementation that you can see the problem with this approach.

The implementation is likely to expand to something like the following:

while (iitr != end)
           *oitr++ = *iitr++;

There is no provision for the exceptional circumstance that the destination iterator fails, because the loop condition simply depends in the source iterator.

So, why doesn't it break out of the copy implementation loop with an exception?

By default IOStreams do not throw exceptions, so the first thing you need to do is to enable them if you want them.

Here we enable exception-throwing on the whole gamut of IO states, reading from an input file:

#include <iostream>
#include <fstream>
#include <iterator>
using namespace std;

int main()
{

     try {
           ifstream file;
           //ios_base::iostate old_flags = file.exceptions();
           file.exceptions(ios_base::eofbit | ios_base::badbit | ios_base::failbit | ios_base::goodbit);
           file.open("test.txt");
           cerr << "Getting... ";
           istreambuf_iterator<char> iitr(file), end;
           while (iitr != end)
                   cerr << *iitr++ << ',';
           file.close();
           cerr << "\nExpect to see this, because no exception is thrown\n";
     }
     catch(ios_base::failure& exc) {
           cerr << '\n' << exc.what() << endl;
     }
}

The EOF condition does not throw, however, because we've been using the stream buffer directly. When you work directly with the stream buffers, you get fast and crude performance. The iterator, works at a lower level than the istream object, and as a consequence no exceptions get thrown when it hits EOF.

You can iterate with the istream object, however, if you use the higher level istream_iterator<char> instead of its more fashionable (?) lower level stream buffer counterpart istreambuf_iterator<char>.

This throws:

#include <iostream>
#include <fstream>
#include <iterator>
using namespace std;

int main()
{
     try {
           ifstream file;
           //ios_base::iostate old_flags = file.exceptions();
           file.exceptions(ios_base::eofbit | ios_base::badbit | ios_base::failbit | ios_base::goodbit);
           file.open("test.txt");
           cerr << "Getting... ";
           istream_iterator<char> iitr(file), end;
           while (iitr != end)
                   cerr << *iitr++ << ',';
           cerr << "\nDon't expect to see this, because an EOF exception will be thrown\n";
           file.close();
     }
     catch(ios_base::failure& exc) {
           cerr << '\n' << exc.what() << endl;
     }
}

So, what my colleague needs to have an IO failure in the output throw is the following:

#include <algorithm>
#include <fstream>
#include <iterator>
#include <iostream>

int main(int argc, char* argv[]) {
    std::ifstream fin(argv[1]);
    if (!fin) 
        return 1;

    std::ofstream fout(argv[2]);
    if (!fout) 
        return 2;

    // Throw as soon as it is unable to write to the memory stick
    fout.exceptions(std::ios_base::badbit);

    // Work with stream iterators (not buffer iterators) when you want to catch exceptions.
    // Where you won't want to catch exceptions, use the faster stream buffer iterators.
    std::istreambuf_iterator<char> iitr(fin), end;
    std::ostream_iterator<char> oitr(fout);


    try {
        std::copy(iitr, end, oitr);
    }
    catch (std::ios_base::failure& exc) {
        std::cerr << '\n' << exc.what() << std::endl;
    }
}

Going at a lower level than istream is not a panacea for performance.

MySQL "server has gone away"

2009-10-09T11:04:00.000-07:00

I've been bitten by MySQL connection has gone away problems before. Last time it was a JDBC connection pool in a Java daemon application that maintained connections to a MySQL server and/or a persistent connection in a PHP script in the same project. This time it was a FastCGI script for MovableType that was reported to be bombing out with an HTTP/500 server error every so often.

The problem in every case was that I was working with a system that was primed for high load, but it was encountering periods of sustained inactivity beyond - incredible though it seems - the 8 hour timeout, after which MySQL rejects idle connections.

The fix for a Perl script was not obvious. If you look through the FastCGI reference, there isn't an obvious parameter to ensure that the FastCGI is killed after a period of inactivity. The trick is to get the FastCGI script to restart regardless of the fact that there has been no apparent failure. That counts out the -restart parameter. It looks like we need to set -singleThreshold to be something more than 0, which keeps an idle instance running ad infinitum, and less than 100, which presumably causes the FastCGI processes to be dropped immediately.

If that is right it is much easier than the heart-beat fix we needed to apply to the Java application to keep the connection alive. To be on the safe side, we should timeout before the SO_KEEPALIVE, which is likely to be 2 hours for a TCP Socket; though we are dealing with a Unix Domain socket, so that shouldn't apply to out FastCGI Perl script.

YSlow compromise

2009-10-09T08:32:00.000-07:00

Firefox has a great add-in for web developers by Yahoo called YSlow. This makes you confront all of the bad things, which hand-on-heart you know you were doing wrongly in your web page by grading you. No one likes an E/F grade, do they?

The trouble with YSlow is that you soon come to realise that there are real world compromises you need to make - e.g. putting your static content onto an affordable CDN and having it served gzip-compressed to those who can accept it.

If your CDN is Amazon Cloudfront, which puts S3 onto edge servers, you can do clever things with individual files to get them to be delivered gzip-compressed, but you have to get very clever if you are going to handle files, which need to be served clients that do not accept gzip encoding. Since most modern browsers accept gzip encoding, I wonder if it is really worth making special provision for those that don't?

I'd find it hard to argue the case for going down that loathsome site best viewed with path just for the sake of gzip optimisation, but aren't we all demanding modern standards-compliant browsers anyhow, so can we ignore the old ones? Can I stick my neck out and safely compress all the static content without consulting the accept header?

Probably yes.

Personal firewall software like Zone Alarm clobber the accept header. However, I believe [Warning: Not tested!] they relay the gzip-compressed content nevertheless. I've not tested this to confirm, but I'm inclined to think that nowadays we can gzip and improve the experience of the masses at the expense of the few, delivering it on cheap edge servers via a CDN.

I wonder if we'll be penalised by Search Engines for adopting this approach? Googlebot requests with Accept-Encoding: gzip,deflate, so no problems there. It looks like Bing does too.

I put a post on Experts Exchange, and Simple CDN was recommended as an affordable alternative to Cloudfront. I probably ought to shop around more, but I want to try to stick with Amazon AWS, if I can, to keep my accounts uncomplicated.

Sticking with Cloudfront/S3, there is a simple and elegant solution provided in a post here. Bearing in mind a root index page can't be contrived in S3, we have to host a site index page in EC2 or equivalent. A server-side script or web application, can inspect the "accept encoding" header and redirect to (say) zip.mysite.com if gzip is accepted and flab.mysite.com, if it isn't. The static content is uploaded to each, gzipped on zip and uncompressed on flab. Hopefully, Google will not penalise for duplicate content. I hate having to worry about SEO.

Standards for PSPs

2009-10-01T05:07:00.000-07:00

I'm pitching for a job where a web site provides the front for a product that is going to have partner-distributors in a load of different countries. When someone buys a product from the site, they do so from their local distributor in a local currency with the distributor's shipping fees and price.

As each new distributor comes on board, he takes a region from the global distributor and then handles orders for all countries within his allocated region. Credit card payments go directly to the distributor's bank account. The global distributor is not in the loop.

To achieve this, we need to handle whatever PSP the partner is able to work with. The partner needs to have an acquiring bank account, which the PSP is willing and able to play ball with in a cost-effective way. Because this has to do with money movements, there is lots of haggling in the set-up of merchant account, what might look like a promising nearly globally applicable PSP turns out to be an expensive option.

Here in the UK, if (say) you have a NatWest/RBS Streamline acquiring account, your main PSP options are Sage Pay (formerly known as Protx), which is developer-friendly, or you the RBS group's own WorldPay, which is less so. PSPs naturally prefer to work with off-the-shelf shopping basket software presumably to avoid the hassle of dealing with developers for bespoke solutions. Off-the-shelf shopping basket software appear to be restricted to certain PSPs. The lazy developer is tempted to be drawn into a PSP lock-in. You soon find, though, that Sage Pay can only be used for UK acquiring accounts and WorldPay is really only competitive for UK acquiring accounts. I suspect that rates change and it is wise to be flexible, because partners may change their PSP because they change their acquiring bank. I'm faced with the prospect of dealing with more PSPs as partners come on board. I expect they are going to be local offerings in different countries.

It would be all well and good if there was a standard for interfacing with PSPs.

There are a number of different ways to work with PSPs:

You can do as much as possible on your site, including collecting the credit card details. The PSP is completely white-labeled, because you pass all the details it needs to work with via an API from your web site. I've done this sort of thing in the past with Verisign/Paypal's PayFlow Pro, and it isn't much fun; you need to have a secure site for handling those credit card numbers and you also have to get with it with PCI DSS.
You can set up the order, billing address and shipment address on your site and then pass those details and the buyer to the PSP to handle credit card input and payment. I like this approach, because the buyer needs only to trust the PSP with his credit card details. I am expecting to go this route, if the PSPs all play ball.
You can have the PSP handle the billing address and delivery address too. I am less keen on this approach, because there is less flexibility.

I'd like to see these approaches standardised, so that the same calling parameters can be used for all PSPs and the results are formatted in a standard way. Without standardisation and with the prickliness of dealing with financial institutions, the PSP integration is really irksome. I'd like to be able to visit the PSP site and see that it supports a standard interface for the second approach and not have to go through the rigmarole of applying for a test account with that PSP.

It should be possible, shouldn't it?

Uniform initialisation in C++

2009-09-20T02:59:00.000-07:00

Uniform initialization (to use the American spelling), is sure to make you start noticing initializer lists, when your bleeding edge colleagues start adopting some of the goodies in the next incarnation of the C++ standard.

The C++0x standard will have us initialising a vector of strings with an initializer list, using one of the following constructs:

std::vector<std::string> v = { "alpha", "beta", "gamma" };
std::vector<std::string> v({ "alpha", "beta", "gamma" }); 
std::vector<std::string> v{ "alpha", "beta", "gamma" };

I like the first form, because it looks like array initialisation and {} alreadylooks like a std::initializer_list<> to me. I'm comfortable with the expectation that my compiler will not construct a temporary vector and copy construct a vector or worse still default construct and assign a vector from that temporary. I expect elision.

The second form is more explicit: you know that it is constructing a vector<> from a std::initializer_list<>, but the first is more legible, because there is less puctuation.

It is the third form that is tough for a change-adverse C++ veteran to swallow, because it looks like a class declaration or a scope gone wrong. Thankfully regular functions, which take initializer list parameters do not have an equivalent of the third form, so why have it for an object construction?

Presumably because we should not expect the compiler to elide the assignment / copy construction, the second form is preferred over the first by the followers of Scott Meyers, who famously said we should "Prefer initialization to assignment in constructors", and the third form was therefore there to stop the next generation of C++ from coming to look like punctuation gibber.

Sorry Scott, but I prefer assignment in constructors. Nevertheless, I'll come round to the third form, as long as I'm not task-switching between C++ and Java projects. Undeniably, initializer lists are a good thing.

What no index page on S3?

2009-09-18T12:40:00.000-07:00

If you look at my drab company home page in Cloudfront, you have to look at http://cdn.seseit.co.uk/index.html because http://cdn.seseit.co.uk/ gives you an S3 bucket listing, when you make its ACL public read.

S3 users have been crying out for a solution for this for some time now. I should add my voice.

Scaling out copying ephemeral storage

2009-09-18T07:51:00.000-07:00

I'm still on my static publishing theme.

The situation is this:

I have a Page Server in EC2 which is reasonably up to date and is updating itself all the time from S3.
It may be one of several Page Servers in varying states of up-to-date-ness, using the trick I outlined in my last post to update their content from pages published by Publishers.
Page Servers serves static-ish content, by which I mean HTML with SSI and PHP. That's why the content isn't simply in S3.
I want to instantiate another Page Server, starting with a snapshot taken of the block storage used for the docroot. That way the new Page Server will be quick to catch up with the others, lagging behind only by the length of time it takes to instantiate and initialise the new EC2 instance.
I'm not using EBS for the docroot. I am saving money, using ephemeral storage, accessed via /mnt.

So how do I get a copy of [most of an] an existing /mnt onto a new instance without taking the Page Server off line for reads by Apache.

Here's how:

First find out how much data there is to copy over e.g.:

# On the running instance
rainbow1-ec2:~# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             147G   48G   92G  35% /mnt

With (say) Elasticfox (or EC2 command line tools), you will need to create an EBS volume which is a bit bigger than this to allow a backup/restore file to be copied onto it. This file will include some additional meta data. We might use a 50G volume in the example shown.

Attach the EBS volume to (say) /dev/sdp and create a file system on it:

# On the running instance
rainbow1-ec2:~# mkfs.ext3 /dev/sdp
rainbow1-ec2:~# mkdir -m 000 /mnt
rainbow1-ec2:~# mount -t ext3 /dev/sdp /mnt2

Stop data from being written to /mnt and flush writes. In my static publishing design, the writers are replication processes, running on the Page Servers copying data from S3. Having stopped the writers, you call sync. Then any cached writes will be flushed to disk and the disk will be ready for a low-level copy or backup:
```
# On the running instance
rainbow1-ec2:~# killall myreplicationprocesses
rainbow1-ec2:~# sync
```
You can stop accidental writes to /mnt, by making is temporarily read-only:
```
rainbow1-ec2:~# mount -o remount -o ro /dev/sda2
```
At this point, Apache should be able to read from the file system, but nothing should be writing to it. Any temporary directories should be kept out of /mnt for this approach to work.

We can now create a "backup" of /mnt onto the EBS volume:

# On the running instance
rainbow1-ec2:~# dump -0 /dev/sda2 -f /mnt2/dump.dat

# On the new instance
rainbow2-ec2:~# mkdir -m 000 /mnt2
rainbow2-ec2:~# mount -t ext3 /dev/sdp /mnt2
rainbow2-ec2:~# cd /mnt
rainbow2-ec2:~# restore -rf /mnt2/dump.dat

The running instance should be returned to read+write and the replication processes can be started again now:
```
# On the running instance
mount -o remount -o rw /dev/sda2
rainbow1-ec2:~# start.myreplicationprocesses 
```
The EBS volume should be unmounted before being detached, using Elasticfox or equivalent:
```
# On the running instance
rainbow1-ec2:~# umount /mnt2
```
The EBS volume should be attached now to the new instance. This is done with Elasticfox or equivalent. Let's say we use /dev/sdp again:
```
# On the new instance
rainbow2-ec2:~# mkdir -m 000 /mnt2
rainbow2-ec2:~# mount -t ext3 /dev/sdp /mnt2
```
The directory structure can then be "restored" onto the new instance before un-mounting, detaching and deleting the EBS volume:
```
# On the new instance
rainbow2-ec2:~# cd /mnt
rainbow2-ec2:~# restore -rf /mnt2/dump.dat
rainbow2-ec2:~# umount /mnt2
```

If we are trying to save money, you may wonder why we don't go via S3 rather than the temporary EBS volume. The answer is that we may not have enough space on the root file system.

My SimpleDB binlog for static publishing

2009-09-18T03:16:00.000-07:00

If you've come here expecting some sort of SimpleDB binlog to fulfill a MySQL-ish need like replication or rollback, you've come to the wrong place.

What I'm trying to achieve was outlined in my previous post. I want to publish PHP or HTML-that-uses-SSI in S3 and have that pulled onto Page Servers into ephemeral storage EC2, by which I mean the /mnt directory so that pages can be served from the local file system. I've not junked the idea of using an EBS volume, but /mnt makes sense as long as I can initialise new instances when I scale out in a timely manner.

I want Page Servers to pull content from S3 that they need to catch up with. I'm calling the replication list a binlog, because it is analogous to MySQL replication, when slaves need to catch up with the master by going through the master's binlog from the last known master status. No doubt this approach is used in a zillion other scenarios in CS and no doubt there is a better way to refer to this mechanism, but MySQL is a common shared experience, so let's use that as the analogy and call the list a binlog.

Here's what I need to achieve:

A Publisher (one of several masters) publishes a file and copies it to the safe storage of S3.
It then updates the binlog so that a Page Server can pull the content off S3 and store it in local ephemeral storage in its docroot in /mnt.
Each Page Server has a concept of something like a master status in the binlog and knows where to pick up from in its replication process.

There is an improvement, which I'd like to achieve in that a file - synonymous with a S3 object key - which appears twice in the binlog is only fetched once. Confused? Consider a page which is published and republished before a Page Server can fetch it. You only want to fetch it once, but Publisher(s) will have put it into the binlog twice in the interim.

You might be wondering why SimpleDB and why not SQS. SQS is fine for FIFO messaging, but doesn't handle different readers having different positions in the queue - to do that the ReceiveMessage would have to be able to support a request indicating where to pick up from, which it does not per the WSDL 2009-02-01.

So time for another back-of-an-envelope design:

We create a domain for the application's binlog
We implement a non-quite unique auto-increment in a unique item+attribute, with keyname called index and attribute called index. Every time this gets incremented the attribute is replaced.
Typically there will be one binlog entry per index value stored in an item, whose name corresponds to the index value at the time the entry was added.
We expect, however, that there will be items with multiple binlog entries, because SimpleDB's eventual consistency dicates that an unique auto-increment implementation equivalent to a primary index is impracticable.
Binlog entries have an item name that corresponds to the index and have a non-unique attribute named "s3-key", corresponding to the S3 bucket object key for the file to be replicated. That keyname corresponds to the docroot path of the published file.
Page Servers must be able to handle items with multiple "s3-key" attributes.

This falls within SimpleDB limits, as long as we do not wind up with greater than 1 billion files in the queue, which would be a sick publishing system! We are using a large number of items per domain and only one attribute per item, unless we want to add diagnostics to our binlog entries.

The SimpleDB items will need to be cleaned up, when they are older than (say) 48 hours (TBA). To accomplish this, the SimpleDB domain ought to have some additional house-keeping items for clean-up.

No NAS in EC2

2009-09-17T09:04:00.000-07:00

My current application is a content publisher. I need to publish static content in Amazon's EC2 cloud. I need to do so in a manner that scales out well.

Amazon's preferred handling for static content is S3 and, where content is served world-wide, its CDN CloudFront. I'd be very happy with that approach if I could wean my template designer colleagues from PHP. As long as the static content isn't really static, we cannot use S3 for page-serving.

Let's assume that PHP and SSI are order of the day, and we are therefore obliged to serve not-so-static content from an elastic farm of Apache "Page Servers" (Apache+PHP) in EC2. Note: S3 and therefore CloudFront does not support ESI, unlike Akamai, which is expensive; ESI is a CDN's equivalent of SSI.

The data centre precedent for this requirement is publishing to a shared directory on a fault-tolerant NAS. That works for PHP or HTML+SSI, but NFS exports aren't easy to contrive in the cloud. In EC2, we don't have a good NAS implementation. You could roll your own NFS export and implement your own fault tolerance, but it is a square peg for a round hole.

Let's do some soul-searching about the need for the NAS. Let's look at our demands, and see if we can wean ourselves off those beloved NFS shares.

When you are publishing static content in a site that predominantly serves blog content, you can build the following argument against using an NFS share in favour of duplicating content on all Page Servers:

Writes are relatively infrequent.
We are like a broadcaster. Our authors have relatively little to say, but we say the same things to everyone. That means that all content can be expected to be read often from all page servers.
Local storage is cheap and gives us the most efficient file I/O.
Having copies of a file on all instances of clients is wise because clients in the cloud are relatively volatile. This gives us redundancy, which is a Good Thing.
Repeatedly reading the same content from an NFS export (e.g. a NAS) is an inefficient use of bandwidth, when the content could be local.

What if all content servers could have their own copy of the content?

I'd first like to consider using the EC2 ephemeral storage, which EC2 Linux users affectionately know as /mnt. For many purposes has been supplanted by the relatively recent launch of EBS. I'd then like to consider EBS.

These scale-out thoughts are not limited to MovableType architecture, but since that's what I have in my text editor for now, lets consider content publishing in MT to give us a context. I want to instantiate as many Page Servers in a MT Advanced Configuration, as my not-so-static (by which I mean PHP) elastic demand requires. Since I am going to scale out with EC2 instances running Apache+PHP, I can sensibly use and initialise the ephemeral storage which I get with each of these instances. If an instance goes bad, the server and its locally mounted docroot are terminated together. I can also do likewise with a locally attached EBS volume (i.e. in the same availability zone as the EC2 instance).

Instantiating a Page Server that has its docroot on ephemeral storage (/mnt) goes as follows:

I instantiate the new EC2 Page Server instance.
I use rsync to synchronise its /mnt docroot with a known working Page Server instance. This is not all that quick, because it is done at the file system level, synchronising individual files.
I add the new Page Server to the load balancer when it is up top date.

My alternative is to instantiate new instances with the docroot in EBS as follows:

Keep the docroot on an EBS volume formatted with XFS so it can be conveniently frozen, when needs be. There needs to be one EBS volume per Page Server instance.
Freeze, snaphot and unfreeze an EBS volume attached and mounted on a known working Page Server instance. This is quick, being done at the block device level.
Create a copy of that volume in another instance of an EBS volume.
Instantiate a new Page Server.
Attach the new EBS volume to the new Page Server instance.
Attach the EBS volume.
Mount the EBS volume and start Apache on the new instance.
I add the new Page Server to the load balancer when Apache is started.

There are virtues in both approaches. EBS seems cooler, but there is a cost implication going that root. The virtue of block copying EBS volumes versus working with the file system in ephemeral storage depends on how quickly rsync is able to copy the docroot from a known working Page Server instance and how resource-hungry that process is. The ephemeral storage also needs to be large enough in the required EC2 instance type.

In either case, this gives is a Page Server with snapshot of the docroot which we expect to be somewhat out of date, by the time the server goes on-line. How can we keep Page Servers docroots synchronised?

MovableType handles this in the application tier, using the Schwartz publishing queue to send files to Page Servers. At the time of writing there are some gremlins in that procedure, but the idea is that a Publisher has the responsibility for copying its published file to all Page Servers.

There are obstacles that need to be overcome:

When a Page Server grabs a snapshot, it should register itself as a recipient for the Publishers, so it can subscribe to new updates. This creates a problem for Publishers, if it is not immediately available for file copy (a problem when the docroot is in an EBS mount rather than ephemeral storage). The registration process itself isn't part of an MT installation - this is a configuration setting requiring - (does it?) - a restart to take effect. It would need some proper engineering.
When a Page Server becomes unavailable, the source Publisher has to handle timeout etc, before skipping it. Again, in MT this would require some work.

I prefer the idea that the Page Server is responsible for fetching its content. This scales better.

Let's come up with a back-of-an-envelope design:

The Publisher has pulled off a publish request from the Schwartz PQ. It has published content to a local file system and is just about to mark the job as done.
Before disposing of the job, MT currently puts another request into the PQ to get the file RSync'ed, but we know that approach is fragile. Either we need it to put the file somewhere safe and to send a message message all Page Servers to get them to collect it, or we need to make it practicable for the publish request to be reinstated in the Schwartz queue in the event that the Publisher becomes unavailable when a Page Server wants to fetch the page...
Page Servers fetch the file, when they know it has been created or modified.

I prefer the former approach in 2, using safe storage, because I can see race conditions if things get unstable and re-publish requests are order of the day. In the AWS world, safe storage means S3. If our published content is in S3, it will not need republishing in the event of a Publisher outage.

Here's a bum steer, but bear with me:

We can mount a PersistentFS to turn an S3 bucket's objects into a block storage device. This allows one EC2 instance to write to the file system and many to read from it and the I/O is efficient. A Publisher can publish directly to the PersistentFS. These file systems will come at a price in the near future (the only cost is S3's at the time of writing), but this should still be competitive with taking an NFS approach with EBS for persistence.

In the AWS world, "messaging" typically means using SQS, but that really only works for 1-to1 messaging. We need something more like a MySQL binlog, which Page Servers can slave to, where each Publisher is a master with a master status and each Page Server is a slave. Using an approach whichis analgous to MySQL replication, we can have Page Servers act as PersistentFS slaves. Every time a Publisher publishes a file, its path is put into the binlog, which is also on the PeresistentFS filesystem. Page Servers, keep a track of entry number and binlog file number just like MySQL slaves keep a track of offset and binlog file. Polling the mtime in the binlog is a simple way to see that it has been appended.

So, each Publisher should have a PersistentFS to write to. When the file is written, it is reliably written in S3 and a "binlog" is updated. Like MySQL binlogs can only be XX days old maximum and Publishers are responsible for purging old ones. Each Page Server is a read only client, copying files to its ephemeral storage, which it uses as docroot.

Bah! I didn't see this in the small print:

PersistentFS does not support changing or writing to a file system from one computer while it is mounted read-only on another computer.

What a shame. It isn't going to be as simple as I hoped. It looks like we cannot use S3 as a block device for this after all. We need to copy published files individually into it and we need to refine the binlog idea, because that's lost its home now.
Here's my proposed approach (revised):

Use local ephemeral storage (/mnt) for publishing files. When the file is written, it is written into persistant storage in S3 and a binlog is updated. Only then is the publishing job marked as done.
The binlog is not a file. Rather it is a log record in Amazon's SimpleDB. Not sure about the details yet.
Like MySQL, binlog files (or SimpleDB values) can only be XX days old maximum and Publishers are responsible for purging old binlogs.
Page Servers poll the for binlog updates. They then walk through the new entries copying the files from S3 to the local ephemeral file system on /mnt.

Let's re-visit the argument in favour of NFS:

NFS is easy to set up. You mount the same docroot in Publishers and Page Servers.
Scaling out is simply a matter of adding read-only Page Servers or read+write Publishers, which point to the same shared directory.

Now that the PersistentFS idea has been scrapped, neither the Page Servers nor the Publishers need to mount anything more than the usual ephemeral storage. So this should be easier or system administration.

The only thing that needs to be worked on is the SimpleDB. Conceptually a binlog does not need to be synonymous with an individual publisher, because all Publishers write to the same S3 bucket and SimpleDB binlog. We just need to see this working with key-value pairs.

Ideally the static content that can remain in S3 - content that doesn't use SSI and is not PHP - should remain there. Mixing that content with the content served by the EC2 Page Servers requires some discipline (e.g. a name like static.page-server.website.com mapping to the CloudFront CNAME), but it will result in the best content delivery.

This is sketchy I know... but it looks like the start of a plan.

SQS WSDL upgrade

2009-09-12T05:35:00.000-07:00

I was wrongly alarmed that support for the WSDL implementation, which I am currently depending upon in a web service in distributed environment was at its end of life. The alarm was raised by an e-mail from AWS reminding me about the end of life of an old version of the WSDL for its SQS web service, which provides robust message queues.

...we'd like to provide an update on the "end-of-life" schedule for WSDL versions 2006-04-01 and 2007-05-01. As previously communicated, Amazon SQS users will have until November 6, 2009 to complete their migration to WSDL version 2009-02-01 or 2008-01-01, after which the old WSDL versions will no longer be available.

Foolishly I read that November was the end of the line for the 2008-01-01 WSDL, so I panicked. Yes I know, they were perfectly clear in that notice, but I'm human. One day I may have to confront the upgrade though, so perhaps there is a lesson to learn.

I use two clients for SQS. My Java applications use typica 1.3 and my Perl scripts use an old version of the Perl Library for Amazon SQS. These both date back to the 2008-01-01 WSDL version of the service and create queues, which require a client that respects that version of the WSDL.

Amazon have written up some notes about migrating to the new version of the WSDL. We cannot expect a 2009-02-01 client to play ball with a 2008-01-01 queue, so migrating to 2009-02-01 needs to be a concerted process, which isn't easy in my application's production environments.

I have a number of 2008-01-01 queues in production, which are kept fairly busy. I was curious to see how an upgrade to the 2009-02-01 Perl library would cope with one of these queues. I was annoyed that the API had changed slightly. The change seems unnecessary. With a couple of small edits to a Perl script to send a message to the queue, I was ready to go, and ready to get a status 400 internal error. That was no great surprise, because Amazon say:

Queues created with the previous WSDL versions cannot be accessed with version 2008-01-01 or 2009-02-01, and queues created with version 2008-01-01 or 2009-02-01 cannot be accessed with the previous versions. During the migration period, you can use the previous WSDL versions to send requests to queues created with previous WSDL versions, and you can send requests using the new versions to queues created with the new versions.

So, I wanted to know what would happen with typica 1.6 in my Java application. I was pleasantly surprised to find that the typica upgrade works fine with a 2008-01-01 queue. I'll keep testing but as long as I don't create any queues with typica 1.6 during the migration, I may be able to upgrade all of my Java wars and SiteSuite libraries and continue to plot along with 2008-01-01 and then upgrade my Perl scripts at a leisurely pace. Or perhaps I should replace my Perl scripts with BeanShell scripts and standardise on typica.

Coordinating distributed environments with different SysAdmins in each environment is awkward and something to be avoided is possible. Wherever you can depend on backward compatibility, you problems go away. 1 : nill to the Java library this time.

September 18th update: It transpires that typica 1.6 needs you to us a different namespace for the 2009-02-01 WSDL, with the existing namespace you invoke the 2008-01-01 WSDL. So it it no cleverer than the Perl API. It would be nice if someone could write a high level wrapper for these libraries to get the right WSDL for the right queue. Having said that, Amazon ought perhaps to be doing that themselves.

MovableType publishing in EC2 with dedicated Publishers and RSync

2009-09-11T05:54:00.000-07:00

I've been thinking about scaling out MT publishing again.

A really satisfying way to publish static content would be to put it out onto S3, and I've posted thoughts about an approach which would do that in the MT forums. That's an ambitious goal and ultimately a good one, but let's walk before we run and first consider publishing to MT Page Servers in EC2. It is something we have to do anyhow, unless we can convince MT template authors to avoid resorting to generating PHP, where client-side JavaScript would suffice.

Rather than publish on a NAS which creates a single point of failure headache in Amazon's cloud, you might choose to synchronise published folders on page servers with Unison. That approach is fine for a system which does a limited amount of publishing, but feels wrong for a busy MT system and I expect will prove to benchmark poorly. I haven't tested this, but Unison crawls through the entire directory tree, which has got to be expensive if the tree is substantial. Let's take it as read that Unison is expensive for a large blog server deployment.

It is much better to use a message queue which has been tipped off about published pages, and what better queue to use for that purpose than one that already exists? MT queues jobs in the Schwartz message queue, and there is a hook already in place to get it to do so to get rsync to broadcast published files. See Byrne Reese's MT Operations Manual about this (look for RSync).

Let's consider this in a scaled out HA deployment. MT describe an advanced configuration where Publishers are separate from Page Servers. That is a good model to work from for a deployment with elastic scaling.

A Publisher publishes a file to its local file system. A Publisher may be doing this because it picked up the publishing job from the Schwartz message queue in the first place. It could be a dedicated host handling publish requests from the queue. Alternatively the Publisher may be local to the App Server, because the blog has been set up to use static (i.e. immediate publishing). It is good to support static publishing as well as queued publishing, so let's assume we have to deal with either. Let's assume that comments are always published via the queue, though. Otherwise, we need also to consider a Publisher that is synonymous with a Comment Server too.

So do the rsync request needs to be fielded by the server that has done the publishing, since the file is going to be local to its file system? That would mean that run-periodic-tasks needs to be run on all potential Publishers and MT would need to be designed to have Rsync requests picked up by all Publisher servers. That would be an elaborate and expensive way to defer a job on the local machine. Happily, I believe that the design is cleverer than that. But I'm seeking some clarification from Byrne Reese on this to confirm.

The job processor - a Publisher that subscribes to run-periodic-tasks to pull job requests from the Schwartz queue (as apposed to a static Publisher) - is [** I believe **] able to use an rsync source from a server other than itself. The rsync destinations (SyncTarget) are the Page Servers. We only really care that content gets to Page Servers. Publishers in an Advanced Configuration do not need to have a complete set of the published content. Page Servers do.

So what steps are involed in scaling out? How do we add Page Servers to the EC2 configuration?

The first thing to do is to set up the new Page Server as a SyncTarget in all units which are configured to run-periodic-tasks in their crontabs. The Publishers that run-periodic-tasks subscribe to the Schwartz message queue and publish static content that has been queued. Similarly, but not as the same job (because content may be published immediately) these units may pick up rsync requests for specific files. We need to know that new published content is being moved onto the new Page Server before it goes on line.
The next thing to do is to rsync the entire content of the docroot from another Page Server.
Then add the new Page Server to the load balancer.

Page Server AMIs should be set up with the rsync package pre-installed along with MT. Publishers also need rsync to fetch files from Publisher sources.

Update 12th September: It looks like there is a design flaw in MovableType 4.31 with respect to the RSync implementation in that it assumes that the file has been published locally. For a scaled out EC2 system with no NAS, this is a problem. I'm going to see if the RSync request in the Schwartz job queue has (or can be easily made to have) the host name of the host which published the file and then... perhaps... work on a patch to fetch the file from that unit, if an EC2 scale-out is what my client wants.

Linux bind mounts to EBS in EC2 and MySQL

2009-09-10T10:31:00.000-07:00

I've been making extensive use of bind mounts lately for setting up a persistent database in EC2, which appears to be in the root file system. Following a recommendation by the brilliant Eric Hammond, I put my MySQL data and config files onto an XFS-formatted EBS volume so that it persists.

To minimise changes to set-up files, I moved the /etc/mysql, /var/lib/mysql and /var/log/mysql directories to {EBS-MOUNT}/etc/mysql, {EBS-MOUNT}/var/lib/mysql and {EBS-MOUNT}/var/log/mysql and re-created /var/lib/mysql and /var/log/mysql as empty directories, which I bind-mounted to the directories in the EBS volume. This is the same approach that Schlomo Swidler takes in his excellent article about attaching an EBS volume at start-up. However, I made the mistake of putting my mounts in the ephemeral storage on /mnt, which was a bad plan with hindsight, because /mnt itself is mounted separately on /dev/sdb and isn't something you bundle with an AMI.

I noted my set up procedure, but it occurred to me that unlike soft symlinks to directories, directories which are bind-mounted are not all that obvious when you are trying to remember which you handled in that way, and you don't want to dig through notes.In a wiki by Chris Siebenmann, I found an article explaining that /etc/mtab shows the bind mounts, but it isn't main-stream system administration.

I was curious to see what would happen in /etc/mtab was bundled in an AMI, which was booted without the benefit of Schlomo's recipe for attaching an EBS on start-up. What would retro-fitting the volume entail.

My MySQL EBS volume was mounted in /mnt/mysql. I don't recommend this, because the /mnt directory is itself mounted in ephemeral storage. The volume concerned was attached to a High CPU extra large 64-bit c1.large Debian Lenny EC2 instance with a MySQL slave set up with server-id = 2. To instance another MySQL slave, I was going to have to create a snapshot and change that ID in a fresh EBS volume created from that snapshot. But first, I needed to create an AMI for the MySQL master/slave, because I hadn't already done this. The initial set-up had been a quick response to a hosting emergency.

I followed Eric Hammond's familiar procedure for creating HA AMI for a dedicated master/slave 64-bit instance. With the MySQL config in my master or slave EBS volume and the root filesystem simply pointing to these, there ought to be no differences between the a master and slave AMI.

It occurred to me, however, that when I was bundling my AMI, I needed to exclude the very directories that I was concerned about. Otherwise, the bundling process would bundle /etc/mysql, /var/lib/mysql and /var/log/mysql into the AMI unaware of the fact that this data lives in the EBS volume.

So here's my bundle volume command:

sudo -E ec2-bundle-vol           \
-r $arch                       \
-d /mnt                        \
-p $prefix                     \
-u $AWS_USER_ID                \
-k /root/.ec2/pk-*.pem         \
-c /root/.ec2/cert-*.pem       \
-s 10240                       \
-e /mnt,/tmp,/root/.ssh,/root/.ec2,/var/log/mysql,/var/lib/mysql,/etc/mysql

Unlike Eric Hammond, I've been keeping AWS authentication keys in /root/.ec2, but otherwise it it only /etc/mysql, /var/lib/mysql and /var/log/mysql that needs special handling.

I hate having special handling, though. It made me think again about my use of bind-mounts. Had I been less driven by fashion, I'd simply have used soft symlinks for these directories. Directories have to use soft symlinks and links between mounts have to be soft anyhow.

Had I used soft symlinks:

I wouldn't have obfuscated things and ls -l or find . -type l would have shown the symlinks and made it clear what I was doing. AN Other would have had a better time with that than having to rummage around in /etc/mtab to see bind mounts.
I wouldn't have needed a special procedure for the bundle volume, cleverly excluding my bind mounts, because soft symlinks wouldn't have been followed and the MySQL data and configuration files on my EBS volume wouldn't have been bundled into the AMI, if I omitted the exclude.

So I ask myself, What good reason is there to use bind mounts, when good old soft symlinks will do?

Using Unison for a HA docroot instead of NFS

2009-09-09T06:40:00.000-07:00

I thought that synchronising a docroot between two "masters" in an HA web site, would require running rsync at both ends. There is something ugly about that, and like master-master replication in MySQL it is bound to be inefficient.

I'm not quite sure why I overlooked unison. There is a succinct article written by Pablo, which put me straight and I like this approach. Unison is genuinely bidirectional and easy to install from package managers.

My mindset is this:

I have a database-driven site, and only the database really needs to be bang up to date. I've scaled "up" rather than out my database, because I can get away with avoiding clustering and my database can cope with lots of web page server connections.
Static content published in the docroot of my web page servers ought to be current, but it can lag by 5 minutes on different nodes.
I'm really not keen on using NFS, because you are stuck with something expensive if you aren't going to wind up with a single point of failure. I'd rather publish locally and using synchronisation to keep all of my nodes up to date with published content.
Content that is published can have a crude conflict resolution. If a file is published simultaneously on two nodes, the nodes should prefer their local copy. It isn't a big problem than one node may be out of date, because the next update is bound to propagate anyhow.

With this mindset, I have a quick any simple Unison profile, which prefers local copies when there are conflicts and otherwise updates writes on all nodes. The strategy isn't good enough for a database, but it is fine for published content.

So unlike my database, I scale "out" my web page servers and avoid the misery of NFS.

Sharing a DEV environment in AWS

2009-09-07T10:56:00.000-07:00

If you are familiar with Amazon's Web Services and its flagship service EC2, you'll be familiar with the plethora of public AMIs which can get you kicked off with a pre-installed system in the cloud.

Someone very clever has hopefully gone to the trouble of creating a base installation of your favourite Linux distro – say Debian Lenny 5.0. That gives you the bare bones of a system, including a package manager, which lets you add the packages which your application depends upon. Someone else may have gone to the trouble of taking that AMI and added packages to it so that it is now (say) basic Debian based web server with PHP support or with Tomcat for Java. They then bundle and register another AMI. If they make their AMI public or make it accessible to you through an ACL, you have something nicer to start with for your project.

Something you very soon come to realise, though, is that your system is volatile. While your instance is up and running, you may build upon your application and its data, but when it is terminated, the fruit of your efforts is lost. Your instance runs on commodity hardware, and you should not depend on the hardware not to crash.

So, if your application is worth its salt, you should re-bundle and register your own AMI from it. It is a relatively big deal doing this and you don't really want to do this too frequently. The AMI is stored in Amazon's robust S3 simple storage service. You can set up an ACL to make it accessible to as many or as few people as you want and distribute it to your clients, colleagues or collaborators in the general public in that way. That way you don't have to re-install all of your dependencies when you re-launch your AMI. This is fine but the process can take ~30 minutes and it is tempting to fail to save changes when things take that long.

If your data is of value, you can save it independently in an S3 bucket. And make it accessible to as many or as few as you like, using ACLs.

If you have a lot of data files, which you want to preserve, you should consider mounting a directory with persistent storage.

There are two alternative approaches:

You can set up a PersistentFS persistent file system in S3. This is fairly quick to set up. Data persists in S3 objects as tarballs representing the block device - i.e. you don't see individual files in S3, so this is not a convenient way to publish static data. These S3 objects are efficiently written to from a cache file which is local to the EC2 instance. This is efficient for most purposes but inefficient for I/O intensive data - e.g. database.
You can set up an EBS volume for data, which is local to the EC2 instance region, but which has a lifetime which is independent of any EC2 instance. EBS volumes are block devices, which may be attached to any one EC2 instance in the same region, and can be high level formatted with (say) an ext3 or XFS file system. When the attached EC2 instance is terminated, the EBS block persists, but it exists as a single instance and should therefore be backed up onto S3 is the data is valuable. There is a convenient mechanism for creating "snapshots" of EBS volumes into S3. These can be converted back into new EBS volumes in any region you choose.

The relative merits of each approach are covered by PersistentFS.com in a forum post.

For a development environment, EBS seems like a natural mount to work with, because things change around a lot and creating snapshots of EBS volumes is rather like committing changes to a repository, when things are stable. You can create snapshots of versions.

A very clever extension to this approach is presented in an excellent article by Shlomo Swidler, where he proposes mounting an EBS volume with a root file system on it. Linux has the ability to change its root file system after booting. Shlomo's article was written after this idea was mooted and expanded upon in a developer discussion, which he credits in his article. Shlomo describes how to make a self-contained EBS, with your complete development system, which is "pivot-booted" from a "bare bones" AMI. Schlomo's preferred poison is the Alestic Ubuntu AMI, and he creates the bootable EBS volume and a pivot-boot AMI from it. I created a bootable EBS volume from an Alestic Debian AMI from his instructions, but struggled to get my pivot-boot working. Rather than persevering with my Debian pivot-boot, though, I've been using N Martin's public pivot-boot AMI from nimlabs, which is described in the developer discussion. I don't care which distro N Martin uses for the pivot boot AMI (actually it is Ubuntu Hardy), because my distro prepared using Shlomo's instructions applied to Debian Lenny is boot-strapped from my EBS volume. I do care that N Martin set up his pivot-boot to expect a root file system in /dev/sdj, because that's where I need to attach my bootable EBS volume.

So I now have my entire DEV environment on an EBS volume, which Amazon charges $0.10 per GB month for. I minimise my EC2 usage by terminating EC2 when I go home. When I go on holiday or want to put a project on ice, I create a snapshot and kill my EBS volume, because data is more durable in S3 than on an EBS volume. It also costs less in S3, because despite being charged $0.15 per GB month, the volume is compressed and S3 snapshots store only incremental changes.

So I want my EBS volume to be as small as possible and I have the lack of foresight to start with a 1G volume. My project dependencies increase - my package manager has been busy - and I outgrow that volume. What do I do now?

I do the following in Elasticfox (a Firefox extension):

I shut down my EC2 instance to create a snapshot of my 1G EBS volume. Why shut it down to create a snapshot? Because it is a root file system and I don't want to create a snapshot it while it is busy or the snapshot will be unbootable. If it is XFS, xfs_freeze cannot be used on the root file system, and if it is ext3 freezing is complicated and is bound not to work on a root file system anyhow. My EBS volume is high level formatted with an ext3 file system. That means I need to un-mount the volume before creating a snapshot, and that means terminating the EC2 instance.
I then create the snapshot.
I create a 2G EBS volume from the snapshot.
I start up an EC2 instance. This doesn't need to be a pivot boot instance, because we are going to use it to grow the ext3 file system; it should not boot from the 2G EBS volume at this point. Rather than choosing my usual Debian Lenny 5 base AMI, I pivot-boot off the 1G EBS volume, which I attach as /dev/sdj (N Martin's preference).
When the instance is up, I attach the new 2G volume to it as /dev/sdk – i.e. something other than /dev/sdj.
I don't mount /dev/sdk. I work with the EBS block device as follows:
1. I check the ext3 file system on the block device: e2fsck -f /dev/sdk
2. I grow the file system to fill the 2G volume: resize2fs /dev/sdk
I can terminate my EC2 instance now, though I don't need to do this to create a snapshot of the "grown" 2G volume to supersede the snapshot of the 1G volume, because the EBS volume is not mounted and therefore is not busy.
I detach the 2G volume from /dev/sdk on the EC2 instance, if it is still running.
I use EBS volumes created from a 2G snapshot henceforth and attach it to /dev/sdj of N Martin's pivot boot AMI instance.

Now I have a 2G development system for my project, a procedure for growing it if needs be and a procedure for creating robust snapshots in S3.

I need to share my development system and I hit an irritating omission in AWS's current system. Unlike the manifests and parts of AMIs, EBS snapshots are not regular S3 objects that appear in buckets which you specify. That means you cannot set up ACLs to allow specific users or the general public to access your snapshots. This seems arbitrary and I'm sure Amazon will address this oversight, but in the interim you have to do the following to share your EBS snapshot with others so that you can share your development system, using Elasticfox to work EC2 and S3Fox to work with the S3 buckets and object ACLs:

Important... Make sure that your root file system doesn't contain anything that you do not want to share. If you have your authentication details in /mnt, you are OK. However, if they are in /root/.ec2 on the booted EBS, you need to beware that this will be distributed, and should be moved away before creating a snapshot. You will be fine if you followed Shlomo's advice: "The certificate and private key credentials should be copied to the instance in the /mnt partition so they don't get bundled into the image and copied to the bootable EBS volume."
You need to start an EC2 instance, if you do not already have one running. The instance does not need to be a pivot boot, but you will want to install some S3 tools onto it to upload the file to S3: a great choice is s3cmd. If you have this on your development system, you can launch that with the pivot boot as usual – you will be using a 2^nd EBS volume if you do.
You should create an EBS volume from your latest snapshot.
Attach the EBS volume to (say) /dev/sdk
Copy the contents of that block device into a file in /mnt, using dd. If it is a 2G volume, you will create a 2G file: dd if=/dev/sdk of=/mnt/x.dat
Compress the file - this compressed to 90% of its size for me: gzip /mnt/x.dat
Upload the compressed file into an S3 bucket.
Set up the ACL in the S3 bucket so the file may be read by everyone, who you want to read it.

The person wishing to use your snapshot should:

Instantiate a bare bones EC2 instance
Attach a new 2G EBS on (say) /dev/sdk
Fetch the x.dat.gz from your bucket, using (say) s3cmd. A good destination for this is /mnt.
Uncompress: gunzip /mnt/x.dat.gz
Copy the contents of the uncompressed file onto the block device: dd if=/mnt/x.dat of=/dev/sdk
Terminate the EC2 instance or detach the EBS volume from it
Start an instance of the N Martin's public pivot boot AMI ami-2feb0f46 nimlabs/pivot-sdj-20080824.manifest.xml
Attach the EBS volume as /dev/sdj
Wait for it to boot – just like you

All of this is I expect blindingly obvious to Shlomo et al, and I hope the EBS sharing requirement will soon becomes obsolete (I know that Schlomo is working on the Amazon folk to that effect), but that's my recipe for now for sharing a nifty development system without sharing your AWS account.

A production system is an altogether different kettle of fish.

Update: 24th September

EBS volumes may now be shared, so no need to mess around directly with S3.

Here's what they now say:

Snapshots can be shared using AWS Management Console or using API calls. You have full control over what accounts you share each snapshot with, including the option to share it with the entire AWS community.
AWS Management Console:
Log in to the AWS Management Console, click the Amazon EC2 tab and click Snapshots on the left navigation pane
Right-click on the snapshot you wish to share and select Snapshot permissions
Add the AWS account numbers of the developers who you want to grant access or share it publicly
Hit Save to apply the permissions

More on this here.

That makes a shared DEV environment more of a snap, doesn't it?

NFS always bites you

2009-09-03T15:00:00.001-07:00

When I was working for the forerunner of a company that runs managed services for e-mail filtering, the system administrators hated NFS with a vengeance. No doubt that was because they literally lost sleep when alerts systems indicated that NFS has gone down in one data centre or the other.

Once again, I've found myself trying to figure out why three servers in a production cluster have terrible update times. Once again, it was NFS that was letting us down. If my understanding was right, the NetApp cluster that's used as a NAS was rebooted at some time and no one noticed that rpc.statd had died on those hosts. My guess is that rpc.lockd gets in knickers in a twist about lock recovery and that somehow makes the NFS client go slowly for writes.

I'm certainly not a SysAdmin, and no doubt my read of the situation is naïve, but one way or another I reckon that we are paying the price of the simplicity of NFS.

Why isn’t this MovableType?

2009-09-03T03:49:00.001-07:00

I've been working extensively with MovableType, which ticks a lot of boxes for me.

These are the main points for me:

My primary client uses MT as its primary CMS. [That is a good enough reason on its own.]
It scales nicely, publishing static content.
It allows a software developer to get his hands dirty, using Perl, which I like for irrational reasons.

So why aren't I using this now?

Well... Google provides free hosting and it is good to see other technologies, if I'm going to improve what I do with MT.

I am for example trying to get AtomPub to work properly, and was curious to try out the Microsoft Word Client. This post comes from a Word 2007 blog post. The integration with Blogger looks to be mature, and I've had teething problems with MT, using a Perl client. This post should be proof of concept for Word 2007 AtomPub implementation – not that I can see a compelling reason to use that as a UI under normal circumstances!

As part of my experiment, I want to see if adding this paragraph in Word - having previously published the entry - makes the post update rather than appending a new entry. If this get's repetitive, we have a thumbs down!

My apprenticeship

2009-09-03T03:25:00.000-07:00

My first IT employer described what he did as the trailing edge of technology. Trail blazers had beaten down a path and made massive profits from being the first to provide the world with the dubious benefits of IVR technology for premium rate telephone services. His aim was to mop up the more complicated work, which they didn't have the time or inclination to tackle and perhaps do simple things better. He did very well at that, and my apprenticeship was a privileged one. He was conservative about change and a strong believer in KISS.

He was still using the 1987 TurboC compiler in the mid '90s, because he has such an in-depth understanding of its foibles. You could fit the compiler, an editor, Norton Utilities and a copy of LANtastic on a 720K diskette, which fit nicely into your pocket (unlike the 360K ones, which corrupted if you sneezed), and you were set up to trouble-shoot any problems in the early '90s, as long as you were dealing with DOS.

I failed utterly to convince him about the virtues of C++ and the benefits of OOP. While I have subsequently prospered adopting new technologies and trusting new libraries, which he would have steered clear from, I still hear echoes of his disdainful wisdom, when adopting a new technology for the sake of being fashionable or short-sighted expediency causes a project to collapse.