pelure

Generating Source Files with Leiningen

2013-10-28T00:00:00Z

Recently, we needed to include some generated source files in a project. The source code generation was project specific, so we didn’t want to have to create a leiningen plugin specifically for it. To get this to work required using quite a few of leiningen’s features.

This post will explain how to use lein to customise you build to generates a source file, but many of the steps are useful to implement any form of lein build customisation.

The Generator

The source code generator is going to live in the my.src-generator namespace. Here’s an example, that just generates a namespace declaration for the my.gen namespace under target/generated/my/gen.clj.

(ns my.src-generator
  (:require [clojure.java.io :refer [file]]))

(defn generate []
  (doto (file "target" "generated" "my" "gen.clj")
    (-> #(.getParentFile) #(.mkdirs))
    (spit "(ns my.gen)")))

Development only code

The source generation code should not be packaged in the jar, so we place it in dev-src/my/src_generator.clj, and add dev-src and the generated source directories to the :dev profile’s :source-paths. The :dev profile is automatically used by leiningen unless it is producing a jar file. When producing the jar, the dev profile will not be used, so dev-src will not be on the :source-path (we add the generated directory to the base :source-path below).

:profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}}

Running project specific code with leininingen

The run task can be used to invoke code in your project. To use lein’s run task we need to add a -main function to the my.src-generator namespace.

(defn -main [& args]
  (generate))

In the project.clj file we also tell lein about the main namespace. In order to avoid AOT compilation of the main namespace, we mark it with :skip-aot metadata.

:main ^:skip-aot my.src-generator

Customising the jar contents

The generated files need to end up in the jar (and possibly be compiled), so we put them on the :source-paths in the project. If we had wanted to include the sources without further processing, we could have added the generated directory to :resource-paths instead.

:source-paths ["src" "target/generated"]

Extending the build process

Now we can tell lein to generate the source files whenever we use the project. We do this by adding the run task to the :prep-tasks key. Leiningen runs all the tasks in :prep-tasks before any task invoked by the lein command line.

The tricky bit here is that the run task will itself invoke the :prep-tasks, so we want to make sure we don’t end up calling the task recursively and generating a stack overflow. To solve this, add a gen profile, and disable the prep tasks in it. We use the :replace metadata to ensure this definition takes precedence. See the leiningen profile documentation for more information on :replace and it’s sibling :displace.

:gen {:prep-tasks ^:replace []}

Then use this profile when setting the :prep-tasks key in the project.

:prep-tasks [["with-profile" "+gen,+dev" "run"]  "compile"]

Now when we run any command, the sources are generated.

Adding an alias

Finally we may want to just invoke the source generation, so let’s create an alias to make lein gen run the generator. We need the gen profile for this, or otherwise the generator will run twice.

:aliases {"gen" ["with-profile" "+gen,+dev" "run"]}

The final project.clj

For reference, the final project.clj looks like this:

(defproject my-proj "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :source-paths ["src" "target/generated"]
  :main ^:skip-aot my.src-generator
  :prep-tasks [["with-profile" "+gen,+dev" "run"]  "compile"]
  :profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}
             :gen {:prep-tasks ^:replace []}}
  :aliases {"gen" ["with-profile" "+gen,+dev" "run"]})

Conclusion

This required using many of lein’s features to get working - hopefully you’ll find a use for some of them.

Alembic Reloads your Leiningen project.clj Dependencies

2013-08-29T00:00:00Z

You’re working away in a Clojure REPL, when you realise you need to add a dependency. You add the dependency to your leiningen project.clj file and then? Instead of shutting down your REPL, loosing whatever state you have built up, you can use Alembic to load the new dependencies. Simply call (alembic.still/load-project).

Of course, it still has to work within the confines of the JVM’s classloaders, so you can only add dependencies, and not modify versions or remove dependencies, but this should still cover a lot of use cases.

To use alembic on a single project, simply add it as a dependency in your :dev profile in project.clj:

:profiles {:dev {:dependencies [[alembic "0.2.0"]]}}

To make alembic available in all your projects, and it to the :user profile in ~/.lein/profiles.clj instead:

{:user {:dependencies [[alembic "0.2.0"]]}}

Alembic also allows you to directly add dependencies without editing your project.clj file, using the distill function. Use this if you are just exploring libraries, for example.

Finally a big thank you to Anthony Grimes and the other flatland developers for removing classlojure’s dependency on useful, which should make this all much more robust.

Evaluate and Format Clojure in Emacs Markdown Buffers

2013-08-26T00:00:00Z

When writing documentation or blog posts about Clojure code, it is very useful to be able to format Clojure code blocks using clojure-mode and evaluate code with nrepl.el.

This can be enabled using mmm-mode, which allows a single buffer to use different major modes for different sections of the buffer (and is not limited to just web modes). Install mmm-mode using M-x package-install mmm-mode, or using M-x el-get-install mmm-mode from the excellent el-get, or by checking the project from github and installing manually.

To configure this for clojure and markdown, add this in your init.el or .emacs file.

(require 'mmm-auto)
(mmm-add-classes
 '((markdown-clojure
    :submode clojure-mode
    :face mmm-declaration-submode-face
    :front "^```clj[\n\r]+"
    :back "^```$")))

(setq mmm-global-mode 'maybe)
(mmm-add-mode-ext-class 'markdown-mode nil 'markdown-clojure)

After evaluating the above, or restarting emacs, you can test multi-mode support by opening a markdown document, or creating a new one, and adding a clojure source block, e.g.:

```clj
(defn my-fn [x]
  (inc x))

(my-fn 1)
```

Inside the code block you can format and evaluate your code as in any clojure-mode buffer, and the code will display exactly as in a .clj file. By default the evaluation uses a running inferior lisp process, which you must start yourself. To use a running nrepl session instead, use M-x nrepl-interaction-mode inside the code block.

Using with AsciiDoc

This technique is not limited to clojure and markdown, but could be made to work whenever you would like differing major modes in distinct parts of your Emacs buffers. You can add class to mmm-mode appropriately, for as many major mode combinations as you need. The regions for each major mode are detected using regular expressions (or by some function).

For example, if you’re writing asciidoc, you might use:

(mmm-add-classes
 '((asciidoc-clojure
    :submode clojure
    :face mmm-declaration-submode-face
    :front "\\[source, clojure\\][\n\r]+----[\n\r]+"
    :back "^----$")))
(mmm-add-mode-ext-class 'adoc-mode nil 'asciidoc-clojure)
(mmm-add-mode-ext-class 'doc-mode nil 'asciidoc-clojure)

Summary

mmm-mode allows you to flexibly use multiple major modes in different parts of a single emacs buffer. Here we have shown how to use it for clojure-mode code blocks in markdown or asciidoc, but it is in no way limited to this, and it allows some fine grained customisation to the appearance and behaviour of each major mode block. I’m sure you’ll find your own uses for mmm-mode.

Snarf PGP Keys from Signed Messages in Emacs mu4e

2013-08-25T00:00:00Z

I just moved to mu for reading my email. One feature I was missing was the ability to receive PGP keys for signed messages.

When you receive a signed message, mu shows the verification status in the Signature field in the message view (see MSGV-Crypto). If you don’t have the sender’s PGP key on your keyring, this will show unverified. Click on the Details link within field will show the sender’s key id. To manually import the key you can use gpg:

$ gpg --recv <the-key-id>

This seemed a little labourious, so some automation was in order. mu4e allows you to define actions that can be run on messages (or attachments), so I just wrote an action to do this.

(defun mu4e-view-snarf-pgp-key (&optional msg)
  "Snarf the pgp key for the specified message."
  (interactive)
  (let* ((msg (or msg (mu4e-message-at-point)))
          (path (mu4e-message-field msg :path))
          (cmd (format "%s verify --verbose %s"
                 mu4e-mu-binary
                 (shell-quote-argument path)))
          (output (shell-command-to-string cmd)))
    (let ((case-fold-search nil))
      (when (string-match "key:\\([A-F0-9]+\\)" output)
        (let* ((cmd (format "%s --recv %s"
                            epg-gpg-program (match-string 1 output)))
               (output (shell-command-to-string cmd)))
          (message output))))))

This works by parsing the output of the mu program itself, as displayed in the Details window, to obtain the PGP key id. It then executes the gpg --recv command, parsing in the parsed key id.

To install the action, we simply add it to mu4e-view-actions:

(add-to-list 'mu4e-view-actions
             '("Snarf PGP keys" . mu4e-view-snarf-pgp-key) t)

How to Build Clojurescript Libs with JavaScript Dependencies

2013-08-16T00:00:00Z

Using JavaScript dependencies in a Clojurescript library seems to be hard. It took me many hours to understand how it should work. A big thanks to Chas Emerick for setting me straight on most of this.

Luke Vanderhart posted a general introduction to using Javascript libraries in Clojurescript. Go read it if you haven’t already - this post assumes you have.

While that post is an excellent description of using JavaScript in a Clojurescript application, it doesn’t really address JavaScript in Clojurescript libraries, which has the additional problem of how to ensure the JavaScript dependency is available in the consumer of the library. A Clojurescript library should definitely be capable of providing it’s dependencies, but should also allow the consumer to override the version of these dependencies.

Don’t package the JavaScript

The first approach is to simply not provide the JavaScript at all. This is the approach taken by jayq for example. The consumer of jayq, or any library that uses jayq, is required to provide jQuery through the JavaScript runtime. This can take the form of a <script> link in the browser page, or a call to webPage#injectJs in phantomJS. The compile :libs or :foreign-libs options can not be used to provide the dependency, as there is no way for the compiler to know that jayq depends on the namespace provided by these options.

For the consumer of the library to use compiler:optimizations other than :whitespace, they will need to provide an :externs file.

Package JavaScript

The second approach is to package the JavaScript via a Clojurescript namespace. This involves adding a require on a namespace to the code that directly depends on the JavaScript, and arranging for that Clojurescript namespace to load the JavaScript, using either of the compiler:libs or :foreign-libs options.

The Clojurescript library can make the JavaScript library available in its resources. The library consumer can then use resource via the :libs or :foreign-libs options, depending on whether or not the JavaScript contains a goog.provides call.

If the library is packaged with a goog.provides call, then the consumer can not replace the version using :libs [""] - the use of an explicit prefix in :libs is needed to prevent more than one JavaScript implementation trying to provide the clojure namespace, or the use of :foreign-libs where the namespace is explicitly mapped.

For examples, the pprng library packages its dependency with a goog.provides call, allowing the use of :libs [""] to pull in the dependency. The papadom library on the other hand provides vanilla javascript dependencies, and requires the use of the more verbose :foreign-libs option.

If the JavaScript is to be provided in the runtime, then the consumer will have to provide an empty namespace definition to satisfy the require in the Clojurescript library, and the:externs file as in the first case.

Postscript

There are several assumptions in much of the documentation that I didn’t see explicitly explained. I’ll record these here for posterity.

A clojurescript library is always a source code library. There is no such thing as the linking of compiled clojurescript artifacts.

Neither:libs nor:foreign-libs actually changes how the JavaScript is accessed within the clojurescript code. If you include jQuery via a :libs, and a require, you still access it through js/jQuery. The require of the namespace specified by goog.provide, or the namespace specified in the :foreign-libs’ :provides key, simply ensures the JavaScript is loaded.

The choice of compiler :optimizations affects what information you need to provide, and this differs depending on whether you are providing javascript libraries through the runtime (e.g. $lt;script> tags in the browser), or through :libs or :foreign-libs compiler options. The simplest here is to use the compiler options. When providing the JavaScript via the runtime, then everything should also just work if you are using no optimisation, or just :whitespace, but as soon as you try anything else, you will need to provide an :externs definition for the JavaScript libraries.

Exploring a todo app with core.async

2013-08-15T00:00:00Z

We’re going to build an equivalent of the AngularJS TODO example using core.async, and a templating library, papadom, that I’ve written to help in this.

Clojurescript recently gained a CSP implemetation via core.async, similar to Go’s channels, or CML’s channels (CML also has a nice select). Bruce Haumann started exploring this with ClojureScript Core.Async Todos, and David Nolen has been looking at how to use core.async for responsive design. In this post, we’ll take the TODO example, and take it a little further.

Basic Display

We’ll start with just displaying a list of todo items. For this we’ll need a template, so we’ll just write this in HTML, and add a t-template attribute, which enables us to use mustache style templating of values to display. This doesn’t use mustache sections for looping, in order to preserve valid HTML markup.

<h1>TODOS</h1>
<ul class="unstyled">
  <li t-template="todos">{{text}}</li>
</ul>

To get this to show something we’ll need some code:

(ns app
  (:require
   [papadom.template :refer [compile-templates render]]))

(defn start []
  (compile-templates)
  (render {:todos [{:text "learn papadom" :done false}
                   {:text "write a papadom app" :done false}]}))

When you call app.start() from the page containing the above template, you’ll see a list of two todo entries.

Adding an event

Now we have something displayed, lets add a checkbox to mark todo items as done:

<ul class="unstyled">
  <li t-template="todos">
    <input type="checkbox" t-prop="done" t-event="done"
           t-id="index" index="{{@index}}">
    <span>{{text}}</span>
  </li>
</ul>

The t-prop attribute tells the template to which data value to display as the checkbox.

The t-event attribute specifies that we want an event. When the checkbox is clicked, we will get a core.async message with a :done event type. We need to know which todo was clicked, so we use the t-id attribute to list the attributes whose values should be sent as the event data – in this case the index, which has a value based on handlebars style @index property.

Now we need some code to process the events. To do this we’ll define an app function that will be passed a state atom containing a map with our todos state, and a core.async channel from which to read events. The function will loop over events, and dispatch them as required.

(defn app
  [state event-chan]
  (go
   (loop [[event event-data] (<! event-chan)]
     (case event
       :done
       (let [nv (boolean (:checked event-data))]
         (swap! state assoc-in [:todos (:index event-data) :done] nv)))
     (recur (<! event-chan)))))

When the app function receives a :done event, it will update the state atom appropriately. Now we have our state updating, we’ll need to display it, which we can again do with the render function.

(defn show-state [state]
  (render "state" state))

We still need to get show-state called, and we’ll arrange this in a modified start function. This will create an atom for the state, and add a watch on the atom that will call show-state.

(defn start
  []
  (let [event-chan (chan)
        state (atom nil)]
    (compile-templates)
    (template-events event-chan)
    (add-watch state :state (fn [key ref old new] (show-state new)))
    (reset! state {:todos [{:text "Learn papadom" :done false}
                           {:text "Build a papadom app" :done false}]})
    (app state event-chan))))

We’ve also added a core.async channel, event-chan, which we’ve passed to template-events to arrange delivery of the events defined in our template. We pass this channel to the app function to start processing the events.

This shows the basic structure of the application.

Adding New Todo Elements

To allow you to add new todo items, we’ll add a form to our template, specifying a t-event attribute, which will cause an event to be sent when the form is submitted, with the form’s input values as the event data.

<form t-event="add-todo">
  <input type="text" t-prop="text" size="30" placeholder="add new todo here">
  <input class="btn btn-primary" type="submit" value="add">
</form>

To process this new event, we’ll add a case to the app function loop’s case form.

:add-todo
(swap! state update-in [:todos]
       conj {:text (:text (input-seq->map event-data))
             :done false})

This uses the input-seq->map helper to convert the data from the form into a map, and we extract the :text value (defined by t-prop in the input element).

And we’re done. To see a full working example have a look at the template and code in the todo example of papadom. To run the example:

git clone https://github.com/hugoduncan/papadom.git
cd papadom/examples/todo
lein ring server

Configuration in Templates is not Configuration as Code

2010-10-04T00:00:00Z

If you have configuration that uses template configuration files, you are practising neither configuration as code nor configuration as data. Having configuration locked away in template files reduces its visibility, and makes it hard for you to query it. It might be easier to write configuration code to use templates, but it will come back to bite you.

One of the first things I implemented in pallet was a templating mechanism, because configuration management tools use templates, right? I even built a template selection mechanism, just like Chef's.

I have come to realise however, that having configuration in template files is not particularly useful. There are three major problems you are likely to encounter. Firstly template files are not visible, secondly you can not query the data in the template files, and lastly you will need to touch multiple files to add or modify parameters.

Visibility at the point of usage is important, especially in a team environment. If you have to find the template file and look at its content when reading your configuration code, then the chances are you assume it hasn't changed, and skip the contents. Making an analogy to the world of software development, templates are like global variables in one sense. You can change the operation of a program with a global variable modified in some obscure place, and in the same way, you can change your system configuration by changing a template file, tucked away in some folder, and not visible from where you are actually calling your configuration crate/recipe.

The ability to query configuration settings allows not just finding out, for example, which directory a log file is in, but also enables you to put tools on top of your configuration data. Template configuration files suffer on two counts here - they are separate text files that require parsing to be read, and the format of each configuration file is different.

The last point concerns the flexibility of your configuration. If you have used template files, with hard coded parameter values, and you then want to modify your configuration to dynamically set one of those hard coded values, you have to modify all the specialised versions of the existing templates, and specify the value in code. You have to touch multiple files - lots of room for making typos.

My goal for pallet then, is to have all configuration supplied as arguments to crates. For most packages a hash map is sufficient an abstraction for providing the data, but when this gets too cumbersome, we'll use a DSL that mirrors the original configuration file language.

Goodbye hidden configuration!

Configure Nagios using Pallet

2010-08-18T00:00:00Z

Basic Nagios support was recently added to pallet, and while very simple to use, this blog post should make it even simpler. The overall philosophy is to configure the nagios service monitoring definitions along with the service itself, rather than have monolithic nagios configuration, divorced from the configuration of the various nodes.

As an example, we can configure a machine to have it's ssh service, CPU load, number of processes and number of users monitored. Obviously, you would normally be monitoring several different types of nodes, but there is no difference as far as pallet is concerned.

We start by requiring various pallet components. These would normally be part of a ns declaration, but are provided here for ease of use at the REPL.

(require
  '[pallet.crate.automated-admin-user
    :as admin-user]
  '[pallet.crate.iptables :as 'iptables]
  '[pallet.crate.ssh :as ssh]
  '[pallet.crate.nagios-config
     :as nagios-config]
  '[pallet.crate.nagios :as nagios]
  '[pallet.crate.postfix :as postfix]
  '[pallet.resource.service :as service])

Node to be Monitored by Nagios

Now we define the node to be monitored. We set up a machine that has SSH running, and configure iptables to allow access to SSH, with a throttled connection rate (six connections/minute by default).

(pallet.core/defnode monitored
  []
  :bootstrap [(admin-user/automated-admin-user)]
  :configure [;; set iptables for restricted access
              (iptables/iptables-accept-icmp)
              (iptables/iptables-accept-established)
              ;; allow connections to ssh
              ;; but throttle connection requests
              (ssh/iptables-throttle)
              (ssh/iptables-accept)])

Monitoring of the SSH service is configured by simply adding (ssh/nagios-monitor).

Remote monitoring is implemented using nagios' nrpe plugin, which we add with (nagios-config/nrpe-client). To make nrpe accessible to the nagios server, we open the that the nrpe agent runs on using (nagios-config/nrpe-client-port), which restricts access to the nagios server node. We also add a phase, :restart-nagios, that can be used to restart the nrpe agent.

Pallet comes with some configured nrpe checks, and we add nrpe-check-load, nrpe-check-total-proces and nrpe-check-users. The final configuration looks like this:

(pallet.core/defnode monitored
  []
  :bootstrap [(admin-user/automated-admin-user)]
  :configure [;; set iptables for restricted access
              (iptables/iptables-accept-icmp)
              (iptables/iptables-accept-established)
              ;; allow connections to ssh
              ;; but throttle connection requests
              (ssh/iptables-throttle)
              (ssh/iptables-accept)
              ;; monitor ssh
              (ssh/nagios-monitor)
              ;; add nrpe agent, and only allow
              ;; connections from nagios server
              (nagios-config/nrpe-client)
              (nagios-config/nrpe-client-port)
              ;; add some remote checks
              (nagios-config/nrpe-check-load)
              (nagios-config/nrpe-check-total-procs)
              (nagios-config/nrpe-check-users)]
  :restart-nagios [(service/service
                    "nagios-nrpe-server"
                    :action :restart)])

Nagios Server

We now configure the nagios server node. The nagios server is installed with (nagios/nagios “nagiospwd”), specifying the password for the nagios web interface, and add a phase, :restart-nagios, that can be used to restart nagios.

Nagios also requires a MTA for notifications, and here we install postfix. We add a contact, which we make a member of the "admins" contact group, which is notified as part of the default host and service templates.

(pallet.core/defnode nagios
  []
  :bootstrap [(admin-user/automated-admin-user)]
  :configure [;; restrict access
              (iptables/iptables-accept-icmp)
              (iptables/iptables-accept-established)
              (ssh/iptables-throttle)
              (ssh/iptables-accept)
              ;; configure MTA
              (postfix/postfix
               "pallet.org" :internet-site)
              ;; install nagios
              (nagios/nagios "nagiospwd")
              ;; allow access to nagios web site
              (iptables/iptables-accept-port 80)
              ;; configure notifications
              (nagios/contact
              {:contact_name "hugo"
               :service_notification_period "24x7"
               :host_notification_period "24x7"
               :service_notification_options
                  "w,u,c,r"
               :host_notification_options
                  "d,r"
               :service_notification_commands
                 "notify-service-by-email"
               :host_notification_commands
                  "notify-host-by-email"
               :email "my.email@my.domain"
               :contactgroups [:admins]})]
  :restart-nagios [(service/service "nagios3"
                     :action :restart)])

Trying it out

That’s it. To fire up both machines, we use pallet’s converge command.

(pallet.core/converge
  {monitored 1 nagios 1} service
  :configure :restart-nagios)

The nagios web interface is then accessible on the nagios node with the nagiosadmin user and specified password. Real world usage would probably have several different monitored configurations, and restricted access to the nagios node.

Still to do…

Support for nagios is not complete (e.g. remote command configuration still needs to be added, and it has only been tested on Ubuntu), but I would appreciate any feedback on the general approach.

Mocking Clojure Functions with Atticus

2010-05-18T00:00:00Z

I dislike most mocking, and try and avoid it as much as possible. Sometimes it is however the only realistic way of testing. I did a quick survey of mocking tools in clojure, and found them very much reflecting the Java mocking libraries. Clojure has a few more dynamic capabilities than Java, so I thought a little about how these could be used to make a simple mocking facility, and atticus is what I came up with.

There is a consensus that mocking should be implemented by binding a function’s var to a new function for the duration of a test, and atticus does this too. Atticus’ premise is that we can do simple mocking by declaring the mock function as a local function definition, and have the local function do the argument checking, return value setting, etc. The simplest case would be something like below, which checks the value of its argument and specifies a return value:

;; pull in namespaces
(use ‘clojure.test)
(require ‘atticus.mock)

;; define test which mocks f
(deftest mock-test
  (atticus.mock/expects
    [(f arg “Check argument”)
       arg)] ; set the return value
    ;; in a real test case this would be called
    ;; indirectly by some other function
    (is (= 1 (f 1)) “Call mocked function”))

At the moment, I have added two macros to this. once, which checks a function is called once, and times, which checks that a function is called a specific number of times. The macros are used to wrap the body of the mock function, which keeps the function’s expected behaviour in one place.

;; define test, that should be called just once
(deftest mock-test
  (atticus.mock/expects
    (f arg “Check argument”)
         arg))) “Call mocked function”))

;; define test, that should be called exactly twice
(deftest mock-test
  (atticus.mock/expects
    (f arg “Check argument”)
         arg))) “Call mocked function”)
    (is (= 1 (f 1)) “Call mocked function”))

So what do you think, is this a reasonable approach? Not having the explicit calls to returns, etc, might be seen as a loss of declarative clarity, but I for one prefer this, as it gives you the full power of the language to test the arguments and set the return value.

References

Provisioning Cloud Nodes with Pallet

2010-05-12T00:00:00Z

I recently needed to move a server from dedicated hosting to a cloud server. The existing server had been configured over time by several people, with little documentation. I want to make sure that this time everything was documented, and what better way than doing that than using an automated configuration tool.

Looking around at the configuration tools, I couldn't find one I really liked, so I started Pallet. I'll explain why I didn't use an existing tool below, but first I wanted to show how to manage nodes in Pallet.

  ;; Pull in the pallet namespaces
  (require 'pallet.repl)
  (pallet.repl/use-pallet)

  ;; Define a default node
  (defnode mynode [])

  ;; Define the cloud account to use
  (def service
    (compute-service "provider" "user" "password"
                      :log4j :ssh))

  ;; Create 2 nodes
  (converge {mynode 2} service)

This example would create two nodes (cloud vm instances) with the tag "mynode" in your cloud account, as specified in the service. This would give you the smallest size, ubuntu image on most clouds. Of course, to do anything serious, you would want to specify the image you would like, and you would probably like some configuration of the nodes. So carrying on the above example:

  ;; Pull in the needed crates
  (use 'pallet.crate.automated-admin-user)
  (use 'pallet.crate.java)

  ;; Define a new node that will use the Java JDK
  (defnode javanode
    [:ubuntu :X86_64 :smallest
     :os-description-matches "[^J]+9.10[^32]+"]
    :bootstrap [(automated-admin-user)]
    :configure [(java :openjdk :jdk)])

  ;; Create a new node, and remove the previous ones
  (converge {javanode 1 mynode 0} service)

This would stop the two nodes that were created before, and create a new one, with the specified ubuntu version. On first boot, it would create a user account with your current username, authorize your id_rsa key on that account, and give it sudo permissions. Every time converge is run, it also ensures that the openjdk JDK is installed.

The configuration to be applied is specified as a call to a crate - automated-admin-user and java in the example above. Crates are just clojure functions that specify some configuration or other action on the nodes (they're equivalent to Chef's recipes, which Pallet can also execute using chef-solo). Pallet can be extended with your own crates, and crates can specify general actions, not just configuration. lift is a companion to converge, and can be used to apply crates to existing nodes (including local VM's). The hypothetical example below would execute my-backup-crate on all the "mynode" nodes.

  (defnode mynode [] :backup [(my-backup-crate)])
  (lift mynode service :backup)

This was just a quick overview of Pallet, to give you an idea of what it is. One big area of Pallet not demonstrated here is its command line tool. But that is a topic for another post.

Why Write another Tool?

Now you've seen some examples, I'll try and explain the features that make Pallet distinct from other configuration tools out there.

No Dependencies

The machines being managed require no special dependencies to be installed. As long as they have bash and ssh running, they can be used with pallet. For me this was important - it means that you can use pretty much any image out there, which is great for ad-hoc testing and development.

No Server

Pallet has no central server to set up and maintain - it simply runs on demand. You can run it from anywhere, even over a remote REPL connection.

Everything in Version Control

In pallet, all your configuration is handled in SCM controlled files - there is no database involved. This means that your configuration can always be kept in step with the development of your crates, and the versions of the external crates that you use.

Jar File Distribution of Crates

Custom crates can be distributed as jar files, and so can be published in maven repositories, and be consumed in a version controlled manner. Hopefully this will promote shared crates.

Provisioning, Configuration and Administration

Pallet aims quite wide. You can use it for starting an stopping nodes, for configuring nodes, deploying projects and also for running administration tasks. To be honest, this wasn't an initial design goal, but has come out of the wash that way.

Interested?

Hopefully this has whetted your appetite, and you'll give pallet a try. You can get support via the Google Group, or #pallet on freenode irc.

Shell Scripting in Clojure with Pallet

2010-05-03T00:00:00Z

Let's face it, many of us hate writing shell scripts, and with good reason. Personally, it's not so much the shell language itself that puts me off, but organising everything around it; how do you deploy your scripts, how do you arrange to call other scripts, how do you manage the dependencies between your scripts? Pallet aims to solve these problems by embedding shell script in clojure.

Embedding in Clojure

Embedding other languages in lisp is not a new idea; parenscript, scriptjure (which Pallet's embedding is based on), and ClojureQL all do this.

So what does shell script in clojure look like? Some examples:

(script
  (ls "/some/path")
  (defvar x 1)
  (println @x)
  (defn foo [x] (ls @x))
  (foo 1)
  (if (= a a)
    (println "Reassuring")
    (println "Ooops"))
  (println "I am" @(whomai))

which generates:

ls /some/path
x=1
echo ${x}
function foo(x) {
ls ${x}
 }
foo 1
if [ \\( \"a\" == \"a\" \\) ]; then echo Reassuring;else echo Ooops;fi
echo I am $(whomai)

The aim is to make writing shell script similar to writing Clojure, but there are obvious differences in the language that limit how far that can be taken. To run the code above at the REPL, you'll have to use the pallet.stevedore package.

Escaping back to Clojure

Escaping allows us to embed Clojure values and expressions inside our scripts, in much the same way as symbols can be unquoted when writing macros.

(let [path "/some/path"]
  (script
    (ls ~path)
    (ls ~(.replace path "some" "other)))

We can now write Clojure functions that produce shell scripts. Writing scripts as clojure functions allows you to use the Clojure namespace facilities, and allows you to distribute you scripts in jar files (which can be deployed in a versioned manner with maven)

(defn list-path [path]
  (script
    (ls ~path)
    (ls ~(.replace path "some" "other)))

Composing scripts

Pallet allows the scripts to be combined. do-script concatenates the code pieces together.

(do-script
  (list-path "path1")
  (list-path "path2"))

chain-script chains the scripts together with '&&'.

(chain-script
  (list-path "path1")
  (list-path "path2"))

checked-script finally allows the chaining of scripts, and calls exit if the chain fails.

(checked-script "Message"
  (list-path "path1")
  (list-path "path2"))

Conclusion

Writing shell script in Clojure gives access to Clojure's namespace facility allowing modularised shell script, and to Clojure's packaging as jar files, which allows reuse and distribution. The ability to compose script fragments leads to being able to macro-like functions, such checked-script, and you could even use Clojure macros to generate script (but I haven't thought of a use for that, yet).

The syntax for the embedding has arisen out of practical usage, so is far from complete, and can definitely be improved. I look forward to hearing your feedback!

UPDATE: stevedore now requires a binding for *template*, to specify the target for the script generation. This should be a vector containing one of :ubuntu, :centos, or :darwin, and one of :aptitude, :yum, :brew.

Swank Clojure gets a Break with the Local Environment

2010-03-31T00:00:00Z

Recently I got fed up with a couple of warts in swank-clojure, so I made a couple of small fixes, and that lead to a couple of new features. Using SLIME with Clojure has never been as smooth as using it with Common Lisp, and the lack of debug functionality beyond the display of stack traces is particularly onerous. Recently, George Jahad and Alex Osborne’s debug-repl showed the possibility of adding a break macro to enter the debugger with the call stack intact and local variables visible. This functionality is now in swank-clojure.

Consider the following example, adapted from debug-repl:

  (let [c 1
        d 2]
    (defn a [b c]
      (swank.core/break)
      d))
  (a "foo" "bar")

Running this now brings up the following SLDB debug frame:

BREAK:
  [Thrown class java.lang.Exception]

Restarts:
 0: [QUIT] Quit to the SLIME top level
 1: [CONTINUE] Continue from breakpoint

Backtrace:
  0: user$eval__1666$a__1667.invoke(NO_SOURCE_FILE:1)
  1: user$eval__1670.invoke(NO_SOURCE_FILE:1)
  2: clojure.lang.Compiler.eval(Compiler.java:5358)
  3: clojure.lang.Compiler.eval(Compiler.java:5326)
  4: clojure.core$eval__4157.invoke(core.clj:2139)
  --more--

As you can see, the stack trace reflects the location of the breakpoint, and there is a CONTINUE restart. Pressing "1", or Enter on the CONTINUE line, or clicking the CONTINUE line should all cause the the debugger frame to close, and the result of the function call, 2, to be displayed in the REPL frame.

Enter, or "t", on the first line of the stacktrace causes the local variables to be displayed:

BREAK:
  [Thrown class java.lang.Exception]

Restarts:
 0: [QUIT] Quit to the SLIME top level
 1: [CONTINUE] Continue from breakpoint

Backtrace:
  0: user$eval__1666$a__1667.invoke(NO_SOURCE_FILE:1)
      Locals:
        b = foo
        d = 2
        c = bar
  1: user$eval__1670.invoke(NO_SOURCE_FILE:1)
  2: clojure.lang.Compiler.eval(Compiler.java:5358)
  3: clojure.lang.Compiler.eval(Compiler.java:5326)
  4: clojure.core$eval__4157.invoke(core.clj:2139)
  --more--

Pressing enter on one of the local variable lines will pull up the SLIME inspector with that value. If you go back to the REPL without closing the SLDB frame, there will be no prompt, but pressing enter should give you one. The local variables are then all be avaiable for evaluation form the REPL.

Should an error occur while you are using the REPL, you will be placed in a nested debug session, with an "ABORT" restart to return to the previous debug level.

Finally, restarts are now displayed for each of the exceptions in the exception cause chain.

  (let [c 1
        d 2]
    (defn a [b c]
      (throw (Exception. "top" (Exception. "nested" (Exception. "bottom"))))
      d))
  (a "foo" "bar")

This will bring up the debugger with 2 cause restarts, which can be used to examine the related stack traces.

top
   [Thrown class java.lang.Exception]

Restarts:
  0: [QUIT] Quit to the SLIME top level
  1: [CAUSE1] Invoke debugger on cause  nested [Thrown class java.lang.Exception]
  2: [CAUSE2] Invoke debugger on cause   bottom [Thrown class java.lang.Exception]

Backtrace:
   0: user$eval__1752$a__1753.invoke(NO_SOURCE_FILE:1)
   1: user$eval__1756.invoke(NO_SOURCE_FILE:1)
   2: clojure.lang.Compiler.eval(Compiler.java:5358)
   3: clojure.lang.Compiler.eval(Compiler.java:5326)
   4: clojure.core$eval__4157.invoke(core.clj:2139)
  --more--

The break functionality is known only to work from the REPL thread at the moment. With that small proviso, I hope you enjoy the new functionality - at least it provides a basic debug functionality until full JPDA/JDI integration is tackled.

Benchmarking Clojure Code with Criterium

2010-02-19T00:00:00Z

I have released Criterium, a new project for benchmarking code in Clojure. I found Brent Broyer's article on Java benchmarking which explains many of the pitfalls of benchmarking on the JVM, and Criterion, a benchmarking library in Haskell.

The main issues with benchmarking on the JVM are associated with garbage collection, and with JIT compilation. It seems from Broyer's articles that we can mitigate the effects but not completely eliminate them, and Criterium follows his advice. Both of the above libraries use the bootstrap technique to estimate mean execution time and provide a confidence interval, and Criterium does likewise. At the moment the confidence intervals are biased and I still need to implement BCa or ABC to improve these.

One of the functions that I wanted to benchmark originally involved reading a file. Criterium does not yet address clearing I/O buffer cache, and I am not sure on the best way forward on this. On Mac OS X, the purge command can be used to clear the caches, and on Linux this can be achieved by writing to /proc/sys/vm/drop_caches. On the Mac at least, this causes everything to grind to halt for about five seconds, and there are then some file reads as whatever processes are running read things in again. This sort of behaviour doesn't lend itself to inclusion in a timing loop... Any suggestions?

A Clojure library for FluidDB

2009-09-13T00:00:00Z

FluidDB, a "cloud" based triple-store, where the objects are immutable and can be tagged by anyone, launched about a month ago. As a another step to getting up to speed with Clojure, I decided to write a client library, and clj-fluiddb was born. The code was very simple, especially as I could base the library on cl-fluiddb, a Common-Lisp library.

I have some ideas I want to try out using FluidDB. It's permission system is one of it's best features, together with the ability to use it for RDF like triples means that it could provide a usable basis for growing the semantic web. My ideas are less grandiose, but might take as long to develop, we'll see...

Setting up clojure and compojure with maven

2009-09-06T00:00:00Z

I wanted to experiment with building a webapp using Clojure, so I tried setting up the Compojure web framework. I am new to clojure, so I am not sure if this is the preferred way of doing things, but here goes anyway.

There seem to be several ways to set up clojure in emacs. I ended up following Bill Clementson's instructions. A couple of years ago I had some experience using maven, so decided to use this to manage my classpath. Installing maven on my mac was simple with macports (sudo port install maven).

Setting up a POM for maven took longer than expected. Stuart Sierra's post pointed me to the formos maven repository containing the clojure snapshots. With some help from google, I also found the maven-clojure-plugin, which is a maven plugin for compiling clojure, and the clojureshell-maven-plugin which will start a swank session (or bare REPL) using the pom information.

With the basic clojure and maven setup in place, it was time to move on to compojure. I added the Compojure git repository into Bill Clementson's clj-build script, ran it to clone the repository, and then built it using ant (ant deps; ant). Jim Downing instructions for installing compojure into your local maven repository (mvn install:install-file -DgroupId=org.clojure -DartifactId=compojure -Dversion=1.0-SNAPSHOT -Dfile=compojure.jar -Dpackaging=jar) work smoothly.

Product Development Flow

2009-08-30T00:00:00Z

I have spent the last few months with my latest start-up, Artfox, where I have been trying to push home some of the lean start-up advice expounded by Eric Ries and Steve Blank. I was hoping that "The Principles of Product Development Flow", by Donald Reinertsen, might help me in making a persuasive argument for some of the more troublesome concepts around minimum viable product and ensuring that feedback loops are in place with your customers as soon as possible. Unfortunately, I don't think that this is the book if you are looking for immediate, practical prescription, but it is a thought provoking, rigorous view of the product development process, that pulls together ideas from manufacturing, telecommunications and the Marines.

Perhaps Reinertsen's most accessible advice is that decisions in product development should be based on a strong economic foundation, pulled together by a concept of the "Cost of Delay". Rather than on relying on prescriptions for each of several interconnected metrics, such as efficiency and utilisation, Reinertsen suggests that economics will provide different targets for each of these metrics depending on the costs of the project at hand.

His proposition that product development organisations should measure "Design in Process", similar to the idea of "Intellectual Working In Process" proposed by Thomas Stewart in his book "Intellectual Capital", is what allows him to make the parallels to manufacturing and queueing theory and enables the application of the wide body of work in these fields to product development.

His practical advice, such as working in small batches and using a cadence for activities that require coordination, will come as no surprise to practitioners of agile development, and Reinertsen provides clear reasoning of why these practices work.

During my time at Alcan, and later Novelis, I gave a lot of thought to scheduling, queues and cycle times in a transformation based manufacturing environment, and I found that this had many parallels to his view of the product development process, and little in common with what Reinertsen describes as manufacturing, which seems to be limited to high volume assembly type operations. I found many ideas that could be usefully taken back to a manufacturing context.

If you look at this book as an introduction to scheduling, queueing theory and the reason's behind some of agile development practices, then you will not be disappointed.

Rails Environments For Lisp

2009-04-07T00:00:00Z

The facility of Ruby on Rails' test, development and production environments is one of those features that goes almost unremarked, but which makes using rails more pleasant. No doubt everyone has their own solution for this in other environments, and while I am sure Common Lisp is not lacking in examples, I have not seen an idiomatic implementation. In developing cl-blog-generator I came up with the following solution.

Configuration in Common Lisp usually depends on using special variables, which can be rebound across any block of code. I started by putting the configuration of my blog into s-expressions in files, but got tired of specifying the file names for different blogs. Instead, I created an association list for each configuration, and registered each using a symbol as key. I can now switch to a given environment by specifying the symbol for the environment.

The implementation (in src/configure.lisp under the GitHub repository) consists of two functions and a special variable. SET-ENVIRONMENT is used to register an environment, and CONFIGURE is used to make an environment active. The environments are stored in *ENVIRONMENTS* special as an association list. An example of setting up the configurations can be seen in the config.lisp file. In creating the configurations I drop the '*' from the special names.

I'm relatively new to CL, so let me now if I have overlooked anything. Writing this post makes me think I am missing a WITH-ENVIRONMENT macro ...

Twitter is More Secure than My Credit Card

2009-04-02T00:00:00Z

Twitter now lets developers build applications that take actions on your behalf without you ever having to divulge your password. Instead of asking you for your password, these applications ask Twitter to ask you for permission, and you give permission to the application while logged in to Twitter. What's even better is that you can revoke the application's permissions, from within Twitter, at any time, without having to change your password. The OAuth protocol makes this possible, and does so in a very secure manner.

Compare this to on-line transactions involving your credit card; you have to divulge your user-name (the name on your credit card) and your password (your credit card number) for every transaction. Some sites even store your card details, and you rely on trust, and the vendors good standing with the credit card company, that they will not make further transactions using your card. What happens should you find your card being abused? You have to go through the hassle of cancelling your card and obtaining a new card number, which you then have to divulge to all the companies that make regular charges to your card, instantly creating the opportunity for further abuse. To make matters worse, the credit card companies even give us these cards with our passwords on them (the card number), violating the "never write down your password (or PIN)" rule.

OAuth is an open protocol developed by some of the major web companies. The protocol is gaining traction rapidly, and the OAuth site lists some of the OAuth service providers (that is the sites that allow OAuth authorisation). Part of the reason for the rapid adoption is that is not really new technology. As mentioned in the OAuth design goals, the protocol is essentially a standardisation of the proprietary protocols used by Google’s AuthSub, AOL’s OpenAuth, Yahoo’s BBAuth and FlickrAuth and Facebook’s FacebookAuth.

Eran Hammer-Lahav provides a getting started with OAuth guide, Google provide a good overview of uses for OAuth, and the OAuth wiki provides links to all the details.

cl-blog-generator Gets Comments

2009-03-31T00:00:00Z

I have now added a comment system to cl-blog-generator. My requirements were for a simple, low overhead, commenting system, preferable one that could possibly be fully automated.

The comment system was inspired by Chronicle's, with a slight modification in approach - the comments are never saved on the web server, and are just sent by email to a dedicated email address. Spam filtering is delegated to the whatever spam filtering is implemented on the mail server, or in your email client. The comment emails are then processed in CL using mel-base and written to the local filesystem. Moderation can optionally occur on the CL side, if that is preferable to using the email client.

There is still some work left to do - I would like to be able to switch off comments on individual posts, either on demand on after a default time period - but I thought I would let real world usage drive my development.

Search Across Open Browser Tabs

2009-03-28T00:00:00Z

I am an Opera user, these days mainly because it gives me integrated mail, feed and news reading, so that everything that comes from the web appears in one place. The last significant innovation I remember was the introduction of tabs, and that was some time ago (long before it made its way into IE, for example). I am a heavy user of tabs - it is not unusual for me to have over fifty pages open, as I tend to just open pages and rarely close them again. This means that the tab icons are unreadable, and Alt+Tab (I'm on a mac) produces three or four columns to scroll through to select the tab I'm after. I dream of a better tab navigation model, and would love to be able to search across all the open tabs. Surely it wouldn't be that hard to implement.

Maybe I'm being a little harsh as there have been some other innovations. Firefox's add-ons work seamlessly, we can use arbitrary search engines thanks to OpenSearch, and Conkeror gives full screen browsing with emacs key bindings, but for the most part it has been better support for standards that has dominated the release notes.

In preparing this post, I found that Google's Chrome has search over your page history. Sounds like that might fulfill my wish, though I wonder how its process per tab model will scale to lots of tabs. I just have to wait for its release on mac OS X.

Blog Site Generators

2009-03-27T00:00:00Z

I recently uploaded some links to my cl-blog-generator project, and have been getting some feedback with comparisons to other blog site generators, or compilers, such as Steve Kemp's Chronicle, or Jekyll as used on GitHub Pages. Compared to these, cl-blog-generator is immature, but takes a different approach in several areas that Charles Stewart suggested might be worth exploring. I look forward to any comments you might have.

Formatting

All the blog generators seem to use a file based approach for writing content, but they differ in the choice of input formats supported, and in the approach to templating. cl-blog-generator is the least flexible, requiring input in XHTML, while Chronicle allows HTML, Textile or Markdown, and Jekyll Textile or Markdown. For templates, Chronicle uses Perl's HTML::Template, and Jekyll uses Liquid. cl-blog-generator uses an approach which substitutes content into elements identified with specific id's or classes, similar to transforming the templates with XSLT.

cl-blog-generator's choice of XHTML input was driven by a requirement to enable the validation of post content in the editor, which is not possible using Chronicle's HTML input because of the headers and lack of a body or head element, and a desire to be able to use any CSS tricks I wanted, which ruled out Textile and Markdown, or any other markup language. The lack of an external templating engine in cl-blog-generator was driven by simplicity; I couldn't see a use for conditionals or loops given the fixed structure of the content, and this choice leads to templates that validate, unlike Jekyll, and which are not full of HTML comments. The current id and class naming scheme in cl-blog-generator could certainly use some refinement to improve the flexibility of the output content format, and I would definitely welcome requests for enhancements should the scheme not fit your requirements.

Database and Two Phase Publishing

Perhaps the most significant difference in approach for cl-blog-generator is its use of a database and an explicit publish step. With cl-blog-generator a draft can exist anywhere in the filesystem, and must be "published" to be recognised by the blog site generator. The publishing process fills in some default metadata, such as post date, if this is not originally specified, copies the modified draft to a configurable location, and enters the metadata into the database. This ensures that the post is completely specified by its representation in the filesystem, and that the database is recreatable.

The database enables the partial regeneration of the site, without having to parse the whole site, and makes the linking of content much simpler. However, having Elephant as a dependency is probably the largest impediment to installation at present.

On Titles, Dates, Tags and Filenames

cl-blog-generator's input XHTML has been augmented to add elements for specifying post title, date, update date (which I believe is missing from the other systems), slug, description, and tags. On publising (see next section), any of these elements that is missing, except the mandatory title, is filled in with defaults.

Both Chronicle and Jekyll use a preamble to specify metadata, with the filename being used to generate the post's slug. Jekyll also uses the filename and its path for specifying the post date, and tags.

Bells and Whistles

Finally, here is a grab bag of features.

Chronicle comes with a commenting system.
cl-blog-generator generates a meta description element, which is used by search engines to generate link text. It also generates meta elements with links to the previous and next posts.
Jekyll has a “Related posts” feature for generating links to similar posts.

Chronicle and Jekyll both have migration scripts for importing content.
Chronicle has a spooler for posting pre-written content at specific times

Frameworks and Productivity

2009-03-20T00:00:00Z

Yesterday was frustrating; I spent far too long trying to debug some problems in a Rails application I am writing. Rails, and frameworks in general, are supposed to give us improved productivity by hiding the complexity and mechanics of the task at hand. This is great as long as the framework behaves as expected, but invariably causes problems when things go wrong.

The missing field

My application uses associations, and I had a belongs_to association that was supposed to be populated in a before_validation_on_create callback. In my tests I noticed that the linked model was not being instantiated. After much searching, it turns out I had forgotten to create the foreign key field. Unfortunately Rails was silent on this issue and the belongs_to association code seemed to execute quite happily without the field.

Can’t dup NilClass

My models also use has_many associations, which I could populate with no problems. When I tried to access the association though, I kept getting Can’t dup NilClass errors. This one turned out to be an issue with the generated collection.build method. As noted in the documentation by the somewhat cryptic Note: This only works if an associated object already exists, not if it‘s nil!, the method fails if the collection is empty (at least that’s what I think it means). Explicitly instantiating the associated model and then adding it to the collection fixed the problem.

Bad choice of name

In my application I had a model named Target, which meant that models that associate with Target, such as my TargetProfile model, have a target attribute.

Unfortunately the target attribute in TargetProfile always returned the instance of TargetProfile - not quite what is expected. The problem was caused by the fact that ActiveRecord’s AssociationProxy, used to implement associations between models, has a target attribute. The documentation contains another warning Don‘t create associations that have the same name as instance methods of ActiveRecord::Base, but mentions nothing of AssociationProxy, which isn’t even part of the documented api. I call this broken encapsulation.

The Rails Way

Rails heavily promotes testing and Test Driven Development (TDD), which is linked to the “fail early, fail fast” paradigm. It seems strange that known issues (as partially documented) are allowed to persist and cause silent failures. All the issues above could have been flagged by Rails.

Conclusion

I am not picking on Rails, however, as these type of issues seem to occur in many frameworks. No software is bug free, and a framework has to work hard to hide the complexity and mechanics of its domain. When things go wrong, we are always left questioning whether the issue is in the framework or in our application and we invariably end up getting to know the implementation details of the framework, which does little for our productivity.

Create a Catalog for XHTML on OS X

2009-03-11T00:00:00Z

While trying to validate the output of cl-blog-generator I needed a local DTD for XHTML. The textproc/xmlcatmgr package in Darwin Ports creates a catalog at /opt/local/etc/xml/catalog, but it does not include XHTML. A flattened XHTML DTD can be found in the w3 validator library and installed the DTD's under /opt/local/share/xml/, but I couldn't find a catalog file for it. It turns out it is pretty simple to write the catalog file; the Wikipedia XML Catalog entry has an example that contains what is needed. Save the example next to the XHTML DTD's as catalog.xml and adjust the paths, then add a "nextCatalog" entry in /opt/local/etc/xml/catalog pointing at the catalog.xml file.

Now I can use (setf cxml:*catalog* (cxml:make-catalog '("/opt/local/etc/xml/catalog"))) and CXML will use the local DTD specified in the catalog.

Parsing YAML Dates in Rails Gives Surprising Results

2009-03-07T00:00:00Z

Today's surprise was that "2009-01-01" and "2009-1-1" get parsed differently by the YAML parser in Rails. The former gets converted to a Date, while the latter becomes a String. It confused me for a while, as the problem only showed up when I wanted to send the dates to a Flot chart. Looking at the standard, it's conforming behaviour. Must be me that is non-conforming then...