He allowed himself to be swayed by his conviction that human beings are not born once and for all on the day their mothers give birth to them, but that life obliges them over and over again to give birth to themselves.” ― Gabriel García Márquez, Love in the Time of Cholera
Edit
This is a re-write - the original is below. It now aims to provide some recommendations on how to release for consumption by a git dependency. It all seems very obvious in retrospect
Firstly only publish a single artifact from a single git repository, or several artifacts with identical versions if it is a monorepo. You can imagine schemes that would work with multiple artifacts with independent versions, but tooling is going to have a hard time with any such scheme. This was the piece of the puzzle I was missing in my original post.
Once we've agreed to the above, then it becomes simple - versions become monotonically increasing, and easy for tooling to deal with.
Just put a git tag on a release in the same way you would a version published to maven. For a good scheme see Golang Modules. Make sure the scheme you choose sorts well with version-clj (h/t @borkdude for both).
Remember you can have multiple tags for a given sha, so you could tag it v1.0.0-alpha to start with and promote it to v1.0.0, if that is your cup of tea.
Many thanks to the collective wisdom of Michiel Borkent @borkdude, Alex Miller @puredanger, Sean Corfield and Erik Assum @slipset on the clojurians slack.
Original post below
When you want to consume a library using git dependencies, you go to the project's GitHub page, lookup the SHA from the README, put it in your deps.edn, and your done, right? – But what happens when you want to upgrade? rinse and repeat? How do you even know that a new release is available?
A git repository is an append only log of changes to a project, and Together with the repository url, the SHA forms a content based addressing scheme to a particular state of the project. This is a natural identifier for that particular project state.
As consumers of a library, we aren't concerned with every single commit made to the repository - we want to know the SHA that the project's maintainers consider to be a release. We might not want the main branch HEAD commit, depending on the branching model used by the project developers.
A good CI pipeline takes an immutable project artifact, and put's it through increasingly vigorous testing. It might start of as an alpha, or a release candidate, and as confidence is increased through testing, it can be promoted to a full release.
An artifact built from the contents of a single git SHA fits this model nicely. In an open source world, we can think of a SHA as an alpha release, that gets tested by a small number of people, and then gets published as a release - tools.build seems to follow this model for example, with announcement of alphas on #tools.build, followed by release announcements, for the same SHA, on #announce after a few people have tried it.
So which SHA do we want to put in our deps.edn?
In the maven world, versions are ordered, so when a new version is published, it is a signal that can be used by tooling to determine if an update is available.
In the git world, a SHA is not ordered, so how do we know when a new version is available? Should we check slack, or a blog, or the project's home page? or could we as project authors provide data to allow automation of this process?
To provide release information, we could put the version information into the repository itself.
There are many release schemes, but we can model a project's releases as falling into release streams. Examples of this are "stable" or "alpha" or "v4.x". A release for each stream is then just a SHA associated with the stream.
A natural way to present this would be as a map in an EDN file (or JSON, or YAML, this doesn't need to be clojure specific).
{:stable {:git/tag "v1.0.0" :git/sha "abcdef"}
:alpha {:git/tag "v1.1.0" :git/sha "abcdef"}
:head {:git/tag "master" :git/sha :latest}}A polylith or other monolith repository could have different streams for the various artifacts it published.
If we decide on this format, we then need to make the file discoverable. One way one be to take it from a git repository's default branch, which seems like a good default.
The final piece of the puzzle would be for tooling like antq and clojure-dependency-update-action to use this information.
I'm sure I can't be the first to have thought of this.
What do you think - is this useful? How could the idea be improved?
]]>“My mind turned by anxiety, or other cause, from its scrutiny of blank paper, is like a lost child–wandering the house, sitting on the bottom step to cry.” — Virginia Woolf
I was inspired to write some blog posts, which led me to realise my current blog and blogging setup were completely broken.
Michiel Borkent (@borkdude) recently wrote about his migration from Octopress. His requirements were very similar to mine, so I copied and modified. Thank you Michiel!
His blog, REPL adventures, is well worth the read.
With a static web site, the perennial problem is how to enable discussions. Some people just punt pn this, and point to reddit, but Michiel's solution is to use github discussions. As a way of owning the discussion, this has a lot of appeal.
I think it could be taken further. It would be great if we could automate the creation of a blog post topic when creating a blog post. Unfortunately the gh command line client doesn't support discussions yet, so that would require using Github's GraphQL API - more work than I wanted to do for now.
One downside though, is that the discussions are not visible on the blog pages, where discussion could easily engender more discussion. I wonder if a Github Action could be triggered by conversation activity, and automatically republish the post with the discussion to date at the end of the post.
There are many blog site generators (I used Hugo, of course), even if we limit ourselves to clojure:- bootleg, nota, cryogen, and static to name a few.
These are usually feature rich. The price for those features though, is extra complexity.
Michiel's blog uses babashka tasks to add a post, render posts, etc. These are extremely quick to run and make maintaining the blog simple. it does just what he needs, and no more.
This reminds me of project automation, and the tools.build approach of using composable code tasks to build just what is needed.
Maybe there is an opportunity to take the same approach for building a blog or static site. If we could pick from a selection of configurable tasks, maybe we wouldn't need to write our own.
Speaking of which, I have tried writing my own before, in common-lisp: cl-blog-generator.
So I have a blogging setup. Now I just need to write something.
]]>This post will explain how to use lein to customise you build to generates a source file, but many of the steps are useful to implement any form of lein build customisation.
The source code generator is going to live in the my.src-generator namespace. Here's an example, that just generates a namespace declaration for the my.gen namespace under target/generated/my/gen.clj.
(ns my.src-generator
(:require [clojure.java.io :refer [file]]))
(defn generate []
(doto (file "target" "generated" "my" "gen.clj")
(-> #(.getParentFile) #(.mkdirs))
(spit "(ns my.gen)")))The source generation code should not be packaged in the jar, so we place it in dev-src/my/src_generator.clj, and add dev-src and the generated source directories to the :dev profile's :source-paths. The :dev profile is automatically used by leiningen unless it is producing a jar file. When producing the jar, the dev profile will not be used, so dev-src will not be on the :source-path (we add the generated directory to the base :source-path below).
:profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}}The run task can be used to invoke code in your project. To use lein's run task we need to add a -main function to the my.src-generator namespace.
(defn -main [& args]
(generate))In the project.clj file we also tell lein about the main namespace. In order to avoid AOT compilation of the main namespace, we mark it with :skip-aot metadata.
:main The generated files need to end up in the jar (and possibly be compiled), so we put them on the :source-paths in the project. If we had wanted to include the sources without further processing, we could have added the generated directory to :resource-paths instead.
:source-paths ["src" "target/generated"]Now we can tell lein to generate the source files whenever we use the project. We do this by adding the run task to the :prep-tasks key. Leiningen runs all the tasks in :prep-tasks before any task invoked by the lein command line.
The tricky bit here is that the run task will itself invoke the :prep-tasks, so we want to make sure we don't end up calling the task recursively and generating a stack overflow. To solve this, add a gen profile, and disable the prep tasks in it. We use the :replace metadata to ensure this definition takes precedence. See the leiningen profile documentation for more information on :replace and it's sibling :displace.
:gen {:prep-tasks }Then use this profile when setting the :prep-tasks key in the project.
:prep-tasks [["with-profile" "+gen,+dev" "run"] "compile"]Now when we run any command, the sources are generated.
Finally we may want to just invoke the source generation, so let's create an alias to make lein gen run the generator. We need the gen profile for this, or otherwise the generator will run twice.
:aliases {"gen" ["with-profile" "+gen,+dev" "run"]}For reference, the final project.clj looks like this:
(defproject my-proj "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.4.0"]]
:source-paths ["src" "target/generated"]
:main
:prep-tasks [["with-profile" "+gen,+dev" "run"] "compile"]
:profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}
:gen {:prep-tasks }}
:aliases {"gen" ["with-profile" "+gen,+dev" "run"]})This required using many of lein's features to get working - hopefully you'll find a use for some of them.
]]>project.clj file and then? Instead of shutting down your REPL, loosing whatever state you have built up, you can use Alembic to load the new dependencies. Simply call (alembic.still/load-project).Of course, it still has to work within the confines of the JVM's classloaders, so you can only add dependencies, and not modify versions or remove dependencies, but this should still cover a lot of use cases.
To use alembic on a single project, simply add it as a dependency in your :dev profile in project.clj:
:profiles {:dev {:dependencies [[alembic "0.2.0"]]}}
To make alembic available in all your projects, and it to the :user profile in ~/.lein/profiles.clj instead:
{:user {:dependencies [[alembic "0.2.0"]]}}
Alembic also allows you to directly add dependencies without editing your project.clj file, using the distill function. Use this if you are just exploring libraries, for example.
Finally a big thank you to Anthony Grimes and the other flatland developers for removing classlojure's dependency on useful, which should make this all much more robust.
clojure-mode and evaluate code with nrepl.el.This can be enabled using mmm-mode, which allows a single buffer to use different major modes for different sections of the buffer (and is not limited to just web modes). Install mmm-mode using M-x package-install mmm-mode, or using M-x el-get-install mmm-mode from the excellent el-get, or by checking the project from github and installing manually.
To configure this for clojure and markdown, add this in your init.el or .emacs file.
(require 'mmm-auto)
(mmm-add-classes
'((markdown-clojure
:submode clojure-mode
:face mmm-declaration-submode-face
:front "^```clj[\n\r]+"
:back "^```$")))
(setq mmm-global-mode 'maybe)
(mmm-add-mode-ext-class 'markdown-mode nil 'markdown-clojure)
After evaluating the above, or restarting emacs, you can test multi-mode support by opening a markdown document, or creating a new one, and adding a clojure source block, e.g.:
(defn my-fn [x] (inc x)) (my-fn 1)
Inside the code block you can format and evaluate your code as in any clojure-mode buffer, and the code will display exactly as in a .clj file. By default the evaluation uses a running inferior lisp process, which you must start yourself. To use a running nrepl session instead, use M-x nrepl-interaction-mode inside the code block.
This technique is not limited to clojure and markdown, but could be made to work whenever you would like differing major modes in distinct parts of your Emacs buffers. You can add class to mmm-mode appropriately, for as many major mode combinations as you need. The regions for each major mode are detected using regular expressions (or by some function).
For example, if you're writing asciidoc, you might use:
(mmm-add-classes
'((asciidoc-clojure
:submode clojure
:face mmm-declaration-submode-face
:front "\\[source, clojure\\][\n\r]+----[\n\r]+"
:back "^----$")))
(mmm-add-mode-ext-class 'adoc-mode nil 'asciidoc-clojure)
(mmm-add-mode-ext-class 'doc-mode nil 'asciidoc-clojure)
mmm-mode allows you to flexibly use multiple major modes in different parts of a single emacs buffer. Here we have shown how to use it for clojure-mode code blocks in markdown or asciidoc, but it is in no way limited to this, and it allows some fine grained customisation to the appearance and behaviour of each major mode block. I'm sure you'll find your own uses for mmm-mode.
When you receive a signed message, mu shows the verification status in the Signature field in the message view (see MSGV-Crypto). If you don't have the sender's PGP key on your keyring, this will show unverified. Click on the Details link within field will show the sender's key id. To manually import the key you can use gpg:
$ gpg --recv <the-key-id>
This seemed a little labourious, so some automation was in order. mu4e allows you to define actions that can be run on messages (or attachments), so I just wrote an action to do this.
(defun mu4e-view-snarf-pgp-key (&optional msg)
"Snarf the pgp key for the specified message."
(interactive)
(let* ((msg (or msg (mu4e-message-at-point)))
(path (mu4e-message-field msg :path))
(cmd (format "%s verify --verbose %s"
mu4e-mu-binary
(shell-quote-argument path)))
(output (shell-command-to-string cmd)))
(let ((case-fold-search nil))
(when (string-match "key:\\([A-F0-9]+\\)" output)
(let* ((cmd (format "%s --recv %s"
epg-gpg-program (match-string 1 output)))
(output (shell-command-to-string cmd)))
(message output))))))
This works by parsing the output of the mu program itself, as displayed in the Details window, to obtain the PGP key id. It then executes the gpg --recv command, parsing in the parsed key id.
To install the action, we simply add it to mu4e-view-actions:
(add-to-list 'mu4e-view-actions
'("Snarf PGP keys" . mu4e-view-snarf-pgp-key) t)
Luke Vanderhart posted a general introduction to using Javascript libraries in Clojurescript. Go read it if you haven't already - this post assumes you have.
While that post is an excellent description of using JavaScript in a Clojurescript application, it doesn't really address JavaScript in Clojurescript libraries, which has the additional problem of how to ensure the JavaScript dependency is available in the consumer of the library. A Clojurescript library should definitely be capable of providing it's dependencies, but should also allow the consumer to override the version of these dependencies.
The first approach is to simply not provide the JavaScript at all. This is the approach taken by jayq for example. The consumer of jayq, or any library that uses jayq, is required to provide jQuery through the JavaScript runtime. This can take the form of a <script> link in the browser page, or a call to webPage#injectJs in phantomJS. The compile :libs or :foreign-libs options can not be used to provide the dependency, as there is no way for the compiler to know that jayq depends on the namespace provided by these options.
For the consumer of the library to use compiler:optimizations other than :whitespace, they will need to provide an :externs file.
The second approach is to package the JavaScript via a Clojurescript namespace. This involves adding a require on a namespace to the code that directly depends on the JavaScript, and arranging for that Clojurescript namespace to load the JavaScript, using either of the compiler:libs or :foreign-libs options.
The Clojurescript library can make the JavaScript library available in its resources. The library consumer can then use resource via the :libs or :foreign-libs options, depending on whether or not the JavaScript contains a goog.provides call.
If the library is packaged with a goog.provides call, then the consumer can not replace the version using :libs [""] - the use of an explicit prefix in :libs is needed to prevent more than one JavaScript implementation trying to provide the clojure namespace, or the use of :foreign-libs where the namespace is explicitly mapped.
For examples, the pprng library packages its dependency with a goog.provides call, allowing the use of :libs [""] to pull in the dependency. The papadom library on the other hand provides vanilla javascript dependencies, and requires the use of the more verbose :foreign-libs option.
If the JavaScript is to be provided in the runtime, then the consumer will have to provide an empty namespace definition to satisfy the require in the Clojurescript library, and the:externs file as in the first case.
There are several assumptions in much of the documentation that I didn't see explicitly explained. I'll record these here for posterity.
A clojurescript library is always a source code library. There is no such thing as the linking of compiled clojurescript artifacts.
Neither:libs nor:foreign-libs actually changes how the JavaScript is accessed within the clojurescript code. If you include jQuery via a :libs, and a require, you still access it through js/jQuery. The require of the namespace specified by goog.provide, or the namespace specified in the :foreign-libs' :provides key, simply ensures the JavaScript is loaded.
The choice of compiler :optimizations affects what information you need to provide, and this differs depending on whether you are providing javascript libraries through the runtime (e.g. $lt;script> tags in the browser), or through :libs or :foreign-libs compiler options. The simplest here is to use the compiler options. When providing the JavaScript via the runtime, then everything should also just work if you are using no optimisation, or just :whitespace, but as soon as you try anything else, you will need to provide an :externs definition for the JavaScript libraries.
Clojurescript recently gained a CSP implemetation via core.async, similar to Go's channels, or CML's channels (CML also has a nice select). Bruce Haumann started exploring this with ClojureScript Core.Async Todos, and David Nolen has been looking at how to use core.async for responsive design. In this post, we'll take the TODO example, and take it a little further.
We'll start with just displaying a list of todo items. For this we'll need a template, so we'll just write this in HTML, and add a t-template attribute, which enables us to use mustache style templating of values to display. This doesn't use mustache sections for looping, in order to preserve valid HTML markup.
<h1>TODOS</h1>
<ul class="unstyled">
<li t-template="todos">{{text}}</li>
</ul>
To get this to show something we'll need some code:
(ns app
(:require
[papadom.template :refer [compile-templates render]]))
(defn start []
(compile-templates)
(render {:todos [{:text "learn papadom" :done false}
{:text "write a papadom app" :done false}]}))
When you call app.start() from the page containing the above template, you'll see a list of two todo entries.
Now we have something displayed, lets add a checkbox to mark todo items as done:
<ul class="unstyled">
<li t-template="todos">
<input type="checkbox" t-prop="done" t-event="done"
t-id="index" index="{{@index}}">
<span>{{text}}</span>
</li>
</ul>
The t-prop attribute tells the template to which data value to display as the checkbox.
The t-event attribute specifies that we want an event. When the checkbox is clicked, we will get a core.async message with a :done event type. We need to know which todo was clicked, so we use the t-id attribute to list the attributes whose values should be sent as the event data – in this case the index, which has a value based on handlebars style @index property.
Now we need some code to process the events. To do this we'll define an app function that will be passed a state atom containing a map with our todos state, and a core.async channel from which to read events. The function will loop over events, and dispatch them as required.
(defn app
[state event-chan]
(go
(loop [[event event-data] (<! event-chan)]
(case event
:done
(let [nv (boolean (:checked event-data))]
(swap! state assoc-in [:todos (:index event-data) :done] nv)))
(recur (<! event-chan)))))
When the app function receives a :done event, it will update the state atom appropriately. Now we have our state updating, we'll need to display it, which we can again do with the render function.
(defn show-state [state]
(render "state" state))
We still need to get show-state called, and we'll arrange this in a modified start function. This will create an atom for the state, and add a watch on the atom that will call show-state.
(defn start
[]
(let [event-chan (chan)
state (atom nil)]
(compile-templates)
(template-events event-chan)
(add-watch state :state (fn [key ref old new] (show-state new)))
(reset! state {:todos [{:text "Learn papadom" :done false}
{:text "Build a papadom app" :done false}]})
(app state event-chan))))
We've also added a core.async channel, event-chan, which we've passed to template-events to arrange delivery of the events defined in our template. We pass this channel to the app function to start processing the events.
This shows the basic structure of the application.
To allow you to add new todo items, we'll add a form to our template, specifying a t-event attribute, which will cause an event to be sent when the form is submitted, with the form's input values as the event data.
<form t-event="add-todo">
<input type="text" t-prop="text" size="30" placeholder="add new todo here">
<input class="btn btn-primary" type="submit" value="add">
</form>
To process this new event, we'll add a case to the app function loop's case form.
:add-todo
(swap! state update-in [:todos]
conj {:text (:text (input-seq->map event-data))
:done false})
This uses the input-seq->map helper to convert the data from the form into a map, and we extract the :text value (defined by t-prop in the input element).
And we're done. To see a full working example have a look at the template and code in the todo example of papadom. To run the example:
git clone [https://github.com/hugoduncan/papadom.git](https://github.com/hugoduncan/papadom.git)
cd papadom/examples/todo
lein ring server
If you have configuration that uses template configuration files, you are practising neither configuration as code nor configuration as data. Having configuration locked away in template files reduces its visibility, and makes it hard for you to query it. It might be easier to write configuration code to use templates, but it will come back to bite you.
One of the first things I implemented in pallet was a templating mechanism, because configuration management tools use templates, right? I even built a template selection mechanism, just like Chef's.
I have come to realise however, that having configuration in template files is not particularly useful. There are three major problems you are likely to encounter. Firstly template files are not visible, secondly you can not query the data in the template files, and lastly you will need to touch multiple files to add or modify parameters.
Visibility at the point of usage is important, especially in a team environment. If you have to find the template file and look at its content when reading your configuration code, then the chances are you assume it hasn't changed, and skip the contents. Making an analogy to the world of software development, templates are like global variables in one sense. You can change the operation of a program with a global variable modified in some obscure place, and in the same way, you can change your system configuration by changing a template file, tucked away in some folder, and not visible from where you are actually calling your configuration crate/recipe.
The ability to query configuration settings allows not just finding out, for example, which directory a log file is in, but also enables you to put tools on top of your configuration data. Template configuration files suffer on two counts here - they are separate text files that require parsing to be read, and the format of each configuration file is different.
The last point concerns the flexibility of your configuration. If you have used template files, with hard coded parameter values, and you then want to modify your configuration to dynamically set one of those hard coded values, you have to modify all the specialised versions of the existing templates, and specify the value in code. You have to touch multiple files - lots of room for making typos.
My goal for pallet then, is to have all configuration supplied as arguments to crates. For most packages a hash map is sufficient an abstraction for providing the data, but when this gets too cumbersome, we'll use a DSL that mirrors the original configuration file language.
Goodbye hidden configuration!
]]>Basic Nagios support was recently added to pallet, and while very simple to use, this blog post should make it even simpler. The overall philosophy is to configure the nagios service monitoring definitions along with the service itself, rather than have monolithic nagios configuration, divorced from the configuration of the various nodes.
As an example, we can configure a machine to have it's ssh service, CPU load, number of processes and number of users monitored. Obviously, you would normally be monitoring several different types of nodes, but there is no difference as far as pallet is concerned.
We start by requiring various pallet components. These would normally be part of a ns declaration, but are provided here for ease of use at the REPL.
(require '[pallet.crate.automated-admin-user
:as admin-user] '[pallet.crate.iptables :as 'iptables] '[pallet.crate.ssh :as ssh] '[pallet.crate.nagios-config
:as nagios-config] '[pallet.crate.nagios :as nagios] '[pallet.crate.postfix :as postfix] '[pallet.resource.service :as service]) Now we define the node to be monitored. We set up a machine that has SSH running, and configure iptables to allow access to SSH, with a throttled connection rate (six connections/minute by default).
(pallet.core/defnode monitored [] :bootstrap [(admin-user/automated-admin-user)] :configure [;; set iptables for restricted access
(iptables/iptables-accept-icmp)
(iptables/iptables-accept-established)
;; allow connections to ssh
;; but throttle connection requests
(ssh/iptables-throttle)
(ssh/iptables-accept)]) Monitoring of the SSH service is configured by simply adding (ssh/nagios-monitor).
Remote monitoring is implemented using nagios' nrpe plugin, which we add with (nagios-config/nrpe-client). To make nrpe accessible to the nagios server, we open the that the nrpe agent runs on using (nagios-config/nrpe-client-port), which restricts access to the nagios server node. We also add a phase, :restart-nagios, that can be used to restart the nrpe agent.
Pallet comes with some configured nrpe checks, and we add nrpe-check-load, nrpe-check-total-proces and nrpe-check-users. The final configuration looks like this:
(pallet.core/defnode monitored [] :bootstrap [(admin-user/automated-admin-user)] :configure [;; set iptables for restricted access
(iptables/iptables-accept-icmp)
(iptables/iptables-accept-established)
;; allow connections to ssh
;; but throttle connection requests
(ssh/iptables-throttle)
(ssh/iptables-accept)
;; monitor ssh
(ssh/nagios-monitor)
;; add nrpe agent, and only allow
;; connections from nagios server
(nagios-config/nrpe-client)
(nagios-config/nrpe-client-port)
;; add some remote checks
(nagios-config/nrpe-check-load)
(nagios-config/nrpe-check-total-procs)
(nagios-config/nrpe-check-users)] :restart-nagios [(service/service
"nagios-nrpe-server"
:action :restart)]) We now configure the nagios server node. The nagios server is installed with (nagios/nagios "nagiospwd"), specifying the password for the nagios web interface, and add a phase, :restart-nagios, that can be used to restart nagios.
Nagios also requires a MTA for notifications, and here we install postfix. We add a contact, which we make a member of the "admins" contact group, which is notified as part of the default host and service templates.
(pallet.core/defnode nagios [] :bootstrap [(admin-user/automated-admin-user)] :configure [;; restrict access
(iptables/iptables-accept-icmp)
(iptables/iptables-accept-established)
(ssh/iptables-throttle)
(ssh/iptables-accept)
;; configure MTA
(postfix/postfix
"pallet.org" :internet-site)
;; install nagios
(nagios/nagios "nagiospwd")
;; allow access to nagios web site
(iptables/iptables-accept-port 80)
;; configure notifications
(nagios/contact
{:contactname "hugo"
:servicenotificationperiod "24x7"
:hostnotificationperiod "24x7"
:servicenotificationoptions
"w,u,c,r"
:hostnotificationoptions
"d,r"
:servicenotificationcommands
"notify-service-by-email"
:hostnotification_commands
"notify-host-by-email"
:email "my.email@my.domain"
:contactgroups [:admins]})] :restart-nagios [(service/service "nagios3"
:action :restart)]) That's it. To fire up both machines, we use pallet's converge command.
(pallet.core/converge {monitored 1 nagios 1} service :configure :restart-nagios) The nagios web interface is then accessible on the nagios node with the nagiosadmin user and specified password. Real world usage would probably have several different monitored configurations, and restricted access to the nagios node.
Support for nagios is not complete (e.g. remote command configuration still needs to be added, and it has only been tested on Ubuntu), but I would appreciate any feedback on the general approach.
]]>I dislike most mocking, and try and avoid it as much as possible. Sometimes it is however the only realistic way of testing. I did a quick survey of mocking tools in clojure, and found them very much reflecting the Java mocking libraries. Clojure has a few more dynamic capabilities than Java, so I thought a little about how these could be used to make a simple mocking facility, and atticus is what I came up with.
There is a consensus that mocking should be implemented by binding a function's var to a new function for the duration of a test, and atticus does this too. Atticus' premise is that we can do simple mocking by declaring the mock function as a local function definition, and have the local function do the argument checking, return value setting, etc. The simplest case would be something like below, which checks the value of its argument and specifies a return value:
;; pull in namespaces (use 'clojure.test) (require 'atticus.mock);; define test which mocks f (deftest mock-test (atticus.mock/expects [(f [arg] (is (= arg 1) "Check argument") arg)] ; set the return value ;; in a real test case this would be called ;; indirectly by some other function (is (= 1 (f 1)) "Call mocked function"))
At the moment, I have added two macros to this. once, which checks a function is called once, and times, which checks that a function is called a specific number of times. The macros are used to wrap the body of the mock function, which keeps the function's expected behaviour in one place.
;; define test, that should be called just once (deftest mock-test (atticus.mock/expects [(f [arg] (atticus.mock/once (is (= arg 1) "Check argument") arg))] (is (= 1 (f 1)) "Call mocked function"))
;; define test, that should be called exactly twice (deftest mock-test (atticus.mock/expects [(f [arg] (atticus.mock/times 2 (is (= arg 1) "Check argument") arg))] (is (= 1 (f 1)) "Call mocked function") (is (= 1 (f 1)) "Call mocked function"))
So what do you think, is this a reasonable approach? Not having the explicit calls to returns, etc, might be seen as a loss of declarative clarity, but I for one prefer this, as it gives you the full power of the language to test the arguments and set the return value.
I recently needed to move a server from dedicated hosting to a cloud server. The existing server had been configured over time by several people, with little documentation. I want to make sure that this time everything was documented, and what better way than doing that than using an automated configuration tool.
Looking around at the configuration tools, I couldn't find one I really liked, so I started Pallet. I'll explain why I didn't use an existing tool below, but first I wanted to show how to manage nodes in Pallet.
;; Pull in the pallet namespaces (require 'pallet.repl) (pallet.repl/use-pallet);; Define a default node (defnode mynode [])
;; Define the cloud account to use (def service (compute-service "provider" "user" "password" :log4j :ssh))
;; Create 2 nodes (converge {mynode 2} service)
This example would create two nodes (cloud vm instances) with the tag "mynode" in your cloud account, as specified in the service. This would give you the smallest size, ubuntu image on most clouds. Of course, to do anything serious, you would want to specify the image you would like, and you would probably like some configuration of the nodes. So carrying on the above example:
;; Pull in the needed crates (use 'pallet.crate.automated-admin-user) (use 'pallet.crate.java);; Define a new node that will use the Java JDK (defnode javanode [:ubuntu :X86_64 :smallest :os-description-matches "[J]+9.10[32]+"] :bootstrap [(automated-admin-user)] :configure [(java :openjdk :jdk)])
;; Create a new node, and remove the previous ones (converge {javanode 1 mynode 0} service)
This would stop the two nodes that were created before, and create a new one, with the specified ubuntu version. On first boot, it would create a user account with your current username, authorize your id_rsa key on that account, and give it sudo permissions. Every time converge is run, it also ensures that the openjdk JDK is installed.
The configuration to be applied is specified as a call to a crate - automated-admin-user and java in the example above. Crates are just clojure functions that specify some configuration or other action on the nodes (they're equivalent to Chef's recipes, which Pallet can also execute using chef-solo). Pallet can be extended with your own crates, and crates can specify general actions, not just configuration. lift is a companion to converge, and can be used to apply crates to existing nodes (including local VM's). The hypothetical example below would execute my-backup-crate on all the "mynode" nodes.
(defnode mynode [] :backup [(my-backup-crate)]) (lift mynode service :backup)
This was just a quick overview of Pallet, to give you an idea of what it is. One big area of Pallet not demonstrated here is its command line tool. But that is a topic for another post.
Now you've seen some examples, I'll try and explain the features that make Pallet distinct from other configuration tools out there.
The machines being managed require no special dependencies to be installed. As long as they have bash and ssh running, they can be used with pallet. For me this was important - it means that you can use pretty much any image out there, which is great for ad-hoc testing and development.
Pallet has no central server to set up and maintain - it simply runs on demand. You can run it from anywhere, even over a remote REPL connection.
In pallet, all your configuration is handled in SCM controlled files - there is no database involved. This means that your configuration can always be kept in step with the development of your crates, and the versions of the external crates that you use.
Custom crates can be distributed as jar files, and so can be published in maven repositories, and be consumed in a version controlled manner. Hopefully this will promote shared crates.
Pallet aims quite wide. You can use it for starting an stopping nodes, for configuring nodes, deploying projects and also for running administration tasks. To be honest, this wasn't an initial design goal, but has come out of the wash that way.
Hopefully this has whetted your appetite, and you'll give pallet a try. You can get support via the Google Group, or #pallet on freenode irc.
]]>Let's face it, many of us hate writing shell scripts, and with good reason. Personally, it's not so much the shell language itself that puts me off, but organising everything around it; how do you deploy your scripts, how do you arrange to call other scripts, how do you manage the dependencies between your scripts? Pallet aims to solve these problems by embedding shell script in clojure.
Embedding other languages in lisp is not a new idea; parenscript, scriptjure (which Pallet's embedding is based on), and ClojureQL all do this.
So what does shell script in clojure look like? Some examples:
(script (ls "/some/path") (defvar x 1) (println @x) (defn foo [x] (ls @x)) (foo 1) (if (= a a) (println "Reassuring") (println "Ooops")) (println "I am" @(whomai))
which generates:
ls /some/path x=1 echo ${x} function foo(x) { ls ${x} } foo 1 if [ \( \"a\" == \"a\" \) ]; then echo Reassuring;else echo Ooops;fi echo I am $(whomai) The aim is to make writing shell script similar to writing Clojure, but there are obvious differences in the language that limit how far that can be taken. To run the code above at the REPL, you'll have to use the pallet.stevedore package.
Escaping allows us to embed Clojure values and expressions inside our scripts, in much the same way as symbols can be unquoted when writing macros.
(let [path "/some/path"] (script (ls ~path) (ls ~(.replace path "some" "other)))
We can now write Clojure functions that produce shell scripts. Writing scripts as clojure functions allows you to use the Clojure namespace facilities, and allows you to distribute you scripts in jar files (which can be deployed in a versioned manner with maven)
(defn list-path [path] (script (ls ~path) (ls ~(.replace path "some" "other)))
Pallet allows the scripts to be combined. do-script concatenates the code pieces together.
(do-script (list-path "path1") (list-path "path2"))
chain-script chains the scripts together with '&&'.
(chain-script (list-path "path1") (list-path "path2"))
checked-script finally allows the chaining of scripts, and calls exit if the chain fails.
(checked-script "Message" (list-path "path1") (list-path "path2"))
Writing shell script in Clojure gives access to Clojure's namespace facility allowing modularised shell script, and to Clojure's packaging as jar files, which allows reuse and distribution. The ability to compose script fragments leads to being able to macro-like functions, such checked-script, and you could even use Clojure macros to generate script (but I haven't thought of a use for that, yet).
The syntax for the embedding has arisen out of practical usage, so is far from complete, and can definitely be improved. I look forward to hearing your feedback!
UPDATE: stevedore now requires a binding for template, to specify the target for the script generation. This should be a vector containing one of :ubuntu, :centos, or :darwin, and one of :aptitude, :yum, :brew.
]]>Consider the following example, adapted from debug-repl:
(let [c 1
d 2] (defn a [b c] (swank.core/break) d)) (a "foo" "bar") Running this now brings up the following SLDB debug frame:
BREAK: [Thrown class java.lang.Exception]Restarts: 0: [QUIT] Quit to the SLIME top level 1: [CONTINUE] Continue from breakpoint
Backtrace: 0: user$eval1666$a1667.invoke(NOSOURCEFILE:1) 1: user$eval__1670.invoke(NOSOURCEFILE:1) 2: clojure.lang.Compiler.eval(Compiler.java:5358) 3: clojure.lang.Compiler.eval(Compiler.java:5326) 4: clojure.core$eval__4157.invoke(core.clj:2139) –more–
As you can see, the stack trace reflects the location of the breakpoint, and there is a CONTINUE restart. Pressing "1", or Enter on the CONTINUE line, or clicking the CONTINUE line should all cause the the debugger frame to close, and the result of the function call, 2, to be displayed in the REPL frame.
Enter, or "t", on the first line of the stacktrace causes the local variables to be displayed:
BREAK: [Thrown class java.lang.Exception]Restarts: 0: [QUIT] Quit to the SLIME top level 1: [CONTINUE] Continue from breakpoint
Backtrace: 0: user$eval1666$a1667.invoke(NOSOURCEFILE:1) Locals: b = foo d = 2 c = bar 1: user$eval__1670.invoke(NOSOURCEFILE:1) 2: clojure.lang.Compiler.eval(Compiler.java:5358) 3: clojure.lang.Compiler.eval(Compiler.java:5326) 4: clojure.core$eval__4157.invoke(core.clj:2139) –more–
Pressing enter on one of the local variable lines will pull up the SLIME inspector with that value. If you go back to the REPL without closing the SLDB frame, there will be no prompt, but pressing enter should give you one. The local variables are then all be avaiable for evaluation form the REPL.
Should an error occur while you are using the REPL, you will be placed in a nested debug session, with an "ABORT" restart to return to the previous debug level.
Finally, restarts are now displayed for each of the exceptions in the exception cause chain.
(let [c 1
d 2] (defn a [b c] (throw (Exception. "top" (Exception. "nested" (Exception. "bottom")))) d)) (a "foo" "bar") This will bring up the debugger with 2 cause restarts, which can be used to examine the related stack traces.
top [Thrown class java.lang.Exception]Restarts: 0: [QUIT] Quit to the SLIME top level 1: [CAUSE1] Invoke debugger on cause nested [Thrown class java.lang.Exception] 2: [CAUSE2] Invoke debugger on cause bottom [Thrown class java.lang.Exception]
Backtrace: 0: user$eval1752$a1753.invoke(NOSOURCEFILE:1) 1: user$eval__1756.invoke(NOSOURCEFILE:1) 2: clojure.lang.Compiler.eval(Compiler.java:5358) 3: clojure.lang.Compiler.eval(Compiler.java:5326) 4: clojure.core$eval__4157.invoke(core.clj:2139) –more–
The break functionality is known only to work from the REPL thread at the moment. With that small proviso, I hope you enjoy the new functionality - at least it provides a basic debug functionality until full JPDA/JDI integration is tackled.
]]>I have released Criterium, a new project for benchmarking code in Clojure. I found Brent Broyer's article on Java benchmarking which explains many of the pitfalls of benchmarking on the JVM, and Criterion, a benchmarking library in Haskell.
The main issues with benchmarking on the JVM are associated with garbage collection, and with JIT compilation. It seems from Broyer's articles that we can mitigate the effects but not completely eliminate them, and Criterium follows his advice. Both of the above libraries use the bootstrap technique to estimate mean execution time and provide a confidence interval, and Criterium does likewise. At the moment the confidence intervals are biased and I still need to implement BCa or ABC to improve these.
One of the functions that I wanted to benchmark originally involved reading a file. Criterium does not yet address clearing I/O buffer cache, and I am not sure on the best way forward on this. On Mac OS X, the purge command can be used to clear the caches, and on Linux this can be achieved by writing to /proc/sys/vm/drop_caches. On the Mac at least, this causes everything to grind to halt for about five seconds, and there are then some file reads as whatever processes are running read things in again. This sort of behaviour doesn't lend itself to inclusion in a timing loop... Any suggestions?
FluidDB, a "cloud" based triple-store, where the objects are immutable and can be tagged by anyone, launched about a month ago. As a another step to getting up to speed with Clojure, I decided to write a client library, and clj-fluiddb was born. The code was very simple, especially as I could base the library on cl-fluiddb, a Common-Lisp library.
I have some ideas I want to try out using FluidDB. It's permission system is one of it's best features, together with the ability to use it for RDF like triples means that it could provide a usable basis for growing the semantic web. My ideas are less grandiose, but might take as long to develop, we'll see...
]]>I wanted to experiment with building a webapp using Clojure, so I tried setting up the Compojure web framework. I am new to clojure, so I am not sure if this is the preferred way of doing things, but here goes anyway.
There seem to be several ways to set up clojure in emacs. I ended up following Bill Clementson's instructions. A couple of years ago I had some experience using maven, so decided to use this to manage my classpath. Installing maven on my mac was simple with macports (sudo port install maven).
Setting up a POM for maven took longer than expected. Stuart Sierra's post pointed me to the formos maven repository containing the clojure snapshots. With some help from google, I also found the maven-clojure-plugin, which is a maven plugin for compiling clojure, and the clojureshell-maven-plugin which will start a swank session (or bare REPL) using the pom information.
With the basic clojure and maven setup in place, it was time to move on to compojure. I added the Compojure git repository into Bill Clementson's clj-build script, ran it to clone the repository, and then built it using ant (ant deps; ant). Jim Downing instructions for installing compojure into your local maven repository (mvn install:install-file -DgroupId=org.clojure -DartifactId=compojure -Dversion=1.0-SNAPSHOT -Dfile=compojure.jar -Dpackaging=jar) work smoothly.
I have spent the last few months with my latest start-up, Artfox, where I have been trying to push home some of the lean start-up advice expounded by Eric Ries and Steve Blank. I was hoping that "The Principles of Product Development Flow", by Donald Reinertsen, might help me in making a persuasive argument for some of the more troublesome concepts around minimum viable product and ensuring that feedback loops are in place with your customers as soon as possible. Unfortunately, I don't think that this is the book if you are looking for immediate, practical prescription, but it is a thought provoking, rigorous view of the product development process, that pulls together ideas from manufacturing, telecommunications and the Marines.
Perhaps Reinertsen's most accessible advice is that decisions in product development should be based on a strong economic foundation, pulled together by a concept of the "Cost of Delay". Rather than on relying on prescriptions for each of several interconnected metrics, such as efficiency and utilisation, Reinertsen suggests that economics will provide different targets for each of these metrics depending on the costs of the project at hand.
His proposition that product development organisations should measure "Design in Process", similar to the idea of "Intellectual Working In Process" proposed by Thomas Stewart in his book "Intellectual Capital", is what allows him to make the parallels to manufacturing and queueing theory and enables the application of the wide body of work in these fields to product development.
His practical advice, such as working in small batches and using a cadence for activities that require coordination, will come as no surprise to practitioners of agile development, and Reinertsen provides clear reasoning of why these practices work.
During my time at Alcan, and later Novelis, I gave a lot of thought to scheduling, queues and cycle times in a transformation based manufacturing environment, and I found that this had many parallels to his view of the product development process, and little in common with what Reinertsen describes as manufacturing, which seems to be limited to high volume assembly type operations. I found many ideas that could be usefully taken back to a manufacturing context.
If you look at this book as an introduction to scheduling, queueing theory and the reason's behind some of agile development practices, then you will not be disappointed.
]]>The facility of Ruby on Rails' test, development and production environments is one of those features that goes almost unremarked, but which makes using rails more pleasant. No doubt everyone has their own solution for this in other environments, and while I am sure Common Lisp is not lacking in examples, I have not seen an idiomatic implementation. In developing cl-blog-generator I came up with the following solution.
Configuration in Common Lisp usually depends on using special variables, which can be rebound across any block of code. I started by putting the configuration of my blog into s-expressions in files, but got tired of specifying the file names for different blogs. Instead, I created an association list for each configuration, and registered each using a symbol as key. I can now switch to a given environment by specifying the symbol for the environment.
The implementation (in src/configure.lisp under the GitHub repository) consists of two functions and a special variable. SET-ENVIRONMENT is used to register an environment, and CONFIGURE is used to make an environment active. The environments are stored in ENVIRONMENTS special as an association list. An example of setting up the configurations can be seen in the config.lisp file. In creating the configurations I drop the '*' from the special names.
I'm relatively new to CL, so let me now if I have overlooked anything. Writing this post makes me think I am missing a WITH-ENVIRONMENT macro ...
Twitter now lets developers build applications that take actions on your behalf without you ever having to divulge your password. Instead of asking you for your password, these applications ask Twitter to ask you for permission, and you give permission to the application while logged in to Twitter. What's even better is that you can revoke the application's permissions, from within Twitter, at any time, without having to change your password. The OAuth protocol makes this possible, and does so in a very secure manner.
Compare this to on-line transactions involving your credit card; you have to divulge your user-name (the name on your credit card) and your password (your credit card number) for every transaction. Some sites even store your card details, and you rely on trust, and the vendors good standing with the credit card company, that they will not make further transactions using your card. What happens should you find your card being abused? You have to go through the hassle of cancelling your card and obtaining a new card number, which you then have to divulge to all the companies that make regular charges to your card, instantly creating the opportunity for further abuse. To make matters worse, the credit card companies even give us these cards with our passwords on them (the card number), violating the "never write down your password (or PIN)" rule.
OAuth is an open protocol developed by some of the major web companies. The protocol is gaining traction rapidly, and the OAuth site lists some of the OAuth service providers (that is the sites that allow OAuth authorisation). Part of the reason for the rapid adoption is that is not really new technology. As mentioned in the OAuth design goals, the protocol is essentially a standardisation of the proprietary protocols used by Google’s AuthSub, AOL’s OpenAuth, Yahoo’s BBAuth and FlickrAuth and Facebook’s FacebookAuth.
Eran Hammer-Lahav provides a getting started with OAuth guide, Google provide a good overview of uses for OAuth, and the OAuth wiki provides links to all the details.
]]>I have now added a comment system to cl-blog-generator. My requirements were for a simple, low overhead, commenting system, preferable one that could possibly be fully automated.
The comment system was inspired by Chronicle's, with a slight modification in approach - the comments are never saved on the web server, and are just sent by email to a dedicated email address. Spam filtering is delegated to the whatever spam filtering is implemented on the mail server, or in your email client. The comment emails are then processed in CL using mel-base and written to the local filesystem. Moderation can optionally occur on the CL side, if that is preferable to using the email client.
There is still some work left to do - I would like to be able to switch off comments on individual posts, either on demand on after a default time period - but I thought I would let real world usage drive my development.
]]>I am an Opera user, these days mainly because it gives me integrated mail, feed and news reading, so that everything that comes from the web appears in one place. The last significant innovation I remember was the introduction of tabs, and that was some time ago (long before it made its way into IE, for example). I am a heavy user of tabs - it is not unusual for me to have over fifty pages open, as I tend to just open pages and rarely close them again. This means that the tab icons are unreadable, and Alt+Tab (I'm on a mac) produces three or four columns to scroll through to select the tab I'm after. I dream of a better tab navigation model, and would love to be able to search across all the open tabs. Surely it wouldn't be that hard to implement.
Maybe I'm being a little harsh as there have been some other innovations. Firefox's add-ons work seamlessly, we can use arbitrary search engines thanks to OpenSearch, and Conkeror gives full screen browsing with emacs key bindings, but for the most part it has been better support for standards that has dominated the release notes.
In preparing this post, I found that Google's Chrome has search over your page history. Sounds like that might fulfill my wish, though I wonder how its process per tab model will scale to lots of tabs. I just have to wait for its release on mac OS X.
]]>I recently uploaded some links to my cl-blog-generator project, and have been getting some feedback with comparisons to other blog site generators, or compilers, such as Steve Kemp's Chronicle, or Jekyll as used on GitHub Pages. Compared to these, cl-blog-generator is immature, but takes a different approach in several areas that Charles Stewart suggested might be worth exploring. I look forward to any comments you might have.
All the blog generators seem to use a file based approach for writing content, but they differ in the choice of input formats supported, and in the approach to templating. cl-blog-generator is the least flexible, requiring input in XHTML, while Chronicle allows HTML, Textile or Markdown, and Jekyll Textile or Markdown. For templates, Chronicle uses Perl's HTML::Template, and Jekyll uses Liquid. cl-blog-generator uses an approach which substitutes content into elements identified with specific id's or classes, similar to transforming the templates with XSLT.
cl-blog-generator's choice of XHTML input was driven by a requirement to enable the validation of post content in the editor, which is not possible using Chronicle's HTML input because of the headers and lack of a body or head element, and a desire to be able to use any CSS tricks I wanted, which ruled out Textile and Markdown, or any other markup language. The lack of an external templating engine in cl-blog-generator was driven by simplicity; I couldn't see a use for conditionals or loops given the fixed structure of the content, and this choice leads to templates that validate, unlike Jekyll, and which are not full of HTML comments. The current id and class naming scheme in cl-blog-generator could certainly use some refinement to improve the flexibility of the output content format, and I would definitely welcome requests for enhancements should the scheme not fit your requirements.
Perhaps the most significant difference in approach for cl-blog-generator is its use of a database and an explicit publish step. With cl-blog-generator a draft can exist anywhere in the filesystem, and must be "published" to be recognised by the blog site generator. The publishing process fills in some default metadata, such as post date, if this is not originally specified, copies the modified draft to a configurable location, and enters the metadata into the database. This ensures that the post is completely specified by its representation in the filesystem, and that the database is recreatable.
The database enables the partial regeneration of the site, without having to parse the whole site, and makes the linking of content much simpler. However, having Elephant as a dependency is probably the largest impediment to installation at present.
cl-blog-generator's input XHTML has been augmented to add elements for specifying post title, date, update date (which I believe is missing from the other systems), slug, description, and tags. On publising (see next section), any of these elements that is missing, except the mandatory title, is filled in with defaults.
Both Chronicle and Jekyll use a preamble to specify metadata, with the filename being used to generate the post's slug. Jekyll also uses the filename and its path for specifying the post date, and tags.
Finally, here is a grab bag of features.
Chronicle comes with a commenting system. cl-blog-generator generates a meta description element, which is used by search engines to generate link text. It also generates meta elements with links to the previous and next posts. Jekyll has a "Related posts" feature for generating links to similar posts. Chronicle and Jekyll both have migration scripts for importing content. Chronicle has a spooler for posting pre-written content at specific timesYesterday was frustrating; I spent far too long trying to debug some problems in a Rails application I am writing. Rails, and frameworks in general, are supposed to give us improved productivity by hiding the complexity and mechanics of the task at hand. This is great as long as the framework behaves as expected, but invariably causes problems when things go wrong.
belongsto association that was supposed to be populated in a beforevalidationoncreate callback. In my tests I noticed that the linked model was not being instantiated. After much searching, it turns out I had forgotten to create the foreign key field. Unfortunately Rails was silent on this issue and the belongs_to association code seemed to execute quite happily without the field. has_many associations, which I could populate with no problems. When I tried to access the association though, I kept getting Can't dup NilClass errors. This one turned out to be an issue with the generated collection.build method. As noted in the documentation by the somewhat cryptic Note: This only works if an associated object already exists, not if it‘s nil!, the method fails if the collection is empty (at least that's what I think it means). Explicitly instantiating the associated model and then adding it to the collection fixed the problem. In my application I had a model named Target, which meant that models that associate with Target, such as my TargetProfile model, have a target attribute.
target attribute in TargetProfile always returned the instance of TargetProfile - not quite what is expected. The problem was caused by the fact that ActiveRecord's AssociationProxy, used to implement associations between models, has a target attribute. The documentation contains another warning Don‘t create associations that have the same name as instance methods of ActiveRecord::Base, but mentions nothing of AssociationProxy, which isn't even part of the documented api. I call this broken encapsulation. While trying to validate the output of cl-blog-generator I needed a local DTD for XHTML. The textproc/xmlcatmgr package in Darwin Ports creates a catalog at /opt/local/etc/xml/catalog, but it does not include XHTML. A flattened XHTML DTD can be found in the w3 validator library and installed the DTD's under /opt/local/share/xml/, but I couldn't find a catalog file for it. It turns out it is pretty simple to write the catalog file; the Wikipedia XML Catalog entry has an example that contains what is needed. Save the example next to the XHTML DTD's as catalog.xml and adjust the paths, then add a "nextCatalog" entry in /opt/local/etc/xml/catalog pointing at the catalog.xml file.
Now I can use (setf cxml:catalog (cxml:make-catalog '("/opt/local/etc/xml/catalog"))) and CXML will use the local DTD specified in the catalog.
Today's surprise was that "2009-01-01" and "2009-1-1" get parsed differently by the YAML parser in Rails. The former gets converted to a Date, while the latter becomes a String. It confused me for a while, as the problem only showed up when I wanted to send the dates to a Flot chart. Looking at the standard, it's conforming behaviour. Must be me that is non-conforming then...