Generating Source Files with Leiningen

Recently, we needed to include some generated source files in a project. The source code generation was project specific, so we didn't want to have to create a leiningen plugin specifically for it. To get this to work required using quite a few of leiningen's features.

This post will explain how to use lein to customise you build to generates a source file, but many of the steps are useful to implement any form of lein build customisation.

The Generator

The source code generator is going to live in the my.src-generator namespace. Here's an example, that just generates a namespace declaration for the my.gen namespace under target/generated/my/gen.clj.

(ns my.src-generator
  (:require [clojure.java.io :refer [file]]))

(defn generate []
  (doto (file "target" "generated" "my" "gen.clj")
    (-> #(.getParentFile) #(.mkdirs))
    (spit "(ns my.gen)")))

Development only code

The source generation code should not be packaged in the jar, so we place it in dev-src/my/src_generator.clj, and add dev-src and the generated source directories to the :dev profile's :source-paths. The :dev profile is automatically used by leiningen unless it is producing a jar file. When producing the jar, the dev profile will not be used, so dev-src will not be on the :source-path (we add the generated directory to the base :source-path below).

:profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}}

Running project specific code with leininingen

The run task can be used to invoke code in your project. To use lein's run task we need to add a -main function to the my.src-generator namespace.

(defn -main [& args]
  (generate))

In the project.clj file we also tell lein about the main namespace. In order to avoid AOT compilation of the main namespace, we mark it with :skip-aot metadata.

:main ^:skip-aot my.src-generator

Customising the jar contents

The generated files need to end up in the jar (and possibly be compiled), so we put them on the :source-paths in the project. If we had wanted to include the sources without further processing, we could have added the generated directory to :resource-paths instead.

:source-paths ["src" "target/generated"]

Extending the build process

Now we can tell lein to generate the source files whenever we use the project. We do this by adding the run task to the :prep-tasks key. Leiningen runs all the tasks in :prep-tasks before any task invoked by the lein command line.

The tricky bit here is that the run task will itself invoke the :prep-tasks, so we want to make sure we don't end up calling the task recursively and generating a stack overflow. To solve this, add a gen profile, and disable the prep tasks in it. We use the :replace metadata to ensure this definition takes precedence. See the leiningen profile documentation for more information on :replace and it's sibling :displace.

:gen {:prep-tasks ^:replace []}

Then use this profile when setting the :prep-tasks key in the project.

:prep-tasks [["with-profile" "+gen,+dev" "run"]  "compile"]

Now when we run any command, the sources are generated.

Adding an alias

Finally we may want to just invoke the source generation, so let's create an alias to make lein gen run the generator. We need the gen profile for this, or otherwise the generator will run twice.

:aliases {"gen" ["with-profile" "+gen,+dev" "run"]}

The final project.clj

For reference, the final project.clj looks like this:

(defproject my-proj "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.4.0"]]
  :source-paths ["src" "target/generated"]
  :main ^:skip-aot my.src-generator
  :prep-tasks [["with-profile" "+gen,+dev" "run"]  "compile"]
  :profiles {:dev {:source-paths ["src" "dev-src" "target/generated"]}
             :gen {:prep-tasks ^:replace []}}
  :aliases {"gen" ["with-profile" "+gen,+dev" "run"]})

Conclusion

This required using many of lein's features to get working - hopefully you'll find a use for some of them.

Discuss this post here.

Published: 2013-10-28

Alembic Reloads your Leiningen project.clj Dependencies

You're working away in a Clojure REPL, when you realise you need to add a dependency. You add the dependency to your leiningen project.clj file and then? Instead of shutting down your REPL, loosing whatever state you have built up, you can use Alembic to load the new dependencies. Simply call (alembic.still/load-project).

Of course, it still has to work within the confines of the JVM's classloaders, so you can only add dependencies, and not modify versions or remove dependencies, but this should still cover a lot of use cases.

To use alembic on a single project, simply add it as a dependency in your :dev profile in project.clj:

:profiles {:dev {:dependencies [[alembic "0.2.0"]]}}

To make alembic available in all your projects, and it to the :user profile in ~/.lein/profiles.clj instead:

{:user {:dependencies [[alembic "0.2.0"]]}}

Alembic also allows you to directly add dependencies without editing your project.clj file, using the distill function. Use this if you are just exploring libraries, for example.

Finally a big thank you to Anthony Grimes and the other flatland developers for removing classlojure's dependency on useful, which should make this all much more robust.