Sunday, 30 December 2007

Compiling Common Lisp

When a language is pre-compiled (like C or C++) there is a step of compiling and linking the source code into a form that can be executed directly on the target hardware. The assembling of source into an executable form is usually handled by a build system like SCons or Make.

The CPython implementation of the Python language uses an interpreter. There is no compile step involved (it is implicitly compiled to bytecode).

Common Lisp implementations are somewhere in between in that there is a compile step but you can still use them as interpreters. For example, SBCL creates these files called FASLs which seem to stand stand for "FASt Loading". Their use should be self-explanatory. The formats are implementation-dependent meaning that they are not portable between implementations (not like .class files for Java, for example.) I couldn't figure out what SBCL stores in it's fasls but my guess is that it is a bytecode.

While developing a library, it is useful to have an interpreter-like environment. However, when you are using a library, you don't want to load and recompile the source every single time. What you want to do is compile the library's files into FASL format and have your implementation load them when necessary.

That is where ASDF comes in. ASDF, which stands for "Another System Definition Facility, is a way to define your projects so that they can be compiled and loaded along with whatever they depend on, in the right order. The rest of this post covers what I did to get a project and a testing system setup. Installing ASDF is out of scope here but if you use SBCL, you already have it.

First, the file system layout that I tend to use:

sohail@dev:~$ cd project/
sohail@dev:~/project$ find . -not -iname "*.fasl" -and -not -iname "*~"

ASDF looks for .asd (a system definition?) files in whatever paths are defined in the list asdf:*central-registry*. So the first thing you want to do is add the above path to the list. The way I did it (which is not optimal) is I modified ~/.sbclrc to include the following lines:

(require :asdf)
(push #p"/home/sohail/project/" asdf:*central-registry*)

Open up src/package.lisp in Emacs and enter the following:

(defpackage project
(:use common-lisp)
(:export some-function))

This uses the standard defpackage macro to define a Common Lisp package. A package is a mechanism to map symbols to names. The above exports a symbol called some-function. Next, open up src/code.lisp and enter the following:

(in-package #:project)

(defun some-function (a)
(format *standard-output* "a is: ~A" a))

This uses the standard in-package macro to set the current package to be project. If you are using Slime, you can try loading code.lisp (C-c C-l) and if you haven't loaded package.lisp, you should get a message telling you that project does not designate a package. To make it work, load package.lisp and then code.lisp. In a large project, it would be impossible to remember which order to load things in.

The next thing to do is to make this loadable via ASDF. To do this, open up project.asd and enter the following:

;;; Define a package to define our system definition
(defpackage #:project-asd
(:use :cl :asdf)) ;; Use ASDF (obviously!)

;;; All our definitions
(in-package project-asd)

;;; Our system is called project
(defsystem #:project
:name "My project!"
:components ((:module "src"
:components ((:file "package")
(:file "code"
:depends-on ("package"))))))

We document that code.lisp depends on package.lisp. Another way to write the defsystem could have been:

(defsystem #:project
:name "My project!"
:components ((:file "src/package")
(:file "src/code" :depends-on ("src/package"))))

I prefer using a module because if you have multiple modules that have dependencies, the dependencies are easier to define. For example, you might have a "model" module that depends on the "database" module.

Now go to the REPL and type (asdf:oos 'asdf:load-op #:project). Assuming you set up asdf:*central-registry* as above, you should get output like this:

; compiling file "/home/sohail/project/src/package.lisp" (written 30 DEC 2007 05:23:41 PM):
; compiling (DEFPACKAGE PROJECT ...)

; /home/sohail/project/src/package.fasl written
; compilation finished in 0:00:00
; compiling file "/home/sohail/project/src/code.lisp" (written 30 DEC 2007 05:23:37 PM):
; compiling (IN-PACKAGE #:PROJECT)
; compiling (DEFUN SOME-FUNCTION ...)

; /home/sohail/project/src/code.fasl written
; compilation finished in 0:00:00

Even though we told ASDF to "load" the system, since we hadn't compiled the files, ASDF compiled them for us, in the right order. Type (project:some-function 5) if you want to convince yourself that it worked!

Now we want to add a package to test our code. To do this, open up project-test.asd and enter the following:

(defpackage #:project-test-asd
(:use :cl :asdf))

(in-package project-test-asd)

(defsystem #:project-test
:name "Tests for my project!"
:depends-on (#:project)
:components ((:module "test"
:components ((:file "package")
(:file "tests"
:depends-on ("package"))))))

The only new thing here is the use of the :depends-on keyword argument to defsystem. Here, we are telling ASDF that before loading/compiling project-test, the project system must have done so successfully.

Mechanically, we add the following to test/package.lisp:

(defpackage #:project-test
(:use :cl) ;; Could also use the project package but I like to qualify symbols
(:export run-tests))

And the following into test/tests.lisp:

(in-package #:project-test)

(defmacro assert-string-equal (form1 form2)
`(if (string= ,form1 ,form2)
(print "Passed!")
(print "Failed!")))

(defun run-tests ()
(let (output)
(let ((*standard-output*
(project:some-function 5)
(setq output (get-output-stream-string *standard-output*)))
(assert-string-equal "a is: 5"

Now save all those files, go back to the REPL and type:

(asdf:oos 'asdf:load-op #:project-test)

You should see "Passed!" output.

Before you start writing your own test framework, take a look at FiveAM. You can also see my thoughts about FiveAM.

Happy Lisping!


xach said...

SBCL does not enable an interpreter by default. Everything you write or evaluate is compiled first.

SBCL fasl files contain machine code for your platform (x86, x86-64, PPC, etc).

I don't recommend defining a new system for your .asd files (#:project-asd). Just use (asdf:defpackage ...)

Sohail Somani said...

Thanks for clarifying.

Did you mean asdf:defsystem?

I don't remember where I picked up the habit of defining a package for the .asd but I realize its not necessary!