Matt Andrews
Software engineer making apps – that aren’t apps – and more at the FT. 会说汉语.
Makefile reuse and inheritance

Reusable Makefile

Trends like microservices and technologies like AWS Lambda mean the way applications are built is changing. Before, a single monolithic application may have been contained within a single repository but now one product might be delivered by dozens or even hundreds of little sub-applications each in individual repositories where every one of those responsible for a small number of tasks.

Why you might do this is a topic for another time, but this approach makes maintaining build automation tools across an application harder.

Front end build automation tools like gulp and grunt have mature plugin framework. That means it’s easy to share solutions to common problems. You can create plugins that are downloaded at build time and can be shared across different repositories.

There is no package manager or registry for makefile. But there’s nothing to stop you putting Makefiles in npm (or bower, nuget, composer, etc) modules. You could even use git submodules.

Once you have a shared bit of makefile you can then use include to include it in another makefile.

include

The include [filenames…] ‘directive’ of makefile reads in the contents of all the filenames into the parent makefile before continuing.

filenames… can also be shell patterns.

For example adding include *.mk will include all files in the same directory that end with .mk.

The Financial Times’ Makefile for front end applications

The pattern we adopted was to have a single, centralised Makefile that got committed to each our applications in a file called n.Makefile. That was then pulled into each repository’s actual makefile by an include n.Makefile added at the top. That centralised makefile contains a simple update script that allows each repository to be upgraded to the latest version by running make update-tools.

That centralised makefile contained standard patterns for installing dependencies, building assets, deployments and linting.

Part of the philosophy behind it was to ensure that it was easy to override any part of it.

For example, if a developer wanted their repository use the default install task provided by the central makefile (run by the developer by typing make install) they wouldn’t need to do anything besides include n.Makefile. It would be provided by default.

However, if the developer wanted to write their own install task all they need to do is implement an install task in their Makefile. They can even call make install-super anywhere in their install task to run the shared makefile’s install task as well.

This might appear at first glance to be quite similar to inheritance except that make does not support inheritance. So how did we achieve this?

We basically hacked inheritance into make by exploiting wildcards. In our shared makefile instead of defining an install task, we define an instal% task.

This means that if there isn’t a task called install running make install will run the steps defined in the instal% task and if an install task gets added that will run whenever a developer runs make install instead.

For example:-

n.Makefile

1
2
instal%:
echo "shared install"

Makefile

1
2
3
4
5
include n.Makefile

install:
echo "repo install"
make install-super

Will do this:-

1
2
3
4
5
$ make install
echo "repo install"
repo install
echo "shared install"
shared install

We use -super as a suffix by convention but the way we’ve achieved inheritance actually means install-super could be swapped with any string that matches instal% and isn’t equal to install.

Make for hipsters

Introduction

Most tools fall into one of two types. Tools that are nice to use and tools that are ubiquitous. Make is the second type. Initially released in 1977, it has stubbornly refused to go away for four decades.

For our front end projects at the FT we built an incredibly powerful suite of gulp-based build tools (and then wrapped them in an even more feature-rich set of build tools). But, with an ever increasingly large dependency tree, they started to take more and more minutes to install. That was fine when we were making use of all their features, but often when we were building little prototypes or simple APIs we didn’t really need Sass or Gulp but ended up depending on it anyway because it was all bundled together.

When we started migrating some parts of our apps to AWS Lambda, which for the types of functions we were building, has no front end, forcing developers to install and use these gulp based tools on every build seemed crazy.

I took another look at make and these are the notes from what I learnt.

Warning: these notes are probably factually inaccurate, almost certainly promote bad practise, and are definitely delightfully hacky. Enjoy.

Make for beginners

Getting started

make is a command line tool. If you create an empty new directory and run it, you’ll get an error like:-

1
2
$ make
make: *** No targets specified and no makefile found. Stop.

You need to create a makefile to tell make what to do.

Advanced: What should I call my makefile?

By default, make will try the following filenames in the following order: GNUmakefile, makefile and Makefile but your makefile can have any filename you like.

If you’ve called it a name other than one of the defaults, you need to tell make that filename, which you can do by providing the -f option.

1
$ make -f MyMakefile

Make’s documentation recommends Makefile as the filename.

Rules

The fundamental feature of build automation tools like Make is to allow developers to define tasks. In makefile these are called rules.

A simple makefile rule looks like:-

1
2
target … : prerequisites …
[hard tab] recipe

An example of a makefile with a single, very basic rule could be:-

1
2
dothings:
echo "hello world"

In this case we have a single ‘target’, dothings, and a single line ‘recipe’, echo "hello world". The recipe simply prints out the text ‘hello world’ into the terminal.

Try this by runing that rule by typing make dothing into your terminal. It should output someting like:-

1
2
3
$ make dothings
echo "hello world"
hello world

To do more than one task, one at a time, just like out the tasks using spaces to separate them.

For example, for this make file:-

1
2
3
4
5
6
7
8
dothis:
echo "this"

dothat:
echo "that"

dotheother:
echo "theother"

Leads to this…

1
2
3
4
5
6
7
$ make dothis dothat dotheother
echo "this"
this
echo "that"
that
echo "the other"
the other

Advanced: only outputting the output

Make has done a little more than we’ve asked it too. In addition to running the ‘recipe’ for dothings, it has also output the recipe itself.

If you want to stop make from doing this either add an @ to the beginning of the line in the makefile, or use the -s option when you run make on the command line.

1
2
dothings:
@echo "hello world"

Or

1
2
$ make -s dothings
hello world

Doing two things at once

By default Make runs one task a time. You can allow make to do more than one thing at a time through the -j or --jobs option. So make dothis dothat dotheother -j2 will run two ‘jobs’ at a time, make dothis dothat dotheother -j will not limit the number of things it does at once.

Intermediate make

Writing the perfect install script

We use npm to manage dependencies for our NodeJS applications but the these notes apply to most other package managers (e.g. bower, composer, maven, etc).

npm explained in 30 seconds:-

  • When you run npm install, npm will download your project’s dependencies into a folder called node_modules (equivalent to bower’s bower_components or composer’s vendor directories)
  • npm dependencies are listed in a package.json file, typically stored in the root of the repositories (equivalent to maven’s pom.xml or ruby’s Gemfile)

We wanted to have a single rule that you could run on within any of our repositories that would install each repository’s dependencies. The first version looked a bit like this:-

1
2
install:
npm install

That worked and is simple except that if you ran make install twice in quick succession the second time, although faster than the first, would be quite slow.

One feature of make is that it checks if a file or folder matching the rule name exists first before executing that rule’s recipe — and it won’t run the recipe if that file or foler already exists.

So we can improve our make install rule by:-

  • adding a new makefile rule called node_modules,
  • making node_modules a ‘prerequisite’ of install,
  • and moving npm install to be run in the node_modules rule.
1
2
3
4
install: node_modules

node_modules:
npm install

This is great because once you have successfully installed all the repository’s dependencies if you run make install into your terminal make will instantly tell that there is Nothing to be done for 'install'.

Unfortunately this has introduced a problem. If you add a new dependency, for example, to your package.json file, make install will fail to run npm install and that new dependency will not be installed.

I actually lied to you earlier.

When I said that ‘one of the features of make was that it checks if a file or folder matching the rule name exists first before executing that rule’s recipe — and won’t run the recipe if that file or folder already exists’ I was oversimplifying things.

If that task’s name matches a file and that that has prerequisites that match files that exist, make will look the last modification date of those files with the files. If those files have changed more recently than the files that match the task name, it will run the recipe.

That’s quite difficult to understand so if you don’t get it, follow the example and hopefuly it will be a little clearer…

Our makefile will now look like this:-

1
2
3
4
install: node_modules

node_modules: package.json
npm install

Create a package.json file containing:-

1
2
3
4
5
{
"dependencies": {
"inherits": "^2.0.1"
}
}

Then run make install:-

1
2
3
$ make install
npm install
[email protected] node_modules/inherits

(Note: I’ve removed some npm warning that aren’t important for this discussion)

Then run make install again:-

1
2
$ make install
make: Nothing to be done for `install'.

Then edit package.json and change it to:-

1
2
3
4
5
6
{
"dependencies": {
"inherits": "^2.0.1",
"path-is-absolute": "^1.0.0"
}
}

Now run make install again and you’ll see that npm install runs again:-

1
2
3
$ make install
npm install
[email protected] node_modules/path-is-absolute

(Note: I’ve removed some npm warning that aren’t important for this discussion)

I now have to admit this whole section is a lie.

A perfect install script isn’t possible as it’s not smart enough to know to only run npm install if something about the dependencies within the package.json file changes.

But still, it’s pretty good.

Using the makefile language

What you need to know

  • You can write simple programs with the makefile language.
  • A typical makefile will contain a mixture of the makefile language and bash.
  • A single line can contain both the makefile language as well as bash.
  • bash can only be written in the recipe section of rules.
  • You can write the makefile language anywhere within a makefile.
  • Every line of bash is isolated from every other.
  • The parts of the makefile written in makefile are executed once when the makefile is compiled.

Basic functions

A function call looks like this:-

1
$(function argument,argument,…)

(You can use either normal () brackets or curly {} brackets)

A very simple (and not very useful) example of a function is subst, which does a find and replace of text in a string.

1
$(subst from,to,text)

For example:

1
2
secure:
echo "$(subst http,https,http://mattandre.ws)"

Results in:

1
2
3
$ make secure
echo "https://mattandre.ws"
https://mattandre.ws

Note: you might be tempted to add some whitespace to make things look a bit neater, for example:-

1
2
secure-with-space:
echo "$(subst http, https, http://mattandre.ws)"

Don’t do this. That whitespace matters to make and will try to replace the string http with the string https (rather than the expected https) in the string http://mattandre.ws (rather than the expected http://mattandre.ws). Sometimes this doesn’t matter but often it will. It’s best to not include any whitespace between arguments in Make function calls.

In the above example results in *two additional and probably unwanted spaces being added to the beginning of the domain name.

1
2
3
$ make secure-with-space
echo " https://mattandre.ws"
https://mattandre.ws

You can read more about make functions in the manual or see some creative uses of them in the Financial Times’ shared front end application Makefile.

Writing npm modules in ES6 that run in ES5

Writing ES6 is lovely, but using it is a nightmare,” a colleague of mine remarked today.

Actually, he used a swear word.

The problem is we’re far from being ready to assume that every app or module that depends on our npm modules is capable of understanding ES6. If we want to be able to write our modules in ES6 we must transform them.

What should and should not be committed into source control

  • Only the original source files should be committed into repositories.
  • Never commit anything that is automatically built or compiled by a tool or script.
  • Never commit css that has been built from Sass.
  • Or javascript that has been transpiled by babel.

Why?

  • It’s confusing for developers coming to the project fresh. It’s often not clear which files they should edit.
  • Built files always drift out of sync with source files because someone always forgets to rebuild before committing.
  • It makes using GitHub’s web UI to make changes impractical or often impossible.
  • It messes up diffs and commit history.
  • Just never commit built files*.

* O.K. so this is like any other rule—break it before doing something even worse—but except on those occasions, definitely never do it!

Writing npm modules in ES6 so that they run in ES5

There are some convenient hooks in npm scripts where you can integrate any build steps for npm modules. One of them is prepublish that will, as the name suggests, run before npm pushes your module to the registry.

The following snippet in your package.json will convert all the files in src from ES6 to ES5 and pop the result a new folder called build:

1
2
3
4
"main": "build/main.js",
"scripts": {
"prepublish": "babel src --out-dir build"
}

Additionally, you can create an .npmignore file with src/ in it to prevent the original pre-transpiled ES6 code from being published to the npm registry.

Side effects

This has annoying consequences. Run git status after an npm publish and you’ll notice that, as expected, the built files have been generated—and git will tempt you to commit them.

As I’ve hopefully convinced you, committing built files is a Bad Idea™. Instead, you might consider adding /build/ to your .gitignore file.

This will solve the immediate issue of stopping you accidentally committing your built files into git but will create another issue.

npm publish will exclude all files matching the rules in .gitignore from being published to the registry. If you added /build/ to your project’s .gitignore the built JavaScript won’t be published and apps and modules depending on your component will break.

To fix this simply create a .npmignore file — its mere existence will prevent npm looking at our .gitignore file and our code will be properly published to the npm registry.

Use a .npmignore file to keep stuff out of your package. If there’s no .npmignore file, but there is a .gitignore file, then npm will ignore the stuff matched by the .gitignore file. If you want to include something that is excluded by your .gitignore file, you can create an empty .npmignore file to override it. — https://docs.npmjs.com/misc/developers

For our project we’ll probably want an .npmignore file that looks like this:-

1
src/

Side Effects, round 2

The npm command line tool will allow you to install dependencies from the registry — the normal way—or directly from git.

For example if you run npm install --save strongloop/express it will bypass the npm registry and go straight to GitHub to download express from there.

Because we’ve pointed the main property of the package.json of our module to a file that doesn’t exist in git and therefore doesn’t exist when our module is installed this way it will not work.

The fix? Commit the built files.

Haikro: better Heroku deploys for Node.js

There’s a lot to love about Heroku. Servers spin up instantly. Code deployments are quick. You can rollback to any old version of your application in just one click.

And much, much more.

But there’s a lot I don’t like about Heroku for Node.js web applications

By default it runs npm install to install dependencies as part of every deploy. Although it has some magic to cache those dependencies and the NPM registry is a lot more reliable than it used it be this still introduces some risk that if npm install can’t run, you won’t be able to deploy your app.

Also as our websites have gotten more complex, to keep our codebases tidy we’ve started using tools like SASS and Browserify to split our CSS and front-end JavaScript up across multiple files. This means that it’s quite normal for applications that I work on to need to be ‘built’ before they can be published on the web. If you’re using Heroku and need to run a lot of built steps as part of your deploy the options are even worse than for node_modules. You either need to remember to rebuild and commit the files that get generated into git before deploying — or run your entire build process on Heroku itself.

Running your build process on Heroku turns out to be quite difficult. Often build processes rely on a lot of tools — SASS, for example, relies on Ruby. If you run your build process on Heroku you’ll end up installing a lot of tools onto your web server that will only ever get used once, when the site is deployed. This slows down your deploy and makes it more fragile.

It is true that if you check your node_modules and, if you have them, build JavaScript and CSS files into git Heroku won’t need do all these steps on deploy but I prefer not to do that because doing this ruins ‘diffs’ previews between commits and means that you can’t make quick edits to code via the GitHub UI — need to run the whole build process for every change.

Introducing Haikro

Heroku actually supports two mechanisms for deploying code. You can either use Heroku toolbelt and (typically) typing git push heroku or they now also have a new API that can be used for deployments.

Unfortunately that new API is very sensitive to the format of the applications you give it to run. Because of this I’ve written a small wrapper around that API that can be dropped into any Node.js project which means that the code that is deployed onto Heroku no longer needs to be the same code that is checked into git. I’ve called it Haikro.

1
2
3
4
5
./node_modules/.bin/haikro build deploy \
--app my-heroku-app \
--heroku-token $(HEROKU_AUTH_TOKEN) \
--commit `git rev-parse HEAD` \
--verbose

I’ve tried my best not to reinvent too much and so pretty much everything about how you write Node.js apps for deploying via git push heroku should work for Haikro too, for example:-

Specific version of Node.js

To specify a particular version of Node.js add an ‘engines.node’ property to your package.json file with the semver of your desired Node.js version:-

1
2
3
4
5
[…]
"engines": {
"node": "0.10.x"
}
[…]

Procfile

Also Procfiles for web nodes will continue to work (but not yet for worker nodes):-

1
web: node server/app.js

Bringing this together

This means you can pre-download your dependencies and run your build steps locally or as part of your Continuous Integration process (I’ve tested Codeship, Travis and CircleCI) and then all Heroku needs to do is run your application.

Continue to a full worked example using Grunt, SASS and Express…

Or, take a look at the Haikro repository.

Yet another task runner

Now you’ve all heard of Grunt, Gulp, Broccoli and Duo I think it’s time for me to announce a task runner for JavaScript projects that I’ve been ahem working on. It’s called *scripts.

Adding tasks

Tasks are simply added to your project’s package.json like this:

1
2
3
4
5
6
7
[..]
"scripts": {
"test": "npm run jshint && npm run lintspaces && mocha",
"jshint": "jshint *.js",
"lintspaces": "lintspaces -ntd spaces -i js-comments -s 2 *.js"
},
[..]

Running tasks

Simply type npm run <scriptname> into your Terminal. For the most commonly run script, test, you can even just type: npm test.

Pass information from task to task

By using the back-tick ` and npm run‘s silent mode you can even pass information from task to task:

1
2
3
4
5
6
7
8
[..]
"scripts": {
"test": "npm run jshint && npm run lintspaces && mocha",
"jshint": "jshint `npm run -s js-files`",
"lintspaces": "lintspaces -ntd spaces -i js-comments -s 2 `npm run -s js-files`",
"js-files": "find . -name '*.js' ! -path './node_modules/*'"
},
[..]

Here I am using npm run -s js-files to get a list of all the JavaScript files in my project, which are then being linted by Lintspaces and JSHint via npm run jshint and npm run lintspaces.

Plugins

It comes with support for JSHint, Browserify, and more – in fact because it works any tool that has a command line interface directly it supports everything! And you can say goodbye to installing foo-contrib-bar.

Shut up, Matt

If you haven’t guessed by now this post is intentionally a little bit tongue-in-cheek and provocative – but it is also a serious suggestion and I’m not the first person to suggest it.

Using npm scripts as your task runner has a number of quite compelling advantages:

  • If you have node, npm scripts is already installed.
  • With npm scripts you will have fewer dependencies because you install tools directly rather than the task-runner specific version of each tool, which makes them quicker to install and easier to update.
  • Reduces the number of files in your repositories (no need for an additional Gruntfile, Gulpfile, etc)

One obvious downside is that for complex projects the scripts section of your package.json files will start to get a little crowded. But for those cases there’s a natural upgrade path to make

Trollface Emoji

* This might be a lie.