Writing npm modules in ES6 that run in ES5

Writing ES6 is lovely, but using it is a nightmare,” a colleague of mine remarked today.

Actually, he used a swear word.

The problem is we’re far from being ready to assume that every app or module that depends on our npm modules is capable of understanding ES6. If we want to be able to write our modules in ES6 we must transform them.

What should and should not be committed into source control

  • Only the original source files should be committed into repositories.
  • Never commit anything that is automatically built or compiled by a tool or script.
  • Never commit css that has been built from Sass.
  • Or javascript that has been transpiled by babel.

Why?

  • It’s confusing for developers coming to the project fresh. It’s often not clear which files they should edit.
  • Built files always drift out of sync with source files because someone always forgets to rebuild before committing.
  • It makes using GitHub’s web UI to make changes impractical or often impossible.
  • It messes up diffs and commit history.
  • Just never commit built files*.

* O.K. so this is like any other rule—break it before doing something even worse—but except on those occasions, definitely never do it!

Writing npm modules in ES6 so that they run in ES5

There are some convenient hooks in npm scripts where you can integrate any build steps for npm modules. One of them is prepublish that will, as the name suggests, run before npm pushes your module to the registry.

The following snippet in your package.json will convert all the files in src from ES6 to ES5 and pop the result a new folder called build:

  "main": "build/main.js",
  "scripts": {
    "prepublish": "babel src --out-dir build"
  }

Additionally, you can create an .npmignore file with src/ in it to prevent the original pre-transpiled ES6 code from being published to the npm registry.

Side effects

This has annoying consequences. Run git status after an npm publish and you’ll notice that, as expected, the built files have been generated—and git will tempt you to commit them.

As I’ve hopefully convinced you, committing built files is a Bad Idea™. Instead, you might consider adding /build/ to your .gitignore file.

This will solve the immediate issue of stopping you accidentally committing your built files into git but will create another issue.

npm publish will exclude all files matching the rules in .gitignore from being published to the registry. If you added /build/ to your project’s .gitignore the built JavaScript won’t be published and apps and modules depending on your component will break.

To fix this simply create a .npmignore file — its mere existence will prevent npm looking at our .gitignore file and our code will be properly published to the npm registry.

Use a .npmignore file to keep stuff out of your package. If there’s no .npmignore file, but there is a .gitignore file, then npm will ignore the stuff matched by the .gitignore file. If you want to include something that is excluded by your .gitignore file, you can create an empty .npmignore file to override it. — https://docs.npmjs.com/misc/developers

For our project we’ll probably want an .npmignore file that looks like this:-

src/

Side Effects, round 2

The npm command line tool will allow you to install dependencies from the registry — the normal way—or directly from git.

For example if you run npm install --save strongloop/express it will by the npm registry and go straight to GitHub to download express from there.

Because we’ve pointed the main property of the package.json of our module to a file that doesn’t exist in git and therefore doesn’t exist when our module is installed this way it will not work.

The fix? Commit the built files.

Haikro: better Heroku deploys for Node.js

There’s a lot to love about Heroku. Servers spin up instantly. Code deployments are quick. You can rollback to any old version of your application in just one click.

And much, much more.

But there’s a lot I don’t like about Heroku for Node.js web applications

By default it runs npm install to install dependencies as part of every deploy. Although it has some magic to cache those dependencies and the NPM registry is a lot more reliable than it used it be this still introduces some risk that if npm install can’t run, you won’t be able to deploy your app.

Also as our websites have gotten more complex, to keep our codebases tidy we’ve started using tools like SASS and Browserify to split our CSS and front-end JavaScript up across multiple files. This means that it’s quite normal for applications that I work on to need to be ‘built’ before they can be published on the web. If you’re using Heroku and need to run a lot of built steps as part of your deploy the options are even worse than for node_modules. You either need to remember to rebuild and commit the files that get generated into git before deploying — or run your entire build process on Heroku itself.

Running your build process on Heroku turns out to be quite difficult. Often build processes rely on a lot of tools — SASS, for example, relies on Ruby. If you run your build process on Heroku you’ll end up installing a lot of tools onto your web server that will only ever get used once, when the site is deployed. This slows down your deploy and makes it more fragile.

It is true that if you check your node_modules and, if you have them, build JavaScript and CSS files into git Heroku won’t need do all these steps on deploy but I prefer not to do that because doing this ruins ‘diffs’ previews between commits and means that you can’t make quick edits to code via the GitHub UI — need to run the whole build process for every change.

Introducing Haikro

Heroku actually supports two mechanisms for deploying code. You can either use Heroku toolbelt and (typically) typing git push heroku or they now also have a new API that can be used for deployments.

Unfortunately that new API is very sensitive to the format of the applications you give it to run. Because of this I’ve written a small wrapper around that API that can be dropped into any Node.js project which means that the code that is deployed onto Heroku no longer needs to be the same code that is checked into git. I’ve called it Haikro.

./node_modules/.bin/haikro build deploy \
	--app my-heroku-app \
	--heroku-token $(HEROKU_AUTH_TOKEN) \
	--commit `git rev-parse HEAD` \
	--verbose

I’ve tried my best not to reinvent too much and so pretty much everything about how you write Node.js apps for deploying via git push heroku should work for Haikro too, for example:-

Specific version of Node.js

To specify a particular version of Node.js add an ‘engines.node’ property to your package.json file with the semver of your desired Node.js version:-

	[…]
		"engines": {
			"node": "0.10.x"
		}
	[…]

Procfile

Also Procfiles for web nodes will continue to work (but not yet for worker nodes):-

web: node server/app.js

Bringing this together

This means you can pre-download your dependencies and run your build steps locally or as part of your Continuous Integration process (I’ve tested Codeship and Travis) and then all Heroku needs to do is run your application.

Continue to a full worked example using Grunt, SASS and Express…

Or, take a look at the Haikro repository

Yet another task runner

Now you’ve all heard of Grunt, Gulp, Broccoli and Duo I think it’s time for me to announce a task runner for JavaScript projects that I’ve been ahem working on*. It’s called scripts.

Adding tasks
Tasks are simply added to your project’s package.json like this:

[..]
  "scripts": {
    "test": "npm run jshint && npm run lintspaces && mocha",
    "jshint": "jshint *.js",
    "lintspaces": "lintspaces -ntd spaces -i js-comments -s 2 *.js"
  },
[..]

Running tasks
Simply type npm run <scriptname> into your Terminal. For the most commonly run script, test, you can even just type: npm test.

Pass information from task to task
By using the back-tick ` and npm run‘s silent mode you can even pass information from task to task:

[..]
  "scripts": {
    "test": "npm run jshint && npm run lintspaces && mocha",
    "jshint": "jshint `npm run -s js-files`",
    "lintspaces": "lintspaces -ntd spaces -i js-comments -s 2 `npm run -s js-files`",
    "js-files": "find . -name '*.js' ! -path './node_modules/*'"
  },
[..]

Here I am using npm run -s js-files to get a list of all the JavaScript files in my project, which are then being linted by Lintspaces and JSHint via npm run jshint and npm run lintspaces.

Plugins
It comes with support for JSHint, Browserify, and more – in fact because it works any tool that has a command line interface directly it supports everything! And you can say goodbye to installing foo-contrib-bar.

Shut up, Matt
If you haven’t guessed by now this post is intentionally a little bit tongue-in-cheek and provocative – but it is also a serious suggestion and I’m not the first person to suggest it.

Using npm scripts as your task runner has a number of quite compelling advantages:

  • If you have node, npm scripts is already installed.
  • With npm scripts you will have fewer dependencies because you install tools directly rather than the task-runner specific version of each tool, which makes them quicker to install and easier to update.
  • Reduces the number of files in your repositories (no need for an additional Gruntfile, Gulpfile, etc)

One obvious downside is that for complex projects the scripts section of your package.json files will start to get a little crowded. But for those cases there’s a natural upgrade path to make

Thank you and good night

* This might be a lie.

DOM Event Delegation without jQuery

When building the FT’s and Economist’s HTML5 apps we felt that as we were targeting only the latest browsers shipping the entirety of jQuery would be a bit – well – wasteful. What we wanted were small focused components that could be swapped in and out that we could pull in (initially) via npm, bower or (later) our build service. This thinking has since spread to the rest of the FT, who are now also moving away from jQuery.

So what is jQuery?

According the documentation, it’s quite a lot of things – as a very crude measure, its API docs have 593 articles. I don’t think I’m unusual in thinking that the vast majority of that I’ve never used and probably would never use.

For me what has made jQuery so helpful are its wrappers that make Ajax, DOM manipulation, DOM transversal and listening to events simple to do.

Since the ‘invention’ of jQuery the browser has completely changed. Whilst we sometimes need to maintain backwards compatibility, for example our stubborn friends in China who just refuse to let go of IE6, the browser now provides a huge amount of what jQuery gave us natively. (Specifically I’m thinking of things like querySelectorAll, classList, the new array methods)

Also after using TJ Holowaychuk’s SuperAgent I can’t look at jQuery’s Ajax API without seeing its idiosyncrasies. (Why is type not method!?)

What to do about event delegation?

But there was a piece of jQuery we needed that was missing in the component world and that is a nice library to help with event delegation (read more about Javascript event delegation and why you might use this pattern on Site Point). So we built one and called it FT DOM Delegate (or ‘dom-delegate’ on the npm or bower registries).

Without the baggage of an old API that everyone already knows we were able to start from scratch. So this is what we did:-

You decide which DOM element to listen to events on

var bodyListener = new Delegate(document.body);
var targetedListener = new Delegate(document.getElementById('my-el'));

Rather than listening to all events on the same place (usually document.body) FT DOM Delegate allows you to create more focused DOM Delegates that only listen to events within a specific element. This is really helpful for creating self-contained widgets or in single page applications like our’s where we dynamically load pages without refreshing the page (where each page might require a different set of event listeners).

Delegates can be killed

targetedListener.destroy();

Just one call to destroy and all events will be unbound, event listeners removed. On single page apps with views being rapidly created and destroyed – this is essential to prevent memory leaks.

We actually went a step further to make delegates recyclable. Via the delegate’s root method you can trivially attach and detach delegates to DOM nodes. This is useful as it allows you to completely re-render the pages’ HTML in javascript without having to re-attach all the event listeners individually.

<body>
  <section id="pane-1">
    <button>Click me</button>
  </section>
  <section id="pane-2">
    <button>No, Click me</button>
  </section>
</body>
var pane1 = document.getElementById('pane-1');
var pane2 = document.getElementById('pane-1');

var dd = new Delegate();
dd.on('click', 'button', function() {
  console.log("button clicked");
});

dd.root(pane1);
// Clicking 'Click me' => console.log
// Clicking 'No, click me' => nothing

dd.root(pane2);
// Clicking 'Click me' => nothing
// Clicking 'No, click me' => console.log

Delegates can be created without the DOM

Because Delegates can be detached from the DOM we realised that we didn’t actually need any DOM at all to be able to set up event listeners.

You can set up a delegate’s event listeners whenever you like, and when you are ready to actually start receiving those events, simply attach the delegate:-

var dd = new Delegate();
dd.on('click', '.close', function() {
 closeOverlay();
});

// ** some time later **
var overlay = document.getElementById('overlay');
dd.root(overlay);

Use capture for pros

Another area we felt was missing from other event libraries was that whilst they were extremely helpful in basic cases – because of the need to support legacy IE they didn’t give you access to decide whether you wanted your event listeners to be capturing or not. (For more detail on how DOM events and ‘useCapture’ work read my former colleague Wilson Page’s article on Smashing Mag)

Basically all events start at the document body then step through the DOM until they hit the element where the event (for example a click) was triggered. Then, if the event can bubble, it reverses back through the DOM until it hits the document body again. (Not all events bubble – for example error and blur events)

As the event moves from the document body towards the target element it is said to be in its capturing phase, when it reaches the target it is at target and is in its bubbling phase when it reverses back up through the document.

Sometimes when you are adding listeners you will want to specify which stage of the event flow you are interested in. Our DOM Delegate library allows you to do this via its fourth parameter:-

delegate.on('click', '.js-btn', function() {
  console.log("Caught event during capturing phase!");
}, true);
delegate.on('click', '.js-btn', function() {
  console.log("Caught event during bubbling phase!");
}, false);

Sensible defaults for capture phases

Some events don’t bubble – e.g. error, blur, focus, scroll, and resize – so for these (unless you specify otherwise) we set useCapture to be true by default.

This is handy as we like to handle all image load failures by hiding them, which we can do with just a few lines of code:-

var dd = new Delegate(document.body);
dd.on('error', 'img', function() {
  this.style.display = 'none';
});

Events for the future

With this library we believe we’ve made a really nice and absolutely tiny event delegation library that gives you as much power as the browser native methods – with some helpful methods that allow to you to easily tidy up after yourself. And we’re one step closer to kicking our jQuery addiction.

And, of course, it’s open source: https://github.com/ftlabs/ftdomdelegate


Update: Google also have a small javascript library that does a very similar job

Playing with Channel messaging

Whilst building some recent experiments with ServiceWorkers I’ve discovered a whole new API that I never knew existed: Channel messaging. Paper-clipped onto the end of the HTML5 Web Messaging specification, Channel messaging enables:

…independent pieces of code (e.g. running in different browsing contexts) to communicate directly
http://www.w3.org/TR/webmessaging/#channel-messaging

As far as I can see, they’re basically Javascript wormholes between different tabs and windows.

Wormholes in the real world

How do they work?

To create a new wormhole you call the MessageChannel constructor in the normal way:

var wormhole = new MessageChannel();

The wormhole has two portals, which are wormhole.port1 and wormhole.port2 and to send objects between one and the other you can postMessage the data on the sending port and listen to message message events on the receiving port.

One small complexity is that you won’t be able to listen to any of the incoming messages until start has been called on the receiving port.

Note: any data sent before the port has been opened will be lost – and there’s no way to interrogate the MessageChannel to find out whether a port is open or not.

Also note: as postMessage is asynchronous you can actually swap the wormhole.port2.start() and wormhole.port1.postMessage(‘HELLO’); around and it will still work.

var wormhole = new MessageChannel();
wormhole.port2.addEventListener('message', function(event) {
  console.log('port2 received:'+event.data);
});
wormhole.port2.start();
wormhole.port1.postMessage('HELLO');

See this for yourself on JSBin

It’s no fun to talk to yourself

Let’s now see if we can use a Shared Worker to wire two browser windows up with each other and see what we are able to send, window to window, tab to tab. The full code is up on GitHub and you can try it out there.

For this we’ll need two files: index.html and agent.js.

/agent.js

var mc;
onconnect = function(e) {
  var port = e.ports[0];
  if (mc) {
    port.postMessage({port:mc.port2}, [mc.port2]);
    mc = undefined;
  } else {
    mc = new MessageChannel();
    port.postMessage({port:mc.port1}, [mc.port1]);
  }
};

This is the SharedWorker. Every odd browser window that connects to it (ie. the 1st, 3rd, 5th, etc), it creates a new MessageChannel and passes one of the ports of that MessageChannel object to that browser window. It will also keep hold of a reference to the most recently created MessageChannel so that it can give the other port of it to the ‘even’ connecting browser windows (the 2nd, 3rd, 4th, …).

This allows the SharedWorker to hook up the browser windows, after which it can simply get out of the way – allowing the browser windows to talk to each other directly.

/index.html

&lt;!DOCTYPE HTML&gt;
&lt;title&gt;MessageChannel Demo&lt;/title&gt;
&lt;pre id=&quot;log&quot;&gt;Log:&lt;/pre&gt;
&lt;script&gt;
  var worker = new SharedWorker('agent.js');
  var log = document.getElementById('log');
  worker.port.onmessage = function(e) {
    window.portal = e.data.port;
    window.portal.start();
    window.portal.addEventListener('message', function(e) {
      log.innerText += '\n'+ (typeof e.data) + ' : ' + e.data;
    });
  }
&lt;/script&gt;
&lt;button onclick=&quot;window.portal.postMessage('hi');&quot;&gt;Send 'hi'&lt;/button&gt;
&lt;button onclick=&quot;var now = new Date();window.portal.postMessage(now);&quot;&gt;Send a date object&lt;/button&gt;
&lt;button onclick=&quot;var node = document.createElement('div');window.portal.postMessage(node);&quot;&gt;Send a dom node&lt;/button&gt;

This code will connect to the SharedWorker, wait for the SharedWorker to send it one of the ports of the MessageChannel (which the SharedWorker will create) and when it gets one, it will start listening to message events and print out the data it receives onto the web page.

I’ve also added some buttons so that it’s easy to test sending bits of data between the two browser windows. (Remember, you need to have two browser windows open for this to work)

Uncaught DataCloneError: Failed to execute ‘postMessage’ on ‘MessagePort’: An object could not be cloned.

Not every kind of javascript object can be sent in this way (which is why DOM nodes fail). According to the specification:

Posts a message to the given window. Messages can be structured objects, e.g. nested objects and arrays, can contain JavaScript values (strings, numbers, Dates, etc), and can contain certain data objects such as File Blob, FileList, and ArrayBuffer objects.
http://www.w3.org/TR/2012/WD-webmessaging-20120313/#posting-messages

Fixing Layout thrashing in the real world

This is sort of a rehash of Wilson Page’s ‘Preventing Layout Thrashing’ but I wanted to look at the topic with a practical example from the FT Web app. Much credit goes to him, Jamie Blair and Ada Edwards for their work creating, and then taming, FastDOM.


Layout Thrashing occurs when JavaScript violently writes, then reads, from the DOM, multiple times causing document reflows.
http://wilsonpage.co.uk/preventing-layout-thrashing/

Often talks on the subject of Layout Thrashing will start with a simple code sample that causes layout thrashing – and show how a developer might rework it to prevent it. Here’s one originally from one of Andrew Betts’s presentations:-

var h1 = element1.clientHeight;           // Read (measures the element)
element1.style.height = (h1 * 2) + 'px';  // Write (invalidates current layout)
var h2 = element2.clientHeight;           // Read (measure again, so must trigger layout)
element2.style.height = (h1 * 2) + 'px';  // Write (invalidates current layout)
var h3 = element3.clientHeight;           // Read (measure again, so must trigger layout)
element3.style.height = (h3 * 2) + 'px';  // Write (invalidates current layout)
etc.

Then there’ll be an example of how you might fix it:-

var h1 = element1.clientHeight;           // Read
var h2 = element2.clientHeight;           // Read
var h3 = element3.clientHeight;           // Read
element1.style.height = (h1 * 2) + 'px';  // Write (invalidates current layout)
element2.style.height = (h1 * 2) + 'px';  // Write (layout already invalidated)
element3.style.height = (h3 * 2) + 'px';  // Write (layout already invalidated)
etc.

Often though many presenters (myself included) will stop there and leave the pesky implementation details up to the developer to sort out.

The problem is nobody actually codes like this.

This is a screenshot from the project that I work on, the FT Web app:

webapp

When we can use CSS (which is immune to layout thrashing) to layout our pages we do but sometimes it’s not possible to do everything we need to in CSS. To give an example of the layout thrashing challenges we have within the web app I’ve highlighted three of the components that each require a little bit of javascript.

Component 1

It’s quite hard to see on the screenshot but we are required to add an ellipsis (…) when the text – which is displayed in two columns – overflows the component (look closely at the bottom right corner). Currently, multi-line ellipsis across multiple columns where the number of lines displayed is dynamic, dependent on the amount of space available, is not possible in CSS. Because of this we created the open source library, FT Ellipsis.

For ellipsis to work it is first required to measure the size of the container, and the number of lines contained within it; and then it has to write the ellipsis styling / insert any additional helper elements into the DOM.

Component 2

The amount of space allocated to component 2 is equal to the height of the window minus the header above it and the advert beneath it – this is done in CSS. However we want to be able to vary the number of lines shown per item. The more space available, the more lines shown – and this is not currently possible in CSS.

To achieve this layout we must first measure the amount of space there is available and then write the appropriate styles into the DOM to clip the text at the point where the component is not able to comfortably show any more text.

Component 3

Finally component 3 is a scrollable column. We would love to do this with pure CSS however the scrolling support for sub-elements on a page on touch devices is currently quite poor and so we must use a momentum scrolling library instead – we use FT Scroller but another popular open source scrolling library is iScroll.

In order to set up a scroller we must first measure the amount of space is available and then add some new layers and apply some new classes on elements on the page.

But we’ve used components! How could we be possibly causing layout thrashing?

Because we want to keep each component completely independent from every other, we store each component’s Javascript in a separate file. Taking the first component as an example, its implementation would look a bit like this:-

[...]

// Javascript to run when the component
// has been inserted into the page
insertedCallback: function() {
  this.ellipsis = new FTEllipsis(this._root);

  // Calculate the space available and figure out
  // where to apply the ellipsis (reads only)
  this.ellipsis.calc();

  // Actually apply the ellipsis styling/actually
  // insert ellipsis helper `div`s into the page.
  // (writes only)
  this.ellipsis.set();
}

[...]

At first glance this seems sensible and their own each component will be as performant as it can be.

Except when we bring the three components together

Because each setupCallback first does a bit of reading followed by a bit of writing, as the browser iterates through and runs them we will inadvertently cause ourselves Layout Thrashing – even though there is no code where it seems we have interleaved DOM reads and DOM writes.

So we created FastDOM.

FastDOM provides a common interface for batching DOM read/write work and internally it uses requestAnimationFrame.

Here’s the code sample above rewritten with FastDOM:

[...]

// Javascript to run when the component
// has been inserted into the page
insertedCallback: function() {
  this.ellipsis = new FTEllipsis(this._root);

  fastdom.read(function() {

    // Calculate the space available and figure out
    // where to apply the ellipsis (reads only)
    this.ellipsis.calc();

    fastdom.write(function() {

      // Actually apply the ellipsis styling/actually
      // insert ellipsis helper `div`s into the page.
      // (writes only)
      this.ellipsis.set();
    }, this);
  }, this);
}

[...]

So now when the ‘setupCallback‘s are run for each of the components, we don’t touch the DOM at all. Instead, we tell FastDOM that we want to do a bit of reading and then a bit of writing – and then allow FastDOM to sensibly order those operations. This eliminated Layout Thrashing.

Except that we had caused ourselves a thousand other problems instead

As the FT Web app is a single page app we are constantly loading and unloading pages, bringing new content and layouts into view – only to destroy them shortly after. In some circumstances the lifetime of any one of those pages can be very short. Sometimes even shorter than the lifetime of a requestAnimationFrame timeout.

And when that happened there would be nothing to unschedule the work we had deferred and, even though the DOM element already no longer existed, those FastDOM callbacks would try to do the work that had been assigned to them. Chrome Dev Tools console was full of errors.

We could have simply added a check at the beginning of every FastDOM callback to see if the element still existed but that would have to have been added in hundreds of places – and would probably be forgotten often. We needed to find a proper solution.

FastDOMs for everybody

In order for FastDOM to be effective there can only be one of them active on a page – it needs to be in overall control of all the DOM reads and writes in order to schedule them all at appropriate times. However, the downside of having a single queue for reads and writes is that it is very difficult to unschedule all the work scheduled by a single component.

What we needed was a way for each component to maintain its own queue of FastDOM work – whilst still leaving scheduling and processing of work to the single app-wide FastDOM.

So we created Instantiable FastDOM.

Instantiable FastDOM

The name is actually a little confusing because Instantiable FastDOMs aren’t really FastDOMs at all – they’re queues of work that has been scheduled in FastDOM (we’re thinking about changing this).

They are intended to be used by components so that components can easily clear any outstanding FastDOM work when they are destroyed.

So here is the code sample above rewritten with Instantiable FastDOM:

[...]

// Javascript to run when the component
// has been inserted into the page
insertedCallback: function() {
  this.ifd = new InstantiableFastDOM();
  this.ellipsis = new FTEllipsis(this._root);

  this.ifd.read(function() {

    // Calculate the space available and figure out
    // where to apply the ellipsis (reads only)
    this.ellipsis.calc();

    this.ifd.write(function() {

      // Actually apply the ellipsis styling/actually
      // insert ellipsis helper `div`s into the page.
      // (writes only)
      this.ellipsis.set();
    }, this);
  }, this);
},
removedCallback: function() {

  // Clear any pending work
  this.ifd.clear();
}
[...]

We have a winner

At long last we had a solution that:

  • eliminated layout thrashing;
  • didn’t cause hard javascript errors for its edge cases;
  • and didn’t add too much additional complexity to the implementations of each of our components.

What about 3rd parties?

No matter how performant and well written your application is, all that hard work can be completely undone by the potentially-not-as-well-informed developers of the widget you’ve been forced to embed on your application.

As usual there’s no magic answer – except that to hope that authors of Javascript libraries which interact with the DOM will split their instantiation logic up so that DOM reads and DOM writes can be run separately and users of those libraries can (if they want to) schedule those pieces of work in a sensible order.

Our ellipsis library, FTEllipsis, is our first example of an open source library that provides this flexibility by separating its instantiation logic into: calc (which only does DOM reads) and set (which only does DOM writes).

Layout Thrashing isn’t going away

As web components get ever closer and websites start to be built that adhere to their principles, if we don’t start using tools like FastDOM, those components are going to merrily read and write to the DOM without pausing to consider what other components might be doing – and Layout Thrashing is going to become harder and harder to avoid.

A terrible solution to Service Workers on http

I’ve been following the Service Worker http/https discussion quite closely lately. After years of waiting Service Worker is starting to feel really close. We even have our ‘first’ real demo outside of browserland.

Anyway here’s my terrible idea about serving Service Workers through http:-

Warning: This is a terrible idea and I can’t believe I’m actually going to suggest it. I have already wasted too many hours of my life creating and managing developer keys for native blackberry apps. Here goes.

So native apps solve this problem of verifying the authenticity of by requiring developers to sign apps with keys.

(Brushing all the implementation technicalities aside) could it be technically possible to verify the authenticity of a ServerWorker if it could be signed with a developer key – in the same way Android/Blackberry, etc native apps are?

Maybe this would be a lose, lose situation – doesn’t exactly make Service Worker that usable and also doesn’t improve the overall security of the web in the same way https does.

Also you’d probably want some kind of 3rd party to tie that key to the domain. Hmm. Sounds a lot like what https already does.

What does this actually solve

Well it means you don’t have to install a certificate for your domain, which means – if you’re using GitHub pages or Heroku – you can have a custom domain without paying $20/m (for Heroku, not possible at all for GitHub pages on https :( ). You might be using Cloudflare (who generously offer free http CDN services). If that’s the case you’ll save paying them some money too.

But it really solves it in the wrong place. https is the right choice and most of this just reimplements that, but badly. Basically, for developers to embrace this I think Heroku needs to lower the price it charges for running a small site behind https and/or GitHub need to start allowing people to upload their own certificates to GitHub pages.

Thank you for coming on this journey with me.

Beware of embedding tweets in full screen single page apps

Using components built by other people is fundamental to the success of any piece of technology. The more high quality physical and virtual components you can pull together, the less you need to build from scratch and the faster you can build things. We’ve been sharing and reusing code since the beginning of the web – and almost every web company that I can think of offers some way to embed their content on your site.

That’s all fine until you find the component does something that you don’t expect it to. For example, if the creator of the component made an assumption that is not true for your application, instead of saving you time it can cause problems for your application or the component itself. This happened to us when we tried to embed Tweets in the FT Web App.

This is an embedded Tweet:
(Please excuse the shameless self promotion)

One of the features the javascript Twitter use for embedded tweets on external websites has is that if you click reply or retweet instead of taking your user away from your website to Twitter, it will helpfully open a new, smaller window in which the user can use to post Tweets from, like this:

The problem is that the way this is implemented is that it doesn’t just affect the behaviour for links within the <blockquote class="twitter-tweet"> elements, it will listen to clicks on all links anywhere anywhere on your web page – and if the link is to a URL containing twitter.com/intent it will open a small new window.

To see this behaviour click here.

Interestingly it’ll also match links to others domains, as long as they contain the pattern twitter.com/intent/. Eg. http://mattandre.ws/twitter.com/intent/tweet. Play around with this on JSBin.

After a bit of digging, hidden in the minified code Twitter encourage you to use, are these few lines that are responsible for this:-

function m(e) {
  var t, r, i, s;
  e = e || window.event, t = e.target || e.srcElement;
  if (e.altKey || e.metaKey || e.shiftKey) return;
  while (t) {
    if (~n.indexOf(["A", "AREA"], t.nodeName))
      break;
    t = t.parentNode
  }
  t && t.href && (r = t.href.match(o), r && (s = v(t.href), s = s.replace(/^http[:]/, "https:"), s = s.replace(/^\/\//, "https://"), g(s, t), e.returnValue = !1, e.preventDefault && e.preventDefault()))
}

[...]

var o = /twitter\.com(\:\d{2,4})?\/intent\/(\w+)/, u = "scrollbars=yes,resizable=yes,toolbar=no,location=yes", a = 550, f = 520, l = screen.height, c = screen.width, h;
b.prototype = new t, n.aug(b.prototype, {render: function(e) {
  return h = this, window.__twitterIntentHandler || (document.addEventListener ? document.addEventListener("click", m, !1) : document.attachEvent && document.attachEvent("onclick", m), window.__twitterIntentHandler = !0), s.fulfill(document.body)
}}), b.open = g, e(b)

For most ordinary websites this behaviour wouldn’t be surprising – and probably even desired.

But our site ain’t no ordinary website. It’s one of those modern new fangled offline-first single page apps called the FT Web app.

Most of our users use our application full screen after it has been added to their (typically) iOS home screen. The problem is that we need to be in complete control (within javascript) of what happens when the user clicks any link because the default behaviour is fairly ugly (the application will suddenly close and the link will be opened in Safari). In order to make that experience a little less awful, in order to support external links like we first show the user a popup warning them that they’re about to leave the app like this:-

I’d be the first to admit that this isn’t exactly the pinnacle of user experience – it reminds me of the Microsoft Office paperclip helpfully double checking that you’re absolutely “sure you wanna exit?” but it’s the best we can do for now.

When we tried to start using Twitter’s embedded Tweet functionality we found that the code we’d carefully crafted to stop web links from inadvertently closing our full screen web app was being completely bypassed. In the end decided not to use Twitter’s javascript library.

It’s a little bit unfair that I’ve singled out Twitter, especially as they do provide the raw CSS to style Tweets without the Javascript that does all the weird stuff. In fact we’ve ended up shunning lots of different libraries for similar reasons (eg. jQuery and numerous advertising libraries) and every now and again one of our advertisers creates an advert that breaks critical features of our web application, which never fails to create a little excitement in the office. For being so adverse to externally written code, we’ve gained something of a reputation internally.

The fundamental problem is that unless you use an iframe to embed content (like YouTube does) – which causes numerous other problems for our web app so we don’t support either :( – the web is not encapsulated. If you add a 3rd party library to your web page, that library can do what it wants to your page and, short of just removing it, there isn’t always much you can do about it if it does do something you don’t agree with.

Guidance

If you’re building websites in non-standard ways (full screen ‘web apps’; packaged/hybrid apps; single page apps and/or offline first apps) don’t automatically assume that because you’re using ‘web technologies’ you will be able to use every existing library that was built for the web. All libraries – even modern, well written ones like the one Twitter use for embedding Tweets – are built with certain assumptions that may not be true for your product.

In the future Web components (via Shadow DOM) will finally bring the encapsulation that the web needs that will help us address some of these problems.

Hopefully iOS will also make the way it handles links in full screen web apps a little better too.

ServiceWorker and https

I recently attended State of the Browser in London and in one of the talks was a very convincing argument from Jake Archibald about the justification for requiring https for ServiceWorkers (the new HTML5 API that will replace web underdog AppCache).

So the logic goes:-

HTTP is wide-open to man-in-the-middle attacks. The router(s) & ISP you’re connected to could have freely modified the content of this page. It could alter the views I’m expressing here, or add login boxes that attempt to trick you. Even this paragraph could have been added by a third party to throw you off the scent. But it wasn’t. OR WAS IT? You don’t know. Even when you’re somewhere ‘safe’, caching tricks can make the ‘hacks’ live longer. We should be terrified that the majority of trusted-content websites (such as news sites) are served over HTTP.

To avoid giving attackers more power, service workers are HTTPS-only. This may be relaxed in future, but will come with heavy restrictions to prevent a MITM taking over HTTP sites.

http://jakearchibald.com/2014/service-worker-first-draft/

This makes sense. If I’m on a dodgy network I might not be too surprised if I saw something strange happening.

But I would be very surprised, and quite alarmed, if upon returning to the safety of my own home wifi, a trusted 3G connection or VPN if that strangeness didn’t go away. ServiceWorker, if it were enabled on http, would be wide open for this sort of attack.

This is where I start to disagree

The problem I have with this logic is that this sort of attack is already possible with AppCache. All you would need to do is serve a HTML page to any URL on a domain with manifest attribute like this:

<html manifest="my-evil-appcache.manifest">

And you would have hijacked that URL forever (or until they clear their browser cache). (Because the next time the user loads that page, the AppCache will try to do an update, attempt to download my-evil-appcache.manifest – and it’ll get a 404 – preventing it from updating… locking it in that state forever…)

For extra fun if you make sure to add a line with a forward slash in it you’d also get control of that domain’s home page:

CACHE:
/

So we should definitely require https on ServiceWorker then?

There’s an argument that says because you can take control of more URLs on a domain (with AppCache you have to specify them individually) so ServiceWorker is higher risk:-

But I feel that actually it only takes one URL to be compromised (say the basket of a popular shopping website that wasn’t served on http, which it would be safe to assume most users would go from straight to a payment gateway) for evil AppCache-savvy geniuses to cause some real harm.

All or nothing

I really don’t see how the ServiceWorker is any more dangerous than AppCache and we’ve had AppCache since Chrome 4 (2010) / even earlier in Firefox ~3.5 (~2008).

That said it does feel a bit risky and requiring https would be a good thing for the web in general, but I think it should be fixed for both technologies and for the browser to require https for both AppCache and ServiceWorker or just leave them both open. I just don’t think requiring https for one and not the other achieves very much…

(Alternatively give us an end-of-life date for AppCache 😉 )

HTML5 Offline Workshop this Autumn in Freiberg

I’m going to teaching a workshop on offline technologies in Freiberg at Smashing Conference this Autumn covering:-

  • A brief history of the offline web
  • Patterns for offline web applications
  • Cookies and Local Storage
  • IndexedDB and WebSQL
  • AppCache and ServiceWorker
  • Offline data sync strategies
  • Open-source libraries that can help us
  • Fallback techniques for older browsers, search-engine crawlers and users that do not need an offline experience

Come along, if you like :-)