andyhume.net

HTML 5 introduced a much improved API to the browser’s history object. Specifically history.pushState() and history.replaceState() which allow developers to manipulate the contents of the history stack, and to detect changes to the active history entry using a new window event, popstate.

This is one of my favourite features which browsers are starting to support (including IE 10), because it allows us to throw away the large and non-trivial code typically used to store the state of the current page in the hash fragment. Before this new API, the only way of manipulating the URL was to update the fragment after the hash, and you’ll have likely seen particularly interactive web applications doing exactly this, including current versions of Gmail.

The first place I saw the new history API really well implemented was Github, where browsing a repository only loads the portions of the page which show the directory tree, and the rest of the page stays in its current state. This improves performance dramatically for a few reasons. Firstly, much smaller chunks of data are being sent over the wire, and secondly the browser doesn’t have to parse, reflow, and repaint as much of the page, or re-execute page-load JavaScript, etc…

The history API is pretty much win win in my book. Except when it’s used in the wrong scenario.

To understand what I mean you need to remember a little bit about the semantics of URIs, and resource-oriented principles. Without getting too bogged down, a URL points at a distinct resource or object, and a hash fragment (or fragment identifier) points to a particular location within that resource.

This leads to a couple of important principles:

If you’re updating the URL with pushState then that new URL must point at a new and distinct resource that exists on your server.

Once the URL has been updated and you refresh the browser, a document which reflects that changed URL must be returned. It’s not correct to return a page identical to the last and then do some client-side routing in JavaScript to setup the state of the page. That’s no different to reading the hash fragment and setting up state. In fact it’s worse, because you are exposing many URLs on the web which all return exactly the same document (apparently Google doesn’t like that either). It’s worse than a #! URL in many ways because it implies that a distinct resource will be returned when in reality it will not.

If you’re updating the state of a single page, continue to use a hash fragment to capture state.

This is particularly true if all you’re doing is exposing or drawing attention to a specific part of a document (eg, opening an accordion widget, or sliding through a carousel). It’s much more painful working with hash fragments than it is the history API, but there are good and well tested libraries available for doing this now, and I believe the semantic distinction is important to make.

It’s true that URLs are an important interface for humans using your web site, and that friendly structured URLs look nicer than those with query strings and hash fragments. But just as important are the principles of REST, of resource-oriented architectures, and of URL design. These are the things the web is built on, and if you start messing with them, the web will burn you.

Media queries are clearly a huge step forward for responsive layouts. Letting you apply different CSS based on the width of the device or the viewport opens up all sorts of possibilities for adapting content to best fit a given space. Media queries work really well when you want to adjust the core layouts of the site, but they’re less suited to changing styles at a smaller more granular level.

When I’m building a specific front-end component, I take great care to make sure it is completely decoupled from any particular layout. It’s not concerned with the width at which it might be displayed at: it could be moved from a small side bar into a larger main column and its width simply adapts to whatever its parent element constrains it to. At Clearleft we take this as far as building all the content components of a site outside of any layout constraints whatsoever.

However, there are occasions when you might want to adapt the styles within the component itself depending on the width at which it is currently being displayed at. There’s a very simple example on the Fontdeck sign up page where the form labels pop up above the text inputs if the viewport is below a certain width.

The example on Fontdeck uses a media query to adjust the display of the label elements when the viewport is more than 1000px, but surely there’s a better way than that? For example, if I moved that form into the right hand column, wouldn’t it be better if it knew what width it was being rendered at and automatically adjusted its layout to the narrower width version. In other words, the layout of the component responded based on the width of its containing element, rather than on the essentially arbitrary width of the viewport.

For fun, here’s some made-up syntax (which Jeremy has dubbed ‘selector queries’) that describes this:

I think there’s a valid use case for this capability. Being able to adjust a single component based on its current layout is extremely powerful. It allows more re-usability of code, and it encourages designers to think about how an individual component adapts on its own, outside of any given layout. Decoupling individual content components from their layout is a key principle of OOCSS, and this encourages that principle.

On a current development project I’m using a JavaScript technique to achieve nearly the same result. It uses a small script (about 1.5k minified) to add class names to elements depending on their width.

Here’s how you use it:

The above code tells the JavaScript to apply a class of wide when the elements offsetWidth is greater than 300 pixels and a class of small when it is less than 10 ems. It looks a little odd of course. I’d rather not have that presentational information in a data attribute (it’s hardly data)—but of course the best place for all of this would be in CSS, so how it’s exposed in the markup is kind of a moot point right now.

The script is on Github, along with an example test page (yes, try resizing the browser and changing text size). It works in all modern browsers back to and including IE6.

If you think about it, responsive layout is not a new thing. Open a simple HTML file in a web browser, and the content automatically adapts to fit the width of that browser. The web is responsive on its own—by default. It’s us that’s been breaking it all these years by placing content in fixed-width containers. Despite the fact that people have been arguing against this for years (please excuse the uncool URI).

Mark Boulton describes current thinking on this as a shift, rather than a trend. I prefer to think of it as a trend, but a trend that is coming to an end. Just as the Web Standards movement ended the unsatisfactory trend of presentational HTML and tables for layout, responsive layout ends the now old-school trend of designing for a 960px (or whatever) width Photoshop page.

The web has a way of fixing its bad habits over time. It’s smart like that.

What is responsiveness?

Responsiveness is what a website does when it’s loaded into an unknown browser on an unknown device by an unknown individual. It’s how you deal with “the most hostile software development environment ever imagined” (via Douglas Crockford). Like progressive enhancement it’s a strategy that frees you to work with the web rather than fight against it.

Optimising for One Web, instead of specific browsers/devices/individuals, is an ideal that is a profound part of being a web developer. It has to be at the heart of everything you design and build. If not, you’re doomed to write code and support an ever increasing array of client platforms from whatever the browser landscape throws at you in the coming years. Good luck with that.

You’ll note however, that I describe it as an ideal. It’s important to understand and accept that current technology and methodology won’t always get you where you need to be. Pretending we can create a perfect solution just because we’ve discovered media queries is as dangerous as ignoring responsive design altogether.

To begin with there are a bunch of things we can’t easily feature detect. Then there are bugs and quirks in the way browsers implement some of the newer technologies which make it difficult to work with them cross-browser. Then, you might find performance is so important that you want to feature detect completely through UA parsing (that’s an edge-case if ever there was one, but it’s legitimate).

Being a web application does not excuse you from these principles. I work mostly on content based sites these days, but that’s not my background. For the bulk of my career I worked on applications with 30-40,000 lines of JavaScript. The non-legacy bits that I was in charge of were built with the principles of progressive enhancement, and they worked on my fat desktop, my skinny mobile (when the CPU could keep up), and they worked when JavaScript was disabled. It wasn’t perfect, but that ideal we were aiming for normally kept us on course, and for the size of application we were dealing with we had a pretty maintainable and scalable codebase that lasted a good number of years and lived through a good number of browser and device releases (including the first generation iPhone).

The best web developers understand the medium, and work to its ideals. But they know what the current limits are and where they need to work around the problem in a way that gets the job done for 95% of users, even if it means a one-off well-understood browser detect.

For me responsive design is another example of web design getting back some of its Dao. That’s why it’s not an added extra or a feature. It’s core to what you do.

A few weeks ago, Andy Budd tweeted, One of the biggest problems in this industry is the assumption that complex, messy problems can have simple, well defined solutions., and then, Platonic idealism is a noble cause. However at its best it’s unrealistic and at its worst, actively damaging.

Okay, here’s the old adage: Good developers are lazy.

I understand the sentiment behind this. Good developers want to fix problems once, forever, then go to the pub. They want to write maintainable code, so they can update it easily when the requirements change, then go to the pub. They want to devise a model that captures every permutation of the design, then tweak the inputs and watch as everything goes beautifully according to plan.

This is noble, and is the right approach to building software and programming computers. But it’s only half the story.

The problem is, to quote Steve McConnell: nobody is really smart enough to program computers. Rarely is it as simple as you think it is, and rarely have you devised a model that captures every permutation.

There is a model that captures every permutation, and every nuance; but you’re not smart enough to figure it out. Even if you were, it would be ten times the amount of code and much more difficult to understand and explain. It wouldn’t be worth the time and effort because your simple model is probably 99% good, it just needs a few well-understood, well-coded and well-documented exceptions. The kind of approach this theoretically ‘good developer’ would rather not take.

In my experience this problem is particularly prevalent in UI code. A user interface is designed to interact directly with humans. Human minds work at a level of complexity that you aren’t able to easily model in code (duh). Designers spend a lot of time figuring out how these UIs should behave. They try and understand a user’s mental model of how something works, but they also test it and look at the behaviours that real people exhibit. Real people exhibit behaviours that often seem unintuitive. It’s sometimes hard to believe that people behave in particular ways until you actually see it happening.

This is why it’s a good idea to invite your developers to usability testing. If the developers aren’t exposed to why a UI is designed in a particular way, they’ll push back in favour of their simpler more consistent and understandable model of how it should work. “Well, the data once normalised is clearly structured like this, so why on earth would you expose it like that?”.

At Microsoft (where developers generally hold the balance of power anyway) I’ve seen particularly influential lead developers simply override the decisions of UX designers because a certain nuanced interaction doesn’t fit the way they’ve modelled it in code. This is not the right thing to be happening, and if you ever catch-me doing it then point me right back here.

Good developers work to an ideal, but they know when and how to work around the exceptions.

Towards the end of last year, Scott Jehl and the Filament Group released a small JavaScript library which experimented with the idea of “Responsive Images”. As the demo explains it can be used to load optimized, contextual image sizes in responsive layouts depending on the screen resolution. It cleverly uses HTML’s base element to redirect all initial image requests to a small 1×1 pixel place holder, so as to avoid loading more than one image.

This was principally developed to allow mobile devices to download smaller and more light-weight images, choosing which image to load based on the screen resolution of the device. As an approach this is useful in a very practical way, but it’s also an example of where one feature of a device is used to make assumptions about another feature. In this case, it’s the assumption that a small screen resolution means a low-bandwidth or low-speed connection.

This itself is driven by the sometimes misleading assertion that, “Mobile users need to save bandwidth”. That may be the case, but there is another rule that trumps it: “All users need to save bandwidth”. Performance matters, a lot, across every type of device and across every type of network connection.

A content first approach

Recently I’ve worked on a number of liquid width sites that have images styled at 100% width. This means the image could be displayed at varying sizes depending on the size of the browser’s viewport. This often leads to a scenario where quite high resolution images are loaded into the page for users that are only ever going to see squashed versions of the images.

The content driven approach to fixing this is to decide which image to load based on whether the image will be stretched beyond its true pixel width. If you stretch an image beyond its true width it begins to look pixelated or blurry. In this scenario, we want to load in a higher resolution version of the image.

I like the theory behind this approach because it embraces the principles of responsive design and applies them based on the best possible display of content rather than on essentially arbitrary and device specific pixel widths.

The code (forked from the Filament Group’s starting point) is available on Github.

At the moment, I would class it as proof-of-concept rather than bullet-proof, but I think it’s an interesting approach which is certainly worth sharing and discussing at this point.

Having said that, there is a small example of it working in the wild. This page for the Brighton Digital Festival contains images which respond in various ways to the size of the browser’s viewport. This is an interesting example because the images don’t always get smaller as the viewport gets smaller. Start with a wide browser window and the images are actually quite small. As you make the viewport smaller you will see the images decrease gradually in size and then jump to a larger size as the layout’s media queries kick in. If you refresh the page at this point you should see higher resolution images load in place of the original images.

This is a short comment on Ted Dziuba’s version of why MacOS X is an Unsuitable Platform for Web Development. Ted is right and wrong at the same time.

Unless you’re building the simplest of web sites with static pages or just PHP you shouldn’t develop on OS X. I used to. I used to compile MySQL and Apache from source, upgrade Python, install Django and a bunch of other Python tools, and leave myself in a hell of a mess, because yes, package management on OS X is a pain in the proverbial. And I hated doing this on the same machine that was my primary machine for email, web browsing, and other ‘desktop’ type tasks.

The obvious solution to this is virtualisation. Using Parallels, VMWare, or VirtualBox (my current choice) you can spin up as many sand-boxed environments as you like (which is why I don’t find much value in virtualenv). Using the same mechanisms you use to create production machines will get you most of the way to a development machine. Take a snapshot of that (or better still, use something like Vagrant) and you’ve got re-usable and shareable development images.

Why is this good?

You’ll catch bugs early. By developing code in an environment that closely matches production you’ll develop code that is more likely to work in that environment. You’ll get less bugs reported to you when your code moves into staging, or worse still production.

You’re sandboxed. You can feel free to install whatever dev tools you want on that machine, with no danger of screwing up your main desktop machine, or bloating it with crap you don’t need. Secondly, if you need a different environment for different projects you don’t have to manage them within one OS. I can’t imagine the hell on earth I’d go through to setup Apache with one lot of virtual hosts pointing at Python and Django, others pointing at Ruby on Rails, etc… all on OS X, where I write my email and sometimes open things like Photoshop and Fireworks.

You can still use OS X editors. I normally use Macfusion to mount appropriate directories of my VMs on OS X, so that I can use Textmate and other OS X tools to edit and manage code. I’m sure there is a better more fashionable approach than Macfusion, but I’m a creature of habit and it works for me.

The best thing about this is, despite OS X being an unsuitable platform for web development, you can still use it as your primary OS. It is a better, more stable, more pleasing and usable platform than any version of Windows or Ubuntu Desktop.

When we had new people joining at Multimap (or ‘onboarding’ at Microsoft) one of the initial big hurdles for them was learning the ins and outs of the approximately 30,000 lines of JavaScript that powered the site.

One thing I used to recommend people do was to turn on a little event logging switch we had in the code that logged to the console every event fired. The application made heavy use of custom events, so by watching these events occur in real time as you navigate and use the application, you can get a pretty good insight into what is going on in the guts of it. For example, a typical output might look like:

dragStart
dragEnd
mapMove
locationUpdate
geocodeSend
geocodeReceive
adUpdate
searchSend
searchReceive
panelOpen

Not only does this give you a broad overview of what might be going on, but knowing the names of the events gives you a starting point for jumping into the code. You can now search and find the module that triggers that event, as well as every piece of code that is listening for it and waiting to do stuff.

Recently I wanted to do the same thing in another JavaScript application which is using jQuery to do its custom events.

The above code monkey patches jQuery’s trigger event and logs the name of the event to the console. It doesn’t work for the browser’s built in events, (which is probably fine as you generally know what browser events are occurring), but it does work for all of jQuery’s built in events and any custom events that you trigger yourself.

I think this is a helpful way of starting to learn a codebase. In this case, it’s not so much new people joining that might benefit, but a client’s development team who’ll be taking this over eventually.

I’ve heard a lot of people recently describing HTML5 localStorage as being a replacement for cookies, or cookies on steroids. Even the masterful Bruce Lawson said pretty much this in his fantastic Web Anywhere talk at SxSW last month. I cringe every time I hear someone say this as it’s such an obviously dumbed down version of reality.

So, to state the obvious…

Cookies are small pieces (about 4 KB max) of data stored on the client which are sent in the headers of an HTTP request to the server, typically to allow for sessions or some other kind of state to be handled across requests. They contain expiry information and are typically created by the server using HTTP headers. There is also a JavaScript API that lets you access and edit cookies, although the less said about that the better. Cookies will continue to be a critical part of your web applications despite the introduction of localStorage.

localStorage is a persistent key-value store in the browser with a full JavaScript API to set/get/remove values and events to track changes programatically. It typically allows for up to 5MB of storage, although this can be increased at the user’s discretion. Data in localStorage isn’t sent to the server as it is with cookies.

localStorage allows for a whole range of uses that cookies do not, and cookies allow for a whole range of uses that localStorage does not. There is admittedly a small area of cross-over, but explaining it in those terms is backwards, hugely undervaluing the scope and importance of localStorage, and frankly condescending to the people you are explaining it to. Please stop doing it.

The next time you’re called to simply explain localStorage or sessionStorage in HTML5, don’t mention cookies. Just simply say, “localStorage is a powerful key-value store in the browser which you can access through a JavaScript API.”

Some of the best-supported HTML5 JavaScript APIs right now are the localStorage and sessionStorage APIs in the Web Storage specification. The latest version of all modern browsers (including IE8) support these, and some support the Web SQL Database API (although this spec is no longer active and isn’t being taken further).

What all of these APIs have in common is that anything a web developer chooses to stash in them is stored unencrypted on the user’s hard-drive for an indefinite amount of time. The spec suggests user-agents MAY allow for configuration of expiry times, but I don’t know of a browser that currently does that. This means that there is a constraint on the nature of the data that should be written to these storage systems.

Over the years, two things that I’ve been trained to have flags pop-up in my head for is “unencrypted data”, and “never expires”. Not major red flags that warn “here’s a massive security hole”, but small blueish flags that warn “there’s a limit to what we should do here”.

I don’t believe this is a flaw in any WHATWG or W3C specification (and the members of these groups that I spoke to don’t believe it’s a significant issue), but I do believe there’s a conversation that needs to take place amongst web developers. A conversation that makes people aware of the limits of client-side storage mechanisms, and gets them to take the question of what they store there seriously.

Mobile Gmail

Gmail’s mobile web app stores data client-side using the Web SQL Database API. It doesn’t just store small pieces of supporting data; it stores the full headers and body of your email locally in a SQLite database that can be read by any sufficiently privileged process running on that machine.

Again, there isn’t necessarily an inherent security problem here. If your machine or device has been compromised then it’s been compromised, and there’ll likely be any number of attack vectors that you’re open to. On the other hand, when I log into Gmail over SSL, carry out all communication in an encrypted format, read my email, then log out and leave the building, I don’t expect my emails to be sitting in plain text in a store that has no expiry directive.

There are certainly counter-arguments to this, most of them along the lines of the horse having already bolted, and if you use the web then data gets downloaded, unencrypted, and exposed in various ways. These are fair arguments, but it doesn’t mean we should forget about the whole thing and not consider how and what we explicitly ask a browser to store.

Things to consider when using localStorage

• What could an attacker do with this data? Make a judgement call based on the sensitivity of the information you are storing and the value of it to an attacker.
• Is sessionStorage more appropriate? Do I really need to persist this data (or a subset of it) across multiple browser sessions, or is the value mainly in sharing data between pages, or tabs within a single session?

This is a response to a blog post by Joe Hewitt, titled Web Technologies Need an Owner.

“Many people seem to assume that the Web will one day become the one and only client computing platform on Earth…”

I don’t think I have ever heard that seriously expressed by anyone. Let alone “many people”.

“…therefore it must not be controlled by anyone.”

Right. It must not be controlled by anyone. Because the goals of the web are lofty. They go infinitely beyond the goals of profit-making organisations creating a platform to make “real developers happy and productive”. The web does not need to compete with other client platforms – its goals are different.

“This is a dangerous assumption. The HTML, CSS, and JavaScript triumvirate are just another platform, like Windows and Android and iOS.”

No they are not. They are different at a fundamental level because of the open and uncontrolled platform that they enable. You don’t have to play if you don’t want to. Take the best of HTTP and go back to iOS, Android or where ever you want. Stop pitching the web in a battle against corporately owned platforms running on corporately controlled hardware and pretending one can win against the other. They’re different things and it’s not going to play out so black and white.

“…except that unlike those platforms, they do not have an owner to take responsibility for them.”

I have many many friends that take responsibility for the web every time they go to work, every time they publish a new article or commit to Github. I trust them so much more than I trust the powers that run your “cutting edge” platforms.

“The Web has no one who can ensure that the platform acquires cutting edge capabilities in a timely manner (camera access, anyone?). The Web has no one who can ensure that the platform makes real developers happy and productive. The Web has no one to ensure that it is competitive with other platforms, and so increasingly we are seeing developers investing their time in other platforms that serve their needs better.”

Much of that’s true, but I don’t see a problem with it. I don’t believe the web will suffer long-term because camera access is still a few years away (the cynic in me would say organisations like Apple and Google are stifling the capabilities of their browsers to ensure longer-term advantages for their native platforms. Sucks to be a web developer, hey?).

The web doesn’t have to ensure anything, because it’s not pitching itself against the things you think it’s pitching itself against.

“My prediction is that, unless the leadership vacuum is filled, the Web is going to retreat back to its origins as a network of hyperlinked documents.”

The web IS a network of hyperlinked documents. It’s not its origins. It’s its present, and its future. Please let it be its future.

“…but it will no longer be your primary window. The Web will no longer be the place for social networks, games, forums, photo sharing, music players, video players, word processors, calendaring, or anything interactive.”

I don’t agree, but I could live with that. I don’t mean to sound arrogant, but I have loftier goals than peddling any of those things you mention.

“Newspapers and blogs will be replaced by Facebook and Twitter”

… Um. I have no idea what that means.

“Let’s face facts: the Web will never be the dominant platform. There will forever be other important platforms competing for users’ time.”

Of course. Other important platforms that push the boundaries. That show standards bodies like the W3C where we want to go, and what might be possible. Much like Flash has done for the past 10 years. This isn’t a bad thing – and as a web developer I’m not scared of it, or wanting to fight against it. I want the web to do the things it’s good at, and I want it to get better and better at those things.

“The Web will be just another app that you use when you want to find some information.”

Just an app that you use when you want to find some information? I don’t want to say more. You’ve said all I could say.

“And so I end on a sad note.”

I’m sorry you’re sad. Don’t worry about the inertial scrolling – it’ll all come out in the wash.