Making Bash Feel Modern

The Bash shell has a bunch of options that just aren’t turned on by default that change how it feels quite a bit. With the hype from zsh, I figured I’d mention the ones I use.

  • autocd is a no brainer. If you type out a name of a directory, it’ll cd into it. Tab completion still works of course, so it just saves typing ‘cd ’ all the time. It’s simpler, shorter, cleaner.
  • cdspell and dirspell are closely related; they try to find and fix typos in directory names, and then will run the new command for you. I haven’t compared their behavior to zsh’s one for one, but they seem to deal with small mistakes which is nice.
  • globstar allows you to use ** to recursively match directories. For example, **/index.php will match any index.php in the current directory or any subdirectory. Powerful and simple.
  • dotglob might not be as useful to everyone, but it let’s wildcards match hidden (dot) files. I’m used to it, and I like having all files be treated the same.
  • extglob also not might be as useful to everyone, but I find it useful. It enables “extended dash regex”, which you’d have to read more about in the bash man pages. The most common thing I use is !(…) to negate something. For example, rm !(*.php) would delete all non-php files.

Don’t get me wrong, there are some tricks that zsh (for example) can pull off that bash can’t, but most of the functionality does exist in bash, its just off by default. Most of these settings I’ve incorporated into overshell as well (whenever I get around to polishing that.)

Overload - Improved Aliases for the Command Line

What is this for?

So, first off, I use terminals (with bash) every day. They’re easily some of the most productive tools at my disposal for a wide range of tasks, from filtering data, manipulating files, or even opening tabs in batches.

Two of the most common tasks I do on the command line are using subversion and uploading files into development environments. I found myself wanting to tweak some of the behavior of subversion commands (svn add, for example) and sought out how I could do that, and came up with overload.

So what is overload?

Overload lets you override subcommands with shell functions. For example, I can define a behavior for ‘svn add’ without affecting the behavior of ‘svn.’ Further, I can define new subcommands for existing commands.

Here’s some examples.

svn_upstatus() { $original_command up "$@" $original_command st "$@"; }

This is a snippet from my .overloadrc file in my home directory. If I type ‘svn upstatus’ in a terminal, it calls overload, which looks for a function ‘svn_upstatus’ in .overloadrc, and then calls that function if it finds one (which it does.) $original_command is a special variable that overload sets before calling the function to be the overridden command (for me, /usr/bin/svn.) This is to avoid commands being remapped multiple times. Overload passes all remaining arguments into the function normally, so ‘svn upstatus app’ will call ‘svn up app’ and ‘svn st app’ in this case.

in my .overloadrc You’ll also see an svn_us function defined, which is the shorthand I use for svn_upstatus, but I felt it was good practice to define the long version as well.

I also have an override for ‘svn commit’ (and ‘svn ci’) to always call svn update before committing (just in case.)

A more interesting example inside my overload is ‘svn add’. For ‘svn add’, I check to see if there’s any additional arguments, and if there are, I let svn add work normally. If there are no arguments after ‘svn add’, I find all unknown files in the current directly and add them. Without this, ‘svn add’ with no additional arguments just throws and error complaining that you didn’t tell subversion what to add.

How to use it

  1. Put the script somewhere convienent and make sure its executable ($HOME/bin/overload, for example.)
  2. To your .bashrc, add ‘source $HOME/bin/overload’ followed whatever commands you want to use overload with (for example, source $HOME/bin/overload svn hg).
  3. Create a .overloadrc file in your $HOME directory and define your functions in there; you can use mine if you’d like to start.
  4. Open a new terminal or such, and start using overload.

How it works

The main ‘trick’ to overload is how it overrides ‘svn’ for example. It’s done by adding a line to your .bashrc file like this:

source $HOME/bin/overload svn hg

Yes, source first. Overload detects when its called by source and knows its configuring an environment and switches its behavior. The above example causes overload to configure the environment to use overload for svn and hg. It does this by creating functions (functions has higher priority than even the path) called ‘svn’ and ‘hg’ that call overload like this:

svn() { overload svn "$@" }

The rest isn’t quite so tricky. When overload is called, it sources your $HOME/.overloadrc, then looks for the longest function name it can find that matches its parameters underscorized. If it find a function, it calls it passing the unused parameters along.

Get it

The current version of overload and my current overloadrc are both on GitHub for your enjoyment.

Tumblr Editting

I’m not quite sure why Tumblr’s window for posting online is so small. So, I’ve made a chrome extension that’s just some CSS tweaks to that do a poor man’s job of making Tumblr’s dashboard and edit panes fluid width. I’ll throw up the extension after I do some more tweaking and try it out, for for now, here’s a screenshot:

Mind the Gap

Sorry that it’s been a while without me sharing any thoughts, but frankly, I haven’t had many thoughts that have felt interesting enough to share. I did write an entry on the iMarc blog about a month ago, but otherwise, I’ve been trying to spend a tad more time learning and less teaching than usual.

My keyboard obsession has really kicked up a few more pegs even since I wrote the article for the entry for iMarc. I’ve been spending time learning about setxkbmap and xmodmap and learning about a lot of the older modifiers that mostly vanished with the origins of modern day PCs.

Modifier keys are keys that do nothing by themselves, but modify the behavior of other keys. Shift is the most common of these. Control, Alt, Command, Fn, and the Windows keys are all othre examples of modifiers. Any off the the shelf mainstream keyboard, especially from 5–10 years ago would have four:

  • Shift,
  • Control,
  • Alt (also called Option on modern macs), and
  • Windows or Command
tBut this is far from a complete list. Most international keyboard layouts also include an AltGr key for typing non-ascii characters. The name AltGr loosely comes from Alternate Graphic. On keyboards with AltGr keys, some keys will have a second glyph (or graphic) on them, usually in the lower right. By holding the AltGr key, that other symbol is typed instead. Further, holding Shift and AltGr simultaneously will typically try to produce a capitalized version of the glyph (if applicable.) On modern macs, the Alt (Option) key behaves more similarly to how the AltGr behaves than Alt on other systems.

I don’t think I can say definitively, but from what I can tell, the space cadet keyboard (above) is right at the pinnacle of modifiers. It had seven modifiers:

  • Shift,
  • Control,
  • Top,
  • Front (or Greek),
  • Meta,
  • Super, and
  • Hyper.

I believe that Meta or Super is the closest to the modern Windows or Command key. I believe the other along with Control and Hyper were used for issuing other commands and keyboard shortcuts primarily, and not for symbols. Top and Front are loosely equivalent to two separate AltGr keys, where Top would access the other symbol on top of the key, and Front would access the third set of symbols printed on the front side of the keys.

Most people found this number of modifiers overkill, and while these will be remembered for a long time as being some of the most powerful and well constructed (hall effect mechanical key switches?) ever built, this layout was certainly overkill for what home users would have adopted then.

I’m not sure when it first happened, but I believe the Fn key came out of 90’s laptop keyboards. The Fn key is implemented very differently than the the other modifier keys; while it does switch the behavior of the keys while held, its not handled by the operating system, but rather by the keyboard itself. This means that the operating system didn’t need to implement any kind of additional support for it by moving the complexity into the keyboard itself. One of the unfortunate results of this is that while you can (more or less) remap any other key on the keyboard, the Fn key cannot be remapped unless there’s some special implementation for it (like a BIOS setting, or a dip switch on the bottom of the keyboard.)

This also moved over into lots of desktop keyboards. Nearly every Apple keyboard has an Fn key, and its fairly common for it to be used with the top row of F keys to access volume and media controls. Further, some keyboards (like Apple ones) realized that the Fn function of the keys were more useful for most users than the F keys they were under, and so they swapped the behavior; media keys by default, and you hold Fn to access the original F key.

So, we’re back up to five mainstream modifiers. However, as computing devices get smaller, we’re revisiting keyboards to see how much we can get rid of.

Possibly the easiest target these days is the CapsLock key. This key is rarely useful, provides no new functionality, and is in the home row, making it one of the most easily pressed keys on the keyboard. Google chromebooks changed this key into a search key; many other keyboards (or users at least) remap this key to be another Control key or sometimes a Backspace key.

Numeric pads are another large target that have been in decline for years now. They not only take up a lot of space, but they also greatly increase the distance between the right hand’s home position on the keyboard and the mouse. Along the same lines, the Insert key is slowly vanishing, and the rest of that cluster (Home, End, Page up, Page Down, Delete) are finding themselves new homes.

The arrow keys also, also slowly getting shoved into the main section of the keyboard. Sometimes they are at a reduced size (like Apple keyboards), sometimes they are full size and at the expense of the right modifier keys, and some keyboards even put them underneath letters so that you have to hold Fn to access them. For example, the KBC Poker hides them under WASD (matching a typical gaming convention), the KBt Pure puts them under OKL;, and the Happy Hacker Keyboard Pro (one of the most expensive, high end programming keyboards available) puts them under [;/’ in a diamond shape.

The HHKB is a great example of eliminating as many keys as possible; the whole F key row is collapsed down into the row below it and all keys to typically to the right of the enter key are merged in as well without making any extra small keys and only cutting into the size of the right Shift and Backspace keys. While I feel that some of the other ultra minimal layouts have solved some problems with the HHKB, I still feel that this direction merits attention as we try to find better ways to  fit keyboards onto tablets, tablet cases, and laptops.

Something that does seem to be missing from the HHKB and its peers is the curved, ergonomic shapes popularized in the early 2000s. I’m not convinced that the curved layouts were really a fad that will die out, so I am curious so see where these concepts will come back together.

Ultimately, I don’t think we’re able to drop the keyboard yet. Touchscreens on tablets may be more versatile, but they’ve been shown to be slower and have ergonomic problems after much heavy use. (This often reminds me of Star Trek: the Next Generation, where day in and day out everyone is seen swatting away at giant glass panels and walls.) The bar for replacing keyboards is set very high as well, as hot keys are a dominant way we control the computer itself.

If I had to hazard a guess of what concepts will go into replacing the keyboard as the fastest, most practical input device, I’d have to credit Swype. Swype is an Android only keyboard alternative, where instead of tapping on each letter of a word, you touchdown on the keyboard on the screen on the first key on the word, and drag your finger to each following letter in the word, and then release, drawing some kind of bizarre squiggle. Swype uses not on the motions you’ve made, but the timing of the motions, to guess the word you meant and does a fairly good job of it. It only requires one ‘tap’ per word, opposed to a tap per letter on a keyboard, and while its still not as fast a full size keyboard, it is the fastest input method for a cellphone sized device. Swype has all the benefits of gestures, but makes additional use of where on the screen you’re drawing the gesture, and that allows it to accept a large volume of input data and process it in a meaningful way.

I’m curious what the ‘full screen’, swype-inspired solution would be, which can make sure up to ten simultaneous contact points (instead of one) and a much larger surface.

Apple at the reins

Today is the last day Flash is available for Android devices. From my point of view, I will remember this as Steve Jobs’ (or at least, Steve Jobs’ Apple’s) crowning achievement: convincing their user base that they were better off letting Apple make decisions for them.

Enter Apple’s decision and Steve Jobs’ Thoughts on Flash. Jobs rant makes some valid points (proprietary, video encoding, touch.) Still, many of these aren’t really solved by not allowing for flash.

iPhone apps are certainly proprietary and worse, tied to exclusively Apple’s platform. Jobs mentions apps as solutions to ‘filling gaps’ (YouTube, games) but ultimately, Apple has pushed people to choose less open solutions controlled by Apple instead of Flash, which, in 2010, was available on nearly every platform.

HTML/CSS/Javascript also assumes a pointing device. Hover states are used constantly and are a prevalent problem with mobile sites. We run into this in particular with hover based drop down menus on mobile devices.

Apple’s motive was all about maintain control over what applications you could run on your iPhone. iOS-only applications entrench Apple’s devices by preventing alternatives from having access to any existing iOS software. Apple even went as so far in iOS 4 to prohibit other developer tools that could target multiple platforms including iOS (Flash-to-iPhone, MonoTouch, Unity3D, etc.) Most of these were exclusively targeting mobile devices. This throws out nearly all of Jobs’ points. Apple wants apple-only developers. Apple wants to make it as difficult as possible for you to switch to a non-Apple device.


People know how to figure out what applications they like or want to run. We ask each other, find sites with recommendations, and sometimes even just try out applications. This is how we figure out what software to run on our desktops or apps to install on our phones. We’re also smart enough to recognize patterns. Most people can recognize signs of a poorly made windows or java applications. We’re capable of determining what’s good and what isn’t. At the very worst, we know who to ask.

There is no App Store for Windows. The most popular desktop platform in the world let’s users find and install any software they’d like, and let’s their users determine what works well and what doesn’t. Apple’s App Store not only enforced rules against Flash or tools like Unity, but rules about design and content. Apple censors its own user base at its own discretion. Applications like Google Voice were even censored.

Something like 10% of iOS devices in the united states (and like 40% in China) are jailbroken simply to get around the limits the App Store imposes. Apple tried to make jailbreaking an iPhone illegal in the US, but eventually lost. Apple wants to maintain control over its user base.

So, today shouldn’t really be celebrated. Today is just another consequence of Apple abusing its user base and notable head start in the smart phone market to force decisions on an entire industry. The outcome was certainly better for Apple; less options is always worse for us. I’m still hoping that this kind of aggressively selfish behavior died off with Steve Jobs.


I can’t help but contrast this sort of compliance with the response SOPA and the other related proposals garnered from the community. SOPA’s stated goal was to solely to stop illegal activity, but there was a near unanimous rebuttal that we didn’t want to hand the ability to blacklist or censor the internet to the government. It’s curious that some people are willing to hand it over to Apple, who have already proven that they have no intention to only target illegal things.

Boy Scouts

I was a member of the Boy Scouts of America from a being a tiger scout (the lowest rank in cub scouts, I was probably seven) all the way up until I was eighteen and an eagle scout. I spent two summers working as a counselor at Camp Bell in Gilmanton Iron Works. Camp Bell was only in its second year when I worked there, being adjacent to Hidden Valley and on a newly reacquired section of the reservation. I also was a member of the Order of the Arrow and participated in some events related to that and helped staff some events because of that.

My personal experience through all of this was a resounding positive. I’d break down all of scouting into three messages that are continuously enforced:

  • You can do it. Especially in younger scouting, a lot of the message is showing kids what they’re capable of.
  • We can do it. Eventually, the focus shifts to showing kids what they’re capable of as a group working together.
  • They can do it. Lastly, the focus shifts to showing kids how those around them, peers or otherwise, are capable of a lot as well, and through responsibility and respect, we can help everyone around us do more.

Hidden Valley focuses primarily on the first of these three, whereas Camp Bell was more focused on the second. All of these messages are fairly universal, and I think help balance some of the competitive nature that most boys end up with at these ages.

These messages are not religious. While a scout,

  • I attended meetings held in church basements at two different catholic churches. We helped the churches by cleaning the grounds and other such tasks, but there was no religious context forced upon us.
  • Nearly all meals or events did involve some kind of blessing, but it was nearly always as universal as possible. It was expected that everyone be respectful during these, but nothing beyond that.
  • At no point was I forced to participate in any particular religious context. If there was any message, it was solely that churches often do take on responsibilities for the community and we were also capable of taking on: things like looking after moments and memorials, public property, etc.

The scout promise I learned was

On my honor I will do my best to do my duty to God and my country and to obey the Scout Law; to help other people at all times; to keep myself physically strong, mentally awake, and morally straight.

and the scout law:

A Scout is trustworthy, loyal, helpful, friendly, courteous, kind, obedient, cheerful, thrifty, brave, clean, and reverent.

"Reverent" here really could be thought of as honorable as respectful, and ultimately, not the source of the problem. "God" is. The Girl Scouts here have elected to allow the word "God" to be replaced a more suitable word for each person, however as far as I am aware, the boy scouts have not. Still, in practice, this reference to God felt no more weighty than it does singing "God Bless America", or say "under God" in the pledge of allegiance. It was there, but certainly never a focus of scouting.

While working at the scout camp, I got to meet lots of other scouts and scout leaders. All in all, there were very few problems; the most problematic people we dealt with on a regular basis were parents who didn’t trust their children, let alone us with their children. Some parents were entirely unwilling to believe that their child was capable of independence, and seemed  hellbent on making sure they didn’t learn it.

However, there was a specific week each summer that which had an abnormal share of problematic children. In this area, nearly all LDS troops calibrated their schedules so they all attended scouts camps simultaneously. That week was not enjoyable.

Now, there were staff members that were members of the LDS Church who felt the same way about this week, and there were plenty of kids who were well behaved. However, as near as I could tell, there was some significant extra pressure exerted on these children, whether by the church or congregation, to ensure that they went to scout camp. A kid who doesn’t want to be at scout camp as no interest in independence, cooperation, or respect, and is usually very disruptive in an environment that is designed to let kids self-manage. Interested LDS scouts had to miss out on their own experience trying to maintain control over others that weren’t; overall, we were limited to what we could do with the group and spent much more time solving disputes and enforcing consequences than we did for the rest of the summer.

I saw this same pattern both summers, and also got to see the difference when staffing events both groups attended. It was consistent.

Scouting’s implementation is very much up to each separate community. The national policies are no better a representation of scouts than our country’s national policies are for us. Worse, individuals have even less say in who is on the national council than we do in our own government.

I have no intention of distancing myself from the organization; I have every intention of making it known that the organization is run by individuals who have every intention of discouraging and eliminating the kind of intolerance the national council is promoting. Stripping medals from eagle scouts is an act of desperation from a national council that has no real way to control its own organization’s implementation and direction. Just as it takes citizens to change our countries laws, it will take scout leaders to change the scouting community. By their own design, they have trained the future generations of scout leaders to be able to reason independently, work together, and take on the responsibility of changing the scouting organization. It’s just a matter of time.

Unfiltered Part 2: Consequences

If you’d like, you can read what I wrote on this back in April.

The next generation won’t see people. Each generation has learned how to better filter the information than the generation before it. The media industry has been trying to find more effective ways of advertising or pushing opinions on us for decades, and so far, each generation has learned how to better cope with the newer tricks. The newest tricks these days are deeply connected to social media like Facebook, even finding ways to convince other people to do the advertising for them. People are already ready learning how to filter out noisy updates, meaningless status messages, and even poorly formed opinions. It only makes sense that we’d eventually give way to a generation even better at discerning value in people than we have.

One of the lowest forms of internet advertising, the popup, is almost over. I’m sure there there will always be bastions on the internet that will find more and more creative ways to open new windows and tabs without permission, but at least mainstream media and advertising have all realized that we’re so used to knowing that there will be nothing in the popup, that I’d go so far as to say it ads a negative connotation to the ad. I will think worse of your company if your site opens popups, and I’ll think worse of your company if it advertises in popups. So, this marketing fad has died off almost completely.

Still, not to be mean, but there are generations that are still susceptible to this kind of advertising. In particular, I’ve run into a number of examples where popups pose as system utilities or such that have still succeeded.

As for me personally, I had nearly forgotten that there even were ads in Gmail (which I use every day) until some time last week I thought about it again. I’m constantly exposed to two or three ads while using Gmail, and yet they might as well not be there. I’ve learned to selectively ignore those parts of Gmail entirely.

Gmail was hardly the first place I ran into this kind of advertising; in fact, I’d say in many ways the popup filled, flash banner ads that were ubiquitous to the pre-Google online advertising were much worse.

I like to spend time thinking about what skills the generations behind me will have that mine don’t. One major transition we’ve seen is a shift from the brightly colored, “THIS IS AN AD” styles to the ones these days which pose as personal messages, simple black on white text, or even things that are re-posted by others on Facebook. We are probably conditioning the next generation to be adept at telling the difference between honest thoughts and contrived message constantly, as advertising constantly tries to find ways to blur the distinction.

On the radio this morning, they made a big deal talking about protecting children from cell phones, social media, and the internet in general. I find this infuriating. So much of a child’s life revolves around them learning how to interact with others under safe circumstances. An unsupervised child on a bike on the street is more capable of hurting themselves than anywhere on the internet. We should be far more capable of managing to supervise a stationary child on a computer.

The internet itself is our most efficient tool for learning and communicating to date. Especially those of us working in offices probably rely on our ability to learn new information, related to whatever topic is at hand, on a daily basis. This is more valuable skill than bike riding that we should be encouraging, fostering, and teaching to children. We don’t ask everyone in the park to leave if we want to go there with our kids; we simply teach our kids how to identify dangerous situations and avoid them. Why would our strategy for online communication be any different?

Don’t get me wrong, I’m not against going outside. It’s important to push children to do a wide range of activities, including lots of active ones given our current health problems. Still, the internet is the most effective tool for accessing information worldwide. All we need to do is keep them away from the handful of people who are looking to take advantage of them, just like we do in public, and their own curiosity and interests will take over.

In a twisted way, its our own media that’s improving the culture. With media teetering on and off the edge of credibility, with advertising working to impersonate people, we are learning to adapt. While some generations may resent having to constantly adapt to the newest strategies and others may simply fall victim to it, our youngest generations have grown up with the most clever forms that we have invented yet and are still coping. A generation that’s grown up always knowing that images are photoshopped. A generation that’s always had access to resources like Wikipedia. A generation that’s not only always had the ability to check whether what their being told by their parents is factual, but have been taught through Facebook and social media the necessity of questioning whether something is factual or not.

I only hope we can find the most effective ways to communicate what little knowledge and skills we’ve gained to the next generation, to add to the collective pool of information as much as we can so it can be there when they want it.

Retina is a Lie

  • iPhone 4, 960x640 at 3.5” ‒ 329.65 PPI
  • Galaxy Nexus, 1280x720 at 4.65” – 315.83 PPI
  • Motorola Droid (original), 854x480 at 3.7” – 264.77 PPI
  • iPad 3, 1536x2048 at 9.7” – 263.92 PPI
  • Nexus 7, 1280x800 at 7” – 215.93 PPI

So, either the Motorola Droid also has a retina display, or the iPad 3 doesn’t.

UPDATE: So, it’s been pointed out to me that the definition of a retina display adapts based on the average viewing distance of the device. So, at ~12 inches, 300 PPI qualifies, however, at ~24 inches (twice the distance), it only take 150 PPI. Even at 18 inches the Nexus 7 qualifies, so I’d say it has a retina display as well.

Yes, another reskin

Just tried to add a little more detail, as well as build out some mobile styles again. I think my tests will be working to integrate some HeyDings, Heydon Pickering’s icon OFL icon font.

Also, I wanted to point out that apparently someone else in 2012 thought ‘Because Robots’ was a good name for their site, and wanted to mention them. It doesn’t appear that the wayback indexed the site back when it was running a TiddlyWiki which is unfortunate, because I used to have a whole page of ‘because robots’ sayings that were mostly culture references. Some of the ones I remember are

  • because robots never sleep, they wait
  • because robots grew up on the streets of sydney
  • because robots need more cow bell

A PHP Web Server on Windows (without Apache)

With PHP 5.4, there’s a built in web server. Turns out, it’s pretty easy to use this as a web server for windows and skip installing nginx or apache entirely. I do recommend using NSSM, the ‘Non-Sucking Service Manager’, as it makes creating the service much easier.

  1. Install PHP 5.4 from the windows installer (or whatever you want to use.)
  2. Fire up NSSM.
  3. Create your docroot: mine’s C:\var\www.
  4. Select php.exe as the executable and set the arguments to something like “-S 0.0.0.0:80 -t C:\var\www”.
  5. Create the service; use windows services (services.msc) to start/stop the server, and you can setup things like starting on boot and such here too.

MVC and the Web

Once you pass a certain scale for a project, the architecture becomes critical. This happens way before its important for performance. It compartmentalizes behavior of your application into smaller pieces that can be worked on separately and efficiently. This is my first attempt at trying to describe where I think most implementations of MVC for the web fall short; I hope to provide a working alternative in the future. Until then I realize I’m just criticizing, but I still wanted to share my thoughts.

MVC

Model-View-Controller is a popular architectural pattern that’s become fairly ubiquitous. For a standard, desktop application, it splits your code into three distinct sections.

  • Models are data structures that represent the data being worked with.
  • Views are definitions for the interface of your application and reflect the state of your models.
  • Controllers translate input in the views into actions on the models.

People don’t seem to agree on how these communicate. Some people like views to display data directly from models, whereas others like the controller to sit in between all communication between models and views, whereas other implementations link views to directly pull content from the models themselves.

Multi-tier

While there’s some implementations that could be seen next steps to MVC (PAC, for example, or HMVC), there’s also multi-tier. A typical multi-tier architecture might have

  • A data tier, representing the data itself, but unlike models, with no business logic,
  • A logic tier, where business logic is performed, and
  • A presentation tier, which handles both display and processing user input.

And the web?

Both of these architectures, however, weren’t designed with the web itself in mind, and translate somewhat awkwardly. In a typical, request based website,

  • User input is contained in the request but does not go through a view.
  • Initial requests contain all the input from the user.
  • All input processing must be completed before any interface is presented to the user.
  • All data must be retrieved from a persistence layer each time, so there’s no transient states for models.
  • Views typically do not update to reflect the current state of models.

Typically, these differences lead to the follow simplification of MVC for the web:

  • Requests to directly to a special controller, a router, which chooses which other controller should handle the request.
  • The controller loads models from persistence and trigger business logic processing on them.
  • The controller loads a view, passing data from the models or the models themselves on to the view.

So what’s wrong?

MVC works best views and controllers are as small as they can be. A single interface should be composed of multiple views and controllers. With a unique view and controller for each request, every request needs to implement all the decisions necessary to determine what models load and what business logic to trigger for every part of the current interface. In MVC, a controller should map specific user input to triggering specific business logic on models. Clicking on a button on might trigger the delete user controller, which would load up the related user model, and trigger delete on the model. The delete user controller should not be doing things like loading up recent posts for a section in the sidebar, or even be responsible for determining of the current user has permissions to delete someone else.

In practice, most websites implementations of MVC implement models correctly, but that’s about it. Most routers route to a single method as the controller. Most views are full page templates. Instead of MVC, I’d rather call it MTP:

  • Models that represent the data,
  • Templates for the page to be rendered, and
  • Procedures that call business logic on models and pass the results to templates.

But wait, we have band-aids.

All things considered, MTP architecture has been implemented enough times that I’d say we’ve gotten good at it. For example, templates can include other templates (often called partials). Hooks can be defined so that multiple procedures can be called by the router instead of just one. Data can be passed between procedures and onto template through some kind of data object that can be manipulated by each procedure along the way. Some templates (sometimes called modules) can even contain their own simple procedures.

So what’s wrong again?

Templates and procedures, even with a router, are much worse than views and controllers. Briefly, before I go through an example,

  • Every procedure needs to be aware and check for all state data. Even with hooks and composition, either the procedure or the router does all the work of checking for permissions, or trying  catching any errors separately.
  • Every procedure needs to be aware of all the data necessary for every template that will be used.
  • Sub-templates (partials) can only use subsets of the data the parent template has.

Let’s compare

Let’s talk through the differences between a MTP and MVC approach to handling an administrator deleting a user.

Routing

The differences start with routing. In our implementation of MTP,

  • The router routes a procedure.
  • A ‘data object’ is instantiated that is passed by reference to the procedure and all of the hooks. This is used to pass data between hooks, the procedure, and the template.
  • Hooks may be called by the router before the procedure.
  • The procedure is called.
  • Different hooks may be called.
  • In a final step or hook, the data object is matched with a template to render HTML.
Whereas, in MVC,
  • The router routes to a ‘top level’ view.
  • The view loads any child views.
  • Each child view probably has a template, and returns rendered HTML (or equivalent) to its parent.
  • Lastly, the top level view is rendered to HTML.

Accessing Models

Again, for our implementation of MTP,

  • Hooks and procedures both loads models.
  • Data from models (or even entire models) are put into the data object.
  • Order of execution is critical, as both hooks and procedures may depend on others to be run first to load certain models into the data object.
  • In the final step, the template references any data from models (or models themselves) to render HTML.

And for MVC,

  • Views load the models they require.
  • Child views may be passed models from their parent, or ideally, child views would have read only access to their parents, however, in PHP this may just have to be a best practice.
  • The only order of execution that matters is enforced by having child views and parent views.
  • Each view is only responsible for returning back rendered HTML (or equivalent) its its parent, so in the final step, the top level is just composing views.

But what about controllers?

Backing up to how controllers are defined:

  • Controllers translate input in the views into actions on the models.

For the web, this is the most difficult part to translate. For most applications, input happens asynchronously, as events in particular views, and controllers are in a sense glorified event handlers. For the web, all input is given with the initial request. Secondly, on the web, input is only scoped to a top level view.

It would be easy to take top level views and add input handling into them… and that’s where MTP comes from. But, when we mix the controllers into these views, we lose the separation of input handling from visual presentation. Or worse, we mix input handling into the models themselves.

Views should call controllers and pass input and related models to them. Let’s talk through an example that shows the differences between all three:

Example: a company directory

Models

  • Person – represents a person in the company directory. Has accessors (getName, getPhoneNumber) and mutators (setName, delete).
  • Department – represents a department of the company.

Controllers

  • PersonController – handles editing a single user.
  • DepartmentController – handles editing a department.
  • LoginController – handles logging in a user.

Views

  • desktop-view – top level view for desktops. depending on URL, may load
  • desktop-login-form – child view that shows up on the desktop site and is used to login. Passes input to LoginController.
  • view-user ‒ child view that loads a Person and display their information.
  • view-department – child view that loads a Department, every Person in the department, and displays the relevant information.
  • update-profile – allows a regular user to update their own details. Passes input to PersonController, and uses view-user to display the new version of the profile.
  • update-my-department – allows a uer to update their Department’s details. Passes input to DepartmentController, and uses view-department to display the new version of the department.
  • mobile-view – top level view for mobile devices. depending on URL, may load
  • mobile-login-form – child view that is used by mobile devices to login. Passes input to LoginController.
  • mobile-view-user – different, mobile optimized alternative to view-user.
  • mobile-department – similiarly, a different mobile optimized alternative to view-department.
  • mobile-update-profile – allows a use to update their own details from their phone. Passes input to PersonController.

Desktop vs. mobile is certainly an easy way to see two different views that should share the same controller, but certainly doesn’t need to be this strong of a separation. Any time you have multiple views (pages) that should share input handling benefit from this. Things like having a login form on multiple pages. If there’s an admin and a regular user view that allow editing a profile, they should share a controller. Right now, we often link users off to a common page for editing a user to minimize this problem. Moving towards having separate controllers makes things like implementing editing in place or “double click to edit” without redirecting much easier to implement.

Conclusions

As I said initially, I hope to implement what I’m proposing so I can share that next, but I wanted to share my thought process so far. As I’m still in the design phase for this, I’m very interested in feedback.

Just Relax: Why it’s difficult to plan for scaling

Diablo 3 was released yesterday, and the authentication servers in particular are having trouble keeping up with the load. Parts of the community are being supportive, but other parts are being obnoxious enough that I wanted to rant. So, here’s the rant for any of you interested enough to read it.

Computers are awesome at doing one thing at a time. That’s all they did at first, we just figured out how to make them very efficient at switching between a lot of different tasks.

The ‘speed’ of a processor (say, 3 Gigahertz) refers to the clock rate of the processor. This doesn’t exactly translate to how many instructions a processor can perform per second, but its closely related. (This is affected by other things, not the least of which is the architecture of the system itself.)

These days, most home systems have multi-core processors. Here, each core can do its own instruction at the same time, So that unrelated ‘tasks’ can be done in parallel. High end computers often have multiple processors, each which can have multiple cores. Something like the authentication system for Blizzard is almost certainly composed of multiple servers.

Problems don’t arise from CPUs not being fast enough or there not being enough servers.

If you wanted to design a car to go 1000 miles per hour, you’d might start by building a car that can go 10 mph and test it up to 100 mph. You’d be able to see things like whether your engine keeping up or whether the wheels stayed attached, but you probably would never discover that at 300 mph, there’s enough upward force generated by the air moving under the car that it’ll lose traction and flip over. The point is, some problems, even with computers, you can’t predict by simulating anything less than full load.

Now pretend you’re Blizzard. You have a new game with over 2 million pre-orders alone. Do you try to simulate a million people connected? How would you even know that you’ve properly simulated a million computers, each with different hardware and operating systems in a way that’s realistic? Or do you run betas and try to find all the serious issues with a hundredth of the real player base, and try to plan to scale from that?

Lastly, remember that this login system isn’t just returning a list of online/offline. Its loading up characters, progress, and even dealing with trying to verify your account is valid and paid for and you’re not some kind of hacker trying to break into someone else’s account.

So, let them have some slack. It’s a good game. If you have to, pretend that the real launch isn’t for a week or two and play it then; there’s no subscription you’re wasting, you’ll be able to play this game now as it is or you can wait for the fixes.  Either way, hopefully if you bothered to read all this it offered a little food for thought about why this is hard.