(... an old rambly email, pulled from the dusty attic for your perusal ...)

So, it is Monday afternoon, December 26.   I love this week, between Christmas and New Year.   There is no pressure to get things done, yet I often get a lot done.   Mostly it is done at a high level, in my brain, rather than at a low level, with my fingers.   This email is part of my work for this week.

So, I went for a bike ride this morning with a good friend who is not a programmer.   He’s actually a financial guy, buys and sells bonds or something, he’s a neighbor and the father of my daughter’s best friend and a nice guy and a fellow bike rider.   And a smart guy.   So I was explaining why I like this week, and that my goal for this week was to plan out next year, and how measuring what we do (at Aperio) was one thing I wanted to improve, significantly, and that was my goal.  

How did this come up?   Maybe you don’t care but hey I’m going to write about it anyway.   See both of us have been eating like crazy the last few days and both of us are conscious of the fact that an hour’s worth of bike riding is equivalent to one piece of See’s candy.   So we’re both getting fat(ter) and we were talking about how we’re going to lose the weight in January.   And we differ in our approaches to this in one important aspect, I weigh myself every day, and he never weighs himself, ever.   So I was telling him that measuring is the key to improving.

Why does it work?   I think it works in environments where you don’t know all the factors, and where the main factor is the behavior of people.   Measuring somehow changes the behavior of people, in subtle ways, in ways that they cannot themselves control.   And that is how the magic is done.

Imagine measuring the output of a completely automated factory.   No people at all.   Would the simple act of measuring change the output of the factory?   No, it would not.   You would know the output, and that might be helpful all by itself, and over time you would know the variability of the output and things like that, but the output itself would not be affected.

Now imagine measuring the output of a factory which contains humans.   But you don’t tell the humans that you’re measuring the output.   Would this simple act change the output of the factory?   No, it would not.   Exactly like the case of the fully automated factory, you would know the output but the output itself would not be affected.

Now imagine measuring the output of a factory with humans, but this time you tell the humans that you’re measuring the output.   You don’t tell them the output, you don’t feed back what you measure in any way.   At time zero you tell them that you’re measuring, and that’s it.   Would this simple act change the output of the factory?   I claim that it would.   Somehow in ways you might not be able to explain, knowing that the output is being measured would change the humans’ productivity.   So not only would you know the output, you would have affected the output.

Given that the output is affected, would it be increased?   Aha!   You might say “sure, people want to do a good job, so knowing their output is being measured, they will do a better job, and the output will be increased”.   That makes sense if people know what to do to increase output.   If the correlation between their activities and the output of the factory is well understood, then knowing their output was being measured would cause people to work “better”, and the factory output would be increased.

But what if the correlation between people’s activities and the output of the factory was not straightforward?   What if it was complicated and not well understood?   It is conceivable that people would change their behavior in the factory thinking their changes would increase output, not realizing that in actuality their changes decreased output.   So knowing that their output was being measured would cause the output to change, but it would not necessarily cause the output to increase.

In many complicated environments it is tough to know which changes have which effects.   In all likelihood some changes cause increases in output, some don’t, and the overall mix of changes causes a mix of offsetting changes in output.   Overall the change might be positive, but not necessarily optimal.

Finally imagine measuring the output of a factory with humans, telling them that you’re measuring, and periodically telling them the output, too.   Would this simple act change the output of the factory?   Yes, it would.   And I claim that this is the best thing to do because the people in the factory can learn from the effects of their changes, and optimize their changes to increase output.   They might make some changes which decrease output, but after learning, they would change back.   They might try some things without knowing whether they would increase or decrease output, and learn which do and which don’t.   Over time the factory would asymptotically approach optimality.

I know I’m jumping around here, sorry, this is a bit of stream-of-consciousness here.   Please bear with me, I think this is all going to make sense in the end.

Think about trying to lose weight.   This is a fairly simple system – at least the inputs - with one measured output.  However there are many things you can do, in complex combinations, to affect the system.   Sure, eating See’s candy is known to be suboptimal.   But how about eating stew?   You have to eat something, so maybe that is good?   How about drinking Diet Coke?   Is that better or worse than drinking water?   How about riding?   Sure, exercise is known to be helpful, but you can ride in the morning, ride in the afternoon, ride every other day.   You can eat before you ride, or not, you can eat after your ride, or not.   Complicated.   Without getting periodic feedback on the system – without weighing yourself – how can you optimize?

Okay, now let’s talk about Aperio.  (Here’s the part where, hopefully, it will start to make sense.)  I really want to ship releases on time.   I want to be able to say, in December, that we’re shipping release 8 in May, and I want to publish the high-level features of release 8, and I want to know that we’re going to ship that release on time, in May, with those features.   How can we do that?

First, we need enough resources.   We need sufficient people, computers, tools, etc. to actually be able to create the contents of release 8 by May.   (Actually the contents need to be created by March, so they can be tested in April, so they can be released in May.)   In the past maybe we didn’t have enough resources do to what we wanted in the timeframes that we wanted.   Now I think we do.

Next, we need to manage all the distractions that can interfere with getting release 8 built.   Some of these distractions are important parts of what we do, they create value.   We have to support sales as they sell systems to prospects.   We have to support operations as they make systems, install them, and support existing customers.   And we have to have fun!   We have to take vacations, and fix bugs in old releases, and hook up webcams so we can see our dogs from work.   We have to configure new computers, and install various software tools, and play games.   There is a lot to do.

Each day there are a million little decisions to make which contribute to whether or not we achieve our goals.  How can we optimize that decision making?

I can try to manage the distractions.   I can try to make all those little decisions.   But that is inefficient.   In fact, it just doesn’t work.   (For one thing, it annoys the people in the factory :)   I can make high-level rules, or share my experience.   Some of these things are obvious – kind of like knowing that eating See’s candy will increase your weight.   Some of these things are less obvious – kind of like discovering that eating after a ride is better than eating before a ride.   Communicating high-level generalizations is good, but still suboptimal.

But what about this: I tell you the goals – you, the thinking humans in the factory - and let you make all the little decisions in ways which enable us to achieve our goals.   Just like with a complicated factory, some of the cause and effect is pretty unobvious.   We try stuff, and we measure, and we learn what works and what doesn’t.   And over time we improve.   That is the magic.

I hope you’re with me so far, that you’ve been able to filter all the babble and get the essence, because I think this is pretty important.   I don’t want you to feel that we’re measuring because we read it somewhere in a book, or because managers all take a class in measurement.   Or because I just have this urge to write long emails about stuff, so I invented this theory about measuring.   There is a method to this madness, I’ve honestly thought about this a lot.   I want to improve, and I think measuring is the key to improving.

If I’m right, and this really works, and there is a magic to measurement, then maybe this will help.   So what do we measure?  How do we work the magic?   We need to decide what we want to optimize, and this will tell us what to measure.  

If we want to improve productivity, we should measure productivity.   I’m not saying that’s easy – because productivity is complicated, and measuring it is complicated – but it can be measured.  Essentially productivity is the amount of work that gets done in a certain amount of time.  Get more work done in less time, and you’re more productive.   Measuring time is easy, but measuring work is not.  I don’t want to run back through how we plan to measure productivity, only to note that we do, and that the hope is that in so doing, we can all iterate into a more efficient workflow.

There are all kinds of work products that are valuable but tough to measure.   Helping salespeople to close a sale.   Helping operations to satisfy customers.   Helping each other to have fun.   (Foosball, anyone?)   Even code is hard to measure, because what we really want is functionality for customers, and how do you quantify that?   So we approximate, but measuring something is worthwhile, because measuring is the key to improving.

Back to what I wrote earlier.   “I really want to ship releases on time.   I want to be able to say, in December, that we’re shipping release 8 in May, and I want to publish the high-level features of release 8, and I want to know that we’re going to ship that release on time, in May, with those features.”   What I really want is to become more predictable.   Unfortunately as everyone knows building software is hard to predict, so how do we make it more predictable?

Following the logic of this thread, if we want to improve predictability, we should measure predictability.   This is way better than me telling everyone what to do, and also better than me making general suggestions to guide what to do.   Sure, we know breaking projects down into pieces helps.   Sure, we know writing specs helps.   Sure, we know eliminating distractions, and setting priorities, and establishing goals all help.   But there might be a lot of other things which help, too.   I’d rather have 20 thinking humans on this problem than one.   By measuring our predictability, and sharing the measurements with you, I think we can all iterate into better predictability.

And that is the magic.