Shipping vs. Learning

“What did you ship last quarter?”

“When is this going to ship?”

“Real artists ship.”

The verb “ship” has a long history in the software development world and before that, the physical world. In the physical world, it originally meant “to transport something on a vessel”, and in the software world, it meant “to press a tape/disk/CD and send it out to consumers”. Since then, it has come to simply mean “release”, and even then, usually not in any sort of final form.

Everyone inside tech companies loves shipping. It’s the culmination of a lot of hard work and creativity from designers, engineers, PMs, researchers, and any number of other people, and when it’s good it puts a dent in the universe. It is no wonder then that so much of the machinery of tech organizations is centered around shipping.

But should it be? Especially given how much shipping itself has changed in the last couple of decades?

Here is a non-exhaustive list of problems that can occur when everything is oriented around shipping:

• Teams ship inferior or non-impactful things in order to say they shipped something
• Team A ships more than Team B and thus appears more competent, even though Team B’s mission is more difficult or process more rigorous
• Teams ship things that teach their company nothing
• Teams manipulate (or very liberally analyze) data in order to justify shipping
• Teams aren’t given enough runway to do great work because of the constant pressure to ship ASAP
• Team members are given (or not given) raises, promotions, or other praise because of how often (or seldom) they ship

Whenever I talk to design leaders from other organizations, I’m struck by how similar everyone’s challenges are. Over coffee with my friend Cap Watkins — Buzzfeed’s Head of Design — we talked about if there was a better way to measure progress than shipping.

Learning as a prime objective was what kept coming up. Organizations that learn the quickest seem the most likely to succeed over the long haul. This is not an original thought, for sure, but most companies don’t seem to have operationalized it formally.

Maybe shipping is actually just a subset of learning, or at the very least, in service of it. Put differently, shipping can be a great way to learn, but it’s not the only way, and it’s not always the best way. If all of that is true, how might we re-orient the development process around learning?

A provocative approach would be to prioritize learning as the most important regular deliverable of engineering/product/design teams. Let’s say you are a manager, executive, or anyone else interested in a particular product within a company. Here is the sort of progress report you are probably used to seeing from whoever is leading the team building the thing:

SEPTEMBER

• Launched movie recommendations
• Hired 2 engineers
• Fixed 30 bugs
• Went into beta with collaborative filtering
• Drafted product plan for internationalization

When written more with more structure, they can take the form of OKRs (Objectives & Key Results) or KPIs (Key Performance Indicators).

While useful in showing that the team isn’t sitting on their hands, it doesn’t really hold anyone accountable for what good all of that is doing for the organization. What caused the team to launch movie recommendations and what will that mean for the business? What will 2 more engineers accomplish? What is the impact of the bugs fixed? What do we hope to learn from the collaborative filtering beta and how will we know we should even launch it? What have we learned about internationalization in terms of how much bigger it will make our business or how much of an effort it will be?

A list like the one above is at best a tactical inventory, but at worst a veil that masks how little actual learning may be occurring. In other words, it may make you look like you’re moving forward (good!) at the expense of ignoring how little of a difference you are making (bad!). What if, instead of delivering something like that every week, month, or quarter, you delivered something more like this:

SEPTEMBER

• What we learned: Removing steps from our onboarding experience did not reduce user confusion. Instead, clearing up language such that users felt they were making progress resulted in the greatest gains (link to more details here).
• Cost to learn: One researcher, one prototyper, one week of calendar time, and $2000 in participant fees.
• Plan to proceed: Productionize new language within one month.
• What’s needed to proceed: Some internationalization help… we’ll outsource this.

 

• What we learned: We have entire tab in our app — Theaters — that only 1% of our users regularly use. Additionally, we have another feature — Friends — buried under a sub-menu, that almost half of our users use regularly. We aren’t sure what the effects would be if replace Theaters with Friends, but it seems like the right thing to do.
• Cost to learn: One PM, from one day of digging through data.
• Plan to proceed: Run an experiment swapping in a Friends tab in place of the Theaters tab and see what effect it has on metrics.
• What’s needed to proceed: We have no dedicated data scientist in our group. Would be great to have one, but short of that, if we could just get a day per week of someone’s time, we’d take that.

 

• What we learned: Our projects take at least 30% longer than they should because of an outdated build process that has too many dependencies. In addition to the cost in extra time spent, we have also lost engineers and designers due to this frustration.
• Cost to learn: We’ve been paying this cost in slow development and high attrition for at least two years.
• Plan to proceed: We just dedicated two weeks of an engineer’s time to investigate the issues and what it would take to fix. Our hope is to stack rank what’s easiest and most valuable to fix and then attack the list aggressively.
• What’s needed to proceed: We need air cover from executive leadership for probably about a month while we get this fixed. Meaning, we need to pause our current product work and not have to answer questions about why. This is important work, leverageable by the entire organization.

What we learned is really the key deliverable, and something potentially interesting to many people across the company. Cost to learn is a chance to optimize for and highlight efficiency, as spending a year of engineering time carries much greater opportunity costs than running some efficient, qualitative testing. Plan to proceed lets people know the outcome of the learning and how it affects the way forward. And finally, what’s needed to proceed spells out what a manager, executive, or anyone outside the core team can do to help.

This is a rough sketch and could use plenty of iteration and refinement, but one thing I like about it is that while there are plenty of legitimate and illegitimate excuses for not shipping, there is really no excuse for not learning. In other words, a team could spend an entire quarter (or much longer) shipping absolutely nothing, even though they are doing all the right things. However, if a team is not learning quickly and regularly, they are almost certainly doing something wrong.

I also like that a re-orientation around learning gets people thinking in an entirely different way.

“What’s the minimum we can ship with?” becomes “How might we learn X most quickly?”

“What do we want to release next quarter?” becomes “What is most important for us to learn about our product next quarter?”

“How can we ship more often?” becomes “How can we increase the pace of our own learning?”

None of this means you won’t also have to ask questions about shipping, but shipping simply becomes another way to learn… alongside research, prototyping, and many other methods. It’s still what you ultimately want to get to, but it’s more a by-product of the methodical learning you hold yourself and all teams within the company accountable for every week, month, or quarter.

Some of today’s companies who learn and evolve the quickest have surely already operationalized some of this sort of thinking, even if informally or by accident. If you’re reading this and you’re thinking to yourself “we already do all of that!”, then that is fantastic (and I would love to hear more about your process in the comments)! If, however, this sort of re-orientation sounds refreshing to you, start talking about it with your team and see what steps you can take to try it out on a project or two. I’m curious to hear what works best for everyone.

(This post also available on Medium.)