How to innovate from bad situations — 6.1040 Software Design

Successful innovation from the start

With innovation, where you start from determines where you end up. Unfortunately many starting points can lead you to failure, spending time and money and goodwill to produce something that nobody wants. Of course it's possible that you might recover from a bad start. If you build the wrong solution, you might be able to figure out from feedback what's wrong with it, and iterate until it works. But that's likely to be expensive and slow, and you might not be able to get there from here, because you're too constrained by the path you're on.

Some common mistaken strategies

The first mistake is asking only what and not why. You embrace a solution before you have any reason to believe that it's even relevant. The lure of new technologies can make them seem appealing in their own right. You convince yourself that you need agentic AI, blockchain, quantum etc, but these bring only costs and not benefits if you don't understand why you might want them.

A second mistake is to frame your goal solely in terms of values. You want to be efficient, or customer-friendly, or scalable, or speedy. These might all be good things but they're not sharp enough to guide you. You might refine an overarching value into a collection of more granular values. If you can make these measurable, they may be useful metrics in assessing progress, but they're not enough to set you on a path to innovation.

A third mistake is to place too much confidence in expressed desires. Design thinking suggests that you start by asking what people want or need. This seems plausible, but it doesn't work so well in practice:

When people say they want or need something, the thing they want or need is just a figment of their imagination. As Rolf Faste noted, this is a paradox at the very heart of "needfinding": you're looking for something that doesn't exist. Describing something imaginary is hard.
Users are notoriously unreliable in predicting what they want or need. When given exactly the thing they said they wanted, they often choose not to adopt it, because it turns out that they never really wanted or needed it after all. Or perhaps they did, but there's some other reason that addressing that particular want or need is not enough.
Wants and needs are easily conflated with the features of an imagined solution. A user who wants something is usually just designing it in their head. But users are rarely good designers.

A good strategy: starting with bad things

A better start identifies bad situations to mitigate. You point to a problem that people have today that prevents them from doing something they want to do, or a bad thing that happens that they would prefer does not happen, or just something in a situation makes them unhappy or dissatisfied in some way. This is more effective than identifying needs or wants because:

Bad situations that users complain about are real and concrete. They can be confirmed by direct observation in the situations people are already in, without requiring imagining things that don't exist.
Users are far more reliable in accurately describing their problems than in speculating about what solutions they would accept.
Although designers sometimes like to believe that people will change to adopt innovations, in practice people are very reluctant to change. The best way to succeed when innovating, therefore, is to take a behavior that people already engage in and mitigate the problems that bother them.
As a design goal, preventing a bad thing is something you can evaluate the finished solution on, because you can assess whether the bad thing still happens, or has been eliminated or reduced.

Example: Apple's iPod. Steve Jobs understood that people weren't just looking for a shiny music player. At the time, MP3 players were already available, and making a shinier one would not have been enough. People didn't really know what they wanted or needed. Apple's values of simplicity, attention to detail, user focus, etc could certainly inform the design process but they don't provide direction. Instead, Jobs's genius was recognizing the bad things that a new device and ecosystem could overcome. It was bad that you couldn't take all your music with you; you could only upload a few files to an MP3 player. It was bad that obtaining music files meant ripping CDs or downloading pirated copies online. It was bad that if you wanted a song you had to buy the whole album. The iPod fixed all of these and more, with different design elements addressing different bad things. The tiny 5GB drive and the Firewire port let you upload all your music; the iTunes store let you avoid the tribulations of ripping or pirating; the idea of selling individual songs for $0.99 each saved you from having to buy the whole album.

Users may only be able to recognize a bad situation when it has been mitigated. Perhaps you were never bothered by not being able to take all your music with you everywhere. But once all your friends have an iPod, the bad situation becomes intolerable. This is what makes successful innovations succeed. People don't just want or need the innovation; they find it intolerable to be without it. This is what Merrick Furst calls a "not not": it becomes not acceptable to not have the innovation.

But unlike the elusive needs of needfinding which didn't exist prior to the innovation, a bad situation exists and can be pointed to even if people in that situation don't see it clearly. Once Steve Jobs pointed out to you that you couldn't take all your music with you, the badness was irrefutable.

Tips for identifying bad situations

Ways to identify bad situations

Bad situations can be identified in at least three ways:

Delve into the experiences of users, and search for evidence of things that make them frustrated. The kind of empathetic interviewing that design thinking advocates can be productive, but don't encourage them to tell you about what they want or need. Instead, have them explain bad things: what goes wrong, where they get stuck, what feels burdensome, and so on.
Observe people at work, carefully scrutinizing what they do, and try to recognize frictions and limitations that they themselves might not notice.
Look for workarounds. When people encounter obstacles they often develop ways to work around them. These workarounds are themselves innovations driven by identifying bad situations. Try not to be distracted by the workarounds themselves, but instead to understand why they are there and what problem they address. If you focus too much on the workaround and its inadequacy, you run the risk of inventing a solution that is just a tweak on the workaround.
Consider a situation and try to come up with ways in which a malicious actor might break things. Chances are those things are already happening (and even if they're not, preventing them will likely be useful, especially in critical situations).

Beware features

It's tempting to sidestep the harder work of clarifying a bad situation and instead to just name a missing feature. Just listing needs and wants, without trying to understand their underlying causes, is much less reliable than pointing to bad situations, so it should be resisted. When you name a feature, you're beginning to design the solution before you've understood the problem. Even if a feature isn't explicitly named, saying that the problem is an inability to do something is often the same thing, so when you see a bad situation described in this way, make sure a feature isn't being named and that the real problem is articulated.

For example, a team lead might complain that the email client her team uses to discuss project work doesn't support the kind of reactions with emojis that tools like Slack support. This isn't a bad situation; it's a feature request. Instead, we need to understand what the real problem is and what goes wrong. Perhaps it's that the team lead frequently wants to poll the team to get votes on how a decision should be made, and is frustrated that when she asks for votes, she gets long-winded emails in response and can't get team members to commit. That's a bad situation, and while it might be mitigated by a reaction feature, that's not the only (and probably not the best) solution. A better solution might be a mechanism that supports voting explicitly.

A purported bad situation may sometimes be nothing more than an expressed desire for a particular feature. Be especially careful of pain points expressed as an inability to do something, and when that seems compelling look for real consequences rather than relying on an intuition that the missing action is needed. For example,

Criteria for evaluation

How do you know if your identification of a bad situation is promising?

Look out for features. Make sure the purported bad situation isn't just a proposed feature in disguise. Be especially careful of pain points expressed as an inability to do something, and look for real consequences rather than relying on an intuition that some missing behavior is needed.
Seek confirmation. For example, if your impression is that people spend too much time trying to find movies to watch online, look for some data to corroborate it (hint: you'd be right). Corroborating evidence can be quantitative, but anecdotal reports can be compelling too. And remember to beware of confirmation bias. Your goal should be to obtain evidence that you can objectively assess, not to filter the evidence to support a foregone conclusion.
Make your situation concrete and clear. Try and turn generic and vague situations into ones that are familiar and well understood. For example, the bad situation "people receive too much email" is not as helpful as the situation "when employees arrive at work, they are unable to find the important emails they need to organize the rest of the day because so many new messages arrived overnight". This second situation is much more informative, because it suggests that volume alone may not be the problem.
Check that your situation can act as a yardstick of success. For example, suppose your bad situation is that people in your company are unprepared for meetings so the first half of every meeting is spent catching up and providing background. If you designed a solution involving an agent that sends out briefs in advance of meetings, you will know if it succeeds in mitigating the bad situation if people are getting to the substance of the meeting more quickly. Contrast that with a poorly formulated bad situation that just says "employees are uninformed". You might design agents to help them be more informed, and you might measure whether the agents are used and whether employees say they like them, but you'd never learn whether your solution was producing any real value.
The badness should have a clear cost. The cost need not be measurable in direct financial terms; it may be a cost in attention, or well-being, or relationships. For example, maybe your organization has a cafeteria and people have to wait in line when they go for lunch. Before you identify this as a bad situation that requires agents to predict demand for menu items and to deliver lunch to offices, consider what the real cost is. It's one thing if the line is so long that people skip lunch, have to eat too quickly, or are late for meetings after. But it's another if the line is never longer than a few people and it actually encourages social interaction.
One-off situations are less credible than recurring ones, unless they're potentially catastrophic. For example, suppose your bad situation is that when you run conferences you give people name tags and two people turn up at a conference with the same first and last names. This is unlikely to matter. First, it's a rare event; second, the cost is low or even non-existent (assuming that the name tags are to help attendees make connections). A rare situation of very high cost is another matter. If the name tags are being used in a radiotherapy center to match doses to patients, confusing two patients who happen to have the same name is very bad indeed.
You must have some chance of altering the situation. If the situation is one that is outside your control, and you have little ability to affect it, there may be little point in designing an intervention. An advantage of making the bad situation explicit is that you can ask whose situation it is, and who controls it.

Examples of innovations from bad situations

Here are some examples of innovations that come from mitigating bad situations:

Protective mechanisms. The simplest examples are innovations that are intended to prevent bad actors from causing trouble. The locks on your door prevent thieves from entering your house. Two-factor authentication prevents someone from accessing your bank account if they don't also have your phone.
Punishment mechanisms. Punishing misbehavior deters bad actors, and makes it less prevalent. Auditing with logs doesn't stop employees from doing things they're not supposed to do, but it raises the chance of being caught.
Eliminating friction. Small amounts of friction can make tasks burdensome. Before Zoom, most video conferencing apps required each participant in a meeting to be added explicitly by the host. The linking concept, in which the host could just create a link and share it with participants who could then join by themselves eliminated that friction.
Reducing mental burden. Users can be overwhelmed by choice, or just weighed down by unwanted burdens of searching or filtering. Email triage agents, recommendation engines, and faceted search all address this, by saving the user the effort of evaluating a large number of items.
Preventing errors. In many situations, errors can undermine work and credibility. Checking legal citations in a brief, reviewing code, analyzing the tone of communications--many such activities are becoming more amenable to automation due to AI (but also often more necessary).
Overcoming lack of expertise. A person engaged in a task may find themselves needing expertise they lack. Advising a startup founder on how to register and set up a company; giving instructions to a wilderness hiker on how to reseat a dislocated shoulder; devising investment strategies.
Closing an information gap. A task requires some information that is not available. Getting weather forecasts when packing for a trip; using historical flight performance data to predict likely delays and plan multi-step routes; generating warnings for potential drug interactions.

How bad situations lead to design elements

Sometimes mitigating a single bad thing can make a product successful. For example, when COVID struck, there was a need for video meetings with many participants (eg for university classes). Existing apps like Skype forced the host of the meeting to add the participants to the meeting themselves (and worse, they had to be existing account holders, and the host needed to know their account names). This was a very bad situation that made it impossible to host large meetings. Zoom solved this with the Linking concept, in which the host creates a link, shares it with participants, who can then join the meeting themselves (and moreover, did not have to be account holders). There are surprisingly many product successes that relied on mitigating a single bad thing. Another example is the self-scheduling of tools such as Calendly, which eliminated the difficulty of negotiating meeting times.

But more often the motivation for an innovation will be not just one bad thing but multiple ones. For example, with the iPod we saw that the bad situation wasn't just not having your music with you, but also not being able to acquire the music you wanted. It wouldn't matter not having your music with you if you weren't able to obtain music in the first place.

Each bad thing is likely to become a design element in the solution. In the case of the iPod, we saw that not having your music with you was mitigated by a new disk drive and Firewire port; the difficulty of obtaining music was mitigated by the iTunes store; the annoyance of having to buy a whole album by selling individual songs.

Concepts as design elements

Concept design is an approach to designing systems that are defined by their behavior. These systems include software applications and services, AI agents, and human protocols as embodied in policies and processes. In many cases, the very same concept exists in all these domains. For example, restaurant reservations started out as a human protocol in which diners would call restaurants on the phone; then software services like OpenTable appeared; and now agents can make reservations for you. In all cases, however, the basic activity of reserving a table is the same.

In these behavioral systems, the design elements are distinct activities. Each activity has a well defined purpose that mitigates a particular bad thing, and can be understood in its own right. For example, in a restaurant reservation system, there is not only the activity of reserving, but also activities like notifying (confirming reservations to user), reminding (when a reservation is coming up), paying (if reservations require a deposit), rating (of restaurants), assigning of diners to tables, and so on.

These activities are called concepts, because they have a conceptual integrity by virtue of their independent value, and often their familiarity and possibility for reuse. For example, the same reserving concept might be used for restaurant tables and hair salon appointments; the same notifying concept might be used to inform you when someone replies to your social media post; the same rating concept might be used for products in an online store.

Acknowledgment

The idea of bad situations emerged in discussions with Merrick Furst, whose book The Heart of Innovation argues that to uncover "authentic demand," innovators must map and study the situations people find themselves in to find hidden behavioral conflicts.