Rubric for problem framing — 6.1040 Software Design

A problem framing should include:

Domain. A compelling description of the general problem domain, understandable to someone who has no experience in this domain. For example, if the problem is about customer support tickets, this would explain how tickets are used, and how support staff are organized and conduct their work.
Bad situations. One or more scenarios that point to problems to be prevented or mitigated. A scenario should be as concrete as possible, and should include not only a moment of difficulty or pain but the steps that led up to it and, if relevant, the consequences that follow.
Corroboration. Evidence in support of the prevalence and significance of the bad situations, ideally with citations of reliable sources.
Workarounds and comparables. A description of existing workarounds that people use to mitigate the bad situations, and a list of existing solutions that are relevant, with reasons for why they are insufficient.
Solution sketch. A sketch of a potential solution to the bad situations. The sketch is not a design; its role is to provide some suggestive evidence that the problem is not completely intractable and might be solved without extraordinary effort.

Some criteria for each part:

Domain. The domain description should be comprehensive in providing enough detail that a reader can get a full picture of the background to the bad situations, but it should be focused and cover only enough of the domain that is needed to grasp the problem. In addition to explaining what people do in the domain, it should include the essential motivations that explain why they do those things.
Bad situations. The scenarios should be easy to understand and it should be clear why they are bad. There should be enough detail to be a yardstick against potential solutions. The description should not dwell on user interfaces or other low-level details; the point is to address what happens to someone in a situation, what they can or cannot do, and why they experience discomfort, inconvenience, or worse. The significance of a bad situation should be clear in terms of the harm or cost incurred, and the frequency with which it arises.
Corroboration. The evidence should be directly tied to the bad situations, and should be compelling, and presented fairly and without bias, with judgments on the credibility of sources when appropriate.
Workarounds and comparables. It should be clear why people use the particular workarounds and why they are unattractive. If there are widely used solutions, the list of comparables should include representatives of major classes.
Solution sketch. The idea embodied in the sketch should be easily graspable by a layperson, and its relationship to the bad situations and likelihood of mitigating them should be clear.

Common pitfalls:

Cognitive bias. The designers trying to identify bad situations and users who are asked about them are humans, and therefore suffer from cognitive biases that mislead them. You need to be extremely vigilant and place weight on observations of real problems, and be wary of perceptions of those problems. Ideally, identifying bad situations is done by observing those situations carefully and listening to what users say about them. When a user is very unhappy about something and wants to tell you what the problem is and how it should be fixed, believe the unhappiness and try to understand deeply what is causing it. Be more skeptical of the user's explanation of what the problem is, and very skeptical of hypothesized solutions.
The domain is too broad. For example, if the bad situation is about the difficulty of finding online movies to watch, the domain should not be entertainment or movies generally, but online movie viewing.
The bad situation is vague. For example, everyone knows it's hard to schedule meetings, but that's too vague a problem. A bad situation would explain what goes wrong when the leader of an organization tries to find a meeting time with some members, and then finds that they cannot agree on a time.
Falling for features. Hypothesizing a want or need of a particular feature is not a bad situation. If you feel that a feature is missing, try to understand where that feeling comes from and what bad situation is not being addressed.
Biased evidence. The corroborating sources are cherry-picked and ignore evidence that would point in the other direction.
Low impact. The bad situation is real, but its impact is not significant enough in either harm or frequency to be worth addressing. The workarounds are not annoying enough to merit a new solution.
Irrelevant comparables. The list of comparables includes products that are not relevant to the bad situation. If an existing product might be assumed to have a solution to the bad situation but in fact does not, that would be relevant.
Solution sketch restates the problem. The solution sketch doesn't actually point to a solution, but just restates the problem, for example by saying that an app will be built to address the problem.
Solution sketch is unfocused. The sketch lists a variety of features whose relevance to the bad situation is unclear.
Solution is too ambitious. The proposed solution requires overcoming a hurdle that most experts would view as insurmountable, for example because it would require extravagant use of resources or a major technological advance.
Solution requires change of behavior. The proposed solution would require users to behave in a very different way, or to perform some steps that they are unlikely to be willing to perform.

Competency levels:

Deficient. Parts of the problem framing are missing; or the explanations are not intelligible; or the bad situation is just a feature list.
Emergent. The domain is cogently explained, and the bad situation is believable, but is not expressed with sufficient concrete detail to be a useful yardstick for solutions. Workarounds may be missing or contrived. Analysis of comparables misses important examples, or does not give good reasons for their inadequacy. Solution sketch is vague, overly detailed or implausible.
Competent. All parts are present and clearly understandable. The bad situations are concrete and their costs are clear. The descriptions of workarounds and comparables provide a valuable assessment of the larger context. The solution sketch is realistic and understandable. But the bad situations are not entirely compelling: the costs might not be enough to warrant an innovation, or the reader might be left with the impression that there are likely more significant bad situations to address in the same domain. Or the workaround and comparables seem to be sufficient and the relative benefit of the new solution is unclear.
Expert. The problem analysis provides an "aha" moment that reveals an opportunity for innovation that was non-obvious, by pointing to a bad situation that has not received attention, or that people might not even have realized was bad because they tolerated it or worked around it. But once noticed, the bad situation is clear, as if there was a hole in the world asking for an invention to fill it.