Prologue: A tale of two databases
Many years ago, I joined an organization in the throes of change. The company was in the process of replacing a venerable Lotus Notes–based system with a newer customer relationship management (CRM) product, and I was tasked with integrating data from the CRM with other syndicated and publicly available data sets. The requirements were complex, but the system design evolved through continual, often animated discussions between the development team and key business stakeholders in an environment characterized by openness and trust. The system was delivered on schedule, with minimal rework required.
Five years later, I was invited to participate in a regional project at the same company. The goal was to build a data warehouse for subsidiaries across Asia, driven by the corporate IT office located in Europe. Corporate’s interest in sponsoring this effort was to harmonize a data landscape that was—to put it mildly—messy, with each subsidiary taking a bespoke approach to data management to address local reporting needs. The subsidiaries thought their systems were just fine. They perceived the push for standardization as a corporate power play that would result in a loss of local autonomy over data.
There would be no timely and amicable resolution this time. The discussion began to devolve into heated Skype exchanges between stakeholders, forcing regional IT to step in and call a meeting to resolve the conflict. This episode led me to a broader understanding of the social complexities that lie beneath software development projects, and a deeper appreciation for the context in which such projects unfold.
The wickedness within
A few weeks before the meeting, I’d stumbled upon a 1973 paper in the journal Policy Sciences by UC Berkeley academics Horst Rittel and Melvin Webber titled “Dilemmas in a general theory of planning.” In the paper, the authors define the concept of a “wicked problem”: a complex situation that’s perceived in distinct ways by different stakeholders, making it difficult to translate into a clear problem statement.
Although Rittel and Webber were referring to problems related to town planning, the circumstances they described mapped cleanly to the data warehouse conundrum my team was facing. Consider the following 10 characteristics of wicked problems, taken verbatim from the paper, with my notes from that time:
“There is no definitive formulation of a wicked problem.” The depiction of the conflict depends on who you ask. Is it a corporate power play to wrest decision-making autonomy from subsidiaries or a genuine attempt to standardize reporting across a global company?
“Wicked problems have no stopping rule.” Fundamentally, a data warehouse is never done; it continues to evolve as user requirements change.
“Solutions to wicked problems are not true-or-false, but good-or-bad.” A database design is never “right” or “wrong” (true or false), but some designs are better than others. In our case, the design will depend on the approach we agree on—corporate-focused or subsidiary-focused—and one group’s definition of “good” will be the other group’s “bad.”
“There is no immediate or ultimate test of a solution to a wicked problem.” Again, there’s no objectively “right” answer—which approach qualifies as “better” depends on who you ask.
“Every solution to a wicked problem is a ‘one-shot operation’; because there is no opportunity to learn by trial-and-error, every attempt counts significantly.” Building a data warehouse is an expensive endeavor. It might not be a “one-time” activity like building a bridge, but redesigning a data warehouse from scratch would be costly and implausible. In essence, we have one chance to get it right.
“Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan.” There are, in principle, a huge number of possible database designs.
“Every wicked problem is essentially unique.” There are fundamental principles of data warehouse design, but each data warehouse is singular, reflecting the requirements, capabilities, and technology choices of the organization—and, in our case, the politics at play.
“Every wicked problem can be considered a symptom of another problem.” Our problem is fundamentally one of organizational change, which is, in turn, a response to corporate management’s perception of business inefficiencies.
“The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution.” In our case, the discrepancy is that we have two conflicting interpretations of the problem. One group sees the problem as one of data standardization across the enterprise, while the other perceives it as one of addressing subsidiaries’ local reporting requirements. Which viewpoint stakeholders hold is ultimately a political matter rather than a technical one.
“The planner has no right to be wrong.” The database designer will ultimately be held responsible for the consequences of any design decisions. These decisions will require the designer to (somehow) steer a narrow course between the opposing desires of subsidiaries and corporate in order to satisfy both parties.
How many of these characteristics apply to large-scale software projects you’ve worked on? I’ll wager you’ve encountered at least a few. Software development is, itself, a wicked problem, requiring stakeholders to reconcile differing perceptions of what a product should do and how it should be built. We often assume the right project management methodology—agile, scrum, kanban—will solve the problem. However…
Methodologies (alone) aren’t the answer
In his 2002 book, Facts and Fallacies of Software Engineering, renowned software engineer and author Robert Glass argues that the top two causes of late and over-budget software projects are poor estimation and unstable requirements. Both are related to wickedness.
Complex projects are hard to estimate and requirements difficult to pin down due to internal or end users’ continually shifting perceptions of what they need. As a consequence, runaway software projects often end up failing, delivering suboptimal products that don’t work the way users expect them to.
The problems Glass describes have plagued developers since the early 1960s, when the first large software projects were conceptualized and executed. By the early 1970s, several software project management methodologies had emerged to address them. The most infamous of these is the waterfall approach, first described by Dr. Winston Royce in a 1970 paper titled “Managing the development of large software systems” in the Proceedings of IEEE WESCON. (Royce’s proposal is far more nuanced than the caricature of waterfall that emerged in the 1980s. For example, he pointed out that the one-shot waterfall is risky and recommended feedback between each step—iterations!—as well as doing a pilot implementation prior to the final version.)
The Manifesto for Agile Software Development, from 2001, responds to the perceived failures of waterfall, and while it does address some of waterfall’s shortcomings, it’s no panacea. The fragility of agility lies in the ease with which organizations can ignore the principles that make it successful, for example by implementing management-mandated deadlines rather than negotiated ones. Agile can easily become a collection of empty rituals that hinder progress rather than enable it.
But the manifesto emphasizes an essential truth: The success of complex software projects hinges on trust-based communication. Indeed, three of its four values—individuals and interactions over processes and tools, customer collaboration over contract negotiation, and responsiveness to change over following a plan—are about communication.
For simple projects involving small stakeholder groups, communication is relatively straightforward. But it grows more challenging as projects become more complex and distributed—not only because of the number of people involved, but also because team members’ and other stakeholders’ views and beliefs may differ greatly. In other words, due to issues of wickedness.
How to manage wicked problems
With wicked problems, the trick is to surface and reconcile diverse viewpoints, making multiple perspectives explicit so all parties can develop a shared understanding of contentious issues. Done right, this can lead to a resolution (or, at least, a partial resolution) of the issue.
In a 1970 working paper titled “Issues as elements of information systems,” Rittel and researcher Werner Kunz propose a visual notation called Issue-Based Information System, or IBIS, to help facilitators resolve differences between stakeholders. With IBIS, we can visualize the informal logic of conversation using three types of nodes: questions (or issues), which capture the problem being discussed; ideas, or responses offered to those questions; and arguments for and against those ideas (pros and cons). The notation is useful for managing wicked problems because it can help clarify stakeholders’ viewpoints.
Having used IBIS to map small meetings within my workplace, I was convinced it would be of great value to the upcoming discussion about the corporate data warehouse. The day before the meeting, I met with the project lead to canvas the possibility of using IBIS to map the conversation. After seeing a brief demo, he was quite taken by the idea and excited for me to use it.
The data warehouse meeting took place over the course of a day, with participants drawn from three stakeholder groups: corporate IT, local IT, and business reps from the subsidiaries. Mapping the conversation using IBIS enabled us to agree on the root question (how should we design the data warehouse?), surface different ideas in response to the question (subsidiary-focused versus enterprise-focused design options), and capture arguments for and against those ideas.
I noticed IBIS took the heat out of the discussion by objectifying points and separating opinions from their holders, which made it easier for the facilitator to nudge the conversation in productive directions. By the end of the meeting, we’d reached a consensus on a path forward: a data warehouse design involving a number of subsidiary-focused data marts that would support local reporting while enabling a consistent roll-up of data for corporate reporting. (I wrote about the meeting in more detail in a 2011 paper for the International Journal of Managing Projects in Business titled “Mapping project dialogues using IBIS: a case study and some reflections.”)
You don’t necessarily have to use the IBIS framework to manage a wicked problem. The central takeaway is the mindset with which you approach it. When managing wicked problems, you have to understand the substantive points of difference between stakeholders and find common ground between them, however modest it may be.
As my intellectual hero, Heinz von Foerster, once wrote: “The problem is not truth, the problem is trust.” Where there is no trust there can be no communication, and communication is the key to managing wickedness.
Epilogue: The fate of the two databases
The first database in our tale stood the test of time, remaining a primary source of information for sales and marketing decisions for close to a decade. The second was built four years after the meeting I described using the agreed-upon approach. However, corporate opted to retain control of hosting and development, thereby determining what got implemented and what didn’t. I’m long gone from the company, but I occasionally catch up with old friends from those days. They tell me that adding new functionality involves running a bureaucratic gauntlet that gives them nightmares, and they do all they can to avoid it.
While I understand my former colleagues’ stance, I can’t help but feel it perpetuates the very wicked problem we’d set out to resolve. I sometimes wonder what I would do in their stead. Invariably, I return to the answer I had all those years ago: approach it with a genuine intent to understand the other party’s position, build a shared perception of the disparate facets of the problem, and create trust through empathetic communication. These are the first steps to managing wickedness in town planning, software development, or anything else.