In the European folk tale “Stone Soup,” a group of travelers arrives in a village carrying only an empty cooking pot. The villagers refuse to share food with the hungry visitors, so the travelers fill the pot with water, drop in a stone, and place it over a fire.
When a curious villager asks what they’re doing, the travelers reply that they’re making stone soup. The soup will taste superb, they promise, and they’d be happy to share it with the whole village. All they need is a little garnish to enhance the flavor.
Eager to taste this wondrous soup, the villagers begin to offer ingredients—meat, potatoes, onions, carrots, cabbages—at the travelers’ request, making the soup more and more flavorful. Finally, the travelers remove the stone from the pot and share the soup with the townspeople. Their clever ruse has resulted in a hearty, delicious meal the whole village can enjoy.
Software development shares some interesting parallels with this parable. Starting with a simple base (the stone) and adding layers (the ingredients) over time can help engineers manage risk and demonstrate progress on a project. This affords teams flexibility when a project’s strategy changes or critical dependencies fail to materialize. Taking a “stone soup” approach can help developers avoid getting mired in the planning stages of a project, spending precious weeks trying to predict every possible outcome or contingency.
Pathfinders are end-to-end tasks or workflows that provide value to your users, and they’re a proven method for making your software soup. (They’re also known as “tracer bullets,” as described in Andrew Hunt and David Thomas’s The Pragmatic Programmer.) A team working on an e-commerce website, for example, might build a flow that involves browsing a product catalog, selecting an item, and landing on the product page. By implementing basic functionalities, teams can demonstrate the desired workflow to stakeholders and/or customers, illustrate progress to project owners, and course-correct as needed.
At first glance, pathfinders might appear indistinguishable from prototypes. However, prototypes aim to explore specific aspects of a system and are often discarded and rebuilt after the component has been validated. Pathfinders, in contrast, are usually built on more permanent architecture. This foundation is the stone in your soup, and it’s often left in the delivered codebase. (This works better for software than it does for soup!) Pathfinders—the ingredients—represent simplified but complete implementations of a workflow, and as you add more and more pathfinders to your foundation, your project becomes more robust.
By providing fast feedback loops for complete features rather than requiring precise, up-front planning, pathfinders can help teams deliver complex and ambiguous software projects, especially ones with hard deadlines. Here, I’ll share how two of my teams harnessed this lightweight technique to usher two projects—Adobe Creative Cloud Assets and NodeSource N|Solid—from the pot to the dinner table.
A recipe for Creative Cloud chowder
I joined the Adobe Creative Cloud (CC) team in November 2010, not long after building a metadata service that launched with one of its flagship products, Adobe Creative Suite (CS). At the same time, I inherited the team building Adobe CS Review, a comment and review product that used acrobat.com’s backend. The team was tasked with building the digital asset management service that would become Creative Cloud Assets, the central hub and cloud-based storage for content created in Adobe’s desktop and mobile apps. Our requirements were no more elaborate than “Build a web frontend to the cloud storage, kinda like Dropbox.” We were starting practically from scratch, a pot filled with water.
Both management and the acrobat.com team assumed we would build on the acrobat.com backend, but several issues made this a nonstarter. For one, the backend was designed specifically for Flash or ActionScript clients. For another, it was a massive black box controlled by the acrobat.com team, so we couldn’t easily contribute to its functionality. To top it all off, we had an immovable deadline: the Adobe MAX conference in October 2011, less than a year away.
Knowing Creative Cloud Assets had to support web and mobile clients (and, eventually, desktop ones), we opted for an API-first strategy. Initially, our API would use the acrobat.com backend, but it would abstract things like asset uploads and metadata retrieval enough that we could switch to a different implementation as needed.
Because we were such a new team, and because Adobe was betting big on the cloud, we decided to build a pathfinder to demonstrate we could deliver on time. We leveraged the build and deploy infrastructure we’d created for the metadata service, as well as its feature flag system, user management, and the API service, which we extended to support asset management. This would be the stone in our soup. The only missing piece was a web frontend, which we were able to implement quickly by keeping the user experience minimal. The pathfinder used this web interface to let people view assets in their accounts and upload new ones to the cloud service.
In less than a month, we had a functional, responsive web interface that consumed our API. By focusing on this core workflow and being able to demonstrate working software early on, our team earned the confidence and support of senior leadership.
We soon learned we no longer needed to support the acrobat.com backend and that there would be a big CC offsite happening in a month. These events helped us define our next two pathfinders. The first replaced our API’s usage of the acrobat.com backend with our own code; the second integrated with the first two mobile applications being built for CC.
This second pathfinder was significant for a few reasons. First, the API was already integrated with our web client, so we had a clear model to follow. Second, it helped us identify issues in the API. For example, after observing workflows that required multiple API calls, we modified the API to return more data in a single request.
Focusing on that initial pathfinder allowed us to cut through the ambiguities, build incrementally, and generate confidence among stakeholders. At the offsite, our team was able to demonstrate the Creative Cloud Assets workflow end to end, backed by services running in AWS. We’d proven we could create a satisfying meal with just a few ingredients—and we were able to launch Creative Cloud in just 10 months.
A simmering pot of N|Solid stew
When I joined NodeSource as VP of engineering in April 2015, the company had already launched one product called N|Ship focused on packaging and securing Node.js applications for enterprises. Next in line was N|Solid, an application performance management (APM) solution based on an instrumented version of Node.js that enabled CPU profiling and heap snapshots. Enterprise software has historically been painfully utilitarian, so we planned to differentiate our product through an emphasis on frontend architecture and visual design.
Organizationally, we had a frontend team building N|Ship alongside a backend team building N|Solid and contributing to Node.js. But that changed when our CEO discontinued N|Ship and we had to pivot to shipping N|Solid alongside another project, Node.js Long-Term Support (LTS). Now, our goal was to launch alongside that release—in just three and a half months. That’s a long time for soup to boil, but not to pivot to a new project with a new set of goals.
With an understandably deflated frontend team, an APM backend team with an incomplete product vision, and an upcoming engineering offsite, pathfinders would prove vital to uniting the engineering organization and meeting our launch date.
At the offsite, the N|Solid team demonstrated what they’d built so far and described the visualizations they had in mind for CPU profiles and heap snapshots. With the launch date looming, I suggested reusing most of N|Ship’s plumbing for N|Solid and changing the shape of the data the N|Ship shell expected. Since the N|Ship engineers had the ingredients we needed to start our stone soup, we could spend the bulk of our time implementing those shiny visualizations.
For our first pathfinder, we took a heap snapshot of a process, a straightforward scenario that allowed a stripped-down version of N|Ship to call into the N|Solid server. Within just a day and a half, we had the pathfinder up and running.
Next, we turned to CPU profiles. This second pathfinder generated a profile for a process and visualized the data in a way that would help developers diagnose bottlenecks in their code. The first pathfinder provided the bulk of the infrastructure we needed to receive data, so we spent the majority of our time getting the code for the visualizations right—the essential elements to deliver great UX. Within a week, we had a working demo.
Pathfinders afforded us the speed and structure we needed to build our product swiftly and get early buy-in from management. Over the next few months we continued to iterate on it, addressing stability issues in the N|Ship code and fine-tuning the visualizations. We launched N|Solid 1.0 that October in tandem with the Node.js LTS release. Three months later, we closed a significant sale to a large enterprise client—further evidence that our stone soup had become a nourishing repast.
A bowl of pathfinder potage
Software projects are often fraught with ambiguity and speculation, tight timelines and high stakes. Teams may come to a project with nothing more than an empty cooking pot and a little water. This was the case for Creative Cloud Assets and N|Solid—my teams had firm deadlines but little in the way of concrete requirements.
In the face of myriad unknowns, pathfinders offer an alternative to detailed planning, providing a “decide-as-you-go” development model with tight feedback loops. The recipe is straightforward: Start simple with that first architectural stone, then build incrementally, adding ingredient after ingredient. As you continue to iterate and meticulously prioritize what to add next, your product experience will become richer and more appetizing to customers. The result will be a scrumptious soup for all to enjoy.