How to Make a Good Secret Sauce
Knowledge moats (or secret sauces) are one of the most fundamental type of moat in business. They consist of the information, data and know-how in systems, processes and people that is difficult to copy by competitors. They include collections of recipes, special tools and the accumulated knowledge of employees. For companies that build technology, the knowledge moat can be vast and complex often including hundreds of thousands of lines of code designed to analyze and automate aspects of an industry. Knowledge and information can be dry subjects but keep reading for my favorite ways of exploiting them as well as an appearance by Chuck Norris.
Knowledge is a map
How do you build a good knowledge moat? One thing that helps is to have a sense of the scale and dimensions that knowledge can have and a sense of the amount of work needed to accumulate it. There are different ways to talk about knowledge and information. Books and encyclopedias are often used to portray knowledge. However, one metaphor I like, which emphasizes important characteristics of knowledge, is to talk about it as a map. In business, we are usually talking about a map of the challenges, the problems, the pains, the dangers and the roadblocks customers encounter, as well as their likes and dislikes, a map that reveals the best routes they can take to navigates their world.
A map is less detailed than the real world
Thinking of knowledge as a map emphasizes a few important aspects. First, a map is a lot less detailed than the real world. It only contains a fraction of the information present in the real world, but it’s still very useful because as some say, you can’t fold a territory and put it in your pocket. A map is information reduced and simplified so that it’s easier to reason about. For digital information, it also means it’s more computationally tractable. Unnecessary details are abstracted away or eliminated where they don’t matter. This reduction in details is also used to emphasize certain aspects, filter out noise and reduce cognitive load.
It’s often easier to read and reason about the world shown on digital maps using normal map mode than using satellite mode. Satellite mode usually contains too much distracting information. You don’t need to know the position of every tree to understand how to get somewhere. Looking at all these trees use up some of your brain’s recognition and processing power.
Some of the same tricks are used with computer models. It is rarely efficient to take into account all the tiny details. For example, simulating a car atom by atom for self driving purposes would take too long even though it would in some sense, be more correct. Machine learning internals are often like maps. They are about reducing and simplifying the data to its essential parts, about reducing the parameter space. The ideas of overfitting vs underfitting usually come into play (these are traps that are easy to fall into, more on this later).
Zooming in can reveal an explosion of details
Another important aspect of maps, especially electronic zoomable maps, is that despite being simplified, zooming in or looking closer, can reveal a lot more details. Geographic terrain has been described as having fractal patterns. As you zoom in on parts of it, more and more details emerge. On a 2D map, zooming in by one factor reveals about a factor squared amount of details.
The analogy fits the exploration of real world problems. When an area important to users is found having complex obstacles and use cases, it is possible to create a ton of difficult to copy value by looking at this area closely, drawing at high resolution to show fine details of the obstacles and clearing the easiest path around them.
Remember the last time you tried to solve a technical problem and as you progressed, you discovered problematic edge cases after edge cases. This was you zooming in on the details of the problem.
This is all very abstract. Let me provide an example. A company that aggressively exploits knowledge moats is Stripe. Stripe noticed that the most painful, difficult and complex part of online payments was the bureaucratic layers, banking red tape, exceptions and edge cases related to fraud prevention, related to legal hurdles when doing transactions across regions and jurisdictions, related to security requirements when capturing credit cards on websites etc. etc. They noticed that as you looked closer and closer, this field had tons of edge cases that were painful to navigate for most businesses. They decided to gather fine knowledge of everything related to online payment obstacles through lots of trial and error and map it all out to be able to build automation that covered these edge cases. They then presented their users with the simplest of APIs that almost never hit unhandled obstacles.
Previous payment systems were a lot more complex to use, putting the burden of handling edge cases on users. They would throw exceptions and errors at developers and users when a transaction veered off from the most common use cases. Merchants often had to contact individuals in the traditional banking system to try to find workarounds for errors. Oftentimes, end users just gave up on trying to pay for something online and went to physical stores to make their purchase. Stripe’s stated mission to “increase the GDP of the internet” means they want to avoid this at all cost.
Stripe’s repeatable strategy seems to be to find bureaucratic realms that have layers and layers of painful and complex hurdles and discover and find or build the best streamlined paths around them. That is why they created Stripe Atlas, initially a tool to make it simple for people outside the US to incorporate a company in the US, something I once tried and failed to do myself because I was hit by a wall of paperwork and regulations.
Efficiency requires getting the right things right
When exploring terrain to gather information for a map, it’s really useful to know which parts are important, which parts needs to be drawn at high resolution and which parts only need low res imagery or simple shapes. This is because you can save a massive amount of time and resources if you can skip irrelevant details in some areas. The almost exponential nature of zooming in multiple dimensions makes it impossible to get everything right at finer levels, success depends on getting the right things right. This is crucial for information gathering efficiency but it also results in more usable, less overfitted data with less distracting noise.
For businesses, overfitting usually means having a product that has too many details specific to a few select customers, details that don’t generalize to a large enough user base or that are simply irrelevant and distracting. Underfitting happens when products are too simple and general, when they don’t capture important details making them easy to reproduce by competition. Products that can be used for many different use cases tend to be underfitted to particular ones and tend to be easily commoditizable (and are often outsourced to low cost countries). Finding the optimal fit requires a lot of experimentation.
For a product team, exploring the right areas efficiently requires having good knowledge of what is important to users, what is difficult and what is painful. It requires the team to have close relationships and continuous feedback from users to help steer the exploration. It requires a lot of trial and error focused on important areas. Sometimes most of the value ends up being in a few key hard to uncover details.
They say, when learning how to write, that condensing and summarizing while maintaining clarity is often the hardest part. The same is probably true when creating products that fit users’ world concisely yet tightly around the important details. A well designed product makes complex scenarios seem simple. It’s easy to forget how many details are dealt with under the hood of such a product, until you use an alternative that doesn’t have these edge cases covered.
It’s somewhat of an art to distinguish areas to gloss over and the more important areas to dig into. With practice, it’s possible to develop a sense of how long it takes to peel off layers and how much value you can uncover in particular areas. Honing this skill and then using this perspective to look at your favorite technologies will make you conscious of how much resolved complexity often hides underneath simple UIs.
Even small details take time to discover and conquer
Since fine maps have tons of details, map building is more marathon than sprint. Every detail whether large or small often takes a similar amount of time to discover, analyze and plan for, even the smaller edge cases revealed by looking very closely at problems. Because of the efforts involved, the work is often only worth doing when the problems apply to a large enough market.
To get a visual sense of how getting to a high resolution in multidimensional space takes very many iterations, look at this picture of Alan Turing refined iteratively by adding simple shapes (or if you like, a video of the process to map Chuck Noris’s face).
When building products, you may start in a fog of war with an MVP designed with broad strokes and then iterate into details. Look again, at how many ovals there are on Turing’s face in the last image in the series. On a finished product, the individual strokes are not necessarily noticeable, but they are there and usually each took time to place.
The learning cadence is crucial
A fast “build-measure-learn” feedback cycle is important to get to the finer details in a reasonable amount of time. Every piece of feedback you get from your users is like one oval added to your map.
Some people call the pace at which you draw new details the heartbeat of the company. The faster you do it, the more you can get, through feedback, to a high resolution picture of the important areas of your user’s world.
Of course exploration and knowledge building is not the end goal. The goal is to plan the best routes around or through obstacles, to build the best roads, bridges, tunnels, railroads or self driving cars that pleasantly take users to their destination.
Copying detailed knowledge is fundamentally difficult
A detailed and well fitted map, neither overfitted nor underfitted is very difficult to copy by competitors. They would need to use your product in countless situations to see how it reacts to all the edge cases. Those that try to copy you might build superficially similar interfaces, but waste efforts getting the wrong details right while never getting to the more important ones (they would almost always be better off studying their own users and building something more fitted to their own context and reality).
I would even say that the difficulty inherent in copying this type of knowledge, is one of the fundamental property of the universe. Information theory and the closely related field of thermodynamics describe how easily, complex correlated patterns, gravitate towards disorder and information loss. Having well fitted knowledge results in expertise, which you can only get through lots of experience as it is fragile and tends to dissipate. This limit in knowledge transmission is what prevents, for example, people from learning to play music or sports just by listening to the pros talking without practicing themselves. In software, it will be very difficult for competitors to reproduce your codebase’s good fit to your industry just by looking at your UIs. It’s what makes a secret sauce based on lots of first hand experience, a good moat.
One powerful thing about computer code compared to, for example, human memory, is that it doesn’t degrade over time. It’s almost immune to thermodynamics. It might need to evolve to maintain fit with a changing world but for the problems that don’t change very much, there is generally little degradation. Well fitted, detailed code that competitors don’t have access to, can thus be a very durable part of your knowledge moat.
Product teams should strive to be aware of the challenges their users face at varying levels of details. Tailoring a solution based on a refined knowledge map that covers important edge cases is what makes a product premium and prevents it from being compared to and copied by low cost alternatives. It allows solutions to take subtle non-obvious paths that are hard for competition to copy. In the best case, your users might forget that all the annoying edge cases exist, until they try your competitor’s product and start tripping on them repeatedly.
