I have seen companies grow astonishingly fast, and I have seen them fall apart with equal force. What stays with me now is how little either outcome feels mysterious in retrospect.
The same patterns keep resurfacing: success reinforcing the decisions that produced it, incentives narrowing what an organization is willing to see, and perfectly rational choices accumulating into a position that becomes harder and harder to escape. Over time, I began to recognize that many of these patterns have close analogues in optimization and engineering.
One of them matters more than most: sometimes the path to a better position runs directly through a worse one.
Every company that survives its first few years gets good at climbing. Cut costs a little, improve conversion a little, ship features a bit faster. Each step nudges the metrics up. That's gradient ascent, and it works until the hill runs out.
The catch is that gradient ascent only reports on the ground under your feet. A company can climb its local hill all the way to the summit, doing everything right by every quarterly number, and still sit far below the real peak, because reaching it required going down first. Cannibalizing your own product. Entering a market where you have no existing advantage. Spending on something with no visible payoff for eighteen months. Each of these moves can read as a mistake on the metrics governing the current business. That's precisely why rivals often avoid them.
This helps explain a pattern that otherwise looks paradoxical: incumbents can continue outperforming a disruptor on the dimensions they already know how to measure even as the basis of competition shifts underneath them. Kodak understood digital imaging. Blockbuster understood that distribution was changing. Among other things, their existing economics made defending the current peak locally rational, a subtler failure than simply missing what was coming. For Kodak, protecting film could make sense inside a system built around film margins, manufacturing assets, distribution, and customers. The hill was real. Its height was the problem.
Optimization offers a useful analogy for escaping this trap: simulated annealing. Rather than always taking the locally improving move, the algorithm sometimes accepts a worse state, especially early in the search. It does not do this because deterioration is inherently valuable, or because random wandering is a strategy. It does it because refusing every temporary loss can trap the search permanently near the first decent result it finds, even if a much better one sits just beyond it.
A strategic pivot is obviously not a literal annealing algorithm. Companies do not choose their next market by sampling random neighboring states. The useful resemblance is narrower: exploration requires tolerating some locally negative moves long enough to discover whether they open a path to a better configuration. Early on, that means greater tolerance for experiments whose immediate economics look worse. As evidence accumulates around a promising direction, that tolerance should shrink and operating discipline should tighten. Exploration gradually gives way to exploitation.
None of this argues for taking every risky bet. Most valleys do not conceal taller peaks, and undisciplined exploration loses to competent optimization more often than it beats it. The narrower claim is more useful: when every decision is locally justified and the business is still structurally stuck, optimizing harder may only move you more efficiently toward the summit of the wrong hill.
But this is where the metaphor stops being merely a strategy problem.
The difficult question is not whether a company can intellectually understand that it needs to cross a valley. It is whether an organization made of individually evaluated people can survive the crossing.
The Incentive Gradient
Simulated annealing works in code because the optimizer is allowed to finish the search. The loss function does not have a career. It is not separately evaluated after each unfavorable iteration. No intermediate state is called into a performance review and asked why the numbers went backward.
A company is different.
A company contains many optimizers operating on different objectives and different time horizons. Each person is running something like a local gradient ascent on their own career, nested inside whatever search the company claims to be running.
That changes the problem completely.
Imagine leadership deliberately accepts a 15% revenue decline in one business line to test an architecture that could support a much larger business three years later. At the company level, this may be a rational exploratory move. But the manager responsible for that business does not get evaluated at the end of the three-year search. He gets evaluated this quarter. His team gets ranked this cycle. His budget gets negotiated against the current numbers. By the time the experiment could converge, the people who absorbed its early losses may already have been punished for producing exactly the result the strategy required.
The organization therefore runs on multiple objectives at once, each tied to a different level and operating on a different clock.
Leadership may be optimizing for enterprise value over a decade.
A division may be optimizing for annual revenue.
A manager may be optimizing for the next promotion cycle.
An employee may be optimizing for whether the current project produces a visible win before performance reviews.
Each objective can be locally rational. Together, they can make the global strategy impossible.
This is why telling an organization to "explore" is weaker than it sounds. A strategic decision changes the stated destination, but it does not automatically change what the people expected to get there are actually being rewarded for. If a manager is still rewarded for quarterly growth, asking that manager to intentionally produce a temporary decline is asking them to work against the very thing the organization continues to reward.
Eventually, the rewarded behavior wins.
This is the deeper reason companies struggle to cross valleys. Exploration and exploitation often require incompatible reward signals, a harder problem than risk aversion, short-term thinking, or a failure of imagination, because it persists even when none of those are present. The existing organization has been tuned, sometimes over decades, to correct deviations from the current business model. That correction mechanism is precisely what makes the company efficient on its present hill. It is also what drags exploratory efforts back toward the local peak before they have traveled far enough to discover anything else.
The solution has to protect exploration from the optimization machinery of the existing business, not simply authorize it.
That means decoupling what each part of the organization is measured against.
A serious exploratory effort needs its own capital, its own metrics, its own time horizon, and often its own organizational boundary. The people running it have to be rewarded for reducing uncertainty, validating assumptions, finding a viable architecture, or reaching deliberately chosen milestones, not for preserving the economics of the business they are trying to replace.
Seen this way, the value of a skunkworks or separately capitalized venture lies in the incentive firewall it creates. Extra resources and freedom matter partly because they help maintain that separation.
The firewall prevents the rest of the organization's optimizer from reaching into the experiment and correcting it back toward the local peak. It gives the new system enough isolation to endure apparently bad intermediate states without being forced to justify itself against metrics designed for a different business.
That distinction matters because exploration cannot merely be authorized. It has to be institutionally protected from the pressure to perform well right now.
Crossing the valley means tolerating moves that look wrong from the hill you're leaving. Almost everyone can see that, once it's pointed out.
Almost no organization is built to let anyone survive doing it. The ones that are doing exactly what I point in this article tend to become the most powerful companies there are.
A Quick Note on the Terms
For readers less familiar with the optimization language used here:
- Gradient ascent - improving a result by repeatedly moving in the direction that appears better from where you currently are.
- Local optimum - the best position available nearby, which may still be far worse than another position elsewhere.
- Simulated annealing - a search method that sometimes accepts a temporarily worse result in order to escape a local optimum and explore whether a better one exists.
- Objective function - the thing a system is effectively trying to maximize or minimize. In a company, that might be revenue, profit, growth, enterprise value, promotion prospects, or any metric that actually shapes decisions.
- Exploration vs. exploitation - the tradeoff between searching for better possibilities and improving what already works.
The mathematics is useful here mainly as a language for patterns that also appear in organizations: what gets rewarded, what gets optimized, and what becomes difficult to leave behind once a system is working.