Stuck in a Local Optimum — Part I.
On the invisible ceilings we've stopped noticing
In Silicon Valley there lives a man who looks like the devil and talks like the devil, and may even think of himself as one. His central thesis revolves around the notion that we as humanity have been stuck in a kind of local optimum, one from which it is quite difficult to make any progress. This is why we have been plateauing for the last 50 years, without any major breakthrough in science or in culture more broadly, except perhaps AI based on large language models. He then blames all sorts of things and proposes that we do all sorts of evil ones.
I will not argue with him in a way that promotes his thinking; instead I will explain my own view of how we got to where we are, because I think his initial premise is true. There is always a grain of truth in any evil — otherwise no one would be tricked into it. The rest of it I have no use for.
A local optimum is a solution that looks good enough to live with for a while — one that may even seem like the best we could ever find, except that it isn’t, and we don’t know it. Suppose we had never invented the steam engine and were stuck with horses as our most advanced means of transport. Maybe there are good reasons to stick with horses. Perhaps, on considering the steam engine, we would see all its implications and decide against it. But prematurely concluding that there can never be anything more effective and efficient than horses is simply not right — it’s not true. It’s living in a lie. And God never wants us to live in a lie. He is the truth, and He wants everyone to know the truth. He wants everyone oriented toward Him.
It’s fairly easy to see the issue in the artificial example of the steam engine. You may find it abstract, even pointless, and think that nothing so obviously stupid could be happening in the world around us. Yet it does — all the time, and almost everywhere. Let me give you some real examples.
In November 2021, the antitrust division of the U.S. Department of Justice (DOJ) sued to block Penguin Random House from buying Simon & Schuster, its major competitor and one of the Big Five publishers. The DOJ argued that the merger would harm authors — specifically the writers of “anticipated top-sellers,” the books that command advances of $250,000 or more — because the two houses would no longer bid against each other for their work. Stephen King testified for the government, against his own publisher, likening the promise that the merged imprints would keep bidding independently to a husband and wife bidding against each other for the same house. In late October 2022 the court agreed and blocked the deal.
The more revealing part of the trial, though, was the portrait of the industry that emerged from the testimony. Publishing, the executives conceded, is an unpredictable, low-margin business in which no one can reliably guess which book will sell. Most titles sell only a handful of copies, and even books acquired for million-dollar advances frequently flop. During the trial a statistic was cited suggesting that half of all published titles sold fewer than a dozen copies — a figure now widely regarded as inaccurate, misread from BookScan’s data. But the deeper claim survives the correction: publishers cannot tell in advance which book will succeed. And the money follows that uncertainty — the bulk of advance spending is poured into a small number of near-guaranteed names: celebrities, franchise authors, brands the houses lean on to stay afloat.
What follows from this, in my reading, is that authors who write books that could influence humanity for centuries are not promoted, not supported, often not even discovered. One could accept that a few potential Shakespeares, Tolkiens, and Hemingways will remain undiscovered forever; yet the reality looks more like all of them remaining undiscovered — or else there being none at all. This is a very likely instance of a local optimum, and it is not acceptable.
My prediction is that at some point in the future, once we have the will, the means, and the skill to tell a fabulous book from a mediocre one, dozens of consequential titles will suddenly surface and change the world. AI could help us get there — but it won’t if we keep insisting that what I’m saying here is naïve nonsense. It won’t if we keep saying that whether a book is good is as irreducibly random as a quantum measurement, and that no one could ever predict it. Nothing will change until we can at least entertain the possibility that the book market is stuck in a local optimum, one caused by cultural disorientation and a flood of mediocre, mass-produced text.
But maybe you think it’s just books. Perhaps you don’t regard books as the crucial cornerstone of civilization that I do. Then let’s take a look at science. I know several scientists, and all of them agree that it is incredibly difficult to get funding to investigate a potentially game-changing subject no one has looked into before, because it’s considered too risky. Science is financed by public money — or, more generally, by the kind of money whose backers, mostly taxpayers, expect to see something in return. And the safest, most reliable way to satisfy those backers is to pursue only the tiniest incremental steps. This means we are measurably moving forward, yet whatever major discoveries do happen, happen by accident — or by mistake.
Consider the story of the mRNA vaccines that saved millions of lives during the Covid pandemic. In the 1990s, Katalin Karikó, a Hungarian biochemist at the University of Pennsylvania, was convinced that messenger RNA — the molecule that carries genetic instructions to the cell’s protein factories — could be turned into medicine. The field disagreed. mRNA was thought too unstable, too inflammatory, too impractical to be worth the money. She couldn’t win grants. In 1995 she was demoted from her faculty track, her salary cut. She kept working anyway. Together with the immunologist Drew Weissman, she eventually found that chemically modifying the building blocks of mRNA let it slip past the immune system’s alarm response — the discovery on which every mRNA vaccine now rests. When they submitted the finding in 2005, both Nature and Science rejected it. It took another fifteen years, and a global pandemic, for the world to notice. In 2023 the two of them were awarded the Nobel Prize in Medicine.
Or consider CRISPR, the gene-editing tool that has become the defining biotechnology of our century. Nobody set out to discover it. In 1987 a group of Japanese researchers led by Yoshizumi Ishino were studying an unremarkable gene in the bacterium E. coli when they stumbled across a strange pattern in the DNA nearby: short sequences repeated at regular intervals, separated by unique spacers. They had no idea what it was. Their paper ends, almost with a shrug, by admitting that the biological significance of these sequences was unknown — and then they moved on. For the better part of two decades the observation sat in the literature, cited once or twice a year.
What rescued it was not a funded program but plain curiosity. In the early 1990s a young Spanish researcher named Francisco Mojica, working on salt-loving microbes for reasons that had nothing to do with any application, noticed the very same repeats in an entirely different organism. That they appeared in creatures so unrelated convinced him they had to matter. He was, for the most part, alone in thinking so. When he tried to publish his interpretation, Nature turned him down. Only years later did anyone grasp that these repeats were a bacterial immune system — and that they could be repurposed into a tool for rewriting the code of life itself.
Two of the most consequential biological advances of our age, and neither arrived the way the system is built to deliver progress. One survived because a single researcher refused to give up on an idea everyone told her to abandon; the other survived because a curious microbiologist chased a pattern no funder would have paid him to chase.
Now set that against physics. If biology shows us invention by stubbornness and accident, theoretical physics shows us the same dysfunction from the opposite direction: a system that lavishes decades of funding, prestige, and talent on a program that has produced no breakthrough at all. The physicist Sabine Hossenfelder (whose YouTube channel I keenly recommend) has spent years making this case, most pointedly in her book Lost in Math. Her charge is that fundamental physics has seen no major breakthrough in more than forty years — and that this is not bad luck but the product of bad incentives. Theorists, she argues, have been seduced by mathematical beauty and elegance into building ever more elaborate structures — string theory, supersymmetry, the multiverse — that are untestable in principle and unconfirmed in practice. When the Large Hadron Collider was switched on, the particles that supersymmetry had been expected to reveal simply failed to appear. Rather than abandon the framework, its defenders adjusted the parameters and pushed the predictions to energies just out of reach, where the theory could once again be neither proven nor disproven.
And the machine rolls on regardless, because the incentives reward it. Publish or perish — and the former you do by working on what everyone else works on. You win grants by extending the fashionable program, not by declaring it a dead end. A young physicist who suspects the whole elaborate construction is just a mathematical fantasy has every professional reason to keep quiet and add another citation to the pile. It is the local optimum again, manifesting in one of the most prestigious fields we have: a beautiful, self-consistent, richly funded structure that may simply be describing nothing.
With finite time and resources, one has to set priorities. That’s true everywhere, science included. But who would dare to tell a more meaningful question to investigate from a less meaningful one? In a world where there is no God, how do you even argue about what makes sense to explore, when making sense doesn’t itself make much sense in the first place?
This obviously doesn’t apply when the question is whether to build atomic weapons capable of utterly destroying civilization. Maybe that was the last question we answered with funding backed by religious fervor. But there seems to be no question of similar magnitude today. Not even the question of getting the climate back on track, which would help preserve livable conditions for our children — nor the question of eliminating poverty, which would substantially expand the size of the world market.
Our society is so full of crucial examples of being stuck in a local optimum that I decided to split this article in two. Next week you can read further examples, along with what they have in common and what we need to do to start making progress.