Yesterday I tried to fix my bicycle. It wasn’t the kind of repair you do once or twice a month — I had to order some special equipment I wasn’t familiar with. Things didn’t quite go as planned, and I was afraid I’d end up causing damage instead of putting things back in order. So I consulted an LLM, which gave me some guidance, but the more I thought about it, the more I realized that if the advice wasn’t completely accurate, I could break the bicycle for good. So I searched the web, but I couldn’t find anything that would confirm, refute, or add to what the LLM had told me. The AI summary at the top of the search results said more or less the same thing as the model I’d already consulted. Then I searched YouTube, hoping to find a video showing the same edge case I’d run into. I couldn’t find one. I couldn’t spend hours trying to verify it, so I decided to take the risk. It was just a bicycle, after all. I ended up damaging it, but only slightly, and a professional will be able to set it right, I hope.
It wasn’t that the LLM was wrong. Based on what I’d learned from my mistake, I asked it a few follow-up questions, and it told me how I could have prevented what happened. So it had been right all along — I’d just prompted it poorly, and under time pressure I made a rushed decision that happened to be the wrong one. The time pressure came from how hard it was to find something a human had written — or at least knowingly approved — that would help me verify the LLM’s output. As an engineer, I’m well aware that there are situations where I don’t know what I don’t know. That’s why my natural tendency is to look for experimental evidence that shows me in detail how something works end to end. I have my own methods for surfacing the important things I’m unaware of — even the ones I don’t understand well enough to ask the right questions about. This time, I failed. It was just a bike; not much was at stake. But it was the first time I’d had to rely solely on an LLM in a make-or-break situation. I pray you never find yourself under time pressure for weightier reasons, deciding on things far more crucial than a bike. Sadly, I know quite a few people who already have.
People who aren’t software engineers by trade would suggest I need to learn how to prompt the AI better, but that’s simply not true. I’m convinced the experience I’m about to relate is impossible to convey convincingly in words; you can only gain it by spending at least a couple of years writing software and looking back at your failures:
You can never properly test your own output by yourself.
Not even if you try hard, not even if you’re the most senior engineer, not even after reading a ton of books. It’s impossible. You need a second pair of eyes, a human — in software we call them QA engineers or testers. There must be a similar role in every field of engineering. Alternatively, the other person may be God — by which I mean you run an experiment that tests your output against reality, which is bound by God’s laws and gives you instant feedback from the hidden Him. Countless times, I’ve watched managers ask me or my peers why we needed testers: “Couldn’t you just be more diligent? Next time, you simply need to try a bit harder to find your blind spots.”
If all you’ve ever done is fill in spreadsheets and check up on other people, or you’ve never worked in the role that does what the company fundamentally does, you can never get it — yet it’s a natural law that you can’t discover your blind spots any way other than with the help of another person. I believe few things are more fundamental to engineering than this — if anything is at all. Needless to say, this applies to our inner lives within the body of Christ.
Part of the reason this finding seems so counter-intuitive is that an experienced worker often produces work that’s almost flawless — or at least perfectly acceptable. Even without a second pair of eyes, such output would pass every quality standard. Some managers are therefore tempted to get rid of quality-assurance people, bypassing them by creating just enough pressure — high enough to make people more vigilant about their blind spots, but not so high that it breaks them. This is a bit like trying to pull yourself out of a swamp by your own hair, and then, when that doesn’t work, trying to fix it by pulling harder. If only we knew which parts of what we produce are flawless and which need fixing — then the second pair of eyes wouldn’t be necessary at all. But who will tell us? And can it be someone who isn’t a “who” at all? Can’t we just ask AI?
It seems this is like the Song That Never Ends: we’re back at square one, at the beginning of this article, where I used an LLM to ask for guidance — but since I didn’t want to break my bicycle for good, I tried to find something a human had written, or at least knowingly approved, that would help me verify the LLM’s output. This recursive line of thought is not unlike the struggle to build a perpetuum mobile; only instead of an infinite source of energy, you’re after an infinite source of manpower — also known as intelligence. You can’t do it. It’s against the laws of nature, and God’s design has no flaw susceptible to a hack. If you’re an atheist, just remove the word “God” and everything still stands. It’s rock solid.
An LLM is like a huge book in which everything is written, and with each new version of the model it’s written more clearly, precisely, and accurately. We just never quite know which parts of the book to read. Or we do know, but it would take us too long to read them, so we end up reading only a short summary, which may not reflect our particular situation well. The problem with the book is that it’s static — it’s not a living person.
We scraped all the books in the world into one ultimate book, and we hope that if the complete wisdom of the universe isn’t there yet, it will be in the next version. We even have a magnificent way now to search the book for whatever we need. The LLM can produce not only what we asked for, but also everything we didn’t ask for but would have, if only we knew what we don’t know. Except that in that case its response would be too long for anyone to read.
As I argue in my book, natural language has a limitation that can’t be eliminated: it isn’t suited to storing knowledge in sufficient detail while still keeping it comprehensible to homo sapiens. We’re at a stage in our cultural evolution where we’re beginning to entertain the possibility that understanding the knowledge doesn’t matter so much, as long as we can use it. Alternatively, we contemplate replacing homo sapiens with something better — something with fewer limitations.
Technically the solution is simple, as I’ve argued in my book and elsewhere. And I’m building one: instead of removing understanding, or the human species, from the equation, I’d dare to suggest we try storing knowledge in something other than natural language. That I would find feasible. And don’t worry, it won’t hurt the LLMs; they’d most likely thrive because of it. But we need to start maintaining this knowledge infrastructure together, collaboratively, with our own eyes on it, so that it carries our seal of approval — possibly with a great deal of help from LLMs.
We may not have enough time left, though.
I’ve just realized I can’t find a bicycle repair shop near me with decent reviews and somewhat heavier machinery. That is, I can’t find it without asking an LLM. These days, all the services that aren’t LLMs or big-tech aggregators are much harder to find — all those sites whose content was created by an independent set of eyes, by real people. Maybe they’ll be gone for good soon, and asking an LLM will be the only option left. But what will the LLMs use for training then, to stay up to date? Especially once we no longer possess the understanding ourselves.
I think we may be losing it.