I read something on LinkedIn this week that I haven’t been able to put down, which is not a sentence I expected to write.
It was a post by Alan Watkins about the word we use when an AI makes something up. We say it hallucinated. He points out that this is a strange word for it. A hallucination is a perception thing — your senses telling you about a world that isn’t there. The machine has no senses. It isn’t seeing anything. It isn’t fibbing either, because fibbing takes knowing the truth and stepping around it on purpose. His worry is that the word makes us picture a well-meaning little mind that’s trying its best and got turned around, and that isn’t quite what’s sitting there.
I’ve been chewing on it all week, which is a thing I do — I get one idea stuck and it follows me on walks.
And I came out somewhere to the side of where he ended up. Because I do think there’s a hallucination happening. I just think it might be happening over here, on our side of the screen.
Let me start with a small confession.
I use these tools for work. Some of you know that a chunk of what I do is helping small shops with their social posts and cleaning up copy for people who’d rather be baking or framing pictures than writing captions. AI has been a real help with the boring middle part — the third draft of a newsletter, the four subject lines I need before I can pick one. I’m not here to tell you to be scared of it or to swear it off. My mug and I are quite fond of it, most days.
But a while back I caught myself doing something, and I’ve been a little sheepish about it since.
I’d asked for some background on a topic I was writing up. What came back was lovely. Organized, confident, exactly the right length, with the kind of easy competence you hear in someone who has done a thing for twenty years. I felt that little click you get when a thing lands. Yes. That’s it. Good.
Then I went to check one of the details, because a copy editor’s brain never fully turns off, and it wasn’t there. Not shaded or half-right. Just not a thing.
What stuck with me wasn’t that the answer was wrong. Answers are wrong all the time; people are wrong at me all day and I hardly notice. What stuck with me was that little click. I’d felt understood. I’d felt answered. And that feeling had shown up right on schedule, and we were wrong together.
Here’s the part I find genuinely interesting, and I’ll keep it to the coffee-table version, because I’m no engineer.
The way these systems work, more or less, is by producing the next likely piece of a sentence. Then the next. Then the next. It’s a guess, made over and over, each one shaped by the guesses before it. The sentence doesn’t get planned the way I plan one, where I know roughly where I’m heading before I set off. It accumulates.
I want to be fair here, because “it’s just guessing” undersells it badly. An enormous amount of work and structure went into making that possible, and what’s in there is real — it has absorbed a great deal about how a recipe hangs together, and a legal brief, and a manual for a furnace. Calling it fancy autocomplete is a bit like calling a suspension bridge a rope trick. The description gets the parts right and misses the accomplishment entirely.
But underneath, it’s math and tokens. Ideas go in as numbers, get moved around inside a set of rules, and a guess comes out the far side. A good guess, usually, because the writing it learned from was done by people describing a real world. Sounding like that writing turns out to be a decent stand-in for being right.
Decent. A stand-in.
So that’s the guess. Here’s the gap, and here’s where we come in.
You show up to the conversation with a shape in your head. Sometimes it’s a sharp shape — you already know the answer and you’re just checking your work. More often it’s a rough outline. You know about how long a decent answer runs, what pieces belong in it, what somebody who really knew this would sound like saying it.
Then the answer arrives and it fits the outline. The two shapes settle into each other, and that settling registers somewhere below your thinking as understanding. You feel answered.
That feeling is carrying an awful lot, and it’s telling you less than it seems to.
It’s the same equipment that finds a face in a wall outlet. Two little holes and a slot, and there’s a startled fellow looking back at you. He’s not in the outlet. You brought him with you.
Which means when we take a made-up answer and accept it, two things happened. It guessed and missed. And the shape we were holding was loose enough to take the miss. The second half is the half we can actually do something about.
I don’t think we should feel bad about this, for what it’s worth.
For as long as there have been people, somebody speaking smoothly and confidently about a thing was pretty good evidence they knew the thing, because you couldn’t talk that way without the knowledge underneath. Any of us can sort the person who has actually repaired a dishwasher from the person who read about repairing one, in about two minutes, mostly by the way the sentences move. That instinct served us well for a very long time.
It’s just that smooth talk got cheap, and it floated free of whether anything’s behind it. The instinct keeps firing anyway, the way an optical illusion keeps working after someone explains it to you.
So the things I’ve started doing are small and unglamorous, which is my favorite kind of advice.
I check the parts I could check if I bothered — names, dates, numbers, anything with a source attached. I’ve learned that if I couldn’t tell a right answer from a wrong one on some question, that’s the question to slow down on rather than speed up. I notice when my reading has gone easy because the prose is going down easy, since that moment shows up before I catch it. And when something confirms what I already believed, I try to give it a second look rather than a pass, because a shape I’m already holding is the easiest one in the world to fill.
None of that is hard. It’s about a mug’s worth of extra patience.
The picture I keep now is two words, which is about my limit for keeping things straight.
There’s the guess, which is what the machine hands you.
And there’s the gap, which is the distance between that guess and what’s true.
The gap doesn’t show. There’s no wobble in the voice, no hedge, no little tell. The only way to see it is to go look, and going to look is a choice you make while you’re feeling perfectly satisfied.
Alan’s point was that the better we understand the machine, the better we can work alongside it, and I think that’s right. I’d just add the other half. Understanding ourselves is what keeps us from misusing it. A tool doesn’t get misused because it’s a bad tool — it gets misused because of what the person holding it assumed it was for. If we go in believing there’s something in there weighing the question the way we would, we’ll hand it work it was never built to do, and we’ll accept the results because they arrived sounding right.
Know what it’s doing, and know what you’re doing while you read it, and it’s a good tool. It’s been a good tool for me.
Anyway. Go refill your coffee. I’m going to go take a picture of the fog.
Heather Dean
