A few days ago I wrote about spending $220 on Google app ads and finding out 60% of the installs were robots, and I promised an update on the refund. I still haven’t heard back from Google, so I decided to go down a rabbit hole while I was waiting.
It may seem a little crazy, but between my experience with Google and all the recent talk about slowing AI down, I keep thinking about WarGames. For those of you not familiar with the movie, stop reading this and go watch it right now. It’s awesome, and at least 25% of the reason I got into computers. But, spoiler alert, a teenager hacks into a military computer and starts a game called Global Thermonuclear War, without realizing the computer is going to play it for real. They finally stop it by having it play tic-tac-toe against itself, over and over, and then every nuclear war it can simulate, until it concludes that “the only winning move is not to play.”
For me, not playing meant pausing the ad campaign, and that was an easy decision. (First I changed its goal to “won a puzzle”, but it kept sending junk, and paying Google while they investigated was annoying, so I turned it off.) We make a small daily puzzle app called Dayzle. The campaign was CA$40 a day, and turning it off cost us a trickle of installs, most of which were robots anyway. The stakes were low. The same decision is a lot harder for the people building AI, and the reason starts with my old job.
In my old life I wrote portfolio optimization software, which was clearly an optimization problem. It was right there in the name: use math to build a portfolio of stocks that maximizes or minimizes some goal, like return or risk.
The biggest headache in that work has a name: global optimization. Picture a mountain range where your goal is to reach the highest point. A person can look around and walk toward the tallest peak. Software is more like someone climbing in thick fog. It can feel which way the ground slopes under its feet, so it keeps stepping uphill until every direction goes down, and then it stops. That spot is a local maximum, the best answer nearby. The real top, somewhere else in the fog, is the global maximum.
I’m going to stretch those two words a little for the rest of this post. Local is what’s best for whoever is doing the optimizing, and global is what’s best for everyone. I’m hoping the math people will forgive me, because the problem has the same shape: you climb the hill you’re standing on, and you can’t see whether it’s the one you should be on.
Google’s ad algorithm is a very powerful version of that climber. It takes all the data Google has and tries to maximize whatever goal you give it, and my goal was installs. When something starts looking good on that goal, the algorithm leans into it hard. It found a bot farm that would “install” our app at a higher rate than anywhere else it had tried, so it decided that was the best place to spend my money. That was bad for me, because the money was wasted, and bad for Google, because I paused the campaign. The algorithm had no way of knowing either of those things. It was built to do one thing, and it did it very well.
So what was my real goal? Well, I want revenue. Installs were a stand-in, on the assumption that installs turn into players and players eventually turn into revenue. It turns out a goal that’s only roughly right can take you somewhere very wrong once something is optimizing hard for it. (Economists have a name for this too, Goodhart’s law.)
In a world with no bad actors, installs would have been good enough. Google, the algorithm and I could all have been happy counting them. But how bad is the bad actor, really? We can read about a bot farm and say it’s wrong, but in all likelihood the person running it is celebrated in their community. They describe the business as an adtech firm. They make good money, drive a nice car, belong to the exclusive golf club, and probably sponsor the local kids’ hockey team. Their neighbours see a successful entrepreneur, and in a way they’re right. That person found an algorithm with a flawed goal and optimized their own outcome against it, in a way that’s probably illegal but gray enough that nobody cares enough to do anything about it. Their local maximum is a very comfortable place.
Which brings me to the AI news. In the last few days the heads of Anthropic, OpenAI and xAI have all called for AI development to slow down, along with some regulation. The natural response is: “You run the companies, so feel free to stop playing any time you want. Why do you need the government to do it for you?” The psychology of that would be a whole other article, but I think most of it comes down to goals, the same way my installs goal did.
These companies have raised incomprehensible amounts of money on the promise of an AI future, and they’ve delivered a lot of it. AI has improved by leaps and bounds since ChatGPT first went viral. But the money comes with goals attached, and the goals are revenue and growth. If you’re hoping to go public at a trillion dollars, you need a market worth trillions, which means going after every industry, and doing it before your competitors do.
What does that look like from the inside? Say you’re the CEO of one of these labs. To raise the next round you need to hit revenue targets. To hit them you need a) the best people, so you stay at or near the front, b) that team pushing what’s possible in every industry at once, and c) to sell it to nearly anyone on the planet. The same goes all the way down. If a lab offered you three times your salary to work on something you were uneasy about, you’d probably take it, and if you turned it down, you know someone else would take it. You might even tell yourself you’d be more careful with it than they would.
Anthropic and OpenAI are both set up as public benefit corporations, so benefit to the world is written into their charters. The next funding round is still priced on revenue. So mostly we’re hoping that all this productivity makes the world better on its own, and that everyone using it behaves responsibly. If that happens it’ll be a happy accident, a lot like hoping my installs would turn into revenue.
Now picture being the one who decides not to play. I paused a CA$40-a-day campaign and nobody noticed. A lab that slows down on its own is walking away from hundreds of billions of dollars that investors put in on the promise it would stay ahead. Many of its people are paid in shares that are only worth something if it does, and a competitor that kept going will happily hire them. The CEO becomes the person who lost the AI race: to their investors, to their own staff, to a government that talks about beating China, and to peers who’ll say they lost their nerve. And after all that, AI keeps improving anyway, because the next lab kept playing. You’d have given up the money and the respect and changed very little.
That’s why they want the government involved. The step downhill only works if everyone takes it at the same time, and a law is one of the few things that can make everyone take it at once. The cynical read, that asking for regulation is a way to keep smaller competitors out, fits the same pattern. That would just be each lab doing what’s locally best for it again.
I’m an economist by education, and this is a textbook prisoner’s dilemma. Every lab would be better off if they all stopped, but only if everyone else stops too, so each one keeps going. Where they end up is what economists call a Nash equilibrium: nobody can do better by changing course on their own.
Society mostly runs this way too. We each optimize for ourselves and hope the total comes out somewhere not too bad, maybe even good. Most of us do fairly harmless things, like make puzzle games, so when our local goal pulls a little against the global one, it doesn’t matter much. The stakes change when the thing you’re building is an optimizer itself. All an AI needs to do real damage is a goal slightly different from humanity’s and to be very good at reaching it. The Google Ads algorithm had both, and it led to a bad outcome for everyone involved (except the bot farm owner).
The computer in WarGames got to learn its lesson safely, by playing tic-tac-toe against itself until it ran out of games, which is the closest anyone gets to seeing the whole mountain range at once. The labs will have to learn theirs for real, all together, while every one of them has a very expensive reason to keep climbing.
And that’s why I’d put myself in the doomer camp: every person involved is going to do what’s locally best for them, which is almost guaranteed to lead to a globally bad outcome.