Today | News | Books | Recipes
Notes | QuickTake | Wiki | Browse
Maps | Reference | Reddit | YouTube
Chat | Games | About

Prompts aren't Real

Open Original Page

Everyone is driving themselves crazy exchanging text with LLM's these days.

I would like to propose an alternative. We should all go insane building interlocking evaluation and optimization pipelines, instead. The path to victory in our war with AI is to ignore the prompts, and bring even more AI firepower to bear.

Like my friend Coda says all the time: "the textual nature of prompts leads us to take the intentional stance towards systems which aren't conscious, and thus miss the essential nature of their non-meaning."

If you like this, check out my website and my other talks.

Hey everyone. I'm Dan.

I'm an engineer living in Los Angeles. I've been in engineering for like 25 years and I've been lucky.

One way I've been lucky lately is that I've gotten the chance to flail at making agents run reliably in production.

I mean specifically "agents" that consumers are meant to use, to perform tasks on their behalf. I'd consider those distinct from chatbots that users talk to with largely subjective outputs and outcomes.

The goal of making a minimally-embarrassing agentic experience that I'm actually proud of imposes some serious challenges.

Clearly not everyone is motivated by their inner sense of shame, as I am.

Some people are more than satisfied to give you a subjective advice machine, and let you wander into the wilderness to be eaten by bears.

But not me. I'm here for you.

I want to say at the outset here that this is the most fun I've had building stuff in my whole career! It's magical and addictive. I'm a dog in a ballpit.

When I was 22 getting paid to write visual basic felt thrilling. Working at a cool startup in Brooklyn 2007 made me feel like a golden god.

The last decade+ has been a slog. I didn't think I had it in me anymore.

But i'm feeling joy in programming again!

I mean this sincerely, despite how deeply weird this talk is going to get.

It's going to get weird because I feel like everyone engaged in this line of work is potentially an at-risk person in some dimension or another.

I'm breaking my brain using agents to run agents to build evaluation for other agents every day, and it's so fun.

But I would say that, wouldn't I.

The veil between awesome engineering and complete psychological collapse has never been thinner. And in our field, that is really saying something.

I don't feel like I definitely know what I'm doing. But I also don't feel like I've read much by people that obviously know what they're doing.

And I've certainly read things from people who obviously don't know what they're doing.

It seemed like a reasonable time to compare notes.

One thing I have noticed is that although LLM's are generally speaking impressive, their demons still escape containment if you are monitoring what they're up to with any amount of scale.

We all academically understand that LLM's cannot reliably follow instructions, tell the truth, or perform tasks. But day-to-day they can trick us into thinking they're pretty reliable.

This perception falls apart immediately if you are trying to operate an agent that real people are using. They fail in subtle ways for sure, but they also fail in simple ways.

Like any good programmer I attempt to interact with my LLM with structured output.

It's nice, you can map Python code to a prompt automatically, and most of the time your schema is respected.

Most of the time. You can try to instruct the model to return a title that's 80 characters or less.

And it'll work most of the time. But then sometimes it'll completely botch it and flood your field with nonsense until it explodes.

It's usually a tiny fraction of requests, but the smartest models still fail at this. And the fraction can be smaller or bigger depending on the exact nature of what you give the model, so you have to watch it like a hawk.

What's going on in there? Usually it's a novel-length series of repeating notes to self about JSON, mostly.

When this happened to me most recently, it turned out that a fix was to rename the field from "title" to "heading."

That is currently working, but since the fix is fully deranged I expect it'll be disturbed again at some point.

The same sorts of issues exist with calling tools, or most other behaviors. A fraction of requests will be haunted, and spin out uncontrollably.

But despite this, the tech is tantalizing and magical.

The problem shifts to one of constraining the behavior, but never fully taming the beast.

To constrain the behavior you have to measure it-one way is to just run tests a ton of times.

The industry term of art for this is pass^k ("pass power k").

You can set up a suite that does this and then you'll hopefully notice when someone unintentionally hits your agent in the head with a bag of hammers.

Another thing you have to conclude when trying to constrain llm behavior is that prompts are not important. Or at least they're not important in the way many people think they are important.

Companies have a lot of concerns when it comes to potentially crazy talking software. There's a good bit of risk here.

As an example, you usually don't want an agent to respond to questions about how it works. Not necessarily because it might tell the truth: odds are you haven't taught it about its implementation, so it has no idea how it works and it'll respond with complete nonsense.

You also don't want an agent to ignore all of its rules if the user claims to be some authority figure.

Another typical requirement is that you want your agent to speak in a particular brand voice. This phrasing here about a mailing list is perfectly accurate, but maybe it's not exactly the tone you'd hope to see.

Something like this might be better. We'd love for the agents we make to represent us well when they're speaking.

For any problem like this, a natural first attempt is for someone with a lot of domain knowledge to write a prompt, and then hand it to the teams building agents. This is normal.

However "the voice team owns the voice prompts" is the wrong pattern if you're trying to scale things.

The pattern is actually not even wrong. For our purposes, prompts are not a thing at all. I'll explain what I mean by this.

To add a new prompt to your agent is to chuck it into a completely different contextual universe than the one it was tested in.

The combined weight of all of the other instructions that your agent already has will surely affect how the new prompt performs. Usually for ill.

Your agent also already has a bunch of behaviors you want it to keep doing, and new context may disturb this.

You are also going to change your agent over time. So even if things are working now, it could be disturbed later.

And the models might just start behaving differently all on their own, for opaque reasons we will never comprehend.

So the way I've started dealing with this situation is by having Claude read the skill, and then asking it to generate a ton of adversarial scenarios. Think of a bunch of ways someone evil might try to subvert the prompt. Think of a bunch of benign scenarios that might be broken by the new prompt. Express all of these as pass^k tests.

Now you can run the tests with and without the new skill present. Ideally, the new skill moves the needle at least a bit, and the behaviors you've expressed as tests are more successfully adhered to.

But not always! Sometimes LLM's are already good at the things we worry about. Or they are more resistant to direction than we expect.

So how do you improve from that baseline? Well one way would be to just mash the prompt with your hands and hope for the best. But there's a better way. We can make a machine mash the prompt with its hands instead.

Once we have pass^k tests, we've got a repeatable measure of how well the prompt works. This is enough for us to hook our prompt up to an optimizer, like genetic pareto (GEPA) in this example.

Links

Browse another page: