That one time an agent was more helpful than I wished for

Very often I see people starting a system prompt like "You're a helpful agent/bot that does XYZ", and I've done it, too. But once I was working on an agent to take natural language request and turn it into a SQL query in a very well defined domain. We had added canonical queries, background information, guides how to modify the query, how to add break downs, etc.

And YET, the agent always started with "ok, let me see what kind of tables are out there", and essentially making up stuff as it went along.

So I started tweaking the prompt, and eventually arrived at something like "your job is to translate the users request into sql in the most direct way possible following the guides you have. If you are stuck, don't try to solve it, report back to the user so they can fix it."

At that point the agent started to use the canonical queries as instructed, and guess what, it turned out that the query had a bug! I can only assume that the agent realized that and tried to work around it to be helpful! After I found and fixed the bug, I left the phrasing in because for this specific use case, that's what I wanted.

I learned from this that "helpful" is note just a default friendly persona, it already implies a lot of behavior, and depending on what you want, it might not be the right thing.