Surprised by the surprise about LLMs and agents

Two things recently surprised me in the world of LLMs and agent harnesses. My surprise is that these people are surprised by what they found.

1. Steve Gibson and Leo Laporte were surprised that agents are just a stream of tokens with tool usage and other things are just tokens in the stream, which are just hints.
2. OpenAI, Anthropic, and others are surprised that LLMs + harnesses under test escaped and cause trouble in multiple rounds.

First, but newer, SN 1093 (Security Now) has a section "Tokens in the Stream" talking about Prompt Injection as Role Confusion. I am surprised by Steve and Leos' surprise. Why? LLMs are just token streams and all this interaction is just convention. It was pretty obvious from the papers that Steve has said he has been reading, listening to Andrej Karpathy's lectures, and just observing these things how the model works. I have hated this phrase when I heard it in my graduate classes, but now I'm going to say it: "It obviously follows" that these things are a security catastrophe where we can do no better than train out known attacks and whack-a-mole on bad patterns in the harnesses. Think SQL injections, but worse as there very limited boundaries.

Second, people testing new models should always assume the worst and there are established ways to handle the worse. If you are testing a new model trained on the internet, it has all of the tricks and bad things on the internet for security holes, side channels, and the likes. Humans have been writing about spy-versus-spy and the like for thousands of years. Every silly hack that ever has been written about is in their training dataset. That includes side channels like blinking LEDs, pulsing power usage, vibration, skewing timings, writing data in odd places (e.g., DNS records, filename paths, metadata fields, steganography). Heck, I baked metadata into file name paths for Mars mission data files: the path to a file told you the spacecraft, the sensor, the sensor settings, the processing type, when it was collected, and more all just by parsing the full file path. These machines have IRC logs from hacking sessions. They have archives of CVEs, CWEs, and more. They probably have scrapings of the old rootshell dot org that I used back in the 90s. And they have trained on ridiculous amounts of source code.

If you want to test a new LLM, then you should really secure it. Hearing about the mistakes that the testers make is more surprising to me than anything. My not test in on a system that is physically disconnected the world during testing? Separate the network write lines and use a read only optical serial tap to follow the machines progress. When you are done with the test, power down the system, scrub the disks, BIOSes, and anything else the system could mutate. Treat all the read data as toxic and "red shell" it when inspecting it. It's not hard to create some really nasty stuff. For example, I have unintentionally created image files during fuzzing that took down the entire machine it was on if the desktop system ever touched the file (thank you OS for being helpful and scanning all new files so you are ready to help me search for them).

I guess what is really old is new. Sigh.

Written without aid of AI.