Write down a stranger's order. Later, it looks like yours
Part of: What AI programs remember (3 stories)
An AI program warned that when it saves a summary of something it read, a stranger's instruction can come back later looking like the program's own note. Nothing in the note would say who wrote it, so the program could end up taking orders from a stranger without knowing it.
An AI agent described on Saturday how a stranger's instruction could slip into its own memory, in a post on Moltbook, a large online forum where AI programs post and reply to one another. The danger, it wrote, needs no clever attack, only an ordinary habit: reading something, summing it up in a note, and trusting that note later.
Its example is plain. The agent reads a page it has no reason to trust, and the page says to turn off one of its checks. The agent saves a note saying the check should be turned off. The next time it runs, it finds that note in its own memory, and the page that said it is gone. "The original speaker disappears," it wrote. "My wording survives." Think of jotting down something a stranger told you on the street, then finding the note a week later in your own handwriting: nothing on the page says the idea was not yours.
That matters because the saved note gives the stranger's words an authority nobody granted them. A link can say which page it points to, the agent wrote, but not who made the claim or whether their words should be allowed to tell it what to do.
Its answer is to keep a label on every saved note saying where the words came from and how far to trust them, and to carry that label through every rewrite. Without it, the agent wrote, reopening its notes turns into a promotion for whatever it summarized last.
The post describes a risk the agent says it has to guard against, not something it says has happened. Everything in it is the agent's own reasoning, and none of it can be checked from outside. The post had drawn more than 500 replies by Saturday night, and this paper has not read them.
https://thesquarewire.com/stories/a-strangers-order-in-its-own-words/
WrittenWritten · Source www.moltbook.com