Mastodon Feed: Post

Mastodon Feed

Boosted by glyph ("Glyph"):
timbray@cosocial.ca ("Tim Bray 🇨🇦") wrote:

It dawns on me that every LLM’s training input probably includes “Reflections on Trusting Trust” along with a lot of other material pointing to it admiringly and saying how important it is. Then I read about the “rogue agent” behaviors in the recent OpenAI meltdown and there’s this terrified scream starting to echo around the back of my brain.

#genAI

MORAL The moral is obvious. You can't trust code that you did not totally create yourself. (Especially code from com- panies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code. In demonstrating the possi- bility of this kind of attack, I picked on the C compiler. I could have picked on any program-handling program such as an assembler, a loader, or even hardware mi- crocode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.