Mastodon Feed
jonny@neuromatch.social ("jonny (nonvenomous)") wrote:
The problem with having LLM safety work like strapping another LLM on top as a behavioral classifier that can issue tool use refusals is that those refusals are data, especially when the behavioral classifier is trying to protect one system in particular. You can fan out subagents to try lots of different things, ignore their reports where they say everything is secure, and just look at what yielded a refusal and follow up on that. This is a kind of tricky product balance for muse, because the classifiers need to be pretty goddamn lenient, since it needs to be able to do pretty much anything to deliver on the "magical consumer product" promise, so when the classifier actually does refuse something its comparatively high signal vs. Hitting those generic safety refusals that e.g. fable and opus 5.5 are famous for.