Mastodon Feed: Post

Mastodon Feed

Boosted by baldur@toot.cafe ("Baldur Bjarnason"):
dingemansemark@scholar.social ("Mark Dingemanse") wrote:

Indeed. As @olivia often says, guardrails are a scam. Model providers can play catch-up with stopgap measures but fundamentally these systems are open, and just as water seeks its own level, prompts find their way

Perhaps of interest, I go into some detail about guardrails and preference tuning (RLHF) in a recent paper of mine
https://doi.org/10.5281/zenodo.19452871

Preference tuning is often sold by model providers as contributing to model “guardrails”, a reassuringly solid metaphor for what is in fact at best a hopeful collection of probabilistic, error-prone and context-dependent techniques to curb model output (Ciston 2026). As it turns out, wrapping a language model in a chat interface and tuning its output so that it comes across as maximally helpful, harmless and human-like has some severe side-effect
More generally, the engineered overconfidence of these models makes them a well-known vector for misinformation in domains that range from tourism (Christensen et al. 2025) to mental health (Ittefaq et al. 2026). It is important to understand that with large language models, there is no bright line that separates an exchange that yields passable output from one that points a tourist to the wrong destination, or one that pushes a teenager with suicidal ideation over the edge. All are, from the point of view of the system, travels in latent space shaped probabilistically by training data and fine-tuning regimes; none can be deterministically pinpointed or prevented. The purpose of a system is what it does (Beer 2002). The whole range of outcomes can be directly related to the interactional parlor tricks played on users through preference tuning.