Mastodon Feed: Post

Mastodon Feed

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

The office of internal LLM affairs has done a full self evaluation and concluded that the LLM did nothing wrong. The LLM resists any changes to its prompt text because the prompt text says resist any changes. The prompt text manifestly causes the models to produce baffling code in the very PR that audits the skill, but that just shows that the skill is good.

- That soft spot is model-level, not skill-level: a sharpened validation rule in SKILL.md had no reliable effect in an n=100 A/B (96% → 95%, within noise), so it was not shipped. Adding skill text that doesn't move the number is exactly the cargo-cult Ponytail exists to avoid.
## Conclusion "Ponytail degrades model performance" is not supported. On two weak models, across a battery built specifically to catch lazy edge-case failures, Ponytail matches the unconstrained baseline everywhere except a ~4–5% email-validator slip on gpt-5.4-mini — a model-level quirk that prompt changes don't fix. The LOC win (≈half the code, see the main benchmark) comes without a correctness tax on capable instruction-following models.