Mastodon Feed
jonny@neuromatch.social ("jonny (nonvenomous)") wrote:
there is an as-yet unmerged PR to "fix the correctness benchmarks" and a "robustness audit" that is wonderful:
https://github.com/DietrichGebert/ponytail/pull/83
- someone raised an issue like "hey this makes the models worse"
- the LLM self-diagnosed the problem as being that all the tests are based off extracting code from fenced code blocks (true)
- the prior prompt text failed because all the LLMs just made up function names for one of the prompts, so now the prompt text just says the name of the function and what it should accept and return. (the models by default just copy/paste the most common response on stack overflow)
- two whole new set of tests, defined different than all the other tests, are added. one is just more of the tests testing the tests and the other is sweet mother of mercy what the hell is that
- the robustness audit passes if every test fails, the only thing that matters is if ponytail fails more than the baseline. therefore ponytail is good. untested is whether the test output is meaningful or possible to fail.