Mastodon Feed: Post

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

this is what counts as benchmarking, with code links because this shit makes literally no sense and boils your brain if you try and read it:

• A list of 5 hardcoded 1-sentence prompts
• that get checked against a hardcoded set of if switches to give a name to the prompt, rather than idk labeling the prompts themselves. If a prompt doesn't have a matching word in that switch, the test is marked as pass.
• the task name extracted from the if/switch block selects from a map of functions that print the LLM output into a hardcoded string template > a python file
• where the python code has a hardcoded list of possible function names that the LLM could have generated like validate_email, is_valid_email, etc. if any of those names is defined, get the function by fucking evaling the name.
• if none is found, just look for ANY FUNCTION THAT TAKES ONE PARAMETER IN THE globals() DICT AND SEE IF THAT IS AN EMAIL VALIDATION FUNCTION
• call that in your tests by oh wait no yeah just completely redefining the test prompts in the test code again, that's fine. and the output too so the only thing the tests test are the tests when tested on test data.
• actually half the tests just test for the existence of keywords that are inevitably in the output since they are also in the prompt and they are the most sampled training data in the world
• when you actually run the benchmarks, the plugin actually causes one of the test cases to fail because it invents several function names
• the email validator it writes is just "anything with an @ and a period."
• the only output that's actually reported is lines of code, lower is better.