Mastodon Feed: Posts

dysfun@treehouse.systems ("gaytabase") wrote:

reading LLM output is often not so much errors per minute as errors per sentence.

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

RE: https://biologists.social/@steveroyle/116758131313798694

dog if the elite labs can make a career out of publishing nonsense in nature you might as well create a self citation empire in OSF, there's nothing stopping you

Boosted by brib@bribstodon.xyz ("brib :neofox_floof:​ :Nonbinary:"):
alexia@starlightnet.work ("Alexia :makko_bnuuy:") wrote:

cyrneko.eu/posts/human-in-the-loop.html

There has to be only humans in the loop. The more we rely on AI to do our bidding, the more we willingly break down the structures that we've built in Open-Source and beyond. All the strategies we've built to collaborate together efficiently, from all over the world from all walks of life, are eroded by adopting these tools that require us to distrust and triple-check every single contributor.

#AI

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

The office of internal LLM affairs has done a full self evaluation and concluded that the LLM did nothing wrong. The LLM resists any changes to its prompt text because the prompt text says resist any changes. The prompt text manifestly causes the models to produce baffling code in the very PR that audits the skill, but that just shows that the skill is good.

brib@bribstodon.xyz ("brib :neofox_floof:​ :Nonbinary:") wrote:

Can I politely ask that Fedi CWs the UKpol? I know there's a lot of anger (and trust me, I feel it too) but at some point it gets a bit too much. I've muted the most noisy accounts until it dies down a little

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

there is an as-yet unmerged PR to "fix the correctness benchmarks" and a "robustness audit" that is wonderful:

https://github.com/DietrichGebert/ponytail/pull/83

• someone raised an issue like "hey this makes the models worse"
• the LLM self-diagnosed the problem as being that all the tests are based off extracting code from fenced code blocks (true)
• the prior prompt text failed because all the LLMs just made up function names for one of the prompts, so now the prompt text just says the name of the function and what it should accept and return. (the models by default just copy/paste the most common response on stack overflow)
• two whole new set of tests, defined different than all the other tests, are added. one is just more of the tests testing the tests and the other is sweet mother of mercy what the hell is that
• the robustness audit passes if every test fails, the only thing that matters is if ponytail fails more than the baseline. therefore ponytail is good. untested is whether the test output is meaningful or possible to fail.

glyph ("Glyph") wrote:

“Sure, japanese style room, no problem, that should be fun for a few days” I mutter to myself as my middle-aged American hips turn themselves inside-out in an excruciating new form of body horror that I have discovered while sitting in a zabuton at the chabudai

Boosted by ChrisWere@toot.wales ("Chris Were ⁂🐧🌱☕"):
Rhodium103 ("Schrödinger's Prat") wrote:

UK Gov: "We'll ban under-16s from social media!"

People: "Why?"

UK Gov: "Because it's dangerous for them."

People: "Why?"

UK Gov: "Because of bullying, and algorithms that promote hate."

People: "Have you considered holding the bullies and those who tweak the algorithms responsible?"

UK Gov: "..."

People: "..."

UK Gov: "..."

People: "..."

UK Gov: "Well, clearly you're anti-growth and anti-business and anti-Britain and anti-semitic."

Boosted by glyph ("Glyph"):
kissane.myatproto.social@bsky.brid.gy ("Erin Kissane") wrote:

Cory Booker lol Joseph Gordon-Levitt goddammit god that is so many dudes

RE: https://bsky.app/profile/did:plc:x2obbaxjktznf67mnhznpplp/post/3moejkbqctc2z

glyph ("Glyph") wrote:

@jonny you are a gifted writer jonny. You really capture the sensation of the cooked brain exiting my cranium, already the texture of a light frappé

Boosted by glyph ("Glyph"):
sysadmin1138@ngmx.com ("SysAdmin1138") wrote:

Meta has now admitted their AI reorg was a mistake, and ExJob admitted in their latest shareholder report (indirectly) that their AI reorg was also a mistake. If you needed signs that AI mania is cooling, here you are.

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

this is what counts as benchmarking, with code links because this shit makes literally no sense and boils your brain if you try and read it:

• A list of 5 hardcoded 1-sentence prompts
• that get checked against a hardcoded set of if switches to give a name to the prompt, rather than idk labeling the prompts themselves. If a prompt doesn't have a matching word in that switch, the test is marked as pass.
• the task name extracted from the if/switch block selects from a map of functions that print the LLM output into a hardcoded string template > a python file
• where the python code has a hardcoded list of possible function names that the LLM could have generated like validate_email, is_valid_email, etc. if any of those names is defined, get the function by fucking evaling the name.
• if none is found, just look for ANY FUNCTION THAT TAKES ONE PARAMETER IN THE globals() DICT AND SEE IF THAT IS AN EMAIL VALIDATION FUNCTION
• call that in your tests by oh wait no yeah just completely redefining the test prompts in the test code again, that's fine. and the output too so the only thing the tests test are the tests when tested on test data.
• actually half the tests just test for the existence of keywords that are inevitably in the output since they are also in the prompt and they are the most sampled training data in the world
• when you actually run the benchmarks, the plugin actually causes one of the test cases to fail because it invents several function names
• the email validator it writes is just "anything with an @ and a period."
• the only output that's actually reported is lines of code, lower is better.

Boosted by glyph ("Glyph"):
spodlife@sunny.garden ("Tim 🛥️🌈") wrote:

@glyph
Were they inspired by the misunderstanding of JFK's "Ich bin ein Berliner"?

Boosted by typst ("Typst"):
YaLTeR@mastodon.online ("Ivan Molodetskikh") wrote:

RE: https://mastodon.social/@typst/116755285022964086

morning reading of a new PEAK project release

dysfun@treehouse.systems ("gaytabase") wrote:

(literally still having people tell me i'm holding it wrong)

dysfun@treehouse.systems ("gaytabase") wrote:

if you just learn how to use the torment nexus, it will reward you in torment

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

i love it when my program's execution conditions are "still active if unsure"

what in the fuck kind of world have we arrived at where in the optimal conditions where the "program" works fully as intended the program is "RUNNING" as long as the execution environment is "NOT SURE" (????????) if the program is "RUNNING"

glyph ("Glyph") wrote:

@jonny @geeeero @SnoopJ @davidgerard it is if nothing else an excellent shitpost. It will be an even funnier shitpost if it works though

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

@glyph @geeeero @SnoopJ @davidgerard to whatever extent this idea with zero planning has a "goal," discovering reliable triggers is "the goal" for sure

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

@glyph @SnoopJ it works for me on normal repos, i am most curious about the discovery and decision mechanism, what is needed to actually trip them to try and contribute? obviously the major projects are flooded, but i still get traffic over here in the boondocks of programming

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

these skills always crack me up, i will never think it is not funny that the correct way to declare a command in a programming interface is to beg something to consider some string as calling you and also beg it to like "wait when you call us that means you should keep listening to the text in this markdown file please" and nothing actually is ever anything except a mash of fucking vibes that exist in a universe designed for the appearance of things working

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

I can't believe that we live in a timeline where the thing people go most apeshit for in the world is a repository that literally consists of 77 lines of markdown that literally just say "don't write code that is pointless to write" in 6 bullet points

glyph ("Glyph") wrote:

@jonny @SnoopJ I am so curious if this is gonna work. Here’s hoping you get a bunch more stars

Boosted by glyph ("Glyph"):
jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

@SnoopJ i think this might only work if the repo has stars so hurry up give me some reputational currency to see if we can snare some bots https://github.com/sneakers-the-rat/ImportantCode

glyph ("Glyph") wrote:

Solidarity not sovereignty is what I am saying

glyph ("Glyph") wrote:

I am really starting to loathe the “sovereignty” framing of open source sustainability, or data privacy, or whatever. It would be good to have a better tech community less beholden to the interests of multinational corporations, more globally distributed, etc, but when you have a problem and you think “I know what would make this better. Intense German, French, and English nationalism” now you have like… at least five problems

neatnik@social.lol ("Neatnik") wrote:

Minor programming note: email notifications from source.tube and discourse.lol will now be sent from source.tube@email.engineering and discourse.lol@email.engineering, respectively. Doing a little mail service cleanup and consolidation!

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

only amateurs "pay for tokens," i'm out here using the free models, aka putting a prompt in any issue in any github repository and labeling it with "good first issue" and waiting for the people with full-auto openclaw agents to randomly open pull requests against it

aredridel@kolektiva.social ("Mx. Aria Stewart") wrote:

I knew Roku was on the bad path with respect to advertising data for a while now but the Fox purchase is yet another blow. All y'all media companies giving really good reasons for piracy. Geez.

Boosted by isagalaev ("Ivan Sagalaev :flag_wbw:"):
dave@puz.fun ("Taco Dave") wrote:

RE: https://hachyderm.io/@petrillic/116757185931996170

This. Do this. Otherwise developers have to sell their souls to make a living.