Mastodon Feed
brib@bribstodon.xyz ("brib :neofox_floof: :Nonbinary:") wrote:
@glyph Yeah I get your pain.
Deciphering LLM security reports back when a new vuln was publicised each week was a world of pain in terms of the report's readability, this seems to be somewhat better (although I still spend a lot of time trying to piece together "why did it say that there?). Seeing the LLM-isms also makes me wonder how thoroughly it was human-checked; what sort of issues sneaked through because LLMs are good at hitting the LGTM part of our brains? But you mention it looks well-reviewed, and perhaps some people are better at reviewing LLM output than I am; my ADHD makes it better to do it all manually even when autocompleting something as simple as a function (I say with experience!).
That's not to say the report is necessarily inaccurate; but I guess I wish the writer would use their own words rather than relying so heavily on an LLM's, even if LLMs did the bulk of the analysis. The reasoning behind wanting the writer's own words is that it forces the writer to actually process the LLM output with their brain rather than risking a LGTM moment, and it saves the end user from parsing slop. If I was receiving an issue like this, I would probably encourage the writer to do this unless it was a super-urgent security thing.
When you're not in a position to make that sort of request...? I guess you can't un-see LLM content so if it's needed you might as well use the information. If it's not needed then I'd probably ignore it (and I generally lean towards muting people who post obvious LLM output, although that's usually in the context of Mastodon accounts rather than technical write-ups)