Mastodon Feed
Boosted by pixxl:
david_chisnall@infosec.exchange ("David Chisnall (*Now with 50% more sarcasm!*)") wrote:
I have a lot of sympathy with the lawyer charged with contempt in this story. He used an LLM to summarise a proceedings and it made up nonsense.
Most of us know that this is a risk, but I routinely talk to people who think an ‘AI summary’ of a document is going to be a distillation of information in the information in the source text, rather than some statistically probable sequence of words that would follow a request to summarise the source text. And that’s entirely because that’s precisely how these things are advertised.
EDIT: To be clear, I have zero sympathy with a lawyer who uses LLMs to research precedent and doesn’t check the results. But this case (at least, as reported) was using a tool sold as a summarisation tool to summarise a case. Yes, he probably should have checked the result, but the tool is explicitly sold as being capable of doing this. If he’d used a spreadsheet to calculate damages would you expect him to have separately checked the calculation by hand? No, because spreadsheets are sold as being able to do this and actually do. 90% of the blame lies with the people who sell these tools and misrepresent their capabilities.
EDIT2: Yes, I know you know not to trust an LLM-based summary. You are very smart. Now go and talk to some folks who don’t have Fediverse accounts. You will find lots of intelligent and successful people in various professions who expect these tools to work because trillion-dollar companies are spending billions of dollars of marketing saying that they do. Yes there are (as the judge said) above-the-fold stories about hallucinations but those are typically about search-like use cases. There are lots of papers about how LLMs are bad at summarisation but not many mainstream articles and the ones that exist are drowned out by marketing.