Mastodon Feed
Reblogged by jsonstein@masto.deoan.org ("Jeff Sonstein"):
simon@simonwillison.net ("Simon Willison") wrote:
New release of my llm-mlc plugin for https://llm.datasette.io/ - this plugin allows you to run large language models like Llama 2 using the MLC Python library, which means you get GPU acceleration on M1/M2 Macs! https://github.com/simonw/llm-mlc/releases/tag/0.4
Attachments:
- [The llm mlc download-model command now takes zero or more optional -a/--alias options to configure aliases for the model once it has been installed. #4:
llm mlc download-model Llama-2-7b-chat --alias llama2
Installation instructions are clearer, and show how to install required dependencies first. #6
The plugin no longer crashes llm if it cannot find the dist/prebuilt folder. #9
New options for temperature, top\_p and repetition\_penalty: #7
llm -m Llama-2-7b-chat \\
-o temperature 0.5 \\
-o top\_p 0.9 \\
-o repetition\_penalty 0.9 \\
'five names for a cute pet ferret'][7] ([remote][8])