Mastodon Feed: Post

Mastodon Feed

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

look yall i don't mean to make things personal but like the person who seems to have written the code that was responsible for this vuln likely makes 10x my salary and from what i can tell their main contribution is like... using someone else's library to implement deduplicated file upload for huggingface. motherfucker i can write 800 lines of python to "change everything" by calling someone else's lib. this package is almost literally nothing, just literally a set of python adapter classes that call into pyspark. if what it takes to get a bazillion dollars is to do this but say you're changing the world, i'd do that no problem.

If i may borrow the canonical AI/crowdfunding/infomercial phrasing of "it's not just x, it's x+y" - "AI" is not only a cult, it's a cult of mediocrity.

Let me explain why Hugging Face Datasets storage is faster than S3 + why today's release changes everything 🧵
Hugging Face uses Xet: a dedupe-based storage which enables fast deduped uploads. Unlike traditional remote storage, uploads are faster on Xet because duplicate data is only uploaded once. For example: if some or all of the data already exists in other files on Xet, it is not uploaded again, saving bandwidth and speeding up uploads. Deduplication for Parquet is enabled through Content Defined Chunking (CDC). Thanks to Parquet CDC and Xet deduplication, saving a dataset on Hugging Face is faster than on any traditional remote storage. This is especially efficient for appending rows or columns, and also for insert/delete opetations (provided they don't change every single block of data) Today I released V2 of the pyspark_huggingface package, a Spark Data Source to read/write HF datasets. This release is huge: it includes Parquet CDC and Xet support. Accelerating dataset saving on HF dramatically
[cloc output, the pyspark_huggingface package is 800 lines of python, full output follows. I actually don't know the best way to represent tabular data for screenreaders, so if you are a screenreader user and have some spare time to link me to things, hmu] 15 text files. 11 unique files. 5 files ignored. github.com/AlDanial/cloc v 2.10 T=0.02 s (448.1 files/s, 81561.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 7 239 152 798 Markdown 1 41 0 127 YAML 1 7 0 35 TOML 1 3 0 31 Jupyter Notebook 1 0 540 29 ------------------------------------------------------------------------------- SUM: 11 290 692 1020 -------------------------------------------------------------------------------