Mastodon Feed: Post

Mastodon Feed

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

i think it's probably really good that your security measures are to do this amazing monkeypatch of your own code

Top is some "PatchedParquetWriter" that forces the kwarg "embed_local_files" to be false, even though all the intervening code that would construct this class is owned by huggingface. A contextmanager disallow_embed_local_files creates some ExitStack() that handles arbitrary stacks of contextmanagers and then monkeypatches one place that the ParquetWriter is used (within the datasets.builder module) for the duration of the contextmanager. [python code follows ] class _PatchedParquetWriter(ParquetWriter): # type: ignore[misc] def init( self, *args: Any, use_content_defined_chunking: bool = True, write_page_index: bool = True, **kwargs: Any ) -> None: kwargs["embed_local_files"] = False super().init( *args, use_content_defined_chunking=use_content_defined_chunking, write_page_index=write_page_index, **kwargs, ) class disallow_embed_local_files: def init(self) -> None: self.exit_stack = ExitStack() def enter(self) -> None: self.exit_stack.enter_context(patch("datasets.builder.ParquetWriter", _PatchedParquetWriter)) def exit( self, exc_type: Optional[type[BaseException]], exc_value: Optional[BaseException], traceback: Optional[TracebackType], ) -> None: return self.exit_stack.close()