Mastodon Feed: Post

Mastodon Feed

jonny@neuromatch.social ("jonny (nonvenomous)") wrote:

have you ever written a package that has obligate imports to a module that always fails import unless you've installed its deps independently, because the entire meat of the package is fucking no-op overrides of the thing you are trying to wrap written in the except block of the import statement, but then in the basic methods of the thing where you tried to dodge the deps you made optional there are also obligate imports of the optional dependency, so that your package no longer does anything like what it advertises that it does but it also fucking fails in any context where it would be used except for how every context where you test and deploy it yourself has the dependency?

i haven't, but that's because while i may have been an amateur at one point, by now i think about the things i'm writing and whether or not they make sense and if they don't fucking make any sense then i fix them.

[the main pypark_huggingface.huggingface moduel imports from pyspark_huggingface.datasource unconditionally, (and later uses .datasource.DataSource as a parent class of the main HuggingFaceDatasets class)] from typing import TYPE_CHECKING, Optional from pyspark_huggingface.compat.datasource import DataSource if TYPE_CHECKING: from pyspark_huggingface.compat.datasource import DataSourceWriter, DataSourceReader from pyspark.sql.types import StructType from pyspark_huggingface.huggingface_sink import HuggingFaceSink from pyspark_huggingface.huggingface_source import HuggingFaceSource class HuggingFaceDatasets(DataSource):
[pyspark, and pyspark[sql] are not required dependencies, so the entire .datasource module is written in the except block of the import to pyspark and it just makes everything no-ops.] from typing import TYPE_CHECKING, Iterator, List, Optional, Union import pyspark try: from pyspark.sql.datasource import DataSource, DataSourceArrowWriter, DataSourceReader, DataSourceWriter, InputPartition, WriterCommitMessage except ImportError: class DataSource: def init(self, options): self.options = options
[the get_source method of the main class imports unconditionally from pyspark_huggingface.huggingface_source] class HuggingFaceDatasets(DataSource): def get_source(self) -> "HuggingFaceSource": from pyspark_huggingface.huggingface_source import HuggingFaceSource if self.source is None: self.source = HuggingFaceSource(self.options.copy()) return self.source
[and pyspark_huggingface.huggingface_source imports unconditionally from pyspark.sql, which will always fail on a default install. AKA the basic usage of the package fails unless it's being used within their own environments which install its deps implicitly.] import ast from dataclasses import dataclass from typing import TYPE_CHECKING, Optional, Sequence from pyspark.sql.pandas.types import from_arrow_schema from pyspark.sql.types import StructType [IDE overlay: "module pyspark is not found"]