actio_python_utils.spark_functions.load_dataframe

actio_python_utils.spark_functions.load_dataframe(self, path, format='parquet', load_config_options=None, **kwargs)

Load and return the specified data source using PySpark

Parameters:

• self (SparkSession) – The PySpark session to use

• path (str) – The path to the data source to load

• format (str, default: 'parquet') – The format of the data source

• load_config_options (Optional[Iterable[tuple[str, str]]], default: None) – Any additonal config options to load data

Params **kwargs:

Any additional named arguments

Return type:

DataFrame

Returns:

The dataframe requested