actio_python_utils.spark_functions.load_db_to_dataframe

actio_python_utils.spark_functions.load_db_to_dataframe(self, pgpass_record=None, relation=None, query=None, load_config_options=None, **kwargs)[source]

Return a PySpark dataframe from either a relation or query

Parameters:

  • self (SparkSession) – The PySpark session to use

  • pgpass_record (Optional[PassEntry], default: None) – PostgreSQL login credentials

  • relation (Optional[str], default: None) – The database relation to load

  • query (Optional[str], default: None) – The database query to load

  • load_config_options (Optional[Iterable[tuple[str, str]]], default: None) – Any additonal config options to load data

  • **kwargs – Any additional named arguments

Return type:

DataFrame

Returns:

The dataframe requested