actio_python_utils.spark_functions.convert_dicts_to_dataframe

actio_python_utils.spark_functions.convert_dicts_to_dataframe(self, dict_list=None, iter_func=None, coerce_to_lists_if_needed=True)[source]

Converts either a list of dicts (dict_list) or a function that returns an iterator of dicts (iter_func) to a PySpark dataframe

Parameters:

  • self (SparkSession) – The SparkSession to use

  • dict_list (Optional[Iterable[Mapping[str, Any]]], default: None) – A list of dicts representing rows

  • iter_func (Optional[Callable[[], Iterable[Mapping[str, Any]]]], default: None) – A function that returns an iterator of dicts representing rows

  • coerce_to_lists_if_needed (bool, default: True) – For any column to create, check if any value in a row is a list, and if so, convert any non-lists in the column to a list

Return type:

DataFrame

Returns:

A new dataframe built from the provided rows of dicts