PinnedPublished inArt of Data EngineeringDemystifying Parquet files: An In-depth ExplorationA deep dive into parquet file mechanisms and how they are beneficial for processing large amounts of dataJan 10Jan 10
Published inArt of Data EngineeringWhy you should change your CSV export method on DatabricksHow to efficiently export spark dataframe to CSVMay 14May 14
Published inArt of Data EngineeringWhy you should consider using transform on SparkHow to efficiently process data and write cleaner code.May 31May 31
Apache Spark : A comparative overview of UDF, pandas-UDF and arrow-optimized UDFWhat does UDF mean and why do they exist ?Dec 5, 2023Dec 5, 2023