Understand How Spark divides Jobs into Stages and Tasks? — Knowing what happens when we send a job or application in Spark for execution is really important. We usually ignore it because we don’t realize how crucial it is. But if we want to get why it matters, DAGs and Explain plans can help us figure it out. Today we…

Understand how to write unit test cases for PySpark using PyTest module. — We all write code to develop data applications. To make sure the code is working as expected at ground zero level, we need to write Unit Test Cases. This article will help you start with Unit Test Cases in PySpark. We are going to use PyTest Python module to create…

Understand How Parquet files can be compacted efficiently utilizing RLE (Run Length Encoding) — Parquet files are one of the most popular choice for data storage in Data & Analytics world for various reasons. Few of them lists as: Write once read Many paradigm Columnar storage Preserve Schema Optimization with Encoding etc. Today we will understand how efficiently we can utilize the default encodings…

Understand how Spark estimates the number of Partitions required to read a file — Spark reads file in partitions and each partition is processed to reach the desired result. But, more often we are confused over how the number of partitions is estimated and divided ? Today we will deep dive inside the logic that Spark uses for this calculation. But before we can…

We all have been in scenario, where we have to deal with huge file sizes with limited compute or resources. More often we need to find ways to optimize such file read/processing to make our data pipelines efficient. Today we are going to discuss about one such configuration of Spark…

Optimizations are all around us, few of them are hidden in small parameter changes and few in the way we deal with data. Today, we will understand The effects of Multiline in files with Apache Spark. To explore the effect, we will read same Orders JSON file with approximate 10,50,000…

It is very important to understand the core of data distribution for Apache Spark. Very often we don’t realize the importance and make mistakes without even knowing. One of such mistake is to use row_number() for generating Surrogate Keys for Dimensions. Surrogate Keys are basically synthetic/artificial keys that are generated…

We often run into situations where we have to join two Spark Data Frames on multiple conditions and those conditions can be complex and may change as per requirement. We will work on a simple hack that will make our join conditions way much more effective and simpler to use. …

In this article we are going to design the Batch Loading Strategy to load data in our Data Lakehouse for the series Data Lakehouse with PySpark. This is a well-known and re-usable strategy which can be implemented as a part any Data Warehouse or Data Lakehouse project. If you are…