If you are new to data warehousing or preparing for an Interview, then this Series is for you. This Series follow YouTube videos explaining the Basics of Data Warehouse in simple terms. Make sure to Like and Subscribe to our channel. Click on the links below to follow: Introduction to…

Understand How Parquet files can be compacted efficiently utilizing RLE (Run Length Encoding) — Parquet files are one of the most popular choice for data storage in Data & Analytics world for various reasons. Few of them lists as: Write once read Many paradigm Columnar storage Preserve Schema Optimization with Encoding etc. Today we will understand how efficiently we can utilize the default encodings…

Understand how Spark estimates the number of Partitions required to read a file — Spark reads file in partitions and each partition is processed to reach the desired result. But, more often we are confused over how the number of partitions is estimated and divided ? Today we will deep dive inside the logic that Spark uses for this calculation. But before we can…

We all have been in scenario, where we have to deal with huge file sizes with limited compute or resources. More often we need to find ways to optimize such file read/processing to make our data pipelines efficient. Today we are going to discuss about one such configuration of Spark…

Optimizations are all around us, few of them are hidden in small parameter changes and few in the way we deal with data. Today, we will understand The effects of Multiline in files with Apache Spark. To explore the effect, we will read same Orders JSON file with approximate 10,50,000…

It is very important to understand the core of data distribution for Apache Spark. Very often we don’t realize the importance and make mistakes without even knowing. One of such mistake is to use row_number() for generating Surrogate Keys for Dimensions. Surrogate Keys are basically synthetic/artificial keys that are generated…

We often run into situations where we have to join two Spark Data Frames on multiple conditions and those conditions can be complex and may change as per requirement. We will work on a simple hack that will make our join conditions way much more effective and simpler to use. …

In this article we are going to design the Batch Loading Strategy to load data in our Data Lakehouse for the series Data Lakehouse with PySpark. This is a well-known and re-usable strategy which can be implemented as a part any Data Warehouse or Data Lakehouse project. If you are…

Next, as part of the series Data Lakehouse with PySpark, we need to setup boto3 and Delta Lake to communicate with AWS S3. This will help us to create our default warehouse location for Delta Lake on AWS S3. We will also setup the metastore location for Delta Lake. To…