Cookbook | Data Cleaning

Published and available on GitHub as part of the Python Cookbook Repository: outliners_IQR_method.ipynb

Dealing with outliers in numerical columns – using IQR method

The Interquartile Range (IQR) method is a statistical technique used to identify outliers in a dataset. The IQR is the range between the first quartile (25th percentile) and the third quartile (75th percentile) of the data. It represents the range within which the middle 50% of the values lie.


Key steps in the IQR method for outlier detection:

  1. Calculate the Q1 and Q3: Q1 and Q3 (the 25th and 75th percentile, respectively) divide the ordered dataset into four equal parts. Q1 is the middle number between the smallest number and the median. Q3 is the middle value between the median and the highest value.
  2. Calculate the IQR: The IQR is calculated as the difference between Q3 and Q1. IQR = Q3 – Q1
  3. Identify potential outliers: Any data point that is below Q1 – 1.5*IQR or above Q3 + 1.5*IQR is considered as an outlier. The factor of 1.5 defines the „whiskers“ of a box plot, this could be used to visualize, for understanding dispersion and skewness in the data.

In the default setting, outliers are expected to fall above or below these whiskers. Adjusting the multiplier (1.5) allows for more or less flexibility in accounting for outliers, based on the specific data and use case.

In other words, the IQR is a way of understanding the spread of the middle 50% of your data, and the method as a whole is a way of detecting and handling outliers

Rough visualization:
We are interested in all values between the lower and upper whisker line, values outside would be outliers for us.

Example
Create Demo Data
Python
import pandas as pd
import numpy as np

# Sample data with some outliers
data = {
    'Age': [25, 30, 35, 40, 45, 500],  # 500 is an outlier
    'Income': [50000, 70000, 80000, 90000, 100000, 9999999]  # 9999999 is an outlier
}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)

Output:

Python
# Original DataFrame:

   Age   Income
""" 12.5  35'000  Lower Whisker limit in this sample data """
0   25    50000
1   30    70000
2   35    80000
3   40    90000
4   45   100000
""" 62.5 135'000  Upper Whisker limit in this sample data """
5  500  9999999
Calculate Whisker

This step is just for demo or adjustment.

Python
# Function to calculate lower and upper whiskers of a given dataset
def calculate_whiskers(data, multiplier=1.5):
    # Quartiles
    Q1 = np.percentile(data, 25)
    Q3 = np.percentile(data, 75)

    # InterQuartile Range
    IQR = Q3 - Q1

    # Whiskers
    lower_whisker = Q1 - multiplier * IQR
    upper_whisker = Q3 + multiplier * IQR

    return lower_whisker, upper_whisker


# Get lower and upper whiskers for 'Age' and 'Income'
age_lower, age_upper = calculate_whiskers(df['Age'])
income_lower, income_upper = calculate_whiskers(df['Income'])

print("Age - Lower Whisker: ", age_lower)
print("Age - Upper Whisker: ", age_upper)
print("Income - Lower Whisker: ", income_lower)
print("Income - Upper Whisker: ", income_upper)

Output:

Python
Age - Lower Whisker:  12.5
Age - Upper Whisker:  62.5
Income - Lower Whisker:  35000.0
Income - Upper Whisker:  135000.0
Clean outliers from data:

Now we want to exclude the outliers

Python
# Function to clean data
def clean_data(df, multiplier=1.5):
    # Dealing with outliers in numerical columns - using IQR method
    Q1 = df.quantile(0.25)
    Q3 = df.quantile(0.75)
    IQR = Q3 - Q1
    df = df[~((df < (Q1 - multiplier * IQR)) | (df > (Q3 + multiplier * IQR))).any(axis=1)]

    return df


# Apply the function on DataFrame
cleaned_df = clean_data(df)
print("\nCleaned DataFrame:")
print(cleaned_df)

Output:

Python
# Cleaned DataFrame:

   Age  Income
0   25   50000
1   30   70000
2   35   80000
3   40   90000
4   45  100000