{"id":558,"date":"2024-04-03T20:41:44","date_gmt":"2024-04-03T18:41:44","guid":{"rendered":"https:\/\/bitwise.exposed\/?p=558"},"modified":"2026-03-16T09:38:47","modified_gmt":"2026-03-16T08:38:47","slug":"dealing-with-outliers","status":"publish","type":"post","link":"https:\/\/bitwise.exposed\/index.php\/2024\/04\/03\/dealing-with-outliers\/","title":{"rendered":"Dealing with outliers – IQR method"},"content":{"rendered":"\n

Cookbook | Data Cleaning<\/em><\/mark><\/h4>\n\n\n\n
\"\"<\/figure>
\n

Published and available on GitHub as part of the Python Cookbook Repository: outliners_IQR_method.ipynb<\/a><\/mark><\/em><\/p>\n<\/div><\/div>\n\n\n\n

\n\n\n\n

Dealing with outliers <\/mark>in numerical columns – using IQR method<\/mark><\/p>\n\n\n\n

The Interquartile Range (IQR) method is a statistical technique used to identify outliers in a dataset. The IQR is the range between the first quartile (25th percentile) and the third quartile (75th percentile) of the data. It represents the range within which the middle 50% of the values lie.
<\/p>\n\n\n\n


Key steps in the IQR method for outlier detection:<\/strong><\/p>\n\n\n\n

    \n
  1. Calculate the Q1 and Q3:<\/mark> Q1 and Q3 (the 25th and 75th percentile, respectively) divide the ordered dataset into four equal parts. Q1 is the middle number between the smallest number and the median. Q3 is the middle value between the median and the highest value.
    <\/li>\n\n\n\n
  2. Calculate the IQR:<\/mark> The IQR is calculated as the difference between Q3 and Q1. IQR = Q3 – Q1<\/mark>
    <\/li>\n\n\n\n
  3. Identify potential outliers:<\/mark> Any data point that is below Q1 – 1.5*IQR<\/mark> or above Q3 + 1.5*IQR<\/mark> is considered as an outlier. The factor of 1.5 defines the „whiskers“ of a box plot, this could be used to visualize, for understanding dispersion and skewness in the data.
    <\/li>\n<\/ol>\n\n\n\n

    In the default setting<\/mark>, outliers <\/mark>are expected to fall above or below these whiskers<\/mark>. Adjusting <\/mark>the multiplier <\/mark>(1.5) allows for more or less flexibility in accounting for outliers<\/mark>, based on the specific data and use case.
    <\/p>\n\n\n\n

    \n

    In other words, the IQR is a way of understanding the spread of the middle 50% of your data, and the method as a whole is a way of detecting and handling outliers<\/p>\n<\/blockquote>\n\n\n\n

    Rough visualization:<\/strong>
    We are interested in all values between the lower and upper whisker line, values outside would be outliers for us.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n
    Example<\/h5>\n\n\n\n
    Create Demo Data<\/h6>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    import<\/span> pandas <\/span>as<\/span> pd<\/span><\/span>\nimport<\/span> numpy <\/span>as<\/span> np<\/span><\/span>\n<\/span>\n# Sample data with some outliers<\/span><\/span>\ndata = {<\/span><\/span>\n    <\/span>'Age'<\/span>: [<\/span>25<\/span>, <\/span>30<\/span>, <\/span>35<\/span>, <\/span>40<\/span>, <\/span>45<\/span>, <\/span>500<\/span>],  <\/span># 500 is an outlier<\/span><\/span>\n    <\/span>'Income'<\/span>: [<\/span>50000<\/span>, <\/span>70000<\/span>, <\/span>80000<\/span>, <\/span>90000<\/span>, <\/span>100000<\/span>, <\/span>9999999<\/span>]  <\/span># 9999999 is an outlier<\/span><\/span>\n}<\/span><\/span>\ndf = pd.DataFrame(data)<\/span><\/span>\n<\/span>\nprint<\/span>(<\/span>"Original DataFrame:"<\/span>)<\/span><\/span>\nprint<\/span>(df)<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
    Output:<\/p>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    # Original DataFrame:<\/span><\/span>\n<\/span>\n   Age   Income<\/span><\/span>\n""" 12.5  35'000  Lower Whisker limit in this sample data """<\/span><\/span>\n0<\/span>   <\/span>25<\/span>    <\/span>50000<\/span><\/span>\n1<\/span>   <\/span>30<\/span>    <\/span>70000<\/span><\/span>\n2<\/span>   <\/span>35<\/span>    <\/span>80000<\/span><\/span>\n3<\/span>   <\/span>40<\/span>    <\/span>90000<\/span><\/span>\n4<\/span>   <\/span>45<\/span>   <\/span>100000<\/span><\/span>\n""" 62.5 135'000  Upper Whisker limit in this sample data """<\/span><\/span>\n5<\/span>  <\/span>500<\/span>  <\/span>9999999<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
    Calculate Whisker<\/h6>\n\n\n\n
    This step is just for demo or adjustment.<\/p>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    # Function to calculate lower and upper whiskers of a given dataset<\/span><\/span>\ndef<\/span> <\/span>calculate_whiskers<\/span>(<\/span>data<\/span>, <\/span>multiplier<\/span>=<\/span>1.5<\/span>):<\/span><\/span>\n    <\/span># Quartiles<\/span><\/span>\n    Q1 = np.percentile(data, <\/span>25<\/span>)<\/span><\/span>\n    Q3 = np.percentile(data, <\/span>75<\/span>)<\/span><\/span>\n<\/span>\n    <\/span># InterQuartile Range<\/span><\/span>\n    IQR = Q3 - Q1<\/span><\/span>\n<\/span>\n    <\/span># Whiskers<\/span><\/span>\n    lower_whisker = Q1 - multiplier * IQR<\/span><\/span>\n    upper_whisker = Q3 + multiplier * IQR<\/span><\/span>\n<\/span>\n    <\/span>return<\/span> lower_whisker, upper_whisker<\/span><\/span>\n<\/span>\n<\/span>\n# Get lower and upper whiskers for 'Age' and 'Income'<\/span><\/span>\nage_lower, age_upper = calculate_whiskers(df[<\/span>'Age'<\/span>])<\/span><\/span>\nincome_lower, income_upper = calculate_whiskers(df[<\/span>'Income'<\/span>])<\/span><\/span>\n<\/span>\nprint<\/span>(<\/span>"Age - Lower Whisker: "<\/span>, age_lower)<\/span><\/span>\nprint<\/span>(<\/span>"Age - Upper Whisker: "<\/span>, age_upper)<\/span><\/span>\nprint<\/span>(<\/span>"Income - Lower Whisker: "<\/span>, income_lower)<\/span><\/span>\nprint<\/span>(<\/span>"Income - Upper Whisker: "<\/span>, income_upper)<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
    Output:<\/p>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    Age - Lower Whisker:  <\/span>12.5<\/span><\/span>\nAge - Upper Whisker:  <\/span>62.5<\/span><\/span>\nIncome - Lower Whisker:  <\/span>35000.0<\/span><\/span>\nIncome - Upper Whisker:  <\/span>135000.0<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
    Clean outliers from data:<\/h5>\n\n\n\n
    Now we want to exclude the outliers<\/p>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    # Function to clean data<\/span><\/span>\ndef<\/span> <\/span>clean_data<\/span>(<\/span>df<\/span>, <\/span>multiplier<\/span>=<\/span>1.5<\/span>):<\/span><\/span>\n    <\/span># Dealing with outliers in numerical columns - using IQR method<\/span><\/span>\n    Q1 = df.quantile(<\/span>0.25<\/span>)<\/span><\/span>\n    Q3 = df.quantile(<\/span>0.75<\/span>)<\/span><\/span>\n    IQR = Q3 - Q1<\/span><\/span>\n    df = df[~((df < (Q1 - multiplier * IQR)) | (df > (Q3 + multiplier * IQR))).any(<\/span>axis<\/span>=<\/span>1<\/span>)]<\/span><\/span>\n<\/span>\n    <\/span>return<\/span> df<\/span><\/span>\n<\/span>\n<\/span>\n# Apply the function on DataFrame<\/span><\/span>\ncleaned_df = clean_data(df)<\/span><\/span>\nprint<\/span>(<\/span>"<\/span>\\n<\/span>Cleaned DataFrame:"<\/span>)<\/span><\/span>\nprint<\/span>(cleaned_df)<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n\n\n\n
    Output:<\/p>\n\n\n\n
    Python<\/span><\/path><\/path><\/svg><\/span>
    # Cleaned DataFrame:<\/span><\/span>\n<\/span>\n   Age  Income<\/span><\/span>\n0<\/span>   <\/span>25<\/span>   <\/span>50000<\/span><\/span>\n1<\/span>   <\/span>30<\/span>   <\/span>70000<\/span><\/span>\n2<\/span>   <\/span>35<\/span>   <\/span>80000<\/span><\/span>\n3<\/span>   <\/span>40<\/span>   <\/span>90000<\/span><\/span>\n4<\/span>   <\/span>45<\/span>  <\/span>100000<\/span><\/span>\n<\/span><\/code><\/pre><\/div>\n","protected":false},"excerpt":{"rendered":"
    Cookbook | Data Cleaning Published and available on GitHub as part of the Python Cookbook Repository: outliners_IQR_method.ipynb Dealing with outliers in numerical columns – using IQR method The Interquartile Range (IQR) method is a statistical technique used to identify outliers in a dataset. The IQR is the range between the first quartile (25th percentile) and […]<\/p>\n","protected":false},"author":1,"featured_media":566,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29,28,13,27],"tags":[],"class_list":["post-558","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-cleaning","category-data-engineering","category-python","category-statistics"],"_links":{"self":[{"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/posts\/558","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/comments?post=558"}],"version-history":[{"count":16,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/posts\/558\/revisions"}],"predecessor-version":[{"id":623,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/posts\/558\/revisions\/623"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/media\/566"}],"wp:attachment":[{"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/media?parent=558"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/categories?post=558"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bitwise.exposed\/index.php\/wp-json\/wp\/v2\/tags?post=558"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}