Basic Data Cleaning with the Diamonds Dataset

Introduction to Data Cleaning

Hello! In this lesson, we will dive into the basic concepts of data cleaning using the Diamonds dataset from the seaborn library. Data cleaning is a crucial step in data preprocessing, ensuring that our data is ready for analysis by dealing with inconsistencies, errors, and missing values. Data cleaning involves identifying and handling missing values, correcting errors, and ensuring consistency. By cleaning your data, you improve the quality of your analysis and the performance of machine learning models.

Quick Recap: Loading and Exploring

Let's quickly revisit how to load the dataset, explore its structure, and identify missing values. First, load the Diamonds dataset using the seaborn library: Pythonimport seaborn as sns import pandas as pd # Load the diamonds dataset diamonds = sns.load_dataset('diamonds')import seaborn as sns import pandas as pd # Load the diamonds dataset diamonds = sns.load_dataset('diamonds') View the first few rows to get an initial overview: Pythonprint(diamonds.head())print(diamonds.head()) Output: plaintext carat cut color clarity depth table price x y z 0 0.23 Ideal E SI2 61.5 55.0 326 3.95 3.98 2.43 1 0.21 Premium E SI1 59.8 61.0 326 3.89 3.84 2.31 2 0.23 Good E VS1 56.9 65.0 327 4.05 4.07 2.31 3 0.29 Premium I VS2 62.4 58.0 334 4.20 4.23 2.63 4 0.31 Good J SI2 63.3 58.0 335 4.34 4.35 2.75 carat cut color clarity depth table price x y z 0 0.23 Ideal E SI2 61.5 55.0 326 3.95 3.98 2.43 1 0.21 Premium E SI1 59.8 61.0 326 3.89 3.84 2.31 2 0.23 Good E VS1 56.9 65.0 327 4.05 4.07 2.31 3 0.29 Premium I VS2 62.4 58.0 334 4.20 4.23 2.63 4 0.31 Good J SI2 63.3 58.0 335 4.34 4.35 2.75 You can access a column using either diamonds['cut'] or diamonds.get('cut'). Both will return the 'cut' column, but get is safer as it does not raise a KeyError if the column is missing. Pythonprint(diamonds['cut'].head()) # Or print(diamonds.get('cut').head())print(diamonds['cut'].head()) # Or print(diamonds.get('cut').head()) Output: plaintext0 Ideal 1 Premium 2 Good 3 Premium 4 Good Name: cut, dtype: category Categories (5, object): ['Ideal', 'Premium', 'Very Good', 'Good', 'Fair']0 Ideal 1 Premium 2 Good 3 Premium 4 Good Name: cut, dtype: category Categories (5, object): ['Ideal', 'Premium', 'Very Good', 'Good', 'Fair'] Check the dimensions and basic statistics: Pythonprint(diamonds.shape) print(diamonds.describe())print(diamonds.shape) print(diamonds.describe()) Output: plaintext(53940, 10) carat depth table price x \ count 53940.000000 53940.000000 53940.000000 53940.000000 53940.000000 mean 0.797940 61.749405 57.457184 3932.799722 5.731157 std 0.474011 1.432621 2.234491 3989.439738 1.121761 min 0.200000 43.000000 43.000000 326.000000 0.000000 25% 0.400000 61.000000 56.000000 950.000000 4.710000 50% 0.700000 61.800000 57.000000 2401.000000 5.700000 75% 1.040000 62.500000 59.000000 5324.250000 6.540000 max 5.010000 79.000000 95.000000 18823.000000 10.740000 y z count 53940.000000 53940.000000 mean 5.734526 3.538734 std 1.142135 0.705699 min 0.000000 0.000000 25% 4.720000 2.910000 50% 5.710000 3.530000 75% 6.540000 4.040000 max 58.900000 31.800000 (53940, 10) carat depth table price x \ count 53940.000000 53940.000000 53940.000000 53940.000000 53940.000000 mean 0.797940 61.749405 57.457184 3932.799722 5.731157 std 0.474011 1.432621 2.234491 3989.439738 1.121761 min 0.200000 43.000000 43.000000 326.000000 0.000000 25% 0.400000 61.000000 56.000000 950.000000 4.710000 50% 0.700000 61.800000 57.000000 2401.000000 5.700000 75% 1.040000 62.500000 59.000000 5324.250000 6.540000 max 5.010000 79.000000 95.000000 18823.000000 10.740000 y z count 53940.000000 53940.000000 mean 5.734526 3.538734 std 1.142135 0.705699 min 0.000000 0.000000 25% 4.720000 2.910000 50% 5.710000 3.530000 75% 6.540000 4.040000 max 58.900000 31.800000

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import seaborn as sns
import pandas as pd

# Load the diamonds dataset
diamonds = sns.load_dataset('diamonds')

Python

print(diamonds.head())

plaintext

   carat      cut color clarity  depth  table  price     x     y     z
0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43
1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31
2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31
3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63
4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75

Python

print(diamonds['cut'].head()) # Or print(diamonds.get('cut').head())

plaintext

0      Ideal
1    Premium
2       Good
3    Premium
4       Good
Name: cut, dtype: category
Categories (5, object): ['Ideal', 'Premium', 'Very Good', 'Good', 'Fair']

Python

print(diamonds.shape)
print(diamonds.describe())

plaintext

(53940, 10)
              carat         depth         table         price             x  \
count  53940.000000  53940.000000  53940.000000  53940.000000  53940.000000   
mean       0.797940     61.749405     57.457184   3932.799722      5.731157   
std        0.474011      1.432621      2.234491   3989.439738      1.121761   
min        0.200000     43.000000     43.000000    326.000000      0.000000   
25%        0.400000     61.000000     56.000000    950.000000      4.710000   
50%        0.700000     61.800000     57.000000   2401.000000      5.700000   
75%        1.040000     62.500000     59.000000   5324.250000      6.540000   
max        5.010000     79.000000     95.000000  18823.000000     10.740000   

                  y             z  
count  53940.000000  53940.000000  
mean       5.734526      3.538734  
std        1.142135      0.705699  
min        0.000000      0.000000  
25%        4.720000      2.910000  
50%        5.710000      3.530000  
75%        6.540000      4.040000  
max       58.900000     31.800000

Python

print(diamonds.isnull().sum())

print(diamonds.isnull().sum())

plaintext

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

Python

diamonds.loc[0, 'cut'] = None
print(diamonds.isnull().sum())

diamonds.loc[0, 'cut'] = None
print(diamonds.isnull().sum())

text

carat      0
cut        1
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

carat      0
cut        1
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

Python

diamonds_cleaned = diamonds.dropna()
print(diamonds_cleaned.shape)

diamonds_cleaned = diamonds.dropna()
print(diamonds_cleaned.shape)

Python

print(diamonds_cleaned.isnull().sum())

print(diamonds_cleaned.isnull().sum())

text

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

Python

print(diamonds.isnull().sum())

plaintext

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

Python

diamonds.loc[0, 'cut'] = None
print(diamonds.isnull().sum())

text

carat      0
cut        1
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64

Python

diamonds_cleaned = diamonds.dropna()
print(diamonds_cleaned.shape)

Python

print(diamonds_cleaned.isnull().sum())

text

carat      0
cut        0
color      0
clarity    0
depth      0
table      0
price      0
x          0
y          0
z          0
dtype: int64