Mastering Row Manipulation in Pandas DataFrame

During today's session, we will delve into how to add and remove rows from a DataFrame in Pandas. These are vital tools for data manipulation, whether adding new entries or eliminating unnecessary data.

Consider it analogous to adding a name to your contacts or deleting an item from your shopping list. We will be carrying out similar operations but with a DataFrame. Let's begin:

import pandas as pd

A DataFrame, a central data structure in Pandas, is a tool for storing data in table form. Each row contains values correlated to an individual entry in our data. For instance, each row of a grocery list might represent a unique grocery item.

Each row features an index, a unique identifier. Now, let's create a DataFrame:

import pandas as pd

data = {
    'Grocery Item': ['Apples', 'Oranges', 'Bananas', 'Grapes'],
    'Price per kg': [3.25, 4.50, 2.75, 5.00]
}

grocery_df = pd.DataFrame(data)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
3       Grapes          5.00
'''

Multiple scenarios might necessitate adding new entries to our DataFrame. Let's explore how to accomplish that:

In modern pandas, we use pd.concat() function to incorporate new rows. If you forgot to add 'Pears' to your grocery list, here’s how to do it:

new_row = pd.DataFrame({'Grocery Item': ['Pears'], 'Price per kg': [4.00]})

grocery_df = pd.concat([grocery_df, new_row]).reset_index(drop=True)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
3       Grapes          5.00
4        Pears          4.00
'''

Setting reset_index(drop=True) resets the index to default integers. Without this step, pandas will save the original dataframes' indices, resulting in both 'Pears' and 'Apples' sharing the same index 0.

For multiple rows, you can concatenate them by creating a DataFrame and adding it to the original one:

new_rows = pd.DataFrame({
    'Grocery Item': ['Avocados', 'Blueberries'],
    'Price per kg': [2.5, 10.0]
})

grocery_df = pd.concat([grocery_df, new_rows]).reset_index(drop=True)

print(grocery_df)
'''Output:
  Grocery Item  Price per kg
0       Apples          3.25
1      Oranges          4.50
2      Bananas          2.75
3       Grapes          5.00
4        Pears          4.00
5     Avocados          2.50
6  Blueberries         10.00
'''

You may wonder why we don't include these rows in the original dataframe. Well, it is only sometimes possible. Imagine we have two separate grocery lists coming from different sources, for instance, from separate files. In this case, the only way to combine them into one is to use pd.concat()