Mastering Code Optimization with Numpy and Pandas for Large Datasets

Introduction to Code Optimization

Greetings, scholars! As we progress and leverage insights from Python and its remarkable libraries, Numpy and Pandas, we embark on an important mission today - Optimization . This session is dedicated to learning the art of refining code to enhance computation efficiency and optimize memory usage — an essential requirement when working with large datasets. In Data Science, large datasets are the norm. Handling such volumes of data efficiently and optimum use of system resources is necessary. Code optimization is our key strategy in these situations. It aims to enhance two critical aspects: reducing computation time and improving memory utilization. With these skills, handling large-scale datasets becomes much smoother! Sit tight; we're about to journey through Python, Numpy, and Pandas, exploring the elements they offer for a smooth ride on the road to optimization.

Understanding the Need for Code Optimization

Can you imagine setting off to a neighborhood store by taking a long detour over the hills? It seems incredible, right? That's precisely what inefficient code does. It solves problems using longer, convoluted routes, squandering valuable resources while accomplishing the bare minimum. Here's where understanding algorithmic complexity or Big-O notation becomes significant. Consider algorithmic complexity as a measure of your algorithm's efficiency relative to the input size. Time Complexity and Space Complexity, the two aspects governing this efficiency, dictate how the time is taken for execution and memory usage changes with the input size. A thorough understanding of these can be a game-changer when dealing with large volumes of data. Consider finding a book in a library, for instance. If you have no idea where the book is located, you might scan each aisle until you find it. Although simple, this approach incurs a time complexity of O(n) . However, if the books are sorted, and you follow the binary search strategy, repeatedly halving your search space until you find the book, you'd have a more efficient time complexity of O(log n) . Quite an impressive time-saver!

Memory Management in Python

Python's memory management is governed by the language’s built-in garbage collector . The garbage collector tracks all your objects and discards the ones no longer required, freeing the memory occupied by this unnecessary data. Suppose you assigned substantial data to a variable that isn't needed later. Python's garbage collector would free up the memory preemptively occupied by that variable's data. While Python's garbage collector works effectively, handling large datasets can pose challenges. Consider variables that handle sizable data chunks. If not managed properly, they can consume major memory, even when redundant, leading to a 'Memory Leak.' Consequently, learning to efficiently manage your memory resources by releasing unneeded bulky variables or adopting memory-efficient data types becomes crucial.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import numpy as np
import time

# Define a large array
large_array = np.random.rand(10**6)

# Numpy way
start = time.time()
print("Numpy sum:", np.sum(large_array))  # This calculates the sum using Numpy's vectorized function
print("Time to calculate the sum in a Numpy list:", time.time() - start)
# Prints "Time to calculate the sum in a Numpy list: 0.00037097930908203125"


# Python way of summing elements in an array
start = time.time()
print("Built-in list sum", sum(large_array))  # This calculates the sum using Python's built-in function
print("Time to calculate the sum in a Python list:", time.time() - start)
# Prints "Time to calculate the sum in a Python list: 0.0417783260345459"

# Note: Timing may vary between runs due to system load, CPU throttling, and other environmental factors

import numpy as np
import time

# Define a large array
large_array = np.random.rand(10**6)

# Numpy way
start = time.time()
print("Numpy sum:", np.sum(large_array))  # This calculates the sum using Numpy's vectorized function
print("Time to calculate the sum in a Numpy list:", time.time() - start)
# Prints "Time to calculate the sum in a Numpy list: 0.00037097930908203125"


# Python way of summing elements in an array
start = time.time()
print("Built-in list sum", sum(large_array))  # This calculates the sum using Python's built-in function
print("Time to calculate the sum in a Python list:", time.time() - start)
# Prints "Time to calculate the sum in a Python list: 0.0417783260345459"

# Note: Timing may vary between runs due to system load, CPU throttling, and other environmental factors

Python

df['Type'] = pd.Categorical(df['Type'])
df['MedInc'] = df['MedInc'].astype('category')

df['Type'] = pd.Categorical(df['Type'])
df['MedInc'] = df['MedInc'].astype('category')

Python

# Downcast data type for 'AveBedrms' column
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['Population'] = df['Population'].astype('int32')

# Downcast data type for 'AveBedrms' column
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['Population'] = df['Population'].astype('int32')

Python

# Regular way
df_copy = df[df['Population'] > 1000]
df_copy.dropna(inplace=True)

# Optimized way
df[df['Population'] > 1000].dropna(inplace=True)

# Regular way
df_copy = df[df['Population'] > 1000]
df_copy.dropna(inplace=True)

# Optimized way
df[df['Population'] > 1000].dropna(inplace=True)

Python

import pandas as pd
from sklearn import datasets
import numpy as np

# Load the California Housing dataset
california = datasets.fetch_california_housing()
df = pd.DataFrame(data=np.c_[california['data'], california['target']], columns=california['feature_names'] + ['target'])

def memory_usage_pandas(df):
    bytes = df.memory_usage(deep=True).sum()
    return bytes / 1024**2  # Convert bytes to megabytes

original_memory = memory_usage_pandas(df)

# Optimize memory usage in Pandas using categorical data types
# California Housing dataset does not have any Categorical features, so we will use downcasting
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['AveRooms'] = pd.to_numeric(df['AveRooms'], downcast='float')
optimized_memory = memory_usage_pandas(df)

print(f'Original memory usage: {original_memory} MB')
print(f'Optimized memory usage: {optimized_memory} MB')
print(f'Memory saved: {original_memory - optimized_memory} MB')

import pandas as pd
from sklearn import datasets
import numpy as np

# Load the California Housing dataset
california = datasets.fetch_california_housing()
df = pd.DataFrame(data=np.c_[california['data'], california['target']], columns=california['feature_names'] + ['target'])

def memory_usage_pandas(df):
    bytes = df.memory_usage(deep=True).sum()
    return bytes / 1024**2  # Convert bytes to megabytes

original_memory = memory_usage_pandas(df)

# Optimize memory usage in Pandas using categorical data types
# California Housing dataset does not have any Categorical features, so we will use downcasting
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['AveRooms'] = pd.to_numeric(df['AveRooms'], downcast='float')
optimized_memory = memory_usage_pandas(df)

print(f'Original memory usage: {original_memory} MB')
print(f'Optimized memory usage: {optimized_memory} MB')
print(f'Memory saved: {original_memory - optimized_memory} MB')

Python

import numpy as np
import time

# Define a large array
large_array = np.random.rand(10**6)

# Numpy way
start = time.time()
print("Numpy sum:", np.sum(large_array))  # This calculates the sum using Numpy's vectorized function
print("Time to calculate the sum in a Numpy list:", time.time() - start)
# Prints "Time to calculate the sum in a Numpy list: 0.00037097930908203125"


# Python way of summing elements in an array
start = time.time()
print("Built-in list sum", sum(large_array))  # This calculates the sum using Python's built-in function
print("Time to calculate the sum in a Python list:", time.time() - start)
# Prints "Time to calculate the sum in a Python list: 0.0417783260345459"

# Note: Timing may vary between runs due to system load, CPU throttling, and other environmental factors

Python

df['Type'] = pd.Categorical(df['Type'])
df['MedInc'] = df['MedInc'].astype('category')

Python

# Downcast data type for 'AveBedrms' column
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['Population'] = df['Population'].astype('int32')

Python

# Regular way
df_copy = df[df['Population'] > 1000]
df_copy.dropna(inplace=True)

# Optimized way
df[df['Population'] > 1000].dropna(inplace=True)

Python

import pandas as pd
from sklearn import datasets
import numpy as np

# Load the California Housing dataset
california = datasets.fetch_california_housing()
df = pd.DataFrame(data=np.c_[california['data'], california['target']], columns=california['feature_names'] + ['target'])

def memory_usage_pandas(df):
    bytes = df.memory_usage(deep=True).sum()
    return bytes / 1024**2  # Convert bytes to megabytes

original_memory = memory_usage_pandas(df)

# Optimize memory usage in Pandas using categorical data types
# California Housing dataset does not have any Categorical features, so we will use downcasting
df['AveBedrms'] = pd.to_numeric(df['AveBedrms'], downcast='float')
df['AveRooms'] = pd.to_numeric(df['AveRooms'], downcast='float')
optimized_memory = memory_usage_pandas(df)

print(f'Original memory usage: {original_memory} MB')
print(f'Optimized memory usage: {optimized_memory} MB')
print(f'Memory saved: {original_memory - optimized_memory} MB')