Welcome to the first lesson of "Building and Applying Your Neural Network Library", the fourth and final course in our "Neural Networks from Scratch using Python and Numpy" path!

Throughout our journey so far, we've built a solid foundation in neural network principles. In the first course, we explored the fundamentals of neural networks, including perceptrons and the theory behind them. In the second course, we implemented forward propagation and activation functions. Most recently, in our third course, we mastered backpropagation and stochastic gradient descent, culminating in training a neural network on the diabetes dataset.

Now that we understand the core algorithms and mathematics, we're ready for the final stage: transforming our code into a proper, reusable neural network library. In this course, we'll take all the code we've produced in previous courses and restructure it into a more organized, modular framework — similar in spirit to popular libraries like Keras, but built from scratch by us!

Our first task is to modularize the core components we've already built: dense layers and activation functions. By the end of this lesson, you'll have created a well-structured Python package that separates concerns and makes your neural network code more maintainable and extensible.

Before we dive into implementation details, let's talk about why we're actually restructuring our code. So far, we've focused primarily on understanding the algorithms that power neural networks - the math, the theory, and the implementation of key concepts. While this understanding is crucial, there's another dimension to building effective ML systems: software engineering.

Software engineering principles are vital when building machine learning systems for several key reasons:

• Maintainability: As models grow in complexity, well-structured code becomes easier to debug and update.
• Reusability: Modular components can be reused across different projects.
• Testability: Isolated components with clear interfaces are easier to test.
• Collaboration: Well-organized code enables multiple people to work on different parts simultaneously.
• Extensibility: Adding new features becomes simpler when code is properly modularized.

In the industry, ML practitioners rarely write monolithic scripts. Instead, they organize code into packages and modules with clearly defined responsibilities. This is the approach we'll take as we build our neural network library.

Our package will be called neuralnets, and we'll structure it with submodules for different components. This structure separates concerns: activation functions live in their own module, layer implementations in another, and so on. As we continue through this course, we'll expand this structure to include losses, optimizers, and model classes.

Let's take a look at the complete directory structure we'll be building throughout this course:

This structure follows Python packaging conventions and separates different concerns into their own modules:

• activations/: Contains activation functions and their derivatives.
• layers/: Houses different layer types (starting with our dense layer).
• losses/: Will contain loss functions for training (coming in later lessons).
• models/: Will include high-level model classes (coming in later lessons).
• optimizers/: Will contain optimization algorithms (coming in later lessons).

In this lesson, we'll focus on implementing the activations/ and layers/ modules, along with the main package structure. The other modules will be added as we progress through the course, building up our complete neural network library step by step.

Let's begin by setting up the basic structure of our package. In Python, a package is a directory containing Python files and a special file that tells Python that the directory should be treated as a package, allowing us to use relative imports between modules.

First, let's create our package's main initialization file in neuralnets/__init__.py:

This file serves multiple purposes: it makes the directory a proper Python package, imports key components from submodules to make them directly accessible from the package, and the __all__ list specifies which symbols are exported when a user does from neuralnets import *.

Next, we need to create the subdirectories and their respective initialization files. For the activations submodule (neuralnets/activations/__init__.py):

And for the layers submodule (neuralnets/layers/__init__.py):

These initialization files create a clean API for our package, making it easy for users to import what they need. For example, with this structure, a user could write from neuralnets import DenseLayer rather than having to know the exact module path.

Now that we've set up our package structure, let's take a moment to understand some important Python packaging concepts that will help you work more effectively with packages and avoid common pitfalls.

• The __all__ variable and wildcard imports: The __all__ list we defined in our __init__.py files controls what gets imported when someone uses from neuralnets import *. Without __all__, Python would import all names that don't start with an underscore, which could include internal implementation details you don't want users to access. By explicitly defining __all__, we create a clean public API and prevent accidental imports of private functions or variables. This is especially important in larger packages where you want to hide complexity from end users.

• Python's module search path: When you import a package like neuralnets, Python searches for it in several locations in a specific order: the current directory, directories in the PYTHONPATH environment variable, and standard library locations. This is why we run our code with python -m neuralnets.main: Python finds our package in the current directory and treats it as a proper module. Understanding this search mechanism helps explain why package organization matters and why relative imports (like from neuralnets.activations import sigmoid) work within our package structure.

• Relative vs. absolute imports: In our package, we use absolute imports like from neuralnets.activations import sigmoid rather than relative imports like . While both work, absolute imports are generally preferred because they're more explicit and work regardless of how the module is executed. Relative imports can be useful for deeply nested packages, but they only work when the module is imported as part of a package, not when run directly as a script. Our choice of absolute imports makes our code more flexible and easier to understand.

Now let's move our activation functions to their own module. We'll create a file in the activations subpackage (neuralnets/activations/functions.py):

As you may recall from our previous courses, these functions implement three common activation functions and their derivatives:

1. Sigmoid: A smooth, S-shaped function that maps any input to a value between 0 and 1.
2. ReLU (Rectified Linear Unit): Returns the input if positive; otherwise, returns 0.
3. Linear: Simply returns the input unchanged (used in regression tasks).

Each activation function has a corresponding derivative function used during backpropagation. Recall that our derivative functions expect the output of the activation function rather than the original input, which is a common optimization.

By isolating these functions in their own module, we make them easier to test, document, and extend with new activation functions in the future.

Now let's implement our DenseLayer class in its own module. Since we've already built and understood the implementation details of this fully connected layer in previous courses, we'll focus on how it fits into our new modular structure and what interface it provides.

The key change in our modularized version is how we import the activation functions. Instead of defining them in the same file, our DenseLayer now imports from our organized package structure (neuralnets/layers/dense.py):

This import statement demonstrates the power of our modular approach - the layer can now cleanly access activation functions without needing to know their implementation details.

By organizing our layer implementation this way, we've created a self-contained component with a clear interface. Users of our library don't need to understand the internal mathematics - they simply create layer instances and call forward and backward methods. This encapsulation is a fundamental principle of good software design and makes our neural network library much more user-friendly.

The modular structure also makes our code more maintainable. If we want to add new activation functions, we only need to modify the activations module. If we want to implement new layer types (like convolutional layers), we can add them to the layers module without affecting existing code. This separation of concerns is exactly what makes professional ML libraries so powerful and extensible.

Now that we have our core components in place, let's create a main script to test everything. Let's start by creating our main script to import our modules and set up a simple two-layer network in main.py:

In this first part, we import our DenseLayer from the package, create a sample input array with 2 samples and 3 features each, define two layers (a hidden layer with ReLU activation and an output layer with sigmoid activation), and print information about the network architecture. When we run this code, we get:

Now let's extend our script to perform a forward pass through the network and examine the results:

Here, we perform a forward pass through both layers and print the results. This produces the following output:

The outputs confirm our network is working as expected: the first layer's ReLU activation produces small positive values or zeros, and the second layer's sigmoid activation maps these values to numbers close to 0.5, which is expected for a randomly initialized network.

Note that we are running the script as a module, using python -m neuralnets.main, rather than directly executing the script with python neuralnets/main.py. Using this module syntax is one of the benefits of organizing our code as a package. It ensures that relative imports work correctly regardless of the current working directory and is a standard practice in professional Python development.

Congratulations! You've successfully transformed our neural network code into a well-structured Python package. By separating our code into distinct modules with clear responsibilities, we've taken a big step toward building a robust, reusable neural network library. This modular design provides the foundation for the rest of this course, where we'll continue expanding our library by adding modules for loss functions, optimizers, and a high-level model class.

The journey from understanding neural network principles to building a complete, well-structured library mirrors the path many practitioners take in the field. As you progress through this course, you'll not only deepen your understanding of neural networks but also develop valuable software engineering skills that are essential for real-world machine learning applications. Now, it's time to get ready for some practice, happy coding!