Optimization with Constraints Using SciPy

Welcome to the final lesson of the course: Optimization with Constraints. Building on your knowledge of function optimization using SciPy, we're now going to tackle optimization challenges that involve constraints, which are crucial for many real-world applications. Consider scenarios where resource limitations must be honored — this is where constrained optimization shines. By the end of this lesson, you'll be equipped to solve such problems using SciPy, enhancing your ability to model and solve complex problems.

Before we dive into the code, let's revisit the concept of constraints. Constraints in optimization are conditions that the solution must satisfy. These can be of two types: inequality constraints, which specify that a certain expression should be greater than or equal to zero, and equality constraints, which specify that a certain expression should be exactly zero.

Let's use the following constraint for an example:

$x + y <= 1$

This constraint means that we are limited to the region under the $y = 1 - x$ line:

We will search for a solution only in the grey region, including the black line itself.

In SciPy, we define constraints using a dictionary format. Here's how you define an inequality constraint where the sum of variables x[0] and x[1] is less than or equal to 1:

# Inequality constraint: x[0] + x[1] <= 1
constraint = {
    'type': 'ineq',  # 'ineq' for inequality
    'fun': lambda x: 1 - (x[0] + x[1])
}

In this constraint dictionary, 'type' signifies the kind of constraint ('ineq' for inequality), while 'fun' provides the function definition for the constraint. The lambda function returns a positive value when x[0] + x[1] is less than or equal to 1, which ensures the constraint is met.

Now that we understand constraints, let's formulate and solve a constrained optimization problem using SciPy. Below is a complete code snippet that includes defining the objective function, setting an initial guess, defining constraints, and solving the problem using SciPy's minimize function.

import numpy as np
from scipy.optimize import minimize

# 1. Objective Function
def f(x):
    return 2*x[0]**2 - 5*x[0] + 3*x[1]**2

# 2. Initial Guess
initial_guess = [1, 1]

# 3. Define Constraints
constraint = {
    'type': 'ineq',
    'fun': lambda x: 1 - (x[0] + x[1])
}

# 4. Solving with SciPy's `minimize`
result_constraints = minimize(f, initial_guess, constraints=[constraint])

# Output the result
print("Optimal solution:", result_constraints.x)  # [ 1.1 -0.1]
print("Objective function value at optimal solution:", result_constraints.fun)  # -3.05

Explanation:

1. Objective Function: The function f(x) is defined to be minimized. It is a quadratic function of two variables.

2. Initial Guess: We start the optimization process with an initial guess of [1, 1].

3. Define Constraints: The constraint x[0] + x[1] <= 1 is defined using a dictionary format, specifying it as an inequality constraint.

4. Solving with SciPy's minimize: The minimize function is used to find the values of x that minimize the objective function while satisfying the constraints. The result is stored in result_constraints, and the optimal solution and its corresponding function value are printed.

To better comprehend the optimization results and the role of constraints, visualization is key. We'll use Matplotlib to create a contour plot of the function, showing the feasible region and the optimal solution.

1. Contour Plotting

   Define a grid of x and y values and plot the corresponding function values as contours.

   Python

   import numpy as np
   import matplotlib.pyplot as plt
   
   x = np.linspace(-4, 4, 400)
   y = np.linspace(-4, 4, 400)
   X, Y = np.meshgrid(x, y)
   Z = f([X, Y])
   
   plt.figure(figsize=(8, 6))
   contour = plt.contour(X, Y, Z, levels=50, cmap='viridis')
   plt.colorbar(contour)

2. Plotting Constraint

   Plot the constraint line x + y = 1 and shade the feasible region where solutions are valid.

   Python

   plt.plot(x, 1 - x, 'k-', label='Constraint: x + y = 1')
   plt.fill_between(x, 1 - x, -2, color='gray', alpha=0.3)

3. Highlighting Optimal Solution

   Mark the optimal solution on the plot with a red point for visibility.

   Python

   plt.plot(result_constraints.x[0], result_constraints.x[1], 'ro', label='Optimal solution')

4. Final Touches

   Add labels, a title, and a legend to make the plot informative.

   Python

   plt.xlabel('x')
   plt.ylabel('y')
   plt.xlim((-2, 2))
   plt.ylim((-2, 2))
   plt.title('Contour Plot with Constraint and Optimal Solution')
   plt.legend()
   plt.grid(True)
   plt.show()

The visualization helps you see how the optimization respects constraints and pinpoints the optimal solution.

Here is the output plot:

It shows the target function in the form of a contour plot. The constraint is represented as a black line with grey coloring of the target region. The optimal solution is a red dot. Note how this dot is close to the global function's minimum but is within the specified bounds.

To gain a deeper understanding of the optimization landscape, we can visualize the function and constraints in three dimensions. This 3D plot will provide a more comprehensive view of how the objective function behaves and how the constraints influence the feasible region.

We'll use Matplotlib's 3D plotting capabilities to create a surface plot of the objective function.

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')

# Create a surface plot
ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.8)

# Plot the constraint line in 3D
ax.plot(x, 1 - x, f([x, 1 - x]), 'k-', label='Constraint: x + y = 1')

# Highlight the optimal solution
ax.scatter(result_constraints.x[0], result_constraints.x[1], result_constraints.fun, color='r', s=100, label='Optimal solution')

# Set labels and title
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('f(x, y)')
ax.set_title('3D Surface Plot with Constraint and Optimal Solution')
ax.legend()
plt.show()

Here is the result:

The 3D plot provides a visual representation of the objective function as a surface. The constraint is shown as a line on this surface, and the optimal solution is marked with a red dot. This visualization helps in understanding the spatial relationship between the function's surface and the constraint, offering insights into how the optimal solution is determined within the feasible region.

Congratulations on completing the lesson! You've learned about defining objective functions, handling multivariable optimizations, and incorporating constraints using SciPy. The powerful techniques you've gained will serve you well in tackling a wide range of optimization problems. Prepare to apply these skills in practice exercises that will reinforce and test your knowledge. Well done on reaching this milestone!