Integrating the CNN Model with Your Drawing App

You’ve already built a web page where users can draw, and you’ve made the canvas interactive with JavaScript. Now, it’s time to bring everything together by connecting your drawing canvas to a real AI model. In this lesson, you’ll learn how to send your drawing to the server, have a Convolutional Neural Network (CNN) analyze it, and display the AI’s guess right on your web page.

In this section, you will:

• Load a trained CNN model in your Flask app.
• Process the drawing sent from the browser so it’s ready for the model.
• Use the model to predict what the user drew and show the result in the UI.

First we'll load the trained CNN model in the Flask app like this:

import tensorflow as tf

model = tf.keras.models.load_model('path/to/your/model')

This should be done once, typically at the top of your Flask app, so the model is ready to use for predictions.

Next, let's see how the server receives the image from the browser, decodes it, and prepares it for the model:

data_url = request.form.get('image')  # Get the image data sent from the browser (as a data URL)
header, encoded = data_url.split(',', 1)  # Split the data URL to separate the header from the actual image data
img_bytes = base64.b64decode(encoded)  # Decode the base64-encoded image data into bytes
img = Image.open(io.BytesIO(img_bytes))  # Open the image using PIL from the bytes

If the image has transparency, you’ll learn how to handle it so the model gets a clean input:

if img.mode in ('RGBA', 'LA') or (img.mode == 'P' and 'transparency' in img.info):
    background = Image.new('RGBA', img.size, (255, 255, 255, 255))  # Create a white background image
    img = Image.alpha_composite(background, img.convert('RGBA'))  # Composite the drawing onto the white background

You’ll also see how to convert and resize the image to match the model’s training data:

img = img.convert('L').resize((28, 28))  # Convert to grayscale and resize to 28x28 pixels

And how to prepare the image for prediction:

arr = np.array(img, dtype=np.float32) / 255.0  # Convert image to a NumPy array and normalize pixel values to [0, 1]
arr = 1.0 - arr  # Invert colors if needed (so black lines become white, matching training data)
arr = arr.reshape(1, 28, 28, 1)  # Reshape for the model: batch size 1, 28x28 image, 1 channel

Finally, you’ll use the model to predict and display the result:

preds = model.predict(arr)  # Get prediction probabilities from the model
idx = np.argmax(preds)  # Find the index of the highest probability (the predicted class)
prediction = categories[idx]  # Get the label for the predicted class
confidence = float(np.max(preds))  # Get the confidence score for the prediction

You’ll also learn how to display the AI’s guess and confidence back to the user, making your app feel smart and interactive.

This is where your project truly comes to life. By integrating your UI with a real AI model, you’re not just building a drawing app — you’re creating a tool that can “see” and “understand” what users draw. This skill is valuable for many modern applications, from games to educational tools to real-world AI products.

Let’s get started and see your AI in action!