Tuning Tree Depth to Prevent Overfitting in Decision Trees

Before we dive into tuning tree depth, let’s quickly review what a decision tree is. A decision tree is a type of machine learning model that makes predictions by learning a series of simple decision rules from the data. You can think of it like a flowchart: at each step (or “node”), the tree asks a question about one of the features (for example, “Is the fruit red?”). Based on the answer, it follows a branch to the next question, and so on, until it reaches a final decision at a “leaf” node (for example, “This is an apple”).

Decision trees are popular because they are easy to understand and interpret. Each path from the root to a leaf represents a sequence of decisions that leads to a prediction. However, if the tree is allowed to grow too deep, it can become very complex and start to memorize the training data instead of learning general patterns. That’s why controlling the depth of the tree is important, which is what we’ll focus on in this lesson.

Welcome back! In the last lesson, you learned how to control the complexity of a logistic regression model using L2 regularization and the C parameter. You saw how tuning this parameter helps you find the right balance between underfitting and overfitting. In this lesson, we will look at a different type of model — decision trees — and see how you can control their complexity using a parameter called tree depth.

To help you visualize, here’s a simple example of a decision tree that predicts whether a fruit is an apple or not:

         [Color = Red?]
           /         \
        Yes           No
      [Weight > 150g?]   Not Apple
       /        \
    Yes         No
  Apple      Not Apple

In this tree, the model first checks if the fruit is red. If yes, it checks if the weight is greater than 150g. Based on the answers, it predicts whether the fruit is an apple or not. This tree has a depth of 2, since the longest path from the root to a leaf has two decisions.

Tree depth is a simple but powerful way to manage how much a decision tree can learn from your data. Just like with regularization in logistic regression, setting the right tree depth is important for achieving good performance. If the tree is too simple, it might miss important patterns. If it is too complex, it might memorize the training data and not work well on new data. By the end of this lesson, you will know how to train decision trees with different depths and how to spot when your model is underfitting or overfitting.