Decision Trees and Random Forests

A decision tree is a supervised learning algorithm that models decisions and their possible consequences as a tree-like structure. It is used for both classification (predicting a discrete class) and regression (predicting a continuous value). The tree consists of: - Root node: the topmost node that contains the entire dataset. - Internal nodes: nodes that test a specific feature and split the data into branches based on the outcome. - Branches: the outcomes of a test (e.g., feature <= threshold, or categorical value). - Leaf nodes: terminal nodes that output a prediction (class label or numeric value).

The algorithm recursively partitions the feature space into regions, each associated with a simple model (e.g., majority class). The goal is to create subsets that are as homogeneous as possible with respect to the target variable.

The tree is built using a greedy, top-down, recursive partitioning algorithm. At each node, the algorithm selects the best feature and split point that maximizes the separation of the target variable. The "best" split is determined by a criterion such as: - Gini impurity: measures the probability of misclassifying a randomly chosen element if it were labeled according to the class distribution in the node. Lower Gini is better. Formula: Gini = 1 - Σ(p_i)^2, where p_i is the proportion of class i in the node. - Entropy / Information gain: entropy measures disorder; information gain is the reduction in entropy after a split. Higher information gain is better. Entropy = -Σ(p_i * log2(p_i)). - Mean squared error (MSE): for regression trees, the split that minimizes the total MSE of the two child nodes.

The algorithm stops splitting when a stopping criterion is met, such as:

In Azure Machine Learning, the Decision Tree module (or the underlying scikit-learn implementation) exposes these critical hyperparameters: - Maximum depth (max_depth): the maximum number of levels from root to leaf. Default is None (unlimited), but typical values range from 3 to 20. Deeper trees risk overfitting. - Minimum samples split (min_samples_split): minimum number of samples required to split an internal node. Default = 2. Increase to prevent splits on very small subsets. - Minimum samples leaf (min_samples_leaf): minimum number of samples that must be in a leaf node. Default = 1. Increase to smooth the model. - Criterion: the function to measure split quality. For classification: 'gini' or 'entropy'. For regression: 'mse' or 'mae'. - Max features (max_features): number of features to consider when looking for the best split. For classification, default is sqrt(n_features); for regression, it's n_features. - Splitter: strategy used to choose the split at each node. 'best' (default) chooses the best split; 'random' chooses the best random split.

When you train a decision tree using Azure Machine Learning, you provide a labeled dataset. The algorithm builds the tree by examining all possible splits for each feature. For numeric features, it sorts the values and considers midpoints between consecutive distinct values as potential split points. For categorical features, it considers partitions of the categories.

During prediction, a new sample traverses the tree from root to a leaf, following the branches based on its feature values. The leaf's majority class (classification) or mean value (regression) is the prediction.

Decision trees are prone to overfitting, especially when the tree is deep and captures noise in the training data. Symptoms include high accuracy on training data but poor performance on test data. Mitigation strategies include:

A random forest is an ensemble of decision trees, typically trained with the bagging (Bootstrap Aggregating) method. It combines the predictions of multiple trees to improve accuracy and control overfitting. The algorithm works as follows: 1. Create N bootstrap samples (random samples with replacement) from the original dataset. 2. For each bootstrap sample, train a decision tree, but with a twist: at each node, only a random subset of features is considered for splitting (typically sqrt(p) for classification, p/3 for regression). 3. The final prediction is the majority vote (classification) or average (regression) of all trees.

The combination of bagging (training each tree on a different random subset of data) and random feature selection decorrelates the trees. If all trees were identical, the ensemble would have the same bias as a single tree. By introducing randomness, each tree learns different patterns, and the averaging/voting reduces variance without increasing bias significantly. This makes random forests robust to noise and outliers.

Both decision trees and random forests provide feature importance scores, which indicate how much each feature contributes to the predictions. In random forests, importance is often calculated as the average reduction in impurity (e.g., Gini) across all trees when a feature is used for splitting. This is useful for feature selection and model interpretation.

In Azure Machine Learning, you can build decision trees and random forests using: - Designer: drag-and-drop modules like "Two-Class Decision Forest" or "Multiclass Decision Forest" for classification, and "Decision Forest Regression" for regression. - Automated ML: automatically tries decision trees and random forests among other algorithms. - SDK (Python): using azureml.train.automl or directly using scikit-learn estimators.

The Azure ML Designer provides prebuilt modules with hyperparameter options. For example, the "Two-Class Decision Forest" module has parameters like: - Number of decision trees: default 8. - Maximum depth of the decision trees: default 32. - Number of random splits per node: default 128. - Minimum number of samples per leaf node: default 10.

For classification, common metrics include accuracy, precision, recall, F1-score, and AUC-ROC. For regression, use RMSE, MAE, R-squared. The AI-900 exam expects you to know that random forests generally outperform single decision trees in accuracy but are less interpretable. You should also understand that random forests handle missing values better than decision trees (through surrogate splits or by ignoring missing values during splitting).

In Azure Automated ML, decision trees and random forests are among the algorithms tried automatically. You can constrain the search space by specifying allowed models. Hyperparameter tuning can be done via Azure Hyperdrive, which supports random sampling, grid sampling, and Bayesian optimization. For random forests, typical hyperparameter ranges include n_estimators (10-500), max_depth (1-50), and min_samples_leaf (1-20).

A large bank uses a random forest model to predict whether a loan applicant will default. The dataset includes features like credit score, income, debt-to-income ratio, employment length, and number of late payments. The bank trains a random forest with 500 trees on historical data. The model outputs a probability of default, and the bank sets a threshold (e.g., 0.3) to approve or reject loans. The random forest handles missing values (e.g., some applicants have no employment length) and provides feature importance scores, revealing that credit score and debt-to-income ratio are top predictors. The model is deployed as a web service on Azure Kubernetes Service for real-time scoring. Misconfiguration: if the number of trees is too low (e.g., 10), the model may have high variance; if max_depth is too high (e.g., 100), it overfits. The bank uses Azure ML's automated ML to tune hyperparameters, achieving an AUC of 0.85 on the test set.

A manufacturing company uses decision trees to diagnose machine failures. Each machine has sensors recording temperature, vibration, pressure, and runtime. A decision tree is trained to classify whether a failure will occur within the next 24 hours. The tree's interpretability is crucial because engineers need to understand why a failure is predicted (e.g., if temperature > 80°C and vibration > 0.5 mm/s, then failure likely). The tree is deployed on edge devices using Azure IoT Edge. However, the tree overfits to a specific machine type, so the company uses a random forest to generalize across different machines. The random forest achieves 92% accuracy, but the engineers sacrifice some interpretability. They use SHAP values (Shapley Additive Explanations) to explain individual predictions. Common issue: if the dataset is imbalanced (failures are rare), the model may predict 'no failure' for all cases. They use SMOTE (Synthetic Minority Over-sampling Technique) in the Azure ML pipeline to balance the classes.

A telecom company uses a random forest to predict which customers are likely to cancel their subscription. Features include contract length, monthly charges, tenure, number of customer service calls, and payment method. The random forest is trained on 100,000 customers. The model's feature importance reveals that tenure and contract length are the most important. The company uses the model to target high-risk customers with retention offers. They deploy the model as a batch inference pipeline in Azure ML, scoring 1 million customers weekly. A common pitfall is using default hyperparameters without tuning, leading to suboptimal performance. By tuning n_estimators to 200 and max_depth to 20, they improve precision from 0.65 to 0.72. The model is monitored for concept drift using Azure ML's data drift monitoring.