XGBoost in Python from Start to Finish
Summary
This webinar provides a start-to-finish guide on using XGBoost in Python for customer churn prediction. It covers importing and cleaning data, handling missing values uniquely with XGBoost, and formatting categorical data using one-hot encoding. The tutorial details building a preliminary model, optimizing hyperparameters via cross-validation and grid search, and finally, interpreting and evaluating the optimized model. It emphasizes practical application with a real-world telco churn dataset, aiming to improve prediction accuracy, especially for the minority class (customers who churn).
Key Insights
XGBoost has a unique default behavior for handling missing data; it expects it and uses zero.
XGBoost can natively handle missing values. The strategy is to identify missing values and set them to zero, which XGBoost then interprets correctly, even if zeros already exist in the data or have other meanings.
One-hot encode categorical features to make them usable by XGBoost.
Categorical features (like 'payment_method') are converted into a numerical format using one-hot encoding. This creates new binary columns for each category, preventing algorithms from assuming ordinal relationships between categories.
XGBoost efficiently handles missing data coded as zero using sparse matrices.
When missing data is represented by zeros, XGBoost utilizes sparse matrices, storing only the non-zero values. This makes memory allocation and computation efficient, as zeros are implicitly handled without explicit storage.
The optimized model significantly improves churn prediction accuracy to 84%.
After retraining with optimized parameters and `scale_pos_weight`, the confusion matrix shows a dramatic improvement in correctly classifying churners: 84% accuracy compared to the initial 51%. This comes at a slight cost to accuracy for non-churners.
Improved prediction of churners justifies the trade-off in predicting non-churners for business value.
While the model is slightly less accurate for non-churners, the improved ability to identify potential churners is more valuable to the business, as retaining customers is often more cost-effective than acquiring new ones. This allows targeted retention efforts.
Sections
Introduction and Setup
XGBoost builds powerful, interpretable models using boosted trees, suitable for predicting customer churn.
The webinar introduces XGBoost as a machine learning method that balances predictive accuracy with model interpretability. It will use the telco churn dataset from IBM Base Samples to predict customer attrition using continuous and categorical data.
Key topics include data import, missing data handling, data formatting, model building, optimization, and interpretation.
The tutorial will cover importing data from a file, handling missing data (including XGBoost's unique approach), formatting data with one-hot encoding, building a preliminary XGBoost model, optimizing parameters with cross-validation and grid search, and finally interpreting and evaluating the optimized model.
Prerequisites include basic Python, XGBoost theory, cross-validation, and confusion matrix understanding.
The tutorial assumes familiarity with Python basics, the theory behind XGBoost, cross-validation, and confusion matrices. Links to StatQuest videos for these topics are provided.
Essential Python modules and their installation requirements are listed.
The necessary Python modules are pandas, numpy, scikit-learn, and xgboost. Instructions for installing these and Graphviz (for tree drawing) are included.
Data Import and Initial Cleaning
Load the Telco Churn dataset into a pandas DataFrame for analysis.
The `pd.read_csv` function is used to load the Telco Churn dataset into a DataFrame named `df`. The `head()` function displays the first five rows to inspect the data.
Remove irrelevant 'exit interview' columns that provide perfect, but unusable, future information.
Columns like 'churn_reason', 'cltv', 'churn_score', and 'churn_details' are dropped because they are collected after a customer leaves and would create unrealistic predictive accuracy.
Identify and remove columns with only a single unique value, as they offer no predictive power.
Columns such as 'count', 'country', and 'state' are removed because they contain only one distinct value across all records ('1', 'United States', and 'California', respectively), making them useless for classification.
Drop 'customer_id' and 'lat_long' columns as they are redundant or non-predictive.
'customer_id' contains unique values for each individual and is not useful for prediction. 'lat_long' is redundant as separate 'latitude' and 'longitude' columns already exist. These are removed using the `drop` function.
Replace whitespace in 'city' names with underscores for consistency and drawing tree visualizations.
Whitespace within city names (e.g., 'Los Angeles') is replaced with underscores ('Los_Angeles') using the `replace` function. This is primarily for aesthetic reasons when drawing trees, as XGBoost itself can handle spaces.
Standardize column names by replacing whitespace with underscores for clean processing.
All column names in the DataFrame are processed to replace any whitespace characters with underscores, ensuring a uniform naming convention for easier access and manipulation, particularly for tree visualization.
Handling Missing Data with XGBoost
XGBoost has a unique default behavior for handling missing data; it expects it and uses zero.
XGBoost can natively handle missing values. The strategy is to identify missing values and set them to zero, which XGBoost then interprets correctly, even if zeros already exist in the data or have other meanings.
Identify missing values in 'total_charges' by checking for blank strings, which cause parsing errors.
The 'total_charges' column, initially of 'object' type, contains blank spaces that prevent conversion to numeric. Trying to convert it to numeric raises an error specifically mentioning 'unable to parse string quote nothing or blank space end quote'.
Replace missing 'total_charges' values with 0 for XGBoost's internal handling.
Only 11 rows have missing 'total_charges'. These correspond to customers with 'tenure_months' of 0, indicating they just signed up. These missing values are explicitly set to 0 in the 'total_charges' column.
Convert 'total_charges' to a numeric data type (float64) after handling missing values.
After setting missing values to 0, the 'total_charges' column is converted from 'object' to 'float64' using `pd.to_numeric`. This ensures compatibility with XGBoost, which requires numeric or boolean data types.
Perform a data-wide replacement of all remaining whitespace characters with underscores.
A final `df.replace` operation is performed across the entire DataFrame to replace any remaining blank spaces with underscores. This is mainly to ensure clean presentation in tree visualizations.
Data Formatting for XGBoost
Separate features (X) from the target variable (y) for model training.
The DataFrame is split into two parts: `X` (capital X) contains all features used for prediction, created by dropping the 'churn_value' column from `df`. `y` (lowercase y) contains the target variable 'churn_value' (1 for churn, 0 for no churn).
One-hot encode categorical features to make them usable by XGBoost.
Categorical features (like 'payment_method') are converted into a numerical format using one-hot encoding. This creates new binary columns for each category, preventing algorithms from assuming ordinal relationships between categories.
Use `pd.get_dummies` to perform one-hot encoding on all categorical columns in the feature set.
The `pd.get_dummies` function is applied to the DataFrame `X`, converting all columns with 'object' data types into multiple binary (0 or 1) columns. This results in an `X_encoded` DataFrame with significantly more columns.
Verify the target variable 'y' contains only binary values (0s and 1s).
The `y.unique()` function confirms that the target variable 'churn_value' consists solely of 0s and 1s, which is appropriate for binary classification.
Mini Stat Quest: XGBoost's Handling of Missing Data and Zeros
XGBoost efficiently handles missing data coded as zero using sparse matrices.
When missing data is represented by zeros, XGBoost utilizes sparse matrices, storing only the non-zero values. This makes memory allocation and computation efficient, as zeros are implicitly handled without explicit storage.
XGBoost compares splits using zero-coded missing data against different categories to find the best gain.
When a feature has missing data coded as zero, XGBoost evaluates two potential splits: one treating the missing values as belonging to one category (e.g., blue), and another treating them as belonging to another (e.g., green). It chooses the split that maximizes gain.
Model Building and Initial Evaluation
Split the data into training and testing sets, ensuring stratification due to class imbalance.
The `train_test_split` function is used to divide the data into training and testing sets. Stratification (`stratify=y`) is employed to maintain the original proportion of churned customers (approx. 27%) in both sets, using `random_state=42` for reproducibility.
Instantiate and train an XGBoost classifier ('XGBClassifier') with binary logistic objective.
An `XGBClassifier` is initialized with `objective='binary:logistic'`, `missing=None` (defaulting to zero handling), and `random_state=42`. The model is then trained using the `fit` method on the training data (`X_train`, `y_train`).
Employ early stopping during training to prevent overfitting and find the optimal number of trees.
Early stopping is enabled using `eval_set` and `verbose=True (or False)`. The model trains, monitoring performance on the test set (AUC as the metric). It stops if performance doesn't improve for 10 consecutive trees, using the best iteration found.
Generate a confusion matrix to evaluate the preliminary model's performance.
A confusion matrix is plotted using `plot_confusion_matrix` on the test set predictions. The initial model shows high accuracy (91%) for non-churners but low accuracy (51%) for churners, indicating a significant class imbalance problem.
Address class imbalance using XGBoost's 'scale_pos_weight' parameter.
To improve prediction for the minority class (churners), the 'scale_pos_weight' parameter is introduced. This parameter increases the penalty for misclassifying the minority class, encouraging the model to focus more on correctly identifying churners.
Hyperparameter Optimization using Grid Search and Cross-Validation
Optimize XGBoost hyperparameters like max_depth, learning_rate, gamma, and regularization.
Key hyperparameters to tune include 'max_depth' (tree depth), 'learning_rate' (eta), 'gamma' (minimum loss reduction for pruning), and 'lambda' (L2 regularization). The goal is to improve the model's predictive accuracy, especially for the minority class.
Use cross-validation (GridSearchCV) to find the best combination of hyperparameters.
A grid search is performed over specified ranges of hyperparameters. Due to computational time, the process is divided into two rounds. In each round, a random subset of 90% of the data and 50% of columns per tree are used to speed up cross-validation.
Optimize using AUC as the scoring metric, crucial for imbalanced datasets.
The Area Under the ROC Curve (AUC) is used as the evaluation metric for cross-validation because it is robust to class imbalance, providing a better assessment of performance than accuracy alone.
Iterative optimization process led to final parameters: max_depth=4, learning_rate=0.1, gamma=0.25, lambda=10, scale_pos_weight=3.
Through two rounds of grid search and cross-validation, optimal values were found. The first round helped narrow down ranges, and the second round refined them. `scale_pos_weight` was confirmed to be 3, indicating a significant focus on the minority class.
Building, Evaluating, and Interpreting the Optimized Model
Build the final XGBoost model using the optimized hyperparameters.
A new `XGBClassifier` is trained using the determined optimal parameters, including `max_depth=4`, `learning_rate=0.1`, `gamma=0.25`, `lambda=10`, and `scale_pos_weight=3`. Early stopping is used again.
The optimized model significantly improves churn prediction accuracy to 84%.
After retraining with optimized parameters and `scale_pos_weight`, the confusion matrix shows a dramatic improvement in correctly classifying churners: 84% accuracy compared to the initial 51%. This comes at a slight cost to accuracy for non-churners.
Improved prediction of churners justifies the trade-off in predicting non-churners for business value.
While the model is slightly less accurate for non-churners, the improved ability to identify potential churners is more valuable to the business, as retaining customers is often more cost-effective than acquiring new ones. This allows targeted retention efforts.
Visualize the first tree of the optimized model to understand its decision-making process.
The first tree in the XGBoost ensemble is plotted to illustrate how the model makes decisions. The root node splits on 'contract_month_to_month' < 1, directing observations with missing values or values less than 1 down the 'less than' path.
Interpret tree nodes: splits use feature thresholds and thresholds on negative cases + missing values.
Each internal node represents a split based on a feature and a threshold. The branches indicate paths for 'true' or 'false' conditions. XGBoost includes missing values in the split decision, often grouped with the 'less than' path.
Tree leaves provide incremental probability contributions, summed across all trees for the final prediction.
Unlike traditional decision trees, XGBoost's leaves do not provide direct classifications. Instead, they output small probability increments that are summed across all trees in the ensemble to generate the final prediction probability.
Conclusion
XGBoost successfully built, optimized, and evaluated for customer churn prediction.
The webinar successfully covered data loading, cleaning, missing value imputation, one-hot encoding, model training, hyperparameter tuning with cross-validation, and model interpretation, leading to an improved churn prediction model.
Ask a Question
*Uses 1 Wisdom coin from your coin balance
