Glass-box and black-box AI – what’s the difference?

The need for explainable AI is often put in terms of the type of model being used. While terminology differs, models requiring XAI methods to make them interpretable are often called black boxes. Models that are already interpretable are glass boxes. The main difference between them is their complexity, which also means that the black boxes metaphor is probably not the most suitable for this issue. If you wish, you can also see and inspect the calculations (model weights, bias terms, activation functions) in a complex deep neural network – but its complexity means you’ll have no practical idea of what these numbers actually mean for the task the algorithm is solving. Other, simpler models, such as decision trees, Bayesian classifiers, or linear and logistic regression models, in turn, can be understood simply by looking at the mathematics behind the model.

Let’s consider an example of credit allocation with a binary outcome (credit granted or not) and six features (marital status, current account balance, employment status, credit history, place of residence, and purpose of credit). In a simple logistic regression, each of these features will have an associated coefficient that can be used to understand the effect of that feature on the outcome. For instance, if the married marital status has a coefficient of .12, it means that a married person is 12% more likely to get credit. In a neural network, especially one with multiple layers (usually called a deep neural network), model weights can seldom be interpreted in this manner. Neural networks also compute interactions between features (combinations of individual characteristics) to increase the accuracy of the outcome. On a single hidden layer, these may still be explainable (for example “when someone is married and is unemployed, that combination of features may make credit granting more likely”). However, when you start to compute interactions of interactions, the complexity becomes too large to be followed by human reasoning.

These black box models aren’t only problematic due to their complexity, but also because this complexity may mean that the model is creating new undesirable features. For instance, in the example given above of the interaction of marital status with employment status, the algorithm may combine these features to increase or decrease the likelihood that a non-working spouse gets credit, effectively creating the stay-at-home parent feature. In a glass-box logistic regression, the user gets control and can interpret this kind of process. In a deep neural network, we need XAI methods to find out if this is happening.

Therefore, when deciding on the model to use, consider the following. If explainability isn’t a requirement, use the techniques that give you the best results. Usually, the simpler, the better. Especially for simple problems, simpler models may perform as well as deep neural networks. Before committing to deep learning to follow the latest trends, it’s advisable to consider simpler, glass-box models.

If explainability is a requirement, assess whether it’s more important than accuracy. If explainability is the main concern, consider using a glass-box model like Bayesian classifiers, regression, or decision trees. If accuracy is the top priority and a black-box model performs the best, apply an XAI method on top of it.

Example: Explainable Boosting Machine (EBM)

Explainable Boosting Machine (EBM) is a glass-box AI model that strives to combine interpretability and performance. It's a tree-based, cyclic gradient-boosting Generalized Additive Model (GAM) with automatic interaction detection (1). While EBM is highly explainable, it’s also a GAM in the form of:

g(E[y]) = β₀ + Σ f_j(x_j)

where g is the link function that adapts the GAM to different settings, such as regression or classification (2). GAM allows for comprehending the contribution of each feature to the predictions made by the model. The model is trained using boosting techniques (combining multiple models by learning from mistakes of previously learned models) to enhance its performance.

We assume you’re already familiar with machine learning (ML) and model training. So, we'll skip the model-training details and concentrate on the explanation.

The table below shows the feature space and some records of the dataset.

Let’s look at how an EBM model is built hands-on with the interpret package, one of the well-known libraries for Explainable AI.

import interpret
from interpret.glassbox import ExplainableBoostingClassifier
from interpret import show
ebm = ExplainableBoostingClassifier()
ebm.fit(x_train,y_train)
ebm_global = ebm.explain_global()

Using EBM with this data set, we can achieve an interpretable model with an accuracy of 0.90. From the first graph below, we can understand how each feature contributes toward global model outcomes. It’s shown that in the case of the bank dataset, the duration an individual has had their account is the most effective attribute for the prediction of loan status.

Feature importance distribution on model outcome

Feature score (contribution) to the model prediction.

The second graph shows that as the age feature increases beyond 40, its score on prediction increases proportionally.

We next analyze the explainable model locally, meaning that we're interested in explaining individual data points using the code below.

ebm_local = ebm.explain_local(x_test,y_test)

Features’ contributions to the model’s prediction for a single data point

This graph shows how individual features or combinations contribute to the local prediction. As the figure shows, duration is the most important feature for this particular prediction.

As a bank representative, one could now evaluate whether this model behaves as desired. In case the duration of registration of personal account holders shouldn’t play the most important role with regards to the model’s outcome, then one could adapt the model (for example, by using different features for model training or using feature weights to lower the importance of certain features during model training).