By this point, you should be able to understand and apply some of the most well-known XAI techniques. However, it’s easy to forget that explanations always aim to make someone understand the model better. That someone may be you, looking for insights into the model that you’ve trained, but it could also be a business owner or an end user who needs intuitive ways of understanding how an AI model is making decisions that impact them. In this section, we explore the outputs of XAI techniques and their interpretations. We also dive into some of the practical challenges and concerns that are commonly faced when trying to generate these outputs in practice, such as deployment on edge devices or privacy concerns.
Explainable AI for debugging
Explainable AI provides insights regarding feature importance and provides valuable information that supports data scientists in improving their models. For instance, with global explanations, we can implement dashboards to analyze the errors and discover how the model fails for specific cohorts of data, for example, user groups. The snippet below shows how to get insights from explainability methods using the dashboard provided by Microsoft. Here we use the Adult Census Income dataset for predicting whether a person’s income exceeds $50K/yr or not. The dataset is from the 1994 US Census Bureau database consisting of various demographic, socioeconomic, and employment-related features such as age, marital status, education level, and work hours per week.
from raiwidgets import ResponsibleAIDashboard from responsibleai import RAIInsights rai_insights = RAIInsights( model, train_data, test_data_sample, target_feature, 'classification', categorical_features=categorical_features ) rai_insights.explainer.add() rai_insights.error_analysis.add() rai_insights.counterfactual.add(total_CFs=10, desired_class='opposite') rai_insights.compute() from raiwidgets.cohort import Cohort, CohortFilter, CohortFilterMethods cohort_filter_age = CohortFilter( method=CohortFilterMethods.METHOD_LESS, arg=[65], column='age' ) cohort_filter_hours_per_week = CohortFilter( method=CohortFilterMethods.METHOD_GREATER, arg=[40], column='hours-per-week' ) user_cohort_age_and_hours_per_week = Cohort( name='Cohort Age and Hours-Per-Week' ) user_cohort_age_and_hours_per_week.add_cohort_filter(cohort_filter_age) user_cohort_age_and_hours_per_week.add_cohort_filter(cohort_filter_hours_per_week) cohort_filter_marital_status = CohortFilter( method=CohortFilterMethods.METHOD_INCLUDES, arg=["Never-married", "Divorced"], column='marital-status' ) user_cohort_marital_status = Cohort(name='Cohort Marital-Status') user_cohort_marital_status.add_cohort_filter(cohort_filter_marital_status) cohort_filter_index = CohortFilter( method=CohortFilterMethods.METHOD_LESS, arg=[20], column='Index' ) user_cohort_index = Cohort(name='Cohort Index') user_cohort_index.add_cohort_filter(cohort_filter_index) cohort_filter_predicted_y = CohortFilter( method=CohortFilterMethods.METHOD_INCLUDES, arg=['>50K'], column='Predicted Y' ) user_cohort_predicted_y = Cohort(name='Cohort Predicted Y') user_cohort_predicted_y.add_cohort_filter(cohort_filter_predicted_y) cohort_filter_true_y = CohortFilter( method=CohortFilterMethods.METHOD_INCLUDES, arg=['>50K'], column='True Y' ) user_cohort_true_y = Cohort(name='Cohort True Y') user_cohort_true_y.add_cohort_filter(cohort_filter_true_y) cohort_list = [user_cohort_age_and_hours_per_week, user_cohort_marital_status, user_cohort_index, user_cohort_predicted_y, user_cohort_true_y]
Now we visualize the insights gained from responsible and explainable AI for different cohort of data points.
ResponsibleAIDashboard(rai_insights, cohort_list=cohort_list)
In this tree-shaped graph, you can follow the error rate in the various cohorts. After making different cohorts of the dataset, the model doesn’t perform as expected for those datapoints of people who are married, have higher education, and have minimal capital gain.
Try out the demo at https://raidashboard.azurewebsites.net/
What-if-analysis
Using XAI methods, end users are able to apply changes in the feature values to figure out how the explanation output changes accordingly. This process improves the data analysis since the outputs of feature importance of original data can bring a great deal of knowledge when compared to perturbed features. Perturbation exploration, in other words, looks at the impacts of changes in the data on the model outputs, enabling the users to understand the impact of feature changes. For example, if one has a model that is trying to estimate a good price for a house, one could explore how the outcome changes when changing the postal code or the number of bedrooms in the house.
Adversarial identification
Adversarial identification comprises the detection of malicious data – in other words, data that has been manipulated by an adversary. We’ll look at an example that illustrates how changes in the dataset influence the explanations made by the XAI methods.
In the following, we interpret the outputs of XAI methods and evaluate important features of data that AI models used to boost their inference process. We're interested in evaluating XAI results in case of adversarial activities on data, in other words, seeing if we can spot poisoned or distorted data in our dataset of images. We consider three model-agnostic XAI methods that can be applied for any type of AI model: LIME (Local the Interpretable Model-agnostic Explanation), SHAP (Shapley Addictive Explanations), and occlusion sensitivity. As these methods are model-agnostic, they don’t require any information about the convolutional neural network (CNN) gradients to analyze model behavior. These methods are also perturbation-based, which means that they manipulate the input (image pixels) to extract details that can be linked with the predictions. The images below show that the adversarial attacks (for example steganography) affect the results of XAI. We’ll go through this example more in-depth in section 4.2.
Explanations of original (unpoisoned) data from selected XAI methods.
Examples of poisoned data from selected XAI methods.
Challenges and complexity of XAI Analysis
Explainable AI methods can be difficult to apply in practice as they require instrumentation with the model and data, which requires a considerable amount of time and expertise. At the same time, assessing AI models using these techniques can be difficult due to a variety of variables. Gaining access to both the model and the data needed to train it is a significant hurdle. Businesses that own proprietary models may be unwilling to share them or may only allow limited access to certain stakeholders. Furthermore, worries about data privacy may hinder the sharing of sensitive data used to train the models. Another difficulty is the complexities of the models themselves. Deep learning models, for example, can have millions of parameters as well as complicated, non-linear correlations between input characteristics and predicted output. As a result, understanding how these models behave necessitates specialized knowledge and skills in machine learning, statistics, and data analysis.
Other relevant considerations when using XAI methods
The analysis of AI models using XAI methods for explaining the internal workings of AI black box models can result in complexity and challenges because of resource utilization, algorithmic complexities, and resource requirements. Let’s address some of these complexities below:
In terms of resource usage – XAI Methods like LIME and SHAP require the necessary computation and resources to compute gradients or perturbations for each input. These can considerably raise the computational and resource expense.
In terms of algorithmic complexities – the algorithmic complexity of XAI methods varies depending on the method, type of input data, and the problem that’s being considered. Gradient-based methods, which involve computing gradient at each iterative step, have complexity that’s proportional to the number of input features and also on the number of layers in the model. Perturbation-based methods such as LIME have an algorithmic complexity that varies directly on the number of perturbations and also on the number of input feature vectors. The complexity of rule-based methods depends on the complexity of the rule or algorithmic rule set that the method uses. For instance, for decision trees the complexity depends on parameters like the number of training examples and on the number of input features. The complexity of CFE depends on the optimization technique that is used to produce the counterfactuals, the number of constraints involved, and also on the number of iterations the optimization algorithm uses to generate alternative input instances (counterfactuals).
Resource prerequisites – apart from the above-mentioned complexities, additional resources in the form of labeled or metadata, expert knowledge, or model-specific data may be needed for XAI approaches. For example, there is a need for pre-trained semantic embeddings in some model interpretation techniques like ‘Concept Activation Vectors’, which may not be accessible for all domains of languages or all deployment platforms.
Implementing XAI over edge devices – edge devices often have limited computational resources such as memory, processing power, and storage. Most edge devices rely on battery power and have a limited energy supply. Data availability over edge devices is also an issue due to the limited data storage capacity. This necessitates the need for storing data at some other location than the edge device which gives rise to privacy and connectivity issues. This induces an additional parameter, latency which impacts the overall model performance. Last but not least, model compatibility is also an issue as XAI methods aren’t compatible with all types of models and underlying hardware architecture. Some of the challenges of implementing XAI over edge devices have been overcome by techniques like model compression, optimization of XAI methods for edge devices, and the development of XAI-specific hardware accelerators – but implementing XAI over edge devices still remains a trending area of research with each XAI method posing its own technical challenges.
In this chapter we learned that…
There is an impact of our choice of model on it’s explainability. We got to know the concepts of glass box (inherently explainable) and black box (too complex to trace) models. We found some points to walk through when assessing whether or not explainability is a requirement for a given problem.
Local and global explainability describe if the explanation is local (it explains one specific output of the model, such as SHAP) or global (it explains the behavior of the model in general, such as permutation feature importance).
Model-agnostic or model-specific explainability refers to whether this explainability method can be used on any model (such as global surrogate model, SHAP and LIME) or if it’s tied to a specific model (such as pixel attribution, which is tied to neural networks).
The raiwidgets ResponsibleAIDashboard includes many useful tools for different aspects of responsible AI, including explainability methods.
Explainability tools can be valuable in for example what-if analysis, or exploration of how changing the inputs might impact the outcomes of the system.
Heatmap methods could be used to identify potentially poisoned or data that has been tampered with.
Although useful, XAI analysis itself can be complex, time and resource consuming, computationally heavy, and may not be possible in all environments – for example on edge devices.
