An investigation of the Information Bottleneck (IB) method in supervised learning and how decision trees can be represented in the IB framework

The last two decades have been marked by rapid progress in the fields of machine learning and data science, for which the state of the art has continued to push new frontiers previously thought to be decades away. While remarkable, many of these powerful techniques have important shortcomings such as a high energy usage, risks of bias, and for some a deep lack of theoretical understanding. The Information Bottleneck (IB) method has been proposed as a novel explanation of supervised learning. Originally introduced in the context of Information theory as a principled approach to extract relevant information in a signal, this method has since found a wide variety of applications beyond its initial purpose. In this work, we study the capabilities of the IB framework to evaluate important aspects of supervised models in the important case of classification tasks. Our results indicate that the IB framework is insufficient to adequately evaluate both the quality of a model's fit to the training data, as well as its ability to generalize to unseen observations. Moreover, we also present the first to our knowledge analysis of decision trees in the IB framework.