Information theory and machine learning : information Bottleneck theory applied to classification trees and random forests

The Information Bottleneck (IB) method has been proposed as a theoretical framework to explain the effectiveness of deep learning. It has since become a widely-studied concept in the field of deep learning. However, research on the application of IB theory to other machine learning algorithms is almost nonexistent. In this work we explore the application of IB theory to decision trees and random forests for classification tasks. After establishing the theoretical background, we show evidence against some of the conclusions of earlier work on IB theory in the case of classification tasks. Further, we present a novel analysis of random forests trough the IB method. Finally we apply the IB method to a nondeterministic classification task.