Abstract: Classification is one of the most ubiquitous problems in Machine Learning. Over the past few decades, researchers have developed hundreds of classifiers. The literature proposes dozens of different versions of any basic classification model. Many of those classifiers are rarely used because they are hard to implement or because there is no proper comparison with existing models. Despite the research community has conducted several method comparisons, these comparisons have rapidly aged and failed to encompass a broad range of methods. To palliate those two problems, this paper presents both the largest comparison of classifiers presented so far and also a framework for expanding the comparison in a simple, dynamic way. Thus, we first present the most comprehensive classifier comparison so far with 250 classification methods and 707 datasets. The comparison also takes into account the specific characteristics of each dataset. Then, we present a system where any researcher can submit the implementation of a published method that will be automatically added to the comparison. That system would keep the comparison up to date and offer a relevant way to assess the performance of newly published methods.
Authors: Nicolás E. García-Pedrajas, Juan A. Romero del Castillo, José M. Cuevas-Muñoz, and Javier Pérez-Rodríguez
Supplementary material and datasets: