AAAI Publications, The Thirty-Third International Flairs Conference

Font Size: 
Towards Automatic Clustering Analysis Using Traces of Information Gain: The InfoGuide Method
Paulo Rocha, Diego Pinheiro, Martin Cadeiras, Carmelo Bastos-Filho

Last modified: 2020-05-08

Abstract


Clustering analysis has become a ubiquitous information retrieval tool in a wide range of domains, but a more automatic framework is still lacking. Though internal metrics are the key players towards a successful retrieval of clusters, their effectiveness on real-world datasets remains not fully understood, mainly because of their unrealistic assumptions underlying datasets. We hypothesized that capturing traces of information gain between increasingly complex clustering retrievals---InfoGuide---enables an automatic clustering analysis with improved clustering retrievals. We validated the InfoGuide hypothesis by capturing the traces of information gain using the Kolmogorov-Smirnov statistic and comparing the clusters retrieved by InfoGuide against those retrieved by other commonly used internal metrics in artificially-generated, benchmarks, and real-world datasets. Our results suggested that InfoGuide can enable a more automatic clustering analysis and may be more suitable for retrieving clusters in real-world datasets displaying nontrivial statistical properties.


Full Text: PDF