Successive Halving Top-k Operator

No. 18: AAAI-21 Student Papers and Demonstrations

Volume

Proceedings of the AAAI Conference on Artificial Intelligence, 35

AAAI Student Abstract and Poster Program

We propose a differentiable successive halving method of relaxing the top-k operator, rendering gradient-based optimization possible. The need to perform softmax iteratively on the entire vector of scores is avoided using a tournament-style selection. As a result, a much better approximation of top-k and lower computational cost is achieved compared to the previous approach.

10.1609/aaai.v35i18.17931