Individuals are often more confident in their solutions when working in teams than when working on their own. This confidence boost is observed even when it is not accompanied by a corresponding gain in performance, raising the question of what other factors might be responsible. We address this question by developing a large-scale experimental setting in the form of a two-player online game that allows us to track the confidence of individuals in naturally-occurring online collaborative tasks. This setting enables us to disentangle and compare the effects of different components of the collaborative process on the confidence of each team member. We show that confidence evaluations are subject to social influence: a low-confidence individual receives a confidence boost as a direct consequence of interacting with their teammate, and the extent of the increase depends more on the confidence, rather than on the competence, of the teammate. The resulting framework can enhance our understanding of confidence boost as an often overlooked byproduct of online teamwork and has implications for designing better online collaboration platforms to meet diverse collaborative objectives.