Breaking news, viral videos, and popular memes are all examples of the collective attention of huge numbers of users focusing in large-scale social systems. But this self-organization, leading to user attention quickly coalescing and then collectively focusing around a phenomenon, opens these systems to new threats like collective attention spam. Compared to many traditional spam threats, collective attention spam relies on the insidious property that users themselves will intentionally seek out the content where the spam will be encountered, potentially magnifying its effectiveness. Our goal in this paper is to initiate a study of this phenomenon. How susceptible are social systems to collective attention threats? What strategies by malicious users are most effective? Can a system automatically inoculate itself from emerging threats? Towards beginning our study of these questions, we take a two fold approach. First, we develop data-driven models to simulate large-scale social systems based on parameters derived from a real system. In this way, we can vary parameters — like the fraction of malicious users in the system, their strategies, and the countermeasures available to system operators — to explore the resilience of these systems to threats to collective attention. Second, we pair the data-driven model with a comprehensive evaluation over a Twitter system trace, in which we evaluate the effectiveness of countermeasures deployed based on the first moments of a bursting phenomenon in a real system. Our experimental study shows the promise of these countermeasures to identifying threats to collective attention early in the lifecycle, providing a shield for unsuspecting social media users.