A large proportion of audiences read news online, often accessing news articles through social media like Facebook or Twitter. A distinguishing characteristic of news on social media is that the most prominent (and often the only visible) part of the news article is the headline. We investigate the impact of headline characteristics, including journalistic concepts of news values and linguistic style, on the article's social media popularity. Using a large corpus of headlines from The Guardian and New York Times we derive these features automatically and correlate with social media popularity on Twitter and Facebook. We found most of them to have a significant effect and that their impact differs between the two social media and between news outlets. Further investigation with a crowdsourced study confirms that news values and style influence the audiences' decisions to click on a headline.