Is it Over Yet? Learning to Recognize Good News in Financial Media

Until recently, tracking sentiment in news media required professional annotators to identify the polarity of individual articles so that general trends could be identified. In the work described here we use crowdsourcing to gather non-expert annotations, in conjunction with a supervised learning strategy that generalizes from the manual annotations to label a larger body of news articles. Our analysis of this strategy shows that, while it is effective, there are three key issues that have to be addressed: consensus, coverage, and bias. By obtaining multiple annotations for an article we can establish a consensus for the article. Alternatively we can seek only a single annotation for each article in order to maximize coverage, but without the benefit of a group consensus. With bias, we are not so much concerned by bias among the annotators as by the bias in the learning system which can favor the majority class. In this paper we address these three issues in the context of an analysis of media sentiment towards the Irish economic situation.