The rise of data (big and small) in journalism

NiemanJOurnalismLab reporting:
Viktor Mayer-Schönberger and Kenneth Cukier published their joint tome on big data this week, Big Data: A Revolution That Will Transform How We Live, Work and Think. Mayer-Schönberger, a professor of Internet governance and regulation at Oxford, and Cukier, the data editor of The Economist, argue that having access to vast amounts of data will soon overwhelm our natural human tendency to look for correlation and causality where there is none. In the near future, we’ll be able to rely on much larger pools of “messy” data rather than small pools of “clean” data to get more accurate answers to our questions.
“We are taking things we never thought of as informational and rendering them in data,” Mayer-Schönberger said in a talk Wednesday at the Berkman Center for Internet & Society at Harvard. “Once we think of it as data, we can organize it and extract new information.”
In their book, Mayer-Schönberger and Cukier give a number of examples of industries that will be changed forever by the new messiness of data. Bradford Cross cofounded FlightCaster.com, which predicted U.S. flight delays using data about flight times and weather patterns. The company was sold in 2011, at which point “Cross turned his sights on another aging industry.” He started Prismatic, one of a number of news aggregators that filters content for users by analyzing data about sharing frequency on social networks and user preferences. Mayer-Schönberger and Cukier write:
This is a humbling reminder to the high priests of mainstream media that the public is in aggregate more knowledgeable than they are, and that cuff linked journalists must compete against bloggers in their bathrobes. Yet the key point is that it is hard to imagine that Prismatic would have emerged from within the media industry itself, even though it collects lots of information. The regulars around the bar of the national Press Club never thought to reuse online data about media consumption. Nor might the analytics specialists in Armonk, New York or Bangalore, India have harnessed the information in this way. It took Cross, a louche outsider with disheveled hair and a slacker’s drawl, to presume that by using data he could tell the world what it ought pay attention to better than the editors of The New York Times...
http://www.niemanlab.org/2013/03/were-going-to-tell-people-how-to-interview-databases-the-rise-of-data-big-and-small-in-journalism/

First look: Spundge is software to help journalists to manage real-time data streams

NiemanJournalismLab reporting:
Many power users of Twitter consider TweetDeck essential for managing multiple streams of data. Once you adapt to its overwhelming user interface, the software becomes essential. And addictive.
Imagine being able to add more real-time sources to TweetDeck — RSS feeds, Facebook, Flickr, YouTube — and you have something like Spundge, a web app that launches in public beta today. The software is being marketed initially to journalists and is being tested inside a few news organizations.
“The problem is today’s journalist has to use too many products and applications to do their job, and very few of these were actually built with newsrooms or journalistic workflow in mind,” said Craig Silverman, the corrections guru who is working with Spundge to help develop the product for journalists.
“Spundge is a platform that’s built to take a journalist from information discovery and tracking all the way to publishing, regardless of whatever internal systems they have to contend with,” he told me.
A user creates notebooks to organize material (a scheme familiar to Evernote users). Inside a notebook, a user can add streams from multiple sources and activate filters to refine by keyword, time (past few minutes, last week), location, and language.
Spundge extracts links from those sources and displays headlines and summaries in a blog-style river. A user can choose to save individual items to the notebook or hide them from view, and Spundge’s algorithms begin to learn what kind of content to show more or less of. A user can also save clippings from around the web with a bookmarklet (another Evernote-like feature). If a notebook is public, the stream can be embedded in webpages, à la Storify. (Here’s an example of a notebook tracking the ONA 2012 conference.)
http://www.niemanlab.org/2012/09/first-look-spundge-is-software-to-help-journalists-to-manage-real-time-data-streams/?utm_source=Daily+Lab+email+list&utm_medium=email&utm_campaign=b5bbac0ff5-DAILY_EMAIL

Could data save newspapers?

inma reporting: The world is entering a new era of “Big Data” according to a new McKinsey report which claims that businesses that can get their heads around how to harness the constant flow of information are winning the race of profit and competition.
The news is likely to offend many die hard traditionalists in newsrooms, who are still enraged by the Internet making the word “content” a synonym for journalism. But as obnoxious as it was for purists to contemplate the idea that the poetry of beautiful writing and stunning photography could be belittled by a collective noun, “content” has stretched our perceptions of journalism now. The word has helped us to visualise story telling environments that incorporate rich picture galleries, video, and interactive graphics and information that is both spontaneous and curated. Calling journalism “content” broke an old perception and allowed us to see our product in a different light — the light of our readers and customers (who are now called users, by the way, but we’ll leave that for now).
And so, to data. Data is an interesting one. A big one. Bigger than “content” because data could describe not just what we produce, but, if we get smart very quickly, considering that what we do is data could be our new business model.
It’s even a model that many media companies have proven themselves to be extremely adept at — Dow Jones, Financial Times, Reuters. And when you look at the phenomenon of Google and Facebook, it’s actually the fuel in their tanks that makes us so jealous. The two online monoliths showed us that data is not just for financial boffins and the big end of town — data packaged in a warm and friendly way can change everyone’s lives.
So, “Are you ready for the era of big data?” ask Brad Brown, Michael Chui and James Manyika in the latest McKinsey Quarterly.
“Emerging academic research suggests that companies that use data and business analytics to guide decision making are more productive and experience higher returns on equity than competitors that don’t,” the report says.
It claims that “networked organisations can gain an edge by opening information conduits internally and by engaging customers and suppliers strategically through Web-based exchanges of information. Over time, we believe big data may well become a new type of corporate asset that will cut across business units and function much as a powerful brand does, representing a basis for competition.”
And this is where the idea becomes extremely attractive: data as a brand. Surely companies whose core business is creating stories, photos, images, and video on a 24/7 news cycle — most of it original or a unique understanding of recent events — would know a thing or two about content.
But newspaper companies have always had a lackadaisical attitude to data. While companies such as Google and Facebook — and even Flipboard and Welt — make it their business to hoover up information — much of it ours — and regurgitate it in new formats and contexts, newspaper companies have been happy to throw it out each day — turn it into fish and chip wrapper. For us, the excitement has come not from understanding how what we have done could work in new ways so that we could extract additional value, but on the lure of the next story and doing it better all over again tomorrow.
We even pay extraordinary amounts to third party organisations to provide industry insights and reports into markets which our reporters cover every single day. That’s a huge irony when you think a lot of the information the consultancies are using has come from our own news pages. When it comes to making business decisions, newspapers are insecure about trusting our own insights, nor do we have the best technology for capturing and analysing what we’ve done.
http://www.inma.org/blogs/out-of-the-box/post.cfm/could-data-save-newspapers?utm_source=newsletter&utm_medium=email&utm_campaign=nonmember

ProPublica’s newest news app uses education data to get more social

Niemanlabs reporting:
Yesterday, the U.S. Department of Education’s Office of Civil Rights released a data set — the most comprehensive to date — documenting student access to advanced classes and special programs in public high schools. Shorthanded as the Civil Rights survey, the information tracks the availability of offerings, like Advanced Placement courses, gifted-and-talented programs, and higher-level math and science classes, that studies suggest are important factors for educational attainment — and for success later in life.
ProPublica reporters used the Ed data to produce a story package, “The Opportunity Gap,” that analyzes the OCR info and other federal education data; their analysis found among other things that, overall and unsurprisingly, high-poverty schools are less likely than their wealthier counterparts to have students enrolled in those beneficial programs. The achievement gap, the data suggest, isn’t just about students’ educational attainment; it’s also about the educational opportunities provided to those students in the first place. And it’s individual states that are making the policy decisions that affect the quality of those opportunities. ProPublica’s analysis, says senior editor Eric Umansky, is aimed at answering one key question: “Are states giving their kids a fair shake?”
The fact that the OCR data set is relatively comprehensive — reporting on districts with more than 3,000 students, it covers 85,000 schools, and around 75 percent of all public high schoolers in the U.S. — means that the OCR data set is also enormous. And while ProPublica’s text-based takes on the info have done precisely the thing you’d want them to do — find surprises, find trends, make it meaningful, make it human — the outfit’s reporters wanted to go beyond the database-to-narrative formula with the OCR trove. Their solution: a news app that encourages, even more than your typical app, public participation. And that looks to Facebook for social integration.

The app focuses on measuring equal access on a broad scale: It tracks not only the educational opportunities provided by each school, but also breakdowns of students’ race, disability status, gender, and English proficiency. It also highlights the percentage of teachers with two years’ experience or less — who, as a group, tend to effect smaller achievement gains than their more experienced counterparts — and the percentage of students who receive free or reduced-price school lunch, an indicator of poverty. (More on the developers’ methodology here.)
http://www.niemanlab.org/2011/07/propublicas-newest-news-app-uses-education-data-to-get-more-social/?utm_source=Daily+Lab+email+list&utm_campaign=b3ec0061fe-DAILY_EMAIL&utm_medium=email