What Was Lost is Now Found – How Openness Helps Fuel Big Data

This is Part 3 of a 5-part series devoted to exploring the concept of Big Data to determine what makes it different from other hyped data “revolutions” of the past.

On his first day in Office, President Obama signed the Memorandum on Transparency and Open Government, a platform aimed at fostering greater accountability from the government, by empowering the public with information that had traditionally been hidden.

A variety of initiatives have sprung out of the Open Government initiative, including Recovery.gov, a site that shows how recovery funds are being spent by award recipients and Data.gov, a site aimed at increasing public access to “high value, machine readable datasets generated by the Executive Branch of the Federal Government.”

Obama’s Open Government Initiative is one example (albeit a notable one) of a broader trend taking place across both the public and private domains.  Consider the following:

• In the financial sector, the SEC rulings on fair disclosure placed a mandate on publicly traded companies, to share information in a broad, non-exclusionary manner to the general public.  All “material” information was made publicly available through EDGAR (the SEC’s public database), which instantly surfaced a large amount of financial data via a centralized, Web-accessible system.
• In the technology sector, the practice of providing some level of access to both proprietary information and proprietary applications in exchange for valuable market feedback, brand promotion (and often just dollars) has become something of a standard for larger organizations.  Google has been one of the pioneers in this area, with a variety of API-driven initiatives that have helped the development community build countless applications that would not have been viable in the past.
• In academia, there have been a variety of movements aimed at unearthing a world of student powered knowledge and information that had been previously buried.  The British Library’s EThOS for example, provides researchers with access to any thesis produced through the UK higher education system.

The question of “Why” the world is moving towards greater transparency is a topic of much debate.  The technological determinists will say one thing and the social constructivists will say another.  What they would both agree upon is that there is a movement towards greater transparency and that’s sufficient for the scope of this article.

In this section, we have examined how a shift towards openness on the institutional level has resulted in a massive unearthing of data, which brings us to our second conclusion regarding Big Data:  The broad societal shift towards greater openness and transparency, has created a flow of institutional data to the public sphere that was once hidden.