Facing the Big Data Challenge - Getting Some

When many of us hear “Big Data,” we think about the marvelous new toolsets to process it at scale. That’s great if you are one of the “Haves.” But what about the “Have Nots”? Such as the young company looking to license, curate, crawl, improve, aggregate, etc. This company is interested in guaranteed, simplified access to trusted data at an affordable price.

But it’s not just the young company. With some exceptions, I’d argue everyone is a “Have Not.” In today’s environment of accelerating consumer expectations, anyone building an app or service must continuously look for data which adds value to the end-user. In other words, they must look for access to good, structured data at reasonable terms.

The current landscape and challenges for someone looking for data are:

• Hard to find the ideal right vendor
• Often, data is not conveniently structured, adding to the cost – that is, if it’s even usable at all
• Challenges around determining quality of data
• Licensed data often quite static

Several companies, Factual being one, have been tackling these big problems, attempting to build services and components that will serve “the Have Nots.” Meanwhile, the walls between data silos are crumbling – open government data, API platforms, etc. – creating greater opportunities, as well as challenges, in this new data environment.

In my talk, I’d like to:

• discuss the current environment and the critical need for developers to find access to good, accurate, and affordable data
• explore the challenge of finding and integrating good data, and the tools and techniques for maintaining it
• discuss new algorithmic approaches to handing data, such as rules-based and machine-learned data structuring (i.e., the ability to structure data without needing a human to do it), and de-duplication algorithms for cleaning and improving data
• discuss the opportunities and challenges of enhancing data through crowdsourcing (the wisdom and enthusiasm of crowds, which we’ve seen with such projects like Open Street Maps), as well as through other open data models

My goal with this talk will be to inform the Web 2.0 audience of the current state of things, the challenges and possibilities ahead, and the efforts many of us are currently engaged in to help the “Have Nots” – at least when it comes to data – join the ranks of the “Haves.”