Big Data And Analytics: The Hero Or The Villain?

As the NSA PRISM debacle continues to unfold and spreads across continents it’s probably good to stop and think about the technology and philosophy behind it all. Because this is big data and analytics in its most potent and controversial form, and it’s certainly not the last time we’ll see this hit the headlines.

The NSA-built Accumulo, which is a database design based on Google Big Table. Written in Java, the NSA contributed Accumulo to the Apache Foundation and in 2012 it was promoted from incubation to a top level project. Right now the agency is harvesting petabytes of data in Accumulo, a staggering amount that grows daily. But the cleverest part of all of this is in the analytics, Accumulo was built and extended the Big Table concept to analyze trillions of points in data in order to create intelligence that can detect the connections between those points and the strength of those connections.

If you thought you had it sussed with something like LinkedIn using INmaps (my network is too large to generate one funnily enough) or Facebook Social Graph, think again because through Accumulo the NSA can find out who you are, where you are, who you know and why you know them. [...]