big data

Chris Cook

The NSA data-collection story has prompted a lot of reporting about “metadata” – information about communications between individuals. As the FT has reported:

The practice was revealed by The Guardian, which published an order – signed by the secret Foreign Intelligence Surveillance Court – instructing Verizon to give the NSA the metadata, or information logs, for the calls “on an ongoing daily basis”.

Call logs really matter. Even if you cannot hear the calls, metadata – knowing who called whom – is massively important. It allows you to build a picture of who knows whom and how well. This sounds trivially true, but computing power means it’s extremely easy to pull out this data in real time and in great detail. And if you spot a suspicious group, you can then get another warrant to listen in on their conversations.

Here is an example: I’ve put my Facebook account details into a piece of analytical software – and, 10 seconds later, this is my life:

This is a map of who my friends are based solely on knowing which of them is friends of the others. Nothing else. Just Facebook friend lists. It has worked out that the big clusters are groups of people who know one another – so probably have something in common. The ones at the centre of the packs know everyone else around them, and those at the edge are more peripheral.

 Read more

The advent of big data is forcing visualisation on companies big and small

Some prodigiously talented individuals – such as the Indian mathematician Ramanujan, for example – have always able to extract meaning from vast arrays of data. Everyone else needs pictures.

Stephen Few, whose book Show me the Numbers* is the bible for visualisation specialists, puts it like this: “It’s almost impossible to spot and make sense of patterns without pictures. Patterns that remain hidden in a table of numbers are made visible by the right visualisation. Most models of abstract information work best with simple 2D visualisations that can be viewed and manipulated on a computer.”

The advent of big data as the product of modern business intelligence systems is forcing visualisation on companies big and small.

 Read more