The NSA Isn’t The Only Organization Interested In Big Data

big data

If you thought — for even a little while — that the stuff you post online is private and confidential, then you’re probably now acutely aware that it just isn’t so. Following news that the U.S. National Security Administration (NSA) has been monitoring phone and online communications as part of the USA Patriot Act and the Foreign Intelligence Surveillance Act since 2001, many people are simply shrugging; others are getting snarky (ironically) in social media posts; some are outraged.

Getting a grip on a big data culture

Despite the range of reactions to the recent news, the fact remains: Big data analytics is a big deal these days, thanks to the exploding volume of real-time information that’s available about individuals through digital platforms. The data comes from traditional sales channels as well as the geo-placement applications on smartphones and tablets, web analytics that track online behavior, Facebook updates, and Twitter feeds, credit card transactions, utility bills, and even closed-circuit TV cameras in urban areas and on city buses, taxis, and trains. You can probably assume that you’re being watched, recorded and monitored pretty much all the time, and all the data about your life and behaviors has varying degrees of security.

Big data analytics is a bona fide game changer for cultures everywhere. In 2011 alone, the world generated 1.8 zettabytes of data; that’s 21 zeroes! “That’s roughly the equivalent of every U.S. citizen tweeting three times per minute for 26,976 years,” Bob Boehnlein, general manager for Marketing Operations at Teradata Applications, pointed out. And all types of organizations are scrambling to figure out how to harness and use all this information.

There’s just so much to gain from having accurate, up-to-date information about the people you’re trying to serve. That’s why everyone — retailers, research universities, healthcare providers, politicians, sports franchises, businesses and government agencies — are in the big data mix, trying to find out what makes people tick and how to use the data they’re getting to take care of business and thrive by working smarter. But even the world’s most powerful, profit-hungry enterprises are having a tough time making heads or tails of how to manage all that blasted data they’re gathering.

Even the folks at CERN, the European Organization for Nuclear Research — the geniuses who invented the World Wide Web in 1989 and use the world’s largest and most complex scientific instruments to study the structure of the universe — are struggling to accommodate all the data it collects. In his book Planning for Big Data, Edd Dumbill reports that CERN’s famous Large Hadron Collider system generates so much data that scientists must discard the overwhelming majority of it, hoping they haven’t thrown away the key to the universe in the process. So, seriously, how likely is it that the U.S. government has it figured out?

That’s just the thing that reassures me this PRISM thing is fine — at least for now — for the non-terrorist set. If the world’s strongest, most high performing systems cannot adequately store and process the massive amount of cutting-edge research data being collected, then the fear that anyone is monitoring “every single phone call that every American makes every day,” as expressed by Ron Wyden of Oregon according to the New York Times, seems irrational.

That level of scrutiny on what law-abiding citizens are doing online or talking about just doesn’t seem possible with current resources. Most people are simply not deviant enough to raise a red flag in the complex algorithm that alerts overworked federal authorities to look closer. I mean, can you imagine the agony of being forced to listen in or otherwise monitor every vanilla phone call within and from the United States? Or watch every video on YouTube? Or read every tweet or Facebook post? Sounds to me like a special kind of hell.

Finding the resources to leverage data

Still, big data analytics is very big business these days. The combination of widely accessible mobile technology and the Western world’s digital-obsessed culture makes it possible — and in some industries, unavoidable — for businesses to know as much as possible about their customers’ wants and needs. Consumers have come to expect real-time personalized contact on their preferred terms. In exchange, they routinely share information about themselves and interact freely in digital spaces. To survive, companies must respond.

Now, organizations of all shapes and sizes are frantically trying to find analysts and software that can manage and interpret the vast amount of customer data that’s rushing in from so many different places. Fortune recently reported that Peter Sondergaard, a senior vice president at IT research firm Gartner, predicts that the U.S. will need to fill almost 2 million data analyst positions over the next 3 years (and 4 million positions will open worldwide), generating a total of 6 million jobs by the information economy throughout the next 36 months!

To make all the effort worthwhile, enterprise businesses need to integrate and apply all their customer data to real-world solutions that improve operations and, in many cases, profitability. That’s why the world’s most successful technology companies — including TeraData, IBM, and — are snapping up customer contact expertise through strategic acquisitions; and similarly, retail powerhouses, like Wal-Mart, are absorbing big data processing companies to round out their capabilities. Smart global enterprises know that to succeed in today’s business world, they need to convert vast amounts of consumer data into a comprehensive and cohesive cross-channel brand experience for target buyers.

The information-sharing and -gathering dynamic that’s driving consumer-focused business and making modern marketing so personal, is also giving governments, researchers, educators and other interested parties an unprecedented amount of actionable intelligence. And many of these groups are trying to use this information to provide more effective public services and better meet the expectations of a highly demanding population of citizens — who continue living rather safely in the free-est world available today.