Manage Big Data Like A Pro

Last time I checked, the AT&T global network was reportedly carrying over 43 petabytes of data on the average business day.  That’s what I call Big Data!  To put 43 PB into perspective, that is the digital equivalent of transmitting the U.S. Library of Congress every 12 and 3/4 minutes.  Or imagine a stack of typical CD-ROMs that would stretch over 55 miles high.

There is great value in all of this data we carry – not only for our customers and their users who are sending and receiving all of it, but also for our network operations and security personnel who are analyzing this traffic in order to help provide a safer and more reliable infrastructure for our customers.

Effective data management is critical

For AT&T, the act of carrying all of this content generates hundreds of terabytes of event and log data.  A problem so massive that AT&T Labs had to develop its own data management system, as one didn’t exist back when we started to accumulate all of this data. That system now includes our Daytona® data management system and a network visualization tool that uses powerful computer graphics and data mining to integrate and explore network information and efficiently put this knowledge to work in the AT&T Network.

Monitoring system traffic offers early warnings

All of this data and the ability to efficiently analyze it helps us create valuable models of what “good” traffic looks like, so when we start to detect deviations in these patterns, we have an early warning that something may be brewing in the way of malicious activity or exploits against our network and we can respond to and mitigate the risk, before it ever affects the performance or security of our backbone.

This approach, once reserved for carriers and network providers, is now being adopted by the enterprise. New technology and a growing list of solution providers have made it possible and more affordable for businesses to store and analyze large volumes of data generated by their own corporate network resources.  Now many companies are now putting this data to work with the intent of increasing their security posture.

Analytics improves system security

This trend in big data security analytics is becoming a popular topic of discussion as more sophisticated tools are available to combat the growing number of threats and vulnerabilities. And many of these threats are more sophisticated and multi-layered and cannot be effectively detected using traditional point and perimeter controls. The key to this latest trend is the growing ability to efficiently collect, store, and analyze a complete and comprehensive dataset and actually learn what “normal” activity looks like for your company network. Then, just like on our backbone, use that baseline to quickly analyze traffic and activity to identify potential problems before they can do any damage.

Advanced correlation engines, pattern recognition techniques and whitelisting can all work together, feeding off of terabytes of historical information, both systematically in real-time or on-demand, at the request of IT professionals running queries and correlation requests.

Start with a sound foundation

Cloud-based and mobile architectures are adding even more promise to this already intriguing topic as IT professionals look for ways to integrate these capabilities with structured data infrastructure that already exists within their organizations today.  But before you get too far down the path, remember that the foundation of any good solution is the policies and processes that it is built on. In addition, many experts caution those considering security information and event management (SIEM) solutions to not to try “boil the ocean.”  The idea of deploying a comprehensive defense system can be overwhelming, so don’t feel that it needs to be done overnight or on your own.  You may want to consider a phased implementation and you may want to illicit the help of trained and knowledgeable network and security professionals who can help you de-mystify big data and its role in securing your enterprise.

How is your company managing the security of big data? What are your top concerns­­­­­­?
The Networking Exchange Blog Team About NEB Team