Business and organizations are digitally transforming, this spans from automated customer support systems to the intelligent business analytics. The interconnectedness and the use of disruptive technologies like IoT and cloud are enabling the business to grow faster and be competitive. The mammoth of data generated by IT infrastructure, operational technologies, and IoT is utilized for business analytics, improving operational efficiencies. At the same cyber-attack vector is increasing and cybercriminals are making use of exposure of data to launch high profile attacks and steal the data without getting noticed by cyber defense systems for months together. The following diagram shows the biggest data breaches of the 21^st century. (Source CSOonline)

This indicates the security industry as a whole is not catching enough of the threats and not catching them fast enough, the average dwell time is 99 days as per the FireEye M-Trends 2017 report.

Large volumes of security log data at an accelerating rate needs to be analyzed, This raises multifaceted questions :

• How much data and from which source?
• In what structure and format?

The answers to these questions are needed should be accurate and timely to make beneficial trade-off decisions for earliest identification of  the security incidents, planning for mitigation and containment.

Big data analytics is going to answer these questions.

Without big data analytics, companies are blind and deaf, wandering out onto the Web like deer on a freeway.
Geoffrey Moore, Author of Crossing the Chasm & Inside the Tornado

What is Big Data?

There is a misconception that large amount of data like petabytes is big data. The big data as defined in the early 2000s by industry analyst Doug Laney which is now-mainstream is:

• Organizations collect data from a variety of sources, including business transactions, social media, and information from sensor or machine-to-machine data. In the past, storing it would’ve been a problem – but new technologies (such as Hadoop) have eased the burden.
• Data streams in at an unprecedented speed and must be dealt with in a timely manner. RFID tags, sensors, and smart metering are driving the need to deal with torrents of data in near-real time.
• Data comes in all types of formats – from structured, numeric data in traditional databases to unstructured text documents, email, video, audio, stock ticker data and financial transactions.
• In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. Is something trending on social media? Daily, seasonal and event-triggered peak data loads can be challenging to manage. Even more so with unstructured data.

This means big data provides many inputs if analyzed using latest analysis techniques aided by Machine Learning, can provide answers to:

• Security operations cost reductions
• Reducing threat detection and analysis time
• Developing new product and process to optimized offerings
• Informed and Smart decision making.

“Information is oil of 21^st century and analytics is the combustion engine” – Peter Sondergaard, Gartner Research

Security Operations: Where and How Big Data Can Help?

Most security professionals do not yet fully understand big data, nor how to apply it to their own requirements, so they are unable to weed through all the hype and remain reserved to embrace the technology.

Big data analytics in security involves the ability churn out massive amounts of digital information to analyze, visualize and draw insights that can make it possible to predict and stop cyber-attacks.

When big data is combined with high-powered analytics, following security operations related tasks can be performed:

• Determining root causes of failures, issues, and violations in near-real time.
• Recalculating entire risk portfolios in minutes.
• Detecting fraudulent behavior before it affects the organization.
• Maintaining Visibility “Visibility” to detect, alert on, and investigate attacks by “seeing” the malicious cyber activity.

The diagram below depicts the how the reactive threat analysis can be converted into predictive, actionable threat analysis by utilizing big data and machine learning based analytics.

Need for Big Data Analytics based Security Tools

The current tools like SIEM are not sufficient when it comes to real-time processing of tremendous events logs generated by various log sources including Applications, storage, networking, security, Mobile and IoT devices, cloud platforms and SaaS solutions, etc.  The current SIEM technologies have limited capability to compare identified anomalies to the past events data and identify patterns to detect the threat. SIEM technologies need to be updated with new and different analytics techniques to detect advanced threats.

How Can Big Data Analytics Help?

When it came to threat analysis and support for security operations, we need a big data platform to specialize in analysis and analytics

• The inherent capability of big data for scaling and advanced analytics can be applied to threat detection. Few leaders in security analytics area are currently using it to detect security threats, address scalability requirements for IT event management, and help gauge the effectiveness of other security investments
• Big data natively address ever-increasing event volume and the faster rate at which the new events need to be analyzed. This holds promise for security intelligence, both in the varied ways, it can parse information and by its native capabilities to sift proverbial needles from the monstrous haystack of log data.
• Threat intelligence is truly proactive for assessing the security posture and a threat hunting function. This helps to develop and maintain a baseline threat profile. This threat profile is the data-driven mechanism (as opposed to fear, uncertainty, and doubt) that informs the business and the security practitioners of the most likely who, what, where, when, and how of attacks and the best way to begin looking for them. By constantly assuming compromise in the most likely areas of the infrastructure, organizations can focus their hunting and provide an accurate and factual answer to the question at the top of every executive’s mind: “Am I compromised?”