White

Paper: Leveraging Web Intelligence to Enhance Cyber Security

October 2013

Inside:
New context on Web Intelligence The need for external data in enterprise context Making better use of web intelligence

CTOlabs.com
Web Intelligence: A new category of actionable information
Web Intelligence is the parsing of millions of sources of Internet connected information in a way that is useful to decision-making. It enables the harnessing of the global information grid and adds predictive power to functions such as strategy development, investment decisions and risk assessment/mitigation. This paper, sponsored by Recorded Future, examines this new category of Web Intelligence in a cyber defense context and provides information you can use in deciding the best ways to integrate Web Intelligence into enterprise cyber security operations.

Our Insights into Web Intelligence:

The lead author of this paper led some of the first contributions of all source intelligence to cyber defense in the US Department of Defense and has been an active contributor to the cyber security and technology communities for two decades. For the last four years, the research team at CTOlabs.com has been contributing to studies and analysis and community events on cyber security operations, security technology and analytical tools. We interact with the community through our blog and newsletters, including daily and weekly newsletters tracking cyber security and analytical tools. We leveraged our background in cyber intelligence and technology in producing this assessment. We also checked our assumptions by asking for inputs from a range of enterprise CISOs in the financial, manufacturing and retail sectors.

Web Intelligence and Cyber Security

Web Intelligence can significantly enhance enterprise cyber security operations. In a cyber context, web intelligence is being used to track vulnerabilities being discussed in hacker channels and exploited in successful attacks. Web intelligence also portrays information on the nature of malicious code and its mitigation strategies. Further, it is a means of tracking the technologies and tactics being employed by attackers, as well as the proven best practices being applied to mitigate threats. It is in this last category of information that web intelligence is making its most unique contributions to cyber defense. Web intelligence is bringing new insights into the identity, motivation and intentions of threat actors, and it is doing so in ways that can contribute to predictions of future behavior. Since Web Intelligence can provide enhanced information on threat intentions it enables a shifting of cyber defense to more proactive strategies. For example, information on past behaviors of cyber actors associated with real-world events can lead to predictions on future behaviors associated with coming events. This can lead to predictions of when to expect DDoS 2

Web Intelligence for Cyber Security Operations

attacks or when to expect more focused phishing attacks. In some cases it can also lead to predictions on the nature of the deceptive content that can be used in phishing attacks. With more precise insights, action can be taken to mitigate threats before they strike. Web intelligence also makes critically important contributions to the issue of assessing who is attacking and why. More refined assessments on this critical element can contribute to assessments of an adversarys next step. Web intelligence can help defenders assess whether an attack is hactivism or something more sinister. It can also help in assessments of whether or not others will be targets in particular business partners such as suppliers or customers - and if a more collective defense will need to be mounted.

Web Intelligence from Recorded Future

Recorded Future is a web intelligence company. Their mission is to harness open web sources that publish open information on the web for analysis. They create insight in support of government missions and business decisions.

Recorded Future and their Temporal Analytics Engine organize web information for analysis to yield new insights. Recorded Future specializes in analyzing human writing to detect events, actions and descriptions of actions and then place this information in a time-based (temporal) context. These timelines and topics can be aggregated and correlated to ensure information on the same event can be viewed by multiple angles. This enables analysis in the light of all related information, including historical information. Recorded Future ingests, in real time, over 300,000 real time sources, performing over 50 extractions per second and building a deep history at the same time. They have already amassed a fact based of over 5 billion facts in multiple languages including English, Chinese, Russian, Arabic, Farsi, Spanish, and French.

CTOlabs.com
Background: The Roots of Web Intelligence
The origins of web intelligence for cyber security can be traced to the beginning of organized enterprise cyber security activities that began after the famous Morris Worm of November 1988. In the worms aftermath, responders noted shortcomings in their ability to know information from outside their organizations. Since then: Most major organizations have established dedicated efforts to stay informed on external threats. There has been an explosion in original content publicly available on the web, including blogging, niche publications, social media, but also vast stores of commercial data that were once locked away and inaccessible to others. Increasingly, both threat actors and defenders are openly sharing valuable information on open source web channels, making totally new sources of information available

The Use of Web Intelligence For Cyber Security Today

CISOs who leverage Web Intelligence for Cyber Defense are finding far more utility technical feeds of vulnerabilities and attack signatures. Advanced streams of information on adversaries and their intentions, correlated and assessed, can now be provided in a context ready for use by enterprise cyber security teams Most CISOs we spoke with are in the process of enhancing their ability to use web intelligence, and we believe this will be a high growth segment of the security technology portfolio in all major enterprises. Web Intelligence can contribute to dedicated cyber security efforts by parsing and correlating millions of data sources relevant to computer security. Succinct articulations of threat actors, their capability, history and intentions can be presented along with dynamically updated information on vulnerabilities and methods required to mitigate vulnerabilities. This can all be presented in conjunction with dynamically updated information on international and

Web Intelligence for Cyber Security Operations

regional events that may trigger cyber security events. This automated extraction and presentation of knowledge is already contributing to the situational awareness of several global industries and is now available for general use by cyber defenders everywhere.

Web Intelligence and Enterprise Security Management Suites

Recorded Future provides a means of interacting directly with data and analysis on global events, including cyber security focused information. However, the capability can be even more impactful when considered in the light of existing enterprise capabilities. We believe most enterprises will want to find the optimal connection between their existing security information management systems and Recorded Future. Fortunately, modern security solutions provide data integration APIs to get data in and out. The following provides some context on how Recorded Future fits in the context of major security suites: Tool HP-ArcSight Capability
Focused on logs and events but connectors to Autonomy and Hadoop show potential for future all source capabilities

Web Intelligence Integration

Information from Recorded Future can be easily moved to ArcSight and feeds from ArcSight can be moved back. This later path is being used by enterprises to establish an analytical SIEM that is strong at correlating SIEM incidents with other threat feeds (including Malware IPs, Vulnerabilities, threat intel etc). This can help rapidly prioritize event response. McAfee has always stressed interoperability in their solutions and the ESM architecture allows easy import and export of data. However, we have no examples of the use of ESM as an analytical SIEM or in support of one. Splunk has had strong import and export capabilities in place since their first offering, and these can be automated as desired. The dashboarding capabilities of Splunk can be used as all source displays of information, potentially including interactive connections to Recorded Future. The powerful tools for analysis of ongoing and past events leverage very large datastores and are designed to provide analysts easy ways to export data and analysis. This enables the use of data from NetWitness to power analytical SIEM applications. This can be a powerful contribution to forensic analysis. We are not aware of a smooth way to move information out of the Q1 Radar architecture, however exports based on user-selected criteria can be done. No indications of all source capabilities in future roadmap. Unique clustered columnar database is not designed for use by other systems, but exports of selected information can be made.

McAfee ESM

DPI and log data. Database monitors. No all source capabilities. New release provides speed and scale and ability to add external threat feeds, showing potential for integrating Web Intelligence. Dashboarding capabilities important. Leveraged for log based and network data analysis

Splunk

RSA NetWitness

IBM-Q1 QRadar

Log and event management with behavior analysis. Netflow data a strength. A purpose-built big data SIEM tool. Ability to take data feeds and integrate other information shows promise.

Sensage

CTOlabs.com
Most enterprises are also leveraging link analysis and related investigative tools, including IBMs Analyst Notebook (which is ubiquitous), and the rapidly proliferating Maltego. Some use the advanced capabilities of Palantir. Users of current versions of these systems can rapidly and easily move information to and from advanced web intelligence platforms like Recorded Future.

A User Look at Web Intelligence

Web Intelligence feeds can be presented in interactive interfaces that enable rapid assessment of dynamic information. Interfaces of Recorded Future offer analysts a means interacting with data and forming hypotheses and conclusions quickly. Analysts are presented with polished and sophisticated ways to interact with large stores of correlated and assessed information. Recorded Future also enables direct access to specialized modeling and visualization of events in time and over geography, while still enabling drill-down into sources of any data. The Cyber Intelligence Application on the Recorded Future Enterprise Platform is delivered via software as a service. This simple account-based access to the platform gives access to the full power of Recorded Futures understanding of Internet connected information

Web Intelligence for Cyber Security Operations

Optimizing the use of Recorded Future for Web Intelligence in Support of Cyber Operations
The new field of Web Intelligence is already providing actionable information relevant to cyber security professionals. Recorded Future provides the only automated solution in this space that is capable of ingesting, in real time, the right security related information from the Internet. Their fast and valuable information feeds fill a gap.

Our recommendations:
1. Establish your enterprise vision for the use of Web Intelligence in support of your security posture. 2. Launch a proof of concept leveraging Recorded Futures Software as a Service cyber intelligence application. This application enables rapid delivery of capability that can put Web Intelligence to use in your enterprise almost instantly. During the proof of concept formulate evaluations on criteria like: a. Ability to meet your vision for web intelligence support to cyber operations b. Ability to leverage the full spectrum of intelligence information from the Internet and your internal sources c. Ability to enable shared situational awareness across all levels of your organization d. Ability to drive proactive mitigation of threats.

More Reading
For more federal technology and policy issues visit: CTOvision.com- A blog for enterprise technologists with a special focus on Big Data. CTOlabs.com - A reference for research and reporting on all IT issues. FedCyber.com Focused on federal cyber security J.mp/ctonews - Sign up for technology newsletters including the Security Technology Weekly.

About the Author

Bob Gourley has been active in the cyber defense community since 1998, specializing in intelligence support to cyber operations. He is CTO and founder of Crucial Point LLC and editor and chief of CTOvision.com He is a former federal CTO. His career included service in operational intelligence centers around the globe where his focus was operational all source intelligence analysis. He was the first director of intelligence at DoDs Joint Task Force for Computer Network Defense, served as director of technology for a division of Northrop Grumman and spent three years as the CTO of the Defense Intelligence Agency. Bob serves on numerous government and industry advisory boards. Contact Bob at bob@crucialpointllc.com

For More Information

If you have questions or would like to discuss this report, please contact me. As an advocate for better IT use in enterprises I am committed to keeping this dialogue up open on technologies, processes and best practices that will keep us all continually improving our capabilities and ability to support organizational missions.

CTOlabs.com