In his most recent blog (http://www.sensage.com/blog/category/jim-pflaging-ceo-blog/), Jim Pflaging introduced a very exciting concept – Security Intelligence – and talked about how our most advanced customers are leading the charge in this improved approach to security and compliance management. I think it’s really important to emphasize the process aspect of this pursuit. We get pumped up about the technology because of all it can do, but organizations need to evolve their processes to get the most out of this technology. Even if they don’t consolidate the different groups involved with security (i.e., corporate security, security operations, IT operations, forensic investigation, audit/compliance and risk management), they can improve their coordination across these groups to reduce duplicated effort, human error and incident response times. The SenSage Professional Services team has been helping our customers tackle this sort of process improvement for over five years now and has codified its findings in the Security Intelligence Capability Maturity Model. The model provides a practical methodology to prioritize, plan and measure results in security and compliance improvement efforts, and will be a regular topic in our future blogs. We will be demonstrating the model and our Security Intelligence solutions in our booth #845 at RSA this week. If you’re at the show, stop by to learn more and share your perspective!

permalink

In past blog posts I have often cited the need for a scalable event data warehousing capability to keep up with data collection and analysis requirements to address compliance and security operations. After hearing from dozens of customers about how they’re using SenSage to address their most critical security and compliance challenges, I’ve decided to focus less on event data warehousing and more on how our customers and partners are using SenSage. Towards the end of 2009, we searched for a way to net it out. In the end, it was pretty easy - Security Intelligence. This term sounds lofty at first, but once you learn how we think about it, I think you will find it very down to earth.

Of course, Security Intelligence is a variation of Business Intelligence or BI. BI solutions leverage the data management capabilities provided by data warehouses to deliver decision support information to business managers. Well, that’s exactly what Security Intelligence provides: essential decision support for security, risk management and compliance operations. Done right, Security Intelligence solutions are open, flexible, and scalable like traditional data warehouses while delivering deep security context.

Improved decision support is exactly what today’s security, risk management and compliance professionals are looking for. Detection and response to cyber-threats, regulatory compliance risks and investigating system failures all require thorough but simplified analysis of massive amounts of event data. Whether responding to an incident in real time or drilling through terabytes of related events to investigate the related context or improving a control, security professionals are asking for better decision support solutions.

As compared to Business Intelligence solutions, this is a bit of a niche play. These solutions are tailored to meet the needs of security, risk management and compliance professionals. But compared to the traditional SIEM and log management point products which are built on flat files, Oracle, or, worse, closed database management systems, Security Intelligence is a more flexible and sustainable approach.

SenSage is at the forefront of this technology, delivering Security Intelligence solutions that unify SIEM, log management and controls monitoring through a single analytics environment and data management architecture. Our customers are capturing the benefits of decision support in the security management context, leading to technology consolidation and process improvements not easily accomplished with the point products noted above.

We’ll be talking about Security Intelligence quite a bit in the coming months. Drop me a line, I’d love to hear your perspective.

permalink

These days it seems everyone wants to talk about life in “real-time”. Last week, the San Jose Mercury News, ran a piece called the “Real-Time Web, the valley’s new obsession” (http://www.mercurynews.com/business/ci_13342816). The main theme of the piece, “What’s fast enough?” is thought provoking. In particular, I loved the trendspotter debate about whether Twitter was “real-time” or “near real-time”.

My reaction to the article?  I high-fived my son at the breakfast table. You see, for those of us in the security and data warehousing world, it’s great to see a pervasive, general business debate about the value of real-time analysis — issues we’ve been dealing with for years. Particularly in security, we’ve been debating (or some may say splitting hairs) about “What’s fast enough?” for years. To us, the conversation isn’t new at all; it’s based on real technology and real needs. It is evolving however, from a focus on “real-time vs. near real-time” to one focused on “real-time and all-the-time”.

Why? Five main issues are driving this evolution:

1.  New battles. Speed and accuracy are essential when it comes to new battles such as cyber threats and monitoring core intellectual property. For instance, in markets like healthcare and finance, the cost of a missed security breach can extend to irreparable financial or reputation loss. In defense, it can mean lives are lost.

2.  New data. The key to responding to these new battles is event data — time-stamped, append-only data — and it’s the fastest growing data on the web.

3.  New requirements. Proper response to these battles starts with detecting threats from terabytes of events as they are occurring – in real-time. Proper response extends to analysis of years of collected event data. In some cases, this means sifting through hundreds of billions of records, to find fraud, criminal activity, or, simply, errors. In each case, the need for wickedly fast and complete response is essential.

4.  New technologies. No surprise, vendors are stepping up with solutions to address this large, new market. SIEM and log management firms have been at this for years. More recently, data warehouse firms are joining the fray: Teradata with the launch of their Extreme Data Appliance 1550 and Netezza with their Mantra Compliance Appliance. Set Google Alerts for event data, log management, or real-time analytics and you’ll see what I mean. The vendors are coming.

5.  New expectations. I think the Merc article nailed this one. Thanks in part to the “instant on availability” of social media, SalesForce.com, Zillow, and dozens of other SaaS apps -we’re quickly getting trained to expect results without time-out for manuals or training sessions. In short order, these same expectations will be “table stakes” for business intelligence and security applications.

Last week, I was at an industry gathering hosted by Sierra Ventures – their annual CIO Forum. A few things stood out. The first was the CIO’s peer discussion about delivering “real-time analytics” and self-service to their business users. The second was Joe Tucci’s keynote session about the next wave of IT. In his talk, he stressed the importance of speed and self-service enabled by cloud computing as being at the heart of the next wave of IT. He said the change will be bigger than anything we’ve seen, will give unprecedented power to users, and will have a huge mortality rate for those who can’t adapt.

This echoes what we’ve heard from our customers.  They’re telling us this new wave can’t hit “fast enough”. They’re in the game and deploying solutions to make this vision of “Real-time, All-the-Time” a reality.

permalink

Saw an interesting column in InfoWorld on “Learn to Love Your Log Files” -  http://tinyurl.com/lvhabg

The author, Roger Grimes, highlights a theme that is increasingly getting increased attention – the value of log files.  The article gives practical ideas for implementing and managing log management systems.  He also provides an interesting perspective on how SIEM and log management technologies fit together.

In my opinion, SIEM originated with the vision to be the single-pane of glass – to separate the signal from noise.  From an architectural perspective, data management was generally an afterthought.  Events were normalized and data discarded after a few weeks.  As a result, the initial wave of vendors built their solutions around familiar data management systems such as Oracle databases or flat files.  Over time, the reporting requirements became more demanding and the amount of data to be analyzed increased significantly.

The pendulum has shifted – data management is a central buying criterion for a logging or SIEM solution. Compliance might have started this trend, but now security is giving it the next push.  Why?  Threats are more sophisticated.  Insiders don’t generate failed logons.  So, you need to keep months of valid session detail if you want to find the low and slow anomalies.  In order to keep up with these demands, many customers are expanding their data retention period as well as the scope of data analyzed to include ERP applications, credit card and ATM transactions….their most sensitive data.

The implication of these trends is massive data stores and more sophisticated data analysis – even for small firms.  Log data repositories can easily reach into the 10s of terabytes for small firms and hundreds of terabytes for larger firms.  It’s no surprise that for many organizations, security and event data is their largest single data store.  As a result, customers are looking at long-term ROI and are pulling their enterprise data warehouse architects into data governance & compliance efforts.

Today, people from diverse roles across the enterprise need immediate access to security and GRC information.   Having said that, you can’t trade off accuracy and completeness for ease of use, and it has to be tamper-proof.  The implication is you need a system that is easy to use AND provides reports and trending information that is 100% accurate.  That’s why some vendors who claim to be “Google for Logs” (fast but not 100% accurate) will have difficulty addressing the reporting, forensic, and retention requirements of the log management market.

Check out the article - another good contribution to the conversation.

permalink

I’ve been noticing a lot of discussion online about MapReduce and Hadoop recently. While MapReduce may seem new, implementations have been around for years. Let’s take a closer look.

MapReduce is a software framework introduced by Google to support distributed computing for large data sets on clusters of computers. The objective of MapReduce is to get extremely fast answers from massive amounts of data. In the “Map” step, the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker nodes process the smaller problem, and pass the answers back to its master node. In the “Reduce” step the master node then takes the answers to all the sub-problems and combines them to get the answer to the problem. One example of MapReduce is the Apache project Hadoop, a widely used open-source implementation of MapReduce.

So are these really new concepts? Not really. Some database systems with MPP architecture have been doing this for quite a while. While MapReduce is powerful, one of its drawbacks has been that each step of the MapReduce operation (filtering, grouping, and aggregation) is a separate, high-level programming abstraction that needs to be maintained by a developer and thereby increases data management total cost of ownership.

SenSage has been providing MapReduce capabilities with “in database” analytics commercially available since 2004. You might be saying, “yeah right”. Well, it’s true. We have over 400 deployed customers and patents to back it up.

We’ve simplified the promise of MapReduce. Namely, we’ve eliminated the hassle of intermediate programmatic effort to produce lightning-fast, in-memory analytics. SenSage combined a few pieces of our intellectual property with our MPP share nothing architecture to solve the problem:

• First, the SenSage columnar database supports parallel transformation and partitioning of data. In SenSage, SQL Map is like the group-by clause of an aggregate query. Reduce is analogous to the aggregate function (e.g., average or sum) that is computed over all the rows with the same group-by attribute.
• Second, since day one, SenSage has allowed users to write their own functions in SenSage SQL, which are automatically enabled for parallel execution using our MPP architecture. With Google, Hadoop, and many others, users have to write and maintain their own programs to accomplish the same thing.  With SenSage, users write standard SQL and SenSage does the rest.
• Third is “IntelliSchema” – this is where it gets really cool. This is a SenSage innovation that is an abstraction layer between the original data and the analysis tools, and enables our MapReduce engine to execute queries successfully even if the underlying data schema changes. Intellischema gives our customers the ability to handle a wide variety of data sources and write standardized libraries of analytics while still maintaining the fidelity of the original event data.  This allows any data source to automatically appear in relevant queries and reports.

It’s good to see technologies like MapReduce getting attention in the marketplace. As customers better understand the benefits, they can make more informed buying decisions.

permalink