Around the world, we see increasing requirements for communications service providers to maintain and provide intelligence about their users. Yesterday, a bill was introduced in the U.S. House of Representatives that would require Internet service providers to retain subscriber IP address information for up to eighteen months. The stated goal of this legislation is to assist federal law enforcement in investigations into online child pornography and child exploitation cases. This is certainly a worthwhile endeavor, but the same effort could also be leveraged to assist in cyber-crime and cyber-terrorism cases.

The European Union has had this form of legislation in place since 2006 (Directive 2006/24/EC), and SenSage has helped more than twenty service providers comply with it by storing call detail record (CDR) data for up to two years as required by law. These systems have profoundly reduced the turnaround time for subpoenaed data pertinent to cyber-crime and cyber-terrorism investigations…in some cases from months to minutes or even seconds. What was a best effort process with unpredictable results, is now a proven process that law enforcement can anticipate and leverage in the context of its established checks and balances such as the subpoena process.

For the largest service providers, this means collecting billions of CDRs per day, managing petabyte-level data stores and processing thousands of queries per month with specific response time requirements. This is no small feat and leverages patented technology from SenSage, but in most cases the companies were keeping this data anyway and are now better organized in the way that they manage it.

Meanwhile, society benefits as the digital age rockets forward and law enforcement keeps pace…without slowing it down.

The newly pending legislation in the U.S. is an encouraging step towards achieving a better balance between digital lifestyles, necessary law enforcement and personal privacy. The U.S. can learn from the example set by the EU…perhaps even leapfrog it!

permalink

Verizon Business recently published its 2011 Data Breach Investigations Report (DBIR) and I am once again stunned by the apparent correlation between getting breached and myopic log management:

• 69% of the breaches had log evidence available for forensics
• <1% of the breaches were discovered by internal log analysis and/or review

The report makes a halfhearted attempt to look on the bright side: “…discovery through log analysis and review has dwindled down to 0%…so the good news is that things are only looking up from here.” CEOs reading this will not be amused, particularly if they are among the 86% of breached companies whose internal efforts failed and were notified of the breach by a third party.

When I force myself to think about what choices could lead to this miserable outcome, I come up with three types of companies:

• Type 1 companies don’t read these reports and therefore have no clue that they can help themselves (not a great place to be)
• Type 2 companies have deployed log management products and commit real resources to log analysis, but simply haven’t succeeded in this effort (a better place to be, but not great)
• Type 3 companies read these reports, know that they can help themselves, but have failed to act (probably the worst place to be)

If you are a Type 1 or if you know one, perhaps this latest report and my humble rant will help you and yours open your eyes to the reality of modern day threats, the realities of imperfect security and the necessity of log management and analysis as a compensating control. If you are a Type 2, you should challenge your log management vendor to help you succeed. Chances are, you are struggling with a log management technology that lacks flexibility to include all relevant sources, scalability to store all of the relevant data for extended periods of time, and analytical strength to help your finite staff automate exception filtering. If you are a Type 3, I can’t help you yet, but perhaps I will read more about you in the next DBIR.

permalink

SenSage recently conducted a survey of 383 information security professionals and found that two out of three had encountered obstacles to security data access and analysis while performing their security duties. This clearly validates the need for open data analysis architectures in the SIEM and Log Management market. According to the same survey, the tasks impacted by these obstacles are critical to the perceived effectiveness of log management, compliance reporting, real-time monitoring, forensic investigation and incident response processes in their organizations. I would place the impeded tasks into two groups: traditional but underwhelming and emergent but immature.

In the traditional but underwhelming category we have basic things like “trying to better understand a compliance exception or real-time console alert.” You would think that these tasks would have matured and evolved to a point of effectiveness by now but they haven’t because most SIEM and Log Management offerings do not enable the end user to drill into data behind compliance reports and real-time alerts. In the emergent but immature category, we have more holistic things like “trying to understand how a certain metric is changing over time” and “trying to compare security effectiveness across different groups or environments.” Again, we can trace the struggle here to weak data management scalability and weak data analysis capabilities of most SIEM and Log Management offerings. SenSage specializes in delivering these scalability, drill-down and trend analysis capabilities within its SIEM and Log Management offering, and is encourage by the fact that the industry is starting to acknowledge these challenges and demonstrating an interest in tackling them in order to improve their security postures.

This second annual survey also indicated minor progress in security management process evolution. Specifically, coordination across log management, compliance reporting, real-time monitoring, forensic investigation and incident response processes has improved slightly but remains a challenge. We know that process coordination is challenged by the usual “organizational dynamics” in large companies and government agencies. But we also know that data (fact) helps stimulate cooperation across teams because it cuts through subjective and political behaviors. Other findings include:

• Measurement of these processes is basically flat year over year (measurement is hard, especially when you don’t have the tools
• Consistency of process improvement has increased, but finding the resources needed to implement process improvements remains a challenge
• Perceived effectiveness of these processes has improved slightly year over year, but 57% still believe that they are infective or only somewhat effective

SenSage conducts these surveys to keep tabs on how the problem set is evolving in our market. We continue to believe – and these surveys continue to confirm – that effective data management, scalability and analysis is critical for success in proactive security organizations.

SenSage will present a webinar detailing the survey results, changes in the past year, interesting correlations and emerging use cases for data-driven security management. If you would like to attend, please register here.

permalink

There was much talk about cloud security at the RSA Conference last week. On multiple occasions, I was asked for my opinions about log management in the cloud, so I thought I would repeat my replies here. Note that I see log management in the cloud as more of an outsourcing decision than a technology decision. Overall, I see cloud-based log management services on the rise, but with a few key qualifiers that impact the forecast ahead.

The Choice Comes Down to Personality.
Just as we’ve seen with general outsourcing trends over the years, the companies that are willing to outsource log management are those that have an outsourcing personality type…willing to jettison anything that fails the “core versus context” test. While initiatives related to security have tended to be seen as “sensitive context” and therefore tend to stay in house, tight capital budgets and staffing resources have driven more companies to consider outsourcing log management. On the other side of the coin, I do not see many proactive security organizations outsourcing log management because for them, log management, exception reporting and security data analysis are core to their business or government missions.

The Choice May Be a Short-sighted One.
Some organizations see log management outsourcing as an opportunity to avoid non-compliance risks, hoping to blame the outsourcing provider as a first line of defense in dealing with lapses. This is obviously a short-sighted approach, but it happens and may in some cases deliver the intended benefits.

Outsourcing Works Best in a Vertical Context.
This principle often trumps the first two and creates a situation that makes outsourcing superior to insourcing, even for security functions. This is particularly true in the defense industry, where information access policies (e.g., Unclassified, Classified, Secret, Top Secret) and other control structures (e.g., mission, branch, command) are well established, and where defense integrators have established the individual skill sets and large-scale project management competencies necessary for success. We have also seen this in the health care market, where we have partnered with Cerner to embed our log management, audit and compliance reporting capabilities into their health care IT platform Millennium.

More in the U.S., Less in Europe.
In the U.S., we’ve seen a gradual but persistent warming to the thought of handing security data over to security service providers. Not so much in Europe, where privacy concerns and regulatory obligations make this proposition less attractive. Since the primary drivers in the U.S. are also present in Europe (e.g., compliance mandates amidst staffing and budget pressures), it will be interesting to see if the tendencies shift over time.

In summary, log management in the cloud can make sense in the right situations. Make sure the technology behind your service provider can scale to meet your needs and keeps your data separate from other customers’ data. You should also make sure that the technology will help the service provider reduce costs via data compression and storage optimization, because these advantages will help keep your provider in business or reduce the cost of the service or both. Finally, make sure that you will get standard and custom reports, the latter being delivered via a direct interface of your own. Of course, all of these requirements are satisfied by SenSage SIEM and Log Management solutions for managed security service providers. If you’d like to learn more, send us a note.

permalink

You need a SIEM.  So after reviewing data sheets, seeing a product demo, and participating in a proof-of-concept, you make your choice.

And everything is fine.  For a while.

Then you start to notice things.  Like the fact that as more data is added, queries start taking longer.  Or those custom data sources that you were told could easily be added don’t show up in the out-of-the-box reports.  Or that the product takes a lot more support than you anticipated.

A year down the road, a compliance or security incident comes to light that needs to be investigated.  Critical questions need to be answered, and they need to be answered fast, such as:

• What exactly happened?
• Has this ever happened before?
• What data was affected?
• What users are involved?
• What users/customers are affected?

Answering these questions is critical to securing your data, meeting your compliance requirements, and maintaining your organization’s reputation.  So you begin your investigation, only to find that:

• Analyzing a year’s worth of data actually requires accessing two different data repositories; your SIEM’s short-term database and its long-term “log depot.”
• The short-term database uses out-of-the-box SQL reports to extract data and returns formatted reports with column headings to help user comprehension.
• The long-term log depot uses indexed searches and returns raw data records with the matched values highlighted, requiring the user to understand the data source’s log format.
• It’s up to *you* to correlate the results from both data repositories in order to answer the remaining questions.
• All of this takes a great deal of time, effort, and knowledge.

So when you needed a single place to analyze your data, you got two different repositories.  When you needed efficiency, you got two different methods of analyzing data (sometimes three).  When you needed accuracy, you got two different levels.  When you needed understanding, you got raw log records.  And when you needed quick analysis, you got to manually correlate query results.

That’s because architecture matters.  And because you probably didn’t take that in to account in your evaluation process.

When you saw the product demo, you saw the short-term repository because that’s where the best looking dashboards and reports are.  You might not have even known at that point that there was another repository.  When you did the proof-of-concept it was only over a week so there was no need to involve the long-term log depot.  When you asked questions about retention you were told that was what the log depot was for.  When you asked about investigations, you were told that was what the log depot was for.  But you never saw it.

If you asked about integration, you were told that the short-term database could import data from the long-term log depot to allow you to do the analysis in one place.  While that feature actually exists, its ludicrous to think that you could import a year’s worth of data (or longer) into a database that has to be archived every 30 days.

That’s because architecture matters.

It matters when investigating a security incident.
It matters when trying to size a data privacy breach.
It matters when time is of the essence.
It matters when accuracy is required.
It matters when total cost of ownership is viewed beyond the implementation price.

So architecture should matter to you.

In the coming weeks I will be expanding on this theme and others relating to why SenSage is uniquely positioned to solve your most complex SIEM challenges.  But in the meantime, you can start to see how SenSage’s architecture sets it apart from the rest by starting here:

http://www.sensage.com/products/event-data-warehouse.php

permalink