Struggling to Make Sense of Log DataInfosec Challenge: Detecting APT Threats from Logs
In an era when organizations become overwhelmed with the digital information they accumulate, knowing what to do with that massive amount of data presents a big challenge. That's true with information recorded in logs that track the flow of data through information systems.
See Also: A CISO's Guide to Communicating Risk
Collecting LogsWhat importance do you place on each of the following reasons for collecting logs? Source: SANS institute
Asked to rank the difficulty of using collected log data for specific goals, detecting advanced-persistent-threat-style malware came in first, followed by preventing incidents, supporting forensic analysis and tracking suspicious behavior such as unauthorized access and insider abuse, the survey of more than 600 IT and IT security practitioners showed.
Many organizations struggle with APT, and screening out actionable data from background noise on their networks have proved difficult.
"Even when we look at the 22 percent of respondents who are using SIEM (security information and event management systems) for collecting logs and processing them, nearly the same percentage say it is difficult to prevent incidents and detect advanced threats," wrote Jerry Shenk, the SANS Institute senior analyst who authored the report. "This similarity indicates that log and event management systems, or the way they are being used, have a long way to go in including the critical needle in a haystack that organizations need during a network crisis."
SANS found that 58 percent of organizations use a log manager to collect and analyze data; 37 percent employ SIEM systems in some capacity and 22 percent collect log data and process them exclusively with their SIEM systems.
Industry has responded, and nearly every product managing logs ships with one or more built-in processes to extract, analyze and alert organizations on data. Still, the survey revealed, a significant percentage of organizations - 22 percent - have little or no automation, mostly because of limited resources such as time and money. Respondents cited two other reasons for not investing in automated systems: the lack of management buy-in and insufficient time to evaluate the options available in different SIEM and log management products.
With or without tools, many organizations don't spend much time analyzing logs. Thirty-five percent of respondents said their organizations allot no time to less than one day a week on log analysis. The smaller the organization, the less likely they would spend on log data analysis.
"That is really not very much time spent getting familiar with logs," Shenk said. "Given the advanced threats they are struggling with, we would have expected the time organizations spend on log analysis to increase, not decrease. We cannot stress enough that the best way for organizations to quickly detect abnormalities is to gain an understanding of their baseline or normal activity by reviewing/analyzing log data on a regular basis."