Cooperating Detectors

A malware detector tries to determine whether a program is malicious (examples
of malicious programs are drive-by-downloads, botnets, and keyloggers).
Malware detection is primarily performed at two vantage points: host and
network. This post explains why cooperation between host-based and network-
based detectors is a good thing.

Traditionally, detection has been performed either at the network or host level, but
not both. First, let me examine both approaches separately.

A network-based detector monitors events by examining a session or
network flow and tries to determine whether it is malicious. The
advantage of a network-based detector is ease of deployment -- there
are not that many points of deployment for a network-based detector
(typically they are deployed behind border routers).

Unfortunately, network-based detectors have a limited view of each
network session. In fact, if a session happens to be
encrypted such as is common with VPNs, Skype, and some bots, a
network-based detector is essentially blind. For example, a botmaster
can hide its communication with the bots by simply encrypting the session.

By contrast, host-based detectors have a more comprehensive view of system activities, i.e.,
they have the potential to observe every event at the host, including malicious ones. However, the major drawback of a host-based detector is that it has to be widely deployed. Typically in a managed
network (such as in an enterprise), a host-based detector has to be deployed at
every host.

Cooperation between host-based and network-based detectors can potentially
address the shortcomings of each detector. I've come up with three possible scenarios.

1) Host-based detector helping the network-based detector.
A network-based detector can pull alerts from a host-based
detector and a host-based detector can push alerts to a network-based
detector. This is a simple solution and I suspect the easiest
scenario for cooperation.

2) Queue up suspicious activity on a virtual machine.
If a network-based detector determines that a session is
"suspicious," it can divert the suspicious traffic to a virtual machine
with a host-based detector for more in-depth analysis. The trick here
is figuring out what events are indeed "suspicious" (you do not want
too much traffic to go through the "slow path" corresponding to a
host-based detector). There is already a
startup called Fireeye adapting this solution. I find this line of work quite intriguing.

3) Pushing signatures.
This third scenario has been explored quite thoroughly in academic
literature. It involves the cooperation of host-based and network-based detectors to
push signatures for malware in real-time. For example, if a host-
based detector recognizes an attack, it pushes out a signature to a
network-based detector. The advantage of this
approach is that, by updating a network-based detector, an entire
enterprise can be protected against that particular threat. However, in my
view this is not a good approach in the long run. Hackers are creating malware variants at
an alarming rate and signatures won't be able to keep up.


Popular posts from this blog

Malware Trend in 2007

Web Attacks