DCGUARD: A Holistic Approach for Detecting and Isolating Malicious Nodes in Cloud Data Centers

This paper presents DCGUARD, a unified security approach for detecting and isolating misbehaving computing and forwarding nodes in multi-tenant virtualized cloud data centers. DCGUARD employs technological advancements in Virtual Machine Introspection (VMI), Software-Defined Networking (SDN), and secure probabilistic sketching to detect and isolate parts of the Virtual Machines (VMs) and network switches experiencing malicious behavior dynamically. The main contribution lies in designing a divide-and-conquer strategy that utilizes VMI and network programmability to apply focused distributed task and packet probing mechanisms on portions of the data center network rather than focusing the security functions on the entire physical network. The processing VMs and network switches are recursively partitioned into independent logical groups inspected individually to localize abnormal/malicious computing and switching nodes incrementally. This remarkably enhances the efficiency of the detection mechanisms, which opportunistically approaches a logarithmic time complexity in the number of protocol steps towards convergence (compared to a linear time complexity in traditional intrusion detection systems) when a relatively low number of hostile VMs and switches are present. Real experiments are evaluated, and a test-bed blueprint of the proposed design is emulated in a virtualized cloud environment using the Mininet emulator. The performance, convergence, and accuracy benchmarks corroborate the analytical advantage of the proposed security approach.