Fusco, F: High-speed indexing and archival of network measur

increasingly interested in efficient solutions enabling the collection of exact data streams, and the expedient analysis of large-scale repositories of historical network measurements particularly in case of security breaches. Enabling long-term historical analysis of massive volumes of network monitoring data is required to enable forensics, anomaly detection, and information leakage analysis tasks.Advanced data collection systems are required to enable the archival of high-speed streams of network monitoring data and, most importantly, to enable fast explorations of large-scale repositories. Such systems have to support data archiving under extremely high-speed insertion rates and to produce archives still amenable to indexing and search. Current solutions that address the challenge of lossless storage of massive network monitoring data streams use off-the-shelf compression techniques, like GZIP and BZIP2. The main shortcoming of these solutions is that they do not offer efficient query processing, especially for queries targeting a small part of the dataset, as large data blocks are compressed and then retrieved using expensive decompression operations and serial scans of the archives.In this thesis, we first focus on the storage, indexing, and data querying of high-speed streams of network flow information and we propose an architecture built upon novel lossless indexing and compression algorithms carefully optimized for the network monitoring domain. The architecture is capable of compressing high-speed streams of network flow records in real-time while achieving higher compression ratios than popular general-purpose compressors, and, more importantly, produces compressed archives that support partial decompression. Then, we describe an indexing architecture for packet traces that has been integrated into libpcap, the de-facto reference library for accessing packet trace repositories.We make the following important contributions: (a) we propose a novel compressed bitmap index encoding that outperforms the current state-of-theart both in terms of CPU load and disk consumption when indexing network flow traces and packet traces, (b) we introduce an online stream reordering mechanism that boosts both compression ratios and retrieval time of modern compressors and compressed bitmap indexes, and, (c) we describe RasterZIP, a novel lossless compressor that leverages indexes for providing fine-grained decompression granularity. RasterZip achieves higher compression ratios than general purpose compressors by exploiting data patterns, such as the shared prefixes of reordered IP addresses, that are commonly present in networking data.