Skip to main content

Vox of Dartmouth, the College's newspaper for faculty and staff, ceased publication in February 2010. For current Dartmouth news and events, see:

· Dartmouth Now
· Periodicals
· Events Calendar

Finding Patterns in a Deluge of Data

Researchers say Process Query Systems are best approach

Dartmouth engineers George Cybenko and Vincent Berk think that PQS, or process query systems, are the way to go to make sense of the huge volume of data collected each day from sources such as computer network monitors, video surveillance cameras, financial transaction records, and databases of e-mail exchanges. The duo present their case in a paper published in the January issue of IEEE Computer, the flagship magazine of the Institute of Electrical and Electronics Engineers' Computer Society.

Vincent Berk and George Cybenko
Vincent Berk (left) and George Cybenko advocate Process Query Systems, which detect changes and irregularities in vast quantities of data, for uses such as network security monitoring. (Photo by Joseph Mehling '69)

"PQS closes the gap between gathering a tremendous amount of valuable data and figuring out what the data means," says Cybenko, the Dorothy and Walter Gramm Professor of Engineering at Thayer School of Engineering.

PQS has been an evolving algorithmic and software framework for the past few years. Cybenko and Berk say that it is a useful and incredibly powerful tool to quickly analyze credit reports for identity theft, discover attacks on computer networks, and measure activity at, say, national borders, mall parking lots, or wildlife refuge areas. According to Cybenko, "PQS can do for discrete, categorical data analysis problems what classical times series analysis did for finance and control systems where the data are numerical."

It is based on the premise that sensed environments, be they computer networks, e-mail traffic, or high-security buildings, all consist of processes with distinct states, dynamics, and observables. PQS works to detect and understand the changes or irregularities in these processes. The PQS software is easily installed with the sensor equipment to collect, monitor, and sort a great deal of data.

"I think the most interesting application of PQS to date is in network security monitoring," says Vincent Berk, research associate and lecturer at Thayer School. "Network administrators have many options when it comes to monitoring tools, however none of them are integrated; and, while all of them produce useful information, it's often in hugely impractical quantities. PQS brings together the information, and effectively focuses on the most important issues first. To my knowledge there has not been a new software technology that is this versatile since the introduction of relational databases."

This work is a project of Dartmouth's Institute for Security Technology Studies, and it is supported in part by funding from the U.S. Department of Homeland Security, Science and Technology Directorate, and the Department of Defense (DTO, AFRL, and DARPA).


Questions or comments about this article? We welcome your feedback.

Last Updated: 12/17/08