FBI shows off huge, multi-source linked database

According to the Washington Post, the FBI has just provided a demonstration of a new ginormous database to help the war on terror. Here is an extract (I have some comments below):

FBI Shows Off Counterterrorism Database

The FBI has built a database with more than 659 million records -- including terrorist watch lists, intelligence cables and financial transactions -- culled from more than 50 FBI and other government agency sources. The system is one of the most powerful data analysis tools available to law enforcement and counterterrorism agents, FBI officials said yesterday.

The FBI demonstrated the database to reporters yesterday in part to address criticism that its technology was failing and outdated as the fifth anniversary of the Sept. 11, 2001, terrorist attacks nears.

Privacy advocates said the Investigative Data Warehouse, launched in January 2004, raises concerns about how long the government stores such information and about the right of citizens to know what records are kept and correct information that is wrong.

The data warehouse is an effort to "connect the dots" that the FBI was accused of missing in the months before the 2001 attacks, bureau officials said. About a quarter of the information comes from the FBI's records and criminal case files. The rest -- including suspicious financial activity reports, no-fly lists, and lost and stolen passport data -- comes from the Treasury, State and Homeland Security departments and the Federal Bureau of Prisons.

"That's where the real knowledge comes from . . . sharing information," said Gurvais Grigg, acting director of the FBI's Foreign Terrorist Tracking Task Force, who helped develop the system.

In a demonstration, Grigg sat at a computer and typed in the name "Mohammad Atta," one of the 19 hijackers in 2001. The system can handle variants of names and up to 29 variants on birth dates. He typed "flight training" in the query box and pulled up 250 articles relating to Atta.

Wonder what happens if you type in "J. Edgar Hoover" and "pink tutu"?

... No top secret information is in the system, officials said....

Perhaps I am being overly cynical, but wouldn't you think that in order to be effective, the database should contain top secret intelligence? Surely the results of an interrogation of an insurgent in Afghanistan or the wiretapping of a radical in England would be "top secret" but highly relevant.

If you are going to mung together a big whack of databases to provide some actionable intelligence, shouldn't it include the right databases? Wouldn't that include "top secret" intelligence databases? Wouldn't that help avoid some of the "intelligence failures" that have been fought over the last few years?

... Irrelevant information can be purged or restricted, and incorrect information is corrected, he said. Willie T. Hulon, executive assistant director of the FBI's National Security Branch, said that generally information is not removed from the system unless there is "cause for removal."

Every data source is reviewed by security, legal and technology staff members, and a privacy impact statement is created, Grigg said. The FBI conducts in-house auditing so that each query can be tracked, he said.

David Sobel, senior counsel of the Electronic Frontier Foundation, said the Federal Register has no record of the creation of such a system, a basic requirement of the Privacy Act. He also said the FBI's use of an internal privacy assessment undercuts the intent of the privacy law.

FBI officials said the database is in "full compliance" with the law.

Sobel said he learned under a Freedom of Information Act disclosure last week that the system includes 250 million airline passenger records, stored permanently.

"It appears to be the largest collection of personal data ever amassed by the federal government," he said. "When they develop the capability to cross-reference and data-mine all these previously separate sources of information, there are significant new privacy issues that need to be publicly debated."