Las Vegas Sun

June 4, 2012

Currently: 90° | Complete forecast | Log in

U.S. government seeks to mine social media to predict future

Associated Press

Sunday, Feb. 12, 2012 | 8:40 a.m.

The U.S. government is seeking software that can mine social media to predict everything from future terrorist attacks to foreign uprisings, according to requests posted online by federal law enforcement and intelligence agencies.

Hundreds of intelligence analysts already sift overseas Twitter and Facebook posts to track events such as the Arab Spring. But in a formal "request for information" from potential contractors, the FBI recently outlined its desire for a digital tool to scan the entire universe of social media _ more data than humans could ever crunch.

The Department of Defense and the Office of the Director of National Intelligence also have solicited the private sector for ways to automate the process of identifying emerging threats and upheavals using the billions of posts people around the world share every day.

"Social media has emerged to be the first instance of communication about a crisis, trumping traditional first responders that included police, firefighters, EMT, and journalists," the FBI wrote in its request. "Social media is rivaling 911 services in crisis response and reporting."

The proposals already have raised privacy concerns among advocates who worry that such monitoring efforts could have a chilling effect on users. Ginger McCall, director of the open government project at the Washington, D.C.-based Electronic Privacy Information Center, said the FBI has no business monitoring legitimate free speech without a narrow, targeted law enforcement purpose.

"Any time that you have to worry about the federal government following you around peering over your shoulder listening to what you're saying, it's going to affect the way you speak and the way that you act," McCall said.

The FBI said in a statement to The Associated Press that their proposed system is only meant to monitor publicly available information and would not focus on specific individuals or groups but on words related to criminal activity.

Analyzing public information is nothing new in the world of intelligence. During the Cold War, for example, CIA operatives read Russian newspapers and intercepted television and radio broadcasts in hopes of inferring what Soviet leaders were thinking.

But the rise of social media over the past few years has dramatically changed both the kinds and amount of freely available information. For example, Twitter CEO Dick Costolo said at a recent conference that users of the micro-blogging service send out an average of one billion tweets every three days.

"It really ought to be the golden age of intelligence collection in that you've got people falling all over themselves trying to express who they are," said Ross Stapleton-Gray, a former CIA analyst and now a technology consultant who advises companies on security, surveillance and privacy issues.

As a staffer in the early 1990s in what later became the Office of the Director of National Intelligence, Stapleton-Gray said the U.S. intelligence community's early efforts to better harness the increasing volume of information becoming available on the Internet ran into resistance from old hands who believed that secrets were more valuable than the information anyone could get.

But agencies' requests for better social media tools indicate that resistance has wilted.

The system sought by the research arm of the national intelligence director's office would fuse together everything from Web searches to Wikipedia edits to traffic webcams to "beat the news" by predicting major events ranging from economic turmoil to disease outbreaks.

The Defense Department's tool would track social media to identify the spread of information that could affect soldiers in the field and also give the military ways to conduct its own "influence operations" on social networks to counteract enemy campaigns.

The intelligence director's office and the Defense Department said they could not meet the AP's deadline to answer specific questions about the proposed projects.

The FBI is seeking a web app that would automatically scrape social networks for data that could alert the agency's operations center to breaking crises as they happen and plot them on interfaces like Google Maps

For such systems to work well, their developers would have to overcome several technological challenges, the easiest of which is handling the massive amount of data involved.

Developments in so-called "cloud computing" have made processing big data sets easier than ever before by spreading the work broadly across networks of computers.

Instead, experts in the field say the major hurdle is in effect teaching computers how to read. To sift the valuable information from the mundane, the software must understand the subtleties of meaning in tweets and blog posts to tell the difference between, for example, a serious statement and a joke.

Solving such problems falls to researchers in fields such as natural language processing and computational linguistics _ the same specialties that brought the world the iPhone's Siri voice-activated assistant and IBM's Watson, which trounced its human opponents at Jeopardy.

San Francisco-based Linguastat Inc. worked with the Centers for Disease Control during the 2009 swine flu outbreak to track public fears and concerns on social networks and determine whether the CDC's public health messages were gaining traction. Company co-founder John Pierre said that tracking public sentiment depended on much more than searching social media for specific words or phrases.

"Just because they mention it, do they like it, do they not, are they saying it in the right context? Is it a band called The Swine Flu?" Pierre said.

Authenticity also becomes an issue in analyzing social networks. Computer programs known as "bots" already plague services such as Twitter with junk posts similar to email spam. Researcher Tim Hwang has scripted his own bots to see how much influence they could wield over social networks and says the ability to create bots that closely mimic humans will only improve over time.

This matters in intelligence gathering because bots could fool analysts _ and their software _ into thinking they're witnessing a genuine shift in social trends that in reality could be a government propaganda campaign driven by, for example, Twitter users that don't really exist.

"We have all the data. How do we know what's real and what's not?" Hwang said.

William McCants, an analyst at the Center for Naval Analyses and a former State Department official, monitors al-Qaeda propaganda online. He said he worries that the systems the FBI and other agencies are seeking could create an overreliance on technology at the expense of carefully trained human analysts who are still better at zeroing in on the facts that matter most.

"The more data you use and the more complicated the software, the more likely it is you will confirm a well-known banality," McCants said a friend likes to joke. "You didn't need to be on Twitter to know that a revolution was happening in Egypt."

___

Online:

Proposed FBI social media app: http://bit.ly/AF17HJ

___

Marcus Wohlsen can be reached on Twitter: http://twitter.com/MarcusWohlsen

Discussion: 4 comments so far…

Comments are moderated by Las Vegas Sun editors. Our goal is not to limit the discussion, but rather to elevate it. Comments should be relevant and contain no abusive language. Comments that are off-topic, vulgar, profane or include personal attacks will be removed. Full comments policy. Additionally, we now display comments from trusted commenters by default. Those wishing to become a trusted commenter need to verify their identity or sign in with Facebook Connect to tie their Facebook account to their Las Vegas Sun account. For more on this change, read our story about how it works and why we did it.

Only trusted comments are displayed on this page. Untrusted comments have expired from this story.

  1. By staralioflundnv (Star Ali Mistriel-Kogan)
    Feb. 13, 2012
    6:44 a.m.
    Flag

    HUMANS monitoring human beings. Now let me make a list of how ineffective much of all of this will be. Especially with the US Government. As long as the good ol'boy system reigns, it will be selective information and enforcement. Especially in NEVADA.

    Example: 1-two aging brothers-one brother gets prescriptions from the VA Hospital in Prescott and "gives" bottles of those drugs to his brother: Celebrex, hydrocodone, etc.; spouse discovered these after spouse ended up in the hospital internally bleeding (agitated affects by drugs)spouse had to be lifeflighted. Drugs with explanation turned into the area DEA, Prescott VA Hospital, and reported to BATF and zero happens. Why? nearly all these parties attend same church... or Example 2-illegal explosives,primacord, blasting caps were discovered in family shed, reported to BATF, due to discovery and concern over constant threats to blow up SNWA pipeline project.

    These few examples (there are more) make a case as to how ineffective US Government agencies are. Citizens are encouraged to report anything suspicious. All this makes me believe that in doing what is the "right thing to do" ends up in serious consequences and life endangerment. The US Government is corrupt, as we are sadly finding out. They set up dictators, fund enemy governments, and USE information to their political advantage and most certainly NOT protect the reporting citizen. It is a most sad realization.

    Blessings and Peace,
    Star

  2. By KillerB
    Feb. 13, 2012
    8:28 a.m.
    Flag

    "Social media has emerged to be the first instance of communication about a crisis, trumping traditional first responders that included police, firefighters, EMT, and journalists," the FBI wrote in its request. "Social media is rivaling 911 services in crisis response and reporting."

    Not sure really what's worse here -- this subtle reminder Big Brother is now our government's active anti-Fourth Amendment policy, or people's moronic need to be noticed.

    "These few examples (there are more) make a case as to how ineffective US Government agencies are. . . All this makes me believe that in doing what is the "right thing to do" ends up in serious consequences and life endangerment."

    star -- the point being those agencies exist specifically to monitor normal citizens going about their daily business looking for an excuse to prey on them. It's what I mean by government at every level being the apex predator, and it's actively on the hunt for every one of us. The budget ax needs to start chopping, and government has given us ample proof it won't do it.

    "Is this 1984, or what?" -- the Honorable Alex Kozinski, now chief judge of the 9th U.S. Circuit Court of Appeals, in the Unabomber case

  3. By boftx
    Feb. 13, 2012
    10:12 a.m.
    Flag

    Many, many moons ago, when usenet was still pristine, it was a common practice for us to include snippets of trigger words that were referred to as "spy bait." It was a widely held thought/joke that the NSA read every piece of traffic on the Internet at the time.

    Today, nobody in their right mind would think of joking around like that unless they really enjoy being visited by government agents.

    I miss those days.

  4. By SunJon (Jon Becker)
    Feb. 14, 2012
    3:05 a.m.
    Flag

    "the ability to create bots that closely mimic humans will only improve over time."

    Trying to predict the future using conversations with a 'bot' has many of the same characteristics as chasing flatulence in a wind storm.

Post a comment

Commenting requires registration.

Comments are moderated by Las Vegas Sun editors. Our goal is not to limit the discussion, but rather to elevate it. Comments should be relevant and contain no abusive language. Comments that are off-topic, vulgar, profane or include personal attacks will be removed. Full comments policy.

If you would like to submit your comment as a letter to the editor, you may submit it here.

Most Popular