There’s a problem that constantly plagues us in this day of information overload, and that is the ability to sift the stream of incoming information into the bits that are valuable from those that aren’t. It’s a tough issue that we’ve been working on at Ushahidi, and re-working, a solution on for a while now. Our solution is called SwiftRiver.

SwiftRiver is a free and open source intelligence platform that helps people curate and make sense of large amounts of information in a short amount of time. In practice, SwiftRiver enables the filtering and verification of real-time data from channels such as SMS, Email, Twitter and RSS feeds. It’s especially useful for organizations who need to sort their data by their unique expectations of authority and accuracy, as opposed to popularity. Such organizations include journalists, community based-organizations, PR/marketing, emergency responders, election monitoring groups and more.

SwiftRiver, In Plain English

There’s a torrential river of information that’s constantly flowing on the Internet. If you dive into that river in search of something, chances are you’d drown.

Now before we go any further, let’s first define this river of information. Simply, the “river” is made up by billions of bits of information. In the context of SwiftRiver, we call these things “droplets.” For example, common droplets in the river are tweets, Facebook updates, and blog posts. These are common examples, but by definition, things like text messages, emails, and even rows in a database table are considered droplets, too.

So how does SwiftRiver help you make sense of all these droplets? Well, it analyzes droplets much like your brain analyzes the world around you. For example, when you look at a kitchen table, your brain instantly determines its color, height, location and texture. In the same way, when SwiftRiver looks at a droplet, it determines all its attributes. For example, if SwiftRiver looks at a tweet, it can determine things like location, time, author and meaning (in the form of keywords). Generally speaking, SwiftRiver does this using a process known as “natural language processing.”

Once SwiftRiver analyzes all the droplets, you then have the ability to filter them down from that torrential river to a manageable stream. In addition to filtering, you can run different analyses on them, helping you get the “big picture” of your set of droplets.

SwiftRiver Glossary of Terms

Droplet: The basic unit of content inside SwiftRiver, i.e., a tweet, a Facebook update, a blog post, an SMS text message, etc.

Identity: The originator of a droplet from a specific channel, i.e., a Twitter account, a Facebook account, a phone number, an email address. Identities are automatically extracted when a droplet is “siphoned” from a channel.

Source: Comprises one or more identities, and could be a person or organization. Unlike identities that are automatically extracted, sources are subjective and put together by users in the system.

Channel: The vehicle for transporting a droplet into the river, i.e., RSS, SMS, Twitter, JSON, XML, etc.

River: The torrent of droplets that come from the predefined channels.

Filter: The mechanism for reducing a channel or river from a torrent of droplets to a more manageable set.

Stream: A collection of droplets whose contents are defined by a filter or a combination of filters.

Bucket: A group of hand-picked droplets from a stream.

Trends: A narrative based on the droplets in a bucket. Components: description, time, sources, places, media (links), tags, theme.

Types of stories, i.e., outputs:

• Text
• Maps
• Timeline
• Graphs, Charts, Heatmaps
• Gallery: Photos, Video, Audio

A sneak peak at the rebooted platform

Since the end of this summer, the Ushahidi team has been focused on rebuilding much of the platform so that it would dovetail perfectly with the Ushahidi core platform and Crowdmap. So, it’s a standalone application that anyone can access and use by itself, but it also answers our users’ need for a faster, more nimble way to manage information flow on their deployments.

Back in August, we completed the first iteration of our ID and authentication system, RiverID, the first step needed for us to have a collaborative profile-based tool for SwiftRiver services. In October, we locked four of the Ushahidi team, made up of two designers and two engineers, away in a cabin, deep in the woods of Georgia. Their job: Do two months’ worth of work in six days.

This they did.

While we haven’t released the code yet, the beta will be available in mid-December (three weeks) for the community first, then for the general review. But, in an effort to quench your curiosity, here’s a sneak peak via screenshot.

As mentioned earlier, SwiftRiver will be available in the coming weeks. We will release the roadmap, a new look and feel, and provide an outline on how you can contribute to the next Swift. You can connect to the Swift community via our google group: swiftriver@googlegroups.com

• The US is clueless in response to the Wikileaks release of war documents
• The music and film industries continue to lose to the open web
• South Africa seeks to muzzle the press
• The UAE takes on RIM over the ability to read everyone’s email
• In Australia you can’t link to certain sites from your personal website
• More and more countries require SIM card registration on phones to track users

…it goes on.

This isn’t a conspiracy, it’s a reaction by those in the status quo (be they government, big business or large established organizations) to the inefficiencies that they represent in the system being overcome by changes in technology and culture. As the open web expands it becomes a real threat to controlling governments, even to the relevance of the nation state itself.

The Case of the Russian Fires

Blogging and social media have been utilized for transparency and accountability for a number of years. While that’s interesting in its own right, I find the translation of those online tools into offline activities far more compelling.

The Russian forest fires is a particularly interesting one, as it represents what appears to be a major shift in ownership and attitudes in Russia around governance and responsibility. At the same time, one of the main tools used to organize it was the Ushahidi platform (giving me that little bit of liberty to write some thoughts on the bigger picture).

“On the one hand, cooperation was empowered by a shared understanding that the government has failed to get the situation under control and, moreover, didn’t want to be held accountable for it. On the other hand, it was information technologies that provided both information exchange and tools for coordination and effective collaboration.”

Please, read the full Global Voices article on the Russian fires.

Final Thoughts

Personally, I don’t see too many governments being displaced or replaced by online cooperation alone. Trust, reputation and resources are just a few of the hurdles to overcome before that happens. Instead, I think we’re seeing the continuation of the refinement of mass movements, brought about by the inefficiencies in the system, which catch on faster and are enabled better online and then move offline for impact.

1. Ushahidi takes the lead in deploying the Ushahidi platform
2. Crowdsourced information is statistically representative
3. Crowdsourced information cannot be validated

Lets start with the first. We do not take the lead in deploying Ushahidi platforms. In fact, we often learn about new deployments second-hand via Twitter. We are a non-profit tech company and our goal is to continue developing innovative crowdsourcing platforms that cater to the growing needs of our current and prospective partners. We provide technical and strategic support when asked but otherwise you’ll find us in the backseat, which is honestly where we prefer to be. Our comparative advantage is not in deployment. So the credit for Ushahidi deployments really go the numerous organizations that continue to implement the platform in new and innovative ways.

On this note, keep in mind that the first downloadable Ushahidi platform was made available just this May, and the second version just last week. So implementing organizations have been remarkable test pilots, experimenting and learning on the fly without recourse to any particular manual or documented best practices. Most election-related deployments, for example, were even launched before May, when platform stability was still an issue and the code was still being written. So our hats go off to all the organizations that have piloted Ushahidi and continue to do so. They are the true pioneers in this space.

Also keep in mind that these organizations rarely had more than a month or two of lead-time before scheduled elections, like in India. If all of us have learned anything from watching these deployments in 2009, it is this: the challenge is not one of technology but election awareness and voter education. So we’re impressed that several organizations are already customizing the Ushahidi platform for elections that are more than 6-12 months away. These deployments will definitely be a first for Ushahidi and we look forward to learning all we can from implementing organizations.

The second misconception, “crowdsourced information is statistically representative,” often crops up in conversations around election monitoring. The problem is largely one of language. The field of election monitoring is hardly new. Established organizations have been involved in election monitoring for decades and have gained a wealth of knowledge and experience in this area. For these organizations, the term “election monitoring” has specific connotations, such as random sampling and statistical analysis, verification, validation and accredited election monitors.

When partners use Ushahidi for election monitoring, I think they mean something different. What they generally mean is citizen-powered election monitoring aided by crowdsourcing. Does this imply that crowdsourced information is statistically representative of all the events taking place across a given country? Of course not: I’ve never heard anyone suggest that crowdsourcing is equivalent to random sampling.

Citizen-powered election monitoring is about empowering citizens to take ownership over their elections and to have a voice. Indeed, elections do not start and stop at the polling booth. Should we prevent civil society groups from crowdsourcing crisis information on the basis that their reports may not be statistically representative? No. This is not our decision to make and the data is not even meant for us.

Another language-related problem has to due with the term “crowdsourcing”. The word  “crowd” here can literally mean anyone (unbounded crowdsourcing) or a specific group (bounded crowdsourcing) such as designated election monitors. If these official monitors use Ushahidi and they are deliberately positioned across a country for random sampling purposes, then this becomes no different at all to standard and established approaches to election monitoring. Bounded crowdsourcing can be statistically representative.

The third misconception about Ushahidi has to do with the tradeoff between unbounded crowdsourcing and the validation of said crowdsourced information. One of the main advantages of unbounded crowdsourcing is the ability to collect a lot of information from a variety of sources and media—official and nonofficial sources—in near real time. Of course, this means that a lot more of information can be reported at once, which can make the validation of said information a challenging process.

A common reaction to this challenge is to dismiss crowdsourcing altogether because unofficial sources may be unreliable or at worse deliberately misleading. Some organizations thus find it easier to write off all unofficial content because of these concerns. Ushahidi takes a different stance. We recognize that user-generated content is not about to disappear any time soon and that a lot of good can come out of such content, not least because official information can too easily become proprietary and guarded instead of shared.

So we’re not prepared to write off user-generated content because validating information happens to be challenging. Crowdsourcing crisis information is our business and so is (obviously) the validation of crowdsourced information. This is why Ushahidi is fully committed to developing Swift River by early 2010. Swift is a free and open source platform that validates crowdsourced information in near real-time. Follow the Ushahidi blog for exciting updates!

We’ve had a problem though, our team is small and we’ve had to use the lion’s share of our time and energy just to keep Ushahidi on track. Meaning, we’ve not been able to put as much resources into Swift necessary to make it what it should be by now.

Our answer to this is Jon Gosier.

Jon is the founder of Appfrica Labs in Uganda, a Senior TED Fellow and a name well-known amongst the African technology and blogging crowd (on Twitter at JonGos). Jon will be leading the Swift River initiative from our side. Besides his technical and organizational skills, we’re excited to have Jon on board because he shares the same ethos, energy and African roots as the rest of us.

Jon will also work alongside the rest of the Swift River community, including the ones who came up with the idea; Kaushal Jhalla and Chris Blow who have been doing most of the work over the past year on this project.

A little more on Swift River’s purpose

The purpose of the Swift River initiative is to develop a free and open-source platform to validate crowdsourced crisis information in near real-time. User-generated content is becoming an increasingly important source of information during crises while traditional media continues to play a pivotal role in documenting crises as they unfold. These two trends are expected to continue well into the future. The challenge is how to filter this growing torrent of information while keeping the “floodgates” open?

There is an apparent tradeoff between crowdsourcing (opening the floodgates) and data validation (the filter). One of the strengths of crowdsourcing is the ability to collect a high volume of information from highly diverse sources in near real time. One of the challenges, however, is to validate this vast amount of information in near real-time in order to inform crisis early warning and rapid response operations.

Want to get involved?

• The Swift Mailing List (Discussion)
• The Other Swift Mailing List (Development)
• The Swift Homepage
• The Swift repository

I’ve spent the last couple days holed up in a room in scenic Salzburg, Austria with 20 other people from both traditional journalism and new media backgrounds. Our goal: discuss strategies for more effective engagement and investment in “tomorrow’s media“. Having this mixture of “old school” journalism professionals mixed in with those of us who are only embedded in the blogging and social media fields, but with no classical training in journalism, has led to some very interesting conversations.

Quality

One of these sticking points has been around the journalist’s emphasis on the need for quality in reporting via new media (citizen media, like blogs and social networks). It’s a good point, we all want more quality reporting in any medium, whether it’s by a blogger or a newspaper. There is a stated need for education and training in solid journalism practices.

That might be true. However, it misses one important point, especially when we compare how we talk about the West (US/Europe) and everyone else.

Critical Mass

In the West, there has long been access to computer, the internet and social media tools. There’s a critical mass of bloggers and influencers who use social media tools like Facebook, YouTube or Twitter. You have to ask the question then, has access and familiarity with these digital tools led to the vibrancy that allows greater transparency to happen?

Is it due to having access, and your peers usage, that allows self-learning to take place with the new media and social tools which are available to everyone? Kids, students and the general populace tinker in the space and grows the space organically.

Put another way, a statistically inconsequential number of bloggers and social media influencers from the West were trained in journalism. Why are we expecting that training and education, rather than simple, open access to social media tools will make a difference “over there” when it didn’t to anyone in the West?

[Sidebar: Quality and critical mass are not mutually exclusive. As a new medium and new users get comfortable and grow, quality is evolutionary and grows as well.]

• First, we see an inordinate amount of traffic on the social networks (Twitter, Facebook, etc).
• Second, the aggregators step in to gather the data into one place.
• Third, we see visualizations (maps and graphs).

Social Network Chatter

When there is a “hot flash” emergency, the social media networks start to buzz – Twitter is the biggest and most open example of this. It’s also the place where a lot of misinformation starts to get amplified. It’s not just a place to air your own thoughts, but a place to get a bead on information as it is happening. You can get the information here faster than anywhere else, period.

Ever since the Mumbai terror attacks last year, the community behind Ushahidi has been interested in this. We think this should be done, understanding the probability of information being true by gathering streams of data and using both machine-based and human filtering to make sense of it. The Swift River project was born from this. It is currently being prototyped and iterated on within VoteReport.in project. Another interesting project in the same line is InSTEDD’s Evolve, which has a great amount of potential in just this scenario.

There is another issue at hand here though; the fact that many individuals who have information are not on Twitter, Facebook, or any other big social network. So, while there is a great deal that can be done with the open channels available in the developed world, most of the world is not on those channels when it matters most.

This is where tools like Ushahidi come in. It’s why it was developed, a way to get information from people who aren’t connected online and who might never be. A way for them to share information and receive alerts around emergency incidents.

Aggregating the News

As soon as a new emergency hits, the first thing we see is aggregation of news and data around it. This is good. In the Swine Flu case, I think the example set by Delicious creator Joshua Schachter is one of the best.

“my weekend evening project: RSS + Twitterfeed + Bit.ly + Delicious + Google News = @threatwatch“

People try to find the best places to get new news. People like Jimmy Wales will use his platform to create real-time wiki collaboration. Once in a while you’re fortunate to have institutions like the CDC give out good up-to-date information.

I’m intrigued by the aggregation of information, and it makes sense if you can actually get a handle on the different channels, not just mainstream news, wire services, images and video. It really has to be done on a large scale and it’s not an easy thing to manage in terms of both volume and veracity of information.

Visualizations and Maps

The last, and usually the most helpful to general news seekers, is the maps that have started to crop up. Google Maps Mania has an extensive list. Below is Google’s own aggregation and mapping of Swine Flu:

View H1N1 Swine Flu in a larger map

Timothy O’Brien has some interesting graphs on the data as well, especially the information coming through from Twitter. As time goes on, if this follows the patterns we’ve seen before, we’ll see even better and more professionally designed informatics done by the NT Times and others.

In Summary

What we have is the beginnings of an ecosystem for emergency and disaster information. The projects are disjointed and unconnected, and there’s little hope of making them one cohesive unit (nor should the necessarily be).

What I do hope to see in the future is that the protocols, tools and processes for gathering, making sense of, and then disseminating crisis information becomes more open and standardized. There’s no reason that Ushahidi shouldn’t plug and play well with Evolve, which then feeds into Threatwatch on Twitter and is all part of a mapping and visualization scheme by larger publishers.

It would be very interesting to get some of the minds behind Twitter, Ushahidi, InSTEDD, Facebook, Wikia, Google and others together to better figure out how we can each continue to build independently, yet at the same time work towards a better ecosystem for emergency information.

Swift River was given a lot of thought by Chris Blow and Kaushal Jhalla before the Ushahidi meetings in March 2009. This video is a short version of what was recorded, giving an idea of what was being discussed.

What is Swift?

It’s an initiative that seeks to do two very important things, both of which are crucial for not just Ushahidi, but for many emergency response activities in the future. First, it gathers as many possible streams of data about a particular crisis event as possible. Second, using a two-part filter, that stream of data is filtered through both machine based algorithms and humans to better understand the veracity and level of importance of any piece of information.

You can read about the human interaction part of it here, where I talk about crowdsourcing the filter.

Preferred Futures for African IT Projects – Erik Hersman from Jonathan Marks on Vimeo.

Without the internet the general public is left misinformed about international news. With a death toll of over 5.4 million, the DRC conflict is the deadliest war since WWII. Yet, in 2000, there was so little coverage that reportedly, an average viewer watching 30 minutes of news daily on CNN and BBC would have seen a total of 15 minutes and 29 minutes of coverage, respectively, over a span of one year, in the largest African country whose wealth in natural resources surpasses many. By not maintaining an informed public, an inefficient media does not only fail its audience, but it also fails those who are suffering. NGO’s and humanitarian efforts receive greater governmental funding and private contributions the more the public is aware. The more mainstream media fails to be efficient, the more powerful the internet is in maintaining and informed public.

When violent conflict broke out in 2007, international media institutions, and NGOs received a large portion of their information from bloggers within the DRC, such as Congogirl and Kate Wolf. When reporting incidences in the north kivu province, Ushahidid’s use of mobile phones and the web, in cooperation with NGO workers and local journalists, turned the blogging of 2007 into efficient media through a system of credible web based reporting*. In the last two months, lack of internet access has slowed humanitarian efforts, and weakened incident reporting on the north east province of the tracking map. Unfortunately, national low circulation of print media, lack of radios and tvs, can further fuel conflict.

In 2004, when Gen. Nkunda and Col. Mutebusi jointly took over Bukavu, violent demonstrations toward the UN (there to prevent the incident) broke out in the capital. Meanwhile the DRC government abandoned support, of a member of the presidential guard’s coup attempt. Rumors surrounding the presidential guard’s whereabouts escalated political tensions throughout the country. The rumor mill spun to such an extreme that under the threat of sanctions, the official government restricted the press from releasing any messages that might heighten tension or conflict.

Many services in Africa have built their business models around mobile technology, therefore owning a cell can dramatically improve one’s life. The African mobile market is growing twice as fast as the global market. Ushahidi has used the prevalence of mobiles in Africa, to help formulate a new journalism, however, an effective internet service is crucial in its efficiency as seen in the north eastern province. The deal between Microm DRC, and its high bandwidth, low-latency internet provider, 03b Networks says that its extensive deal will provide internet for the whole country. Financial supported by Google Inc, Liberty Global Inc and HSBC Principal Investments, the network project will combine speed, and global reach to create the first ever ultra low latency, fiber speed satellite network. With communication throughout the DRC intact, the path for a country to create, build, maintain and sustain good governance and protection its human rights can be seen, and the door for global populous to be truly informed opens.

About the guest author:
Sarah Jones is a multimedia journalist currently pursuing her masters in International Journalism at City University in London. Her thesis will feature the experiences and views of those living in the Matonge district of Belgium., She graduated with a B.A. in Communications from Lake Forest College in 2008 while working as the associate producer for the “Someone You Should Know” segment on ABC 7 News Chicago.