There’s a problem that constantly plagues us in this day of information overload, and that is the ability to sift the stream of incoming information into the bits that are valuable from those that aren’t. It’s a tough issue that we’ve been working on at Ushahidi, and re-working, a solution on for a while now. Our solution is called SwiftRiver.

SwiftRiver is a free and open source intelligence platform that helps people curate and make sense of large amounts of information in a short amount of time. In practice, SwiftRiver enables the filtering and verification of real-time data from channels such as SMS, Email, Twitter and RSS feeds. It’s especially useful for organizations who need to sort their data by their unique expectations of authority and accuracy, as opposed to popularity. Such organizations include journalists, community based-organizations, PR/marketing, emergency responders, election monitoring groups and more.

SwiftRiver, In Plain English

There’s a torrential river of information that’s constantly flowing on the Internet. If you dive into that river in search of something, chances are you’d drown.

Now before we go any further, let’s first define this river of information. Simply, the “river” is made up by billions of bits of information. In the context of SwiftRiver, we call these things “droplets.” For example, common droplets in the river are tweets, Facebook updates, and blog posts. These are common examples, but by definition, things like text messages, emails, and even rows in a database table are considered droplets, too.

So how does SwiftRiver help you make sense of all these droplets? Well, it analyzes droplets much like your brain analyzes the world around you. For example, when you look at a kitchen table, your brain instantly determines its color, height, location and texture. In the same way, when SwiftRiver looks at a droplet, it determines all its attributes. For example, if SwiftRiver looks at a tweet, it can determine things like location, time, author and meaning (in the form of keywords). Generally speaking, SwiftRiver does this using a process known as “natural language processing.”

Once SwiftRiver analyzes all the droplets, you then have the ability to filter them down from that torrential river to a manageable stream. In addition to filtering, you can run different analyses on them, helping you get the “big picture” of your set of droplets.

SwiftRiver Glossary of Terms

Droplet: The basic unit of content inside SwiftRiver, i.e., a tweet, a Facebook update, a blog post, an SMS text message, etc.

Identity: The originator of a droplet from a specific channel, i.e., a Twitter account, a Facebook account, a phone number, an email address. Identities are automatically extracted when a droplet is “siphoned” from a channel.

Source: Comprises one or more identities, and could be a person or organization. Unlike identities that are automatically extracted, sources are subjective and put together by users in the system.

Channel: The vehicle for transporting a droplet into the river, i.e., RSS, SMS, Twitter, JSON, XML, etc.

River: The torrent of droplets that come from the predefined channels.

Filter: The mechanism for reducing a channel or river from a torrent of droplets to a more manageable set.

Stream: A collection of droplets whose contents are defined by a filter or a combination of filters.

Bucket: A group of hand-picked droplets from a stream.

Trends: A narrative based on the droplets in a bucket. Components: description, time, sources, places, media (links), tags, theme.

Types of stories, i.e., outputs:

• Text
• Maps
• Timeline
• Graphs, Charts, Heatmaps
• Gallery: Photos, Video, Audio

A sneak peak at the rebooted platform

Since the end of this summer, the Ushahidi team has been focused on rebuilding much of the platform so that it would dovetail perfectly with the Ushahidi core platform and Crowdmap. So, it’s a standalone application that anyone can access and use by itself, but it also answers our users’ need for a faster, more nimble way to manage information flow on their deployments.

Back in August, we completed the first iteration of our ID and authentication system, RiverID, the first step needed for us to have a collaborative profile-based tool for SwiftRiver services. In October, we locked four of the Ushahidi team, made up of two designers and two engineers, away in a cabin, deep in the woods of Georgia. Their job: Do two months’ worth of work in six days.

This they did.

While we haven’t released the code yet, the beta will be available in mid-December (three weeks) for the community first, then for the general review. But, in an effort to quench your curiosity, here’s a sneak peak via screenshot.

As mentioned earlier, SwiftRiver will be available in the coming weeks. We will release the roadmap, a new look and feel, and provide an outline on how you can contribute to the next Swift. You can connect to the Swift community via our google group: swiftriver@googlegroups.com

Photograph from the Wikipedia 2011 Egyptian Revolution article that grew rapidly as events unfolded in Egypt

On the 25th of January this year, Egyptians took to the streets to demand the overthrow of the regime of Egyptian President Hosni Mubarak. At exactly 13:26 UTC on the 25th, Wikipedia editor, The Egyptian Liberal, created a page on .en Wikipedia titled ‘2011 Egyptian Revolution’. Following this, hundreds of edits were made to the page as events in Egypt unfolded.

When a globally-relevant news story breaks, relevant Wikipedia pages are the subject of hundreds of edits as events unfold. As each editor looks to editing and maintaining the quality and credibility of the page, they need to manually track the news cycle, each using their own spheres of reference. The decisions that are made to accept one source while rejecting others remains opaque, as are the strategies that editors develop to alert and keep track of the latest information coming in from a variety of different sources.

Today we launch an exciting new project to study how Wikipedia editors track, evaluate and verify sources on rapidly evolving pages of Wikipedia, the results of which will inform the development of Ushahidi’s Sweeper tool. The project was inspired by a meeting in late 2010 between Erik Möller of the Wikimedia Foundation and Jon Gosier, former director of Swift/Ushahidi who were struck by the common problem users of both systems are faced with when information is coming at them from hundreds or even thousands of different sources during an international news event. In particular they recognised how Sweeper could potentially be used by Wikipedia editors to collaboratively make sense of sources rather than all using independent strategies for tracking and verifying issues of importance.

In the research phase of the project, we’re trying to understand things like: How do editors evaluate sources that may be notable in one region but unknown in another? How do editors verify information in crisis scenarios like earthquakes and political unrest, or even rapidly evolving cultural phenomena when sources are situated outside the frame of reference of most established editors? What techniques do editors use to track issues around pages that they edit? What role does social media play in this process?

In the second phase, we will apply the learnings from the first phase of the project to Ushahidi’s Sweeper and RiverID tools for experimental use in Wikipedia. The research will be helpful in developing algorithms and lenses that users can apply to stream specific types of content according to different levels of authority, accuracy and trustworthiness over time.

The project is seen as a win-win for both Wikipedia and Ushahidi. The Ushahidi team hopes to integrate learnings from the project to improve the way that we manage accurate and trustworthy information. The project has been endorsed by the Wikimedia Foundation who are excited about the potential of working with Ushahidi to support Wikipedia editors.

“Wikipedia has become a go-to resource of high-quality, almost real-time information on rapidly changing events, largely because its editors constantly assess and synthesize information from a broad range of sources. Ushahidi has achieved tremendous social impact by bringing information to people in need, and we are excited to partner with them in exploring how real-time information management in Wikipedia can be made more scalable and effective,” said Dario Taraborelli about the partnership.

According to Ushahidi Executive Director, Juliana Rotich, “The Ushahidi team hopes to integrate the lessons and recommendations from the project to improve the way that we manage accurate and trustworthy information. This is important for Ushahidi deployers and users worldwide who use an array of tools like Ushahidi and Wikipedia, as they aggregate, curate and collaborate.”

The project is being jointly funded by Hivos, the Humanist Institute for Development Cooperation and the Open Society Foundation. See the WikiSweeper project page for more information or to sign up for project updates.

Community Development Open Call

We are restarting the Ushahidi development community open calls and invite you to join a skype chat/ustream call. Our team of developers (Mobile, Swift, and Ushahidi) will be available to answer questions, take your feedback and brainstorm with you on the best ways to open up. This chat will be focused on technical development. We will hold future community chats focused on other topics.

The call will take place at 18:00 EAT/11:00 EDT/08:00 PDT. (Your local time)

This call will be part of the hackathon day and is set to be time zone friendly.

Also, we want to let you know that we are working on streamlining the communication points (wiki, blog, forums, dev chat and mailing lists) to make it easier for you to participate in features, bug fixes and more. Stay tuned in the coming weeks for more on this topic. We know this is a top community concern.

Nairobi Ushahidi Hackathon: Mobile, Crowdmap, Swift River and Ushahidi

We’re holding a Nairobi Ushahidi hackathon on Saturday, August 27, 2011 for some braining and hacking:

Register today:

Schedule:
Date: Saturday, August 27, 2011
Duration: 12:00 – 20:00 EAT
Pizza for 19:00 EAT.

Some of the hacks or bugs that we will focus on include:

• River id and some of its components for Openid, OAuth
• Ushahidi API
• J2ME app
• Your plugin ideas
• Web designers: we could use a hand testing a design theme process.
• Non-techs are welcome to work on documentation, some communications tasks, photos and video fun

We welcome virtual participants to join the hackathon via skype dev chat.

Talk with you on Saturday,

Heather L.

Swift is an open-source initiative who’s goal is to make the process of vetting information more efficient.  The project to date has progressed well thanks in no small part to the following people: Matthew Griffiths (so important to this project I mentioned him twice), Ahmed Maawy, Charl Van Neikerk, Heather Ford, Vladimir Ermakov, The Ushahidi team, Omidyar Networks, Chris Blow, Kaushal Jhalla, Neville Newey, Edmar Ferreira, Pete Warden, Patrick Meier, Anahi Ayala, Ethan Zuckerman, Google’s 2010 Summer of Code Participants (Mang-Git, Soe, Nishith Rastogi), the Guardian’s Activate staff, Product(RED) and many others. This project would be nowhere without you all so thanks for making it happen.

For many of us, this project represents a new way of democratizing access to the tools for understanding and vetting information which is needed by Ushahidi, journalists, and many others.

During the response to the quakes in Haiti in 2010, one of the most frequently requested features from Ushahidi users was the ability to turn any web page into a ‘report’ with the click of a button (as opposed to through aggregation). There were a lot of hacks to get around this, such as taking an RSS feed from delicious.com and feeding it into Ushahidi as a way to have a bookmark button that worked with the platform.

You no longer have to do that, with the latest release of Sweeper you can essentially roll your own open source Delicious or Pinboard.in. This is really powerful when you start activating plugins like our auto-tagger SiLCC or our Push plugins which can output all of your bookmarked content as reports in Ushahidi, or as feed that can be consumed by any application.

We call this little plugin Quiver as it’s where you manually collect and store information using Sweeper which essentially turns it into your own Delicious clone, but with all the contextualization features that people have come to love it for.

So how does it work? It’s simple! Just download and install any version of Sweeper following the current release of v0.3.2 which can be found here.

Once you’ve done that, go to the sources panel.

Select Quiver from the list.

Drag the bookmarklet to your browser.

Done. Sweeper is a tool for the curation of real-time media. Now the sites you find interesting while researching can be mashed up with the content you’re aggregating from the web, twitter, email and other feeds.

Get it from Swiftly.org

The summation of the talk is that the Swift project has been assigned a very complex and incredibly difficult task: to verify and contextualize data from the mobile and social web. How do we do this? This seems to be the part that confuses people. It’s not any of our apps, and it’s not any of our individual APIs that we rely upon to do this. It’s the combination of all these things into one robust algorithm that tries to digitally reconstruct the real-world context, using the features extracted from the content to prioritize and de-prioritize information relevant to that context.

I like to refer to this as folksonomic triage where layers of historic, social, temporal, geospatial and other types of information are layered upon one another to perform a function, and the system (through a process called active learning) then learns how to improve form the user’s interactions. What this attempts to do is allow the human to give the machine algorithm some insight into the types of content they prefer, and the types of content they don’t. A statistical profile of the content features of each type is recorded, with varying degrees of nuance in-between including accounting for bias, crosstalk, irrelevance and falsehoods.

Some of this happens on the application side, some of it happens on the logic/cloud side of things. This is because it’s very important that the user understand that the platform is there to serve them, and not the other way around; algorithms augmenting human decision making. This means we’ve abstracted some elements of the system logic (the elements that everyone needs to re-use over and over again) while the things specific to the use of the platform, are defined in the UI.

Use Cases

We’re really excited to have had a number of really amazing partners new and old using the platform. This includes groups like Newsti.ps who are building a ‘people’s newswire’ using the Swift products.

There are also some really big projects that are occurring. For instance this BBC article profiles PAX which is using our platform to power their conflict early warning platform. They want to index massive amounts of data from around the world and then use what’s captured to spot the historic patterns and trends that then can be used to demonstrate confidence in future patterns.

One of our favorite uses of the Swift platform to date was Product (RED)’s use last year to mashup large quantities of social media activity to power their Turn The World (RED) campaign.

There have been many more uses that we can’t talk about yet, but hopefully those become public soon.

Some Numbers

There are currently eight different code repositories housing the greater Swift project. Each of these API elements is tackled as if it were a single problem. This includes code for location disambiguation, natural language processing, influence detection, reputation monitoring and duplication filtering. You can find more about them here – http://blog.swiftly.org/post/5788873594/resources-for-developers

• These combined repos contain around 150,000 lines of code (not including frameworks like Kohana)
• Over 7,000 downloads of Sweeper to date
• Which theoretically means at least 7,000 users of our APIs
• Sweeper users tend to aggregate thousands of items of content over the life of a deployment which means we’ve taken around 70,000,000 items of unstructured data and done things to it – like add location, tags or filtered out the duplicates. That’s obviously a very liberal extrapolation, but it should give you a sense of the amount of data we’re dealing with.
• As the project moves forward, and all our APIs are finally completed, this number will grow exponentially. With RiverID alone (which tracks the reputation of content and people online) we expect to be indexing over half a billion items of content and actions from the social web alone by the end of the year. That’s just one API, the others will also need to scale on equal terms.

When I first started working on www.Indymedia.org in 2000, I was really excited about the platform it provided: a way for people who witnessed news events to immediately publish text, audio, video and photos to an OPEN newswire.  This was unprecedented on the web at that time, and led to an explosion of open multimedia content-posting sites.  Since its inception at the World Trade Organization protests in Seattle in 1999, the Independent Media Center expanded into over 200 local sites worldwide, all funneling featured content into the main (global) site www.indymedia.org.  In many ways, this could represent the way news organizations operate in the future – but most of the major news companies haven’t caught on to this trend just yet.

I got into the world of journalism because I didn’t trust the media.  Time and again, I’d read, hear or watch news stories that were grossly inaccurate, one-sided and oversimplified.  So I took seriously the slogan, “Don’t hate the media, BE the media”, and helped launch a bunch of indymedia centers and microradio stations all over the world, always with the hope of giving voice to the voiceless, allowing people to tell their own stories and to share in the narrative that was developing about them without the often-damaging involvement of advertising dollars and managing editors who presume to dumb things down for audiences they believe they have to entertain as well as inform.

Now, with more and more people turning away from traditional media to get their news online (see chart), it seems those audiences, about whom so many assumptions were made by the management of media corporations, are trying to find their own way in the new media world and find the news that they think is important and valuable.

Unfortunately, this often means that people seek out only news sources that confirm and uphold their existing points of view, and may be just as full of inaccuracies, speculation and oversimplification as the news media that they were trying to escape.

How can we get through the mess of misinformation to find the real tips of breaking news events, as they’re happening, and get this information out to as broad an audience as possible?

I’ve been working with a team at Stanford this year to use Ushahidi’s Swiftriver platform, and specifically Sweeper (one of the multiple tools in the Swiftriver toolbox) to try to extract real newstips from the deluge of 140-character texts and tweets, and try to figure out which newstips are real and accurate.  Our project description and current newswire is at www.newsti.ps

We’re implementing this in the Occupied Palestinian Territories, an area where many news incidents are under-reported in the US, and others are over-reported, giving US audiences a skewed perspective of the reality on the ground.  We’re using the Swiftriver platform to skim the web and twitter for keywords that are then filtered by keyword, location, reputation and duplication and organized into a database.  Our reporters in different parts of the Palestinian Territories (the West Bank, Gaza and Jerusalem), can follow up on the most poignant of these tips and verify their accuracy.  These reporters have created the International Middle East Media Center (www.imemc.org), currently the most widely-read English-language news site based in the Palestinian Territories.

We’re also working on a way to allow people who witness news events but don’t have the luxury of a smart phone yet (only 2% of cellphone users in the Palestinian Territories have smart phones, and 3G is extremely spotty), to send texts and photos directly into our system as well.  For translation of Arabic texts, we’ve solicited the help of the crowdsourced translation team of www.meedan.net.

Like with Indymedia, we think that this work can be an alternative to the mainstream media – although, as always, they are free to use these news stories, it seems unlikely that many will.  When news corporations are focused on selling advertising dollars instead of providing accurate news for their audiences, they will continue to go the way of the dinosaurs, as they are doing.  Unfortunately what we’re losing right now are lots of good, investigative news reporters who held politicians’ feet to the fire, reported on breaking news events and local issues, investigated wrongdoing by large companies, connected audience members with the stories of people in different circumstances far across the globe, but with whom they could relate due to the strength of the writing and storytelling.  What we’re left with right now, to a large extent, are cable news channels whose focus is on entertainment and advertising, and vitriolic talk radio that exuberantly embraces speculation, rumor and misinformation over fact-checked, accurate news reports.  On the local news front, AOL’s newest branchild, patch.com, threatens to replace real local reporting with half-hearted, badly-written reports that are unapologetically inaccurate.

Can we get a ‘people’s newswire’ based on eyewitness reports of newsworthy events?  I believe we can – if we combine the automation of systems like Swiftriver, the data visualization possibilities of tools like Ushahidi, and the insight of trained reporters who can follow up on potential leads.  Heck, if we can do it in the Palestinian Territories, then we can do it anywhere!

The video below is a short presentation about this project. Be sure to check out our website www.newsti.ps for real-time updates during the upcoming humanitarian flotilla to break the siege on the Gaza Strip.

QR Codes, also known as 2Dimensional barcodes, are evolving as part of an emerging trend called ‘the internet of things’ which describes the augmentation of real-world objects and locations with meta-data that is stored in computer networks.

Using some sort of sensing device, usually a camera, a computing device effectively scans a code and decrypts whatever values it might contain. This can be used to store messages, links or small files — to the tune of around a few thousand bytes. If you put a QR code on an apple, you’ve essentially turned that apple into a real-world data storage device.

So those are the basics of what they are, below we’ll explore how they can be used for disaster response.

But First the Checkin

With Crowdmap Checkins, Ushahidi introduced an open source platform for semi-social location sharing. But the value in disasters is simple, the user performs a very quick physical movement to log data as opposed to taking 5 minutes to fill out a report. In other words, it’s quick and relatively frictionless to do. But I don’t want you think so much about location, think about the function. Quick data capture. That’s what checkins offer, and similarly QR codes can be used to do the same.

If you think of the QR code as an abstraction of a function (the act of checking in) then it can play the same role, as well others (because it carries more data). Rather than clicking a button in an application, all a user needs to do is scan the code with a smart device and the hidden data inside is revealed. Not to mention the fact that any physical object can become the storing device for this data.

Assumptions

Now, the consideration of using such technology in disaster scenarios relies upon a couple of assumptions. One, that smart devices are available and in use. Two, that a sufficient data connection is available. Three, that a printing device is available and portable. Four, that security is not a huge concern as a well placed Sharpie marker could completely disrupt your plans.

Potential Uses

QR Codes and the Ushahidi Platform
View more presentations from Ushahidi

Okay so we’ve covered the why and the why not, what about the acute uses?

• Instructions, messages, or media can be coupled with objects as QR or shortcodes
• Triggers a remote process (real-world API call).
• Triggers a process on the device itself.
• Carry information about the device itself.
• Authentication. Code can serve as an identification measure.

Those are a few of the ideas I came up with. The most interesting to me are encoding functions and processes in the code that are unlocked when scanned. More on this in a future post.

I should add that this post is an exercise in thought, and not something we’re actively working on, although It would be pretty cool.

People have made all sorts of remarks over the years about the effectiveness or ineffectiveness of the Ushahidi platform because that’s where it’s often perceived the most value is. But the value often isn’t in the platform itself, it’s in the momentum built around a given deployment and subsequently the value that collaboration offers others:

This is a diagram illustrating some of the value exchanged in the typical collaborative crisis mapping platform. Green lines represent monetary transactions, white lines indicate direct use or utility, orange lines indicate flows of less tangible value that are more difficult to quantify.

The smaller bubbles indicate what I call ‘multipliers’. Multipliers are players that reinforce what would otherwise be loose ties, making the ties between two entities stronger and more binding.

Jon Gosier, Director Swiftly.org at Net Prophet 2011, May 11, 2011

Currently in Sweeper and soon in Ushahidi, users can translate real-time content from one language into another, on the fly, as they receive it. This is done using our Google Translate plugin. Google Translate currently supports 50+ languages.

For the Sweeper deployment we’re using to monitor the situation in Japan internally, we’re using this feature to monitor events, since we can’t manually translate every single message coming through. We’ve found it a significant timesaver. You can also see below that we’re showing the user what language the message was translated from, or if it’s been translated at all…

Before:

After:

It’s important to understand, that this is machine translation, so it’s far from perfect. But if you’re monitoring feeds from multiple countries across Twitter, RSS, Email or SMS it’s sometimes useful enough to get a quick sense of what’s being said, where to potentially look for more info, or perhaps where to direct human translators.