There’s a problem that constantly plagues us in this day of information overload, and that is the ability to sift the stream of incoming information into the bits that are valuable from those that aren’t. It’s a tough issue that we’ve been working on at Ushahidi, and re-working, a solution on for a while now. Our solution is called SwiftRiver.

SwiftRiver is a free and open source intelligence platform that helps people curate and make sense of large amounts of information in a short amount of time. In practice, SwiftRiver enables the filtering and verification of real-time data from channels such as SMS, Email, Twitter and RSS feeds. It’s especially useful for organizations who need to sort their data by their unique expectations of authority and accuracy, as opposed to popularity. Such organizations include journalists, community based-organizations, PR/marketing, emergency responders, election monitoring groups and more.

SwiftRiver, In Plain English

There’s a torrential river of information that’s constantly flowing on the Internet. If you dive into that river in search of something, chances are you’d drown.

Now before we go any further, let’s first define this river of information. Simply, the “river” is made up by billions of bits of information. In the context of SwiftRiver, we call these things “droplets.” For example, common droplets in the river are tweets, Facebook updates, and blog posts. These are common examples, but by definition, things like text messages, emails, and even rows in a database table are considered droplets, too.

So how does SwiftRiver help you make sense of all these droplets? Well, it analyzes droplets much like your brain analyzes the world around you. For example, when you look at a kitchen table, your brain instantly determines its color, height, location and texture. In the same way, when SwiftRiver looks at a droplet, it determines all its attributes. For example, if SwiftRiver looks at a tweet, it can determine things like location, time, author and meaning (in the form of keywords). Generally speaking, SwiftRiver does this using a process known as “natural language processing.”

Once SwiftRiver analyzes all the droplets, you then have the ability to filter them down from that torrential river to a manageable stream. In addition to filtering, you can run different analyses on them, helping you get the “big picture” of your set of droplets.

SwiftRiver Glossary of Terms

Droplet: The basic unit of content inside SwiftRiver, i.e., a tweet, a Facebook update, a blog post, an SMS text message, etc.

Identity: The originator of a droplet from a specific channel, i.e., a Twitter account, a Facebook account, a phone number, an email address. Identities are automatically extracted when a droplet is “siphoned” from a channel.

Source: Comprises one or more identities, and could be a person or organization. Unlike identities that are automatically extracted, sources are subjective and put together by users in the system.

Channel: The vehicle for transporting a droplet into the river, i.e., RSS, SMS, Twitter, JSON, XML, etc.

River: The torrent of droplets that come from the predefined channels.

Filter: The mechanism for reducing a channel or river from a torrent of droplets to a more manageable set.

Stream: A collection of droplets whose contents are defined by a filter or a combination of filters.

Bucket: A group of hand-picked droplets from a stream.

Trends: A narrative based on the droplets in a bucket. Components: description, time, sources, places, media (links), tags, theme.

Types of stories, i.e., outputs:

• Text
• Maps
• Timeline
• Graphs, Charts, Heatmaps
• Gallery: Photos, Video, Audio

A sneak peak at the rebooted platform

Since the end of this summer, the Ushahidi team has been focused on rebuilding much of the platform so that it would dovetail perfectly with the Ushahidi core platform and Crowdmap. So, it’s a standalone application that anyone can access and use by itself, but it also answers our users’ need for a faster, more nimble way to manage information flow on their deployments.

Back in August, we completed the first iteration of our ID and authentication system, RiverID, the first step needed for us to have a collaborative profile-based tool for SwiftRiver services. In October, we locked four of the Ushahidi team, made up of two designers and two engineers, away in a cabin, deep in the woods of Georgia. Their job: Do two months’ worth of work in six days.

This they did.

While we haven’t released the code yet, the beta will be available in mid-December (three weeks) for the community first, then for the general review. But, in an effort to quench your curiosity, here’s a sneak peak via screenshot.

As mentioned earlier, SwiftRiver will be available in the coming weeks. We will release the roadmap, a new look and feel, and provide an outline on how you can contribute to the next Swift. You can connect to the Swift community via our google group: swiftriver@googlegroups.com

Heather Leson is our “Director of Community Engagement”. This means she’s responsible for coming up with the strategy for how we interact better with our non-tech users, and then implementing it. She’s the one who you’ll be in touch with on the Community.Ushahidi.com side, and who will help muster the right mix of case studies, tools and use-cases that make running your own deployment easier.

Heather comes with a boatload of community work. Hailing from Canada, she has over 10 years of experience in technical incident management, software life cycle development, customer care and communications in Internet technologies. In fact, I first met Heather (virtually) as we each led the Random Hacks of Kindness discussions between the Australia and Kenya teams last year. She was an important part of the growth of CrisisCommons, and has helped run multiple unconferences and hackathons.

Heather comes with the right mix of community engagement background, mixed with our ethos of open source, open ideas and open community. We’re SO excited to have her join us, and we hope that you’ll reach out and start engaging with her right away. After all, she is your ambassador within Ushahidi.

It’s with a touch of sadness, and excitement for Jon, that I’m writing this post as he has decided to follow his dreams and embark on a startup of his own. Matt Griffiths, who’s been working with us on SwiftRiver as well, will be joining him in this new project called MetaLayer. This new company plans to use smartphones and augmented reality, mixed with big data, to contextualize the world. It’s a massive idea, with great potential, and we’re looking forward to seeing them excel in their new startup.

What This Means to Ushahidi

Internally at Ushahidi, the SwiftRiver project keeps on humming with a new website refresh launching, SwiftMeme upgrades and redesign, and the Sweeper app in it’s next iteration. Charl and Ahmed, the core engineers, continue to guide the development cycle. David Kobia, Ushahidi’s Director of Technology Development, will set the roadmap, architecure needs and provide the direction for this project in addition to the other products and services offered by Ushahidi.

As always, we rely on the community too, this isn’t just about the core team doing work and making the platform what it is. The continued help from experts in all fields is needed and we’re asking for your help to ensure that we continue on the high velocity trajectory that SwiftRiver is on.

Why did it work?

The “Trust Bridge”

Over the past three years we’ve seen thousands of Ushahidi deployments (12,000+ as of this writing), some of them big, many of them small. We always think about what makes a deployment successful. Why do some catch on and others don’t?

The common denominator for successful deployments that get the crowd involved, where communities are engaged and using the platform for both sending and receiving information, is that they’re run or endorsed by people or organizations that people trust. These entities form what I call the trust bridge. They’re the necessary glue that brings credibility and trust to a deployment so that people are willing to take part.

In the case of Kenya 2008, the reason that the first version of Ushahidi worked was because the public trusted us, the primary drivers and bloggers. Those of us behind the building of Ushahidi were well known and credible to thousands of readers, who in turn took that trust and passed it on to their friends and family in Kenya and in the diaspora.

The same applies to deployments of the platform in places like New Zealand earlier this year, where an open community worked and drove a successful operation. In Haiti, where a group of trusted organizations and volunteers came together to coordinate a rapid crowdsourcing response. The same applies to many others, including Japan, Australia, Kenya’s referendum and the Russian fires.

Building Your Trust Bridge

How do you get to a place where you’re the trusted entity?

The short response is that it doesn’t come at the time of an emergency. This applies if you’re an individual or a large organization starting something off; you have to have credibility and trust from others in order to gain traction.

The longer answer is that assuming you have that first level of credibility, you still should look for further verification and trust building through other trusted intermediaries vouching for you. There are a couple ways you can build this:

Partnerships – Find other organizations that fill roles that you lack or that share the same mission as you. Encourage them to take on roles that are highly inclusive and transparent to the community that you’re interacting with.

News Media – Contrary to popular belief, crowdsourcing and new media do not do away with traditional media. Over the years, these organizations have built up a certain amount of credibility. If your local media is compromised, you should still reach out to international media for coverage. One of the great benefits of international media is that they attract the diaspora to your cause, who in turn can volunteer or spread the word on the ground through their local family/friend networks.

Community – The most successful deployments of Ushahidi are those that are done by, and for, members of the community which is affected. Instead of closing off operations, messaging and administration of the deployment you should instead be reaching out to and openly involving these members to be a part of it. Give responsibility and extend your reach.

The technology that is the Ushahidi platform, Crowdmap and SwiftRiver are excellent tools. However, deploying of these tools is only the first, simple step, in a successful crowdsourcing and community engagement. Building trust bridges, communicating openly and effectively and running a well administered site are the real juice that makes a deployment catch on and thrive.

The Ushahidi core team works from 7 different timezones ranging from Kampala to Louisville, soon expanding to places like Brazil and Korea. One weekend a year we’re able to get together, in-person, to solidify our connections with each other and talk through the big strategic topics that are best done face-to-face. It could be argued that it’s the most important 3 days of the year for us.

The First XV

2010 was a big growth year for Ushahidi, where we got up to 12 core team members – doubling in size from 2009. We’re adding 3 more people this year, which brings us to 15, a fortuitous number for the team as many of us are big rugby fans.

(Caleb decided to have a little fun, putting us all in our positions based on the date that we joined the team.)

12 Months Later

Last year we met in Miami, as we are this year, and a lot has happened since then. To name the big ones:

• Plugins – extensible way to add new functionality without bloating the core
• Crowdmap – maps for non-developers, also a means to quickly collect reports giving deployers time to install their own server
• SMSSync – simple and robust alternative to Frontline and Clickatell
• iOS – rich smart phone experience
• Checkins – opens platform to entirely new uses
• Stand-By Task Force – game changer in disaster response
• J2ME – extending reaching onto older devices
• Community Site – fantastic documentation
• Map Geometry

Looking at the historical record, it’s been a good year. However, there’s a lot more to do. At this meeting, besides drinking a Mojito on South Beach, we’ll get into some of the big future-looking issues, such as:

Visual Reporting: What’s the perfect Ushahidi dashboard? How do we surface “power stats” for Ushahidi deployments and metrics. Swift-Ushahidi integration visuals on the front and back end.

Knowledge Management: How do we come up with a plan to capture information that we know internally, so that it is shared with deployers and developers better?
The inverse, how do we handle and capture information that our *users* know regularly?

Crowdmap Scalability & Migration: Making sure that even the biggest deployments work on Crowdmap. Adding in new a la carte features, etc.

Of course, this is a chance to discuss some of the more mundane items as well, around operations, funds and how we work towards organizational financial sustainability as well. It also means that we’ll be offline from today until about Tuesday of next week. We’ll be a little slower on email and other communications mediums, but bear with us as it’s for a good cause.

The value of crowdsourced information isn’t always found in its accuracy and it certainly isn’t always easy to make sense of, as people are writing quickly and are in difficult personal situations. The quality of crowd-based information sometimes isn’t as high as if it was from experts, but there is value in both it’s mass (size) and it’s immediacy.

I was a little surprised to see a post by Jim Fruchterman, CEO of Benetech an organization that we greatly respect and admire, talking about how crowdsourced data is not a substitute for real statistics. Those blog posts, and the Fast Company article that they spawned, might be good for making waves and getting attention, but it’s of little value if you want to really dig into the problem and find answers.

This academic debate started after the European Commission’s Joint Research Center did a study on Haiti’s crowdsourced information and found a pattern of aggregated text messages predicting where the damaged buildings were concentrated. The crux of the argument is that this “positive correlation” either is, or isn’t, true. What I find more troubling is that this was a study specifically focused on Haiti, and I’m not sure why Benetech felt it could be extrapolated out to every other earthquake scenario?

Sitting back and thinking about this blogging dialogue, I can say that I too am very interested in knowing if there is an aggregation and prediction pattern that can be used in realtime. If so, responding organizations will be more effective with that knowlege. If not, then it won’t be and that is also good to know.

I’m concerned with how Benetech is positioning their answers on the same findings as being clear cut and right. My colleague Patrick Meier has written a piece digging deeper into the aggregation and patterns studies. He makes a clear case on just how unclear Benetech’s findings are compared to the European Commission’s.

The problem with academic arguments is that they disregard the reality of the mess that is a crisis response. If it were a clinical environment where we all got to sit back, sift data and take our time to make a decision, then Benetech’s expertise in cutting-edge quantitative research on patterns of human rights abuses is undeniably useful. In this case, it isn’t.

The problem is this; we don’t have the days/weeks/months needed to sit back and gather all of the statistical data before a decision is made. In a disaster situation, such as the above referenced Haiti, text messages are important and have a great deal of use at that moment. It doesn’t matter if aggregation patterns might/might not prove to be valuable, because we still haven’t seen anyone use that information in the real world.

Again, I too want to know if there is a true pattern on buildings and text messages. Since Haiti, there have been a couple more large earthquakes and the Ushahidi platform has been used to aggregate data around those as well. I’m sure we could convince the good people who did the New Zealand and Japanese response to share information, and that might give us enough data to make a more educated deduction. A sample of three is more valuable than a sample of one.

Either way, It’s worth pointing out that we’re further ahead than we were before Ushahidi started crowdsourcing crisis information due to the fact that we have realtime information, from the ground, that allows decisions to be made.

You can see it live here: http://demo.ushahidi.com/japan. On the live map, you can toggle the clustered numbers on/off using the “stack” button on the right side of the map.

Japan earthquake Ushahidi data, heatmapped

We’re thinking of doing two things to make this more useful:

1. Making it filterable by category.
2. Creating it as a plugin so that anyone can put it on their own Ushahidi deployment.

We’d like to hear from you. Is this valuable or useful? What other visualizations do you think would make sense to do?

Read more about what’s going on at the iHub blog.

SMSsync v1.0.3

A couple months ago we came out with SMSsync, a simple Android app that turns your phone into an SMS gateway. We’ve been testing the app to find and fix bugs, and you’ve had suggestions on how to make it better. Here are the results:

• Downgraded to Android 1.6 so SMSsync works from 1.6 and above.
• Now SMS can be deleted from the SMS inbox. This is a configurable option.
• Fixed issue with the secret variable.
• Better support for localization – All hardcoded strings have been moved to the string.xml file.
• Supports outbox — for pending messages that manually needs to be synced.
• Improved SMS background service — Now it starts and stops perfectly.
• Added version number to the powered by text on the Settings screen.
• Changed notification Icon to SMSSync’s launch icon.
• Shows status of Pending messages. Whether there are pending messages or not.
• Pre-populates URL field with “http://” when setting up the sync URL.
• Sends an auto response once the SMS is received. This is a configurable option.
• Validates the callback URL

Download the Android app.
Download the source code.

Ushahidi Foundation

• Mission: We have always been, and will continue to be, a non-profit tech company that develops free and open source software for information collection, visualization and interactive mapping. We focus on creating simple tools for crowdsourcing information and changing the way information flows, with a specific focus on information trickling from the bottom up.
• African roots: We’re now up to 12 people on the team, and more than half of us are Africa-based, with team members in Ghana, Kenya and Uganda, with the rest in the US.
• Global focus: Our volunteer network spans the globe with programmers, deployers and translators in every continent (except Antarctica). We might have built the tool for Kenya, but the organization was formed to take the platform global.
• Independent funding: We made a rule within Ushahidi in the very first year that we wouldn’t take any public funding from any government. We haven’t, and don’t plan on changing that anytime soon. Our funding comes from private foundations (Cisco, MacArthur, Omidyar Network, Knight, Hivos, Humanity United), and from select custom projects each year.

The 2011 Focus

The strategy at the Ushahidi organization for the past 3 years has been on getting the right team together to create the best crowdsourcing and visualization platform that we could so that others could do in 3 hours what took us 3 days to do in Kenya. It has also been to figure out how to best cultivate and work with a growing community of volunteer programmers, designers and deploying organizations that have emerged around the platform.

The strategic focus for 2011 takes us into new territory, with a focus on the user community, platform refinement, product packaging, deploying organization partners and funding diversification.

The User Community
2010 saw us shift from primarily putting our focus in the volunteer programmer community, and starting to reach out more to the user and deployer community. We’ve been working hard since this time last year to better understand the needs and create better ways of sharing between these groups. It’s still a work in progress, but you can see the new community section of the site here.

We’re trying to create the community connection point, where people can learn from others who have done similar-type deployments to the one that they are planning, as well as a knowledge library for you to find, read and learn from for your own deployments. Finally, it’s a place where people can connect directly to each other, and hopefully we’ll see a lot more partnerships take place.

Last year also saw the beginnings of the Universities for Ushahidi program (U4U), where Ushahidi and the United States Institute for Peace (USIP) are partnering to develop a unique, open curriculum and training program for students to develop core skills for conflict management, real-time mapping, and promote peacebuilding initiatives in their own communities.

Platform
The Ushahidi platform focus in 2010 was on its basic architecture, making sure that what we had built could be extended (plugins), was easier to customize (theming), had an easier install and update process, and that everything that was supposed to work, worked.

This year, while there will always be some architectural work to do, we’re going to pay a lot more attention to the little things. Our goal is to make both the front-end, and admin side, more user friendly. For us this means simplifying and making the user experience more refined. We’d love to get some of your help as we go through this process, and we’re already seeing a lot of great work from the volunteers on making this happen.

Products
SwiftRiver and Crowdmap were both launched into their betas in 2010. We’re still taking Crowdmap through it’s paces, and we have some big adjustments coming that way from the lessons we learned in this month’s Australia Floods deployment. SwiftRiver has a number of tools within it, with the framework well set, and starting to be fully integrated into the Ushahidi platform.

What we’re doing that’s different in 2011 though is that we’re packaging products that can be sold. While all of our code remains free and open source to download and run on your own, if you want to use our servers and web services, those are available as well. SwiftRiver serves are a great example of this.

Crowdmap, our hosted Ushahidi platform service, is also free and available to anyone to use. Where you’ll see the packaging of our products for purchase here is in the plugins and upgrades that you can do on top of the vanilla installation. Though many will be free, some will be for paid upgrades. Look for packages around SMS bundles, “checkins”, and data export/backups among others.

On top of the Crowdmap and SwiftRiver products, we are bundling some of our tools into specific packages for certain types of uses. For instance, we have a number of plugins that we created specifically for election monitoring efforts in Kenya and Tanzania and we’re going to make those freely available to everyone in the community for use anywhere in the world.

Deploying Partnerships
As Ushahidi we get approached by a lot of organizations to help them do custom deployments for all types of use cases. Due to how our team is made up (primarily focused on platform development) we’ve historically been quite choosy about who we work with, and the type of projects. This will continue, with Ushahidi as an organization taking on a few high-level projects each year. We’ll soon release our pricing plans for this direct engagement with our organization.

At the same time, there are certain people and organizations who have continued to be a strong part of the Ushahidi ecosystem in the last 12-18 months. Our plan is to continue to connect organizations who need help in their deployment, and who are willing to pay for that work, with this growing community of deploying experts. Our purpose is to extend what can be done through crowdsourcing and data visualization far beyond what we alone can do as an organization, and to us that means making sure that others are deeply involved and benefiting from it too.

Funding Diversification
Our funding has traditionally been by private foundations, which is a model we needed to get started, but is one that we hope to diversify. In 2010 approximately 10% of our revenues came via custom work for clients. Our goal for 2011 is to push that number closer to 20% for external custom work, with an additional 10% from the SwiftRiver and Crowdmap products.

An couple examples of this type of work are how SwiftRiver worked with the (RED) campaign on a custom deployment for AIDs Day. Another is the work that we started last year with Humanity United last year to put together an early warning system in Liberia. These are wildly different types of deployments with different engagement timelines, teams and costs – and serve to show just how different the possibilities are for the platform.

This move towards a distinctly different revenue stream strategy will mean some adjustments in our team makeup, including shifting of some individuals into different teams, and likely an addition or two that have an outside focus on community development, project management and customer service. As always, we’ll keep you updated on how and when this unfolds.