There’s a problem that constantly plagues us in this day of information overload, and that is the ability to sift the stream of incoming information into the bits that are valuable from those that aren’t. It’s a tough issue that we’ve been working on at Ushahidi, and re-working, a solution on for a while now. Our solution is called SwiftRiver.

SwiftRiver is a free and open source intelligence platform that helps people curate and make sense of large amounts of information in a short amount of time. In practice, SwiftRiver enables the filtering and verification of real-time data from channels such as SMS, Email, Twitter and RSS feeds. It’s especially useful for organizations who need to sort their data by their unique expectations of authority and accuracy, as opposed to popularity. Such organizations include journalists, community based-organizations, PR/marketing, emergency responders, election monitoring groups and more.

SwiftRiver, In Plain English

There’s a torrential river of information that’s constantly flowing on the Internet. If you dive into that river in search of something, chances are you’d drown.

Now before we go any further, let’s first define this river of information. Simply, the “river” is made up by billions of bits of information. In the context of SwiftRiver, we call these things “droplets.” For example, common droplets in the river are tweets, Facebook updates, and blog posts. These are common examples, but by definition, things like text messages, emails, and even rows in a database table are considered droplets, too.

So how does SwiftRiver help you make sense of all these droplets? Well, it analyzes droplets much like your brain analyzes the world around you. For example, when you look at a kitchen table, your brain instantly determines its color, height, location and texture. In the same way, when SwiftRiver looks at a droplet, it determines all its attributes. For example, if SwiftRiver looks at a tweet, it can determine things like location, time, author and meaning (in the form of keywords). Generally speaking, SwiftRiver does this using a process known as “natural language processing.”

Once SwiftRiver analyzes all the droplets, you then have the ability to filter them down from that torrential river to a manageable stream. In addition to filtering, you can run different analyses on them, helping you get the “big picture” of your set of droplets.

SwiftRiver Glossary of Terms

Droplet: The basic unit of content inside SwiftRiver, i.e., a tweet, a Facebook update, a blog post, an SMS text message, etc.

Identity: The originator of a droplet from a specific channel, i.e., a Twitter account, a Facebook account, a phone number, an email address. Identities are automatically extracted when a droplet is “siphoned” from a channel.

Source: Comprises one or more identities, and could be a person or organization. Unlike identities that are automatically extracted, sources are subjective and put together by users in the system.

Channel: The vehicle for transporting a droplet into the river, i.e., RSS, SMS, Twitter, JSON, XML, etc.

River: The torrent of droplets that come from the predefined channels.

Filter: The mechanism for reducing a channel or river from a torrent of droplets to a more manageable set.

Stream: A collection of droplets whose contents are defined by a filter or a combination of filters.

Bucket: A group of hand-picked droplets from a stream.

Trends: A narrative based on the droplets in a bucket. Components: description, time, sources, places, media (links), tags, theme.

Types of stories, i.e., outputs:

• Text
• Maps
• Timeline
• Graphs, Charts, Heatmaps
• Gallery: Photos, Video, Audio

A sneak peak at the rebooted platform

Since the end of this summer, the Ushahidi team has been focused on rebuilding much of the platform so that it would dovetail perfectly with the Ushahidi core platform and Crowdmap. So, it’s a standalone application that anyone can access and use by itself, but it also answers our users’ need for a faster, more nimble way to manage information flow on their deployments.

Back in August, we completed the first iteration of our ID and authentication system, RiverID, the first step needed for us to have a collaborative profile-based tool for SwiftRiver services. In October, we locked four of the Ushahidi team, made up of two designers and two engineers, away in a cabin, deep in the woods of Georgia. Their job: Do two months’ worth of work in six days.

This they did.

While we haven’t released the code yet, the beta will be available in mid-December (three weeks) for the community first, then for the general review. But, in an effort to quench your curiosity, here’s a sneak peak via screenshot.

As mentioned earlier, SwiftRiver will be available in the coming weeks. We will release the roadmap, a new look and feel, and provide an outline on how you can contribute to the next Swift. You can connect to the Swift community via our google group: swiftriver@googlegroups.com

The joint project combines the efforts of multiple partners including Al-Jazeera, Ushahidi, Souktel, Crowdflower, the African Diaspora Institute and the wider Somali Diaspora. The basis of my pitch to Al-jazeera was to let ordinary Somalis speak for themselves by using SMS to crowdsource their opinions on the unfolding crisis. My colleagues at Al-jazeera liked the idea and their editorial team proposed the following question:

Al Jazeera wants to know: how has the conflict of the last few months affected your life? Please include the name of your hometown in your response. Thank you!

So I reached out to my good friend Jacob Korenblum at Souktel. He and I had been discussing different ways we might combine our respective technologies to help in Somalia. Souktel has been working in Somalia and providing various SMS based solutions to several organizations. Jacob had previously mentioned that his team had a 50,000+ member SMS subscriber list. This proved to be key. Earlier this week, the Souktel team sent out the above question in Somali to about 5,000 of their subscribers. An effort was made to try and select geographically disbursed areas.

We’ve since received well over 2,000 text message replies and counting. In order to translate and geolocate these messages, I got in touch my colleagues Vaughn Hester and Lukas Biewald at Crowdflower in San Francisco. Crowdflower uses micro-tasking solutions to process and structure data flows. They were very keen to help and thanks to their support my Ushahidi colleagues Rob Baker and Linda Kamau were able to customize this Crowdflower plugin to translate, categorize and geo-locate incoming text messages:

They also wrote additional software so that text messages from Souktel could be automatically forwarded to the Crowdflower plugin which would then automatically push the processed SMS’s to a live Ushahidi map hosted by Al-jazeera. While the software development was moving forward, I connected  with colleagues from the Somali American Student Association who expressed an interest in supporting this project. Thanks to them and other members of the Somali Diaspora, hundreds of Somali voices were translated and shared on Al-jazeera’s public Ushahidi map of Somalia within days. But we still need lots of help. So if you speak Somali and English, then simply follow this link.

I wanted this project to serve as a two-way conversation, however, not just a one-way information flow from Somalia to the world. Every report  that gets mapped on an Ushahidi platform is linked to public discussion forum where readers can respond and share their views on said report. So I suggested that Al-jazeera invite their viewers/readers to comment on the text messages directly. The next step will be for Al-jazeera’s editorial team to select some of the most compelling and interesting comments and to text these back to the senders of the original text messages in Somalia. This two-way flow of information can be iterated and scaled given that the technologies and workflows are already in place.

In sum, the purpose of this project is to catalyze global media attention on Somalia by letting Somali voices take center stage—voices that are otherwise not heard in the international, mainstream media. If journalists are not going to speak about Somalia, then this project  invites Somalis speak to the world themselves. The project highlights  these voices on a live, public map for the world to bear witness and engage in a global conversation with people of Somalia, a conversation in which Somalis and the Diaspora are themselves at the centerfold. It is my sincere hope that advocacy and lobby group will be able to leverage the content generated by this project to redouble their efforts in response to the escalating crisis in Somalia.

I very much hope to see this type of approach used again in Somalia and elsewhere. It is fully inline with the motivations that inspired the launch of the first Ushahidi platform almost 4 years ago today: collective witnessing. Indeed, I am often reminded of what my friend Anand Giridharadas of the New York Times wrote last year vis-a-vis Ushahidi. To paraphrase:

They used to say that history is written by the victors. But today, before the victors win, if they win, there is a chance to scream out with a text message, a text message that will not vanish, a text message that will remain immortalized on a map for the world to bear witness. What would we know about what passed between Turks and Armenians, Germans and Jews, Hutus and Tutsis, if every one of them had had the chance, before the darkness, to declare for all time:

“I was here, and this is what happened to me”?

Use #SomaliaSpeaks to witness the project on Twitter.

I want to specifically thank the following individuals who put an incredible amount of time and effort (most pro bono) to make this project happen: Robert Baker, Linda Kamau, Michael Moszczynski, Katie Highet, Jacob Korenblum, Vaughn Hester, Mohammed Dini, Hamza Haadoow, Andrew Jawitz and of course the excellent Al Jazeera team in Doha. Thank you all for going above and beyond to make this happen.

]]> http://blog.ushahidi.com/index.php/2011/12/08/somalia-speaks/feed/ 29 http://blog.ushahidi.com/index.php/2011/11/11/what-is-the-next-step-for-ushahidi-verification/ http://blog.ushahidi.com/index.php/2011/11/11/what-is-the-next-step-for-ushahidi-verification/#comments Fri, 11 Nov 2011 08:57:41 +0000 hford http://blog.ushahidi.com/?p=6164 As Ushahidi ethnographer, my job is to do on-the-ground research on users’ experience with our technology in particular contexts. Something that we’ve been thinking about a great deal as we develop SwiftRiver is the process of verification, the ways in which technology and society work together to create useful, trustworthy and actionable information, as well as where the technology in particular contexts might be failing.

With over 20,000 installations of Ushahidi and Crowdmap since January, 2009, Ushahidi has been used in a number of different contexts – from earthquake support in Haiti, to reports of sexism in Egypt, to election monitoring in the Sudan. In each of these cases, a map is publicized and individuals are encouraged to send reports to it. The process of verifying information reported by the crowd has taken on a variety of different forms depending on the needs and affordances of the environment and the community supporting it.

The memo I just published on scribd introduces the concept of verification, how it has evolved at Ushahidi and in sample deployments, alternative ways of thinking about verification and some suggestions for further research. Its goal is to inform developers and designers as they develop the next generation of Ushahidi and SwiftRiver software to meet the needs of our users rather than prescribing what should be done.

Ushahidi support for verification has until now been limited to a fairly simple backend categorisation system by which administrators tag reports as “verified” or “unverified”. But this is proving unmanageable for large quantities of data and may not be the most effective way of portraying the nuanced levels of verification that can practically be achieved with crowdsourced data.

What research needs to be done to test verification alternatives so that Ushahidi and Crowdmap deployers are provided with due diligence tools that can advance trust in their deployments? Can we do this in a way that doesn’t add any new barriers to entry to those who need to have their voice heard on Ushahidi? How can we ensure that this solution is as close as possible to the needs, incentive systems and motivations of deployers and users? What is the next step for Ushahidi verification?

The Ushahidi core team works from 7 different timezones ranging from Kampala to Louisville, soon expanding to places like Brazil and Korea. One weekend a year we’re able to get together, in-person, to solidify our connections with each other and talk through the big strategic topics that are best done face-to-face. It could be argued that it’s the most important 3 days of the year for us.

The First XV

2010 was a big growth year for Ushahidi, where we got up to 12 core team members – doubling in size from 2009. We’re adding 3 more people this year, which brings us to 15, a fortuitous number for the team as many of us are big rugby fans.

(Caleb decided to have a little fun, putting us all in our positions based on the date that we joined the team.)

12 Months Later

Last year we met in Miami, as we are this year, and a lot has happened since then. To name the big ones:

• Plugins – extensible way to add new functionality without bloating the core
• Crowdmap – maps for non-developers, also a means to quickly collect reports giving deployers time to install their own server
• SMSSync – simple and robust alternative to Frontline and Clickatell
• iOS – rich smart phone experience
• Checkins – opens platform to entirely new uses
• Stand-By Task Force – game changer in disaster response
• J2ME – extending reaching onto older devices
• Community Site – fantastic documentation
• Map Geometry

Looking at the historical record, it’s been a good year. However, there’s a lot more to do. At this meeting, besides drinking a Mojito on South Beach, we’ll get into some of the big future-looking issues, such as:

Visual Reporting: What’s the perfect Ushahidi dashboard? How do we surface “power stats” for Ushahidi deployments and metrics. Swift-Ushahidi integration visuals on the front and back end.

Knowledge Management: How do we come up with a plan to capture information that we know internally, so that it is shared with deployers and developers better?
The inverse, how do we handle and capture information that our *users* know regularly?

Crowdmap Scalability & Migration: Making sure that even the biggest deployments work on Crowdmap. Adding in new a la carte features, etc.

Of course, this is a chance to discuss some of the more mundane items as well, around operations, funds and how we work towards organizational financial sustainability as well. It also means that we’ll be offline from today until about Tuesday of next week. We’ll be a little slower on email and other communications mediums, but bear with us as it’s for a good cause.

“Our community really gives us strength and has made the platform what it is today.” – Brian Herbert

Brian Herbert is the Director of Crowdmap at Ushahidi. He largely runs the Crowdmap.com site, trouble shooting bugs and defining roadmap for Ushahidi’s SAAS offering, he also contributes heavily to core Ushahidi platform code.

What’s your day to day at Ushahidi like?

There is no such thing as a typical day with Ushahidi. There are constantly new challenges taking us from one task to another. Essentially, no person is a 100% software developer or 100% customer support representative. We are really dynamic in that sense; where everyone has their hands in all parts of the business. My time is still spent mostly doing software development on Crowdmap and the Ushahidi Platform, incorporating feedback from deployers into our platform and solidifying our offerings.

How did you get involved in the software/tech space?

In high school (2000) I started a website called bestsiteontheplanet.com, which is no longer in existence. My friends and I like to say it was Facebook before Facebook, with profiles of my friends, message boards and blog posts. The thrill of running a relatively popular website was enough to get me interested in jumping into software development. Between bestsiteontheplanet.com, going to college, working at CNET and teaching technology in the Peace Corps in Kenya, I’ve ended up where I am today, working with one of the best teams imaginable.

What are some of your favorite apps for work and how do you use them?

Aside from webapps, my set of tools are MAMP, GitX, Coda, Skype and Terminal. That’s really all you need to get anything done these days. A few things that I use that make life easier are TinyGrab, Alfred, Sparrow, Dropbox, Adium, Evernote… The list goes on but that’s what happens when every application does one task and does it well.

What are some cool projects you’re working on at Ushahidi or; What excites you about your work right now?

I think our CI (checkins) functionality is going to revolutionize the way people use and submit reports to Ushahidi deployments. Coupling this with Crowdmap and improving our service offerings are going to make it really easy for people to get a robust, dynamic citizen generated data gathering machine up in running in the amount of time it takes to get a Big Mac from the drive thru at McDonald’s. It’s a really exciting time for Ushahidi, but then again it has been exciting from day one, over three years ago.

What helps you make it through each day?

My bottomless cup of coffee from Quills Coffee in Louisville, KY. Rent is cheap!

The one thing you can’t live without?

I couldn’t live without the amazing team of developers, testers and deployers on SwiftRiver and the Ushahidi Platform. Our community really gives us strength and has made the platform what it is today. Oh, and my coffee

We wanted to publish a series of findings from this study, including the material used to present it (the slides).

The research reports in this series were written by Sarah George and Dragana Kaurin with a lot of help from the Ushahidi staff and various deploying individuals and organizations. You can find the raw datasets and collected research in full at http://community.ushahidi.com. We’re sharing these findings with our community, researchers interested in our platform, journalists looking for such information and to anyone else who finds this useful.

Note: Numbers are as accurate as possible but there are some obvious gaps: Downloads does not include some methods of downloading the software. For instance, from github or third-parties, and obviously not downloads that have been shared locally once downloaded. Unique views is data that was collected from some deployments but that was a small minority so the actual number there is likely many magnitudes higher and is not conclusive. For the purpose of this study, a report is any message received by the respective deployment, whether by Twitter, SMS, Email or Web feed, approved or not.

Key Deployment Report (2008 to 2011)
by Sarah George

14 pages. Excerpt. Download the full report.

From the first quarter of 2009 to the present, Ushahidi, SwiftRiver and Crowdmap products have been deployed in a variety of scenarios such as human & natural disasters, elections monitoring & observation, tracking incidents of crime and civil unrest, promoting peace initiatives, documenting the impact of the “Deepwater Horizon” oil-spill disaster, crowdsourcing citizen response to Russian wildfires, visualizing the urban landscape in Prague and New York City or mapping the disruptions caused by the London tubestrikes.

To date Ushahidi has received over 7,761 requests to download the software from organizations and individuals around the world and over 10,521 mobile application downloads. This is a testament to the adoption of the platform worldwide. The platform is easily customized and localized for use in different contexts: it receives inputs from multiple sources (SMS, Email, Web, Twitter), provides online data visualization, and enables users to subscribe to alerts via SMS, Email or RSS.

There are currently 4,980 externally hosted Ushahidi sites that have received more than 50 unique visitors, 6,462 cloud-hosted Crowdmap instances and 47 sites are hosted on Ushahidi servers. In all, externally hosted sites have received over 2,102,959 unique visitors and 190,439 reports. Since launching in August 2010, Crowdmap instances have tracked 539,622 uniques and 35,332 reports. As of December 2010 sites hosted by Ushahidi had received more than 705,734 unique visitors (600,000 attributed to the Haiti and Chile deployments) and 20,804 reports. There have been more than 11,000 Ushahidi deployments to date (see chart below).

The full document is embedded below.

Below you’ll find the slide deck that accompanied this report.

[Guest blog post by Heather Leson, an idea hacker and community builder. Haiti changed her life and inspired her to become a serial volunteer for digital response (CrisisCommons, CrisisMappers, Stand-by Task Force) and an Ushahidi user for the past year. She blogs at textontechs.com and tweets on @heatherleson]

Ushahidi Mappers in Canada!

Be still my prairie girl heart. Laura Madison and Dale Zak spent the winter preparing for the spring floods in Manitoba and Saskatchewan. Extensive flooding is expected in both the Red River Valley and North/South Saskatchewan River regions. This is the first time that Ushahidi has been used to prepare and to report the floods in Canada. In fact, both maps are the first time that full scale Ushahidi maps have been prepared as part of the citizen reporting and digital volunteer response in Canada. There are Canadian university classes learning Ushahidi and CrisisCamp Toronto conducted an Ushahid beta test in February 2011. Laura is very active in the Ushahidi and Stand-by Task Force communities. Dale is active in the Ushahidi team. Each of us has mapped events around the world. And, we are delighted to see mapping come home.

How to help

Ushahidi is an open source project. There are three types of help required: Mappers, Developers, and Digital Volunteers. Maps are community-driven crowdsourcing. In the coming weeks, the needs will change daily or hourly. This is the beauty and curse of volunteer digital response. Both leaders have been in contact with official responders. However, at this time, their efforts are for citizen response and collaboration.

MBFloods and SKFloods maps allow for reports to be filed by webform, Ushahidi app, iphone, android, or email. Maps allow for various layers of useful open data to be added. People will be able to add news reports, pictures and videos. Maps evolve depending on the community use. So the needs will change over time. The work they have done is fantastic. Bring on the crowd! People are needed to file reports, coordinate mapping teams for about 2 weeks. Laura and Dale will be coordinating this adventure via Skype. I am sure that the great mapping communities may lend a hand. If you are an individual volunteer or are part of a volunteer technical community, please consider contacting:

Contact Dale for Saskatchewan: skfloods AT gmail DOT com
Contact Laura for Manitoba: flood DOT mbfloods DOT ca
Participate in the joint Skype channel: Add Laura (organization9) to get started.

Types of help required:

Preparedness

Laura has added layers for the Manitoba RCMP and Manitoba First Nations. She would like to add more layers. Mappers most welcome to churn out KML/KMZ files. She also needs some PHP help and Ushahidi expertise.

Volunteer Recruitment and Training

Both maps will need digital volunteer teams to support the mapping. The types of content you will be adding is geo-location, media monitoring (mainstream, twitter, facebook), and handling the various streams of online reports (webform, apps and email.)

About Ushahidi

To learn more about Ushahidi, see recently released Ushahidi Manual, created by the lovely crisismapper Anahi Ayala Iacucci. It outlines how to get started with Ushahidi and implement a successful deployment. Maps need a process, this Ushahidi Practical Considerations is also very helpful.

Feel free to contact me as well (heather at textontechs dot com) if you want more information or want to be connected to Dale or Laura.

Heather L.

You can see it live here: http://demo.ushahidi.com/japan. On the live map, you can toggle the clustered numbers on/off using the “stack” button on the right side of the map.

Japan earthquake Ushahidi data, heatmapped

We’re thinking of doing two things to make this more useful:

1. Making it filterable by category.
2. Creating it as a plugin so that anyone can put it on their own Ushahidi deployment.

We’d like to hear from you. Is this valuable or useful? What other visualizations do you think would make sense to do?

Since then it has been a crazy ride…from producing an incredible open source platform and working towards scale, to building and working with an incredibly talented team, to seeing multiple uses of Ushahidi around the world, to numerous awards and press mentions.

For me, what has always been the most important aspect of the work we do has remained simple, building a tool that makes it easy for individuals and groups to tell their stories, and making it easy for these stories to be mapped/visualized.

Ushahidi has grown to be that and much more, thanks especially to the wider community – which saw potential uses beyond crisis reporting and who largely shaped our growth and direction to date be it through translation efforts (Ushahidi now available in 10 languages!), or custom themes, or pushing for a hosted version (Crowdmap), or challenging us to address the shortcomings of the platform (through tools like SwiftRiver and our community resources page).

Beyond the growth of Ushahidi as a platform and an organization, I always tell people that I am most proud of the fact that the Ushahidi story has provided an inspiration to other techies in Kenya and Africa – an example of the kind of talent the continent holds, but also a reminder that we have just scratched the surface. And so after 3 years, I feel it is time for me to take on the next challenge. Those of you who know me well know I’ve got a 1001 ideas floating in my head that I need to get out☺

Where I am headed? I will be joining Google in the new year as the Policy Manager for Africa. The role will involve developing policy /strategies on a number of areas of relevance to Google and the Internet in Africa and will involve working with different parties including government leaders, policy makers, regulators, industry groups and so on. It is a huge opportunity to bring Google’s resources to bear as far as the growth and development of the internet in Africa (and hopefully a reminder of why I went to law school in the first place!). I’m very excited about the move and I hope I can continue to lean on your support and insight in my new role.

To my co-founders – the ride continues! To the most amazing team, I am watching this space! To our Board of Directors, thank you for your insight and guidance! To our partners, especially those who took a risk on us in the early days, most grateful! To the wonderful readers of Kenyan Pundit, whose stories and willingness to share in those dark days of 2007-8 – you were my inspiration, thank you! To the wonderful wider community of Ushahidi – volunteers, translators, crowdmappers, critics (yes I love you too!), journalists, people who supported us in the early days when people asked Usha-what?, THANK YOU THANK YOU.

Need to reach me….you all know I live on the internets right…find me @kenyanpundit or kenyanpundit-at-gmail

- O

A few months back Jon Gosier dropped by the Twitter offices in San Francisco to discuss the importance of Twitter as a resource for crisis mapping and humanitarian response community. For our own platform, Twitter often accounts for much of (if not most) of the dataset that groups deploying the Ushahidi platform will map. With it’s 190 million global users and 50 million tweets per day, Twitter has proven itself to be one of the most valuable (and accesible) real-time information sources.

Twitter for Disaster Response

The SwiftRiver project was, in fact, born out of the need to help Ushahidi users cull and curate data because of the overwhelming flood of data aggregated from sources like Twitter, SMS, Web feeds and email. Part of our mission is to make the process of mining these real-time information channels easier for the crisis response groups who need to sort actionable information from the ‘noise’ using the most efficient methods available.

Twitter’s Claire Diaz Ortiz (Manager of Philanthropic and Social Innovation) explains their company’s role in the disaster response community…

It is important to remember that mobile updates in disaster response not only serve to inform, but to provide direct aid as well. As in the Haiti earthquake and the floods in Pakistan, crisis-mapping tools that collate data received via SMS and other means and then plot it on interactive maps take a key role in connecting victims sending mobile updates with rescue workers and aid organizations who can offer help. This week here at Twitter HQ, we were happy to speak with Jon Gosier, Director of Product for SwiftRiver at Ushahidi. Organizations like Ushahidi build interactive mapping tools that democratize information, increase transparency and lower the barriers for individuals to share their stories.

First Responder is an Oxymoron

Below, Jon from Ushahidi’s SwiftRiver team explains how Twitter empowers first-responders, a word we’ve co-opted to refer to the survivors and victims of disaster as opposed to the groups who it traditionally refers to (paramedics, security etc.)