There’s a problem that constantly plagues us in this day of information overload, and that is the ability to sift the stream of incoming information into the bits that are valuable from those that aren’t. It’s a tough issue that we’ve been working on at Ushahidi, and re-working, a solution on for a while now. Our solution is called SwiftRiver.

SwiftRiver is a free and open source intelligence platform that helps people curate and make sense of large amounts of information in a short amount of time. In practice, SwiftRiver enables the filtering and verification of real-time data from channels such as SMS, Email, Twitter and RSS feeds. It’s especially useful for organizations who need to sort their data by their unique expectations of authority and accuracy, as opposed to popularity. Such organizations include journalists, community based-organizations, PR/marketing, emergency responders, election monitoring groups and more.

SwiftRiver, In Plain English

There’s a torrential river of information that’s constantly flowing on the Internet. If you dive into that river in search of something, chances are you’d drown.

Now before we go any further, let’s first define this river of information. Simply, the “river” is made up by billions of bits of information. In the context of SwiftRiver, we call these things “droplets.” For example, common droplets in the river are tweets, Facebook updates, and blog posts. These are common examples, but by definition, things like text messages, emails, and even rows in a database table are considered droplets, too.

So how does SwiftRiver help you make sense of all these droplets? Well, it analyzes droplets much like your brain analyzes the world around you. For example, when you look at a kitchen table, your brain instantly determines its color, height, location and texture. In the same way, when SwiftRiver looks at a droplet, it determines all its attributes. For example, if SwiftRiver looks at a tweet, it can determine things like location, time, author and meaning (in the form of keywords). Generally speaking, SwiftRiver does this using a process known as “natural language processing.”

Once SwiftRiver analyzes all the droplets, you then have the ability to filter them down from that torrential river to a manageable stream. In addition to filtering, you can run different analyses on them, helping you get the “big picture” of your set of droplets.

SwiftRiver Glossary of Terms

Droplet: The basic unit of content inside SwiftRiver, i.e., a tweet, a Facebook update, a blog post, an SMS text message, etc.

Identity: The originator of a droplet from a specific channel, i.e., a Twitter account, a Facebook account, a phone number, an email address. Identities are automatically extracted when a droplet is “siphoned” from a channel.

Source: Comprises one or more identities, and could be a person or organization. Unlike identities that are automatically extracted, sources are subjective and put together by users in the system.

Channel: The vehicle for transporting a droplet into the river, i.e., RSS, SMS, Twitter, JSON, XML, etc.

River: The torrent of droplets that come from the predefined channels.

Filter: The mechanism for reducing a channel or river from a torrent of droplets to a more manageable set.

Stream: A collection of droplets whose contents are defined by a filter or a combination of filters.

Bucket: A group of hand-picked droplets from a stream.

Trends: A narrative based on the droplets in a bucket. Components: description, time, sources, places, media (links), tags, theme.

Types of stories, i.e., outputs:

• Text
• Maps
• Timeline
• Graphs, Charts, Heatmaps
• Gallery: Photos, Video, Audio

A sneak peak at the rebooted platform

Since the end of this summer, the Ushahidi team has been focused on rebuilding much of the platform so that it would dovetail perfectly with the Ushahidi core platform and Crowdmap. So, it’s a standalone application that anyone can access and use by itself, but it also answers our users’ need for a faster, more nimble way to manage information flow on their deployments.

Back in August, we completed the first iteration of our ID and authentication system, RiverID, the first step needed for us to have a collaborative profile-based tool for SwiftRiver services. In October, we locked four of the Ushahidi team, made up of two designers and two engineers, away in a cabin, deep in the woods of Georgia. Their job: Do two months’ worth of work in six days.

This they did.

While we haven’t released the code yet, the beta will be available in mid-December (three weeks) for the community first, then for the general review. But, in an effort to quench your curiosity, here’s a sneak peak via screenshot.

As mentioned earlier, SwiftRiver will be available in the coming weeks. We will release the roadmap, a new look and feel, and provide an outline on how you can contribute to the next Swift. You can connect to the Swift community via our google group: swiftriver@googlegroups.com

“It’s in the excess that you find the gems.” – Jon Gosier

What’s your day to day at Ushahidi like?

To say that each day is unpredictable is an understatement. I’m a meticulous planner and notetaker, it helps me make sure I’m staying on task and getting things done, but the reality is so many random requests or urgent emails come in that all my planning goes out the window to deal with new stuff. So finding the balance between urgent now, and urgent now-now (as my friends say in East Africa) is critical.

My role at Ushahidi is directing the SwiftRiver project, managing the Swift community, designing product, managing staff and developers, business development and a lot of speaking engagements. I also do a lot of design and marketing – the Swiftly.org website, the copy, the videos, the webinars, the newsletters, the print material are also things that I do.

How did you get involved in the software/tech space?

When I was around 10 or 11 my mother got us our first family computer, an IBM PS/1 and I promptly dropped whatever comicbook I was reading at the time to explore this new world of interactive media. I don’t know if I had seen the movie WARGAMES at the time or not, but it was stories like that that fascinated me. I was never that good of a programmer but trying to teach myself what I could in those early days forced me to discipline myself. There was no one in my family who could program and there were no classes I could take (at the time) to learn. So I was content teaching myself.

I’m a very visual person which is what originally drew me to the area of data visualization. I went to an art school, Savannah College of Art and Design (SCAD), where I was focused on a career in Entertainment. However, I worked in the Music industry, the Film industry and Advertising before I finally wound up at a fledgling series of startups in Atlanta, Georgia. In 2008 I moved to Uganda and started my own software company, Appfrica.

What are some of your favorite apps for work and how do you use them?

The basics: Gmail by-far is the web app I spend the most time in. Skype is invaluable for me, anyone who is regularly on conference calls that span seven timezones or more knows what I’m talking about. TimeandDate.com is another one, for scheduling and planning across timezones.

Basecamp from 37 Signals is another app that’s integral to my workflow. I mentioned before that I’m a meticulous planner. It’s because I’m forgetful, if I don’t write it down (preferably on a screen) I don’t remember to do it. Basecamp I like a lot because I can keep appointments as milestones which show up on my calendar. At the same time, I can plan todos and communicate around those same milestones for managing teams and projects.

I try to use all apps at least once. I use CloudApp for sharing screenshots, Github for managing code, GeckoBoard as a visual dashboard, Chrome for browsing the web, Keynote for presentations.

I’m an avid user of the entire Google Suite but in particular Docs and Reader. As a data-junkie, I love an excess of information. I subscribe to every blog under the sun in Google Reader and then just let the ‘magic’ filter they have recommend what’s interesting from what’s aggregated. A lot of people get overwhelmed by the number of unread content in Reader, but for me that’s the point – if it’s interesting it gets surfaced or you can search random words. It’s in the excess that you find the gems.

What are some cool projects you’re working on right now at Ushahidi or; What excites you about your work right now?

RiverID is the big one. For a long time people haven’t really understand what we aim to achieve with Swift. The apps like Sweeper and SwiftMeme are somewhat irrelevant to the greater goal, which is to truly provide a platform for vetting data. RiverID is a global distributed trust system that builds on a lot of public data from past Ushahidi deployments, data from other trust systems as well as the APIs of various social networks. You can read more about RiverID in this NewYorkTimes article.

Beyond that, I’m really excited that we’re moving into various areas of data science. I feel the value we add as a company is that we produce software tools that make data science relatable to the general public.

What helps you make it through each day?

A sloppy Italian Sub for lunch. If you don’t feel like you need a shower afterwards, it’s not a good sandwich. Also, I really like the people at my co-working space Affinity Lab in DC.

The one thing you can’t live without?

Data. No input, no output.

We like to say that our mission with the SwiftRiver project is to democratize access to the tools used for understanding information. To me that means taking the hard-work out of drawing insight from excessive quantities of data, to help humans process things more efficiently. That’s why it was huge honor to announce the SwiftRiver project’s ongoing collaborations with software developer Pete Warden earlier this year.

Earlier this week Pete announced a cool project closely aligned with our mission called the Data Science Toolkit:

The Data Science Toolkit is a collection of data tools and open APIs curated by our own Pete Warden. You can use it to extract text from a document, learn the political leanings of a particular neighborhood, find all the names of people mentioned in a text and more. He unveiled it today at GigaOM Structure Big Data in New York City.

It’s available as a Web service, or you download a virtual machine and host it on your own server.

• Street Address to Coordinates – Street Address to Location calculates the latitude/longitude coordinates for a postal address.
• File to Text – Converts PDFs, Word Documents, Excel Spreadsheets to text. Recovers text from JPEG, PNG or TIFF images of scanned documents.
• Coordinates to Political Areas – Returns the country, region, state, county, constituencies and neighborhood a point is inside.
• Geodict – Geodict pulls country, city and region names from unstructured English text, and returns their coordinates.
• IP Address to Coordinates – IP Address to Location calculates country, state, city and latitude/longitude coordinates for IP addresses.
• Text to Sentences – Removes any parts of the text that look like boilerplate instead of real sentences.
• HTML to Text – Returns the full text that would actually be displayed in the browser when an HTML document was rendered.
• HTML to Story – Takes an HTML document representing a news article or similar page, and extracts just the story text.
• Text to People – Spots text fragments that look like people’s names or titles, and guesses their gender where possible.

The DSTK project joins a number of similar open data science tools on the market. Increasingly there’s a need for people of all types to own and control their own data in ways that are easy to utilize or deploy. It’s one of the reasons people use Ushahidi products, apps like ours lower the barrier to entry for those who want simple ways to collect or visualize data. Hence the reason we’re actively contributing to GeoDict and the greater DSTK initiative.

Find out more at datasciencetoolkit.org.

PDF | Video | High-Res Image

• The US is clueless in response to the Wikileaks release of war documents
• The music and film industries continue to lose to the open web
• South Africa seeks to muzzle the press
• The UAE takes on RIM over the ability to read everyone’s email
• In Australia you can’t link to certain sites from your personal website
• More and more countries require SIM card registration on phones to track users

…it goes on.

This isn’t a conspiracy, it’s a reaction by those in the status quo (be they government, big business or large established organizations) to the inefficiencies that they represent in the system being overcome by changes in technology and culture. As the open web expands it becomes a real threat to controlling governments, even to the relevance of the nation state itself.

The Case of the Russian Fires

Blogging and social media have been utilized for transparency and accountability for a number of years. While that’s interesting in its own right, I find the translation of those online tools into offline activities far more compelling.

The Russian forest fires is a particularly interesting one, as it represents what appears to be a major shift in ownership and attitudes in Russia around governance and responsibility. At the same time, one of the main tools used to organize it was the Ushahidi platform (giving me that little bit of liberty to write some thoughts on the bigger picture).

“On the one hand, cooperation was empowered by a shared understanding that the government has failed to get the situation under control and, moreover, didn’t want to be held accountable for it. On the other hand, it was information technologies that provided both information exchange and tools for coordination and effective collaboration.”

Please, read the full Global Voices article on the Russian fires.

Final Thoughts

Personally, I don’t see too many governments being displaced or replaced by online cooperation alone. Trust, reputation and resources are just a few of the hurdles to overcome before that happens. Instead, I think we’re seeing the continuation of the refinement of mass movements, brought about by the inefficiencies in the system, which catch on faster and are enabled better online and then move offline for impact.

Solar Flashlight & Kazoo created by the students

To give the kids a real-life example of the changes that are happening in their community, we built a modified Ushahidi map with data overlays from the NYC Data Mine and Recovery.gov. I explained that these are the government’s official data related to spending and therefore allows for the reporting of potential fraud, waste, and abuse, but also for innovative new solutions by identifying the gaps. To make it personal, I asked them what they would do if they knew how much money their school was getting compared to the neighboring school. Not only did that set off a flurry of ideas from the students, but the teachers got pretty excited as well. This already started to show that opening government data can impact the lives of everyday New Yorkers and lead to a smarter city by getting citizens young and old involved in urban planning.

Another exercise that I had the kids run through was stating a mock vision: of the year 2020, gasoline would be expensive, the environment polluted, cars more scarce, and so encouraging the city to place a bike rack in front of my office would enable people to bike to work, making the city more peaceful, healthier and cleaner. Plus, if the government thought a bike rack existed where one didn’t, I could let them know about their error. In this case, I overlayed “Bike Racks”, as a set made available in the Geo Data Catalog. I then asked the students to brainstorm their own scenarios for the year 2020. The kids had a field day dreaming up solutions and adding them to the map. You can visit the website or see it embedded below to see their ideas. When working with young people, it’s important to keep in mind their protection and security, and so of course viewers of the site will notice that their personal information remains private.

Our Ushahidi lesson plan builds off of our participatory collaborative learning curriculum from around the world, Project Einstein. In this case, it was exciting to see the successes of incorporating a tool that integrates lessons from across different discipline: geography, computer science, economics, math, art, social studies, etc. But each local context reveals new insights for our culturally-specific programming and this case was no different.

The biggest problem I ran into was spell-check. I noticed that the students were taking an unusually long time to fill in their reports and after looking into it, found that when students were entering in their main body of information, a line would show up automatically under misspelled words. Every time this happened, students would backtrack and try to figure out the right spelling. This happened so often that I estimate it took about twice the time to create each entry as it otherwise would have. In the places where there was no spell-check, like in the titles, the entries are littered with bad spelling, but they were entered quickly. Our work confronts language problems head on, mainly working with visual media such as maps, photos, videos, etc that can allow people to connect beyond these barriers. It’s important to consider language barriers even with native English speakers as well. And in NYC, it’s even more complicated, with our students coming from places as varied as Tibet, Thailand, Congo, Madagascar and Brooklyn.

“OMG kids are on Facebook!” is one of my favorite challenges that also arose quickly. Two skateboarders had finished mapping their vision for perfect place for a skatepark in their community ahead of the other students and got distracted, finding themselves wandering the internet and logging into Facebook. Instead of scolding them and demanding that they go back to our site, I told them that no other students had added a photo to their posting so could they find the best photo to go along with their post, to make it easier for a politician to see exactly what they had in mind. The hunt was on, and they indeed found a great photo, without another distraction. To me, this is a key aspect to the model of 21st century education – information management. Can students find information that is going to add value to their post. Do they know whether it’s creative commons and how publicly it can be used. There are still many steps before getting to that point, but this is a start. Ushahidi serves as a strong tool because the barrier to entry is low, but the opportunity to dig deeper continues. Thanks to the new plug-in architecture, they can even proudly display that skate park on their Facebook wall.

While technology access is growing in our schools, so is censorship. These kids were astonished to see their work up and live on a website that is free and accessible to anyone. Increasingly, there is a limit to what they can access due to filters and firewalls, and what they can publish because the media that they’re producing has work that can’t be licensed, such as the videos they had made about New York that features the song “Empire State of Mind” by Jay-Z & Alicia Keys. Due to alleged copyright violations, their work can’t be screened to other kids. School banned website lists resemble the ones in Tunisia, a notoriously closed society.

To ensure the continuation of the open web and that students are given increasing access to powerful and empowering tools like Ushahidi, Digital Democracy used this instance in our lobbying efforts, testifying to the New York City Council Technology Committee on Open Data. Our testimony, available here, details how free and open source technologies, coupled with open data and progressive 21st century schools can foster positive engagement between students, their government and their community.

Whether working with kids or lobbying to government, I’m thankful to be able to use such a flexible and interesting tool to convey the message that technology can be used for civic engagement. And as a native New Yorker, the diversity of ideas, skills, backgrounds and approaches in this project reminds me how much I love this city. I just hope that we stop censoring it and start supporting more of these kinds of initiatives in the future and by 2020.

Future Now Ushahidi Map

[iframe http://handheldhumanrights.org/nyc/external 600px 475px]

The following visualizations represent the various methods that go into calculating the reputation and veracity scores for users and content within the SwiftRiver platform. They are in part a response to this comment from reader Charles Bernard on this post. His comment:

In many instances, there are entities with a vested interest in preventing valid information regarding things such as voting, battles and even disasters, both natural and man-made.

For nearly any human effort, there exist a group of entities which would profit by either the details or the extent of a problem being kept from the public–and that can include relief agencies.

While tracking particular sources and their validity of reports is a step in the right direction, some entities, in particular governments and large corporations have access to the resources needed to generate thousands or even 100,00s of thousands of false data reports, flooding the system with misinformation.

In other words, what steps are we taking to prevent individuals with malicious intent from gaming SwiftRiver? Here was my response:

With Swift, we aren’t just validating content, we’re also validating users, users validate each other and content validates users. Content can also be used to verify other content. This creates a system that’s difficult to dupe, as one looking to falsify information would need to thousands of false reports from a number of different ‘users’, locations, and media channels.

What would be absolutely possible is for a group to download Swift, set up their own instance with all sorts of fake information and publicize it as fact. However, our distributed, decentralized reputation system River ID would show that outside of that instances ‘ecosystem’ no one trusts those users, or the instance. If the administrators opt out of tracking…they also forfeit any sort of benefits that come from River ID (trust from users who don’t know you or your site). In this case falsifying information is indeed easy, but promoting it becomes self-defeating, as the more people who aren’t under your influence see it, the less authority your Swift instance (with all it’s fake reports) actually holds.

I thought these concepts might be hard to grasp so I made the following Arc Diagrams to give a visual representation of what I actually mean. Click the images for high-versions. In the images below, the light grey color is simply used to indicate that content isn’t important for what that particular chart is showing you.

Fig. 1 Individual Voting Against the Community

Figure 1 represents the most classic scenario of ‘gaming’, spam, bots or human individuals who are trying to vote bogus content ‘up’ so it will be weighted higher than other content. Section “A” represents User 1. Section “B” represents the activity of User 2 (our spammer). Section “E” represents the community within this particular Swift instance. Section “F” represents the users of our distributed trust system River ID or the global SwiftRiver economy. Section “C” represents individual content items. Section “D” represents the source that content is coming from.

The thickness of the lines connecting the users to the content and the source, represents how they’ve voted on those particular things. The thickness of the line for User 2 tells us that he’s rating these things very highly. Perhaps they come from his blog, and he wants them at the top! The thickness of the lines from the local community of the SwiftRiver instance as well as the global users tells us that these content sources are suspect. We can see that User 1 (who represents our average, active user) is voting closer to the how the community is voting, in fact even harsher than the community votes both the content and the source (represented by thinner lines).

This dynamic relationship between users and their interactions with content (in contrast to the local and global community) is considered when scoring users, content, and the sources. In this case the person voting against the tide is actually damaging his or her own reputation both locally and globally. However, this isn’t the only thing we consider, otherwise it would encourage conformity which also isn’t good (sometimes the outlier knows something the rest don’t.)

Fig. 2 Factors Considered in Rating Content

In Figure 2 we can see that things like Time, Location, Activeness as well as Global and Local interaction, are all considered. Time (green) and Location (dark grey) are optional, for scenarios like a conflict or war. The content producer’s location, or proximity to ‘ground zero’ tells the system to factor this in to its score. Also the length of time that content is produced after the initial event may also tell us a lot. Things like ‘time’ and ‘location’ are optional because if your Swift instance is tracking something like a political scandal, time and proximity may not actually add any value to authority calculations.

Purple represents how active Users 1 and 2 are. In and of itself how much someone uses a Swift instance is irrelevants. It could mean that they are an eager member providing valuable assistance, or it could mean they are attempting a brute force attack on the system similar to the Figure 1 scenario. However, when coupled with other factors, frequency of interaction is considered and can positively or negatively weight the score for a user.

Fig. 3 Ratings Visible to Users

In Figure 3 I’m illustrating what information is visibly shared in the scenarios above. The trust the local community has for Users 1 and 2 is displayed. The trust the global RiverID system has for Users 1 and 2 is also displayed. Thus, the trust Users 1 and 2 should have for each other is inferred.

Swift’s strength is in multiple points of redundancy. All scores are calculated against a multitude of other factors which may or may not be independent to the local community. This allows users to build scores more organically than x=bad y=good. There are some probabilistic calculations as well as algorithmic intricacies that make all this a lot more complex (a lot of math beyond my paygrade). We also calculate things like tags and content influence which compound the complexity.

Unless the local Swift instance administrators opt-in to participating in the global Swift ecosystem, their instance only holds authority with the people using it. In theory, their ‘gaming’ would then be contained to their local Swift instance. The fact that global authority isn’t considered would be an indicator that the public shouldn’t trust it. If they do opt-in to the global ecosystem, it becomes increasingly harder to continue gaming the system, as your scores are constantly weighted against the global community’s.

Because Swift is open source, it’s easy to reverse engineer or hack parts of the local system. But this is why we announced Swift Web Services last month, core components to the global system are centralized and well protected. This protects the global ecosystem, but still allows for independent uses of SwiftRiver, and all of it’s components as open, locally deployable apps. Some users, for example election monitors, may not want their SwiftRiver instance online at all. In that case, global authority doesn’t matter, the instance can and should only be influential amongst the people using it. This is why we opted for cloud solutions in addition to local deployment options, yet another redundancy to ensure the platform’s usefulness in multiple scenarios.

Post any follow up questions to the newsgroup or in the comments below.

Swift River was given a lot of thought by Chris Blow and Kaushal Jhalla before the Ushahidi meetings in March 2009. This video is a short version of what was recorded, giving an idea of what was being discussed.

What is Swift?

It’s an initiative that seeks to do two very important things, both of which are crucial for not just Ushahidi, but for many emergency response activities in the future. First, it gathers as many possible streams of data about a particular crisis event as possible. Second, using a two-part filter, that stream of data is filtered through both machine based algorithms and humans to better understand the veracity and level of importance of any piece of information.

You can read about the human interaction part of it here, where I talk about crowdsourcing the filter.