SwiftRiver is Ushahidi’s software platform for managing large streams of data. Over the past 30 days we completely rewrote the SwiftRiver app to make it faster, leaner and easier to use. Because of the rewrite, some things like the veracity slider have been removed, but expect them to return in the next release version 0.3.0 Benga.

This latest release is the most-stable, most-exciting build released to date. There’s a lot of new features in this release, here are some highlights…

TOTAL REWRITE

We completely rewrote the core app from scratch. This allowed us to move up to Kohana 3.x and get rid of a lot of unused code. To give you an idea of how much this helped the last release shipped with a file size of around 10mb, this one is only 3mb!

The drawback was that we lost some functionality. User roles, pairing with Ushahidi, the veracity slider, and other things will return in version 0.3.0 Benga

ONE PAGE WORKFLOW

The latest Swift release really emphasizes speed of workflow. That said, we’ve completely gotten rid of the idea of having a page for working and page for administrating the site. All the action including administrative functions like activating plugins happens on one page now. There’s also no longer any pagination, the work area automatically refreshes and loads new content.

ADMIN BAR

All administrative functions (activating plugins, changing themes) are now controlled by an administrative bar located at the top of the page.

TURBINE

Turbine is our plugin platform. It’s now documented and ready for use. There are two types of Turbine plugins, impulse and reactors. As the name implies, impulse plugins process data before they hit the database. An example would be a plugin that translates text or pre-processes feeds. Reactor plugins process content after the database and allows plugins to take advantage of things like structured data and user interaction. Some plugins will be both impulse and reactor types.

SHIPS WITH PLUGINS

Some plugins will be packaged with future releases, others will be downloaded from our website. This release ships with SiLCC and TagThe.Net, two auto-tagging services, examples of impulse

More release notes for this version can be found at http://swift.ushahidi.com/doc/

Download v0.2.0 Batuque: ZIP

I just had a meeting with some Knight fellows at Stanford who have some very interesting ideas about how to use Ushahidi in a journalistic context — very exciting stuff.

As a way of giving some quick advice, I drew this little chart in the meeting to show what I think is one of the biggest problems with most launches: the “if you launch it they will come” idea. (As David Kobia puts it.)

The simplicity of Ushahidi setup sometimes leads to some crestfallen administrators.

Just because you bought a domain name and ran the Ushahidi installer doesn’t mean that anyone is going to use they system — and even if you somehow get a lot of reports, you might not be relevant to the existing systems (that is, all the other people who are working on the same problem). So as Ory said in Cape Town, “Don’t get too jazzed up! Ushahidi is only 10% of solution.”

Systems like Ushahidi have turned enormous communication barriers into a trivial installation and training process. But there is a whole other 90% of real work.

One way to solve this: forget about crowdsourcing. Unless you want to do a huge outreach campaign, design your system to be used by just a few people. Start with the assumption that you are not going to get a single report from anyone who is not on your payroll. You can do a lot with just a few dedicated reporters who are pushing reports into the system, curating and aggregating sources.

A related post from Wherecamp

Curious yet? You can download Swift by clicking this link or find out more about the product here.

This morning’s release of SwiftRiver v0.0.9 Rumba is the culmination of nearly a year’s worth of planning and a few months of development time. When I came on board in December, the Ushahidi staff, Kaushal Jhalla, Chris Blow, Ed Bice and many others had already done fantastic jobs of making Swift known and prototyping their ideas. But they still needed an application, and to get it done (amidst all the other chaos like Haiti and launching the iHub) Ushahidi needed someone who could focus only on Swift for however long it took to get the job done. As a huge supporter of SwiftRiver for months prior, it was exciting to be asked to take on the task.

My philosophy was, no matter how long it takes to perfect, people want something they can use sooner rather than later. An okay product can always get better and quite a few people were very much looking forward to the final product. That said, our first goal at Team Swift was to get something done that would be tangible — something usable.

This was supposed to take two months. The course was charted, we had small team in place, and we were ready to go. Then Haiti happened and threw our whole organization (and much of the world) into a bit of choas. So Swift was delayed for a few weeks while I did my best to aid with our efforts there.

This actually ended up helped out significantly, as we now had real world scenario (albiet, an unfortunate one) to learn from. As an organization we learned firsthand the crippling effect of having too much information. In the first few days we had 50,000 reports, then 100,000, then 150,000. We were floating in a sea of data (much of it irrelevant noise, cross-chatter, spam and duplicate submissions) that needed to be verified by human volunteers. There had to be a better way to manage; this is where we found the critical need for an application like SwiftRiver.

How do you deal with ‘too much information’? How do you filter it in a way that saves time, without sacrificing accuracy? This is the problem SwiftRiver is attempting to solve.

Rumba marks the first step in this process. With it you can easily aggregate content, tag and vote. There’s also an option to ‘Pair with Ushahidi’, which will allow users to link their Swift instance with an Ushahidi instance. In that scenario, Swift manages all incoming data which is then posted as a Report in Ushahidi. Our next step is to swap out the core (for scalability reasons). And to integrate our natural language processing module SiLCC and our distributed authority system – River ID. We’ve also integrated existing semantic web applications like Open Calais and TagThe.net. You can look forward to all these things and more in our next release, Apala.

For the next few months there will be many versions of Swift in rapid succession as we iterate our way to our Beta release in August. I like to compare this entire process to long-distance running. When a runner sets out, they usually have already set their course, they already know what kind of energy they need to reserve to complete their journey, and they know that maintaining their ‘runners high’ can carry them along, far past the point of exhaustion. So for us, Rumba is like our feet hitting the pavement for a jog, we know the course, we’re saving our energy and we’re focused only on the road ahead.

In the video below Erik Hersman talks about SwiftRiver at TED.

Runner photo by Thomas Hawk

“Release early, release often.” – Eric S. Raymond

About Google Summer of Code

Google Summer of Code is a global program that offers student developers stipends to write code for various open source software projects. We have worked with several open source, free software, and technology-related groups to identify and fund several projects over a three month period. Since its inception in 2005, the program has brought together nearly 2500 successful student participants and 2500 mentors from 98 countries worldwide, all for the love of code. Through Google Summer of Code, accepted student applicants are paired with a mentor or mentors from the participating projects, thus gaining exposure to real-world software development scenarios and the opportunity for employment in areas related to their academic pursuits. In turn, the participating projects are able to more easily identify and bring in new developers. Best of all, more source code is created and released for the use and benefit of all.

Ushahidi GSoC Projects

For projects we’re working on or ideas on what to contribute as part of the GSoC program, please visit http://swift.ushahidi.com/extend/ideas/.

For Potential Applicants

For people interested in participating in Ushahidi’s Summer of Code projects, please review our organization’s projects here and then fill out the application form here.

Other Relevant Links

mailing list – http://groups.google.com/group/swiftriver
IRC channel - irc://irc.freenode.net/#ushahidi
skype public – click here
facebook – http://www.facebook.com/pages/Swiftriver/362720609137

In anticipation of the forthcoming Alpha release of the new Kohana-based Swift River codebase on March 30th, Meedan is sponsoring a hack night and discussion on Friday, April 2, 2010 at 5pm, at the Meedan offices in San Francisco, California.

Proposed goals for the open-invite evening include:

• getting the new app running locally on some developer machines
• making sure that new developers are clear on how to use the git workflow
• discussing new features and the roadmap
• discussing integration potential, future points of collaboration & code sharing

We will probably transition to conversation over dinner.

WHERE?
There are two ways to participate in the hackup:

• In person. In San Francisco, at the new Meedan offices at 972 Mission St. See: http://www.inveneo.org/?q=mission_social
• Online. In the Swift River Public Skype channel

.

WHEN?
The alpha hackup is sandwiched between Where2.0 and Wherecamp — starting at 4/2 5pm PST and last until late, as the Ugandan and Kenyan teams come online (the morning of 4/3 Eastern Africa Time).

WHO SHOULD ATTEND?

• developers who are interested in getting started hacking on the new codebase
• developers with skills in PHP, MySQL, Python or Ruby on Rails
• our colleagues working on like-minded tools
• journalists or researchers who just want to learn more about the platform

RSVP
If you are planning on coming to the event in San Francisco just send an email to cgblow at gmail dot com — we will make sure you have a phone number so you can get in the Meedan office.

Hackathon Photo By SuperAmit

Each release of Swift River will carry the name of a style of African music. The release schedule appears below. I think it’s fitting to able to pay homage the music around me in this way and it actually serves as a starting point for people looking to be exposed new styles of music that they may not already know. The first version of Swift, an early Alpha, will be available on March 31st, 2010 and there will be regular updates and iterations to follow. If you’re interested in how Swift River verifies and filters the crowd, visit us here.

Alpha Releases
0.0.0 Rumba (Release Date: March 31st, 2010)
0.1.0 Apala
0.2.0 Batuque
0.3.0 Benga
0.4.0 Bikutsi
0.5.0 Cape Jazz
0.6.0 Chimurenga
0.7.0 Fuji
0.8.0 Harare
0.9.0 Jit

Beta Releases
1.0.0 Jùjú (August 1st, 2010)
1.1.0 Kizomba
1.2.0 Kuduro
1.3.0 Kwaito
1.4.0 Kwela
1.5.0 Makossa
1.6.0 Malouf
1.7.0 Maloya
1.8.0 Marrabenta
1.9.0 Museve
2.0.0 Mbalax

Non-Beta and Beyond
2.1.0 Mbaqanga
2.2.0 Mbube
2.3.0 Morna
2.4.0 Palm
2.5.0 Raï
2.6.0 Sakara
2.7.0 Sega
2.8.0 Soukous
2.9.0 Taarab
3.0.0 Zouk

Ouagadougou Drummer Photo by Babasteve

In the post Natural Language Processing with Swift River I introduced you to two underlying technologies powering Swiftriver. Victor Miclovich is the Ugandan volunteer developer who’s spent the last few months working to help make these plans reality with SiLCC (Swift Language Computation Core).

Victor Miclovich, SiLCC Developer and Volunteer

How did you get involved with natural language processing technologies? It’s not a field many Africans are known to be active in.

Victor Miclovich: When I got hooked up with writing code, I discovered another side of computing as a kid. That side of computing led me to doing heavy research work and this fired up my inquisitiveness.

NLP wasn’t what I played with first. I started with doing work inside of artificial intelligence which surprisingly had a likeliness to programming. As I matured in the area, I realized that one would never really master everything in A.I. (artificial intelligence) and so I narrowed my work to machine learning which was about 2 years ago.

Machine learning is a wide subject with lots of literature and research work being done in many areas from computer vision, speech recognition and natural language processing…the list is actually endless. I settled for computer vision work and NLP eventually because of their feasibility and ease of access to technology in Africa. I knew that getting a robot built could be a little bit hard! (laughs) That’s how I got involved with NLP technologies; my curiosity drove me to it.

What inspires you as a software developer?

VM: First, it is my drive and passion for technology. Being able to instruct a machine to do your bidding is something that brings a sense of fulfillment. People don’t always follow my instructions.

Secondly, the people (developers) I encounter wherever I work and go bring inspiration to me…this is just my way of saying that Appfricans are my inspiration…their accomplishments and determination is what keeps me going.

How do you see Africa’s role in tech changing over the next 20 years?

VM: Africa’s role in tech is slowly becoming visible. Universities in Africa are slowly churning up new grads every year. These grads have ambition and are tired of staying behind technology. This is what is going to drive the change in tech.

When students or people get tired of being behind, they develop a strong desire for change…we should not be pessimistic about this, we are optimistic! There are many floating examples all over Africa of tech communities and start-ups sprouting up.

You’re very involved in the community and helping the guys coming up behind you, giving gratis lectures and workshops at your university and mentoring your peers in your spare time. Why do you feel this is important?

VM: It is always important to give back to mankind. Philanthropy has it’s rewards. I feel that if I don’t do something, those are years lost to the community. I have lived in a place where I’ve seen folks with lots of potential and those that have made it in life and science (or tech). Many stay arrogant and don’t give back to the community…they end up living lavish lives with lots of wealth and of course, who else will suffer? The community will. It suffers because those well off folks only do things that will help themselves.

On the other side of things, my giving back to the community helps make more folks like me or even better than me. This means that we shall get thinkers rising exponentially and an increase of great ideas that won’t end up being recursively boring but wonderful! These are the main reasons I feel what I’m doing is important.

How has it been working with the global developer community? Have you learned a lot?

VM: Working with a global developer community has been very interesting. I’ve virtually met folks that have done cool stuff with their time and this has been quite inspiring. It has boosted the quality of the work that I do because of the huge amounts I learn from my peers in the global dev community.

You can follow Victor’s work on SiLCC here or on the Swift River mailing-list.

Sean Conner recently asked a great question about integrating a Question and Answer service like Aardvark or Yahoo Answers into Swiftriver. Here is our approach at Team Swift…

In a Swift instance, Aardvark could be used as an additional ‘channel’ of input. Existing channels are Twitter, Email, SMS, News, RSS (any RSS feed), and Other (the catch-all for items coming in via our API). The only thing the Swift app wants to do is receive content, allow users and our algorithms to tag that content, and based on user behavior it scores the originating content source.

As an example for Aardvark: Johhny asks the question “Did an earthquake really happen in Chile?” on Vark.com on Feb 28th, only a day after the quake actually occurs. Robert responds on Vark with “No, at least I haven’t heard of one.” Vark user Jeremy responds with “Actually, Yes. An 8.8 magnitude earthquake occurred in Chile on Feb 27th.” In Swift, the answer and the accuracy of that answer is more important to us than the actual question (which just provides context).

To integrate Aardvark in Swift we’d probably write a module using their API that aggregates Answers with the corresponding Question as the ‘description’. Example of how that data would post to the Swiftriver API:

Title: “Actually, Yes. An 8.8 magnitude earthquake occurred in Chile on Feb 27th.”
Description: “Did an earthquake really happen in Chile? – Johnny”
Time: 17:08 EST
Date: Feb 30, 2010
Source: Jeremy’s user id on Vark.com
Channel: Vark API
Lat: 10.31
Lon: 01.40
Tags: 8.8, earthquake, chile

Title: “No, at least I haven’t heard of one.”
Description: “Did an earthquake really happen in Chile? – Johnny”
Time: 03:10 EST
Date: Feb 30, 2010
Source: Robert’s user id on Vark.com
Channel: Vark API
Lat: 10.31
Lon: 01.40
Tags: heard, chile, earthquake

Within Swift this is the primary information we need to verify information. Users with a careful eye will notice that we’ve included location data that Vark probably may or may not provide. We can easily extract that info from the hosted service SULSa. Here, the source is what we’re scoring. The channel is just an indicator for the user about where the content is coming from. That said, the source is not Vark itself, nor is it the user’s answer on Vark, but rather the user id on Vark.

Thus, if Robert keeps giving inaccurate answers, he maintains a very low score in Swift while Jeremy is viewed as the more trusted authority. Now this approach assumes that Vark.com offers an API that allows for this type of data aggregation which I don’t think they currently do. Perhaps, it’s a question for the Vark team?