Skip to content


Swift River Global Hackathon | April 2

In anticipation of the forthcoming Alpha release of the new Kohana-based Swift River codebase on March 30th, Meedan is sponsoring a hack night and discussion on Friday, April 2, 2010 at 5pm, at the Meedan offices in San Francisco, California.

Proposed goals for the open-invite evening include:

  • getting the new app running locally on some developer machines
  • making sure that new developers are clear on how to use the git workflow
  • discussing new features and the roadmap
  • discussing integration potential, future points of collaboration & code sharing

We will probably transition to conversation over dinner.


WHERE?
There are two ways to participate in the hackup:

WHEN?
The alpha hackup is sandwiched between Where2.0 and Wherecamp — starting at 4/2 5pm PST and last until late, as the Ugandan and Kenyan teams come online (the morning of 4/3 Eastern Africa Time).

WHO SHOULD ATTEND?

  • developers who are interested in getting started hacking on the new codebase
  • developers with skills in PHP, MySQL, Python or Ruby on Rails
  • our colleagues working on like-minded tools
  • journalists or researchers who just want to learn more about the platform

RSVP
If you are planning on coming to the event in San Francisco just send an email to cgblow at gmail dot com — we will make sure you have a phone number so you can get in the Meedan office.

Hackathon Photo By SuperAmit

Posted in Community, Conferences. Tagged with , , , .

Variations on a Theme

In a past life, before developing software, I was a musician. The two have a lot in common actually: recursive pattern, rhythm, syntax, meter. I suppose most developers don’t think of code this way, but I do. It needs to look as good to humans as it does to machines. When I took over development of Swiftriver and I was looking for a theme to weave through all of our releases, it was natural to default to what I love: music.

Drummer in Ouagadougou

Each release of Swift River will carry the name of a style of African music. The release schedule appears below. I think it’s fitting to able to pay homage the music around me in this way and it actually serves as a starting point for people looking to be exposed new styles of music that they may not already know. The first version of Swift, an early Alpha, will be available on March 31st, 2010 and there will be regular updates and iterations to follow. If you’re interested in how Swift River verifies and filters the crowd, visit us here.

Alpha Releases
0.0.0 Rumba (Release Date: March 31st, 2010)
0.1.0 Apala
0.2.0 Batuque
0.3.0 Benga
0.4.0 Bikutsi
0.5.0 Cape Jazz
0.6.0 Chimurenga
0.7.0 Fuji
0.8.0 Harare
0.9.0 Jit

Beta Releases
1.0.0 Jùjú (August 1st, 2010)
1.1.0 Kizomba
1.2.0 Kuduro
1.3.0 Kwaito
1.4.0 Kwela
1.5.0 Makossa
1.6.0 Malouf
1.7.0 Maloya
1.8.0 Marrabenta
1.9.0 Museve
2.0.0 Mbalax

Non-Beta and Beyond
2.1.0 Mbaqanga
2.2.0 Mbube
2.3.0 Morna
2.4.0 Palm
2.5.0 Raï
2.6.0 Sakara
2.7.0 Sega
2.8.0 Soukous
2.9.0 Taarab
3.0.0 Zouk

Ouagadougou Drummer Photo by Babasteve

Posted in news, swift river. Tagged with , .

Mozilla Foundation Supports Ushahidi-Chile

A guest post from the team running Ushahidi’s Situation Room at Columbia University’s School of International and Public Affairs (SIPA).

Congratulations to everyone involved with Ushahidi-Chile at SIPA for receiving a $10,000 grant from the Mozilla foundation! Our project aligned perfectly with Mozilla Foundation’s mission to promote openness, innovation and participation on the Internet. The work of crisis mapping is more than innovative as it directly affects the most vulnerable people at Chile who are struggling after the earthquake. Thanks to this grant, our volunteers will be able to train local Chileans on how to use and manage the platform themselves.

We hope to use a small part of this grant to recognize and thank our wonderful volunteers at SIPA. Although there was an explosion of interest throughout SIPA to help with this initiative, not everyone was able to take significant time out of midterm exams to help. Our core volunteers are motivated and compassionate; the more they help the more they keep coming back. We could not thank them enough for their work, and we are happy that this grant can go in recognizing the hundreds of hours they have put into Ushahidi-Chile. Thank you volunteers!

The majority of this grant will be instrumental in moving this initiative to the next phase: to Chile. We have been communicating with various Chilean organizations to transition the Ushahidi platform to local Chileans. What’s great about the Ushahidi platform is that it is open source and versatile. The immediate need for the platform was to report incidents on the map, but the need is shifting to another direction. What better people to think of how to best utilize this platform than Chileans themselves.

Posted in Crisis, Deployment, Ushahidi Users, disaster. Tagged with , , .

Uganda’s Victor Miclovich talks Machine Learning

If you were there or following South by South West yesterday, you may have heard some chatter on Twitter about the Africa 3.0 talk by Teddy Ruge of Project Diaspora. In his panel he used Skype video to chat in real time with software developers and incubators in Cameroon, Kenya and with my staff in Uganda. Two of the developers from Appfrica, Moses Mugisha and Victor Miclovich appeared with me on camera to speak with the crowd. One of them, Victor, quickly discussed his natural language processing project SiLCC. Here’s a quick interview allowing him more time to explain his background, how he got into semantic programming and why peer learning is critical.

SXSW Africa 3.0 Panel


In the post Natural Language Processing with Swift River I introduced you to two underlying technologies powering Swiftriver. Victor Miclovich is the Ugandan volunteer developer who’s spent the last few months working to help make these plans reality with SiLCC (Swift Language Computation Core).

Victor Miclovich, SiLCC Developer and Volunteer

Victor Miclovich, SiLCC Developer and Volunteer

How did you get involved with natural language processing technologies? It’s not a field many Africans are known to be active in.

Victor Miclovich: When I got hooked up with writing code, I discovered another side of computing as a kid. That side of computing led me to doing heavy research work and this fired up my inquisitiveness.

NLP wasn’t what I played with first. I started with doing work inside of artificial intelligence which surprisingly had a likeliness to programming. As I matured in the area, I realized that one would never really master everything in A.I. (artificial intelligence) and so I narrowed my work to machine learning which was about 2 years ago.

Machine learning is a wide subject with lots of literature and research work being done in many areas from computer vision, speech recognition and natural language processing…the list is actually endless. I settled for computer vision work and NLP eventually because of their feasibility and ease of access to technology in Africa. I knew that getting a robot built could be a little bit hard! (laughs) That’s how I got involved with NLP technologies; my curiosity drove me to it.

What inspires you as a software developer?

VM: First, it is my drive and passion for technology. Being able to instruct a machine to do your bidding is something that brings a sense of fulfillment. People don’t always follow my instructions.

Secondly, the people (developers) I encounter wherever I work and go bring inspiration to me…this is just my way of saying that Appfricans are my inspiration…their accomplishments and determination is what keeps me going.

How do you see Africa’s role in tech changing over the next 20 years?

VM: Africa’s role in tech is slowly becoming visible. Universities in Africa are slowly churning up new grads every year. These grads have ambition and are tired of staying behind technology. This is what is going to drive the change in tech.

When students or people get tired of being behind, they develop a strong desire for change…we should not be pessimistic about this, we are optimistic! There are many floating examples all over Africa of tech communities and start-ups sprouting up.

You’re very involved in the community and helping the guys coming up behind you, giving gratis lectures and workshops at your university and mentoring your peers in your spare time. Why do you feel this is important?

VM: It is always important to give back to mankind. Philanthropy has it’s rewards. I feel that if I don’t do something, those are years lost to the community. I have lived in a place where I’ve seen folks with lots of potential and those that have made it in life and science (or tech). Many stay arrogant and don’t give back to the community…they end up living lavish lives with lots of wealth and of course, who else will suffer? The community will. It suffers because those well off folks only do things that will help themselves.

On the other side of things, my giving back to the community helps make more folks like me or even better than me. This means that we shall get thinkers rising exponentially and an increase of great ideas that won’t end up being recursively boring but wonderful! These are the main reasons I feel what I’m doing is important.

How has it been working with the global developer community? Have you learned a lot?

VM: Working with a global developer community has been very interesting. I’ve virtually met folks that have done cool stuff with their time and this has been quite inspiring. It has boosted the quality of the work that I do because of the huge amounts I learn from my peers in the global dev community.

You can follow Victor’s work on SiLCC here or on the Swift River mailing-list.

Posted in swift river. Tagged with , , , , , .

Election Monitors and the Unwashed Crowd

Polling Station

I’ve been told that crowdsourcing of elections isn’t a wise move. After all, what value will anyone gain from gathering a bunch of yammering “l33t-speak” texting reports from the unwashed masses? Election monitoring should only be done by trained volunteers and their results analyzed by professionals.

Last week I spent a couple days in sessions on election monitoring, with election monitoring professionals, some who have been doing this for many years. I’ll be the first to say that I didn’t know a lot about how they worked, so I listened very closely to understand just what was going on. I was surprised, very surprised, by what I heard.

As most people, I think of election monitors as those guys with a funny hat/shirt that stand to the side and watch and scribble things on a sheet of paper. That’s true, that is what they do.

However, I was surprised to learn that most of these election monitors are not well trained, give mediocre data, and some end up not just objective bystanders, but proactive enforcers of a given political party. In effect they aren’t much, if any, better than those aforementioned “unwashed masses”.

Exacerbating the issue is the fact that these election monitoring groups rarely keep good records of their monitors. Thus, when fraudulent or questionable activity comes in from a polling station, where compromised monitors are working, they aren’t tracked. In the next elections, these same people step forward and are then used again.

Finally, it doesn’t make sense to equate election monitoring with voting booth monitoring. What about everything that happens before and after elections? If a government is going to skew the elections, they’ll make all the preparations beforehand and don’t generally do much on election day; they wind up the clock and just let it tick they way they built it. By only monitoring voting stations one is basically making the case that what happens weeks/days/hours before the actual voting process is completely independent and has no influence on the elections.

Absurd. Elections are a process, not an event.

Election simulation flow

Interesting, and insane…

In the collection of data then (not the analyzing of it), why are election monitors so much more valuable than crowdsourcing from the public?

First off, there is something to be said about having someone sit at the polling station all day long. It could be argued that people just coming in to vote might see something, but someone there all day is much more likely to catch an inconsistency or fraud. (Though, it should be noted that most African voters defend their vote by also staying all day long until the local ballots are counted and announced.)

Second, election monitors are “known” and can therefore be found if a report was given that needs to be followed up on.

In a more sane and perfect world, we can see from these two points that election monitors would be useful. However, we don’t, and yet we keep seeing money get poured into this broken system. Acting as if election monitoring is some sacrosanct fortress, the only right way to make sure elections happen the right way.

A more sane person might suggest that you look at a lot of incoming data from many sources and use it all. Sure, keep some proven election monitoring practices in place, but balance that against the crowd who can provide you with alerts that might never have cropped up on your radar before. Marry that with budget tracking and cleaned census data.

I’m taking a sarcastic tone in this post, as it seems a bit beyond me that the crowdsourcing method gets pilloried when compared to election monitoring. Both are part of a greater ecosystem of data collection that should be happening. One isn’t better than the other overall, and both bring different strengths to the table. In effect, election monitoring is just pseudo-controlled crowdsourcing.

It’s time to start thinking of elections holistically, as a process and with many forms of data input and analysis.

Posted in crowdsourcing, elections. Tagged with , , , , .

Asking Questions, Verifying Answers

vark.com

Sean Conner recently asked a great question about integrating a Question and Answer service like Aardvark or Yahoo Answers into Swiftriver. Here is our approach at Team Swift

In a Swift instance, Aardvark could be used as an additional ‘channel’ of input. Existing channels are Twitter, Email, SMS, News, RSS (any RSS feed), and Other (the catch-all for items coming in via our API). The only thing the Swift app wants to do is receive content, allow users and our algorithms to tag that content, and based on user behavior it scores the originating content source.

As an example for Aardvark: Johhny asks the question “Did an earthquake really happen in Chile?” on Vark.com on Feb 28th, only a day after the quake actually occurs. Robert responds on Vark with “No, at least I haven’t heard of one.” Vark user Jeremy responds with “Actually, Yes. An 8.8 magnitude earthquake occurred in Chile on Feb 27th.” In Swift, the answer and the accuracy of that answer is more important to us than the actual question (which just provides context).

To integrate Aardvark in Swift we’d probably write a module using their API that aggregates Answers with the corresponding Question as the ‘description’. Example of how that data would post to the Swiftriver API:

Title: “Actually, Yes. An 8.8 magnitude earthquake occurred in Chile on Feb 27th.”
Description: “Did an earthquake really happen in Chile? – Johnny”
Time: 17:08 EST
Date: Feb 30, 2010
Source: Jeremy’s user id on Vark.com
Channel: Vark API
Lat: 10.31
Lon: 01.40
Tags: 8.8, earthquake, chile

Title: “No, at least I haven’t heard of one.”
Description: “Did an earthquake really happen in Chile? – Johnny”
Time: 03:10 EST
Date: Feb 30, 2010
Source: Robert’s user id on Vark.com
Channel: Vark API
Lat: 10.31
Lon: 01.40
Tags: heard, chile, earthquake

Within Swift this is the primary information we need to verify information. Users with a careful eye will notice that we’ve included location data that Vark probably may or may not provide. We can easily extract that info from the hosted service SULSa. Here, the source is what we’re scoring. The channel is just an indicator for the user about where the content is coming from. That said, the source is not Vark itself, nor is it the user’s answer on Vark, but rather the user id on Vark.

Thus, if Robert keeps giving inaccurate answers, he maintains a very low score in Swift while Jeremy is viewed as the more trusted authority. Now this approach assumes that Vark.com offers an API that allows for this type of data aggregation which I don’t think they currently do. Perhaps, it’s a question for the Vark team?

Posted in swift river. Tagged with , , , , , .

Ushahidi-Chile: Reflections after Week One

Caroline Stauffer is a member of the core SIPA Team deploying the Ushahidi-Chile platform. She is a graduate student at Columbia University’s School of International and Public Affairs (SIPA) where she focuses on International Media and Communications. She spent the past summer working with the Associated Press in Bangkok, and worked for a nongovernmental organization in the Dominican Republic prior to SIPA.

Continuous aftershocks, office buildings scheduled for destruction, villages without access to water, gasoline shortages, and looting. These are some of the 800+ incidents my fellow students and I mapped during our first week in the situation room at Columbia University’s School of International and Public Affairs (SIPA). The death toll from the February 27 earthquake is much lower than the number of lives lost since January 12 in Haiti, but the 8.8-magnitude quake in southern Chile was one of the largest on record, and coordinating information, needs and responses on the ground is essential.

Near real-time crisis mapping

Near real-time crisis mapping

An initial training session at Columbia University the Monday after the earthquake drew 60 students—during SIPA’s midterm examination period. A core team of six students is working to ensure that the growing list of volunteer names translates into more hours spent monitoring and mapping. Other schools within Columbia University are starting to get involved, and the team has reached out to contacts at New York University. One of our core members took a quick break from the Situation Room to head to United Nations headquarters after individuals from the United Nations Development Program (UNDP) asked for training on the Ushahidi-Chile application. Most importantly, as students continue to monitor traditional and social media, more reports are reaching the Ushahidi platform from on the ground in Chile.

The core SIPA Team

The core SIPA Team

Some students were initially skeptical of Ushahidi-Chile. The Chilean government had said international aid was not necessary, and the value of mapping information from news sources and Twitter—in short, information that can be found elsewhere—was questioned. However, by Monday, messages communicating the needs and locations of people who lacked basic supplies and water were coming in. It seemed that our team had uploaded enough information to be truly useful on the ground. Soon, Chilean organizations including Red Salvavidas and Chile Ayuda were contributing reports daily.

As a journalist, the flow of information within Ushahidi has been immensely interesting to me. The process that our teams of monitors, mappers and administrators go through to produce the Ushahidi page is not unlike the path a journalist takes to report and produce a story. In both cases, the reporter seeks information from overlooked sources and tries to verify the data. Small bits of information combine to produce a larger picture, giving an overview of a particular situation. In Ushahidi, the picture that emerges is an interactive map, rather than a narrated story.

Crisis Mapping volunteers at SIPA

Crisis Mapping volunteers at SIPA

The SIPA team has touched base with technology students from Talca, in one of the hardest regions hit in Chile. The Chilean students now have administrative abilities on the Ushahidi-Chile page. The goal of the SIPA students working with Chile-Ushahidi is to transfer the platform to an organization in Chile by the end of March, but to leave behind a trained group of crisis mappers at SIPA who will be ready to assemble and share information whenever and wherever the next disaster strikes.

Posted in Community, Deployment, disaster. Tagged with , , , .

Natural Language Processing with Swift River

One of the core features of Swift River is the Language Computation Core, or SiLCC as we like to call it (Swift Language Computation Component). Users send feeds to SiLCC which, using a number of machine learning techniques, parses the incoming text and extracts relevant keywords. The idea is that these keywords (tags) can then be used to infer taxonomic relationships between content items. Some camps refer to this as semantic programming, others refer to it as artificial intelligence, but the general concept remains the same: helping programs to perform tasks based on a growing series of complex conditions. In this case ‘auto-tagging’ or ‘predictive tagging’ based on conditions learned from user behavior and preset rules.

The diagrams below illustrate how this dataflow works. Text passing through SiLCC are parsed, tags are extracted, those tags are then reapplied in the Swift River UI. There, Swift attempts to build relationships between tags. (ex. items tagged with “chile” and “earthquake” are likely related. However items tagged ‘chili’ and ‘earthquake’ likely are not.) Of course other factors are considered like date, time, the point of origin and location of the content creator.

One of the services running within SiLCC is another service called SLISa, which we like to call Lisa (because the ’s’ is silent, hehe). SLISa is the Swift Language Improvement Service App and it trains SiLCC to learn from user interaction. When users of Swift edit or flag tags as inaccurate, SLISa is the service that creates all the conditions that helps SiLCC to learn from it’s mistakes and improve for the future.

SiLCC is an open source project being developed in Python using the pyNLP toolkit. There’s several additional layers of text parsing that I haven’t touched upon including how SiLCC deals with SMS txtspk and Twitter picoformats like hashtags but more on that in a future post!

More on SiLCC at http://swift.ushahidi.com/extend/silcc/. If you have a passion for machine learning, large data sets, and intricate algorithms you might also consider joining the Swift River Google Group or our public Skype Chat.

The Alpha release of Swift River, Version 0.0.9 Rumba will be available to the public on March 31, 2010. Developers can find always find the latest working build and issue tracker at http://github.com/ushahidi/Swiftriver.

Posted in swift river. Tagged with , , , .

Training the Ushahidi-Chile Team in a Flash

Mark Belinsky co-founded Digital Democracy with Emily Jacobi. He serves as Technical Director and brings a background in computer science, sociology and film & media studies. He and Emily have worked closely with Burmese populations since 2007.

Working in the tech sphere, it’s the power and passion that people have that never ceases to astound me. Following the Jan. 12 earthquake in Haiti, people around the world contributed an incredible outpouring of support for the people of Haiti. Now, as that support expands to Chile, it is evident that we are participating in a game changing moment.

On January 12th when the earthquake in Haiti struck, Digital Democracy had two team members on the ground looking at the economic livlihoods of young Haitians . Worried for their safety and the greater loss of life, my colleague Emily and I immediately joined Patrick in the Situation Room at Fletcher, and collaborated with the core team to establish a system that allowed hundreds of volunteers to glean emergency instances from the ground and place them on the map for response. Those first volunteers have trained many more, and their tireless work has directly saved lives.

When an 8.8 magnitude earthquake hit Chile on Saturday, we were called on again to train volunteers. The Fletcher Ushahidi team, their hands full with Haiti, was able to rapidly set up the site for Chile.Ushahidi.com, but needed help training new volunteers. Digital Democracy collaborated to adapt the training module for the Situation Room at the Fletcher School at Tufts University, and on Sunday night we conducted an hour+ training on Skype for a core team at SIPA at Columbia in New York City who are now running the Chile operations. Their ability to quickly respond to the devastating emergency in Chile, despite it being midterms week, speaks to the dedication and passion of the volunteers. The 10 people we trained on Sunday trained 40 more the next day at SIPA, plus a growing number of volunteers in Santiago.

For us, this has been a particularly rewarding process, as we’ve been following and working with Ushahidi since fall 2008. Watching Ushahidi evolve from mapping post-election violence in Kenya to elections reporting when we helped with VoteReportIndia, we’ve worked to harness Ushahidi to empower ordinary citizens. At Digital Democracy, we’ve applied Ushahidi to map human rights abuses in Burma through Handheld Human Rights and peace mapping in Kenya through Sisi Ni Amani – “We are Peace.” In each case, we’ve worked with local community organizations to determine needs and apply the technology – Ushahidi – to best meet their needs.

Our mission is to utilize technology for civic engagement, and these examples demonstrate how technology can encourage deeper engagement with the world around us. At the core, it’s not the technology but the people who use it. Ushahidi is a tool that’s a key part of our arsenal because of the open community around it. As technology enables more and more people to contribute to meaningful actions to save lives, I for one am excited to see where it leads.

Posted in Community, Crisis, Deployment, Ushahidi Users, disaster. Tagged with , , , , .

The Need for a Tech Election Monitoring Toolbox

This week in Nairobi has been “Election Monitoring” week due to the NDI/DFID meeting on Tue/Wed and the HIVOS meeting on Thur/Fri. Interestingly enough, both meetings heavily addressed the uses, or lack thereof, of technology in the election monitoring process.

One of the ideas that hit me was the need for a toolbox of technical tools that could be used by election monitoring groups and citizens both before, during and after the elections.

Understanding the Framework

Most of us think of an election as an event, I did too. Koki Muli provided us a with a great framework to understand the election process as a whole, using this visualization for everyone to see that it is indeed a long-term process, not an event.

The Election Cycle

The Election Cycle

Throughout the days of talk, a lot of information was given about elections, and specifically election monitoring as a practice and how it actually goes down in the real world. I’m not an expert in election monitoring, so there was a lot of education for me, but putting on my technology hat I (and the other tech guys present) were able to come up with some ideas for tactical-level solutions that might be useful to these election monitoring groups.

Starting with the Issues

I don’t have time to run through all of the issues, but here are a couple that have a technology component that could be created to help with them.

Legal Framework
The legal framework for elections starts many years before an election happens, but might be the most important element in an election (in a law-abiding state). These are high-brow activities, where law professors and legal experts write in a format that isn’t easily understood by ordinary people.

The question then is, how could this legalese be translated into a format that is digestable by normal citizens, and how could they then give feedback on what they think of these new laws? I’m already thinking of a tool that can be built for this, having a lot in common with (oddly enough) digital Bibles and how they allow multiple levels of commentary by readers and scholars.

Election Monitors
Registration of election monitors is not rocket science. Many times, the election monitoring groups get funds at the last minute and they go out and recruit anyone above a certain education level within the different constituencies. There is little vetting, some training, and no understanding of their values and ethics for when election day actually arrives. It’s no surprise then that when election day comes, many of these election monitors turn out to be little more than party infiltrators, sometimes abusing the system worse than normal citizens.

What tool can we build to support creation of a database of known election monitors, with full information about them, so that when irregularities or criminal activities are done by or with them that it can be tracked. Having a historical database of election monitors will allow for pattern mapping and even a blacklist for the civil society groups when they go out to find monitors for the next elections.

Real-time Data Collection
During the campaigning and the election day itself, it’s possible to collect information on irregularities from both trusted/known sources (including media, government and civil society), but also from ordinary citizens in the Ushahidi-style of crowdsourcing from citizenry. This includes media monitoring as well as gathering of historical and demographic data so that real-time analysis can happen.

Tools like Ushahidi and Managing News can be utilized here, this is what they are built for. The different types of news sources are complimentary, allowing layering of new, real-time data on top of demographic and historical election data.

Onto the Toolbox

These issues and challenges throughout the election cycle cannot be set up and deployed in the last few weeks or months if they are to be effective. We all need to sit down now, come up with the tactical and strategic-level tools that are needed and deploy them within the organizations who can best utilize them in the election process.

  • What tools can be built?
  • What tools exist already?
  • How can we create a toolbox that election monitoring groups, and citizens, can choose tech tools from and deploy?
  • Has this already been started somewhere else?

Add your thoughts and ideas in the comments below.

Let’s see if we can come up with some ideas, as technologists especially, of tools that can work in conjunction with one another to strengthen the election monitoring and the election cycle as a whole.

Reinier Battenberg had a good slideshow with examples of where tools are needed at different parts of the election cycle. I’m including that here as well as food for discussion. (click the image to download his Powerpoint presentation)

10 applications for use in election monitoring

10 applications for use in election monitoring

Finally, here’s a list of election monitoring-specific tools we could think of, what would you add to it?

Posted in Conferences, Kenya, Nairobi, Strategy, Ushahidi, elections. Tagged with , , , , , , , .