Micz Flor of Sourcefabric

This is a guest blogpost from our developer community, written by Micz Flor.
Micz is a media developer, writer and project manager based in Berlin. Since 1995 he has been knee-deep in Internet, organising a variety of events, net projects, magazines and temporary media labs. In 2000 Micz began work as a media developer for Eastern Europe and Asia at the Media Development Loan Fund (MDLF), where he met his fellow Sourcefabric co-founders.

My introduction into developing for Ushahidi was accidental. While I was developing templates explaining the Newscoop CMS mapping feature, I was looking for a valid KML-feed. Our community manager Jakub Górnicki sent me an XML-file from his recent Ushahidi crisis map covering “Heavy Snow in Poland”. He had used this as a layer on the map. The same day, I started abusing Ushahidi layers as a KML-feed validator (I know, that’s not what it was developed for…). Once we put two and two together, the Ushahidi Cooker was born. Ushahidi layers is a really great feature which (in my opinion) is generally underused.

What does the Ushahidi Cooker do?

The Ushahidi Cooker provides a directory of maps and KML-feeds. Or to be more precise: a tool to easily create and manage libraries of maps. These maps are stored by categories and cities, like “Hospitals in Berlin” or “Airports in Indonesia” – anything you like. These maps come with a valid KML feed. And this feed can be called into a Ushahidi map as a layer, providing additional content on the map for the users of your site. Below you can see a video illustrating the process of creating an Ushahidi layer from Newscoop’s Ushahidi Cooker as well as a link to a demo site where you can try the KML-feeds.

Why is this useful?

1.
Using Ushahidi Cooker means that your map is really your map. It is not a Google Map, or in other words: Google’s map. Similar to Ushahidi, Newscoop provides different ways to render your maps. You don’t have to lock your data into a map provider.

2.
With the Ushahidi Cooker you can prepare maps in the background – and call them into different Ushahidi installations when needed. “Bridges in Iceland” could be of interest on a tourist map as well as a crisis map after a volcano eruption. With Newscoop, you can make a copy of any map you have in stock and then make alterations, give it a new name and use it in a new context.

3.
Newscoop is a fully fledged and open CMS for professional journalism. Tying Newscoop into your Ushahidi map gives you the tools in the background to manage articles and other content around the subject of your map.

Try it out and help improve the Ushahidi Cooker

I would be interested to see this tested in a real life situation. Try it out and tell us what you think. If you need it for a crisis situation, we can also help you out with support. If you are interested, contact our community manager Jakub Górnicki (jakub.gornicki at sourcefabric dot org).

Having been “in the field” as a media developer myself, I know that tools which seem to work at first sight only really work once you broke them a few times – and improve them on the way. The Ushahidi Cooker and the Newscoop CMS are free and open source.

The video tutorials show the Ushahidi Cooker in action. Covering the entire process, the installation in Newscoop, creating maps and adding layers in Ushahidi, the total running time is roughly 17 minutes, so it is really very straight forward.

All Video Tutorials:

• Creating Ushahidi layers with Newscoop’s Ushahidi Cooker

(3:23)

• Creating maps with the Ushahidi Cooker template pack (6:56)
• Installing the Ushahidi Cooker template pack in Newscoop (6:20)

Ushahidi Cooker test install
http://ushahidicooker.templates.sourcefabric.org
More information is available on the help page:

Community / Newscoop support
http://forum.sourcefabric.org

Downloads
Download Newscoop
http://www.sourcefabric.org/en/products/newscoop_download/

Download the Ushahidi Cooker for Newscoop
http://www.sourcefabric.org/en/products/newscoop_templates/

(Example 1 | Example 2)

This has obvious benefits:

• You can define an area
• You can define roads or other features not available on the base map
• You can define one report with multiple points and/or multiple geometries

This feature also allows you to do cool things like write plugins that import GPS data. One is already available – GPXer (apps.ushahidi.com)

To illustrate, I’ve imported a recent bike ride into using this plugin which calculates distances, start-times and end-times for you.

A few things to note:

• The report will also be saved with a single point location. The singular point is derived from the centroid of the multiple geometries. This is just to make it easy to cluster, map and find multiple reports. Also this helps with pulling reports via the API into mobile devices.
• This feature is only currently available in the Admin interface.
• The geometry feature is still work in progress. Your feedback will help us make it more robust.

You can download a build with this feature at: https://github.com/ushahidi/Ushahidi_Web

Highlights of Ushahidi 2.0

1. Plugins

The plugin system is something we’ve been working on for many months now, and we talked about it briefly back in July. This system allows us to do two things; First, it allows anyone to extend the capabilities of the platform. Second it allows us to focus on the core application itself. For the past few months, our community has been building plugins that you will soon be able to take advantage of via an Ushahidi App Market. In the meantime here are some resources to let you take advantage of this new functionality:

• Featured Plugins
• Plugin API Documentation (Still in the works)
• App Repository

2. New and Improved API

Henry and Emmanuel have been working hard on the Ushahidi API which exposes the platform to 3rd party applications that you create. This API allows you to post or retrieve information from Ushahidi deployments. This in fact is how our Mobile applications (iPhone, Android, j2me) communicate with the platform. Our goal is to give you as many ways as possible to consume the data that Ushahidi deployments create. More information about the Plugin API can be found here. We hope you find new and interesting ways to use it. Soon, you’ll be able to administer Ushahidi deployments through 3rd party applications via an Admin API.

3. One-Click Upgrades (Beta)

Ushahidi deployments have been notoriously difficult to upgrade. You can now exhale, because we have been working hard at a one-click upgrade mechanism, so that you can upgrade to the latest version without losing some hair. We’re still working out some kinks though, but its closer to 100% done. The system will alert you to new version availability and give you the option to perform the one-click upgrade. Please note that the one-click upgrade will only be available for users of version 2.0. If you have an older version of platform, you will need to update your deployment to version 2.0 manually. Instructions on how to do this are on this wiki entry. Once updated, you will be able to use the one-click upgrade mechanism for future versions.

4. Improved Localization

With the platform’s continued use around the world in countries that speak anything other than English, it is increasingly important to quickly and easily localize the language used on the deployment. We now have a location on the web (Thanks to Brian) where you can help localize language packs for use on your deployment of the platform. Already, many languages have been added and the list of available languages continues to grow. Head over to Tafsiri (Swahili for translate) to get started.

5. SMS Providers

Version 2.0 of the platform now allows you to add other SMS providers into the system via the Plugin API. The platform ships with FrontlineSMS, Clickatell and now SMSSync. SMS providers either provide a mechanism for filtering text messages into the system and/or sending messages from the system.

6. Trusted Reporter Functionality

Popular deployments are sometimes inundated by reports and it becomes increasingly difficult for administrators to read through and approve incoming messages. With Ushahidi 2.0, it is now possible to tag certain phone numbers, email accounts or twitter users as ‘Trusted Reporters’ and their reports will be automatically approved by the system.

7. Improved Theming

With previous versions of the platform, theming was restricted to what you could achieve with CSS styles alone. In Ushahidi 2.0, you can now create new templates and change the structure of pages, allowing you to design completely new looks. The additional benefit is that you never have to tamper with the core code like you had to before.

8. Improved Reports Listing

Caleb has also been hard at work on the reports listing page. We’re trying to make this page as functional as possible and will continue to do so, so that we can convey different sets of information quickly and in a practical way.

9. Improved Reports Detail
The reports detail page has also been updated. You can go for wider or taller maps. You can also switch tabs to view images and other attached media.

10. Improved Scheduler

Certain tasks with the platform are executed at specified intervals like checking email, checking twitter, sending alerts etc. The improved scheduler allows you to set the frequency of such events.

11. Improved Caching and Speed

Moving forward we’ll begin to add some options for administrators to help optimize the loading of the platform. This is especially critical for high traffic deployments.

There are numerous other fixes and improvements and we hope you enjoy the hard work we’ve put in so far. As always we can’t ever get enough feedback so we’ll be waiting to hear from you!

We’ll also be going into detail in the next few blog posts about how to use some of the features I’ve mentioned above.

So what are you waiting for? Come and get it!

All this has been made possible through Cloudvox. Cloudvox is one of the first (if not the first) providers of affordable Asterisk hosting in the cloud. Asterisk is free open source software that turns an ordinary computer into a voice communications server. We’ve had considerable interest in using Asterisk to stream calls into an Ushahidi deployment, but the barriers to entry have been high – especially since we want to make it easy for you to deploy it too. What intrigued me the most about Cloudvox was that it was instant, affordable and most importantly had a good API that we could tap into, to access all the goodness of Asterisk. The icing on the cake was the fact that they make it easy to rent a phone number.

So the next question of course is how this was integrated into Ushahidi. This was achieved via a plugin. We’ll soon be releasing v2.0 of the Ushahidi platform, which includes a new plugin architecture (You can watch the screencast here). Access to the beta version of v2.0 is publicly available too. I wrote a plugin for Ushahidi that integrates cloudvox and allows you to voice enable your Ushahidi deployment in minutes. Here are some of the features:

• People can call a predefined phone number, which then prompts them for two things: a location and the report.
• The voice prompts can be edited/created within your Ushahidi admin. These ‘text’ prompts are what the caller hears when they call via text-to-speech.
• If the robotic text-to-speech prompts don’t sound good, you have the option to upload pre-recorded voice prompts.
• The recorded location and report sound files are saved separately within your Ushahidi deployment and available for an administrator to listen to when they click on the messages tab.
• These calls can then be used to create new reports which include the sound files for playback.

To see (or hear) this plugin in action, please go to http://calldemo.ushahididev.com/ or dial +1(678) 809-2225 . You’ll notice a new item in the ‘How To Report’ box on the right:

You can also view a example of a voice report by clicking here. To better understand how all this works, Caleb gives us a video walkthrough:

With regard to Cloudvox, they charge about 3 cents (US) per minute and include 100 minutes free on sign up. Billing is done on pay-as-you-go. To rent a phone number, it costs $3/month for US/Canadian phone numbers and $10/month for phone numbers in other countries.

Resources:

• Ushahidi Cloudvox Plugin
• Ushahidi 2.0 Beta
• Cloudvox
• Asterisk

• KML/KMZ Overlays
• Statistics
• Theming and more

This installation would also be a true test of how much data the Ushahidi platform could really handle. The Atlanta metro area had about 30,000 crimes reported in 2009 alone, and the question was if we could visualize this data using our platform in a useful way. Here are some of the lessons I took away from this particular project:

Mapping: Clusters

Traditionally we’ve passed the task of clustering the map data on to the users browser via javascript, however this is only useful if you have a thousand or less map points. Even then you can feel the browser groan under the pressure of calculating what point is close to what then placing it on the map. Immediately it is evident that clustering map data points using javascript is not possible and would have to be done on the server using what is known as ‘Distance Based Clustering’, which is basically grouping reports based on their proximity to each other.

There are various methods of doing this which include using the Haversine Formula, the Euclidean distance Formula, or the Manhattan Distance (also known as the Taxicab Geometry). I opted for the latter – which is not very accurate, but creates clusters at the least cost to the server. Since clusters represent a general area, absolute accuracy is not required. I also have to give credit at this point to Mika Tuupola for his good pointers derived from his experience with map clustering.

Mapping: Layers

Adding other layers to the map allows you to combine and visualize multiple data sets at the same time, which can lead to interesting revelations. In this installation, I included the Atlanta PD zones, and Sex Offenders. I’ll probably be adding more layers as I gain access to the data. While working on project I came across a Batch Geocoder that is incredibly useful for making datasets mappable.

Grouping Items

One functionality that we may have forgotten to mention regarding the Mogadishu Release, is that it gives you the ability to group related categories as illustrated above – something that was quite handy in this installation to group different types of crime.

Proximity

Finally, I realized it might be important to list reports happening near a specific address and order them by distance from the point.

Please click below to view the work in progress:

• New CSS based design that is easier to skin!
• 5 minute installer
• Automatic version detection and upgrade
• Themes
• Site Statistics and Analytics
• Admin email notifications on emails and comments
• Add custom pages and tabs
• Auto-Geotagged news feeds
• Convert news feeds to reports
• Layer KML/KMZ files on the map
• Numerous bug fixes

KML/KMZ Data Overlays
You’ll note that you can now do multiple KML/KMZ file overlays onto a map with pre-existing Ushahidi data. The above map shows our demo Mogadishu site with a KML file of Somali pirate attacks.

Statistics
If you’re running an Ushahidi instance, just jump into the new “stats” tab and you’ll be able to see the number of unique visitors, report statistics, country breakdown and category impact.

Custom designs and theming
There’s a new default look/feel for Ushahidi. We redid it from the ground up so that it would be a lot easier for anyone who wanted to customize the look and feel of the site could do so just by working with a CSS file. There are also some basic color themes included in the backend so that those who aren’t CSS-savvy can still brighten it up.

An easier installer
One of the Achilles heels of any open source software application is the horrible experience installing it. We’ve taken our first step in making it even easier to do this on your own server. You have a choice of taking two paths when you begin, a basic one and a more detailed one. Whichever you choose, you can still fine-tune the details later once the site is up and going.

Find a bug? Submit it to bugs.ushahidi.com
Have a question? Ask it on the forums
Want to pitch in? Start on the wiki or download the code

Major contributors

Projects like Ushahidi wouldn’t be where they are without the help of programmers who volunteer their time, brains and energy to solving some very difficult challenges. A BIG thanks goes out to all the devs who made this happen. Major contributors included:

• Jason Mule (Kenya)
• Brian Muita (Kenya)
• Max Froumentein (Norway)
• Soyapi Mumba (Malawi)
• Emmanuel Kala (Kenya)
• Henry Addo (Ghana)
• Dale Zak (Canada)
• Pablo Destefanis

Ushahidi Meetup 09 C from Ushahidi on Vimeo.

Going through the code contribution statistics a little more than a year later proves that we’re actually achieving the equilibrium and momentum necessary to produce useful open source software. The following graphs illustrate contributions (lines of code) and impact by multiple programmers at different phases of the project.

August 2008 to October 2008

November 2009

Code Commits By Time of Day (Current)

The graph above illustrates the fact that Ushahidi’s developers are actively churning out code at practically all hours of of the day, with a concentration between the hours of 8am and 1pm – which is the time between which North American based developers and African/European developers are concurrently making code commits.

I am currently at the Non Profit Software Development Summit in Oakland California interacting with other non-profit open source development projects which include the very established ones like Drupal, Joomla and Plone, and its interesting to see that our development processes have been very similar.

It has been a year since the presidential election that sparked off the crisis in Kenya. The world has moved on to an economic depression, a black president of Kenyan heritage in the United States, and deja vu in the middle east. Everyone (Almost Everyone) looks to 2009 with optimism and expectation. Meanwhile the fate of the displaced people in Kenya is suspended in time. Nothing has really changed except the name; ‘Displaced Person’ has been replaced by ‘Squatter’. The rest of the Kenyans have had a range of mixed emotions in 2008; from shock and horror to outrage and disgust then short-lived exuberance as Obama took office. The crisis itself has been temporarily fixed, but remains unresolved.

I was in Somalia a few months before it disintegrated and I’m surprised at how indifferent I was watching the events unfold on CNN. I however, was very sad to hear some families we knew hadn’t made it out. Still there was some kind of disconnect – because that wasn’t my country, those weren’t my people and most importantly because Kenya was peaceful. Only ‘other’ people went through such tribulations. It is difficult to describe the numbness and helplessness I felt as I watched my country and my people go through a similar incident. This time, I could feel knives being driven into my heart as it all played out on CNN. To everyone else, it was just news – my indifference had come home to roost.

Ushahidi was born out of calamity, to reveal to the rest of us the real cost of crisis to the human race – an attack on our defeatist, apathetic human nature and proof that your 2 cents really can make a difference. We are all bound to each other by invisible strands and our fates invariably intertwined. 2008 has been a year of learning and unlearning. Crisis, whether political, environmental or otherwise is a part of human nature and indeed an intrinsic component of a planet with elements struggling to survive. Ultimately, who we really are is determined by how we deal with a crisis, and how we deal with a crisis is determined by how prepared we are.

Photo Courtesy of Teseum

The most important part of the site to have translated first is the front-end (the part that everyone sees), the admin area is also important, but not nearly as urgent. We couldn’t just go in there and change the words out, what we needed was a programmatic way to do this so that any other future translations in any other language could be done fairly simply (and much faster).

The first step was to pull all the English text content from the site and put it in a flat file document to be translated into French. Kaushal and Ory worked particularly hard finding people to do that translation and then to create a process, using Google spreadsheet docs, so that the next time we have to do it the transition will go a lot smoother. Once those text fields were all finalized we simply funneled that back into the platform and made an option to access it by clicking on a French flag in the site’s header.

For the administrators

It’s important that the administrators are able to translate incoming English reports into French (or other languages in the future). When on the “reports” page of the administration side, you’ll see this new button:

Once clicked, that button takes you to the following page where you can select the language you would like to translate the report into, and begin translating each section:

Challenges

We’re about 80% of the way done. One area that has proved to be particularly challenging is categories. Those are set by the administrator. This means it’s dynamic, not static like the rest of the content on the front-end. We’re working on a way to for administrators to add their categories in multiple languages.

Some languages take more characters (or words) to say something than others. We’re trying to figure out how to make sure that type of change doesn’t break our formatting completely.

Lastly, we’re trying to decide where and when the right time is to use automated computer translation of content. Requiring administrators to translate and retype reports, news headlines and local help organizations is neither efficient or time-saving. We’re interested in figuring out a way to programmatically help administrators manage this.

Thoughts on ways we can do this better?

• Creating new crisis reports tagged by location
• Adding images, video and news links to reports also tagged by location
• Visualizing reports using an interactive timeline and heatmaps
• Allowing users to participate by adding information to reported events
• Credibility/Ranking of reports
• Subscribe to alerts via email or sms that alert you to reports within the proximity of a specific location

 

Mobile-based reporting will soon be introduced into the beta version of the platform along with several other useful features that we could not squeeze into the alpha release timeframe. There’s still plenty to do before the Ushahidi engine is ready for general release  but everything is going well and we’re extremely excited about it – so stay tuned!

In the demo, linked above, you will find data processed from the original Kenya install of Ushahidi. The original version of ushahidi is found at Legacy.Ushahidi.com.

The admin area looks a little different, below is a look at the dashboard. We’ll be opening up a full demo area for anyone to play with the backend of the system soon.

If you’d like to get involved, head on over to the Ushahidi dev wiki, then get in contact with us.