The summation of the talk is that the Swift project has been assigned a very complex and incredibly difficult task: to verify and contextualize data from the mobile and social web. How do we do this? This seems to be the part that confuses people. It’s not any of our apps, and it’s not any of our individual APIs that we rely upon to do this. It’s the combination of all these things into one robust algorithm that tries to digitally reconstruct the real-world context, using the features extracted from the content to prioritize and de-prioritize information relevant to that context.

I like to refer to this as folksonomic triage where layers of historic, social, temporal, geospatial and other types of information are layered upon one another to perform a function, and the system (through a process called active learning) then learns how to improve form the user’s interactions. What this attempts to do is allow the human to give the machine algorithm some insight into the types of content they prefer, and the types of content they don’t. A statistical profile of the content features of each type is recorded, with varying degrees of nuance in-between including accounting for bias, crosstalk, irrelevance and falsehoods.

Some of this happens on the application side, some of it happens on the logic/cloud side of things. This is because it’s very important that the user understand that the platform is there to serve them, and not the other way around; algorithms augmenting human decision making. This means we’ve abstracted some elements of the system logic (the elements that everyone needs to re-use over and over again) while the things specific to the use of the platform, are defined in the UI.

Use Cases

We’re really excited to have had a number of really amazing partners new and old using the platform. This includes groups like Newsti.ps who are building a ‘people’s newswire’ using the Swift products.

There are also some really big projects that are occurring. For instance this BBC article profiles PAX which is using our platform to power their conflict early warning platform. They want to index massive amounts of data from around the world and then use what’s captured to spot the historic patterns and trends that then can be used to demonstrate confidence in future patterns.

One of our favorite uses of the Swift platform to date was Product (RED)’s use last year to mashup large quantities of social media activity to power their Turn The World (RED) campaign.

There have been many more uses that we can’t talk about yet, but hopefully those become public soon.

Some Numbers

There are currently eight different code repositories housing the greater Swift project. Each of these API elements is tackled as if it were a single problem. This includes code for location disambiguation, natural language processing, influence detection, reputation monitoring and duplication filtering. You can find more about them here – http://blog.swiftly.org/post/5788873594/resources-for-developers

• These combined repos contain around 150,000 lines of code (not including frameworks like Kohana)
• Over 7,000 downloads of Sweeper to date
• Which theoretically means at least 7,000 users of our APIs
• Sweeper users tend to aggregate thousands of items of content over the life of a deployment which means we’ve taken around 70,000,000 items of unstructured data and done things to it – like add location, tags or filtered out the duplicates. That’s obviously a very liberal extrapolation, but it should give you a sense of the amount of data we’re dealing with.
• As the project moves forward, and all our APIs are finally completed, this number will grow exponentially. With RiverID alone (which tracks the reputation of content and people online) we expect to be indexing over half a billion items of content and actions from the social web alone by the end of the year. That’s just one API, the others will also need to scale on equal terms.

“I’m a real family guy.” – Matt Griffiths

What’s your day to day at Ushahidi like?

So, I’m pretty sure that I remember my first day working for Ushahidi as being a long hard afternoon spent in Bubbles O’Leary’s (my local Irish bar) with Jon Gos discussing the potential for the SwiftRiver project over several cold Clubs. (Club is what passes for larger here in UG). Its surprising – or perhaps not – to see such an exciting platform grow for those early – and perhaps slightly too enthusiastic – discussions

How did you get involved in the software/tech space?

So I studied Computer Science in London and worked for a tech company all the way through my degree. I worked as a software developer and then as a development team leader and finally as a software architect planning, designing and managing ebusiness, ecommerce and financial system. I did all this for about 8 years before I move to Uganda with my wife where I got deeply involved with the tech community out here. I currently sit on the board of Appfrica (a Ugandan based tech incubator) and I help to run and manage the Hive CoLab which is a free and open tech space used by Uganda tech entrepreneurs, oh and I also work for Ushahidi

What are some of your favorite apps for work and how do you use them?

True to form, I’m going to answer this from a techy point of view: I love FireBug (web developer extension for FireFox) any web devs out there who haven’t heard of it GET IT NOW! It is IMHO the very best aid to client side web development! Then for server side dev stuff, I’m a massive fan of NetBeans (I know, I know: “isn’t that the IDE kids use?”) well not any more, their latest releases are great and it has loads of support for Python, Ruby on Rails and PHP. I’m a big proponent of Unit Tests in code and NetBeans fully supports integrated unit tests in most languages!

Then as far as frameworks go: I like KO3 for PHP, its defiantly the nicest PHP MVC framework around and there is loads of good support for it on the web. Then in the Python space, I have to admire Django (event though I only use it occasionally) just cause it QUICK AS YOU LIKE to develop apps on. That said, I have always preferred frameworks to be a bit less all encompassing and so my recent faves are Flask for web application development and just straight Werkzeug for developing web services on

As a final note, I are really starting to love NoSQL DBs, MongoDB is great and we about to start a project on Apache Cassandra (which is meant to be wicked, if not a bit of a pig to get going)!

What are some cool projects you’re working on right now at Ushahidi or; What excites you about your work right now?

Well, what a question: Obviously we think everything that work on is – well doesn’t everyone . To be serious for a minute, the SwiftRiver platform is really coming alive at the moment and as Director of Platform I get to be involved in all the cool new stuff going on, great things on the horizon are: We are about to make all out web services publicly available so developers around the world can play with out NLP, Geocoding, Duplication filtering services (and many more). Were also about to start development on RiverID, this is a huge project with some sky high aims but if we can pull it off we think it could be one of the best contributions by the open source community to content curation yet! Then we have new releases of our two web apps Sweeper and SwiftMeme soon to launch and the long awaited SwiftMail to come – what a busy time!

What helps you make it through each day?

Ah now that’s an easy one! My 8 month old son, Harley. He’s wicked and nothing to do with work or tech (although he does sit on my lap sometimes while I’m coding – could that be the reason for our bug count, lol).

The one thing you can’t live without?

So, same as above, really. Despite the fact that I work my ass off on SwiftRiver projects all the hours that god sends, I’m a real family guy and like nothing better than sitting down to breakfast with my wife Venita and our son. I work from home a lot which gives me loads of time to spend with my family (even though I then end up working till midnight)! So for me its a balance between working on great cool things that keep my brain and energy up and spending loads of time with my family.

“…there were no mentors to give me insight or expose me to growth opportunities.” – Ahmed Maawy

Ahmed Maawy joined the Ushahidi full-time in 2011 after having worked on the project as a contractor for several months. His primary focus on the applications Sweeper and SwiftMeme.

What’s your day to day at Ushahidi like?

Awesome and interesting. It’s one thing to be a software developer. It’s another to work alongside people who have what it takes to be top-notch developers by world standards. The entire team respects, understands and inspires each other. The flexibility is great. The degree of professionalism from each member of the team is excellent.

Ushahidi is not only a professional team, its a family. It’s not always about “work” but about the “person”.

How did you get involved in the software/tech space?

Personal initiative. I started college having good knowledge of C / C++, Visual Basic, and Borland Delphi. I even used to mentor the senior students on their projects. It is worth noting that from the place I came from, there were no mentors or developers to give me insight or expose me to growth opportunities. I got work before I left college, in which case my first employer, Mr. Mohamed Jaffer from Jafftek Solutions, Mombasa, noticed my talent and poached me. I thank Mr. Mohamed for giving me my first eye opener, out of which I managed to create a professional side of me from my talent by dealing with some rather difficult clients who had no idea about what computers could do for them. I later became an IT Manager for a Textile industry, got back to Software development and later got an opportunity to start work in Nairobi.

My development (mostly web) experience in Nairobi involved mostly online marketing projects as well as airlines reservations and booking engines. This experience gave me a good knowledge of the business and commercial drive behind technology. After that I move to non profit working in the Mobile and Web space on healthcare projects – working as a Software Development Manager for DataDyne.org. This experience made me understand how non-profits operate, which was a shocking difference to me at the begining, having a very strong understanding of the commercial setups in organizations. This is the point in which I learn that apart from free software, there is also free money – money you don’t have to go look for by knocking on the door of each corporate and demonstrating why a solution XYZ could increase their revenue by a good amount – The ROI world of software.

I then moved away from strategy and settled down into what I can do better than many other people. Software Development. I am glad to have joined Ushahidi. Its by far one of the best places any techy can work for.

What are some of your favorite apps for work and how do you use them?

I used to enjoy using Microsoft Visual Studio – under the .Net framework. Not that I don’t at the moment, I still do, but I have very many other applications I use. Although I had good exposure to web design technologies earlier, I tend to work best with code. I have worked mostly in corporate setups integrating enterprise solutions on Microsoft platforms – Sharepoint, Exchange, a bit of Dynamics, thus my initial bias for Visual Studio.

So, apart from Visual Studio .Net, I use IDEs, mostly JetBrains (Rails, PHP, Python), NetBeans, Aptana (for Rails). I do not perfer using plain text editors, not when there are freely available IDEs to do more than what text editors can do. My favourite framework to date is Ruby on Rails, I believe it is the smartest web technology out there (have a look at the case studies you can find for Ruby on Rails deployments – including Rackspace hosting, GitHub, BaseCamp, Groupon, and much more).

What are some cool projects you’re working on at Ushahidi or; What excites you about your work right now?

I mostly major on SwiftRiver and Ushahidi/Swift integration. I have worked on projects such as SwiftMeme, Sweeper and SwiftRiver itsself. I also work work on Android and Ushahidi platform, but am flexible to tackle any of the challenges.

The exciting thing about SwiftRiver is that even if the project is mature, it is still young and there is alot more to come. I like the fact that it is a challenge to transform SwiftRiver into something bigger than what it currently is.

What helps you make it through each day?

To become a person who can transform the society, or to become a person who can influence the society and inspire people to do the right things requires that you go through various stages in your life that can be both pleasant and unpleasant. Unpleasant circumstances may not only be work related, they can be as a result of various other factors. What doesn’t kill you only makes you stronger. What makes me make it through each day is my belief that everything happens for a reason in your life, and how it affects you does not matter. What matters is how you handle the situation.

The one thing you can’t live without?

Very simple – God.

It’s been just over six months that the SwiftRiver platform hit Beta, along with our first application, Sweeper. Today we’re releasing the latest version of Sweeper, a major overhaul that introduces a number of new features and capabilities. A lot of these are major improvements over the previous builds of the Sweeper app based on user feedback over the past few months. Here’s a quick rundown…

• Visual Dashboard – quick visual survey of data being aggregated
• Tag Based Navigation & Channel Based Navigation – sort content by selecting tags
• Content Clustering – sort content by similarity
• JSON output of filtered feeds – output feeds after they’ve been processed and curated by the sweeper app

You can read more about the latest release of Sweeper in this post by Matt at the Swift blog.

Download Sweeper v0.3

To do this we’re using techniques described in this post, to reverse query a database with place names coupled with their geospatial coordinates. Currently, Swift uses Yahoo Placemaker to do this, but much like Ushahidi, Swift is a free an open source project and, that said, we like to ensure that the components of our systems are also free and open source. We came across the GeoDict project while looking for an alternative to Yahoo’s Place maker and we’re excited to announce that our open source communities will be working together to integrate GeoDict’s existing code and roadmap into Swift’s. Followers of the Swift project will recognize that GeoDict is replacing our SULSa project, which we simply didn’t have the internal capacity to build out.

An excerpt from the announcement at the SwiftRiver blog…

What does this mean?

We’re not sure entirely, but there are some things we do know.  Both projects will remain available under the GPL. You’ll see us contribute our staff, time and resources to the development of GeoDict (because it’s an open source project aligned with our greater mission). GeoDict’s community will also actively contribute back to that code, and hopefully they’ll feel welcome enough that they’ll also contribute to SwiftRiver and Ushahidi code base as well.

GeoDict will be fully integrated into the Swift Web Services family of API products which we offer as both free and paid services, but also as open-source code for anyone out there to use on their own terms.

Big thanks to Pete Warden for creating GeoDict and for supporting our project. Welcome to the Ushahidi family!

Read More at Swiftly.org

Ushahidi is a platform for mapping & visualizing data geo-spatially. SwiftRiver is a platform that powers a number apps for for curating and adding context to real-time data feeds. SwiftRiver’s Sweeper app is one of these applications, designed to optimize user time by add intelligent curation tools and filters to help Ushahidi users manage information. It’s the buffer for groups who want to cut down noise from the web.

The optimal workflow is to aggregate all content & citizen reports (from SMS, Tweets, Email and Web feeds) into Sweeper and then to pass only the relevant information along to Ushahidi for mapping. Today, we’re happy to announce a new version of a plugin called Ushahidi Report Push which allows for native support for this.

Anyone who’s used the two products over the past few months is aware that this interoperability has been spotty at best and difficult to get working without customizing code. It’s now all working out of the box (well out of the repos).

Download the plugin here and find instructions in use on our wiki.

Daithi O Crualaoic explains the Guardian's decisions in customizing Ushahidi.

What is Guardian Activate? A Guardian platform aimed at world-changers who have proven that through the use of technology and the Internet, we can make the world a better place. Past speakers at Guardian Activate Summit have included Katrin Verclas (Mobile Active), Rose Shuman (QuestionBox.org), Eric Schmidt (Google) and Ethan Zuckerman (Global Voices).

Our own discussions with the Guardian staff spanned a number of topics:

• Lessons learned from Guardian’s uses and modifications of Ushahidi
• The role open source software like Ushahidi plays in investigative journalism
• Data Visualizing and Informatic Cartography (mapping)
• Exploring the SwiftRiver platform and products
• Ideas for new open source products for newsrooms and journalists

Guardian champions data journalism

We also had a great tour of the massive four executive floors of the Guardian’s operations in London and the opportunity to sit in on a few non-sensitive meetings and editorial discussions. In the past, Ushahidi has collaborated with newsgroups like Al Jazeera, the Guardian, BBC, Thomson Reuters and others. So it was good to finally have an intense week of discussion with one of the world’s foremost leaders in news, to get some insight as to how our products can be improved to aid the journalistic process.

At the 2010 Guardian Activate Summit our very own Juliana Rotich (Program Director, Ushahidi) gave this talk:

On August 30th we’ll release the first public beta of the SwiftRiver platform, an open source toolkit of semantic web technologies. It’s been a busy few months as we’ve been working round the clock to bring you a solid product.

One of the questions I’m asked frequently is “What can I use SwiftRiver for?” Here are a few examples:

1. Monitoring Real-Time Conversations

The most obvious use of Swift, what organizations like Ushahidi will use it for, is to help manage large streams of real-time content. Whether from blogs, twitter, email, SMS or other means, when something happens (ex. the Haiti Earth Quakes or Kanye West and Taylor Swift at the VMAs) there’s a flurry of activity immediately following the event. For someone collecting news on that subject, for whatever reason, we envision that they would download an instance of Swift, and begin monitoring a number of sources discussing said event.

2. Brand Monitoring

Similar to the scenario above, a PR firm or advertising agency might use Swift to monitor mentions of company’s brand online. This would of course include mentions on Twitter, but might also include SMS and Email campaigns.

3. Data Collection / Research

As a research tool, Swift’s veracity algorithms can be used to curate sources and content that the user trusts to offer more accurate information.

4. Sweeping through Email

If you don’t understand anything about the real-time web or the aforementioned ideas. One of the things almost everyone can relate to, is the need for ways to improve email filtering. Swift is something anyone can set up to help them sort their email by ranking the people you are likely to want to hear from higher than the people you don’t. Meanwhile, users can apply language processing tools to automatically sort email by subject, category or sender.

5. Sweeping Through SMS

Even users who don’t deal with the web at all may find use in SwiftRiver. For one, not all real-time data is online. If you’re on a closed network, you can use it to process text messages received from a local gateway. This useful for users of tools like Frontline SMS or Kannell.

6. Creating a Public Aggregator

One of our pilot partners used Swift to create public ‘planet’ style aggregator and news portal. This required some custom work from our end but we’re excited for their launch.

7. Monitoring Hundreds of Blogs/Sources

Perhaps you’re just a person, a blogger or journalists, who consumes large amounts of information on a number of subjects, like me. I currently follow about 2000 blogs in Google Reader. Reader is extremely useful because I can aggregate whatever I want. From the aggregated datasets, I can then choose to read and share whatever I want. Likewise, in Swift having too much information is actually a good thing, there are still serendipitous ways for navigating content (using tags), as well as a number of filters for viewings items in a more structured manner.

8. Building Apps on SWS

A few days a go we received a number of tweets about an app called FlipBoard, asking if Swift was anything like it. SwiftRiver is actually a very different animal. We’re more like the stack that something like FlipBoard would be built on. We offer several advanced tools (social graph mining, natural language processing, location servers, twitter analytics) for free use via our open API platform Swift Web Services. Anyone can use them and thus anyone can build applications on top of them.

We’ve been working with large media organizations around the world to customize such tools for their needs but because our stack is open, so can you!

9. Dashboard and Shared History Across Media Channels

The most basic feature that makes many of the above possible is that Swift allows you to create dashboard that includes messages from a number of sources and lets you sort, search and curate them all any way you want. This might include videos, tweets, email, text messages, blogs. All of the content you have a need to mine for information, for any reason is also possible.

10. Improving Your Blog

In addition to using Swift to collect research, bloggers are using Swift Web Services for their blogs. Users of WordPress or Drupal can add features like auto-tagging and more using Swift Web Services.

These are just some of the ways our alpha testers have been using Swift, there are many more possibilities and we look forward to exploring after our Beta! To find out more about Swift, try these recent posts from Robert Scoble, the BBC and GigaOM.

For the past two weeks I’ve been in the UK doing quite a bit of work to answer questions, conduct interviews and even give a few talks about the SwiftRiver platform. I hosted our second SwiftRiver 101 in central London and held private sessions with a number of media groups interested in finding out more about the platform and it’s capabilities.

I had the pleasure of speaking with Jon Fildes from the BBC who published an interview with myself and Erik Hermsan this morning. The above pic is from a short TED talk I gave on Swift just last week. Those talks are slowly finding their way online, so keep watching TED.com for it’s release.

Read the BBC Profile on SwiftRiver.

For those of you watching attentively, you may have noticed we missed our last release. As we’re getting close to our Beta, we’re focusing less on big public releases and more on the iterative updates that can be found on our Github account.

Director/System Architect Jon Gosier and Lead Developer/Technical Architect Matthew Griffiths, lead the days presentations. Please videos below…

Presentation 1 – Platform Overview

An explanation of the whole SwiftRiver ecosystem by Jon Gosier.

SwiftRiver 101 Session 1 from Ushahidi on Vimeo.

Presentation 2 – Swift Web Services

Detailed explanations of the Swift Web Services: RiverID, SiLCC, SULSa, Reverberations, and SiCDS by Jon Gosier.

Technical Breakout Panel

Detailed explanations of SwiftRiver’s code, redundant data abstraction layers and API by Matthew Griffiths.

Update: Slides from the day. Platform Overview and Web Service Overview