Fail Smart: 8 Points for Pilot Planning

[Ed. Note: This guest post by Alix Dunn, engine room, is very much in line with suggestions in the Ushahidi Toolkits. About Alix: Alix Dunn has a background in research about and training in the use of technology and media for social change. She currently acts as the creative lead for the engine room, and as a consultant for Tactical Tech's Privacy and Expression and Evidence and Action programs. She holds an M.A.in Media Studies from University of Oslo and a B.A. from Colorado College. (Originally posted on The engine room)] This post is about what to consider when you are planning a pilot project to test a bigger data collection project. Data collection can provide information to power your campaign and the process of collecting it can win communities’ support. But it’s not easy to set up a data collection project to inform and power an advocacy campaign. It’s challenging to identify the right strategy and tools. It’s easy to waste time, energy, money, and contacts with an over-reaching project that hasn’t been tested with a pilot. Have you heard yourself describe your project and think: “That project would be amazing if everything worked out perfectly. But I have no idea if it will succeed and I don’t even know where to begin.”? Thinking through how to effectively pilot may be the next logical step in launching your project. But what is a pilot, and how can you go about designing one? This post aims to work through the biggest problems in designing a pilot, and provide some guidance for how to do it effectively.

Informative Failure

Failure can be a wonderful thing. To fail well, it’s important to design a project so that when something always breaks, you can tell what piece was responsible for the failure. For failure to be informative, you must be able to isolate the cause. Anticipating and embracing failure, and isolating its causes can be challenging, but it is much easier if you design a pilot that starts small, changes fast, and is well-documented. If a project starts out too large, it won’t be clear what variables were the source of the failure. If a project changes too slowly, it will be challenging to eliminate the points of failure or stumble on innovative and effective methods. And if a pilot isn’t well documented, it will be much more difficult to identify and monitor failure. But what does it mean to start small? And what pieces and parts of a pilot should you consider in your design and planning?

Pilot Project Components

1. Scope and Issue

Starting a project that covers every issue from the outset is bound to fail — and not in a good way. Saying you want to monitor and support citizens in tracking every component of government service delivery will lead to frustration and chaotic failure. Picking a very focused issue area (rather than maternal health – access to pre-natal vitamins, rather than election “fairness” – opening and closing time of polling stations) can make success easier to track. Selecting small subsets of issues leaves you with tangible data about a specific issue, and it also means more targeted engagement with communities. It gives you great statistics, and it also increases the chances that your campaign will resonate. The next time you ask a maternal health related question, you want pilot community members to remember you and your project.

Do be as specific as possible with your issue that you want to cover in your pilot. Also have a general idea about what outcomes you need to understand how best to scale. An informative pilot is a success in its own right.

Don’t set yourself up for frustration and disappointment by picking an issue to tackle that is too broad.

2. Geography

Geographic scope of a project can play a huge role in the success of a pilot. Starting with a whole country, or even a whole city can doom a pilot. Think about a dead Ushahidi map with 40 reports across a country of 40 million people. That broad geographic scope makes it challenging to get others involved, and it makes the project easy to ignore. Choose one neighborhood and get everyone involved in a pilot, and then you have a story. You can establish relationships with the community and get feedback on what their expectations are for your project. You can carefully document who engages. We’ll talk more later about making sound choices when selecting geographic scope, but keep it in mind when considering other variables.

Do think small. Make sure you think about the smallest, most feasible-sized place to test your project. Pick a geography that you have the resources to cover and cover well. Think about the place where your strongest and most supportive constituency can help you test your idea.

Don’t pick a geographic scope that you don’t have the resources to cover. Don’t introduce your organization or project to an entirely new constituency in a place you don’t know well. And don’t go too big to test variables or make your team feel like they haven’t accomplished anything at the end of the pilot. You want a win and you want a success story.

3. Data Gatherers or Contributors

Your data collection project won’t succeed without people to collect and contribute data. If you already have a community or group of people who you know will be good data gatherers or contributors, start there. If there are other groups of people who already work together or associate who would be good for these jobs, speak with them about collaborating. It might be easiest to work with a group of people who are located near one another. If you know of a well-connected group that doesn’t live near one another, that may work well too.

Do let the pilot community know what you are working towards. Do collect contact information and make sure to follow-up with the community if outreach and community building is a component of your project.

Don’t collect unnecessary information or information that will put your pilot community at risk. Why collect national ID numbers if you don’t need them? If you are planning on sharing information you collect, make sure you tell the people you are collecting information from. There is no better way to burn bridges with your pilot community (and your larger project) than betraying their trust.

4. Audience

Who do you want to win over and who are you trying to reach? Before you step out on a limb and scale a project, it is important to provide likely supporters the chance to help out (through volunteer hours, donations, awareness raising, etc). It is also critical to understand who your message and project are meant to reach. There are lots of resources to do this. Here is a simple starter site for stakeholder analysis. Are you trying to influence government? The general public? Civil society? Women living in poverty in rural areas of your country? Knowing who you want to reach is the most important part of designing a project.

5. Supporters

Some people you reach out to will be more supportive than others. You won’t need to convince them of what you are working on, and you may even be able to get them involved in the project. Will you need more volunteers for the next phase of the project? Then you will need to target your pilot community to build a network of people that are not only willing to participate in your pilot project, but also so committed that they are willing to work with you in coming phases of the project. Is there a vocal politician that can champion your cause? Is there a well-networked neighborhood that would be a helpful, captive audience interested in making your project a success?

6. Budget

Pilots are hard to budget because they change so fast and are hard to predict. Always be on the lookout for hidden costs. Just because the software you want to use is free does not mean that it won’t cost time, energy, and resources to get up and running and to maintain. While technologies make it technically feasible to contact just about everyone in your country, don’t let the reach of technology distract. In mobile projects, starting a data collection campaign with a blast of SMSs to a huge list of numbers is popular. This kind of SMS blast is very expensive. A pilot is a good way to test a project idea on a small sample, and use the pilot as a way to iterate before spending a lot of money on a larger, follow-on bigger blast. Keep in mind that pilots are a great way to prove a concept before your boss or donor spends the big bucks on it. It also shows that you are keen to not waste money.

7. Incorporating a Pilot Phase in Project Proposals

It’s common to include a pilot phase in project proposals, and this is a good first step. But, having a paragraph on piloting won’t help the project if the rest of the proposal still commits to specific activities that haven’t been tested by the pilot phase. Proposals can include flexibility by openly describing the way a pilot may influence the development of the project. Make sure that project benchmarks are specific enough to be useful, but open enough to be adapted when you learn from the pilot phase.Donors like to see openness and honesty about how a project will unfold and how you are preparing for unpredictability. Flexibility is key. And while it can be challenging to plan flexibility into your projects, showing donors that you are considering how to test ideas and design as you learn from testing will show them that you are thinking ahead and thinking creatively.

Do try to use clear and frank language about the importance of running pilots. Be clear about challenges and opportunities and how you plan on managing a responsive and evolving project while also staying on course to meet your obligations to donors. Try to structure pilots in proposals so that they have real capacity to influence the way projects are planned and rolled out.

Don’t just say you are running a pilot as part of your project because you think it sounds good.

8. Results and Narrative

Documenting impact in a pilot project is critical to justify and design scaling. What results do you want your pilot to have? What do you want to learn from your pilot? The outcome of a pilot can be a subset of the outcome you want out of a project, or it can be tangible proof that the project will work (even if you expect the project to work differently than the pilot did). Document, document, document. After a pilot project it’s important to have a compelling story to tell. Who did you reach? What statistics and hard numbers can you show to document your work? How will this pilot function if scaled? What would you change if you could do it again? Let us know if we missed an element of pilot design or if you want to share an example of a successful or failtastic pilot project. We will be putting together a decision process for pilot design as a next installment. We are also happy to talk to you about your project if you want help thinking through how to design a pilot to test your concept. Also, thanks to Chris Albon at FrontLineSMS for helping me think through some of these issues!