where’s the evidence that aidha creates change? randomized evaluation of compass clubs
Impact assessment is something we take seriously at aidha. Our goal is to determine whether our financial literacy interventions (compass savings clubs) and entrepreneurship training (venture clubs) have a meaningful impact on our students. What is the effect of this basket of interventions? Here we want to capture data on key outcomes that we’d like to see in students by the time they finish the 2 year curriculum at aidha. Some we’ve already elaborated in our theory of change (TOC), they include:
1. whether the student is saving regularly (habit being more important that aggregate amount per se)
2. whether she makes a productive investment (in our student’s context, a tricycle, a buffalo, something that will generate future income), and;
3. post aidha, whether she establishes a sustainable business, defined as providing enough income to sustain herself and her dependents (subjectively defined by the graduate). Learning about this post aidha activity requires long term follow up efforts with alumni. Because our students are migrants, we do not know when they might go home to set up a business, if at all. But that’s just another aidha challenge. And in fact, we’re already attempting to deal with it by initiating a whole separate project on this.
Yes, we believe we need multiple projects if we are to invest seriously in impact assessment. One reason is because our outcomes are complex but another is because impact evaluation can take several forms and each form has advantages and drawbacks. See an outline of design features by JPAL folks here.
For example, aidha’s TOC, and the assessments we hang on that, employs the pre/post quasi-experimental model, as we follow students over time. Here, we’re assuming that the program is the only factor influencing any changes in the measured outcome over time. (We’ll share the preliminary results with you in summer 2011). This is a clean straightforward assessment technique BUT we have to admit that aidha like many educational organizations probably faces a self selection bias. Our students are probably not “typical”. They appear to be pre disposed to succeed in saving and enterprise creation. They are probably not representative of the general population (in this case the average domestic worker). Because our students are highly motivated to approach us in the first place, our TOC impact assessment is unlikely to reflect aidha’s true impact on the general population of domestic workers in Singapore (we only need consider the difference between our students, who attend twice monthly 3 hour sessions on financial literacy + leadership skills training, and those who prefer to shop on Orchard road every Sunday).
So, we need another form of impact assessment to complement our TOC project. Really, we want to quantify how large the impact ACTUALLY is, if our students were NOT self selected. Randomized evaluations, or randomized controlled trials, are the gold standard in scientific and social science research. That’s why we’re thrilled that a randomized evaluation of aidha’s compass clubs (see straits times article on the specifics here), is now underway, led by economics profs Rashmi Barua of SMU and Kartini Shastry of Wellesley College with the support of Lehui Liang. These researchers are currently conducting baseline data collection for the study, to test the impact of aidha’s compass clubs. Next steps include the randomization of baseline survey participants into treatment (receive compass club intervention) and control (no intervention), before clubs start next month. We’ll keep you posted on results from this project throughout 2011 – stay tuned!
introducing the Theory of Change
Hello everyone,
For one of the first functional posts on this blog we are going to take you through a process that’s been invaluable to us here at aidha research: the Theory of Change (TOC).
Taken from the theory of change website, a Theory of Change “defines all building blocks required to bring about a given long-term goal. This set of connected building blocks–interchangeably referred to as outcomes, results, accomplishments, or preconditions, is depicted on a map known as a pathway of change/change framework, which is a graphic representation of the change process.”
Here are some of the characteristics of a TOC that sets it apart from other logic models:
- Identification of long-term goals and the assumptions behind them.
- Backwards mapping and identification of pre-conditions necessary to meet a goal.
- Indicators to measure outcomes and assess the performance of an initiative.
- Identification of the interventions that an initiative will perform to create a desired change.
The object is to take the above elements and fit them neatly into a map or framework, much like the one below:
All in all, the Theory of Change “shows a causal pathway from here to there by specifying what is needed for goals to be achieved.” It’s a hot commodity in the social enterprise world and really serves as a way for a community to come together in the planning process and identify any wasted efforts as well as positive causal relationships.
Here are further links for more information:
- A great powerpoint presentation by ActKnowledge on the differences between TOCs and logic models.
- An exhaustive paper by Connell and Kubisch on a Theory of Change approach.
