8: Analyzing Data

Related Resources

Lecture slides (PDF)

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

SHAWN COLE: Great. It's a real pleasure to be here and thank you for listening to me. This is perhaps the capstone lecture of the course or at least the last lecture of the course. And I'm going to try to pick up right where Michael left off talking about intention to treat and moving on to treatment of the treated. But with any luck, they'll be some time at the end where we can have more general discussions or questions if people are still interested about particular topics or have questions. And I'll stay after class to talk to people. And I think you have my email on the slides. So feel free to get in touch with me at any time. We've got this great project. We want to evaluate it or write JPAL.

So what are we going to do today? Look at some challenges to randomized evaluations. So these problems. So basically, when people don't do what you assign them to do because you can't control them. You can control undergraduates in a lab, but in the real world it's a lot harder.

Then we're going to talk about sort of how do you choose which effects to report in your study? So you've got your study, you did a bunch of household surveying, what do you want to report? How credible are your results?

Then we'll spend some time talking about external validity, which is sort of the question, OK, I have a study that I think is internally valid. We did the randomization correctly and the results we think are legitimate. But how much can that tell us about the greater world around us?

And finally, we'll conclude by talking about cost effectiveness. Which, as economists, is very important to us. So we may have a program that's effective, but how do we compare whether we should spend our precious aid or budget dollars on that particular program as composed to a host of other programs?

And so I usually teach at HBS in the case method, which is very different than a lecture format because it's basically the students always talking and me refereeing. I'm not going to do that today, but I'm very comfortable with interruptions, or questions, or requests for clarification, et cetera. And I think we all stay more engaged if there's more discussion. And so that's super valuable. So that's the outline for today.

And the slides up here are going to differ a little bit from the slides that you have printed out because there was some last minute optimization and re-coordination.

So the basic problem, which I think Michael talked about, is that individuals are allocated to treatment groups and they don't receive treatment, or individuals are allocated to the assignment group, but somehow managed to get treatment. So you talked about students didn't show up at school, so they didn't get the treatment on the treatment day. Or the program said you can't give this program, you can't give de-worming medicine to girls over the age of 13 because of health reasons. They may be pregnant and we don't know how the de-worming medicine affects pregnancy. So what do you do?

You came up with the solution of estimating the program effect, ITT. Which is to use the original assignment. So we have our baseline survey, our baseline list of schools or people and we flipped our coins or run our [UNINTELLIGIBLE] code to randomize. So then it would just evaluate them no matter what actually happened and them. We count them in the treatment group or we count them in the control group. And that gives us our intention to treat estimate.

And so the interpretation of that is, what was the average effect on an individual in the treated population relative to an individual in the comparison population? And you've already covered this with Michael? This is just a review. So is this the right number to look for? Well, if you're thinking about putting in a de-worming program, you have to realize that people aren't going to be at school when the government shows up to administer the de-worming program to all their students. And that's that.

So maybe we can just pause for a second and think about some programs that if you're an [? IPARA ?] you think you're going to be involved with, or if you're involved in an NGO, the type of program you're running. And maybe a few people can volunteer what the intent to treat estimate might look like in their evaluation and whether it's something they care about. Put this into practice.

I should be comfortable with long pregnant pauses as well. Any examples? Excellent.

AUDIENCE: So one of the projects that we're working on is looking at the impact of financial education in low income communities in New York City. Obviously we're trying to measure the impact that financial education has, but we're working specifically with a certain NGO to implement that education. Whatever estimate we get that's the intention to treat will just be measuring the impact of the program.

SHAWN COLE: And so what are the compliance problems you anticipate there?

AUDIENCE: Maybe people not showing up for classes or not following through on whatever they're asked to do in the education.

SHAWN COLE: Or people in the control group could get on the internet and download the government's financial literacy program and study very industriously themselves and learn. The intention to treat will tell you what the program does and you could imagine that if only 15% of the people turn up for your meetings, you're going to have a pretty small difference between the treatment group and the comparison group. That'll be an accurate measure of the program, but that's not going to tell us much about how important financial literacy is in making financial decisions. So if maybe your outcome variable is what interest rate do people borrow at, or do they pay their credit cards back on time, you might not find much effect. From that we can't conclude that financial education doesn't affect credit card repayment behavior. We just have to conclude that this particular program we delivered wasn't very effective. So that's an example. Any other examples? The point is hopefully pretty clear. Great.

So I don't know if you went through these calculations. Yeah, OK. It's absolutely simple. You just take the average in this one, average in this one, and it's the difference. So it's not rocket science. So it relates to the actual program and it gives us an estimate of the program's impact.

I guess the second method we're going to talk about, which you talked about in your learning groups this morning too, is treatment on the treated. And maybe you could motivate this by telling the joke about the econometricians or statisticians who went hunting. Have you heard this?

So the first one aims at a deer and shoots and misses 10 meters to the left. And the second one aims at the deer and misses 10 meters to the right. And the third one says, yes, we got it.

So the intention to treat is giving us the average program effect, but maybe we care more about what's the effect of knowing financial literacy? What's the effect of actually changing people's understanding of financial literacy? And that's where the treatment on the treated estimate can provide some assistance.

So again, we went back to the worming example which you talked about. And so we have 76% of the people in the treatment schools got some treatment in the first round. And in the next round it was 72%. So that's actually nowhere near 100%. One-fifth of the students are not getting their de-worming medicine. And some students in the comparison group received treatment also. So for example, I think when you were testing the children and you found that they had worms in the comparison group, the sort of medical protocol required you to give them treatment.

So what would you do if you wanted to know the effect of medicine on the children who took the medicine? And so you can't just compare children who took the medicine with children who didn't take the medicine. That leads to all the same selection problems we had in the first few days of class where the people who decided not to come to school that day or weren't able to make it to school that day are different than the people who did. And the people in the comparison group who went out to the pharmacy and bought the de-worming medicine are going to be different than the people who didn't.

So what we do in this case is something really quite simple and it's at the foundation of the entire field of modern empirical research. But we won't go into all the details, we'll just talk about this treatment on the treated estimator, or ToT. And so what you don't do is just take the change of the people who were treated and the change of the people who were not treated and compare them. That would be just silly because people who are treated are different than the people who are not treated.

So I think conceptually, this is actually a really simple thing that we're doing. So I don't want to get bogged down or confused in the math. But in the ideal experiment, in the treatment group 100% of the people are treated. In the comparison group, 0% of the people are treated. And then the average difference is just the average treatment effect.

But in real life when we do our experiments, we very often have leakage across groups. So the treatment control difference is not 1, 100% in the treatment group treated and 0% in the control group treated. But it's smaller. The formal econometric phrase for this is instrumental variables. We instrument the probability of treatment with the original assignment and this will rescale the difference in means to give us a better estimate of what the effect of the treatment on the people who were treated is.

So this is a simple example. And it turns out it gets more complicated. We're not going to go into all the nuances. But just suppose for simplicity that children who get the treatment have a weight gain of A, irrespective of whether they're in the treatment or in the control school. And children who get no treatment have a weight gain of B. Again, in both schools. And we want to know A minus B, the difference between getting treated and not getting treated.

This is the math. And maybe it looks complicated, but it's really not. I think if we work through the Excel worksheet explaining what we're doing and then go back to the math, it'll become pretty clear.

Imagine we run this experiment with pupils in School 1 and pupils in School 2. We intended to treat everybody in School 1. We intended for everybody in School 2 to be in the control group. Unfortunately, we were only able to treat 6 out of the 10 people in this group and 2 out of the 10 people in this group managed to get treatment somehow.

The formula sort of guides us through what we need to do and then we can talk about the intuition between what it is. Is there a heroic soul who's willing to try and just talk through the math and figure out? A non-heroic soul? Cold call.

So, Will, let's start out with the easy stuff. So the formula we want is the average in the treatment group minus the average in the control group divided by the probability of treatment in the treatment group minus the probability of treatment in the control group. OK, so how do we calculate all of these things?

AUDIENCE: So first you want to look at the change in the treatment group. So you average out the observed change in weight for the treatment group.

SHAWN COLE: Great. OK, so that's three.

AUDIENCE: You do the same calculation for the control group looking at the average change in weight. You take the difference between those two numbers and that's the numerator of the proper fraction.

SHAWN COLE: OK, so that's yt minus-- OK.

AUDIENCE: And then to do the second half of the calculation you've got the denominator. Compare the rate of compliance in the treatment group to the rate of compliance in the control group. So the percentage in the treatment group that actually complied would be 0.6.

SHAWN COLE: 1, 2, 3, 4, 5, 6. Awesome. OK.

AUDIENCE: And for the control group, it would be the 2 out of 10. So the rate that received the treatment. So 0.6 minus 0.2.

SHAWN COLE: 0.6 minus 0.2. OK. And now?

AUDIENCE: Just divide the top and bottom.

SHAWN COLE: OK, so that's the math. See if we got it right. Yes, we got it right. Excellent. So what's the intuition behind what we're doing here?

AUDIENCE: You're sort of doing a weighted average.

SHAWN COLE: OK, we're certainly taking the two averages and we're taking the difference of them. And what do you mean by weighting?

AUDIENCE: You're weighting them by the degree of compliance.

SHAWN COLE: So if the difference in compliance-- keeping the top term the same, so suppose that this is still--

AUDIENCE: If compliance were really horrible you'd end up with no effect. It could swap the difference between the control and the treatment.

SHAWN COLE: OK, so let's take the mental exercise, keeping this yt minus yc the same at 2.1. If the compliance in the treatment group goes down from 0.6 to 0.3, what's going to happen to the ToT effect? Is it going to go up or down?

AUDIENCE: Up.

SHAWN COLE: Why?

AUDIENCE: [UNINTELLIGIBLE PHRASE] The people that you were targeting are not receiving the treatment. So in a sense, the effect that you are trying to describe is not what we would expect would happen because some of these people did not comply with what you were expecting from the beginning.

AUDIENCE: It could go either up or down depending on the second parameter.

SHAWN COLE: OK, so what I want to do is I want to say there are four parameters here. There's the average in the treatment group. There's the average in the control group. There's the probability of treatment in the treatment group and the probability of treatment in the control group. If this goes down, what's the ToT estimate going to do? It's going to go?

AUDIENCE: Increase.

SHAWN COLE: Why?

AUDIENCE: Because you previously underestimated the effect because not that many people received the treatment.

SHAWN COLE: So mathematically, if we're making the denominator smaller, the size of the fraction has to go up. That's just a law of mathematics. But the intuition, what I'm looking for is the intuition. So what's going on?

So suppose I first tell you that this difference is 3. Can you guys not see these numbers here?

AUDIENCE: No.

SHAWN COLE: That's excellent. OK, so let me use a blackboard, I think this'll be a little bit helpful.

AUDIENCE: So were you asking just basically that because not everybody is being treated it's diluting the overall result?

SHAWN COLE: Explain this. Give us the sort of intuition. I said that this treatment group has an average of 3 and the control group has an average of 0.1 I think. No 0.9. And first I told you that the probability of treatment conditional on being in the treatment group is 0.6. But now I tell you actually it's much lower. It's only 0.3. So we know from the math that it's going to be higher, but why? What's the intuition? Why does it have to be higher?

AUDIENCE: Because the gap in outcomes is actually caused by fewer people. Meaning that for those two people it must have been a really big gap to balance out the average to still be a lot higher when everyone else in the treated group actually got zeros.

SHAWN COLE: Yeah, absolutely. So we've got some difference in observed outcomes. And that has to be caused by the fact that there were more people in the treatment group treated than in the control group who were treated. So if there were a whole lot of people in the treatment group treated, then this rate difference could be just that the average score is two higher for each person in the treatment group, if everybody in the treatment group were treated. But if only three people in the treatment group were treated, to raise the average from 0.9 to 3, you have to have a really big effect on those three people. So that's all the treatment on the treated estimator is doing, is it's rescaling this observed effect to account for the fact that the difference is not 1 here.

If we had perfect compliance, if this were 1 and this were 0, then the denominator of the fraction would just be 1. This would look just like the intention to treat and the intention to treat and the treatment on the treated estimators would be the same. But if we have imperfect compliance, then we can scale up the effect to account for the fact that all of the change has to be due to the fact that a smaller number of people were getting the treatment.

AUDIENCE: Now you can only do this because you believe there is no systematic difference for why people were treated or not in between these two groups?

SHAWN COLE: Right. So there's some technical assumptions on how you interpret these effects. But if we agree that the effect is basically a constant effect, so our literacy program has the same effect on everybody, then this is perfectly fine. And we can do this because we randomly assigned the treatment in the control group so ex ante the two groups are the same. Will?

AUDIENCE: So why do you not have to worry that the people in the treatment group who actually comply are the more motivated batch of the treatment group?

SHAWN COLE: Right. That is a concern and it's covered in the papers, which I'll get to at the end. Basically, the sort of aggressive intuition is that this tends to measure the effect on people who are affected by the program. But in general, this is a pretty good way of just scaling up your program effects to account for the possibility of noncompliance.

So moving back to-- sure.

AUDIENCE: Did you answer his question?

SHAWN COLE: Why do we have to worry about it?

AUDIENCE: Yeah. It seemed like it's a serious problem.

AUDIENCE: It started random, but then people who actually got treated, if they were treated because they were systematically different then it's no longer-- your control group is no longer a good representation of the [UNINTELLIGIBLE].

SHAWN COLE: Right. There are technical assumptions about monotonicity and independence that I sort of would rather not go into. But if you'll at least just grant that there's a constant treatment effect, then I think we're fine and by scaling up the impact to account for the non-compliers, we'll be OK.

AUDIENCE: Is the idea-- sorry not to belabor the point. But is the idea that this method will get the correct magnitude of the treatment on those who are treated in this study, but you can't necessarily extrapolate that to say, that would have been the same treatment on the treated. Would you be able to force these other people who might be different in some way to get treated.

SHAWN COLE: Right. This will tell you the correct magnitude of the people who because they were treated, get the treatment. So that's again, a relevant parameter when you're running a program that causes people to take the treatment, you have the effect. If on the other hand you're the government and you have some ability to compel compliance, then you might not get effect.

So you may worry about this. Some people have done studies to try and sort of see how big a problem this is. And one example I can cite is some studies on educational literature. So in the US people have looked at mandatory school attendance laws that say in some states you can drop out of school at the age of 16 and in some states you can drop out of school at the age of 15. And so changes in these laws induce some people to stay in longer. But probably nobody here would have been affected by one of these mandatory schooling laws because we were all planning to go on to college anyway. And so people have compared estimates in the US with those in the UK where the mandatory schooling law were very binding and they found that the point estimates were pretty similar. So that's an example of where it can be reasonable. But this is something that you have to treat with a little bit of caution.

So there are other challenges with this. To get the ToT, we need to know the probability of treatment conditional being on the treatment and the probability of being treated condition on being in the control group. So why might these be hard to get?

AUDIENCE: I actually had a question. I don't know if it's directly related. Does this work if the probability that you're treated in both the treatment and control turn out to be equal or is there probability that--

SHAWN COLE: Great question. So you're anticipating a slide three slides from now. So let's go there and then we'll come back to my other problem. But this is the equation. So what happens if the probability of treatment and the treatment in the control group is the same?

AUDIENCE: Mathematically you have problems.

SHAWN COLE: Mathematically you're dividing something by 0 and you're going to get infinity or negative infinity. So yeah, that's not successful. I mean, another way of putting that is that your experiment failed. You randomly assigned treatment and you assumed that your program, a financial literacy education program, de-worming medicine, is going to deliver a product. But if, in fact, it doesn't change the probability of treatment at all, then your program failed.

It's sort of like if I were to sit here and working with an [? MFI, ?] send them an email and say, I'm assigning these 50 villages to treatment and these 50 villages to control. And then the email gets lost and they don't do any sort of treatment or control. And then they send me the data back a year later and I do the analysis. Well, I'm going to find that there's no difference in probability of treatment between the two groups and I won't have anything interesting to say. So that's definitely one problem. The experiment doesn't work unless the treatment induces a change in probability of treatment between the treatment in the comparison groups. So that's one problem.

There's a second problem with this estimation since we're on this slide, which is perhaps, slightly more technical. But if the difference between the treatment and the comparison group is small. So say it's 10% of the treatment group are treated and none of the comparison group is treated. Then we're estimating the mean of all of the treated people minus the mean of all the control people and dividing by 0.1, which is the same thing as multiplying by 10. So if there's a little bit of noise in these measures, then instead of-- suppose the true mean is 3, but you happened to measure 3.1. Then that noise is going to be blown up by a factor of 10 as well. So the estimate is going to be much less precise when the difference in treatment between the treatment and the comparison group is low. And as you said, if it gets to 0, you're in a pickle.

AUDIENCE: That's a pretty extreme problem. It means I had a treatment group and only 10% of the people.

SHAWN COLE: Sure. That's true, but it's not implausible that you'd want to run that type of experiment. So if you think of-- I don't know the exact numbers on the flu encouragement design papers. To try and understand how effective a flu shot is, you can send out letters to a bunch of people and some of the people-- the control group, you just send a letter saying, just for your information, the influenza season is coming. Please wash your hands a lot. The treatment group you send a letter saying that information plus we're offering free flu shots down at the clinic, why don't you come down? And you may only get a 10% response rate from that letter. But in that case, the treatment is very cheap. It's just $0.32, or whatever a stamp cost these days.

So you could do a study with 100,000 people. Then you would estimate both of these things very precisely because you'd have 50,000 people in the treatment group, 50,000 people in the control group. And you'd have these numbers and so you could come up with an estimate. It's not like we should give up on an experiment that has a low probability of treatment, it's just if we think we're going to be there we want to do our power calculations carefully. And I think you would have seen when you did your-- did your power calculations include noncompliance? OK, so if you start adjusting the power calculations for noncompliance, you see that you need larger sample sizes. And so that's an important lesson as well.

Sticking with that flu example. Why might this be hard? When we sent out these letters to all these people either encouraging them to wash their hands or encouraging them to get flu shots.

AUDIENCE: We have to observe both. Whether there's treatment in your control in particular, that could be really hard to do.

SHAWN COLE: So maybe if you're directing them to your flu clinic, you're going sort of observe everybody you send a letter who comes in for a flu shot. And maybe they're employees of Blue Cross Blue Shield or something and you hope you'll get a good estimate.

But what about this, you've got 50,000 people you've sent letters to in the control group. What can you do? Do you have to phone up all 50,000 people and try and--

AUDIENCE: And say, oh, by the way, did you happen to get a flu shot.

SHAWN COLE: Exactly. So do we need to make 50,000 telephone calls to solve this problem?

AUDIENCE: Randomly sample.

SHAWN COLE: Randomly sample? So pick 500, even 1,000, or who knows? Maybe even 200 people, 300 people will give you a pretty good estimate of what this is. You just need to make sure that it is a randomly selected sample and you're not just asking all of the people in this particular clinic.

We're getting a little bit out of order. I've gone through the math, but PowerPoint wouldn't let you see it. But at least suffice it to say Will talked us through it. It's not a lot of fancy econometrics we're doing here. This is called the Wald estimate if you've taken econometrics. But it's a very simple method for coming up.

So there's some problems or there's some areas where this could be a problem. And Will hinted at one. We can mention a couple others. So how might this treatment on the treatment design fail, let's say, in the letters example, the influenza example? So we're sending out a bunch of letters.

So suppose the treatment group is you send out a bunch of letters saying, it's flu season, come get a flu shot. And the control group is you don't send out any letters. And so what happens? Suppose you get 50% compliance. So treated and the control group, compliance means flu shot. You get 50%. Let's make it easy, in the control group you get 0%. You do a sample and just nobody gets a flu shot. Maybe this is in a developing country where people don't tend to think about flu shots. And suppose you get the flu rate to be 10% here. We could say this is in Mexico. And 15% here. Do I have my math right? Yeah.

So what would the treatment on the treated estimate here be?

AUDIENCE: 20%?

SHAWN COLE: So what's the formula?

AUDIENCE: 10 minus 15. Divided by 0.5.

SHAWN COLE: Minus 0. So it's minus 10. So giving the flu shot to a population will reduce the incidence of flu by 10 percentage points.

So what's an example of how this experiment could fail, or how this number could not be correct?

AUDIENCE: By reminding people to get a flu shot, you remind them that the flu is out there and they might do other things besides get a flu shot.

SHAWN COLE: Absolutely. So now they wash their hands, or they don't go out in public places, or they wear masks. And so you think that you're reducing the flu a lot by these flu shots. But maybe in fact, washing your hands is a lot more important than-- in fact, I think it probably is-- washing your hands off and is a lot more important than getting a flu shot. And that's what's giving you the effect. So if you're scaling up this impact in the treatment on the treated to give yourself credit for the fact that 50% of the people didn't get a flu shot. If instead they're actually washing their hands a lot, then this won't be the correct estimate of treatment on the treated. Is that reasonably clear?

AUDIENCE: I have a question. Will it be a good estimate of the impact of that specific intervention on the treated? So instead of measuring the impact of the flu shot, you're measuring the impact of reminding people about flu shots and giving them access to free ones?

SHAWN COLE: So would you want to scale that up or not? Which estimate would you want to take there?

AUDIENCE: Depends on what you're interested in.

SHAWN COLE: So suppose we're interested in, what's the impact of sending a letter to somebody and offering them a flu shot? Is the correct estimate 5 percentage points or 10 percentage points?

AUDIENCE: 5.

SHAWN COLE: Why?

AUDIENCE: Then your intent to treat is maybe more interesting because you want to take into account that people change their behavior and their compliance rates are [UNINTELLIGIBLE].

SHAWN COLE: Right. So then we're really interested in the package of effects that sending a letter causes, which includes sometimes getting a shot, more likely. But also washing your hands or being more careful. So the intent to treat-- and as we say, so if you have this situation, you should only use intent to treat. And intent to treat is very interesting and it tells us the effect of sending out a letter, but it doesn't necessarily tell us the effect of the flu shot. Anybody have any examples from their own programs or projected projects where they're concerned about this? Maybe you guys can be at least sure to work this in on your presentations tomorrow.

What about when you have spillovers or externalities? So I think Michael talked about spillovers and externalities this morning. But he might not have integrated that with intention to treat. Is that a correct characterization of his lecture? Excellent.

How could we go wrong-- let's stick to something I know well-- with the Balsakhi Program. In the Balsakhi Program, we have treatment and control schools. And sort of the compliance is 20% in the treatment schools and 0% in the control schools. And the change in test scores is, let's say, 1 standard deviation. No, that's going to be way to high. 0.2 standard deviations here and 0 standard deviations here. So quickly, as a review, how do we get the ITT estimator?

This is what we call a chip shot at HBS. But since we're not awarding points for participation there aren't a lot of golfers out there.

AUDIENCE: Intention to treat? It's just 0.2.

SHAWN COLE: OK and how did you get that? You just saw it?

AUDIENCE: You just subtract it. You don't have to do anything.

SHAWN COLE: So it's 0.2.

AUDIENCE: Minus 0.

SHAWN COLE: Minus 0. Over?

AUDIENCE: Wait, are you saying--

SHAWN COLE: Oh, sorry. Intent to treat. Yeah, exactly. Yeah, you're right. Sorry, my bad. So the intent to treat is 0.2. And what's the treatment on the treated?

AUDIENCE: 0.2 minus 0.

SHAWN COLE: Great.

AUDIENCE: Over 0.2 minus 0.

SHAWN COLE: This is confusing. This is the standard deviations in test score and this is the percentage compliance. And so what's that going to give us?

AUDIENCE: 1.

SHAWN COLE: 1. So wow, that's a spectacularly effective program. I was very proud to be associated with it. It raised test scores by 1 standard deviation. Which if you know the education literature, is a pretty big impact.

What might we be concerned about in this case? So what did the Balsakhi Program do? Refresh ourselves. It takes 20% of the students, pulls them out of the classroom during the regular class, and sends them to a tutor. And these are the kids who are often in the back of the class, the teacher's not paying attention to them because they're only teaching to the top of the class. Maybe they're making trouble, throwing things around, et cetera. What could go wrong?

AUDIENCE: You're attributing the fact to just taking this class to just those students in particular. Whereas you're taking them out of the class, so you're making the class that you left behind smaller. So there's effects--

SHAWN COLE: Right, so how does making the class that you left behind smaller matter?

AUDIENCE: Usually it's easier to learn in a smaller group.

SHAWN COLE: That's why we teach a JPAL maximum class size of 15 or 20. We hope that small classes are more effective. And there's also a tracking argument that maybe now the teacher can focus really on the homogeneous group rather than having to teach to multiple levels, et cetera. So there are all these other reasons to think that it may be the case that they're going to be some spillovers. So how would you explain in words to a policymaker why you're not sure that 1 is the right effective treatment on treated? This is particular for the IPA folks who will have to be doing this to earn their paychecks. But anybody is welcome.

AUDIENCE: There were sidebar benefits to the control group.

SHAWN COLE: Right. Well, not the control group.

AUDIENCE: I mean to the--

SHAWN COLE: Untreated students in the treatment group. So we were attributing all of this test gain to sending these kids out to class. But in fact, there could've been some test gain in the other groups. In the extreme, I suppose you could imagine that there's no performance gain for the children who go out to the Balsakhis, but getting those misbehaving children out of the class causes everybody else to learn so much more effectively that it raises the test score. So the program still has an effect, but the way it has an effect is by getting the misbehaving children out of the class.

Now it turns out that that's not the case that there are sort of some interesting ways to try and tease out how big the spillovers were. And I encourage you to read the paper if you're interested. And it turns out that there's not really any evidence of spillovers, so it really does seem like the effect happened through the Balsakhi Program. But it's definitely something we want to be aware about when we're doing our analysis.

OK, so we've already talked about this. If you have partial compliance, your power may be affected. So the intention to treat is often appropriate for program evaluations. It's simple to calculate. It's easy to explain. If you do this program, you'll get a mean change in your outcome of 0.3, 0.4, 0.5. There are a lot of advantages to it. But sometimes you may be interested as well on the program itself. And as we said, it measures the treatment effect for those who take the treatment because they're assigned to it.

If you have people who will never ever ever take the treatment. If you tried to run a randomized evaluation on Christian scientists who refuse medical treatment and you assign them free medical care, you're not going to find any effect. But we can find the effect of the treatment on people who take the treatment when they're offered it.

And so when you're doing the design of your experiment, it's important to think through these issues and think at the end of the day, at the end of the study, in two years time when we've collected all the data and analyzed it, what sort of results are we going to report? How are we going to report them? And how are we going to explain them to other folks?

So that is intention to treat and treatment on the treated. And let's move briefly to the choice of outcomes and covariates.

We always look forward to your feedback from the course. But in my view, the course might benefit from a little bit more focus on some of the practical aspects of doing an evaluation. So for example, survey design and determining what outcomes. Although maybe many of you are already familiar with that. So often when you do these randomized evaluations, you do like a household survey or you have administrative data on the individual and you have 100 or you have 150 variables about them. So you go into the field with a 20 page survey and you sit and you ask, how many cows do you have? How many goats do you have? How much land do you have? What was your monthly income? Are your children going to school, et cetera.

And so if you imagine an outcome like microfinance, which we hope causes people to be more productive and engaged and boosts household income, arguably boosts women's empowerment, et cetera. There are sort of a lot of things that could plausibly happen because you offer people microfinance. But from a statistical side, this offers some challenges. So what's the problem when you're interested in how microfinance affects 40 different outcomes. So education, consumption, feelings of empowerment, number of hours worked. You know, you can come up with a lot of plausible things that you think microfinance would affect.

AUDIENCE: At a 0.5 level too, those will come out significant having 40 even though--

SHAWN COLE: Right. So for people who haven't done hypothesis testing, hypothesis testing, which is what we use to analyze data says, how likely is it that the difference between the treatment and the control group is because of the program or simply because of random chance? And so you can look at the distribution of outcomes in the treatment group and the control group and observe their means and their variances and the number of observations, and you can come up with a rule that says it's very, very unlikely. Only 1 in 100 times would this thing have happened because of random chance. The standard that tends to be used in economics and the medical literature is this 5% p value, which is 1 out of 20 times.

But if you're looking at 40 outcomes, then on average, 2 of them are going to be statistically significant just out of random chance. So if I were to just randomly divide this group, this class into two groups, and started looking at things like US born or foreign born, or Yankees fan or Red Sox fan, or what have you, it wouldn't be too hard to eventually find something that were statistically significantly different between the two groups.

This is a challenge. There are a few ways to deal effectively with this challenge. So this isn't a particularly difficult challenge. So what the medical literature does in which, I think, we're sort of slowly moving towards in social sciences, is you have to stay in advance where you expect to find effects. So if you're going to the FDA to test the efficacy of a drug, when you apply for the phase III evaluation you have to say, I think this is going to cure Shawn's male pattern baldness. And I think that this is going to result in a more charming personality. And then you find Shawn and all of his brothers and you run the experiment. And then if the outcome turns out that the treatment group is now never get sick, simply immune from all diseases. In the past year, nobody in the treatment group got sick, you can't then go out and market that drug as something that cures diseases because that wasn't your stated hypothesis test ahead of time.

So oftentimes, we have a lot of guidance on what hypotheses to test. So if we're doing a financial literacy program, we expect that to effect financial outcomes and not employment and not divorce rates. Although you could tell a story that eventually gets you there. But when you're reporting your results to the world then, what you want to do is report the results on all the measured outcomes, even the ones for which you find no effect. So then, anybody who takes the study can look and say, OK, they're saying that their program has a great effect on income and children's schooling and health. But they tested 200 things, so I'm a little bit skeptical. Or but they only tested 6 things and half of them were large statistically significant impact, so I really believe that study.

Maybe it's a little unfortunate the last class of the course is about sort of challenges with randomized evaluations because I should just take a sidebar to emphasize that these problems we're talking about are not unique to randomized evaluations. Any evaluation you run these risks. So if it's just sort of the standard, what we like to think of as a pretty bad evaluation where you just find some treatment people and find some associated comparison people and do a survey, you're going to run into this exact same problem. So don't take many of these as a criticism as randomized evaluation, but just take them as good science. What to do for good science.

And there are other things you can do which is to adjust your standard errors. So we have a very simple way of calculating standard errors. But if you're testing multiple hypotheses, then you can actually statistically take that into account and come up with sort of corrected or bounds on your standard errors. And those are described in the literature that I'm going to refer to you at the end of the talk as well.

AUDIENCE: What about taking half your data and mining it?

SHAWN COLE: And throwing the rest away?

AUDIENCE: No, I mine half of my data. I just figure out what really matters and that's how I generate my hypothesis.

SHAWN COLE: That sounds reasonable to me.

AUDIENCE: I better have had a big enough sample.

SHAWN COLE: Yeah, you better have had a big enough sample. I think sort of that intuition is-- data mining is problematic, but it is useful when you run a study to report all of your results because if there are some surprising things there, we don't know whether they're there because of chance or because the program had that effect. But that gives us sort of a view on what to do when we test again.

So if we're going to try this microfinance program out in some other country or some other area of the country, we said it had this surprising effect on girl's empowerment. So now let's make that one of our key outcomes and let's test it again. So I think that sounds pretty reasonable. There's a pretty advanced, developed statistical literature on hypothesis testing and how you develop hypotheses. But that sounds not unreasonable to me. Other thoughts?

So another possibility is heterogeneous treatment effects, which is what we often look for. So you might think that the financial literacy program is more effective with women than men because women started out with lower levels of initial financial literacy, or you might think that the de-worming medication is more effective for children who live near the river because then they play outside more often, or something like that. And so it's very tempting to run your regressions for different subgroups. But again, there's this risk that you're data mining.

Suppose I wanted to show that I randomly assigned you this group into a treatment and control group. And I wanted to show that Yankees and Red Sox preferences were significantly different in the two groups. I could probably cut the data in different ways and eventually find some subset of you for whom the treatment and the control variables were actually different. And so you want to be aware of that possibility. And again, like the FDA drug trial way to avoid this is just to announce in advance which subgroups you expect this product to be more or less effective for and make sure you have a sufficient sample size to test statistical significance within those subgroups.

Again, as a service to all the consumers of your studies, report the results on all the subgroups. Even the subgroups for whom the program's not effective.

So there's another problem that-- did we talk about clustering in groups, data in the power calculations? OK, so that is sort of the bane of statistical analysis. Or as we liked to say when I was a graduate student, sort of people used to not really appreciate the importance of these grouped errors. It was much easier to write a paper back then because you found lots of statistically significant results. And once you start using this cluster adjustment, it's a lot harder. Now only 5% of results are statistically significant. So we whined that if only we had graduated five years earlier, it would have been much easier to get our thesis done.

But we here don't care about getting the thesis done. We here care about finding out the truth and measuring the potential for cluster in standard errors, or clustering common shocks from the group is very strong. You can often get estimates of the correlation within groups using survey data that you already have. That will inform your power calculations. But when you're doing your analysis, you need to adjust your statistical results for this clustering.

And in particular, you run into problems when your groups are very small. So if you're thinking, I want to do an evaluation where I had 15 treatment villages and 15 control villages. Well, it's very likely that outcomes are going to be correlated within that village. And then, all of a sudden, you only have 30 clusters. And even the statistical technique isn't great for sample sizes that are that small. You have to use other statistical techniques, which are sort of valid, but less powerful. So we won't go into the randomization inference. Again, it's mentioned in this paper I was talking about. You should be aware of this and I think the most important time to think about this is when you're designing your evaluation, to make sure you get enough clusters.

And if you can, it's almost always preferable to randomize by individual than group. Because randomizing by group requires typically, much larger sample sizes.

AUDIENCE: Can you just say one more thing or give us a reference on what you meant by other statistical techniques that are valid, but not as powerful?

SHAWN COLE: Sure. Let me just skip to this. This should be on your slide package. The ultimate slide is additional resources. And so there's a paper called "Using Randomization in Development Economics Research: A Toolkit, by Esther, Rachel, and Michael. And this goes through everything we've talked about this week in pretty careful detail, works out the math, and gives you references to what you need. So it's in there.

Josh Angrist, who's on the faculty here, has a very good book called "Mostly Harmless Econometrics." But it's designed for academics. It's not really a textbook. But it goes through these things in very, very, very good detail and it's fun to read.

But specifically, the technique is called randomization inference. It was developed by Fisher. And what you do is you basically-- so you have your treatment and your control group and your mean between the treatment and the mean in the control and you test the statistical significance. And then what you do is you just randomly reassign everybody to either treatment or control, regardless of what they actually did, and see if there's a difference between the treatment and control group. And if you do that a hundred times, you can sort of get a sense for how often you find statistically significant differences or not. It's related to bootstrapping. That's a reasonable method, but the problem with that is the statistical power's not very good. So you need a larger sample. And then once you have a larger sample, then you don't need to worry about it because you can cluster.

So another question that's maybe a little bit more technical is when you're doing your analysis, your regression analysis, is what covariates do you want to control for? So we're looking at the effect of financial literacy education on credit card repayment. When we do our statistical analysis, do we want to control for the age of the person, for their gender, for their initial measured level of financial literacy, et cetera.

Now the beauty of randomization is that it doesn't matter. Even if you don't have data on any of these covariates, as long as the program was initially randomly assigned and the sample size is large enough, then you'll be OK. But what the controls can do is that they can help you get a more precise estimate. So if a lot of people's credit card repayment behavior is explained by whether they ate a cookie as a child or not and you happen to have that particular data for people who have been reading The New Yorker, then you can soak up some of the variation and come up with a more precise estimate. Or you can control for age, or control for income, or other things. So it's often desirable to have additional controls.

But what you don't want to do is control for a variable that might have been affected by the treatment itself. So if you're looking at the effect of microfinance on women's empowerment, so that's the goal of your study. And then you would say, well, women who have higher levels of income report higher levels of feeling empowered. So that's an important determinant of feeling empowered. We should include that in our control when we do our analysis. But if it turns out that microfinance increased income and increased control, then we might conclude that there's no effect because we're attributing the effect to the differences in income. Is that clear? A lot of these are fairly nuanced issues and it's often worth consulting an academic or often a PhD student are eager to work on projects like this as well.

Just as a rule, it's important to report the raw differences and the regression adjusted results.

I think we advance a very strong view that randomized evaluation is a very credible method of evaluation. But even still, there are always ways to tweak or twist things a little bit to try and get the results you want. So you could have a survey with a hundred people or a hundred outcomes and only report seven of them. That might make your program look better than it is. So these rules we're proposing are ways to give people an honest and a thorough view of the effectiveness of your program.

So another rule is that when you're reporting your regression results, you should include the results with the covariates, as well as the results without the covariates.

So now let's talk about threats to external validity. So we spent a lot of time so far talking about internal validity, which is sort of was the treatment randomly assigned? Did enough people in the treatment group comply that you have a difference between the treatment and control group? But it's not sufficient to know that we've learned a lot from a randomized evaluation. So there's some threats that just doing the evaluation itself may have some impact above and beyond the program. And so these are called Hawthorne and Henry effects. And maybe we'll go back to the audience for some examples of a Hawthorne effect.

So a Hawthorne effect is when the treatment group behavior changes because of the experiment. What's an example of that? Anybody familiar with the original Hawthorne study?

AUDIENCE: Was is the lights being dimmed or different levels of lights and workers felt like they were getting attention paid to them no matter what the light level was at? It's just that there was something going on and so their productivity was higher.

SHAWN COLE: I actually don't remember the study. Can I just try and rephrase that? You can tell me whether I'm right. So the experiment was to try and figure out how the level of lighting in a factory affects productivity? I suppose. And so they said we're going to raise the level of lighting in this sort of select, maybe even randomly selected. We randomly select 50 out of 100 of our work groups and we roll in an extra light. And they're like, oh great, management really cares about us. They're including us in this survey. They're giving us extra light. We're going to work extra hard. And so you find a higher output from that group.

Alternatively, if you had said, maybe people are getting distracted by having too much light. So management picked these 50 groups, went in and unscrewed some light bulbs. They're working in sort of a dim area, they'd be like, oh, wow. Management really cares about our well being. Now we focus on the natural light and we're going to work really hard. So just the act of sort of being in the treatment causes your behavior to change. And so what's wrong with that?

I mean we're trying to measure the effect of treatment. If the effect of treatment is to increase productivity, fine.

AUDIENCE: Maybe you can not make a generalization to a population that doesn't have change that behavior because doesn't get the treatment already.

AUDIENCE: You run around changing the light levels at different factories and you don't get the effect because the real treatment was people running around and testing and observing and measuring the change in light.

SHAWN COLE: Right. And saying, we're really glad you're part of this study. It's very important.

AUDIENCE: Which was really what got people to work harder.

SHAWN COLE: Right. You might be able to generalize that if we decided to run this study in every factory in our firm, then we might get similar results in different factories. But probably, within a few months, people would sort of just catch on that this is just kind of wacky, and what's going on? It wouldn't really be the effect of the program. Any other examples from your programs of Hawthorne effects? Or things you might be worried about?

AUDIENCE: It seems like a lot of behavior situations, there's a threat to this. Especially if you have some in developing country context where you have foreigners coming in or people from the capitol coming in. Especially if it's something that-- again, with a behavior change where OK, I know I'm supposed to be washing my hands with soap. I normally don't, but I know that the white people get really happy when I do it and they're coming in to evaluate. So I'm going to go ahead--

SHAWN COLE: Right. So if you show up from abroad and put posters encouraging people to wash their hands, people may pay more attention to that. What does that validate or not invalidate? So suppose we did this and we found that the effect was it reduces reported incidence of diarrhea by 15%.

AUDIENCE: If you then don't still have white people coming into the village, then the same effect might not happen.

SHAWN COLE: Right. So I guess it's a little bit nuanced because we should distinguish between the program generalizability, which is the program could be white people come into the village, and Hawthorne effects, which is because I know I'm in the treatment group in the study, I'm going to act differently. So what's another example of really, a Hawthorne effect? I'm sympathetic to yours as a Hawthorne effect, but I want to really sort of nail it.

AUDIENCE: There's one for hand washing where every week people go into the villages and then tell them the importance of hand washing as a way to prevent malaria. And every time they ask them, did you wash hands? Do you wash hands? But that's not sustainable on a long-term basis because-- and at the same time, you're distributing free soap. So how do you separate everything?

SHAWN COLE: I would say again, that's the program. And so we would say if we scaled that program up to all of the country, we'd be fine. If we go in every week and tell people to wash their hands and distribute soap, that's fine.

AUDIENCE: So sometimes on sexual behavior studies, you'll find that a treatment group that is encouraged to adopt condoms, or something. And then in the post-measurement period, they know what the intervention is about. And so if you ask them, what has your sexual behavior been in the last week, they're much more likely to say that they've been using condoms. Or change their partnering habits or these other sorts of things that have nothing to do with--

SHAWN COLE: Right. So the effect of being in the treatment group and knowing that you're getting this treatment might change how you answer the survey questions, even if you didn't behave that way. Any other?

AUDIENCE: For the Hawthorne effect?

SHAWN COLE: So it's certainly a problem with your survey.

AUDIENCE: It might change other aspects of their behavior other than condom use that would be simply because they know that you're looking.

SHAWN COLE: So I mean I would've said something like I know that they really care about me and so because I'm part of this MIT Poverty Action Lab Study, it must be really important that I do this. But if you were to generalize the program, not as part of a study, then people would react to it differently.

The other side of the coin is the John Henry effect, which is people in the comparison group behave differently. What are examples of that?

AUDIENCE: The village in the control group is resentful to the politician who they perceived as determining who got treatment status and so they don't try as hard on whatever's being measured. Or they intentionally turn in a poor performance as a sign of protest.

SHAWN COLE: Right. They're like, why am I in the control group of this study. I wanted to be in the treatment group. I'm not going to use fertilizer because I know this study's about fertilizer or something. Other thoughts?

AUDIENCE: You could do the opposite. I could say, oh, those guys, they got the treatment. But I don't need that. I can do just as well, so I'm going to--

SHAWN COLE: I'm going to pull myself by my bootstraps.

AUDIENCE: [UNINTELLIGIBLE] going to start studying and double up and I'll show them.

SHAWN COLE: That's an interesting problem, is we don't really know which way these effects go ex ante. The Hawthorne or John Henry effects could be positive or could be negative and could be a challenge for the evaluation.

So how do you sort of try and address this to resolve these problems?

AUDIENCE: It'd be hard to do statistically.

AUDIENCE: This could be dangerous I suppose. But if you try to make the people in the control group, for instance, feel special in some other way. But in a way that you can say is not related to anything you're measuring from the treatment.

SHAWN COLE: Right. So we're doing a financial literacy evaluation where we're showing financial literacy videos to the treatment group. In the control group we're bringing them in and we're showing them films about health or something that we don't think will have any effect on financial literacy, but lots of the things are the same.

In the medical literature they do often double blind studies, where you don't even know whether you're in the treatment group or the control group. So you can't get despondent for being in the control group because you don't know you're there. Sometimes these are sort of inevitable and you can't get around them, but you should think about them carefully and try to minimize the risk.

So another problem with evaluations is sort of behavioral responses to evaluations. So we assign some schools to treatment schools and some schools to comparison schools. And the people in the comparison school say, oh. So we give textbooks to the treatment school. So lots of people say, hey, this school's got new textbooks. I'm going to go to this school. And so that increases the class size. So the textbooks may benefit the test score, but the increased class size offsets that and you find no effect of textbooks because the behavioral responses undid this.

Whereas if you were to do it throughout the country and give every school new textbooks, then there'd be no transferring around because there'd be no reason to change schools because your school would have free textbooks as well. So that's sort of another.

AUDIENCE: I just wanted to go back to your question about how to minimize the Hawthorne and John Henry effect. We're doing an impact study, impact evaluation on microfinance product in Mexico. We try not to talk about the study. The people at the top know that they're implementing this study, but the participants don't know anything about the study. And it's very kind of hush hush, basically to try to avoid changing behavior. And obviously that's not always possible. But to the extent that it is, people don't have to know that they're part of a study.

SHAWN COLE: I'm sure everybody here who's lived in the US has participated in a study sponsored by a credit card company that does randomized evaluations to figure out how to get people to sign up for their credit cards. So they randomly send some people 10 point font on the outside letter, some people 12 point font on the outside letter. Some people 16 point font on the outside letter. And they keep track of who responds and they figure out that this is the right font size. And then they say, what color should it be? This is the right color. What should the teaser interest rate be? And so lots of firms do this without you even knowing it. And then you won't get any John Henry or John Hawthorne effects because people won't even know they're in experiments.

Sometimes there are sort of consent issues that you need people's informed consent that preclude that from happening.

There's some issues that we were touching on before, sort of the generalizability of results. So can the program be replicated on a large national scale? So we're going in and giving free soap to these villages, but it would get expensive to give free soap to every village in the country.

The study sample, is it representative? So what's a problem you might run into here?

AUDIENCE: If for logistical reasons, you're only doing it in one state in a country and it's randomized within the state, but then there's just a different culture in the state, or there's a really strong history or traditions in the state that then are not generalizable up to the country as a whole.

SHAWN COLE: A problem you often run into is NGOs will, I think for reasonable reasons say, OK, let's do this study at out best bank branch or at our best district. Because doing a study is pretty hard. You have to have treatment and you have to have control and keep track of who's in which group. And so these people have been with us for five years and they can do the study really well. And so you do the study and you find some nice effect, but that is the effect of putting your best people into the program and you only have 20 of them. And now when you try to scale it up to 500 villages, you just don't have that level of human capital to implement the same quality of program elsewhere.

So sensitivity of results. This is sort of important, but may be second order important. The state of the art and the sciences, we're still looking for things that work and work well. So we're not as worried about figuring out if we give the de-worming tablet every month versus every three weeks, which one is more effective. I mean that's a useful important question and it probably deserves a study. But it's hard enough to get the big picture studies done to then move onto the sensitivity of the results. That said, sometimes there are often interesting economic questions you have. So you want to know whether microfinance has an impact on people's wealth. But you might also care about the interest rate. And so for microfinance to be sustainable, the interest rate has to be high. But for it to generate a lot of income for the borrowers, the interest rate has to be low. So you could try your program at different interest rates and see whether you find the same effect at different interest rates. That would be very interesting.

So there's often a trade-off between internal and external validity. In my experience, I think it's probably reasonable to focus on the first pass on the internal validity. Because the advantage of picking your best branch or picking your good people to get the study done and done well and have a large treatment effect is that we were sure we know what we're measuring. It's often hard to measure effects in the real world. It's not as if the hundred people in this room are the first people to think we should do something to reduce poverty. It's a difficult and thorny problem. And so if we can throw sort of our best program and show that our best program is effective, then we can sort of work on expanding and testing our second best program.

Statistical power is often stronger if you have a very homogeneous sample. So if you can randomize in a set of twins or something like that, you have very good statistical power. But twins might not be representative of the general population.

And then, of course, the study location is almost never random. In Indonesia we did manage to do a nationally representative randomized evaluation of a financial literacy program, but that was just because the world bank was doing a nationally representative survey and we persuaded them to tack the experiment on at the end. But otherwise it's almost always prohibitively expensive to travel around to hundreds of locations.

But at the end of the day, you do care a lot about external validity. You want to know that before you throw a lot of money at the program, can you get the same effect when you scale it up? And is this program effective for large populations?

So in the last 5 or 10 minutes, we'll talk a little bit about cost effectiveness. So you've done your program, you've done your evaluation, you've got the efficacy. You know how much it costs to deliver the program. Now how do you decide which program to pursue?

I guess the important thing in this is pretty obvious. It's just finding a metric that you can use to compare different programs. So in educational programs we often look at years of schooling as an output. Having an extra teacher causes people to stay in school longer, but extra teachers are expensive. You can figure out how much it costs per child year of schooling you create.

In health programs, they have something called a disability adjusted life year, which I'm sure some of you know a lot better than I do. But it's basically an unimpaired year of life with no disability counts as one. If your legs are immobile, then maybe it'd be 0.6 or something. And it sort of gets adjusted down to figure out which health interventions are more or less cost effective. Or you could do cost per death averted.

I think the interesting takeaway here is that doing these types of comparisons can sometimes lead to pretty surprising results. So we know how to get people in school by reducing the cost of education, so the PROGRESA program in Mexico made conditional cash transfers to students' parents who attended school. Providing free uniforms increases attendance. Providing school meals increases attendance. We've looked at incentives to increase learning. But we've also looked at the de-worming case.

If you'd said five years ago to educational people who specialize in education in developing countries, what do you think a very high impact intervention would be? I think very few people would have suggested de-worming. But if you do the math, you can figure out that an extra teacher will induce let's say one year of additional schooling, but costs $55 per pupil. So the cost per additional year of schooling is here for extra teacher. Iron supplements here. School meals here. Deworming here. So it's just tremendously cost effective to provide this de-worming medicine as a means of increasing years of education. Much cheaper than scholarships for girls, et cetera, or school uniforms.

And you could do this calculation not just for education, you could say, there are a lot of things that we care about. We care about health outcomes, human capital investment, externalities. And so an interesting thing that came out of the de-worming study was that if you did the old studies that didn't take into account the externalities and just sort of treated some people in a school but not other people in a school, it didn't look like a very good intervention. Because the kids would keep reinfecting each other even though they'd just been treated, and so it wasn't that cost effective. But once you did the school level randomization and took into account for the externalities, then the program turned out to look very, very cheap as a way of providing education.

It's also an incredibly effective way of improving health outcomes, de-worming. And much more effective, for example, than treating schistosomiasis.

You can do even more calculations. You can say OK, so we know that the deworming medicine is going to increase years of education by 0.2. Well, what's 0.2 years of education worth? There are economists who have done estimates of the returns to schooling in Kenya. They say if you get an extra year of schooling, you're going to get 7% more income throughout your life. And then so you've got 40 years of life at 7% higher income. You can take the present value of that stream of additional wage payments and you can see that, wow, by investing only $0.49, we're going to generate $20 more in wages at net present value, on average. And so if you have a tax rate of 10%, that's clearly a profitable intervention for the government if it's patient enough. Because it'll take in $2 in net present value in taxes at a cost of only $0.49 of delivering. Of course, there may not be any taxes on informal labor in Kenya.

So I think this is an example we like to cite first, because Michael helped prepare this particular lecture the first time around and is fond of that paper. But second, I think it's a very nice example of a program that has a really big macro effect. So basically, it's been adopted nationwide in Uganda, and they're expanding it a lot in Kenya. It's been tried in India and many other countries have realized that this is a tremendously effective program. And the ability to have this randomized evaluation, there's very credible evidence that said, we had these treatment groups, these control groups. We followed people for three years and the reason why our results are so different than the other results that you were citing as a reason not to provide de-worming is because of these externalities. And we can show you why these externalities matter. The credibility of that study really helped to transform policy and literally save thousands, tens of thousands of lives. Maybe more.

Other examples. PROGRESA, which some of you might be familiar with. It's actually the government of Mexico decided to integrate a bunch of randomized evaluations into its social welfare programs. That methodology and the results of what's been shown effective in that program have been adopted throughout Latin America and elsewhere. And Ben Olken, whom you saw earlier, did some experiments on threat of audits in Indonesia for corruption. And the government of Indonesia is increasing the probability of audits as a way of fighting corruption in the nation's fourth most populous country.

So I think the conclusion is that these evaluations, which take a lot of time and take a lot of effort, let's not kid ourselves. If you sign up for one of these, it's going to be a big affair. But it can have a tremendous impact. It's very important to know from your own perspective how effective your program is, but you can influence policy a lot. So I'm just going to conclude with two things. One is this mention of the additional resources, which should be on the JPAL website. This is a book you have to buy for $60. Only buy this if you're already familiar with econometrics. But they're both great treatments of the material we've covered this week. And I believe JPAL is in the process of developing a practitioner's guide as well. This is a much more technical guide that's full of equations. But hopefully, as you've seen throughout the last week, we've tried to explain things in ways that are accessible that you can explain to people who haven't taken econometrics and that will hopefully be coming out pretty soon.

And so if I were just to at least take two seconds to give my perspective, I'm young but I've done a few of these. It's probably helpful when you're doing one of these evaluations to engage in academics.

If you're thinking of doing a study, you just send an email to JPAL. They'll send it out to their network of 15 or 20 people and I don't know what the response rate is. But I think there's a lot of interest in doing these experiments. IPA is another organization that does this. But there's some subtle nuanced issues that require careful thinking through. Or you could just call me up and say, we're going to be doing this study. We're spending a lot of money on it, can we just talk through these issues with you? I'd be perfectly happy to do that.

And then, one thought. Just a few thoughts on keeping your evaluation costs effective. And maybe even think about this when you're proposing your program tomorrow, is that randomizing at the individual level where possible and appropriate is a tremendously useful way to keep costs down. Because their statistical power is so much higher, so to detect any given effect size you can do it with many fewer observations.

Another way to save a lot of money is to use administrative data. So if you can get people to give you access in New York to their checking accounts and credit card statements, and get those reported, at a low cost to you, the administrative data is often of very high quality and can be collected for little money.

And then the third trick is sort of using lotteries to ensure high compliance. So if you sort of announce that you've got this new program and you let everybody who's interested apply, and then you sort of randomly select 60 and have 60 as a control group, then the compliance is going to be pretty high in the treatment group. That Wald estimator, the difference between these two things, is going to be pretty high.

So we're at 2:25, which at MIT is the end of the class. And I guess the end of the course, at least from the lecturing side. So thank you very much and I will be around here to answer questions for the next 15 minutes if people would like to talk.