5: Measurement and Outcomes

Related Resources

Lecture slides (PDF)

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: So we now have, hopefully, I don't know. I've been going to some of the groups and we seem to be there for at least one of the groups I surveyed by the end of yesterday. We have a question. We know what we'd like to evaluate, and we have some kind of a design. So we know how we are going to get a sample. We know how we are going to randomize. And that's a good start.

So what we are going to do today is getting to know that we have a way of randomizing. We're going to get into the details of how we are actually going to go about collecting the data for the evaluation. So it's not going to be so much about the program, it's going to be about thinking how we really want to do the survey. Thinking about what is the sample size we need, how many data points we need, and what data should we go about collecting, and how much is it going to cost, and what are the trade-offs we are facing when we are trying to address those questions. So we could be spending from one week to one year on these questions. In particular, on the survey design aspect. These are complicated questions. Not many people who are much more qualified, certainly, than me on this issue of how to ask the right question correctly. So we are not going to get into any of that.

It's going to remain at a somewhat abstract level of business one. It's going to be both abstract and quite particular in a way. At the abstract level is how do we even think about the type of data we want to collect? And at the concrete level it's we're going to do it for one particular example, which is this Panchayat example which happens to be one that I know pretty well.

AUDIENCE: [UNINTELLIGIBLE]

PROFESSOR: Not making much progress, huh?

So what are the questions we need to know the answer to before getting ready to start our evaluation? One is what data should we collect? It's kind of a big question. What should go in the questionnaire? Anybody among you who's been involved in doing questionnaires knows that you cannot ask a million questions. And so how do we choose what to put in? And then, what systems we should put in place to ensure that the quality of the data is good? And what sample size we need to plan. So not going to do sample size this morning. It has its own little lecture and a firm set of exercises. So that's going to go this afternoon. We are going to be doing a lot of that and a bit of that today.

So let's start with the Panchayat example. So let's talk a bit about the setting. So the setting is a reservation for women in the Panchayat region in India. So what's the Panchayat and what's the reservation policy? So the rule of the game, I don't ask rhetorical questions. So when I ask a question, someone needs to answer otherwise we're stuck.

AUDIENCE: It was a program designed to decentralize the allocation of public resources in villages to the lower level. And at the same time, you wanted to ensure that meant minorities and scheduled tribes in India got represented somehow, and the preferences and opinions at this Panchayat level.

PROFESSOR: That's right. So the policy is Panchayat stands to consider five people. And maybe in some historical India there was this council of five people who make decisions for the villages. Or maybe not, but anyway, that's where the name comes from. And the idea, and that's not only India, it's something that you see in a lot of developed and developing countries today, is that decisions regarding local public goods such as drinking water infrastructure, the local wards, the buildings-- at least the buildings so, for example, schools, health facilities and such things-- would better represent people's need if people had a say in what they wanted. So it's very difficult to know if our bureaucrats sitting in Delhi that such and such village in the middle of Bihar needs a road as opposed to a water well. And so you can get this vast misallocation of money in this way.

So the idea with the Panchayat is that now even though the revenue collection is still pretty much not coming from the villages where there is not much taxation ability, this one defending decision should be taken increasingly at this local level to ensure that better adequation, a better fit between what you build and what people want. The problem with that, of course, is that when you have local decision making it's who is going to be controlling the power. So India, like many other countries, perhaps even more than other countries, has this history of these conditions again some particular group.

So, for example, the scheduled caste who are the former untouchable? The scheduled tribe who don't even have a caste? And so the worries that there are enough people, strong people, in these groups, that at the national level they can organize to make sure that there are policies that help these particular groups. Same thing for women. There definitely are very strong women in India. But in any local villages, you might not have a minority of scheduled casual tribal women who are able to take responsibility to ensure that their group is represented. So they might never be represented, you might have some tyranny of the majority. So that's why the system also introduced this reservation concept, which is to ensure some representation.

So how does a reservation work? How did they put it in place?

AUDIENCE: They reserved a certain number of seats for women in each council.

PROFESSOR: Right. So there is one reservation at the level of the council. So we can say these are various councils. Each council has about, say, 10-12 representatives which represent a population of 10,000 to 12,000. And so the first reservation is within each council. How many seats for women?

AUDIENCE: One third.

PROFESSOR: One third. So each council gets one third for women. And what about SC and ST What do they get?

STUDENT: In proportion to their population?

PROFESSOR: In proportion to their population. Not in that particular village, but in the whole district. So, for example, in Birbhum, there is about, I think, 30% of scheduled caste. So every village needs to have 30% of scheduled caste, even if they have much less. Except that if they have less than five, they're exempted. If you have no scheduled tribe in the village, you can not have a scheduled tribe representative. So that's the first thing. So are we going to be able to evaluate the impact of having this policy which has, within each council, has one third of the seats to a woman.

AUDIENCE: No, because there's no control.

PROFESSOR: Right. That's going to be very difficult because there is no control. So certainly we won't be able to look at the impact of this policy on what the council does as a whole. You might ask whether the particular village where she comes from, or segment of the village where she comes from, gets different good, possibly. But certainly you're not going to be able to compare the GPs since they are different. So what is the second layer of the policy that's going to be more helpful?

AUDIENCE: The Pradhan?

PROFESSOR: At the level of the Pradhan. And how does that work?

AUDIENCE: It's like a rotation?

PROFESSOR: Right. So how does it work for any particular election?

AUDIENCE: It's like it gets sorted or something?

PROFESSOR: Right. So what the deal exactly, is that they rank them. So these GPs have a serial number, which is some number that they had forever and ever. So they rank them by their serial number and by geographic unit. So they first rank them by block, and within each block they rank them by their serial number. And then the constituent three lists. One list of what they call general, one list of SC, and one list of ST. So the general list means it's not reserved for an SC Pradhan. Those SC lists means it's there for a SC Pradhan, and ST means it's there for a ST Pradhan.

And the way I did this selection is that they have to stay with a random number which they use to say, so if you need to reserve it's like a very long table. Because it tells you if you need to reserve five GPs, you pick number 1, 15, 17, and 21 for SC. So they do this list this way. And then this gives them three lists, which are ordered by serial number. And in each list, they count 1-2-3, 1-2-3, 1-2-3, in the first election. All the GP number one in all the lists got reserved for women. In the second election, again this goes through the same process. And all the GP number two are reserved, and the third election, all the GP number three were reserved.

So in principle there is a rotation, though it is not necessarily one for one in the sense that if a GP happens to be number one on this list in the election in 2003, and number two in this list in 2008, they get reserved twice in a row. Possibly if they become number three in this list, they would do that three times in a row. That could happen, too. But twice in a row is pretty frequent, actually. So this is more hopeful because now we have some GPs that are reserved for women. That says those three are reserved for women. And some who are not in the same year. So now we can compare them. Compare the decisions that are made in those places, compare anything we would like to compare, and we have a chance to actually identify the effects. Yeah.

AUDIENCE: Can you compare if in the non-reserve you elect a woman, does that become part of the control group, or?

PROFESSOR: So what's your thinking on that? So two things could happen, right? It could happen that in the places that are reserved for women, they elect a man anyway in defiance? So you don't have that problem, actually. They elected a woman. And the other thing that could happen is what you're talking about, is that some of those places might decide to elect a woman anyway. In fact, about 7% of them do. So what do you think we do with those guys?

AUDIENCE: They're part of the control.

PROFESSOR: We keep them in the control, right? Because we're going to spend more time on that tomorrow, but if we did put them in the treatment, then we don't have a random selection anymore because now in the treatment we have the people who are forced to have a woman that's on them. Plus the places that choose that's not random. So we'll be keeping them here, and this is going to mean that it's going to look a little bit like an encouragement design like you saw yesterday. Which is, in the places that are reserved for women, 100% of them have women. In the places that were not reserved, 7% have women anyway. So the difference is not quite 100%.

So tomorrow we'll deal with, see, so here we are going to focus on the impact of being reserved, and we're not going to be concerned so much about the impact of having a woman. So we are still going to always compare treatment versus control. Tomorrow you can see how, in fact, we can do something about thinking about how we go from the impact of being reserved, to the impact of being a woman if we are willing to make a couple more assumptions.

So that's a better setting. We can now compare those places and even though the Indian government didn't do this beautiful experiment with the view of an evaluation in mind. And they might have. And then they might contact you and say, hey, we have this set up. How should we go about evaluating it? And so that's the name of the game.

So I think we went through this maybe? We can spend a little more time on the goals. So what do we expect from the Panchayati raj? What will give us a first sense of what data we should think about collecting to know what's the goal of the institution. This big untraditional amendment took place in 1993. What was in the mind of the policy makers at that time?

AUDIENCE: That there would be more accountability to the interest in priorities of interested countries?

PROFESSOR: Right.

AUDIENCE: Greater women's empowerment?

PROFESSOR: So I don't know if it's a goal of the Panchayati raj amendment per se, but we could think it was something people had in mind in the reservation policies.

AUDIENCE: That program selection would be more aligned with the local preferences?

PROFESSOR: Right. Program selection might reflect local preferences. We should keep that in mind.

AUDIENCE: To increase the quality and quantity of public goods?

PROFESSOR: Right. We can see that there is more as a result of their accountability, there might be better and more public goods.

AUDIENCE: More participation in public life?

PROFESSOR: Right. We might also think that we might value the participation for its own good. I would think it's a good thing if people are involved in local democracy.

AUDIENCE: They vote more?

PROFESSOR: They vote more, participate in meetings, things like that. So that's kind of the sum of the objective of the Panchayat raj, generally. And within that there was this reservation. And the reservation policy is, was, still is in a sense, quite controversial. It is still controversial in the sense that the next step they are asking themselves now, and in fact I think has always been somewhat lurking in the background, is should there be reservations at other levels.

For example, for foreign peace. Like, should there be reservations in the parliament modeled on the one [UNINTELLIGIBLE] model, but akin to what we have on the Panchayat. And that debate is lively in India, but it's also elsewhere. Many countries have reservation policies. Last I counted, hundreds of some kind of some form of mandated representation for women. So it is not a very prominent policy, and some countries are very opposed to that. For example, the US.

So what are the potential pros and cons of a reservation policy? Since our evaluation is meant in a sense to inform this debate, we should think about what are the questions that are in people's minds?

AUDIENCE: It's not democratic. It's not majority rule.

PROFESSOR: Right. So that's a disadvantage. So it's not democratic, I'll just put it that way. We are constraining people's choices. They might not like it. So that reduces their utility.

AUDIENCE: But there is this structural inequality, so you need to empower your women [INAUDIBLE] do it, [INAUDIBLE] this reservation [UNINTELLIGIBLE]. But at some point in history, you need to do it to push women farther.

PROFESSOR: So I see more than one thing in that comment. One is that in the short run we need to ensure woman's representation. That's the short run. And I think what I also hear in your comment is that--

AUDIENCE: So many noises in the room. It's not working as it should.

PROFESSOR: Right. So one of the reasons why you might want to ensure woman's participation is just because-- just like that. Not even because you think it will affect, again, like democracy here is an end in itself. You might think, well, democracy is not real because women are not participating. I'm interested in having women participating, period. Even if it doesn't change anything to the outcome, that's what makes democracy real. So that could be one outcome. So we are interested in that as an end in itself. And I think one thing that I overhear in your comment that might not be there in reality, is also that maybe that will help jump-start the process where that would not be needed in the long run.

So it would ensure a woman's representation in the long run, and maybe change voter's view. View is such that in the long run, the effect would persist. In which case you might go to a bigger and better democracy where everybody's actually involved. In fact, it might be good for men as well to avail themselves of the possibility of half the population's talents which now, for some reason, becomes shut.

So that's a first set of things which concerns the participation itself.

AUDIENCE: So one of the advantages of ensuring women's representation is that they might then allocate more money to goods or services that benefit women's [INAUDIBLE].

PROFESSOR: So that is certainly something you hear a lot. So we allocate more money to goods and services which, sorry, can you repeat the end?

AUDIENCE: Which benefit women in [UNINTELLIGIBLE].

PROFESSOR: Which benefit women. Which might be good if-- you were saying that woman were not represented adequately before, and all-- yeah.

AUDIENCE: And then there might also, I mean, there's been some literature saying that there's been a spillover effect for children and families, but if [INAUDIBLE].

PROFESSOR: That's also something you hear a lot which is, so this is an argument in favor of redistribution. To half the population there is a group that doesn't get very much in the normal system. They need to get some part of the rent as well. And then that's sort of one step further which is an efficiency argument. You say, as you do that, you will also get a different type of goods.

STUDENT: Increases a woman's empowerment.

PROFESSOR: Right. So as you do that you might also increase a woman's empowerment. I think it's a double positive, but I'll leave it like that. Empowerment. Which may be good again as a redistribution or because we think that women in the house will do things differently.

AUDIENCE: Maybe the same thing as empowerment, but maybe more measurable is that it claims to reverse the trend of increasing gender gap in the population?

PROFESSOR: So one of the possible outcomes would be gender gap. You mean among children or among adults?

AUDIENCE: Eventually it will be among the adults, but it can be noticed [INAUDIBLE] indicators would be like reduction in girl infanticide.

PROFESSOR: So do we think there is a direct rule between what women policy makers can do, in terms of what type of public goods they can provide? Or do you think it's going to be all indirect to doing stuff that benefits women, which increases the power of women, which means that they make decisions that are better for girls in the household?

AUDIENCE: And perception, it will quickly change in our society.

PROFESSOR: So there is an issue of perception as well.

AUDIENCE: [INAUDIBLE] that when you see [INAUDIBLE] it would be more [UNINTELLIGIBLE] to follow.

PROFESSOR: Right. So you might think that might be something that people are also talked about is just, again, not just the aspect of what she might be doing. Just having the woman as the figurehead changes the image in a somewhat more permanent way. You don't just think well, she's there because of the reservation. You see a woman in a position of power and that might change your view of what's possible.

AUDIENCE: As a disadvantage, you also have the potential of the women who have poor job and so the perception actually becomes negative, if it's perceived that the woman might be unqualified.

PROFESSOR: So the first woman might be unqualified, and that would worsen perception. And then the other thing that your comment could be read as is maybe they're not particularly unqualified, but they are perceived as being unqualified. Because they are perceived as being there because of the reservation policy. And, in fact, it's makes it even more difficult for competent women to assert themselves. That's something that has been said a lot about affirmative action in the US. For blacks, for example, now every time you see a black person succeeding you're thinking, why? It's because they got some unfair advantage somewhere, and so that makes things worse.

AUDIENCE: Backlash.

PROFESSOR: So we can write it as backlash. That's a very good point. So that's a question that we can also ask. What's the perception of women, and women in power before and after?

AUDIENCE: We also create this backlash. It's my perception of it, too, but it's a different issue. It's not necessarily clear that women will, in fact, represent women better. It is possible that women will feel the need to compensate and therefore go out of their way to identify with men's issues, and go out of their way to not be perceived as a feminine candidate.

PROFESSOR: Right. So I'm looking for a term for that but I don't find one. Women may overcompensate and not represent women's interests.

AUDIENCE: As an advantage having women on the councils that you bring more information that may benefit women but also may just benefit the entire society. Because you weren't getting this channel of information either directly from their communities, or from their constituents who don't feel comfortable with women [INAUDIBLE].

PROFESSOR: Right. You may be more information. For example, because women speak up more, and so they give their opinion on stuff and there is stuff they might just know better about, even if they don't particularly care more about. Everybody may have the same preferences so that might not be a conflict. If you never hear from the women that the water well is blocked, then you may never think of fixing it even though that's something that would benefit everybody ultimately.

AUDIENCE: An advantage would be building capacity in women for leadership for the higher levels.

PROFESSOR: Right. So we are now talking about spillover over time. That you have one woman one time, and people understand that they are good and then they can continue on. But it can also be, of course, different levels while you're building up a cadre of powerful women.

AUDIENCE: I would think that you could maybe get an income of that, right away, on women. We vote next year to put in a water well. That means that there's a lot more time to do my trading business, [UNINTELLIGIBLE] business, so that right away I might make more money.

PROFESSOR: Right. So one of the ways we can put it into women empowerment, I'm going to add "to," so it could be to income. It could be to time that you generate, because now you spend less time collecting water. Or it could be to the role models as already discussed. But just the fact that I have the public good that is convenient for me, that I need. And presumably if women's power increased, they're going to do some kind of public good that's good for them which might be these types of things that will free times for them.

AUDIENCE: Kind of a counterpoint to the last disadvantage. It's also possible that because a woman doesn't have to compete against a man, they don't have to pander to men, and in a sense, overcompensate to men's preferences as much.

PROFESSOR: Right. So in a reservation-- we have too many advantages.

AUDIENCE: You add to disadvantage also which is that, I think, this may be more indirect in the sense that when you're electing someone on the premise that they'll able to deliver public goods to a specific constituency, you risk perhaps instituting a culture of paternalistic government that is elected just to deliver the pork to a specific constituency.

PROFESSOR: Right. And that is something that has been discussed a lot. For example, with the scheduled caste is that you could have this shifting. So you go it's your turn, it's our turn, it's your turn, it's our turn, and then people are not really watching the scheduled caste guy because they understand this is the scheduled caste guy's time to take their turn, and vice versa. It might also go the other way in the sense that the scheduled caste people who were previously disenfranchised might now feel, well, we might as well get this guy to deliver really well while he is in power for us. Same thing for women. But the effect on accountability is it's a good point. Because since the whole Panchayat is about accountability to the people, the effect of reservation system on accountability, at best, ambiguous.

So we can say this is related to this first point which is it's not democratic anymore. So any tampering with the democracy you might have story going either way. But that's a disadvantage showing, effect on accountability. You're now accountable to nobody. First, you are a lame duck. There is a very poor chance that you get reelected again, so that's your point here. Which in a sense makes you free to pander to your guys which is maybe what we wanted in this case, but on the other hand, makes you maybe less likely to deliver. Sorry, there was a--

AUDIENCE: Another potential disadvantage which is kind of still under whether or not women are in fact representing, is that if there's a culture of women not speaking up or not advocating, then you might end up with no representation at all.

PROFESSOR: Right. So there is a question of who is in charge. So it might be that when it's an election you elect someone, they're in charge. Here it's like maybe no woman really wants to run, so you pick up any figurehead. It might go back to elite control. If nobody who's democratically elected is in a position to exercise the power, whoever is a natural leader will take it back. It might be her husband, it might be anybody else. It's a very pretty reasonably recent effort that local democracy at that level. And it might be crazy to go back to elite control. So who is in charge? And is there a risk of elite control? Which is of course not why we did this in the first place.

AUDIENCE: I guess it builds upon the last point maybe overall less efficiency in elite administration functions because there may be resentment against the women, so less cooperation against them. So maybe there's more stalemate policies being passed.

PROFESSOR: Yeah. So that's a risk of stalemate. So unless you have a burning point, then we stop here because I have no space anymore on the board. So that's it, that's budget constraint for thoughts.

So there is kind of a lot of ideas and so somehow we are going to want them organized in order to go and collect our data. So one thing we could think of doing is that, oh let's postpone the big deorganization of the thoughts. We can see that it can go all over the place, so let's go and collect a bunch of outcomes and see how it goes. So given all of what we have discussed here, what are the possible things that might be affected by this policy? I'd like to have a board. I can just write. Given all the discussion we had, what are the things that we think might be affected by having this reservation policy? A lot of these things already came in, but we can make a little list for ourselves.

AUDIENCE: Choice of which public goods--

PROFESSOR: So one is definitely the public goods. So the disadvantage I'm going to-- so outcomes-- oh, wow. I don't usually use boards, so I'm going to-- ta da. That has to be MIT. So we have a whole [UNINTELLIGIBLE], so one is clearly public good, and potentially we have lots of them. What are the public goods that we can see in villages?

AUDIENCE: Water.

PROFESSOR: Water.

AUDIENCE: Roads.

PROFESSOR: Roads.

AUDIENCE: Schools.

PROFESSOR: Schools.

AUDIENCE: Hospitals.

AUDIENCE: Health centers.

PROFESSOR: Small health centers, yeah.

AUDIENCE: Young centers.

PROFESSOR: Young centers.

AUDIENCE: Large buildings.

PROFESSOR: Large buildings, irrigation, biogas.

AUDIENCE: Electricity.

PROFESSOR: Sorry?

AUDIENCE: Electricity.

PROFESSOR: Electricity, potentially. Sanitation. So like a long list of public goods could go either a way, so that's a long list. Where else?

AUDIENCE: Perceptions about women.

PROFESSOR: Perception of women. So if you want we'll talk a bit more about how we measure perception about women.

AUDIENCE: Participation.

PROFESSOR: Political participation. So attendance at meetings, voting. And of course we have men and women. That might be different.

AUDIENCE: Better governance?

PROFESSOR: So that's kind of the same thing. So better governance. In practice that's going to be whether there is graft, maybe budget utilization. So some measure of corruption. Bribes. You didn't say it actually in the advantages or disadvantages. The women are less corrupt, more corrupt. I guess it came up in the accountability. What else?

AUDIENCE: Household income.

PROFESSOR: Household income. And while we are in the household?

AUDIENCE: Health education.

PROFESSOR: Health education. Any gender differences in these things, both for the household and for the kids. We were talking about gender discrimination within the household. So again it's like a long list of household stuff potentially. Was a woman participating in savings, groups, blah, blah, blah? There is both perception of women politicians, and we also discussed about perception of women in general. What else could we need?

AUDIENCE: Maybe greater employability of women?

PROFESSOR: Right. So that's going to be maybe an income and then employment. That's part of the long list of stuff we might collect in a household.

AUDIENCE: Social cohesion.

PROFESSOR: Right. So that's maybe political participation and social cohesion. Don't know how you measure that, but we can think about it later.

AUDIENCE: Sustainability. Or sort of decentralization of power. The majority that's been in the government, it's like a vicious cycle. They'll do everything to keep the minorities out of the governance, so if you had jump started this process by introducing 30% of women--

PROFESSOR: Right, right, right. So it's perception of women politicians is in a woman's future electoral success. Is it the case that once a place is reserved for a woman, obviously you are the woman. How about the next time when it's not reserved anymore, do you have more woman candidates? Do you have a woman elected?

Women's future electoral successes. That's one of the first things we had discussed. More? I think it's a pretty long list already. And so now that we have this long list, the issue is what do we do? Suppose you had no money problem. Suppose budget was not a problem. Would you just take this long list? So one possible thing is to say, well, we'll think about this thing later. That makes a long list of outcomes, and go and collect data. So we're going to do a household survey and a community survey, and then an audit of what's there in the villages, and we are going to collect a bunch of data. And then this data will come back, and we'll start to look at it. So what are the pluses and minuses of that approach?

AUDIENCE: [INAUDIBLE] statistical significance. If you collect so much data, eventually you'll randomly stumble upon a result, and maybe that's the result that you end up reporting.

PROFESSOR: Right. So in particular, since we have seen at least some of the groups starting to discuss power, so you've seen hypothesis testing. So if I ran a hundred regression and looked for significant results and then independently looked for results in each of them, how many of them would count significant at the 5% level?

AUDIENCE: Five.

PROFESSOR: Five. So if I run 20, I would get two. So by example, suppose I collect 20 public goods, which is not such a large number. And we find that water wells go up in places which have more women, and irrigation goes down, and anything else doesn't change. What can I conclude if I have just gone on to this big fishing expedition?

AUDIENCE: [INAUDIBLE]

PROFESSOR: Well, you don't know whether it was just random. You can definitely make up a story. What is a story you could easily make up on the basis of those results?

AUDIENCE: That women invest in wells because collecting water used to be a woman's job, and men invest in irrigation because farming is a man's occupation.

PROFESSOR: Exactly. So you could say, well, women are not going to benefit from irrigation anyway, they're going to benefit from drinking water a lot. So this is quite consistent with what we were saying about women leaders investing in the goods women want, and so that's great. The only problem is, if we've done that ex-post, someone else could say, well, or an alternative interpretation is you've run 20 regression, you've found two significant, and you're making up a story ex-post to explain the results.

It's not that it's morally wrong to do that, but if you're not sure it is always going to be some suspicion. And then it's a little bit sad to have spent so much money collecting so much data and not to be very sure how to interpret the results. So maybe that's not the right approach. Maybe we need to do something slightly different, and that is we need to try and put a little bit more thought into why we are collecting each piece of data and where it's going to fit in our global explanation of the results.

So if you take this specific example, instead of doing what we just suppose we did, which is making this long list of results and hope for the best. Instead, if we had said we are going to go after this one question, potentially we can have we can have more than one big question. But we are at least going to go after this one question which is, what is it the case that the women leader do what women want? And we had given ourselves the means to first find out what women want for real, and not making it up from the results.

The key is to try to not be in a position where you're going to have to reverse engineer your explanation of the result ex-post. If you want me to be completely honest, it's very difficult not to do that. Because once you see the results you always want to explain them to yourself, and explain to others. So I'm not about to tell you that from a position of where I sit and say well, don't you do that or you will be damned and go to hell. But the truth is that we do it, but the truth is the less we have to do it the better. And you have to do it less if you've been thinking ex-ante about what it is that you want to test, and what are the steps that align themselves in order to get to where you want to be.

And this result might have been totally fine if we had a good way to say this is what woman are really interested in, and we had collected data on that. And then where our test will not be good by good, is it the case that women do different things. But we ask the questions that generally do women go in that direction. And what is the case? Yes, we're missing a moral explaining to us why it is. How do we go from this hypothesis we have that women are going to do what women want, to how is it going to translate into the public goods.

So the hypothesis to test must be defined before the beginning of the experiment or we don't know how to assist their validity. And what we missed when we did this big list of outcomes, or what we would have missed if we just did this big list of outcomes, and then go out and collect the data and then think about how to interpret them, is these steps. Which is go from a discussion which is very likely at an implicit level at the back of our minds. In fact, now it's very much in the front of our minds since we just had it. But usually when you do an evaluation, it's everybody sort of had that in mind, but it might remain implicit and it's much better to make it explicit. First because you're more likely to collect the data that you actually will need. For example, here we might have missed collecting women's preferences. It's not in the list. It's not in the list of data that's here. Why? Because it's not an outcome. So if were just thinking about, oh, what are the effects, and we forget to collect women's preferences. But then we have no good way to interpret the preferences in the context of that model if it was the model.

So if we don't do this thinking implicitly, we might be missing a key step. In particular, in what we call the intermediate variable that might be needed not as a measure of the impact of the program, but as what is going to help us interpret the impact of this program. So you need to try and define the hypotheses before the beginning of the experiment. And this is actually an exercise that's very useful in my experience working with implementation partners. It's very useful for both sides. Because from the side of the evaluation team, strive to understand what is the program you're evaluating. How it connects to what you know about, say, developments, poverty, et cetera. From the side of the partner, many of you have more experience than me on that. At least the half that are actually into implementation. It's like why are we doing this? Sometimes the answer is not as forthcoming as you might hope.

AUDIENCE: The other thing that was noticeable to me when we were doing this is that there are different levels maybe of importance, or that a lot of them are subsets of other things.

PROFESSOR: Exactly. So one of the things that you might do in this process is to prioritize what is sort of the big news, sort of the million dollar question? What are subsidiary? What are things that are going to enlighten whatever impacts you find? So you might also, we are going to do that in a minute, is to think depending on how much money you have-- is it that good? So sorry. We are trying to do something.

Given how much money you have, you might be thinking of some small things, or some things that might-- for example, some of the things, like say, one of the outcomes we are thinking is relative mortality of girls. You might think realistically two years after the women's empowerment is less likely to happen. So if you had an infinite amount of money, you might still collect it as a subsidiary outcome, but you wouldn't want it in a long list up there with whether or not they invest in water wells. There's no way to do this prioritization unless you are in good cause the exercise of thinking to your causal model, linking whatever intervention you have to the results.

AUDIENCE: In this discussion I believe you also raised the possibly of doing like a factor analysis to create an index of different factors so that we could reduce the number of variables.

PROFESSOR: Right. So that's a very good idea. But on the other hand, you have to think about how to-- your instinct is the right one, which is to say how are we going to combine this stuff? So if you take, for example, the 20 public goods. If you're like, but these women are more efficient, they're going to build more goods. Then you might want to do that, or you might want to sum them or to average them somehow, which is what making an index is. To see whether in general all go positive. But if your model is along this line which is women are doing what women want, then you might not get more goods. You may get some of some, and less of others, so you might need something else than an index to deal with this mess.

But you're exactly right with the idea that what we want is a way to not have many, many, many hypothesis, but fewer. So, for example, it can be one, which is a woman does what women want. That would be the first one, and then what comes out of that. If that's your hypothesis, the index you create would be something different than what you would get from a factor analysis. If, on the other hand, it's an education intervention, and you have a math result, and an English result, and science result, and [UNINTELLIGIBLE] results, and geographic results, then you know that they should all up. Or at least that's your hypothesis they should all go up, then you can do something like that to average the effect across the outcomes.

AUDIENCE: I understand why you don't want to throw in 20 different outcomes. But in terms of how you prioritize the outcomes along with their causal model, there are some fields where it's going to be fairly clear. You know, in the health field, they would say, OK, well, our objective is to reduce malaria, so we didn't reduce malaria. But if something that has more intermediate steps along the way, such as a--

PROFESSOR: What was that about?

AUDIENCE: --such as like a conflict mitigation program, where you're saying, OK, we are going to get children into clubs because we believe that this is going to create more solidarity among ethnic groups, and that this is going to lead to conflict reduction. When you have, like, six or seven different steps along this causal model, how and at what point do you prioritize, this is the impact that we're looking for, versus these are the steps that we have take to get to this impact?

PROFESSOR: So that's a great question, and you can answer it at two levels in a way. One is, where and when you're going to see an impact first. For example, in this case, you might think that ultimately what we care is not the number of water wells. Ultimately what we care is the growth of the Indian economy through these complicated channels. But this is not what we are trying to do, because we think it's going to take a bit more time to arrive to this question. Or ultimately what we care about is girl's mortality. And we're thinking that through this complicated path we're going to get there eventually, but this is not the yardstick by which you measure the success of the program because it's in too long time, and after too many other things have diluted this. But at least you want to know that you're going in the right direction, so you might decide to stop by the water wells for the beginning.

Likewise in your program, you might say you're going to produce into groups to work together. You might say, well our first measure of success with this program is whether or not the use opinion of the other guys has changed in a way that we can reliably measure. And then you're going to ask yourself the question of the measurement, and then you can take it at various steps. And one is, am I still going to find an effect after all the things that have happened? So is it fair to my program? And the other is what can I measure? Some things are easier and harder to measure, and they might be at different levels in the chain. It might be that what is happening first is easier to measure, sometimes it's what's happening a little later that's easier to measure. For example, perceptions might be very hard to measure. But whether or not people are willing to work together to build something might be very easy to measure, so you might go for that.

AUDIENCE: And at the same time, you do want to make sure that you're doing something further enough along the chain that there isn't actually impact, and not just an output measure, or--

PROFESSOR: Exactly. So this is where this conversation's going to be helpful. What is our program hoping to do, and how. And sometimes you realize in that conversation is that what is your program hoping to do so squishy that maybe we should do something else. But not the case of your program that you just described which sounded actually quite specific in what it was hoping to achieve.

So here's our example. So we have to define the hypothesis. So, for example, we can take here, we could take several. One possibility is to take this one which is public goods favored by women are more likely to be chosen by women. So to test that, we need to know what women want, and then we need to collect the data on public goods. And then the one thing we are going to do is, we are creating this index of wanted by women more than by men, which is the way we would aggregate. And then the key test would be are the public goods moving towards what women want? And so the one little snag is how are we going to measure women's preferences? What is it that women want? So what do you guys think? What would be possibilities to measure a woman's preferences?

AUDIENCE: Household surveys?

PROFESSOR: So we could ask them, in household surveys. That would be one way. What they care about. Any other ideas?

AUDIENCE: If there were communities where women were well represented, what were the choices of those communities, historically?

PROFESSOR: Well, it's slightly circular because--

AUDIENCE: [INAUDIBLE] that have a majority of women on the local council have higher percentage of wells.

PROFESSOR: Well, yeah. That's a little bit circular, because that means that what we can do that only, A, if we believe that, in fact, it is true that women better do what women want, which is what we are trying to test. And B, if we think there was no selection in which village elected a woman, and we don't believe that which is the whole reason why we are going through this tortuous exercise. So that might--

AUDIENCE: I know in the exercise it talked about the transcripts from some of the meetings, so you could look to see if there were patterns for what issues were raised. Whether or not the council voted on them, it was what they brought to the table.

PROFESSOR: Right. So you could do that. In fact, this is what we did in this case. Cheaper than household survey, is to say, in general, what do women ask about in those meetings? Preferably, not in places where the woman is the head, because that might have changed the dynamic, but in places which are not reserved. What are women talking about? What are men talking about? And the advantage of a household survey is that it's a little bit costly to go and talk about something. In a meeting, you speak up and other people look at you. If you have to go to the leader and have a written complaint it takes some time, and people wouldn't do that unless they cared about it. So maybe that's one way to proceed. But household surveys is also a good way to proceed.

AUDIENCE: But if you're measuring whether women represent women, like you've assumed at that point, but if you're looking at the discussion within the council, you've assumed at that point that the woman that was selected--

PROFESSOR: Alright.

AUDIENCE: --for women.

PROFESSOR: Alright. So I was thinking of not the small council, but the big council when they have this big meeting. What she was referring to is the transcript of the [UNINTELLIGIBLE] meetings. Or even when women have gone to the office of the Pradhan and asked them. So that's different ways, but the key is, we need to have that. Because then we can see it doesn't go into that direction. And now we don't have 20 public goods, we have one hypothesis.

AUDIENCE: Is it necessarily critical that what women are speaking about when they go visit to large meetings that's what they want, and what the men are speaking about, that's what they want? What if a man is speaking on behalf of his wife's interests?

PROFESSOR: Right. So it is not entirely clear. And so this measure of women's and men's preference might not be the best you can think of. The key is to have a good one. I think you're right, which is it could be that women never speak for example, or never complain about anything, and they would still have preferences so what would have to figure something out to get them. To the extent that both genders speak, then you might think that they speaking about what they care about as long as the house is not a fully harmonious unit. Or you might think that it's not, and then for example, women tend to talk about water just because they know better about water as someone suggested earlier. And it doesn't represent needs, it just represents relative advantages.

So that is something that this particular wealth measuring preferences might not be the best one, but the key is to have one. More than one would be even better. So in this case, that's the only when we had so it was a bit of a gamble. But the key thing is that we want to do that. Then there are other things we might be interested in. So once we have that, do women invest more in public goods, then we can ask the next question which is do investing more once we find this that kind of can be our study number one. Which is yes, we are showing that women invest more in the public goods that women prefer.

Then there was this other question that also came about, a completely different question, which is, does having a woman as a policy maker change the perception of women as policy makers. And that we can think of a separate hypothesis altogether that can be tested separately. And it could be that women as leaders do different things and do what women want, but that there is no lingering effect of having a woman because that doesn't affect people's preferences or because there is a backlash. So we can think of these two as different bins. We can even do one study and not the other, of both of them. But they are kind of separate tracks to which to go.

So if we take a minute on people's perception, do women affect the perception of voters of women leaders? How would you go about thinking about this? How would you go about trying to measure people's political preferences?

AUDIENCE: Asking questions about the role of women in the household, or women in the community.

PROFESSOR: So you can ask questions. So what is an issue is this asking questions in this context? Do you like women leaders? What do you think of--

AUDIENCE: People might feel like there's a right answer.

PROFESSOR: People may feel there is a right answer. And what's the right answer, do you think, in this context?

AUDIENCE: I see that you can tell the story right away that it's important to believe in equality between genders. Or that if you're a male, in front of your other male friends you want to appear like a tough guy.

AUDIENCE: Or who's asking the questions.

PROFESSOR: Or who is asking the questions. Or you might think that it's the right time to send a message to say those people from the capital who are imposing this woman on us, better tell them that really we don't like it. So it could really go either way, we don't know. And in a sense that is something that is interesting as well, is to ask which way is what people are willing to reveal to you? In which direction is it best? So then we would need some measure of "to" preferences. To willingness to consider women as policymakers.

AUDIENCE: Their voting preferences in the next cycle?

PROFESSOR: Yeah. So it seems that the litmus test here would be voting. Which is after one cycle of reservation, or two cycles of reservation, are women more likely to be elected. And that seems to be that it would be like the place easier to start or at least to end, which is does it make a difference? And along the way, if you find that you can think, well, maybe this is because of various things. Maybe this is women started to develop their networks. Maybe women figured out that they could do this. So if we wanted to know that it's really true, the change in the perception of women as policy makers, you would try and get a measure of perception. Again, more to eliminate the end line result, and to try to get the measure of pure, just one sort of--

AUDIENCE: Could you also-- so there are women elected to these councils.

PROFESSOR: Yeah.

AUDIENCE: Could you also though measure the women's ascendancy to other kinds of more management posts, like at school level, or at community groups or something like that, as stepping stones on the way to that?

PROFESSOR: Right. That goes into this margin on women's participation. So you're thinking [STUDENT NAME] earlier was thinking up, like do we see more women--

AUDIENCE: At higher levels.

PROFESSOR: --going at higher level. Or you could say do we see more women in position of power within the village.

AUDIENCE: Sink or swim.

PROFESSOR: School headmasters, things like that. School council, other things like that. Going back to the perception question, do you have an idea of how we could go about measuring people's perception of women as policymakers?

AUDIENCE: We could have an index of the satisfaction they have with investments that women have made, and try to relate it to how much they like or dislike the woman who decided about this investment?

PROFESSOR: So we could try satisfaction of the goods. Thinking that in particular if we had an objective quality of the goods, we could say that if their satisfaction of the goods is lower even though the goods are the same, it indicates that they like women less. But that requires having a very good measure of the quality of the goods. Otherwise, someone could always say they are worse in some dimension that you didn't observe.

AUDIENCE: We could do some kind of hypothetical question that involves psychology, or you're trying to--

PROFESSOR: So yeah, great. Continue in this direction.

AUDIENCE: Where you're interviewing people or doing surveys, and you asked them to imagine a scenario where you have two different candidates, and you give them the backgrounds and one's a man and one is a woman, and ask them who they would pick. I mean, that would be a poorly disguised question--

PROFESSOR: If you have the two candidates and you ask them to compare, you might get into the same issue. But just go one step further with this same idea. You're almost there. Or at least that's one way of doing it.

AUDIENCE: You could present two hypothetical CVs to people and say, would like to vote for this person. But on one of the hypothetical CVs, make it up as a woman, and the other make it up as a man, but they have all the same qualifications. And just see if there's a sense that people would vote more for one versus the other.

PROFESSOR: Exactly. You are [? interpreting ?] values to different people. Nothing forces you to present the same questionnaire to every person. You could randomize within your questionnaire what you're going to show them. So you're going to show them, for example, exactly the same CV and ask them, are you going to vote for this person? Or you could present a scenario saying, so this and this happened, and there was a choice to be made, and they decided to do this and what do you think? Was it a good decision? And in one case it's like Mr. So and So decided to do this, and in one case Mrs. So and So decided to do this. If you ask both, you're exactly right, it was poorly disguised. But if you ask only one, people will answer the question. And they don't have enough information to really judge the person fully. Especially if you give them a small scenario and then ask them, will you vote for them?

So then we would bring in, presumably, whatever else their other views on the person. So then you will have compare across surveys whether people tend to rank more highly the survey that has Mr. So and So, versus the survey that has Mrs. So and So. The way we did it in the follow up study to the one that is in the case is we actually taped speeches. So we had speeches that someone had given, and we had a bunch of women record the same speech. And we then played a tape and said, so what do you think? So the advantage is, we don't even have to insist on that it's Mr. So and So, it's just would you listen to this speech. And the voice immediately tells you and then you can compare people's answers. And so now it becomes you have to compare people's answers to this question in the villages that were reserved and the villages that were not reserved, and see whether any gap changes.

An interesting fact is that then you can correlate that to what they actual tell you and see whether, for example, it might be that the real gap doesn't change, but what they tell you narrows, or it could be the opposite. If they're trying to signal their dislike of the policy or something like that. And at the end of the day, all of this leads to sort of I'll find a hypothesis of did this work, in a sense. Which would be what are the vote shares for women in politics? Or if you were less ambitious, something like what are the share of women who are represented in positions of power within the village as you suggested earlier.

So that's something for day two because we were on measurement of outcomes. I just wanted to give you one example so we don't lack of various ways we can collect outcomes. That's one way. Going back to the example of do women leaders better represent the preferences of women. So what we try and do with an evaluation is to try to find out whether the program's just, whether the program is effective, but also why it's really more helpful? Both because well, we have a richer understanding, and because it will enrich our understanding of the program and also make it easier to draw more general lessons. Because for example, this program you evaluate in West Bengal and someone can say, well we need to work in Rajasthan. So one way is to go and do it in Rajasthan. But then you can say, well, but it worked in Rajasthan, too. We need to work in South India and it cannot replicate everywhere, everywhere.

So eventually you want to have the ability to say something. It's not necessarily we'll go with some hypothesis, but to say something that what is your take at the end of the day of the reasons why this program will have such and such effect. So if you say, well what I think is what's happening is when women are elected, when there is a reservation for women they are doing stuff that women prefer. Then you can say, well, in West Bengal what they want is water so that's what they'll do. In, say, Tamil Nadu our water's not so important-- well, it is very important to me-- so in any place where it's not so much of an issue, I guess that wouldn't be India, then they'll do that.

So as long as you tell me what women care about, I can tell you that it's going to go in this direction. It has the advantage that it makes the replication more interesting because if you're doing North Bengal and then you say, well now some goods go up, some goods go down. And then say you replicate in Rajasthan a new level of tests that you are willing to subject yourself to. If some goods go up, some goods go down, it's like well, maybe you don't even need to do the evaluation because probably you'll find it. Whereas after the West Bengal study you say, whatever I find from this however imperfect way to collect preferences is what women prefer, this is the same direction it's going to move in Rajasthan.

So, in fact, in here in this case it's exactly what happened. Because we first did West Bengal, and we find that women preferred water, according to the measure of preferences, and goods went to water. And men prefer schools, which was surprising to us and others, but that's the way it is, and the goods went to schools. So now for Rajasthan, we say, well we are going to do the same thing. Find out what women want, what men want in the same way. It's going to go in this direction. And we find that there in Rajasthan, also women prefer water, but men love roads. So we should have less roads. More water, less roads. While in West Bengal, women also likes the road, for the reason that they work on them. They are the people who do the roads, so it's employment activities for them.

So it's interesting because we have different predictions. Our prediction is that the road will go up in West Bengal where you have women reservation, and they will go down in Rajasthan. We know why, because it's related to the thing. And in fact we can make this leap of faith beforehand. And so that makes it much more powerful once you replicate, rather than I'll replicate as I can. To say I will replicate with a good sense of what I'm expecting to find.

Which brings me to the same thing which is our saying it's a very difficult ex-post not to use the data to learn more than what the hypothesis was at the beginning because, you know, it's sad. So you could think about it in a more constructive way, which is you could think well, this is what was my hypothesis was in the beginning, this is what I find. In addition, I have also these interesting, tantalizing tidbits of results. I'm putting it in front if you admitting that it was not my hypothesis to start with, but I am contesting that it should be the hypothesis of the next study.

I'll give you one very good example of that. There was one project by David McKenzie from the World Bank, and Suresh de Mel, who works in Sri Lanka, and Chris Woodruff in UCSD, and they were interested in the return to capitol for very small entrepreneurs. So what they did is they gave people in Sri Lanka a grant. Small entrepreneurs, people who had about $200 of working capital, that's just what we call a helicopter drop of money. So you get $100 grant or $200 grant. And they did that and they found that at first you get the average and they found great returns to capital. Very high returns to capital. So very beneficial to give people of the other 5% percent a month. So very high return to capital. Great.

And then they decided to do it separately at women and men. And oh, surprise, they found no return whatsoever for the women, and huge return for the men. So you can say, sorry, it was not in your original design. It was not stratified by gender, so we have really no intents that you are-- so we have to throw this result away. Of course, we don't want to throw this result away because that's so surprising and striking that we kind of want to think about it. So what's the idea? You write this up being very explicit that we found this ex-post, but it seems like really robust. We are going to go and do a new experiment, so we could redo it in Sri Lanka or do it in somewhere else. And in this case, our hypothesis is the age zero is the return to capital are the same for men and women, that's what you are trying to reject.

So these scores are good to think about. These evaluations are part of a process, we are not alone. A lot of people are working on this, there will be replication either by you or by others. And being explicit up front about what was your first hypothesis and your current model. And was is it that you found out as well? Our goal is to not get mixed up, and at the same time not to lose the information that is going to be useful in the future.

So we stop at 12:00 whenever we started, or what are the social norms in this?

AUDIENCE: As long as you want.

PROFESSOR: Oh. I have a phone call at 12:30, otherwise--

AUDIENCE: It's 12:15.

PROFESSOR: I'd probably be finished before that. I don't have time to finish what's on the slides anyway.

So I just wanted to give you a sense of what might be a causal model in this case. The whole perception and goal is not there, it's the public goods thing. And it's a way of disciplining all of the outcomes, as well as the various things we spoke about. So you start from reservation, so one thing that reservations definitely do is that they will lead to more women Pradhan. And then the question is whether or not having more women Pradhan will change the public good, and in what way? And there are really two channels to change the preferences, which we have discussed. One is the women as the Pradhan do what women want. And you've not really discussed that, but that comes with its own set of assumptions which is on the one hand the Pradhan are not representing the majority. As always, the majority hasn't changed. We shouldn't see a difference because even if you are saying, well, you don't have to be accountable to the men. That's not true, because they still do it for you. Ex-ante.

Several woman compete, and the issue is what platform are you going to run on? Well you should be running on the platform that is going to get you elected. And whether or not you're a man or a woman, you're elected by the same group of people. So in a totally stand out model where democracy is perfect, who is in charge doesn't matter. Because who is in charge is representing the desire of the majority, what we call the median voter. So if you had perfect democracy, that channel would be killed, and we wouldn't see an impact. On the other hand, if all the decisions were made by a group of elite villagers, that again wouldn't matter because who is in charge doesn't matter.

So the identity of the Pradhan is going to make a difference only in some middle-of-the-road kind of [? ward ?] where the politician has some control over what is going on. It is not completely controlled by a bureaucracy or by elite, and is not completely accountable to the people. But who is he? He cannot fully commit, for example, to a platform. That doesn't seem unrealistic. People make electoral promises and sometimes they go against them. Almost never, but sometimes. But that's something. So if we do learn that the public good prefers to change, we have learned something broader than just impact to this program. We have learned something about politics in India, which is OK, there is some democracy. Some people are contesting that. Some people say the Panchayat is just a face. There is no democracy really. So if you find a difference by the identity of the Pradhan, it shows you by the by that there is some reality in the democratic system, but it's not perfect democracy. So if we've learned something, it can be broader than a new program as well. And that is a lesson you can take elsewhere.

Another channel by which a woman having reservation can influence the representation of women is to more representation. For example, we were talking about more political participation of women if the woman is the head. So one thing that could be true is that they're more likely to show up in meetings, that they're more likely to speak up. For example, because the women have said that the Pradhan has to be at the village meeting, so she better put it at the time where she can go. So not in the middle of the night in a field. And so as long as she can go, a woman can also go.

So to either of these channels, you'd have the fact that the public good will reflect better women's preferences. We have to add another assumption, is that women have different preferences. And if that's the case, then the public good will be different in a specific way, which is towards those different preferences. And then you might have different outcomes. More income for the women, better health and education outcome, if it comes out that it's what women care about. And you'll be able to follow exactly the trace that, you know, if you find like in West Bengal more water. Maybe you're going to be interested in diarrhea. If you find less schools, you're going to be interested in education to see whether education goes down, and things like that.

So now we have the complete channel. And we can now think about all of our variables that we had collected, and we're going to slot them into, well, what are they going to do for us? So all the public goods go here. We need to collect women's preferences somewhere, it will go here. We want to know as a woman are empowered, so we are going to be collecting all of these. Whether people come to the meeting, whether they speak up, how they are answered to once they speak up. This is all going to go here. So now, if we have an infinite amount of money we're still going to collect a large amount of data, but we know in advance what it is we are going to do with them. We can write it down, put it in an envelope, send it to your grandmother, and this is the thing that really gives a lot of credibility to what you're going to do. Another version of sending it to your grandmother that we are going to try and implement here at J-PAL is to allow you to put it on a website, to upload it somewhere where nobody can see it but you, but it's secure, and the data is mapped. So this is whatever it was your analysis plan at the time of the beginning of your study. So you are tying your hand behind your back.

AUDIENCE: Excuse me, but why would you do that?

PROFESSOR: Because we want to-- or maybe someone can answer that question. Why do we do that?

AUDIENCE: I guess it's just so people aren't relying on your personal integrity. You're saying this was our hypothesis. We didn't ex-post change our hypothesis.

PROFESSOR: Right. So the reason why you want to say what was your hypothesis in advance, is because then you can attest it. Whereas if you take something you can always reverse engineer it. And again, I have no big issue with that, with reverse engineering. Personally, I think it's useful, but it needs to be very clear what was there before and what was reverse engineered after, otherwise we have no notion of statistical tests.

AUDIENCE: I'm just pushing out a little bit further, too. What if you just had a question where you just don't know, right? You're like, wow, we really think there might be an effect of this on that, or we're not sure whether the effect would be upward or downward. Do you still have to just pick one for the sake of having a hypothesis that you're testing?

PROFESSOR: I think you wouldn't want to embark in an evaluation without at least having a sense of why it would go up and why it would go down. So take this specific example. To start with, you shouldn't really know whether the water wells are going to go up and down, because it's going to depend on what women want. So I'm not making a stance. It would be silly to write to my grandmother. I'm betting that the women water wells will go up. Because they could go up or down depending on what women want. Of course, you may have a very strong prior that it's what women want, but the statement would be of the form, if it is the case that women have a strong preference for water, water should go up.

So if you have some uncertainty, it's probably because there is an if somewhere that you're not thinking about that you're expliciting now. I think if thinking sufficiently hard about something, you can know in what condition you would go up and down. There are a lot of programs which could go up and down, that's why we evaluate them. I mean, sometimes you think that they should really go up, but it could also be zero. And it's good to know, if this and this happened then this effect would be expected. If this and this doesn't happen, then I wouldn't see this effect, so you could write that. And in fact, I think these types of statements, in a sense, are almost more informative than this will happen.

So let me stop here. What's in the rest of the slides is kind of little bit random-- not randomized, but not random in the sense of randomized, but random in the sense of every which way I pass out the advice on how to collect data and how to enter data and things like that. For those of you who are going to IPA training after this, you're going to be sick of it by the end of the three days so that's not needed. For the other ones, it's in the slides and it's really-- that's the problem of it being too short anyway. So it is pretty self-explanatory and relatively common sense and not sufficient anyway, but a starting point. Thank you very much.