Correct foundational estimands, randomization, and p-values

causal-inference-for-the-brave-and-true/01-Introduction-To-Causality.ipynb

@@ -40,7 +40,7 @@
     "\n",
     "ML is notoriously bad at this inverse causality type of problem. They require us to answer \"what if\" questions, which economists call counterfactuals. What would happen if I used another price instead of this price I'm currently asking for my merchandise? What would happen if I do a low sugar one instead of this low-fat diet I'm in? If you work in a bank, giving credit, you will have to figure out how changing the customer line changes your revenue. Or, if you work in the local government, you might be asked to figure out how to make the schooling system better. Should you give tablets to every kid because the era of digital knowledge tells you to? Or should you build an old-fashioned library? \n",
     "\n",
-    "At the heart of these questions, there is a causal inquiry we wish to know the answer to. Causal questions permeate everyday problems, like figuring out how to make sales go up. Still, they also play an essential role in dilemmas that are very personal and dear to us: do I have to go to an expensive school to be successful in life (does education cause earnings)? Does immigration lower my chances of getting a job (does immigration causes unemployment to go up)? Does money transfer to the poor lower the crime rate? It doesn't matter the field you are in. It is very likely you had or will have to answer some type of causal question. Unfortunately for ML, we can't rely on correlation-type predictions to tackle them.\n",
+    "At the heart of these questions, there is a causal inquiry we wish to know the answer to. Causal questions permeate everyday problems, like figuring out how to make sales go up. Still, they also play an essential role in dilemmas that are very personal and dear to us: do I have to go to an expensive school to be successful in life (does education cause earnings)? Does immigration lower my chances of getting a job (does immigration cause unemployment to go up)? Does money transfer to the poor lower the crime rate? It doesn't matter the field you are in. It is very likely you had or will have to answer some type of causal question. Unfortunately for ML, we can't rely on correlation-type predictions to tackle them.\n",
     "\n",
     "Answering this kind of question is more challenging than most people appreciate. Your parents have probably repeated to you that \"association is not causation\", \"association is not causation\". But actually, explaining why that is the case is a bit more involved. This is what this introduction to causal inference is all about. As for the rest of this book, it will be dedicated to **figuring out how to make association be causation**.\n",
     "\n",
@@ -125,20 +125,20 @@
    "source": [
     "To get beyond simple intuition, let's first establish some notation. This will be our everyday language to speak about causality. Think of it as the common tongue we will use to identify other brave and true causal warriors, and that will compose our cry in the many battles to come.\n",
     "\n",
-    "Let's call $T_i$ the treatment intake for unit i. \n",
+    "For the tablet example, each unit is a school. Let's call $T_i$ the treatment intake for school $i$. \n",
     "\n",
     "$\n",
     "T_i=\\begin{cases}\n",
-    "1 \\ \\text{if unit i received the treatment}\\\\\n",
+    "1 \\ \\text{if school } i \\text{ received the treatment}\\\\\n",
     "0 \\ \\text{otherwise}\\\\\n",
     "\\end{cases}\n",
     "$\n",
     "\n",
-    "The treatment here doesn't need to be a medicine or anything from the medical field. Instead, it is just a term we will use to denote some intervention for which we want to know the effect. In our case, the treatment is giving tablets to students. As a side note, you might sometimes see $D$ instead of $T$ to denote the treatment.\n",
+    "The treatment here doesn't need to be a medicine or anything from the medical field. Instead, it is just a term we will use to denote some intervention for which we want to know the effect. In our case, the treatment is a school providing tablets to its students. As a side note, you might sometimes see $D$ instead of $T$ to denote the treatment.\n",
     "\n",
-    "Now, let's call $Y_i$ the observed outcome variable for unit i.\n",
+    "Now, let's call $Y_i$ the observed outcome variable for school $i$.\n",
     "\n",
-    "The outcome is our variable of interest. We want to know if the treatment has any influence in it. In our tablet example, it would be the academic performance.\n",
+    "The outcome is our variable of interest. We want to know if the treatment has any influence on it. In our tablet example, it is the school's average academic performance.\n",
     "\n",
     "Here is where things get interesting. The **fundamental problem of causal inference** is that we can never observe the same unit with and without treatment. It is as if we have two diverging roads and we can only know what lies ahead of the one we take. As in Robert Frost poem:\n",
     "\n",
@@ -153,15 +153,15 @@
     "\n",
     "As for the notation, we use an additional subscript:\n",
     "\n",
-    "$Y_{0i}$ is the potential outcome for unit i without the treatment. \n",
+    "$Y_{0i}$ is the potential outcome for school $i$ without the treatment. \n",
     "\n",
-    "$Y_{1i}$ is the potential outcome for **the same** unit i with the treatment.\n",
+    "$Y_{1i}$ is the potential outcome for **the same** school $i$ with the treatment.\n",
     "\n",
-    "Sometimes you might see potential outcomes represented as functions $Y_i(t)$, so beware. $Y_{0i}$ could be $Y_i(0)$ and $Y_{1i}$ could be $Y_i(1)$. Here, we will use the subscript notation most of the time.\n",
+    "Sometimes you might see potential outcomes represented as functions $Y_i(t)$, so beware. $Y_{0i}$ could be $Y_i(0)$ and $Y_{1i}$ could be $Y_i(1)$. Here, we will use the subscript notation most of the time. The subscript $i$ refers to a particular school. When we omit it, $Y_0$ and $Y_1$ denote the potential-outcome random variables for a school drawn from the population, so expectations such as $E[Y_0]$ average over schools.\n",
     "\n",
     "![img](./data/img/intro/potential_outcomes.png)\n",
     "\n",
-    "Back to our example, $Y_{1i}$ is the academic performance for student i if he or she is in a classroom with tablets. Whether or not this is the case, it doesn't matter for $Y_{1i}$. It is the same regardless. If student i gets the tablet, we can observe $Y_{1i}$. If not, we can observe $Y_{0i}$. Notice how in this last case, $Y_{1i}$ is still defined, we just can't see it. In this case, it is a counterfactual potential outcome.\n",
+    "Back to our example, $Y_{1i}$ is the average academic performance at school $i$ if that school provides tablets. Whether or not the school actually does so, $Y_{1i}$ is still defined. If school $i$ provides tablets, we observe $Y_{1i}$; otherwise, we observe $Y_{0i}$. In the latter case, $Y_{1i}$ is the counterfactual potential outcome that we cannot see.\n",
     "\n",
     "With potential outcomes, we can define the individual treatment effect:\n",
     "\n",
@@ -171,11 +171,13 @@
     "\n",
     "$ATE = E[Y_1 - Y_0]$\n",
     "\n",
-    "where, `E[...]` is the expected value. Another easier quantity to estimate is the **average treatment effect on the treated**:\n",
+    "where `E[...]` is the expected value over the population of schools. Another easier quantity to estimate is the **average treatment effect on the treated**:\n",
     "\n",
     "$ATT = E[Y_1 - Y_0 | T=1]$\n",
     "\n",
-    "Now, I know we can't see both potential outcomes, but just for the sake of argument, let's suppose we could. Pretend that the causal inference deity is pleased with the many statistical battles we fought and has rewarded us with godlike powers to see the potential alternative outcomes. With that power, say we collect data on 4 schools. We know if they gave tablets to its students and their score on some annual academic tests. Here, tablets are the treatment, so $T=1$ if the school provides tablets to its kids. $Y$ will be the test score."
+    "The vertical bar $|$ reads as \"given\" or \"conditional on.\" Thus, the $ATT$ is the expected treatment effect among schools that received the treatment ($T=1$).\n",
+    "\n",
+    "Now, I know we can't see both potential outcomes, but just for the sake of argument, let's suppose we could. Pretend that the causal inference deity is pleased with the many statistical battles we fought and has rewarded us with godlike powers to see the potential alternative outcomes. With that power, say we collect data on 4 schools. We know whether each school provided tablets to its students and its average score on an annual academic test. Here, tablets are the treatment, so $T=1$ if the school provides tablets to its students. $Y$ is the school's average test score."
    ]
   },
   {
@@ -293,11 +295,11 @@
     "\n",
     "$ATE=(-50 + 0 - 200 + 50)/4 = -50$\n",
     "\n",
-    "This would mean that tablets reduced the academic performance of students, on average, by 50 points. The $ATT$ here would be the mean of the last column when $T=1$:\n",
+    "This would mean that tablets reduced schools' average academic performance by 50 points. The $ATT$ here would be the mean of the last column when $T=1$:\n",
     "\n",
     "$ATT=(- 200 + 50)/2 = -75$\n",
     "\n",
-    "This is saying that, for the schools that were treated, the tablets reduced the academic performance of students, on average, by 75 points. Of course we can never know this. In reality, the table above would look like this:"
+    "This says that, among the treated schools, tablets reduced average academic performance by 75 points. Of course we can never know this from observed data alone. In reality, the table above would look like this:"
    ]
   },
   {
@@ -411,23 +413,23 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This is surely not ideal, you might say, but can't I still take the mean of the treated and compare it to the mean of the untreated? In other words, can't I just do $ATE=(600+750)/2  - (500 + 600)/2  = 125$? Well, no! Notice how different the results are. You've just committed the gravest sin of mistaking association for causation. To understand why let's look into the main enemy of causal inference.\n",
+    "This is surely not ideal, you might say, but can't I still take the mean of the treated and compare it to the mean of the untreated? In other words, can't I just compute $(600+750)/2  - (500 + 600)/2  = 125$? We can compute it, but we cannot call it the $ATE$ without further assumptions. Notice how different it is from both the $ATE$ and $ATT$ above. We have committed the gravest sin of mistaking association for causation. To understand why, let's look into the main enemy of causal inference.\n",
     "\n",
     "## Bias\n",
     "\n",
     "Bias is what makes association different from causation. Fortunately, it can be easily understood with our intuition. Let's recap our tablets in the classroom example. When confronted with the claim that schools that give tablets to their kids achieve higher test scores, we can refute it by saying those schools will probably achieve higher test scores anyway, even without the tablets. That is because they probably have more money than the other schools; hence they can pay better teachers, afford better classrooms, etc. In other words, it is the case that treated schools (with tablets) are not comparable with untreated schools. \n",
     "\n",
     "Using potential outcomes notation, we would say that $Y_0$ of the treated is different from the $Y_0$ of the untreated. Remember that the $Y_0$ of the treated **is counterfactual**. We can't observe it, but we can reason about it. In this particular case, we can even leverage our understanding of how the world works to go even further. We can say that, probably, $Y_0$ of the treated is bigger than $Y_0$ of the untreated schools. That is because schools that can afford to give tablets to their kids can also afford other factors that contribute to better test scores. Let this sink in for a moment. It takes some time to get used to talking about potential outcomes. Reread this paragraph and make sure you understand it.\n",
     "\n",
-    "With this in mind, we can show with elementary math why it is the case that association is not causation. Association is measured by $E[Y|T=1] - E[Y|T=0]$. In our example, this is the average test score for the schools with tablets minus the average test score for those without them. On the other hand, causation is measured by $E[Y_1 - Y_0]$.\n",
+    "With this in mind, we can show with elementary math why association is not generally causation. Association is measured by $E[Y|T=1] - E[Y|T=0]$. In our example, this is the average test score for the schools with tablets minus the average test score for those without them. Causal effects can target different populations: the $ATE$, $E[Y_1-Y_0]$, averages over all schools, while the $ATT$, $E[Y_1-Y_0|T=1]$, averages over treated schools. The decomposition below relates the observed association to the $ATT$.\n",
     "\n",
-    "Let's take the association measurement and replace the observed outcomes with the potential outcomes to see how they relate. For the treated, the observed outcome is $Y_1$. For the untreated, the observed outcome is $Y_0$.\n",
+    "Let's take the association measurement and replace the observed outcomes with the potential outcomes to see how they relate. We use the **consistency** relationship $Y=TY_1+(1-T)Y_0$: for a treated school the observed outcome is $Y_1$, and for an untreated school it is $Y_0$.\n",
     "\n",
     "$\n",
     "E[Y|T=1] - E[Y|T=0] = E[Y_1|T=1] - E[Y_0|T=0]\n",
     "$\n",
     "\n",
-    "Now, let's add and subtract $E[Y_0|T=1]$. This is a counterfactual outcome. It tells what would have been the outcome of the treated, had they not received the treatment.\n",
+    "Now, let's add and subtract $E[Y_0|T=1]$. Adding and subtracting the same quantity leaves the expression unchanged, but lets us regroup it into a causal-effect term and a selection-bias term. This quantity is counterfactual: it is the expected outcome the treated schools would have had without treatment.\n",
     "\n",
     "$\n",
     "E[Y|T=1] - E[Y|T=0] = E[Y_1|T=1] - E[Y_0|T=0] + E[Y_0|T=1] - E[Y_0|T=1]\n",
@@ -481,13 +483,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now that we understand the problem let's look at the solution. We can also say what would be necessary to make association equal to causation. **If $E[Y_0|T=0] = E[Y_0|T=1]$, then, association IS CAUSATION!** Understanding this is not just remembering the equation. There is a strong intuitive argument here. To say that $E[Y_0|T=0] = E[Y_0|T=1]$ is to say that treatment and control group are comparable before the treatment. Or, when the treated had not been treated, if we could observe its $Y_0$, its outcome would be the same as the untreated. Mathematically, the bias term would vanish:\n",
+    "Now that we understand the problem, let's look at the solution. If $E[Y_0|T=0] = E[Y_0|T=1]$, treated and untreated schools are comparable under the control condition. Equivalently, the treated schools' average counterfactual outcome without tablets equals the untreated schools' observed average outcome. This makes the bias term vanish, so the association identifies the **causal effect on the treated**, the $ATT$:\n",
     "\n",
     "$\n",
     "E[Y|T=1] - E[Y|T=0] = E[Y_1 - Y_0|T=1] = ATT\n",
     "$\n",
     "\n",
-    "Also, if the treated and the untreated only differ on the treatment itself, then, $E[Y_0|T=0] = E[Y_0|T=1]$ and we have that the causal impact on the treated is the same as in the untreated (because they are very similar).\n",
+    "This equality alone does not imply that the treatment effect is the same for treated and untreated schools. In particular, $E[Y_1|T=1]$ could differ from $E[Y_1|T=0]$. It therefore identifies the $ATT$, but not necessarily the population-wide $ATE$. Using consistency makes the derivation explicit:\n",
     "\n",
     "$\n",
     "\\begin{align}\n",
@@ -497,13 +499,13 @@
     "\\end{align}\n",
     "$\n",
     "\n",
-    "In this case, the **difference in means BECOMES the causal effect**:\n",
+    "In this case, the **difference in means becomes the causal effect on the treated**:\n",
     "\n",
     "$\n",
     "E[Y|T=1] - E[Y|T=0] = ATT\n",
     "$\n",
     "\n",
-    "Additionally, if the treated and the untreated only differ on the treatment itself, we also have $E[Y_1|T=0] = E[Y_1|T=1]$, that is, we make sure that both treated and control groups respond similarly to the treatment. Now, besides being exchangeable prior to the treatment, treated and untreated are also exchangeable **after the treatment**. In this case, $E[Y_1 - Y_0|T=1]=E[Y_1 - Y_0|T=0]$ and\n",
+    "To identify the $ATE$ as well, we also need comparability under treatment: $E[Y_1|T=0] = E[Y_1|T=1]$. When both potential-outcome equalities hold, treatment assignment does not change the mean of either potential outcome. Then the average effect is the same in treated and untreated schools, $E[Y_1 - Y_0|T=1]=E[Y_1 - Y_0|T=0]$, and\n",
     " \n",
     "$\n",
     "E[Y|T=1] - E[Y|T=0] = ATT = ATE\n",
@@ -522,7 +524,7 @@
     "\n",
     "![img](./data/img/intro/anatomy2.png)\n",
     "\n",
-    "In the right plot, we depicted the bias that we’ve talked about before. We get the bias if we set everyone to not receive the treatment. In this case, we are only left with the $T_0$ potential outcome. Then, we see how the treated and untreated groups differ. If they do, something other than the treatment is causing the treated and untreated to be different. This something is the bias and is what shadows the actual treatment effect.\n",
+    "In the right plot, we depicted the bias that we’ve talked about before. We get the bias if we set everyone to not receive the treatment. In this case, we are only left with the $Y_0$ potential outcome. Then, we see how the treated and untreated groups differ. If they do, something other than the treatment is causing the treated and untreated to be different. This something is the bias and is what shadows the actual treatment effect.\n",
     "\n",
     "Now, contrast this with a hypothetical situation where there is no bias. Suppose that tablets are randomly assigned to schools. In this situation, rich and poor schools have the same chance of receiving the treatment. Treatment would be well distributed across the tuition spectrum.\n",
     "\n",