Okay, In this video we're going to talk about another problem of low-powered studies. And that is that low-powered studies tends to inflate effect sizes, which is obviously something undesirable. We're going to consider two different cases in the course this video. One is, or first we're going to consider the case when the null hypothesis is false. In other words, in cases where there truly is an effect to detect. And then we'll also consider a case where the null hypothesis is true. So when there actually isn't any effect to detect in the first place, okay? The phenomenon that we're going to describe is called the winner's curse. And the winner's curse involves getting inflated effect sizes among statistically significant effects. So importantly, this inflation of effect size is only occurs when we use some particular thresholds to decide whether or not there's an effect present. So for example, the common practice of using a critical p-value save 0.05 to decide whether or not there's an effect present. That kind to practice leads to these inflated effect sizes among our significant results. Okay. This just highlights another reason for why. Using a threshold value, a threshold p-value to decide on significance is, is a bad idea, as we talked about in other videos. Now, importantly, this phenomenon, or the winner's curse, is most prominent in low-powered studies. So the extent to the winner's curse becomes greater and greater as our power decreases. Okay, so why is that the case? Why is that the winner's curse becomes more prominent in low-powered studies. Let's imagine that we had an experiment that was very low-powered. In order to effect, Yes, sorry, in order to detect an effect of a given size. In that case, it's very unlikely for that experiment to detect that effect unless by random chance. And by that I mean by unless due to sampling error. The samples that we use happen to yield a large effect. Because it's only large effects that a lobe Howard study will be able to. It's only with large effects that a low-powered study we will be able to detect a significant result. Okay? So that's the main reason you, how, how prominent are these? Is this inflation of effect size. Well, I've pulled this figure here from this very nice paper which you've talked about in other videos. I apologize for the strange font here. Something strange happened when I copied and pasted this figure from the original paper. What this figure shows is the relative bias. Research find Dina as a percent against the statistical power of the study. And these results are based on simulations that are performed by these authors. In what you can see is that as the statistical power decreases, the bias increases. Excuse me. The authors have highlighted these two points on this figure because these represent the range of what they found to be the plausible range of typical statistical power for experiments in neuroscience. So this, I believe corresponds to about 8%, and this corresponds to around 31%. This range of power is also what's typical in ecology and evolution for low to medium powered studies. So I believe for low-powered studies, we, sorry, not for low-powered. I meant this is also the range of the power that we have. Any ecology evolution for studies that aim to detect small effects. Sorry, what I said a moment ago was nonsense and I should have said small effects. Okay? So with about 31% power, we get about a 25 percent increase in the effect size. Whereas with about percent power, we get about a 50% increasing effect size. That's astonishing, okay, and that's, that's something that we really need to be aware of when we're reading the literature and when we're performing our experiments. That's actually the main point for this video. So if you want to stop watching the video there, then feel free. What I'm going to do next is I'm actually going to demonstrate the winner's curse using some scripts in R, which I'll create on the fly so you can see how it's done. Okay? So if you're not interested in seeing that, you can just stop the video. Now you've gotten the main point. The main point is that low-powered as we, the main point is this. As our power decreases, the amount of bias in our results among statistically significant findings will increase. Okay, now, let's demonstrate that. So to do that, we're going to go into are. And what we're gonna do is we're going to create two data sets using a random number generator. And to do that, we need to specify a few variables. We need to specify a sample size. And I'm going to specify two different sample sizes. One I'll call n small, which we'll say is five. And another one, which we'll say is enlarge and we'll say that's a 100. Okay? So I'm saying that my small sample size be five and my large sample size to be a 100. We also need to specify amount of variation in our data. Okay, so I'm going to say the standard deviation. I'm going to say this is true for both of the scenarios that we're going to consider are called the standard deviation of. Both. Let's set that to I don't know. Let's set it to five as well. Okay? Multiples of fiber friendly. And now we want to choose some sort of effect size. Okay? So let's imagine we're going to do that by specifying two mean values. Okay? So we'll say with a mean value of one group, which we'll call mean one. And we'll say it has a mean value of 0. And for other group, call that, say the mean values equal to, let's say five. Okay, so we have lots of fives in our center scenario. Okay? So what we're gonna do is we're going to generate two sets of data. Call the first one, data one. And we're going to use a random number generator, which we're going to use our norm. And we're going to use it to generate as many random numbers as we have samples. Okay, So the way we use this is first we specify the number of samples are random number of numbers the, that we want to draw. And so let's say that that's equal to n small. So we're going to draw, in this case, five random numbers because that's our sample size. And we want this to have a mean of group 1. Sui say mean one. And then we specify the amount of variation. So we'll say SD, both. Okay, So let's just show how this works. Oops, I need to tell R all of these things. Let's just highlight all this will just show you how this works. There we go. We've now gotten five random numbers that were drawn from a normal distribution with a mean of equal to mean one which is 0, and a standard deviation of five. Okay? We're going to repeat that. So those the data we generated for one of our groups, sorry, I should've said this earlier. We're going to demonstrate the winner's curse using a t-test. So we're going to be comparing the mean values of two different groups. So I have not mentioned that earlier. That is an important thing to say. So now all I'm gonna do here is you're doing the exact same thing as we did above, except we are now saying that the mean value For this group will be equal to mean 2. So in this case the mean will be equal to 5 for this group. Okay? Now, what we're gonna do is we're going to run a t-test. And we're gonna save the output in an object which we call t out. And we're going to run a t-test or we're going to set it up where we can just specify the data in our two groups by listing them. So data dot one and data, whoops, don't 2. Okay. So let's just run all this and say what we get. So first of all, our data and group one is that our data and group two is that and our t-test will have they will perform a t-test comparing these data to those data. And we're going to store that output in T out. Okay? And here's the output and you can see that we can obtain our p-value here. Okay? He wants to obtain that p-value because we want to identify the cases where we have a significant result. So we can do that by specify a going to this, this object which contains our output. And so if you say T dot out and we say dollar sign, we can say p-value. And now pull that, we're pulling out only the p-value from this output up here, okay, so the p-value is 0.205. That's been rounded. Cuz you can see the more precise p-value is 0.2049984. Okay? Now what we're gonna do is we're setting up our, our simulation in such a way that when our p-value is less than 0.05, you want to store the effect size. So I'm gonna go back up here because we're going to create a vector where we're going to store our effect sizes. And so we'll call this effect size is small. This is for the small, for the small sample size. And we're going to start out with it being empty. Okay, so we're just going to do that by saying it's null to show you that it's empty. There's nothing in there. Okay? Now if you want to store a mean value in there, we can say, let's say we want to store the effect size. Then what we can do is we can say effect size small. What we're gonna do is we're gonna take our original vector, which is, which is currently empty. And we're going to add to it the difference between the mean of this data set and the mean of that data set. Okay? So to take this vector, an add on, this value represents the difference between our means. We're just going to, whoops, we're just going to go like this. We're going to first of all use our C function. And we're first just going to list the vector that we're starting with. And we're just going to attach to that the difference between our two different means. And just to make things a little bit clear, we're going to take the absolute value of that difference because it's possible for one means be larger than the other. So you can either have group 1 be larger or smaller than the other, and we don't really care which way round it goes. We just wanted about the absolute. Value the effect size. So I'm going to say absolute value of the mean of data one minus the mean of data to. Okay? Now let's just prove to ourselves this works properly. So let's say the, sorry, the mean of data one minus the mean of data two is equal to this. Okay? So now if we run this, what we should find is we'll end up adding or end up putting the effect size of positive because we've taken the absolute value positive 5.146 and placed it into this vector. Okay? So now if we say, there we go, okay? So that's what's going on here. What we'd like to do is we only wants to store our effect sizes. If we have a p-value that's less than 0.05, okay? Because that's the conditions in which the winner's curse operates. It's on the operates when we're deciding whether or not something is significant based on some threshold value. Okay, so to do that, we're going to say if our p-value is less than 0.05, then we want this to happen. Okay? So in other words, we're setting up where we've set up this IF statements so that only in cases where the p-value that results my t-test is significant, where we actually store our effect size in this, in this vector. That's what I'm looking for. Okay. So let's just I was just going to say let's, let's just run this. But we've already run all the bits, which is why I've stopped and pause me and post. What we want to do now is we want to repeat this process many times. Let's say let's run it 1000 times. So to do that, we're going to use a for loop. So I'm going to say for i in one to 1 thousand. So this code allows us to run the code, or this code allows us to run the code that we're going to put within these curly brackets, 1000 times. And so guess what? We're going to run this code 1000 times. Okay? So that's how we're going to obtain our data, okay, for only the small sample size. In order to demonstrate that the winner's curse becomes more prominent when we have small sample, when we have a small sample size, we want to compare the output that we're going to get here were the case we have a large sample size. So to do that, I'm just going to repeat all this. I'm just going to copy this code and put it down here. But I'm going to change the vector. Call that vector, call at large, that large, that large. And the other thing that we need to change is we need to change the sample size to a change in the number of samples that we draw from our normal distribution. Otherwise, the code here for this halving when we have a large sample size is identical to the code. For the case we have a small sample size, okay? And so we're going to be storing our effect sizes with a small sample size and this vector AD with a large sample size in this vector. Okay? Just making sure I've done this appropriately. Okay? So now just thinking ahead, what we'd like to do is we eventually want to plot our data. So I'm going to plot the data using a boxplot. So what I'm gonna do is I'm gonna take my, the two vectors that the container effect sizes and I'm just going to add them to or stick them together to create one long vector that will contain all of our effect size. I'm going to call that E size for effect size. And so we're just going to combine these, say effect size small and then a fixed size large. Ok. And now created another variable called sample size. This is going to be a variable that lets us know which type of experiments these, these values come, came from. Okay? So what we're gonna do is we're going to use a repeat function. So we're going to, we're going to repeat the text. Small, so small n. And we're going to repeat it as many times as we have observations in our vector that contains our effect sizes from a small sample size. And so we can determine the number of the number of data points in this vector by using the function length. So what I'm doing here is I'll just highlight this to show you what I'm doing is actually i'm, I'm not going to highlight this quite yet. I'll show you, I'll show you after this has run, I'll show you how this works. Okay? Now I'm just going to repeat this again. I'm going to repeat my repeat. But I now want to say large and large. Okay? And then finally, what we're going to do is we're going to create a box plot. And we're going to put as our Y variable in our box plot the effect size. And we're just going to plot that against, where is my tilda against sample size. Okay? So can I just highlight all of this and run it? And here's that we get, which you can see here. Remember that the true effect size was equal to five because a true difference between our groups is five, as we've indicated here. When we have a large sample size, typically we end up getting an effect size that is around the true value. Okay, So we've been, we've been able to estimate that relatively well. In when we have a small effect size, sorry, when we have a, I might have misspoken when we have a large sample size. So when we have a high-powered study, you can see that the effect size that we infer when we have a large sample size is pretty close to the true value. Okay? I would not be surprised if the mean of this is slightly higher than, than that, because that's what we would predict from the winner's curse, okay? In fact, we'll check that in a moment. What I really want you to see, however, is that when we have a small sample size, the effect size that we estimate is much, much greater. Okay? It's somewhere up around 8.5 or nine typically. And so that's almost, we've almost doubled the true effect size by having experiments with such a small sample size. Okay. I just want to show you, go back to things that I promised that I would do and I want to show you how this statement works. Oops, I forgot a bracket. There we go. Oh, stop that. There we go. Okay. So we had a number of cases. So according to this, we had 17980009192. We had a 100 or 292 effect sizes in our small when we had a small sample size. Okay, apologies if I've swapped the term effect size and sample size and number of times in this video. Using something that a term where everything ends with the word size gets to play with the mind a little bit. Okay. I would like you to point out, so here what I'm what I'm trying to show you here. So i'm, I'm waffling a little bit. We've printed the term small n 290 two times, I believe I've counted that right? So how big is our vector? 292? Okay, so that's how that code works. What we'd like to do now, just to test whether or not this is slightly larger than the true effect size. Is. Let's go for the mean. Effect size. Large, 0, slightly smaller. I didn't expect that. So the winner's curse when it's talking about this bias. This is a bias that we expect to occur on average. Okay? So that does not mean it's going to happen every time. It means that there's going to be a tendency for this to happen. In this particular example, we got our average effect size is almost bang on if anything, ever so slightly smaller than the true effect size. This is one of the times where by chance it might end up being lower. But on average, we expect the effect size, but a large sample size to still be larger than the true effect size. That's pretty much what I wanted to show you in our first case. Case, what we've demonstrated here is that when you have a small sample size, so a low powered study, we end up, or we can end up drastically over estimating the effect size. What I want to point out now in a very simple way is that we get the same result even if there's no effect. Okay, So even if the null hypothesis is true, and we can do that with the exact same code. All I've done, however, is I've changed the mean value of the second group from being fired being 0. So that now the mean value for each of the groups truly is identical. Okay? So let's run this. And here we go. Okay? What you can see here is that the effect size once again, is much larger for the situation where we have a small sample size compared to have a large sample size. And of course, in this case, we can see the winner's curse occurs when you have a large sample size because the true sample size should be 0. Yet, the typical effect size when we have a large sample size is somewhere close to maybe about 1.5. So in this example, we do see the winner's curse coming to fruition and when we have a large sample size, okay. That's it I wanted to show you. Okay, so what we've demonstrated is regardless of whether or not there is actually an effect in an experiment, regardless of whether or not the null hypothesis is true or false. Small, small sample sizes or low power will tend to inflate our effect sizes. So let's wrap this up. Okay. This is what I'd like to end with from this video. I want you to recognize now that if someone comes up to you and says, Hey, look at this result, my experiment is really small, so it was low-powered, but it's still detected this big affect, my results must be really important. Now you can say, actually, that's exactly what we expect to happen with a low powered study. Even if there's no effect at all. Sorry, I'm fat. I'm going to end the video and say hope it's been helpful. And say, thank you very much.