Okay, In our previous video, we discussed three factors that determine the power of an experiment. The first is the sample size. The second is the amount of variation that we expect to have within our data. And the third is the effect size that we seek to be able to detect or understand in our experiment. When we're designing an experiment, it's very wise to go about the process of performing a power analysis for reasons we've either already discussed or will discuss in future videos. And so when we sit down to conduct a power analysis will already have at least one experimental design in mind. Perhaps you might have several experimental designs in mind. And we want to determine the relative power of those different, there's different approaches. The point though is we'll already have at least one experimental design in mind. And our goal is to determine what sample size we will need for that given design in order to achieve a certain level of power, statistical power. In order to do that, in order to perform a power analysis, we need, I've information about the other two factors that determine power. We need to have some sense a priori of the amount of variation that we expect to find within our data. And we need to have a good sense of a reasonable effect size to try to detect, given the biological question that we hope to answer with our experiments. The goal of this video is to provide some advice on obtaining reasonable values of standard deviation and to discuss the kind of things that we might consider when trying to settle on an appropriate effect size for your expense. I'm trying to give you guidance on deciding these particular values which you then can use. During a power analysis. We're going to start by talking about the variation and we expect to find within our data. And I've just terms this inherent variation. I could have just called this variation or variance. We can measure this variation in a number of ways. We might measure in terms of variance, or we might measure it in terms of standard deviation. I'm just going to talk about this variation in terms of standard deviation. So that's, I've abbreviated that is SD, either with or without periods here. Okay. So we're going to start by talking about how we can find information that would inform our sense of an appropriate standard deviation to consider in a power analysis. What I'm referring to is the variation that we might expect to find within groups if we were doing say, a t-test. So if we're comparing two different groups in order to, in order to perform a power analysis for an experiment to be analyzed, the t-test, we'd want to know how much variation that would be within each of the two groups that we seek to compare. As one example, if we were doing something like a one-factor general linear model, where we are comparing three different groups than we'd want to know about the amount of variation we expect to find within each of those groups. Alternatively, if we were fitting a line through our data, then we might be talking about that. Then in that case, we'd want to know about the amount of variation we expect to find around that line that we're fitting through our data. So that's what I mean when I'm talking about the understanding the variation in our data, okay? That's who want to go to quantify in order to perform a power analysis. The first question to ask yourself when trying to find a reasonable expectation for this variation is, are you the first person to ever study this system that you're using? Usually the answer is going to be no. Very often, scientists perform repeated experiments using the same kinds of biological material or the same biological system. So if you are working at a system that other people have worked in before, and the first thing to do is just go check the literature. When you do that though, I want to remind you of a number of different ways that we could calculate our standard deviation. First of all, I'll remind you the standard deviation is just equal to square root of the variance. So if the authors report variance and standard deviation is standard deviation that you want, then you can just take their estimate of the variance, take the square root of it, and that gives you the standard deviation. Alternatively, it's very common for authors to publish estimates of standard error. And I want to remind you the standard error is calculated as the standard deviation divided by the square root of the sample size. As a result, if you know your standard error and if you know the sample size, then you can calculate the standard deviation as the product of the standard error times the square root of the sample size. Sometimes this information won't be given to you in texts, but instead it might be given or presented in the form of a figure. So this figure here presents mean values with standard errors. So this arrow is just point you to this value at this bar here, which represents a standard error bar. If we wanted to determine standard deviation that was associated with each of these three groups, then we would look in the methods section to determine what the sample size was in each of these. And then we'd also want to be able to obtain a good measure of the length of these bars. There are a number of software programs. Sorry, there's there's a variety of software is what I mean to say that can help you obtain data from figures like this. So I've used the program data FEF, which is freely available web. A plot digitizer is also recommended for this purpose, quintessence graph grabber is another recommended product and there are a number of other programs available to you as well. The point This video is nots to give you a tutorial and how these various resources work. But to point out they exist so that if you wanted to get a good estimate of the length of an error bar, there is software available to help you. Now, I want to give one word of caution here with which is that if you are going to the previous literature, then you want to be careful about which studies you use to estimate your standard deviation. And that's because studies that have used a particularly small sample size are likely to have unreliable estimates of the standard deviation. And it's also likely that those estimates of the standard deviation may be biased. So if you have the option, you would you, would, you be best to try and choose previous sources of information that use larger sample sizes than to use studies that have small sample sizes. Okay, so that's just a word of caution. Also, just as another point of warning. If you are obtaining data from figures like this, then you want to make sure that you understand the experimental design and the data analysis that we used to generate the standard errors. And you want to do that so that you can appropriately interpret what these values mean. Now, so at this point in history hits very common for journals to require that authors make their data publicly available by depositing their data in some sort of data archive. There are a number of date data archives available. Dryad is a very commonly used one. It's a place where I've stored my data before. And I'm pointing this out because it very well could be the case that a previous study might have performed an experiment that seemed ideal for you to obtain a measure, the standard deviation, but they might not have reported an appropriate value. If the archive their data. Than what you can do is you can go obtain those data and then analyze those data yourself in order to obtain a measure standard deviation that you believe is appropriate for your experiment. So that's just another potential approach for obtaining your standard deviation or you are an estimate of standard deviation. Now, if it turns out that you really are the first person to be collecting relevant data for, for your question, then really the only solution to get a good sense of the standard deviation is to conduct a pilot study. Conducting a pilot study though has its own questions about how you design the pilot study. So for example, how big should your pilot study be in order to get a reliable estimate of your standard deviation? I have created this plot here to help answer this question. So this plot shows the relationship between the standard error of estimate of variance and the sample size. And what? I'll explain this a little bit more in a moment. I just want to tell you what to pay attention to and what not to pay attention to. When we're looking at this figure, what I want you to focus on is the shape of this curve. And to not pay attention to these specific numbers here because these numbers can change over a variety of scenarios. But the shape of this curve is going to be more of central interest. Now, if you recall, standard error provides us a sense of how precisely we can measure something. And if we're conducting a pilot study in order to estimate standard deviation or variance, then its variance or standard deviation, that's the thing we're trying to estimate. So we would like to be able to estimate variance with as small a standard error as would be reasonable. Okay? And what I want you to see here is that as our sample size increases, the standard error for all our estimates are of, of our variance decreases. In other words, as our sample size increases were able to estimate our variance, which gives us a standard deviation. We're able to estimate our variance or standard deviation better and better and better as our sample size increases. Now, we don't really wants to be conducting a massive experiment is a pilot study because really we want to be focusing our resources on the proper experiments that will follow this pilot study. So what's the smallest num? What we should be asking ourselves is, what's the smallest number of samples that we might want to use that could give us a reasonable estimate of our standard deviation. And the answer to that is somewhere in this vicinity, somewhere between seven to ten samples. Because after this point, you can see this line is relatively flat. And so we end up having diminishing returns in terms of the increased precision with which we can estimate variance versus the amount of effort that we have to expense to get that. So going more than 10, it really doesn't pay off to a great degree. But if we go much less than that, you can, then we can see that our uncertainty in our estimate of our variance increases really quickly. So the main point here of this slide is that if you're going to conduct a pilot study in order to estimate your standard deviation, then it's wise to have, say, seven to ten independent samples in your pilot study. Okay, So usually seven or eight independent samples or ten, provides a reasonable estimate of variance or a variation. On this note, though, I'd like to draw your attention to the fact that if you're in the field of biomedical science, it's very common for published studies to have sample sizes that are smaller than seven or eight. And what that means is that many of the biomedical science studies that have been published use experiments that are not even big enough to represent informative pilot studies. So again, this is, I'm really just trying to emphasize that we need to be very careful about the sources of information that we use to inform our choices of standard deviation. If you were in a situation where the only data available was from biomedical science studies that had small sample sizes. But I would say use that information. That information is better than no information whatsoever. But just be careful when you use it. Okay? For example, you might want to assume that the standard deviation for your experiment might be slightly larger than the standard deviation that's obtained in a published study with a very small sample size just to compensate for a potential bias. Okay? Standard deviation is a relatively simple measure to obtain. Effect size is a bit more tricky and requires a lot more thought. And it's, our discussion of effect size is going to really fill most of this video. I'm going to start with a few general comments or general thoughts about effect size before I get to giving three specific bits of advice for deciding on an effect size. Okay? So the first general point that I want to make when thinking about effect size is the kind of question that you should be asking yourself when deciding on an effect size, okay? Deciding on an effect size. We specifically need to be thinking carefully about the biology, not the statistics, but the biology. And what we need to ask ourselves is, what is the smallest effect that you think would be biologically important to detect? That's a really important question to ask. Because when we design an experiment that can reliably detect the smallest effect size that you think is important. Then what that means is we put ourselves in a position such that if our experiment fails to detect an effect of a particular phenomenon that you're interested in. Then what you can say is that you can be confident that if there is an effect, then that effect is likely smaller than the size that you designed your experiments to detect. And if the effect size is smaller than that, then we would argue that the real, that if there is an effect size, it's going to be so small that it's biologically unimportant. So if you take this approach when designing your experiments, you put yourself in a very powerful position to be able to make effective statements about the phenomena that you want to understand. Now, how do we go about choosing this effect size, the smallest level that we might want to detect. In some cases, the answer to that question is very easy. So for example, let's imagine that you wanted to develop a new vaccine or a new drug. In that case, we would almost certainly wants the new drug or the new vaccine to be at least as effective as a currently existing drug or vaccine. And so in that case, what we might do is you might take the effect size for the currently used vaccine or drug and set that as the minimum effect size that we would like to be able to detect. Because, because ideally for trying to create a more effective vaccine or a more effective drug than we would expect that the effect sizes that would interest us would be greater than the previous effect sizes. Not sure why I stalled there. Okay. Another situation in which should be fairly straightforward to decide upon an effect size is it the biology that you're considering has some particular market value, so it has a commercial value. So for example, let's imagine that you are developing a new variety of food. And this new variety might be less susceptible to damage during shipping. And as a result, this new variety might cost more to make, but losses a product or in shipping might offset those costs. Okay? So in this case, the minimum effect size we might want to consider would be the minimum effect size that will be financially viable for the people that were interested in the product. Okay? So these are the relatively easy situations. In some cases, effect sizes or the biological meaning or interpretation of particular effect size can be very difficult to interpret. So here's an example from a paper that comes from my own research area. I won't tell you which paper it comes from. But what this figure shows is we have something called Tajima's D on the y-axis, which is basically a value or a statistic that describes certain aspects of genetic variation within a population. And the people conducting this study wanted to be able to compare to GIM is D among different classes of genes that are expressed in different ways. How they're male bias, Unbiased are female biased. And they found statistical evidence for a difference between the male bias genes. And these are the groups and that's what the star up here as Mintz to alert us to. I read this paper that when I looked at this, I thought, okay, yeah, there's, there's a star there. But does this difference actually matter? Does it actually matter biologically? And the answer is, I don't know. And Apparently neither do the people who wrote the paper because they never talk about that in the original paper. So in this context, this just point in examples like this. If you really don't have a good sense of How important an effect size of a given size would be biologically, it's kind of difficult to say anything meaningful about your results. We return to that, okay, in a little bit. In some cases, biologists might be interested in effect sizes that are very, very small. This can occur especially in the area of evolutionary biology. Because we know week. Well, because we know that genes, a very small effect can have important consequences for evolution over very long time periods. In fact, it's well known now or it's generally established that most of the effects of the genes in any organism's genome tend to have relatively small effects. Yet, just look at the diversity around us. You can see all the different kinds of species that evolved. We can see the variation among individuals within species. A lot of that variation has arisen due to genes of small effect. So for some biological questions, very small effect sizes will matter. I'm just point out some data here to emphasize this point. So this is just a really cool study where the authors of this study, Bellman at all, wanted to understand the genetic architecture of stature in cattle. Those details don't really matter. What I want to point out is that they had an enormous sample size of over 58 thousand cattle in their study. And with this large sample size, they were able to detect SNPS or genetic markers that could explain 2.1% of the variation in stature. Some people might call this a small effect size. Some people might call this a large effect size depending on your perspective. The point is that with large sample sizes, we can detect small effects. That's emphasized by this other study here on, on a related topic. I'm not going to say much here except to say that one of the general conclusions that we get from this kind of work is that we often finds if thousands of genes can affect a particular trait. And that's true for many traits. And usually each, usually many genes have very, very small effects. Okay? My point here is just to say that in some cases, the effect sizes that concern us might be very, very small. So if your interest was in that genetic architecture of particular traits, then you're asking a question that involves finding very small effect sizes and you would need to have really large sample sizes. Okay? So that's one example of where small effect sizes can be really important. In other types of questions. Small samples, sorry, small effect sizes will not be important. I'm going to end this kind of introductory general thoughts with kind of philosoph, philosophical question. And this touches on the gene expression data that I was talking about back here. Okay. I want to point out that if it's truly, truly unclear, what effect size would actually be relevant to understanding the biological phenomenon that you're interested in. Then what that suggests is what your field of research really needs Is other studies in order to be able to answer that very question. If we have no sense of what effect size would actually be relevant to understanding the system, then one of the things needs to be sorted out in that area of research is what effect size would be relevant. So what I'm trying to say here is that if you truly have no sense of what effect size matters, then you need to do more research in order to understand the biological question that your aunts that you're asking in the first place. So I don't want to beat a dead horse here, but I'll just try to be a little bit more concrete. Let's imagine a situation where we performed an analysis. Like we got back here and we got evidence for there being a difference between our different groups. But we found that there was a small difference. If we didn't know whether or not that difference actually mattered, then what we would need to do is further research to understand the biological consequences of small differences like this. Are there any One last question on this kind of philosophical point is that let's imagine that you conducted a study like this one where you've got a small p-value. If you truly have no idea whether or not the differences between your groups are big enough to be biologically relevant, then does that p-value actually tell you anything? I'm not sure. It might depend on the kind of question you're asking. But I would argue that often it won't. Because if you don't know whether or not an effect as biologically meaningful, then it becomes very difficult to argue that you found anything important. Okay, so that's, that's, those are my general thoughts on effect size. Kind of some philosophical perspectives. What I'd like to do now is much more quickly. Provide you with three guiding points for thinking about effect size, OK, and decided on an effect size for your study. The first is to consider role for theory. And then we can think about relative effect sizes. And then finally, the most obvious thing which is to look at previous research. Some areas of biology. So I'm going to start talking about theory. Some areas of biology are rich with mathematical theory where the theory helps us to gain. The three basically represents mathematical hypotheses for how systems work. And I'm just going to show you three different quick examples that are relevant to my own research. The first comes from local adaptation, and I'm just showing you a couple of notable studies that consider the, the evolution of local adaptation. That the main thing that I want to point out from all this theory is that often local adaptation can be thought of as occurring when you have a particular balance between migration between populations and natural selection. And so this theory provides a framework that tells you how migration and natural selection might balance against one another in various ways to predict when local adaptation will and will not occur. So my point here is that if you for example, had a good estimate of migration, so if those were data you already had in hand, and your goal was to understand natural selection as it pertains to local adaptation. Than what you could do is you could take information for this one variable on migration, go to the theory and then ask, okay, for particular outcome that I'm interested from the theory. How strong with the Natural Selection need to be in order to, in order to detect the thing that I'm interested in. And then that would give you a reasonable estimate of the effect size for natural selection that you would hope to obtain. Okay? Um, I apologize for not actually explaining what local adaptation is. I'm going to be like that for the next two examples. Really, you don't need to understand the specifics of the biology for, for what I'm trying to explain here, what I'm trying to say is that if you have some mathematical framework that connects a variety of variables, then if you understand one of those variables with previous data, then you can use that theory to make predictions about the effect size you would need to study the other variable. Here's another example of my own. This relevant to my own research, which involves mating system evolution. So the evolution of the ability to mate with yourself. This is a very big area with lots of theoretical research. And all of these models consider something called inbreeding depression. Which is the decrease in fitness that arises from mating with close relatives. Basically, it's, it's the reason why we have laws in many nations for why you can't marry a cousin or why you can't marry a close relative. Okay. And if you were interested in studying inbreeding depression and measuring inbreeding depression in a particular species. And you wanted to know kind of what effect size of inbreeding depression would be a reasonable size to detect. You could go to these theoretical models to get a good sense of a reasonable effect size for inbreeding depression. And my third quick example. One of the aspects of biology, the interests me is potential for conflict, genetic conflicts between females and males. And there can be a number of different evolutionary outcomes that depend on how strong natural selection is on females versus males. And there's a lot of theory that addresses those evolutionary dynamics and that explore how natural selection, how those dynamics depend on natural selection for females and males. The point here though, is. Let's imagine that you wanted to measure natural selection and males in this context for sexual conflict. If you had good estimates of natural selection and females and you had a particular evolutionary outcome that you want to do explain. You could go to the appropriate models, use your measure of selection or use your estimate of selection females to then determine from those models how strong selection to males might needs to be in order to find the outcome that you're interested in. So once again, we have an example we can is one estimate of one thing. Put that into the theory and then let that inform your choice of an effect size for another aspect of biology. Okay? So the point here is that theory can be exceedingly helpful to help refine your hypotheses and to be able to ask biologically meaningful questions with biologically meaningful effect sizes. This goes back to the question earlier about what to do if you truly have no idea of what would be a reasonable effect size. This is one place where theory can really help tease apart what a meaningful effect size might be. Okay, so we're going to shift gears now and I'm going to give a seconds totally different bit of advice on how to determine irrelevant effect size. And that is to ask a very different biological question. Okay? And I'm going to explain this by way of example. Let's imagine that we were interested in explaining some physiological process. And we already had some data that explore that physiological process in females versus males, okay? And from that previous work, we had a sense of the size, the difference between females and males with respect to this physiological process. Okay? What we could do then is let's say, let's say that we were interested in understanding some other aspect of biology as it pertains to this physiological process. So for example, let's imagine you want to understand how stress influences this physiological process. So we wanted to compare this physiological process in a stressful environment versus a non-stressful environment. In that case, it might be difficult to come up with a particular effect size that might interest us. But what I want to propose, we might think about effect sizes for our stress treatments in terms of relative effect sizes. So in terms of effects relative to something like the effect of sex. So in other words, if we already knew that there was an effect of sex, let's say that was this big on our physiological process. Then what we might ask, what might use? You might use that effect size as a benchmark for understanding the effective stress. We might say, okay, if the size of differences between the sexes has this big of an effect on physiological processes that I want to know whether the effect of stress at least as big as the effective sex. Or maybe want to be more conservative and say, You might want to ask whether the effect of stress is at least half as big as the effective sex. Okay? The point here is that even if you don't have a strong intuition for what effect size might be biologically meaningful. Understanding a process in terms of relative effect sizes can still give me some meaningful biological insight. Because you can start to make conclusions about what's types of factors most important for the phenomenon you're interested in. In this case, some physiological process. Is sex more important or a stress more important, for example. And the last option is the most obvious. And that is that we can always refer back to previous research in order to get estimates of effect size. And I'm not going to say much more on this except for a, just a word of warning. And that is, you really want to be careful about what studies you use to inform your sense of effect size. So in particular, effect sizes that come from low-powered studies are. So studies with very small sample sizes often cannot be trusted. In fact, low-powered studies often upwardly bias the effect sizes that are reported. And that's something we're to talk about in future videos. So if you decide to choose an effect size, not by answering the question, what's the smallest effect size that I think is biologically important. But instead, by just basing it off of previous research findings, then I caution you to choose the previous research really carefully and to think about how to use that information wisely. So just to summarize, we've talked about how to obtain information about inherent variation and effect sizes to inform our power analyses. And I pointed out there's a lot of different things to consider and a number of different approaches to take in order to get these estimates. The really the main thing that I want to emphasize though, is that when we're making these decisions, the hardest decisions often involve thinking about biology and not statistics at all. This is hard work, but it's work that can really pay off. Because when we've done this hard thinking about things like effect sizes, we will be that much more prepared when we sit down to write down a manuscript that reports the results from the study that we've conducted. And this kind of thinking can really help us to produce a much higher quality publication. I'm going to end this video there and say hope it's been helpful and thank you very much.