Okay, in this video, I'm going to introduce a topic of questionable research practices. This is a very general introduction. In fact, I'm going to say very little about questionable research practices. Specifically. Instead, what I want to do is I want to explain why we need to visit this topic in the first place. And I want to start just by asking this question. How do we judge whether a scientist is successful? I know from my own experience applying for jobs, that when people are looking for job applicants in academia, they often want to Get, get information about how many publications you've produced. They want to see where those publications are, so what journals they appear in. They also often wants evidence that you are capable of acquiring grants. These are often things that are celebrated in science and these are often criteria that we use to judge how successful a scientist is. Of course, there are other aspects as well. So a scientist might have a very good reputation, for example, they might be very good thinker. They might have a real talent for seeing the next big issue that needs to be tackled. Um, or they might have extraordinary technical skills that allow them to ask questions that other people may not be able to. They might just be an excellent mentor, I shouldn't say just because that's so critical to the, the life of science. My point here is that these are often ways that we might use to judge whether or not a scientist is successful. What we don't often ask, however, is how robust a scientist's research is. They might produce lots and lots of papers emanate, publish in all the places where everyone dreams to publish. But ultimately, the ultimate test, whether or not a scientist a successful, is a measure of how reliable their science actually is. I might sound really skeptical our jaded by bringing this up. But as you'll see in a moment, this is not, this is not, this is not a trivial issue. I'll just jump into why you might ask this question. Okay? In 2012, this paper came out, which shook a lot of the scientific community. It addresses the question of how reproducible research is. In other words, there is a question of how many experiments or results that scientists produce can actually be replicated. If another group of scientists tries to recreate their experiment and obtain the same result. I would argue that that is a true measure of the reliability of someone science. And if someone's results are not reproducible, then that really calls into question how useful the original results are. I should say before going on to this question of how reproducible sciences has been floating around for a long time before this paper came out. But this paper was one of the first big papers that really grabbed people's attention for this issue. And there's a lot of good mess. There are many good messages in this paper, but I want to highlight one. I've given this paragraph here which I've just copied and pasted out of the paper. I'm not going to read it all to you. So if you want to see exactly what it says and just pause the video and have a look. Here's a message in a nutshell. What these authors point out is that a biotechnology firm called Amgen decided to take 53 landmark studies in cancer research and ask whether or not they could reproduce the original results that were reported in these landmark studies. And what they found was shocking. They found that only six out of those, 53 or 11% of those original studies could be replicated. This is shocking because it really calls into question. I've just, you know, what do we actually, what is a scientific study worth in essence? And it also really is a call to action to all scientists, all scientists who try to understand what we can do to make our research more reproducible so that our science can have more meaningful impact. So it can actually stand the test of time. This original study that I've just mentioned was a big step in trying to understand what's now called the reproducibility crisis. And there's been a lot of reflection and action in various communities. And I'm going to just highlight some of those actions in this video. This figure here comes from a survey that was published by nature in 2016 where they surveyed over 1000, 500 scientists and ask them a number of questions about reproducibility. And here's this most basic question. Do you think that there's reproducibility crisis? And over 50%, or 52 percent of respondents said yes, there is a significant crisis. 38% said yes, a slight crisis. And the minority of 10 percent or less did not seem to be as bothered with this. This study asked a number of other things. I'm just going to highlight one of them. This is a question that I found particularly exciting where they asked these 1500 scientists, have you failed to reproduce an experiment? And this is both in reference to trying to we produce someone else's work and also try to reproduce your own work. And what you can see here. So these bars represent percentages. You can see the percentages are not small. So there is a large percentage of these respondents who have failed to reproduce both other people's work, which is given by the dark red lines and their own work. What this demonstrates or what this provides additional evidence for, I should say, is that many people, do you have firsthand experience with the fact that scientific results can be on reproducible. I want to point out here that we shouldn't get too carried away with these large percentages. Because this question and these data do not relate to how frequently do you fail to reproduce an experiment. So for example, in chemistry, where over 80% of the respondents failed, reported to have her failed to reproduce an experiment. It's possible that they met each. Scientists that they surveyed, might have tried to reproduce a 100 experiments, but only one of them was or was not reproducible. Pain. If that were the case, then we might worry much less than if half of the time people experienced or people failed to reproduce either their own work or someone else's work. So this figure is a bit shocking. But we shouldn't get too excited by it, okay? Because it doesn't tell us anything about how frequently work is your reproducible. There have been some concerted efforts, however, within some communities to quantify how reproducible studies are. So this is an amazing paper which came out, I believe in 2012, where the authors of this paper conducted that they chose 100 experimental and correlational studies from psychology. And they works to reproduce these original studies. And when they did this, they reproduce the studies in a way that we produced two versions had even greater power than the original, than the original studies. And what they found, however, is that roughly speaking, they only managed to replicate roughly 40 percent of the results. Again, that's shocking and that's a huge call-to-action course. So here, here's one example for psychology and I'll just remind you of the clinical, sorry, the cancer research study that I pointed to earlier, which seemed to have a around a 10 percent reproducibility rate. So why do these problems with reproducibility arise? There are a number of potential sources of problems that can lead to air reproducible data. One source of problems is certainly experimental design. So here are some results from this paper here by elastic at all, which came out in PLOS Biology in 2018. And what this paper's concerned with largely is the issue of pseudo replication. Where if data are analyzed incorrectly, then we might call them pseudo replicated. See, replication has a particular definition. It's something that we're going to talk about a lot when we talk about experimental design, which is going to come in the near future in these videos. So I'm not going to talk about it very much right now. What I'd like to point out though, is that when experiments are analyzed incorrectly so that the analysis is pseudo replicated, that basically leads to results that are, that are just not reliable. And logic at all surveyed the literature for examples of animal studies. And they focused on, focused on particular, on a particular form of experimental designed for animal studies. Say look for experimental innovation. They look for examples where experimental interventions were applied to parents, but the effects are examined in the offspring. And they chose this for reasons that had to do with the common expectations for these types of designs. And here's what we find are, here's what they found. They found that in about a quarter of the cases that they examined, it was clear that the analyses were not performed correctly. So that's about a quarter of results. We're are now unreliable. Sorry. When I say yes and no, I should have I should explain what the question is. The question is, is the experimental unit correctly identified? This means, did the researchers analyze a data appropriately? So I don't remember what I said before, but if I yes is a good thing. So that means 25 percent of studies that logic at all examined did analyze a data appropriately. I think I said, give the reverse message moment to go and so, apologies. So about 25 percent of these types of studies are reliable. About 45% did not. So that means at the very least, 45 percent of studies of this type have unreliable results or they report unreliable results as well. For about 30 percent of the studies, we simply can't tell whether or not the data were analyzed appropriately. I would argue that this case of unclear is just as bad as cases where the where the analysis was not performed correctly. Simply because in cases where the where it's unclear without the data. In cases where it's unclear whether the data were analyzed appropriately, we simply don't know whether or not to trust the study. And if we don't know how to trust it, if we don't know whether or not to trust it. What do you take from it? Experimental power is also a common flaw in experimental design. And this paper here assesses the overall power of studies in the area of neuroscience. And what they find, what they estimate is that the average power for a neuroscience study is around 20 percent, which is very low. And again, this is something we're going to talk about. When we talk, we're going to talk about more. When we talk about experimental design and power. Specifically. Why does this matter? Why does this low-power matter? Well, it matters for a bunch of reasons which we'll talk about in the future. But in the current context. We can connect this paper with this one. You can see that we have the same author on both. And so these papers very much mirror each other. And what we get from this latter papery is that given power of around 20 percent, it's not unreasonable to expect that about 50 percent of significant p-values that are published to be false positives. So collectively, these two papers suggest that the fraction of significant p-values that are reported for neuroscience that are false positives, maybe disturbingly high. We're going to talk about these aspects of experimental design more when we talk about experimental design, that's, that's our next major topic. In the next few videos that follow this one, we're going to focus on something a little bit more subtle, which is the area of questionable research practices, which can be broken down into three general types. There's cherry picking, p-hacking, and harking, where harking stands for hypothesizing after results are known. And all of these can basically involve very subtle research practices, but that can have big consequences for the reliability of your research. So that's pretty much all I'm going to say about questionable research practices in this video. I just want to make it clear in this video why we're thinking about these things and why they're so important. This video has been a bit of a bummer. So I want to end with some perspective. I've, I've rained down a lot of doom and gloom here. So I wants to end this video with this message. Despite all the doom and gloom, science works. We know it works. Without science working. We could not develop vaccines or have excellent medical procedures. I myself have benefited from them when I've had some major surgery. We know that our understanding of genetics and evolution have led to very efficient ways of breeding new crops and breeding animals for agriculture. Genetic engineering would not be possible without science actually working. We cannot put people on the moon without science actually working. So it's clear, it's perfectly clear that science does work. What's an issue with or the central issue in this video is that science doesn't work, however, as often as it should. In other words, we can do a lot better. And I also want to point out that this call to arms, to do better is not just so we can feel better about ourselves as scientists. There, there are major ethical implications here. Because when science is not done as well as it can be, we waste a large number of animal lives. If you're doing an animal related research, the amount of resources, the amount of public money that's spent on research and potentially just wasted is astonishing. And also simply the amount of time that graduate students and postdocs spend in the lab on research that may be unreliable is depressing the amount of time that Professor spent writing grants for work that might not actually be reliable is probably something that many people don't want to think about, but it's something we do need to think about. I said I was going to end this on a, on a happy note. And it seems like, like I've gone all doom and gloom again. So I'll just say it again. Science works. It does work. And science is hugely beneficial for the world. The point of these videos is going to be to point out a number of places where we can work hard to do better. And it makes science work even better than it already does. And on that note, I'll end the video and I'll say, thank you very much.