Introductory Lecture
Slides
The critical researcher: considering validity and reliability
Being an effective researcher requires that you develop the ability to approach your learning critically. A significant aspect of this is to be able to recognise and produce good quality research. Two measures that researchers apply throughout the research process are validity and reliability.
In academic research the terms validity and reliability are often confused, in part because they are often considered to be interchangeable. However, this is not the case and each concept refers to a specific measure. Further confusion can also be caused due to the many different ways that the two concepts can be applied to a piece of research. Nevertheless, by ensuring a separation between the two measure we can actually enhance the quality of research at all stages: design, data-collection, analysis and evaluation. When both measure are effectively applied, it should result in research that is thorough, more critical, less-open to biases, and more generalizable.
Validity
Validity is directly related to the conclusions that you reach at the end of your research. A valid conclusion is one that is securely supported by the evidence presented.
There are two main types of validity:
Internal validity is concerned with the rigour of the study, in particular the quality of control exhibited. Is the research honest, reflecting reality, and has it accounted for known variables? Moreover, it is pertinent to ask both during the research design process and later when evaluating it: Are the methods I have chosen to use valid for the task?
External validity or ‘generalizability’ asks what relevance the research has beyond the group or instance that was investigated? Sample size, characteristics, geographic location, and historical factors can all pose risks to the external validity of research. For example, should only a small group of people respond to a survey, the data collected can not be considered to representative of a larger group.
Reliability
Reliability asks: If we repeat the research will we achieve the same outcome? In other words, would another group of researchers using the same methods obtain the same results?
Different results can be achieved due to a range of factors however the most common are:
Error or variance on the part of the research subject. Asking someone the same question on different days is likely to garner different responses. This is natural and is to be expected in qualitative research, particularly with surveys, questionnaires and interviews.
Bias can occur on both the part of the research subject and the researcher themselves. In terms of the subject, they may alter their responses to a survey or during an interview due to their relationship with the researcher. In terms of the the researcher, bias can be exhibited at all stages of the research process. During the design stage, bias can be exhibited in the wording of questions. Later in the process, a researcher may present their findings in such a way that fits with their hypothesis ignoring data to the contrary.
Ensuring quality and credibility
Ultimately it is very difficult to control every single aspect of research. Unreliability can never be completely ruled out therefore, when presenting research it is imperative to be honest and critical in order to avoid claims of invalidity and ensure credibility:
- How was the research conducted: enquiry, approach, methods?
- Who conducted the research and what are their credentials?
- What secondary sources were referenced and are they reliable?
- What methods of analysis and interpretation were applied?
- Is there a clear, critical voice?
- Is the research written to an appropriate standard?
Reading
- Graham R. Gibbs (University of Huddersfiled): Reliability, validity, generalizability and credibility. Pt .1 of 3: Research Quality [Video]
- Graham R. Gibbs (University of Huddersfield): The Quality of Qualitative Research. Part 2 of 3 on Research Quality and the Research Process [Video]
- Blaxter, et al., (2010) Advantages and Disadvantages of Surveys from How to Research, McGraw-Hill International [Section – click on ‘page 79’ to view/read]
- University of Surrey: Evaluating Secondary Sources [Guide]
- Johns Hopkins University: Evaluating Information Found on the Internet [Guide]
Twitter Discussion
Archive – Wednesday 27th March
Assignments
1. Read and critique the #egg research project conducted by your group. Analyse and evaluate the validity and reliability of your research.
- Write and publish a 500-750 word critique (blog post or Google document), making reference to both the article and your weekly reading. References should adhere to the Harvard system.
2. Complete a blog post reflecting on the week’s assignment and your progress (include a link to the published assignment)
- How did this week’s reading and assignment build on last weeks?
- How has it added to your understanding of conducting effective research?
- How much progress do you believe you have made as you approach the halfway point of the course?
3. Read and comment on the blog posts of the 3-4 members of your comment group. (Discuss their responses to the questions and share your learning from the week)