As I get closer to the end of my fellowship and the writing up of my research I have become preoccupied with whether it is any good. Running through my mind are a series of questions: is the investigation thorough and the argument logical? Has the methodology been appropriate, given the questions I am asking? Am I generating new insights that make a useful contribution to the field? Needless to say, I am keen to produce research of high quality and have been paying attention to how others define this, comparing their ideas with my own sense of what constitutes valuable work.
I have written in a previous blog on the concept of ‘care’ in relation to qualitative and arts-based research and how this makes more sense to me than notions of ‘rigour’ derived from scientific research. I have the same unease with the concept and language of ‘validity’ and ‘reliability,’ both of which are frequently invoked alongside rigour to suggest that qualitative research is or isn’t of high quality and worth believing. Yet often I am unclear what validity or reliability criteria are being applied and not sure if these are appropriate for my own or others’ qualitative, practice-based studies. I have, therefore, been reading up and will briefly outline what I have gleaned in relation to these terms in the hope that it will bring some clarity.
Validity, I have discovered, refers to the credibility of any research. In other words, are the research methods and findings authentic and useful. There are two aspects of validity, internal and external, that are brought into play within the scientific paradigm. Internal validity refers to the procedures or methods used in the research and the extent to which they are appropriate to address the research question. In the case of quantitative research, for instance, will they measure what they need to measure. So, for example, in a study of the relationship between an arts intervention and children’s attainment in schools, any research setting out to test and measure this would need to employ methods that could reveal the strength of the causal relationship between the two variables. That is, between the art project and the children’s attainment.
External validity refers to the extent to which the results from any research can be generalised to situations beyond the study itself. In the case of the quantitative children’s attainment research, external validity would be granted by whether the findings could be confidently applied, say, to a different school with different students. Again, within the scientific paradigm, reliability concerns the repeatability of findings. So, with the arts and children’s attainment research, the more times this study is implemented and produces the same results, the greater the reliability of the findings.
But are these scientific criteria always relevant in the art museum? Take the question of reliability as determined by reproducibility. With my research, the aim has not been to undertake a quantitative study that could be repeatable by a different researcher. I am seeking to unearth and understand how specific people perceive and enact research practices and identify the implications of these perceptions for the sector. I am concerned with internal validity, although I would not use this term, in the sense that I have employed methods best suited to explore my question. But external validity poses more difficulties, since I am not concerned with the extent to which my results are generalisable. So, is my research ‘unreliable’ or should different criteria of trustworthiness be applied here?
The academics Norman Denzin and Yvonna Lincoln draw together alternative validity criteria, all of which are potentially relevant to the museum research context. The first of which is transparency. Because I am not testing a hypothesis (as with the arts and young people’s attainment study mentioned above), the validity or believability of my research needs to come through the clarity of my enquiry process – I must make explicit what I’ve done – and the credibility of the findings in relation to that process. High quality research is that which provides convincing evidence and a systematic logical argument to support the findings. In some ways it can be seen to resemble, as my colleague Christopher Griffin pointed out, ‘a GCSE maths question’ in that a researcher needs to show ‘their working out.’
Reflexivity is also important. To avoid the criticism that my research is merely a collection of anecdotes and personal impressions I need to recognise my situation, acknowledge my subjectivity and make explicit my values. So, in my study, validity will be established and quality ensured, not via my objectivity, but through my reflecting explicitly on what takes place and my responses to it.
Going still further Denzin and Lincoln argue that any social-justice oriented approach to research should be evaluated according to its emancipatory potential. Here validity should be judged in terms of emotionality and caring, personal accountability and the representation of the experience of oppressed people. These are clearly some way from any scientific criteria of reliability, yet they resonate with my experience of art museum practice and my research, which aspires to some degree to address issues of emancipation and representation.
For these reasons it is these criteria of quality, rather than the understandings of validity and reliability drawn from scientific research, that I am bringing to my own study.
PRAM 