What exactly is research impact? Exploring the concepts, vocabularies and logics used in evaluating the impact of social research

This article-based dissertation explores the vocabularies of research impact and their logic in evaluating the impact of social science research. The motive for this study stems from the increasing emphasis on social impact in Finnish research evaluation culture and practices. In practice, impact has become a criterion for quality judgements in competitive funding and formal approval for completed research projects, as well as a developmental tool for university faculties and the legitimation of research policies. Broad academic literature have welcomed the cause of impact thinking with practical frameworks for research evaluation, while a small body of literature has taken a conceptual distance from the practical discussion. This dissertation continues the exploration of the latter discussion by collating its crucial arguments and developing them further. This dissertation brings together the previous notions of critical academic discussions and literature by elaborating on their conceptual contributions to the discussion about impact. The dissertation attempts to develop the critical discussion further by trying to understand why the vocabularies of impact appear in such discordant and often simplistic ways in policies, public discussions and evaluations. It also seeks to understand why impact is construed as a diminished version of the complex social relations of knowledge within official evaluation frameworks for funding and research development.

This dissertation presents discursive notions of the policy frames used to approach impact and the academic discussion on the concept of impact. In addition, this dissertation investigates the logical aspects of a practical impact evaluation. Conceptually and methodologically, the dissertation contributes to the discussion by developing a critical approach that recognises the discursive accounts behind impact thinking, the in-betweenness in the operationalisation of impact via formalised methodical choices and the logical prospects and limitations to constructing and construing impact evaluation.

The conceptual framework of the dissertation relies on previous conceptualisations of research utilisation, public engagement and transdisciplinarity. The dissertation uses these conceptualisations to show which aspects are familiar in the vocabularies of impact and how they construct the concept in various contexts of talk and practice. The dissertation also uses a constructive understanding of evaluation studies to comprehend liminal spaces in impact evaluation. The liminality perspective provides insights into institutional purposes and tensions in evaluation approaches. The methodological approach of the dissertation underlines contextual vocabularies that represent a broader social and cultural understanding of knowledge practices, research evaluation and higher education. The preoccupation with impact is considered a political, social and cultural phenomenon that shapes the understanding of how research knowledge interacts with social agents and how to operationalise it in evaluation. These premises counter purely prescriptive research agendas that attempt to develop ways to construct simulacrums of real research impacts for policy purposes.

The three articles in this dissertation investigate the discursive and logical characteristics of research impact and its evaluation. They are based on research material from policy documents/recommendations, expert interviews, funding proposals, mid-term research reports and funding calls in Finland. Article I illuminates the vocabularies of impact in policy and academic discussions. Article II clarifies the guiding principles of impact evaluation and their dilemmas. Article III explores the relationship between researchers’ pre-evaluative strategies of impact depictions and guidance for proposals and evaluation. Each of the articles contributes original typographies, identifying new usages of the concept of impact. They also highlight what the concept of impact means for formal evaluation practices and possibilities for social scientists in expressing the social opportunities of knowledge.

This dissertation concludes that it is not possible to comprehensively understand research impact evaluation without a critical meta-evaluative perspective. Although practical-prescriptive studies of impact evaluation have achieved conceptual sophistication, they have not been able to solve the logical problems that lead to profound methodical and practical problems. This is because of their lack of understanding of the notion that impact evaluation is based on divergent ideals of knowledge and knowledge use, which have unbalanced representations in research policy and evaluation. This dissertation contributes to the debate by arguing that impact has become an amorphic concept. This is realised by absorbing divergent vocabularies that construct conceptually entangled understandings of the social capabilities of research knowledge and possible ways to operationalise this understanding in evaluation. However, because the emphasis of impact thinking is on incumbent public policy rationales for rating research productivity and legitimising research as socially responsible, it often relies on a reduced understanding of social science research and its possibilities for social change. This is why impact evaluation is often premised on a formalised logic that identifies similar threads of impact that have been preconstructed in its vocabularies for research priorities and evaluation guidance. The dissertation suggests that the only way to solve the contradictory ideals within impact thinking requires a shift from research impact evaluation towards more open organisational and institutional learning of knowledge use in local communities.