A Framework for the Assessment of Research and Its Impacts

Figure 1

Figure 2

Dimensions of methodology: subject and means in our framework_

Dimension	Type/category	Content
Subject(of the assessment)	Output (baseline)	Result of a transformation process which uses inputs to produce products or services
	Productivity and efficiency	Partial or total factor productivity with respect to a reference
	Effectiveness	Considering inputs and outputs, and accounting for the aims of the activities
	Impact	All contributions of research outside academia
Means(of the assessment)	Quantitative approaches
	Qualitative approaches
	Quali-quantitative approaches

Definitions of education, research, and innovation_

Term	Definition
Education	In general, education is the process of facilitating the acquisition or assignment of special knowledge or skills, values, beliefs, and habits. The methods applied are varied and may include storytelling, discussion, teaching, training, and direct research. It is often done under the guidance of teachers, but students can also learn by themselves. It can take place in formal or informal settings and can embrace every experience that has a formative effect. Education is commonly organized into stages: preschool, primary school, secondary school, and after that higher education level. See the International Standard Classification of Education (ISCED, 2011) for a more technical presentation.
Research	According to the OECD’s Frascati Manual (2002), research and development (R&D) is the “creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture, and society, and the use of this stock of knowledge to devise new applications.” The term R&D covers three activities: “basic research, applied research and experimental development. Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view. Applied research is also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily toward a specific practical aim or objective. Experimental development is systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products, or devices, to installing new processes, systems, and services, or to improving substantially those already produced or installed. R&D covers both formal R&D in R&D units and informal or occasional R&D in other units.” See also the more recent Frascati Manual (OECD, 2015b).
Innovation	According to the OECD (2005), an innovation is “the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organization or external relations. The minimum requirement for an innovation is that the product, process, marketing method or organizational method must be new (or significantly improved) to the firm. Innovation activities are all scientific, technological, organizational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations. Innovation activities also include R&D that is not directly related to the development of a specific innovation.”

A characterization of the data dimension in our framework_

Dimension	Characterization
Availability	usability
	sampling vs census
	freely, controlled or undisclosed
	consumption vs participation
	open, institutional provided
	commercial vs publicly available
	privacy/confidentiality (see Ekbia et al. (2015))
Interoperability	a very high level is obtained by an OBDM approach (see Daraio et al. (2016b))
Unit-free property	independence of the data from the unit of analysis

A non-exhaustive overview of the literature on the theory dimension_

Literature stream	References
Economics of science and technology	Audretsch et al. (2002), Stephan (2012), Mirowski & Sent (2002)
Theories of growth	Solow (1957), Abramovitz (1956), Nelson & Phelps (1966), Romer (1986; 1994), Aghion & Howitt (2009)
Quantitative science and technology research	Egghe & Rousseau (1990), Egghe (2005), Moed, Glänzel, & Schmoch (2004), Ding, Rousseau, & Wolfram (2014), Cronin & Sugimoto (2014; 2015), Glänzel et al. (in press)
Economics of innovation, innovation studies	Fagerberg, Martin, & Andersen (2013), Hall & Rosenberg (2010)
Science of science, science and society	Fealing et al. (2011, p. 4), Gibbons et al. (1994), Etzkowitz & Leydesdorff (2000), Edquist (2001), Aghion, David, & Foray (2009), Helbing & Carbone (2012), Scharnhorst, Borner, & van den Besselaar (2012), Nowotny, Scott, & Gibbons (2001), Barré (2004)
Economics of knowledge	Antonelli & Link (2014)
Economics of education	Blaug (1966), Johnes & Johnes (2004), Checchi (2006), Hanushek, Machin, & Woessmann (2016)
Education and society	Roper & Hirth (2005), Teixeira et al. (2004), Teixeira & Dill (2011)
Societal impact, science and public policy studies	Bornmann (2013), Ebrahim & Rangan (2014), Perkmann et al. (2013), Veugelers & Del Rey (2014), Hill (2016)
History of science	Godin (2002; 2004), Gingras (2016)

Toward an ethics of research assessment? Some connections of our framework with MacIntyre’ oeuvre_

Enabling condition	Potential connection	MacIntyre work
Convergence	Invitation to overcome the fragmentation of knowledge and excessive specialization.	The end of educationMacIntyre (2006)
Mixed methods	The need to go beyond a pure quantitative approach (abstract representation of the reality) and include qualitative cases (narratives and storytelling).	After virtue, MacIntyre (2007), & Whose justice, which rationality?MacIntyre (1988)
Knowledge infrastructures	Retrieve the values of tradition in communities of practice that regulate themselves by defining their own standards.	After virtueMacIntyre (2007)