Evidence-based Nomenclature and Taxonomy of Research Impact Indicators

Improvement / Addition / ReductionThis function of indicator explains the addition or enhancement of an existing phenomenon in quantitative or qualitative form. (Example: Improvement in economic gains such as increased employment, health cost cut (Weiss, 2007))

CreationThis function of indicator focuses on the creativity in the form of the development of new knowledge, theory, technique, method, technology, approach, opportunity or any workflow. (Example: Creation of prevention methods for clinical practice (Trochim et al., 2011))

RecognitionThis function explains the recognition of effort in the form of outstanding quality by the peers or experts such as in the form of awards, promotions, meritorious selection and work showcasing etc. This recognition can be of the research, the researcher or the research institute. (Example: Receiving an award on research (Kuruvilla et al., 2006))

Obsoleting / ReplacingThis function elaborates the policy, law, regulation to obsolete or disuse the existing phenomena to overcome the future negative impacts. (Example: Change in law to obsolete the existing method of drug approval (Maliha, 2018))

Context (C)

Target (t)Contextual targets in research impact science include knowledge, service, policy, law, guideline, system, technology, procedure, method, framework, workflow, publication, patent, product, stakeholder, citation, literature gaps, intellectual challenges, scholarly issues, relationships, collaborations, and networks etc. These are the key areas but usually partial in contextual understanding.

Domain (d)The contextual domain is the main area or field of interest of the indicator system designer such as health, education, economy, environment, academia, medical science, chemistry, history, multidisciplinary etc. The main body of knowledge and elaboration of indicators are always from the domain language. The domain is the main component of the indicator, which specialised the context and application of the indicator. However, the level of the domain is subject to the interest and perspective of impact evaluator.

j_jdis-2020-0018_apptab_001_w2aab3b7c20b1b6b1ab2b2ab2Aa

Ref	Indicator	Reference	Measure	Qualitative/Quantitative
1	Identification of research gaps, questions and new research dimension	(Heller & de Melo-Martín, 2009; Kuruvilla, Mays, Pleasant, & Walt, 2006; W. M. Trochim, Marcus, Masse, Moser, & Weld, 2008; Weiss, 2007)	Yes / No^Ŧ	Qualitative
2	Development of a new technique for data collection and new data	(Heller & de Melo-Martín, 2009; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
3	Creation of a research method or extension of existing by involving new approach and technique	(Kuruvilla et al., 2006; W. M. Trochim et al., 2008)	Yes / No^Ŧ	Qualitative
4	Defining the concept and subject vocabulary in a more comprehensive way	(Mankoff, Brander, Ferrone, & Marincola, 2004; W. M. Trochim et al., 2008)	Yes / No^Ŧ	Qualitative
5	Formation of research groups and collaborate in multidimensional research	(S. R. Hanney, Grant, Wooding, & Buxton, 2004; Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; W. M. Trochim et al., 2008)	Yes / No^Ŧ	Qualitative
6	Recruitment of skilled researchers	../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_68 (Heller & de Melo-Martín, 2009)	How many researchers are recruited?	Quantitative
7	Development of communities of science, new grant programmes; replication and new research	(S. Hanney, Buxton, Green, Coulson, & Raftery, 2007)	Yes / No^ŦŦ	Mixed
8	Effective planning and addressing future research	../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_51 (Gordon & Meadows, 1981)	Yes / No^Ŧ	Qualitative
9	Research capacity building for an individual or a group of researchers	(Buxton & Hanney, 1996; Raftery, Hanney, Greenhalgh, Glover, & Blatch-Jones, 2016)	How many researchers are trained?	Quantitative
10	Preparing a better procedure for researchers induction	(Heller & de Melo-Martín, 2009; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
11	Improvement in ethical approval processes for better decisions and timeliness	(Pober, Neuhauser, & Pober, 2001; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
12	Formation of new research teams and projects	(Pober et al., 2001)	How many projects and teams are established?	Quantitative
13	Successful completion of ongoing research with the achievement of set targets	(Weiss, 2007)	Yes / No^ŦŦ	Mixed
14	Retention of research team by involving in productivity and future research	(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Nathan, 2002)	How many members are retained?	Quantitative
15	Advancement in numbers and quality of research and research teams	(Nathan, 2002; Pober et al., 2001; W. M. Trochim et al., 2008; Weiss, 2007)	Yes / No^ŦŦ	Mixed
16	Enhancement of research process, behaviour and procedural protocols	(Heller & de Melo-Martín, 2009; Pober et al., 2001; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
17	Recognition and leadership of researchers in the research domain	(Kuruvilla et al., 2006; Pober et al., 2001)	Yes / No^Ŧ	Qualitative
18	Improvement of research communication between researchers and research organizations	(Heller & de Melo-Martín, 2009; Mankoff et al., 2004)	Yes / No^Ŧ	Qualitative
19	Serving of research staff on a higher level in more advanced organizations at national and international level	(Kuruvilla et al., 2006; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
20	Improvement in research culture and overall environment	(Heller & de Melo-Martín, 2009; Kessler & Glasgow, 2011; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)	Yes / No^Ŧ	Qualitative
21	Identification and overcoming of the research process constraints	(Heller & de Melo-Martín, 2009; Pober et al., 2001)	Yes / No^Ŧ	Qualitative
22	Improved willingness and tangible measures for practice-based and applied research	(Westfall, Mold, & Fagnan, 2007)	Yes / No^Ŧ	Qualitative
23	Development of improved analytical methods for existing data	(Kessler & Glasgow, 2011; Kuruvilla et al., 2006; W. M. Trochim et al., 2008; Weiss, 2007)	Yes / No^Ŧ	Qualitative
24	Improvement in multi-disciplinary research methods	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
25	Creation of methods for cross domains results in interpretation and synthesis	(Kuruvilla et al., 2006; Pang et al., 2003)	Yes / No^Ŧ	Qualitative
26	Embracing the innovative methods for measuring the research outcome	(Dougherty & Conway, 2008; W. M. Trochim et al., 2008)	Yes / No^Ŧ	Qualitative
27	Discovery of new or advanced research findings	(Lavis, Ross, McLeod, & Gildiner, 2003; Mankoff et al., 2004)	Yes / No^Ŧ	Qualitative
28	Discovery of novel knowledge or innovative techniques	(S. R. Hanney et al., 2004; Kalucy, Jackson-Bowers, McIntyre, & Reed, 2009; Lavis et al., 2003; W. Trochim, Kane, Graham, & Pincus, 2011)	Yes / No^Ŧ	Qualitative
29	Demonstration of an efficient way of treatment	(Lavis et al., 2003; W. Trochim et al., 2011; Woolf, 2008)	Yes / No^Ŧ	Qualitative
30	Development of new research devices or products for better results	(ARC, 2018; Kalucy et al., 2009; Lavis et al., 2003; Mankoff et al., 2004; Pang et al., 2003)	Yes / No^ŦŦ	Mixed
31	Obtaining patents for new devices or products	(ARC, 2018; Kuruvilla et al., 2006; Lavis et al., 2003; Lewison, 2003; Sarli, Dubinsky, & Holmes, 2010)	How many patents are obtained?	Quantitative
32	Identification or validation of new biomarkers for better healthcare	(Lavis et al., 2003; Zerhouni, 2007)	Yes / No^Ŧ	Qualitative
33	Use of research outcomes and discoveries into the advancement of research related to animals and humans	(Pober et al., 2001; Woolf, 2008; Zerhouni, 2007)	Yes / No^Ŧ	Qualitative
34	Receiving an award on research	(Kuruvilla et al., 2006)	How many awards are received?	Quantitative
35	The increment in number and proportion of research grant submissions and awards	(ARC, 2018; Lavis et al., 2003; Lewison, 2003; Weiss, 2007)	What is the proportion of success of grant award?	Quantitative
36	Increase in the quantity of publications in high ranking journals as a research outcome	(Buxton & Hanney, 1996; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; Weiss, 2007)	How many publications are produced in high ranking journals? ^ŦŦŦ	Quantitative
37	Increase in the total impact factor gained by publishing research in high ranking journals	(ARC, 2018; Archambault & Lariviere, 2009; RAND Europe, 2006; Weiss, 2007)	How much impact factor is gained? ^ŦŦŦ	Quantitative
38	Increase in the conference papers and presentations organized on national or international levels.	(ARC, 2018; Kalucy et al., 2009; Lewison, 2003)	How many conference papers and presentations are produced? ^ŦŦŦ	Quantitative
39	Increase in the number of citations of research outcome	(ARC, 2018; Garfield, 2006; S. R. Hanney et al., 2004; Kuruvilla et al., 2006; RAND Europe, 2006; Weiss, 2007)	How many citations are obtained? ^ŦŦŦ	Quantitative
40	Increase in media appearance of researchers or research organizations for their findings and its relation to the public	(Kuruvilla et al., 2006; Lewison, 2003)	How many times appeared in media?	Quantitative
41	Popularity and acceptance of research-based knowledge and techniques in masses (e.g. change in community-based health practice or education system)	(Kalucy et al., 2009; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; Weiss, 2007)	Yes / No^Ŧ	Qualitative
42	Participation of researchers as a member of the research journal editorial board or become a journal editor	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
43	Dissemination and reach of research outcome to more audiences	(Kalucy et al., 2009; Kuruvilla et al., 2006; Weiss, 2007)	Yes / No^Ŧ	Qualitative
44	IF2-Index	(Boell & Wilson, 2010)	Index Value ^ŦŦŦ	Quantitative
45	h-Index	(Hirsch, 2005)	Index Value ^ŦŦŦ	Quantitative
46	Contemporary h-Index	(Sidiropoulos, Katsaros, & Manolopoulos, 2007)	Index Value ^ŦŦŦ	Quantitative
47	Individual h-Index	(Harzing, 2010)	Index Value ^ŦŦŦ	Quantitative
48	Hi-Index	(Zhai, Yan, & Zhu, 2013)	Index Value ^ŦŦŦ	Quantitative
49	H2-Index	(Vanclay & Bornmann, 2012)	Index Value ^ŦŦŦ	Quantitative
50	M-Quotient	(Hirsch, 2005)	Index Value ^ŦŦŦ	Quantitative
51	G-Index	(Egghe, 2006)	Index Value ^ŦŦŦ	Quantitative
52	Y-Index	(Fu & Ho, 2014)	Index Value ^ŦŦŦ	Quantitative
53	PRP-Index	(Vinkler, 2014)	Index Value ^ŦŦŦ	Quantitative
54	IFQ2A index	(Torres-Salinas, Moreno-Torres, Delgado-López-Cózar, & Herrera, 2011)	Index Value ^ŦŦŦ	Quantitative
55	DCI-Index	(Järvelin & Persson, 2008)	Index Value ^ŦŦŦ	Quantitative
56	R-& AR-Indices	(Jin, Liang, Rousseau, & Egghe, 2007)	Index Value ^ŦŦŦ	Quantitative
57	AHP Index	(Wang, Wen, & Liu, 2016)	Index Value ^ŦŦŦ	Quantitative
58	Altmetric	(A. E. Williams, 2017)	Altmetric Attention Score	Quantitative
59	STAR Metrics	(Largent & Lane, 2012)	Index Value	Quantitative
60	ResearchGate-Score	(Hoffmann, Lutz, & Meckel, 2016)	Index Value ^ŦŦŦ	Quantitative
61	Crown indicator	(Moed, De Bruin, & Van Leeuwen, 1995)	Index Value ^ŦŦŦ	Quantitative
62	Societal Quality Score	(Mostert, Ellenbroek, Meijer, van Ark, & Klasen, 2010)	Index Value	Quantitative
63	PlumX Metrics	(Lindsay, 2016)	Index Value ^ŦŦŦ	Quantitative
64	Positive reviews of creative publications and performances	(Grant, Brutscher, Kirk, Butler, & Wooding, 2010)	Yes / No^Ŧ	Qualitative
65	Non-academic publications in government reports	(Penfield, Baker, Scoble, & Wykes, 2014)	How many publications are done in government reports?	Quantitative
66	Non-academic citations in government reports	(Penfield et al., 2014)	How many citations are made in government reports?	Quantitative
67	Number of industrial contracts	(ARC, 2018)	How many industrial contracts are obtained?	Quantitative
68	Amount of industrial and academic funding	(ARC, 2018)	How much funding is secured?	Quantitative
69	Community awareness of research; Collaborative projects with end users	(S. Hanney et al., 2007)	Yes / No^Ŧ	Qualitative
70	Facilitation and participation in expert panels for research enquiries; external institution; steering committees and advisory boards	(S. Hanney et al., 2007)	Yes / No^ŦŦ	Mixed
71	Use of research outcomes, discoveries or clinical trials as a best practice	(Lewison, 2003; W. M. Trochim et al., 2008; Woolf, 2008)	Yes / No^Ŧ	Qualitative
72	Use of research outcome in efficiency and better performance of services	(Woolf, 2008)	Yes / No^Ŧ	Qualitative
73	Provision of diversified and efficient intervention and treatment options for clinicians	(Dougherty & Conway, 2008)	Yes / No^Ŧ	Qualitative
74	Improved client care	(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Mankoff et al., 2004; Pang et al., 2003; Pober et al., 2001; W. Trochim et al., 2011; Weiss, 2007; Westfall et al., 2007)	Yes / No^Ŧ	Qualitative
75	The decrease in events of work-environment mistakes	(Donaldson, Rutledge, & Ashley, 2004)	What is the decrease rate of work-environment mistakes?	Quantitative
76	Increase in the provision of training of healthcare improvement from the healthcare providers to the support staff	(S. R. Hanney et al., 2004; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)	How many support staff are trained?	Quantitative
77	Improvement in technologies and information systems for social applications	(B. Haynes & A. Haines, 1998; Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
78	Increase in training development for system improvement	(S. R. Hanney et al., 2004; Kuruvilla et al., 2006; Lewison, 2003; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)	How many trainings are developed for healthcare improvements?	Quantitative
79	Creation of prevention methods for clinical practice	(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Mankoff et al., 2004; Pang et al., 2003; Pober et al., 2001; W. Trochim et al., 2011; Weiss, 2007; Westfall et al., 2007)	Yes / No^Ŧ	Qualitative
80	Adapting evidence-based practices	(Donaldson et al., 2004; Dougherty & Conway, 2008; Grant, Cottrell, Cluzeau, & Fawcett, 2000; Kuruvilla et al., 2006; Westfall et al., 2007)	Yes / No^Ŧ	Qualitative
81	Improvement in patient outcomes	(Donaldson et al., 2004; Dougherty & Conway, 2008; Lewison, 2003; Weiss, 2007)	Yes / No^Ŧ	Qualitative
82	Improvement in health behaviours enthusiasm of patients and general masses	(Kuruvilla et al., 2006; Lewison, 2003; Woolf, 2008)	Yes / No^Ŧ	Qualitative
83	Development and promulgation of guidelines and policies	(Dougherty & Conway, 2008; Grant et al., 2000; S. R. Hanney et al., 2004; Brian Haynes & Andrew Haines, 1998; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; W. Trochim et al., 2011)	Yes / No^Ŧ	Qualitative
84	Progress in personal circumstances-based healthcare e.g. based on genetic sequencing	(Mankoff et al., 2004; Zerhouni, 2007)	Yes / No^Ŧ	Qualitative
85	Strengthening of service-client relationship	(Woolf, 2008)	Yes / No^Ŧ	Qualitative
86	Research outcome translation into medical practice for improvement	(Dougherty & Conway, 2008; Kessler & Glasgow, 2011)	Yes / No^Ŧ	Qualitative
87	Strengthening human protection through improved policies and better procedures	(Weiss, 2007)	Yes / No^Ŧ	Qualitative
88	Improvement in regulation for introducing advanced technologies, tools and techniques	(Lewison, 2003)	Yes / No^Ŧ	Qualitative
89	Compliance of ethical guidelines in research	(Kuruvilla et al., 2006; Weiss, 2007)	Yes / No^Ŧ	Qualitative
90	Development of community-based awareness	(Sarli et al., 2010)	Yes / No^Ŧ	Qualitative
91	Betterment of policies, guidelines and reimbursement systems for service providers	(Sarli et al., 2010)	Yes / No^Ŧ	Qualitative
92	Increased empowerment of service users	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
93	Support of research outcome and information for political decision and policy-making	(Buxton & Hanney, 1996; B. Haynes & A. Haines, 1998; Kalucy et al., 2009; Pang et al., 2003)	Yes / No^Ŧ	Qualitative
94	Improved public awareness about the environment and culture, Public behaviour change and advocacy; Increased literacy and numeracy rates	(Grant et al., 2010; Raftery et al., 2016)	Yes / No^Ŧ	Qualitative
95	Improvement in health literacy of health users and patients	(Kuruvilla et al., 2006; Pang et al., 2003)	Yes / No^Ŧ	Qualitative
96	Improvement in health status of health users and patients	(Dougherty & Conway, 2008; Kuruvilla et al., 2006; Weiss, 2007)	Yes / No^Ŧ	Qualitative
97	Establishment of public health, education or any other social schemes for a region	(Woolf, 2008)	Yes / No^Ŧ	Qualitative
98	The decrease in social disparities	(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Zerhouni, 2007)	Yes / No^Ŧ	Qualitative
99	Improvement in inter-organizational coordination for betterment in social sector	(Sarli et al., 2010)	Yes / No^Ŧ	Qualitative
100	Increase in planning efforts and program implementation related to social issues	(Heller & de Melo-Martín, 2009; Woolf, 2008)	Yes / No^Ŧ	Qualitative
101	Improvement in health user about health research	(Weiss, 2007)	Yes / No^Ŧ	Qualitative
102	Increased empowerment and knowledge of health users about health issues	(Kuruvilla et al., 2006; Weiss, 2007)	Yes / No^Ŧ	Qualitative
103	Better communication and perception of health users about health risks	(Kuruvilla et al., 2006; Pober et al., 2001; Weiss, 2007)	Yes / No^Ŧ	Qualitative
104	Expansion of health education, literacy and other social advantages	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
105	Improvement in Occupational Health and Safety Environment	(Raftery et al., 2016; V. Williams, Eiseman, Landree, & Adamson, 2009)	Yes / No^Ŧ	Qualitative
106	Disuse the law for obsoleting the existing method of drug approval	(Maliha, 2018)	Yes / No^Ŧ	Qualitative
107	Commercialization of new discovery, product or technology	(Kuruvilla et al., 2006; Lavis et al., 2003; Woolf, 2008)	Yes / No^Ŧ	Qualitative
108	Improvement in cost reducing techniques and effectiveness	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
109	Improvement in economic gains such as increased employment, health cost cut	(Aries & Sclar, 1998; S. R. Hanney et al., 2004; Kalucy et al., 2009; Kuruvilla et al., 2006; RAND Europe, 2006; Weiss, 2007)	Yes / No^ŦŦ	Mixed
110	Development of new job opportunities and growth in the specific economic sector or geographical region	(Aries & Sclar, 1998)	Yes / No^ŦŦ	Mixed
111	Development of medicinal products and therapeutic procedures	(S. Hanney et al., 2007; Sarli et al., 2010)	How many products or procedures are developed?	Quantitative
112	Improvement in a business environment, commercialization, technology incubation, products and processes	(Buxton & Hanney, 1996)	Yes / No^Ŧ	Qualitative
113	Reduction in work loss due to illness and increased benefits from a healthy workforce	(CAHS, 2009)	Yes / No^Ŧ	Qualitative
114	Increased Royalties, employment, Licences; creative works commissioned	(Grant et al., 2010)	Yes / No^ŦŦ	Mixed
115	Creation of new knowledge about sustainable development and environmental protection for better future of the world	(Kuruvilla et al., 2006)	Yes / No^Ŧ	Qualitative
116	Improved environmental quality and sustainability	(Engel-Cox, Van Houten, Phelps, & Rose, 2008)	Yes / No^Ŧ	Qualitative
117	Reduced emissions; regeneration or arrested degradation of natural resources	(Raftery et al., 2016)	Yes / No^ŦŦ	Mixed
118	Improved awareness of environmental impacts and legislation for protection	(CAHS, 2009)	Yes / No^Ŧ	Qualitative
119	development of mitigation methods for reducing environmental hazards and losses from natural disasters	(Grant et al., 2010)	Yes / No^ŦŦ	Mixed

Język:: Angielski

Częstotliwość wydawania:: 4 razy w roku
Dziedziny czasopisma:: Informatyka, Technologia informacyjna, Zarządzenie projektami, Bazy danych i eksploracja danych

Kanał RSS czasopisma

Evidence-based Nomenclature and Taxonomy of Research Impact Indicators

Kategoria artykułu: Research Paper

Data publikacji: 03 lip 2020

Zakres stron: 33 - 56

Otrzymano: 28 sty 2020

Przyjęty: 20 maj 2020

DOI: https://doi.org/10.2478/jdis-2020-0018

Słowa kluczowe
Research impact, Nomenclature of indicators, Scientometrics, Classification of indicators

© 2020 Mudassar Arsalan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1

Figure 2

Nomenclature of Indicator with Examples_

j_jdis-2020-0018_apptab_001_w2aab3b7c20b1b6b1ab2b2ab2Aa

Evidence-based Nomenclature and Taxonomy of Research Impact Indicators

Mudassar Arsalan

Omar Mubin

Abdullah Al Mahmud

Kategoria artykułu: Research Paper

Data publikacji: 03 lip 2020

Zakres stron: 33 - 56

Otrzymano: 28 sty 2020

Przyjęty: 20 maj 2020

DOI: https://doi.org/10.2478/jdis-2020-0018

Słowa kluczoweResearch impact, Nomenclature of indicators, Scientometrics, Classification of indicators

© 2020 Mudassar Arsalan et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1

Figure 2

Nomenclature of Indicator with Examples_

j_jdis-2020-0018_apptab_001_w2aab3b7c20b1b6b1ab2b2ab2Aa

Słowa kluczowe
Research impact, Nomenclature of indicators, Scientometrics, Classification of indicators