Acceso abierto

Evidence-based Nomenclature and Taxonomy of Research Impact Indicators

Mudassar Arsalan
Mudassar Arsalan
, 
Omar Mubin
Omar Mubin
 y 
Abdullah Al Mahmud
Abdullah Al Mahmud
   | 03 jul 2020
Journal of Data and Information Science's Cover Image
Acerca de este artículo
Artículo anterior
Artículo siguiente
Cite
Article Category: Research Paper
Publicado en línea: 03 jul 2020
Páginas: 33 - 56
Recibido: 28 ene 2020
Aceptado: 20 may 2020
DOI: https://doi.org/10.2478/jdis-2020-0018
Palabras clave
© 2020 Mudassar Arsalan et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1

Evidence-based taxonomical characteristics of indicators, (A) Scale of indicators, (B) Complexity of indicators, (C) Functions of indicators, (D) Domains of indicators, and (E) Target areas of indicators.
Evidence-based taxonomical characteristics of indicators, (A) Scale of indicators, (B) Complexity of indicators, (C) Functions of indicators, (D) Domains of indicators, and (E) Target areas of indicators.

Figure 2

Cross-constructs distribution of indicators characteristics, (A) Functional distribution of target areas in indicators, (B) Domain distribution of target areas in indicators, and (C) Functional distribution of domains in indicators.
Cross-constructs distribution of indicators characteristics, (A) Functional distribution of target areas in indicators, (B) Domain distribution of target areas in indicators, and (C) Functional distribution of domains in indicators.

Nomenclature of Indicator with Examples.

Functions (F)
Improvement / Addition / ReductionThis function of indicator explains the addition or enhancement of an existing phenomenon in quantitative or qualitative form. (Example: Improvement in economic gains such as increased employment, health cost cut (Weiss, 2007))
CreationThis function of indicator focuses on the creativity in the form of the development of new knowledge, theory, technique, method, technology, approach, opportunity or any workflow. (Example: Creation of prevention methods for clinical practice (Trochim et al., 2011))
RecognitionThis function explains the recognition of effort in the form of outstanding quality by the peers or experts such as in the form of awards, promotions, meritorious selection and work showcasing etc. This recognition can be of the research, the researcher or the research institute. (Example: Receiving an award on research (Kuruvilla et al., 2006))
Obsoleting / ReplacingThis function elaborates the policy, law, regulation to obsolete or disuse the existing phenomena to overcome the future negative impacts. (Example: Change in law to obsolete the existing method of drug approval (Maliha, 2018))
Context (C)
Target (t)Contextual targets in research impact science include knowledge, service, policy, law, guideline, system, technology, procedure, method, framework, workflow, publication, patent, product, stakeholder, citation, literature gaps, intellectual challenges, scholarly issues, relationships, collaborations, and networks etc. These are the key areas but usually partial in contextual understanding.
Domain (d)The contextual domain is the main area or field of interest of the indicator system designer such as health, education, economy, environment, academia, medical science, chemistry, history, multidisciplinary etc. The main body of knowledge and elaboration of indicators are always from the domain language. The domain is the main component of the indicator, which specialised the context and application of the indicator. However, the level of the domain is subject to the interest and perspective of impact evaluator.

j.jdis-2020-0018.apptab.001.w2aab3b7c20b1b6b1ab2b2ab2Aa

RefIndicatorReferenceMeasureQualitative/Quantitative
1Identification of research gaps, questions and new research dimension(Heller & de Melo-Martín, 2009; Kuruvilla, Mays, Pleasant, & Walt, 2006; W. M. Trochim, Marcus, Masse, Moser, & Weld, 2008; Weiss, 2007)Yes / NoŦQualitative
2Development of a new technique for data collection and new data(Heller & de Melo-Martín, 2009; Sung et al., 2003)Yes / NoŦQualitative
3Creation of a research method or extension of existing by involving new approach and technique(Kuruvilla et al., 2006; W. M. Trochim et al., 2008)Yes / NoŦQualitative
4Defining the concept and subject vocabulary in a more comprehensive way(Mankoff, Brander, Ferrone, & Marincola, 2004; W. M. Trochim et al., 2008)Yes / NoŦQualitative
5Formation of research groups and collaborate in multidimensional research(S. R. Hanney, Grant, Wooding, & Buxton, 2004; Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; W. M. Trochim et al., 2008)Yes / NoŦQualitative
6Recruitment of skilled researchers../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_68 (Heller & de Melo-Martín, 2009)How many researchers are recruited?Quantitative
7Development of communities of science, new grant programmes; replication and new research(S. Hanney, Buxton, Green, Coulson, & Raftery, 2007)Yes / NoŦŦMixed
8Effective planning and addressing future research../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_51 (Gordon & Meadows, 1981)Yes / NoŦQualitative
9Research capacity building for an individual or a group of researchers(Buxton & Hanney, 1996; Raftery, Hanney, Greenhalgh, Glover, & Blatch-Jones, 2016)How many researchers are trained?Quantitative
10Preparing a better procedure for researchers induction(Heller & de Melo-Martín, 2009; Sung et al., 2003)Yes / NoŦQualitative
11Improvement in ethical approval processes for better decisions and timeliness(Pober, Neuhauser, & Pober, 2001; Sung et al., 2003)Yes / NoŦQualitative
12Formation of new research teams and projects(Pober et al., 2001)How many projects and teams are established?Quantitative
13Successful completion of ongoing research with the achievement of set targets(Weiss, 2007)Yes / NoŦŦMixed
14Retention of research team by involving in productivity and future research(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Nathan, 2002)How many members are retained?Quantitative
15Advancement in numbers and quality of research and research teams(Nathan, 2002; Pober et al., 2001; W. M. Trochim et al., 2008; Weiss, 2007)Yes / NoŦŦMixed
16Enhancement of research process, behaviour and procedural protocols(Heller & de Melo-Martín, 2009; Pober et al., 2001; Sung et al., 2003)Yes / NoŦQualitative
17Recognition and leadership of researchers in the research domain(Kuruvilla et al., 2006; Pober et al., 2001)Yes / NoŦQualitative
18Improvement of research communication between researchers and research organizations(Heller & de Melo-Martín, 2009; Mankoff et al., 2004)Yes / NoŦQualitative
19Serving of research staff on a higher level in more advanced organizations at national and international level(Kuruvilla et al., 2006; Sung et al., 2003)Yes / NoŦQualitative
20Improvement in research culture and overall environment(Heller & de Melo-Martín, 2009; Kessler & Glasgow, 2011; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)Yes / NoŦQualitative
21Identification and overcoming of the research process constraints(Heller & de Melo-Martín, 2009; Pober et al., 2001)Yes / NoŦQualitative
22Improved willingness and tangible measures for practice-based and applied research(Westfall, Mold, & Fagnan, 2007)Yes / NoŦQualitative
23Development of improved analytical methods for existing data(Kessler & Glasgow, 2011; Kuruvilla et al., 2006; W. M. Trochim et al., 2008; Weiss, 2007)Yes / NoŦQualitative
24Improvement in multi-disciplinary research methods(Kuruvilla et al., 2006)Yes / NoŦQualitative
25Creation of methods for cross domains results in interpretation and synthesis(Kuruvilla et al., 2006; Pang et al., 2003)Yes / NoŦQualitative
26Embracing the innovative methods for measuring the research outcome(Dougherty & Conway, 2008; W. M. Trochim et al., 2008)Yes / NoŦQualitative
27Discovery of new or advanced research findings(Lavis, Ross, McLeod, & Gildiner, 2003; Mankoff et al., 2004)Yes / NoŦQualitative
28Discovery of novel knowledge or innovative techniques(S. R. Hanney et al., 2004; Kalucy, Jackson-Bowers, McIntyre, & Reed, 2009; Lavis et al., 2003; W. Trochim, Kane, Graham, & Pincus, 2011)Yes / NoŦQualitative
29Demonstration of an efficient way of treatment(Lavis et al., 2003; W. Trochim et al., 2011; Woolf, 2008)Yes / NoŦQualitative
30Development of new research devices or products for better results(ARC, 2018; Kalucy et al., 2009; Lavis et al., 2003; Mankoff et al., 2004; Pang et al., 2003)Yes / NoŦŦMixed
31Obtaining patents for new devices or products(ARC, 2018; Kuruvilla et al., 2006; Lavis et al., 2003; Lewison, 2003; Sarli, Dubinsky, & Holmes, 2010)How many patents are obtained?Quantitative
32Identification or validation of new biomarkers for better healthcare(Lavis et al., 2003; Zerhouni, 2007)Yes / NoŦQualitative
33Use of research outcomes and discoveries into the advancement of research related to animals and humans(Pober et al., 2001; Woolf, 2008; Zerhouni, 2007)Yes / NoŦQualitative
34Receiving an award on research(Kuruvilla et al., 2006)How many awards are received?Quantitative
35The increment in number and proportion of research grant submissions and awards(ARC, 2018; Lavis et al., 2003; Lewison, 2003; Weiss, 2007)What is the proportion of success of grant award?Quantitative
36Increase in the quantity of publications in high ranking journals as a research outcome(Buxton & Hanney, 1996; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; Weiss, 2007)How many publications are produced in high ranking journals? ŦŦŦQuantitative
37Increase in the total impact factor gained by publishing research in high ranking journals(ARC, 2018; Archambault & Lariviere, 2009; RAND Europe, 2006; Weiss, 2007)How much impact factor is gained? ŦŦŦQuantitative
38Increase in the conference papers and presentations organized on national or international levels.(ARC, 2018; Kalucy et al., 2009; Lewison, 2003)How many conference papers and presentations are produced? ŦŦŦQuantitative
39Increase in the number of citations of research outcome(ARC, 2018; Garfield, 2006; S. R. Hanney et al., 2004; Kuruvilla et al., 2006; RAND Europe, 2006; Weiss, 2007)How many citations are obtained? ŦŦŦQuantitative
40Increase in media appearance of researchers or research organizations for their findings and its relation to the public(Kuruvilla et al., 2006; Lewison, 2003)How many times appeared in media?Quantitative
41Popularity and acceptance of research-based knowledge and techniques in masses (e.g. change in community-based health practice or education system)(Kalucy et al., 2009; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; Weiss, 2007)Yes / NoŦQualitative
42Participation of researchers as a member of the research journal editorial board or become a journal editor(Kuruvilla et al., 2006)Yes / NoŦQualitative
43Dissemination and reach of research outcome to more audiences(Kalucy et al., 2009; Kuruvilla et al., 2006; Weiss, 2007)Yes / NoŦQualitative
44IF2-Index(Boell & Wilson, 2010)Index Value ŦŦŦQuantitative
45h-Index(Hirsch, 2005)Index Value ŦŦŦQuantitative
46Contemporary h-Index(Sidiropoulos, Katsaros, & Manolopoulos, 2007)Index Value ŦŦŦQuantitative
47Individual h-Index(Harzing, 2010)Index Value ŦŦŦQuantitative
48Hi-Index(Zhai, Yan, & Zhu, 2013)Index Value ŦŦŦQuantitative
49H2-Index(Vanclay & Bornmann, 2012)Index Value ŦŦŦQuantitative
50M-Quotient(Hirsch, 2005)Index Value ŦŦŦQuantitative
51G-Index(Egghe, 2006)Index Value ŦŦŦQuantitative
52Y-Index(Fu & Ho, 2014)Index Value ŦŦŦQuantitative
53PRP-Index(Vinkler, 2014)Index Value ŦŦŦQuantitative
54IFQ2A index(Torres-Salinas, Moreno-Torres, Delgado-López-Cózar, & Herrera, 2011)Index Value ŦŦŦQuantitative
55DCI-Index(Järvelin & Persson, 2008)Index Value ŦŦŦQuantitative
56R-& AR-Indices(Jin, Liang, Rousseau, & Egghe, 2007)Index Value ŦŦŦQuantitative
57AHP Index(Wang, Wen, & Liu, 2016)Index Value ŦŦŦQuantitative
58Altmetric(A. E. Williams, 2017)Altmetric Attention ScoreQuantitative
59STAR Metrics(Largent & Lane, 2012)Index ValueQuantitative
60ResearchGate-Score(Hoffmann, Lutz, & Meckel, 2016)Index Value ŦŦŦQuantitative
61Crown indicator(Moed, De Bruin, & Van Leeuwen, 1995)Index Value ŦŦŦQuantitative
62Societal Quality Score(Mostert, Ellenbroek, Meijer, van Ark, & Klasen, 2010)Index ValueQuantitative
63PlumX Metrics(Lindsay, 2016)Index Value ŦŦŦQuantitative
64Positive reviews of creative publications and performances(Grant, Brutscher, Kirk, Butler, & Wooding, 2010)Yes / NoŦQualitative
65Non-academic publications in government reports(Penfield, Baker, Scoble, & Wykes, 2014)How many publications are done in government reports?Quantitative
66Non-academic citations in government reports(Penfield et al., 2014)How many citations are made in government reports?Quantitative
67Number of industrial contracts(ARC, 2018)How many industrial contracts are obtained?Quantitative
68Amount of industrial and academic funding(ARC, 2018)How much funding is secured?Quantitative
69Community awareness of research; Collaborative projects with end users(S. Hanney et al., 2007)Yes / NoŦQualitative
70Facilitation and participation in expert panels for research enquiries; external institution; steering committees and advisory boards(S. Hanney et al., 2007)Yes / NoŦŦMixed
71Use of research outcomes, discoveries or clinical trials as a best practice(Lewison, 2003; W. M. Trochim et al., 2008; Woolf, 2008)Yes / NoŦQualitative
72Use of research outcome in efficiency and better performance of services(Woolf, 2008)Yes / NoŦQualitative
73Provision of diversified and efficient intervention and treatment options for clinicians(Dougherty & Conway, 2008)Yes / NoŦQualitative
74Improved client care(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Mankoff et al., 2004; Pang et al., 2003; Pober et al., 2001; W. Trochim et al., 2011; Weiss, 2007; Westfall et al., 2007)Yes / NoŦQualitative
75The decrease in events of work-environment mistakes(Donaldson, Rutledge, & Ashley, 2004)What is the decrease rate of work-environment mistakes?Quantitative
76Increase in the provision of training of healthcare improvement from the healthcare providers to the support staff(S. R. Hanney et al., 2004; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)How many support staff are trained?Quantitative
77Improvement in technologies and information systems for social applications(B. Haynes & A. Haines, 1998; Kuruvilla et al., 2006)Yes / NoŦQualitative
78Increase in training development for system improvement(S. R. Hanney et al., 2004; Kuruvilla et al., 2006; Lewison, 2003; Mankoff et al., 2004; Pober et al., 2001; Sung et al., 2003)How many trainings are developed for healthcare improvements?Quantitative
79Creation of prevention methods for clinical practice(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Mankoff et al., 2004; Pang et al., 2003; Pober et al., 2001; W. Trochim et al., 2011; Weiss, 2007; Westfall et al., 2007)Yes / NoŦQualitative
80Adapting evidence-based practices(Donaldson et al., 2004; Dougherty & Conway, 2008; Grant, Cottrell, Cluzeau, & Fawcett, 2000; Kuruvilla et al., 2006; Westfall et al., 2007)Yes / NoŦQualitative
81Improvement in patient outcomes(Donaldson et al., 2004; Dougherty & Conway, 2008; Lewison, 2003; Weiss, 2007)Yes / NoŦQualitative
82Improvement in health behaviours enthusiasm of patients and general masses(Kuruvilla et al., 2006; Lewison, 2003; Woolf, 2008)Yes / NoŦQualitative
83Development and promulgation of guidelines and policies(Dougherty & Conway, 2008; Grant et al., 2000; S. R. Hanney et al., 2004; Brian Haynes & Andrew Haines, 1998; Kuruvilla et al., 2006; Lewison, 2003; Pang et al., 2003; W. Trochim et al., 2011)Yes / NoŦQualitative
84Progress in personal circumstances-based healthcare e.g. based on genetic sequencing(Mankoff et al., 2004; Zerhouni, 2007)Yes / NoŦQualitative
85Strengthening of service-client relationship(Woolf, 2008)Yes / NoŦQualitative
86Research outcome translation into medical practice for improvement(Dougherty & Conway, 2008; Kessler & Glasgow, 2011)Yes / NoŦQualitative
87Strengthening human protection through improved policies and better procedures(Weiss, 2007)Yes / NoŦQualitative
88Improvement in regulation for introducing advanced technologies, tools and techniques(Lewison, 2003)Yes / NoŦQualitative
89Compliance of ethical guidelines in research(Kuruvilla et al., 2006; Weiss, 2007)Yes / NoŦQualitative
90Development of community-based awareness(Sarli et al., 2010)Yes / NoŦQualitative
91Betterment of policies, guidelines and reimbursement systems for service providers(Sarli et al., 2010)Yes / NoŦQualitative
92Increased empowerment of service users(Kuruvilla et al., 2006)Yes / NoŦQualitative
93Support of research outcome and information for political decision and policy-making(Buxton & Hanney, 1996; B. Haynes & A. Haines, 1998; Kalucy et al., 2009; Pang et al., 2003)Yes / NoŦQualitative
94Improved public awareness about the environment and culture, Public behaviour change and advocacy; Increased literacy and numeracy rates(Grant et al., 2010; Raftery et al., 2016)Yes / NoŦQualitative
95Improvement in health literacy of health users and patients(Kuruvilla et al., 2006; Pang et al., 2003)Yes / NoŦQualitative
96Improvement in health status of health users and patients(Dougherty & Conway, 2008; Kuruvilla et al., 2006; Weiss, 2007)Yes / NoŦQualitative
97Establishment of public health, education or any other social schemes for a region(Woolf, 2008)Yes / NoŦQualitative
98The decrease in social disparities(Heller & de Melo-Martín, 2009; Kuruvilla et al., 2006; Zerhouni, 2007)Yes / NoŦQualitative
99Improvement in inter-organizational coordination for betterment in social sector(Sarli et al., 2010)Yes / NoŦQualitative
100Increase in planning efforts and program implementation related to social issues(Heller & de Melo-Martín, 2009; Woolf, 2008)Yes / NoŦQualitative
101Improvement in health user about health research(Weiss, 2007)Yes / NoŦQualitative
102Increased empowerment and knowledge of health users about health issues(Kuruvilla et al., 2006; Weiss, 2007)Yes / NoŦQualitative
103Better communication and perception of health users about health risks(Kuruvilla et al., 2006; Pober et al., 2001; Weiss, 2007)Yes / NoŦQualitative
104Expansion of health education, literacy and other social advantages(Kuruvilla et al., 2006)Yes / NoŦQualitative
105Improvement in Occupational Health and Safety Environment(Raftery et al., 2016; V. Williams, Eiseman, Landree, & Adamson, 2009)Yes / NoŦQualitative
106Disuse the law for obsoleting the existing method of drug approval(Maliha, 2018)Yes / NoŦQualitative
107Commercialization of new discovery, product or technology(Kuruvilla et al., 2006; Lavis et al., 2003; Woolf, 2008)Yes / NoŦQualitative
108Improvement in cost reducing techniques and effectiveness(Kuruvilla et al., 2006)Yes / NoŦQualitative
109Improvement in economic gains such as increased employment, health cost cut(Aries & Sclar, 1998; S. R. Hanney et al., 2004; Kalucy et al., 2009; Kuruvilla et al., 2006; RAND Europe, 2006; Weiss, 2007)Yes / NoŦŦMixed
110Development of new job opportunities and growth in the specific economic sector or geographical region(Aries & Sclar, 1998)Yes / NoŦŦMixed
111Development of medicinal products and therapeutic procedures(S. Hanney et al., 2007; Sarli et al., 2010)How many products or procedures are developed?Quantitative
112Improvement in a business environment, commercialization, technology incubation, products and processes(Buxton & Hanney, 1996)Yes / NoŦQualitative
113Reduction in work loss due to illness and increased benefits from a healthy workforce(CAHS, 2009)Yes / NoŦQualitative
114Increased Royalties, employment, Licences; creative works commissioned(Grant et al., 2010)Yes / NoŦŦMixed
115Creation of new knowledge about sustainable development and environmental protection for better future of the world(Kuruvilla et al., 2006)Yes / NoŦQualitative
116Improved environmental quality and sustainability(Engel-Cox, Van Houten, Phelps, & Rose, 2008)Yes / NoŦQualitative
117Reduced emissions; regeneration or arrested degradation of natural resources(Raftery et al., 2016)Yes / NoŦŦMixed
118Improved awareness of environmental impacts and legislation for protection(CAHS, 2009)Yes / NoŦQualitative
119development of mitigation methods for reducing environmental hazards and losses from natural disasters(Grant et al., 2010)Yes / NoŦŦMixed
eISSN:
2543-683X
Idioma:
Inglés
Calendario de la edición:
4 veces al año
Temas de la revista:
Computer Sciences, Information Technology, Project Management, Databases and Data Mining
RSS Feed de revista