Research Impact (RI) is a broad topic of scientometrics to support the progress of science and monitoring the influence of efforts made by the government, institutions, societies, programs, and individual researchers. There are several documented and popular RI assessment methods developed by individuals and organisations for evaluating the research of a particular programme or general-purpose. This intent has created the diversity in evaluation methods, frameworks and scope. Some approaches focus only on the impacts related to academic recognition and use, such as Bibliometric Measures. However, the growing technology, educational networking, effective and targeted research strategies, and regular monitoring of RI are reducing the gap between the research producers and consumers. As a result, the horizon of RI is expanding and covering other areas of impacts such as on economy, society, and environment.
Many individuals and organizations have introduced measures and indicators for assessing the RI. Nevertheless, due to diversity in nature and scale of RI, not a single method is considered robust and complete (Vinkler, 2010). Therefore, new measures and indicators are being introduced on time to time basis according to the interest and availability of resources of the method designers (Canadian Academy of Health Sciences, 2009). Additionally, higher availability of national and international funding for health sciences is critically influencing the science of RI assessment (Heller & de Melo-Martín, 2009). It means there are more indicators, measures, and frameworks for health-related research than any other areas of science. Resultantly, there is a considerable gap available for generalizability and transformability of health-related efforts to rest of the science.
At large, this study aims to discover the evidence-based diversity of RI indicators and to develop a method. In this regard, Nomenclature of RI indicators is developed based on divide and rule principle to achieve the objective. Additionally, taxonomical analysis is presented based on the primary components of nomenclature. This effort is a step forward to develop a robust and inclusive RI assessment method. The concept of this paper was initially presented in the 17th International Conference on Scientometrics and Informetrics (ISSI 2019) in the form of a poster (Arsalan, Mubin, & Al Mahmud, 2019).
As per the broader understanding, metrics and indicators are different. It is clear from the semantics as “indicator” indicates the research impact, whereas metrics are the measurement of research impact. According to P. Vinkler (2010), an indicator should be read as a meaningful representative, which indicate the performance of a system as per its design objective. Metrics, on the other side, provide additional quantitative information about the impact of a system (Lewison, 2003). However, diversity of effects can create a problematic situation to measure all aspects quantitatively. Therefore, pragmatically indicators have the potential to illustrate useful and broader impacts as compared to the metrics (Vinkler, 2010).
There are multiple reasons for diversity in research impact indicators. For instance, Bennett et al. (2016) explained 10-points criteria for making research impact indicators from a technical and contextual point of view. In this regard, indicators should be specific, validated, reliable, comparable, substantial, accessible, acceptable, appropriate, useable, and feasible. Consequently, diversity in criteria required range of indicators to fulfil the conditions in diversified context. Canadian Academy of Health Sciences (2009) explained another reason for this diversity, i.e. strategy of indicator selection. This strategy describes three basic principles.
The indicator should answer the specific question of evaluation
The indicator should satisfy the level of aggregation
The indicator should be read with other indicators to complement the strength of evaluation
In other words, every indicator only explains the impact of research in minimal dimension, covers a very specific level of aggregation, and has a very limited power of defining the research impact (REF, 2012). As a result, we need a bundle of indicators that fulfil the strategic requirements of evaluators.
“Nomenclature” is a combination of two Latin words viz. “nomen” means “name” and “calare” means “to call”. It is a scientific process in any discipline to assign the names to essential components according to the predefined rules and standards (Hayat, 2014). Generally, these rules are outlined in the form of a classification scheme. Therefore, for nomenclature, the classification system is highly significant. Longabaugh et al. (1983) introduced the problem-focused nomenclature in medical science, which is a coding system with a specific objective. They argued that the problem-focused approach provides better control for organizing and problem management. The similar concept can be applied in any branch of science to organize the objects concerning a problem-focused classification system.
Classification and organization of research impact indicators are not new (Vinkler, 2010). However, the nomenclature or taxonomy approach is missing. Therefore, standardization is globally missing. Every effort of research impact assessment distinctly organized the indicators according to the technical and contextual requirements. Nonetheless, based on the context, the classification scheme of indicators can be arranged in four groups.
Impact Categories and Domains
Impact Time and Pathways
Impact in Specific Dimension
In many research impact assessment methods, the adopted organization of impact indicators is based on impact categories and domains. These methods are wide-scope and open to select indicators in any of their classes (Bernstein et al., 2006). Payback framework for assessing the impact of health research is one of the classical methods falls under this group, for instance (Buxton & Hanney, 1996). It was developed by the Health Economics Research Group at Brunel University in 1996 by Buxton and Hanney (1996). It organizes the indicators in multi-dimensional categories including knowledge, research benefits, political, and administrative benefits, health sector benefits, and broader economic benefits.
The second group that follows the impact time and pathways are based on the concept of output and outcome. The understanding of the difference between output and outcome was first explained by United Way of America (1996) in the form of logic modelling. This model explicitly defines inputs, process and outputs in the form of resources, activities and products, respectively. Whereas, the outcome is a benefit to the population of interest. Weiss (2007) split the outcomes of health research into initial, intermediate, and long-term impacts. This time-bound approach represents a sequence or a chain of effects. For instance, awareness of new research in decision-making community is an initial outcome. That awareness can lead to a change in clinical practice as an intermediate outcome. Ultimately, the long-term outcome is the improvement in the health of patients.
The third approach is exclusive. Many organizations and individual researchers are keen to know the impact of research only in one area in depth. One example is the monetary value approach presented by Deloitte Access Economics (2011). In this approach, all indicators and measures are solely related to the economic impacts of research. Some other methods are organization-specific where a scoring system is limited in scope and developed in a local context. We cannot fit them in any above mentioned organized structure, for instance, The Wellcome Trust's Assessment Framework (Wellcome Trust, 2009), Matrix Scoring System (Wiegers et al., 2015), and Royal Netherlands Academy of Arts and Sciences Approach (VSNU, KNAW, & NWO, 2009).
Although the organizations of indicators within a research impact framework has been a mandatory part of every evaluation method, there is still a need to organize the indicators based on criteria and rules. A classic example of diversity and heterogeneity can be seen in REF (2012) where more than 100 indicators are applied based on subject domains and target areas of socio-economic interest. There is still a need to adopt a mechanism where these indicators can be generalized and transformed on taxonomical structures.
We systematically explored the literature databases, including Scopus, WebMD, ACM DL, IEEE Xplore, Web of Science and Google Scholar to collect research articles providing RI assessment indicators and methods. In many cases, organizations published the frameworks and guidelines in the form of technical reports; therefore, grey literature was also considered.
Multiple combinations of literature-searching keywords were used with their synonyms. These include but not limited to the “research impact”, “research productivity”, “research quality”, “research impact indicators”, “research impact assessment”, “research impact assessment method”, “research impact assessment framework”, “scientometric indicators”, “bibliometric indicators”, “economic indicators”, “social indicators”, and “environmental indicators”. The purpose of using a combination of these keywords was to identify theoretical or applied studies related to the research impact assessment. In theoretical or conceptual studies, we found the constructs and mechanisms of research impact assessment methods applied studies provided the demonstration of assessment methods in the form of case studies. We also found some review articles, which provided a comparison of different RI assessment approaches. However, in this study, we mainly focused on the preparation of RI indicators. Due to using multiple combinations of keywords and databases, we found the significant repetition of the same studies, which we removed with the help of EndNote software. In this study, we extracted indicators from conceptual studies. We used NVivo 12 software for annotation and coding. For deciphering the nomenclature, indicators were disintegrated based on their lexical and conceptual structures as discussed in the Results section. For improving the result of coding, inter-coder reliability was applied on 10% of data and conflicts were resolved with the help of discussion.
The base of the cognitive structure of defined nomenclature in this study is the “every indicator is a contextual-function to explain the impact”. The primary constructs of an indicator are function and context. Function refers to the “correspondence”, “dependence relation”, “rule”, “operation”, “formula” or “representation” as defined by Vinner and Dreyfus (1989). It explains the relationship between the two domains “research” and “impact”. In other words, impact (y) is a function of research (x), i.e. y=f(x). At large, in scientometrics understanding, the functional operation can be “improvement”, “recognition”, “reduction”, “replacement” etc. (see Table 1 for examples). The indicator is a subjective measure of a system-dependent phenomenon which is always described in its contextual understanding by a system designer (Vinkler, 2010). Therefore, the indicator's function is always applied in a specific context. For instance, “improvement in patient care system”, in this indicator, the patient care system represents the context of the healthcare system, and it is critically important for researchers, funders, institutes and support organisations related to the health sciences (Trochim et al., 2011).
Structure of Indicator (I) = F + C Where, F = Function, and C = Context Whereas C = t + d Where, t = target area, and d = impact domain
Structure of Indicator (I) = F + C
Where, F = Function, and C = Context
Whereas C = t + d
Where, t = target area, and d = impact domain
Nomenclature of Indicator with Examples.
|Improvement / Addition / Reduction|
|Obsoleting / Replacing|
The result of the literature search is more than one thousand studies (1,152), where research impact was published in the form of theoretical papers, case studies and review articles. However, after excluding studies where research impact was assessed in case-studies by using any method developed by elsewhere, only 36 conceptual studies were left. In the conceptual studies, we found out more than 500 research impact indicators. For this study, we selected 119 indicators for preparing the nomenclature (see Appendix 1).
In many cases, an indicator is self-explanatory and well written in a proper construct-based structure such as
This contextual nomenclature of indicators allows focussing on context and function irrespective of the selection of the words and lexical structure of the indicator. Additionally, it strengthens the idea of contextual generalisability, which is very helpful in extending the applications and scope of the indicators. For example, in
In analyzed indicators, most of the indicators are functionally related to the improvements in the current state of affairs (63%), mainly focused on future research, services and methods (Figure 1). However, recognition of research (23%) in the form of bibliometric, rewards and other citations is also considerably highlighted in the literature-based list of indicators. Creativity and development (14%) are also the prevailing influence of research, which is reflected in indicators mentioning the creation of new knowledge, technique, research teams, drugs etc. More than half (59%) of the indicators attempt to explore the impact in the academic domain (Figure 2), e.g. Where and how the research is recognised? What knowledge, methods and collaborations are formed? What challenges, issues and gaps are addressed? Knowledge domains related to the social systems and services are second in coverage (26%) that primarily focus on the healthcare, education and justice systems. Economic policies and services also have a good share (11%) in literature-based indicators. Although, during the last two decades, the impact of research on improving the environment and sustainability has also emerged in various indicators, its representation is quite low.
In this study, 119 indicators were interpreted and coded for nomenclature and taxonomy. However, the inclusion of more indicators may change the results of classification. Another aspect which may affect the outcome of the study is consistency in interpretation and coding of indicators. Although it was improved by using the intercoder reliability method on 10% indicators, rule-based text mining techniques may improve the results.
The study categorized the research impact indicators based on their characteristics and scope. Furthermore, a concept of evidence-based nomenclature of research impact indicator has been introduced to generalize and transform the indicators. For building nomenclature and classification, one hundred and nineteen indicators were selected and coded in NVivo software. The nomenclature was developed based on the principle “every indicator is a contextual-function to explain the impact”. Every indicator was disintegrated in three parts (essential ingredients of nomenclature), i.e. Function, Domain, and Target Areas. It is observed that in literature, the primary functions of research impact indicators are improvement, recognition and creation/development. The focus of research impact indicators in literature is more towards the academic domain, whereas the environment/sustainability domain is least considered. As a result, research impact related to the research aspects is considered the most. Other target areas include system and services, methods and procedures, networking, planning, policy development, economic aspects and commercialisation etc.
The study provided a novel approach in scientometrics for generalizability and transformability of research impact indicators. It explored the diversity of indicators and demonstrated the generalization based on fundamental constructs, i.e. function, domain and target area. As a result, a research impact indicator can be modified and applied to multiple research disciplines.
Nomenclature of Indicator with Examples.
|Improvement / Addition / Reduction|
|Obsoleting / Replacing|
|1||Identification of research gaps, questions and new research dimension||(||Yes / No||Qualitative|
|2||Development of a new technique for data collection and new data||(||Yes / No||Qualitative|
|3||Creation of a research method or extension of existing by involving new approach and technique||(||Yes / No||Qualitative|
|4||Defining the concept and subject vocabulary in a more comprehensive way||(||Yes / No||Qualitative|
|5||Formation of research groups and collaborate in multidimensional research||(||Yes / No||Qualitative|
|6||Recruitment of skilled researchers||../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_68 (||How many researchers are recruited?||Quantitative|
|7||Development of communities of science, new grant programmes; replication and new research||(||Yes / No||Mixed|
|8||Effective planning and addressing future research||../../../../Google Drive/Working/001 PhD Work/00103 Paper 1/Drafting/Indicators v2.xlsx - RANGE!_ENREF_51 (||Yes / No||Qualitative|
|9||Research capacity building for an individual or a group of researchers||(||How many researchers are trained?||Quantitative|
|10||Preparing a better procedure for researchers induction||(||Yes / No||Qualitative|
|11||Improvement in ethical approval processes for better decisions and timeliness||(||Yes / No||Qualitative|
|12||Formation of new research teams and projects||(||How many projects and teams are established?||Quantitative|
|13||Successful completion of ongoing research with the achievement of set targets||(||Yes / No||Mixed|
|14||Retention of research team by involving in productivity and future research||(||How many members are retained?||Quantitative|
|15||Advancement in numbers and quality of research and research teams||(||Yes / No||Mixed|
|16||Enhancement of research process, behaviour and procedural protocols||(||Yes / No||Qualitative|
|17||Recognition and leadership of researchers in the research domain||(||Yes / No||Qualitative|
|18||Improvement of research communication between researchers and research organizations||(||Yes / No||Qualitative|
|19||Serving of research staff on a higher level in more advanced organizations at national and international level||(||Yes / No||Qualitative|
|20||Improvement in research culture and overall environment||(||Yes / No||Qualitative|
|21||Identification and overcoming of the research process constraints||(||Yes / No||Qualitative|
|22||Improved willingness and tangible measures for practice-based and applied research||(||Yes / No||Qualitative|
|23||Development of improved analytical methods for existing data||(||Yes / No||Qualitative|
|24||Improvement in multi-disciplinary research methods||(||Yes / No||Qualitative|
|25||Creation of methods for cross domains results in interpretation and synthesis||(||Yes / No||Qualitative|
|26||Embracing the innovative methods for measuring the research outcome||(||Yes / No||Qualitative|
|27||Discovery of new or advanced research findings||(||Yes / No||Qualitative|
|28||Discovery of novel knowledge or innovative techniques||(||Yes / No||Qualitative|
|29||Demonstration of an efficient way of treatment||(||Yes / No||Qualitative|
|30||Development of new research devices or products for better results||(||Yes / No||Mixed|
|31||Obtaining patents for new devices or products||(||How many patents are obtained?||Quantitative|
|32||Identification or validation of new biomarkers for better healthcare||(||Yes / No||Qualitative|
|33||Use of research outcomes and discoveries into the advancement of research related to animals and humans||(||Yes / No||Qualitative|
|34||Receiving an award on research||(||How many awards are received?||Quantitative|
|35||The increment in number and proportion of research grant submissions and awards||(||What is the proportion of success of grant award?||Quantitative|
|36||Increase in the quantity of publications in high ranking journals as a research outcome||(||How many publications are produced in high ranking journals? ||Quantitative|
|37||Increase in the total impact factor gained by publishing research in high ranking journals||(||How much impact factor is gained? ||Quantitative|
|38||Increase in the conference papers and presentations organized on national or international levels.||(||How many conference papers and presentations are produced? ||Quantitative|
|39||Increase in the number of citations of research outcome||(||How many citations are obtained? ||Quantitative|
|40||Increase in media appearance of researchers or research organizations for their findings and its relation to the public||(||How many times appeared in media?||Quantitative|
|41||Popularity and acceptance of research-based knowledge and techniques in masses (e.g. change in community-based health practice or education system)||(||Yes / No||Qualitative|
|42||Participation of researchers as a member of the research journal editorial board or become a journal editor||(||Yes / No||Qualitative|
|43||Dissemination and reach of research outcome to more audiences||(||Yes / No||Qualitative|
|44||IF2-Index||(||Index Value ||Quantitative|
|45||h-Index||(||Index Value ||Quantitative|
|46||Contemporary h-Index||(||Index Value ||Quantitative|
|47||Individual h-Index||(||Index Value ||Quantitative|
|48||Hi-Index||(||Index Value ||Quantitative|
|49||H2-Index||(||Index Value ||Quantitative|
|50||M-Quotient||(||Index Value ||Quantitative|
|51||G-Index||(||Index Value ||Quantitative|
|52||Y-Index||(||Index Value ||Quantitative|
|53||PRP-Index||(||Index Value ||Quantitative|
|54||IFQ2A index||(||Index Value ||Quantitative|
|55||DCI-Index||(||Index Value ||Quantitative|
|56||R-& AR-Indices||(||Index Value ||Quantitative|
|57||AHP Index||(||Index Value ||Quantitative|
|58||Altmetric||(||Altmetric Attention Score||Quantitative|
|59||STAR Metrics||(||Index Value||Quantitative|
|60||ResearchGate-Score||(||Index Value ||Quantitative|
|61||Crown indicator||(||Index Value ||Quantitative|
|62||Societal Quality Score||(||Index Value||Quantitative|
|63||PlumX Metrics||(||Index Value ||Quantitative|
|64||Positive reviews of creative publications and performances||(||Yes / No||Qualitative|
|65||Non-academic publications in government reports||(||How many publications are done in government reports?||Quantitative|
|66||Non-academic citations in government reports||(||How many citations are made in government reports?||Quantitative|
|67||Number of industrial contracts||(||How many industrial contracts are obtained?||Quantitative|
|68||Amount of industrial and academic funding||(||How much funding is secured?||Quantitative|
|69||Community awareness of research; Collaborative projects with end users||(||Yes / No||Qualitative|
|70||Facilitation and participation in expert panels for research enquiries; external institution; steering committees and advisory boards||(||Yes / No||Mixed|
|71||Use of research outcomes, discoveries or clinical trials as a best practice||(||Yes / No||Qualitative|
|72||Use of research outcome in efficiency and better performance of services||(||Yes / No||Qualitative|
|73||Provision of diversified and efficient intervention and treatment options for clinicians||(||Yes / No||Qualitative|
|74||Improved client care||(||Yes / No||Qualitative|
|75||The decrease in events of work-environment mistakes||(||What is the decrease rate of work-environment mistakes?||Quantitative|
|76||Increase in the provision of training of healthcare improvement from the healthcare providers to the support staff||(||How many support staff are trained?||Quantitative|
|77||Improvement in technologies and information systems for social applications||(||Yes / No||Qualitative|
|78||Increase in training development for system improvement||(||How many trainings are developed for healthcare improvements?||Quantitative|
|79||Creation of prevention methods for clinical practice||(||Yes / No||Qualitative|
|80||Adapting evidence-based practices||(||Yes / No||Qualitative|
|81||Improvement in patient outcomes||(||Yes / No||Qualitative|
|82||Improvement in health behaviours enthusiasm of patients and general masses||(||Yes / No||Qualitative|
|83||Development and promulgation of guidelines and policies||(||Yes / No||Qualitative|
|84||Progress in personal circumstances-based healthcare e.g. based on genetic sequencing||(||Yes / No||Qualitative|
|85||Strengthening of service-client relationship||(||Yes / No||Qualitative|
|86||Research outcome translation into medical practice for improvement||(||Yes / No||Qualitative|
|87||Strengthening human protection through improved policies and better procedures||(||Yes / No||Qualitative|
|88||Improvement in regulation for introducing advanced technologies, tools and techniques||(||Yes / No||Qualitative|
|89||Compliance of ethical guidelines in research||(||Yes / No||Qualitative|
|90||Development of community-based awareness||(||Yes / No||Qualitative|
|91||Betterment of policies, guidelines and reimbursement systems for service providers||(||Yes / No||Qualitative|
|92||Increased empowerment of service users||(||Yes / No||Qualitative|
|93||Support of research outcome and information for political decision and policy-making||(||Yes / No||Qualitative|
|94||Improved public awareness about the environment and culture, Public behaviour change and advocacy; Increased literacy and numeracy rates||(||Yes / No||Qualitative|
|95||Improvement in health literacy of health users and patients||(||Yes / No||Qualitative|
|96||Improvement in health status of health users and patients||(||Yes / No||Qualitative|
|97||Establishment of public health, education or any other social schemes for a region||(||Yes / No||Qualitative|
|98||The decrease in social disparities||(||Yes / No||Qualitative|
|99||Improvement in inter-organizational coordination for betterment in social sector||(||Yes / No||Qualitative|
|100||Increase in planning efforts and program implementation related to social issues||(||Yes / No||Qualitative|
|101||Improvement in health user about health research||(||Yes / No||Qualitative|
|102||Increased empowerment and knowledge of health users about health issues||(||Yes / No||Qualitative|
|103||Better communication and perception of health users about health risks||(||Yes / No||Qualitative|
|104||Expansion of health education, literacy and other social advantages||(||Yes / No||Qualitative|
|105||Improvement in Occupational Health and Safety Environment||(||Yes / No||Qualitative|
|106||Disuse the law for obsoleting the existing method of drug approval||(||Yes / No||Qualitative|
|107||Commercialization of new discovery, product or technology||(||Yes / No||Qualitative|
|108||Improvement in cost reducing techniques and effectiveness||(||Yes / No||Qualitative|
|109||Improvement in economic gains such as increased employment, health cost cut||(||Yes / No||Mixed|
|110||Development of new job opportunities and growth in the specific economic sector or geographical region||(||Yes / No||Mixed|
|111||Development of medicinal products and therapeutic procedures||(||How many products or procedures are developed?||Quantitative|
|112||Improvement in a business environment, commercialization, technology incubation, products and processes||(||Yes / No||Qualitative|
|113||Reduction in work loss due to illness and increased benefits from a healthy workforce||(||Yes / No||Qualitative|
|114||Increased Royalties, employment, Licences; creative works commissioned||(||Yes / No||Mixed|
|115||Creation of new knowledge about sustainable development and environmental protection for better future of the world||(||Yes / No||Qualitative|
|116||Improved environmental quality and sustainability||(||Yes / No||Qualitative|
|117||Reduced emissions; regeneration or arrested degradation of natural resources||(||Yes / No||Mixed|
|118||Improved awareness of environmental impacts and legislation for protection||(||Yes / No||Qualitative|
|119||development of mitigation methods for reducing environmental hazards and losses from natural disasters||(||Yes / No||Mixed|Editorial board publication strategy and acceptance rates in Turkish national journals Research misconduct in hospitals is spreading: A bibliometric analysis of retracted papers from Chinese university-affiliated hospitals The need to develop tailored tools for improving the quality of thematic bibliometric analyses: Evidence from papers published in Sustainability and Scientometrics The notion of dominant terminology in bibliometric research