Reflections on Tools and Methods for Differentiated Assessments of Individual Scientists, Groups of Scientists and Scientific Journals

<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis-2019-0012_s_006_w2aab3b7b2b1b6b1aab1c17b1Aa"><h3>Purpose</h3><p>This paper describes an alternative way of assessing journals considering a broader perspective of its impact. The Area-based connectedness (ABC) to society of journals applied here contributes to the assessment of the dissemination task of journals but with more data it may also contribute to the assessment of other missions.</p></sec><sec id="j_jdis-2019-0012_s_007_w2aab3b7b2b1b6b1aab1c17b2Aa"><h3>Design/methodology/approach</h3><p>The ABC approach assesses the performance of research actors, in this case journals, considering the characteristics of the research areas in which they are active. Each paper in a journal inherits the characteristics of its area. These areas are defined by a publication-based classification. The characteristics of areas relate to 5 dimensions of connectedness to society (news, policy, industrial R&amp;D, technology and local interest) and are calculated by bibliometric indicators and social media metrics.</p></sec><sec id="j_jdis-2019-0012_s_008_w2aab3b7b2b1b6b1aab1c17b3Aa"><h3>Findings</h3><p>In the paper, I illustrate the approach by showing the results for a few journals. They illustrate the diverse profiles that journals may have. We are able to provide a profile for each journal in the Web of Science database. The profiles we present show an appropriate view on the journals’ societal connectedness.</p></sec><sec id="j_jdis-2019-0012_s_009_w2aab3b7b2b1b6b1aab1c17b4Aa"><h3>Research limitations</h3><p>The classification I apply to perform the analyses is a CWTS in house classification based on Web of Science data. As such the application depends on the (updates of) that system. The classification is available at <ext-link ext-link-type="uri" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://www.leidenranking.com">www.leidenranking.com</ext-link></p></sec><sec id="j_jdis-2019-0012_s_010_w2aab3b7b2b1b6b1aab1c17b5Aa"><h3>Practical implications</h3><p>The dimensions of connectedness discussed in this paper relate to the dissemination task of journals but further development of this method may provide more options to monitor the tasks/mission of journals.</p></sec><sec id="j_jdis-2019-0012_s_011_w2aab3b7b2b1b6b1aab1c17b6Aa"><h3>Originality/value</h3><p>The ABC approach is a unique way to assess performance or impact of research actors considering the characteristics of the areas in which output is published and as such less prone to manipulation or gaming.</p></sec></abstract>

Measuring Societal Impact Is as Complex as ABC

<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis-2019-0013_s_006_w2aab3b7b3b1b6b1aab1c18b1Aa"><h3>Purpose</h3><p>The main goal of this paper is to show that international peer review can work in China’s context with satisfactory outcomes. Moreover, this paper also provides a reference for the practice of science and technology management.</p></sec><sec id="j_jdis-2019-0013_s_007_w2aab3b7b3b1b6b1aab1c18b2Aa"><h3>Design/methodology/approach</h3><p>This paper starts with a discussion of two critical questions about the significance and design of international peer review. A case study of international peer review of CAS Centers for Excellence is further analyzed.</p></sec><sec id="j_jdis-2019-0013_s_008_w2aab3b7b3b1b6b1aab1c18b3Aa"><h3>Findings</h3><p>International peer review may provide a solution to address the problem of quantitative oriented research evaluation in China. The case study of research evaluation of CAS Centers for Excellence shows that it is possible and feasible to conduct an international peer review in China’s context. When applying this approach to other scenarios, there are still many issues to consider including individualized design of international peer review combined with practical demands, and further improvement of theories and methods of international peer review.</p></sec><sec id="j_jdis-2019-0013_s_009_w2aab3b7b3b1b6b1aab1c18b4Aa"><h3>Research limitation</h3><p>1) Only the case of international peer review of CAS Centers for Excellence is analyzed; 2) A relatively small number of respondents were surveyed in the questionnaire.</p></sec><sec id="j_jdis-2019-0013_s_010_w2aab3b7b3b1b6b1aab1c18b5Aa"><h3>Practical implications</h3><p>The work presented in this study can be used as a reference for future studies.</p></sec><sec id="j_jdis-2019-0013_s_011_w2aab3b7b3b1b6b1aab1c18b6Aa"><h3>Originality/value</h3><p>Currently, there are no similarly detailed studies exploring the significance and methodology of international peer review in China.</p></sec></abstract>

Practice and Challenge of International Peer Review: A Case Study of Research Evaluation of CAS Centers for Excellence

<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis-2019-0014_s_007_w2aab3b7b4b1b6b1aab1c17b1Aa"><h3>Purpose</h3><p>This study expands on the results of a stakeholder-driven research project on quality indicators and output assessment of art and design research in Flanders—the Northern, Dutch-speaking region of Belgium. Herein, it emphasizes the value of arts &amp; design output registration as a modality to articulate the disciplinary demarcations of art and design research.</p></sec><sec id="j_jdis-2019-0014_s_008_w2aab3b7b4b1b6b1aab1c17b2Aa"><h3>Design/methodology/approach</h3><p>The particularity of art and design research in Flanders is first analyzed and compared to international examples. Hereafter, the results of the stakeholder-driven project on the creation of indicators for arts &amp; design research output assessment are discussed.</p></sec><sec id="j_jdis-2019-0014_s_009_w2aab3b7b4b1b6b1aab1c17b3Aa"><h3>Findings</h3><p>The findings accentuate the importance of allowing an assessment culture to emerge from practitioners themselves, instead of imposing ill-suited methods borrowed from established scientific evaluation models (Biggs &amp; Karlsson, 2011)—notwithstanding the practical difficulties it generates. They point to the potential of stakeholder-driven approaches for artistic research, which benefits from constructing a shared metadiscourse among its practitioners regarding the continuities and discontinuities between “artistic” and “traditional” research, and the communal goals and values that guide its knowledge production (Biggs &amp; Karlsson, 2011; <xref ref-type="bibr" rid="j_jdis-2019-0014_ref_007_w2aab3b7b4b1b6b1ab1b6b7Aa">Hellström, 2010</xref>; Ysebaert &amp; Martens, 2018).</p></sec><sec id="j_jdis-2019-0014_s_010_w2aab3b7b4b1b6b1aab1c17b4Aa"><h3>Research limitation</h3><p>The central limitation of the study is that it focuses exclusively on the “Architecture &amp; Design” panel of the project, and does not account for intra-disciplinary complexities in output assessment.</p></sec><sec id="j_jdis-2019-0014_s_011_w2aab3b7b4b1b6b1aab1c17b5Aa"><h3>Practical implications</h3><p>The goal of the research project is to create a robust assessment system for arts &amp; design research in Flanders, which may later guide similar international projects.</p></sec><sec id="j_jdis-2019-0014_s_012_w2aab3b7b4b1b6b1aab1c17b6Aa"><h3>Originality/value</h3><p>This study is currently the only one to consider the productive potential of (collaborative) PRFSs for artistic research.</p></sec></abstract>

Disclosing and Evaluating Artistic Research

<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis-2019-0015_s_005_w2aab3b7b5b1b6b1aab1c17b1Aa"><h3>Purpose</h3><p>To develop and test a mission-oriented and multi-dimensional benchmarking method for a small scale university aiming for internationally first-class basic research.</p></sec><sec id="j_jdis-2019-0015_s_006_w2aab3b7b5b1b6b1aab1c17b2Aa"><h3>Design/methodology/approach</h3><p>An individualized evidence-based assessment scheme was employed to benchmark ShanghaiTech University against selected top research institutions, focusing on research impact and competitiveness at the institutional and disciplinary levels. Topic maps opposing ShanghaiTech and corresponding top institutions were produced for the main research disciplines of ShanghaiTech. This provides opportunities for further exploration of strengths and weakness.</p></sec><sec id="j_jdis-2019-0015_s_007_w2aab3b7b5b1b6b1aab1c17b3Aa"><h3>Findings</h3><p>This study establishes a preliminary framework for assessing the mission of the university. It further provides assessment principles, assessment questions, and indicators. Analytical methods and data sources were tested and proved to be applicable and efficient.</p></sec><sec id="j_jdis-2019-0015_s_008_w2aab3b7b5b1b6b1aab1c17b4Aa"><h3>Research limitations</h3><p>To better fit the selective research focuses of this university, its schema of research disciplines needs to be re-organized and benchmarking targets should include disciplinary top institutions and not necessarily those universities leading overall rankings. Current reliance on research articles and certain databases may neglect important research output types.</p></sec><sec id="j_jdis-2019-0015_s_009_w2aab3b7b5b1b6b1aab1c17b5Aa"><h3>Practical implications</h3><p>This study provides a working framework and practical methods for mission-oriented, individual, and multi-dimensional benchmarking that ShanghaiTech decided to use for periodical assessments. It also offers a working reference for other institutions to adapt. Further needs are identified so that ShanghaiTech can tackle them for future benchmarking.</p></sec><sec id="j_jdis-2019-0015_s_010_w2aab3b7b5b1b6b1aab1c17b6Aa"><h3>Originality/value</h3><p>This is an effort to develop a mission-oriented, individually designed, systematically structured, and multi-dimensional assessment methodology which differs from often used composite indices.</p></sec></abstract>

Methods and Practices for Institutional Benchmarking based on Research Impact and Competitiveness: A Case Study of ShanghaiTech University

<abstract xmlns="http://www.w3.org/1999/xhtml"><sec id="j_jdis-2019-0016_s_012_w2aab3b7b6b1b6b1aab1c17b1Aa"><h3>Purpose</h3><p>Communicating scientific results to the public is essential to inspire future researchers and ensure that discoveries are exploited. News stories about research are a key communication pathway for this and have been manually monitored to assess the extent of press coverage of scholarship.</p></sec><sec id="j_jdis-2019-0016_s_013_w2aab3b7b6b1b6b1aab1c17b2Aa"><h3>Design/methodology/Approach</h3><p>To make larger scale studies practical, this paper introduces an automatic method to extract citations from newspaper stories to large sets of academic journals. Curated ProQuest queries were used to search for citations to 9,639 Science and 3,412 Social Science Web of Science (WoS) journals from eight UK daily newspapers during 2006–2015. False matches were automatically filtered out by a new program, with 94% of the remaining stories meaningfully citing research.</p></sec><sec id="j_jdis-2019-0016_s_014_w2aab3b7b6b1b6b1aab1c17b3Aa"><h3>Findings</h3><p>Most Science (95%) and Social Science (94%) journals were never cited by these newspapers. Half of the cited Science journals covered medical or health-related topics, whereas 43% of the Social Sciences journals were related to psychiatry or psychology. From the citing news stories, 60% described research extensively and 53% used multiple sources, but few commented on research quality.</p></sec><sec id="j_jdis-2019-0016_s_015_w2aab3b7b6b1b6b1aab1c17b4Aa"><h3>Research Limitations</h3><p>The method has only been tested in English and from the ProQuest Newspapers database.</p></sec><sec id="j_jdis-2019-0016_s_016_w2aab3b7b6b1b6b1aab1c17b5Aa"><h3>Practical implications</h3><p>Others can use the new method to systematically harvest press coverage of research.</p></sec><sec id="j_jdis-2019-0016_s_017_w2aab3b7b6b1b6b1aab1c17b6Aa"><h3>Originality/value</h3><p>An automatic method was introduced and tested to extract citations from newspaper stories to large sets of academic journals.</p></sec></abstract>

An Automatic Method to Identify Citations to Journals in News Stories: A Case Study of UK Newspapers Citing Web of Science Journals

AHEAD OF PRINT

Volume 9 (2024): Issue 1 (February 2024)

Volume 8 (2023): Issue 4 (November 2023)

Volume 8 (2023): Issue 3 (June 2023)

Volume 8 (2023): Issue 2 (April 2023)

Volume 8 (2023): Issue 1 (February 2023)

Volume 7 (2022): Issue 4 (November 2022)

Volume 7 (2022): Issue 3 (August 2022)

Volume 7 (2022): Issue 2 (April 2022)

Volume 7 (2022): Issue 1 (February 2022)

Volume 6 (2021): Issue 4 (November 2021)

Volume 6 (2021): Issue 3 (June 2021)

Volume 6 (2021): Issue 2 (April 2021)

Volume 6 (2021): Issue 1 (February 2021)

Volume 5 (2020): Issue 4 (November 2020)

Volume 5 (2020): Issue 3 (August 2020)

Volume 5 (2020): Issue 2 (April 2020)

Volume 5 (2020): Issue 1 (February 2020)

Volume 4 (2019): Issue 4 (December 2019)

Volume 4 (2019): Issue 3 (August 2019)

Volume 4 (2019): Issue 2 (May 2019)

Volume 4 (2019): Issue 1 (February 2019)

Volume 3 (2018): Issue 4 (November 2018)

Volume 3 (2018): Issue 3 (August 2018)

Volume 3 (2018): Issue 2 (May 2018)

Volume 3 (2018): Issue 1 (February 2018)

Volume 2 (2017): Issue 4 (December 2017)

Volume 2 (2017): Issue 3 (August 2017)

Volume 2 (2017): Issue 2 (May 2017)

Volume 2 (2017): Issue 1 (February 2017)

Volume 1 (2016): Issue 4 (November 2016)

Volume 1 (2016): Issue 3 (August 2016)

Volume 1 (2016): Issue 2 (May 2016)

Volume 1 (2016): Issue 1 (February 2016)

Journal of Data and Information Science

Journal of Data and Information Science (JDIS, formerly Chinese Journal of Library and Information Science), sponsored by the Chinese Academy of Sciences (CAS) and published quarterly by the National Science Library of CAS, is the first internationally published English-language academic journal in Library and Information Science and related fields from China.  The Journal of Data and Information Science (JDIS) focuses on data-based research oriented toward the exploration of scientific research and innovation. The main areas of interest are science of science, evidence-based policymaking, research evaluation, computational social science, and scientometrics/bibliometrics/altmetrics/ informetrics. Emphasis is given to research that focuses on data, analytics, and knowledge discovery, and supports decision making and science policy. This includes modeling, innovation, data security, media and communications, and social development. Topics may include studies of metadata or full content data, text or non-textural data, structured or non-structural data, domain-specific or cross-domain data, and dynamic or interactive data.  Specific topic areas may include (but are not limited to):    Knowledge organization  Knowledge discovery and data mining  Knowledge integration and fusion  Semantic Web  Science of science  Bibliometrics and scientometrics  Analytic and diagnostic informetrics  Competitive intelligence  Predictive analysis  Social network analysis and metrics  Semantic and interactively analytic retrieval  Evidence-based policy analysis  Intelligent knowledge production  Knowledge-driven workflow management and decision-making  Knowledge-driven collaboration and its management  Domain knowledge infrastructure with knowledge fusion and analytics  Training for data &amp; information scientists  Development of data and information services    JDIS publishes theoretical and empirical work. Systematic reviews are welcome and applied research in development of advanced methods, services, and best practices is also an important part. But simple application of established informetrics on a specific research field or country is out of the scope.Welcome to submit your papers to JDIS.  Why subscribe and read  JDIS is the first and only English journal from China in Library and Information Science and related fields. With an aim to disseminate the cutting-edge research in these fields, it is devoted to the study and application of the theories, methods, techniques, services, and infrastructural facilities using big data to support knowledge discovery for decision and policy making. The basic emphasis is big data-based, analytics centered, knowledge discovery driven, and decision making supporting. JDIS has gathered a big body of high profile experts across the world who contribute their research to the journal. The international authors account for around 62% in its first publication year (2016).  Why submit  JDIS is the first and only English journal from China in Library and Information Science and related fields. It owns a number of world front-line scholars as editorial board members or reviewers. The turnaround time on average for a manuscript from submission to final decision is less than two and a half months.  Archiving  Sciendo archives the contents of this journal in Portico- digital long-term preservation service of scholarly books, journals and collections.  Plagiarism Policy  The editorial board is participating in a growing community of Similarity Check System's users in order to ensure that the content published is original and trustworthy. Similarity Check is a medium that allows for comprehensive manuscripts screening, aimed to eliminate plagiarism and provide a high standard and quality peer-review process. 

Journal of Data and Information Science

Volume 4 (2019): Issue 3 (August 2019)

Editorial

Reflections on Tools and Methods for Differentiated Assessments of Individual Scientists, Groups of Scientists and Scientific Journals

Research Paper

Measuring Societal Impact Is as Complex as ABC

Practice and Challenge of International Peer Review: A Case Study of Research Evaluation of CAS Centers for Excellence

Disclosing and Evaluating Artistic Research

Methods and Practices for Institutional Benchmarking based on Research Impact and Competitiveness: A Case Study of ShanghaiTech University

An Automatic Method to Identify Citations to Journals in News Stories: A Case Study of UK Newspapers Citing Web of Science Journals

Buscar

Todos los volúmenes y ediciones en esta revista