Large language models and scientific publishing

How a systems perspective can help us with the interdisciplinarity puzzle

<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems. This article assesses whether full counting changes results compared to fractional counting in the case of the UK's Research Excellence Framework (REF) 2021. For this assessment, fractional counting reduces the number of journal articles to as little as 10% of the full counting value, depending on the Unit of Assessment (UoA). Despite this large difference, allocating an overall grade point average (GPA) based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98. The largest changes are for Archaeology (r=0.84) and Physics (r=0.88). There is a weak tendency for higher scoring institutions to lose from fractional counting, with the loss being statistically significant in 5 of the 34 UoAs. Thus, whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall, it may be worth examining in the few UoAs in which it makes the most difference.</p>
</abstract>

Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021

<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Regression is a widely used econometric tool in research. In observational studies, based on a number of assumptions, regression-based statistical control methods attempt to analyze the causation between treatment and outcome by adding control variables. However, this approach may not produce reliable estimates of causal effects. In addition to the shortcomings of the method, this lack of confidence is mainly related to ambiguous formulations in econometrics, such as the definition of selection bias, selection of core control variables, and method of testing for robustness. Within the framework of the causal models, we clarify the assumption of causal inference using regression-based statistical controls, as described in econometrics, and discuss how to select core control variables to satisfy this assumption and conduct robustness tests for regression estimates.</p>
</abstract>

Causal inference using regression-based statistical control: Confusion in Econometrics

<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Purpose</h3>
<p>In recent decades, with the availability of large-scale scientific corpus datasets, difference-in-difference (DID) is increasingly used in the science of science and bibliometrics studies. DID method outputs the unbiased estimation on condition that several hypotheses hold, especially the common trend assumption. In this paper, we gave a systematic demonstration of DID in the science of science, and the potential ways to improve the accuracy of DID method.</p>
</sec>
<sec><h3>Design/methodology/approach</h3>
<p>At first, we reviewed the statistical assumptions, the model specification, and the application procedures of DID method. Second, to improve the necessary assumptions before conducting DID regression and the accuracy of estimation, we introduced some matching techniques serving as the pre-selecting step for DID design by matching control individuals who are equivalent to those treated ones on observational variables before the intervention. Lastly, we performed a case study to estimate the effects of prizewinning on the scientific performance of Nobel laureates, by comparing the yearly citation impact after the prizewinning year between Nobel laureates and their prizewinning-work coauthors.</p>
</sec>
<sec><h3>Findings</h3>
<p>We introduced the procedures to conduct a DID estimation and demonstrated the effectiveness to use matching method to improve the results. As a case study, we found that there are no significant increases in citations for Nobel laureates compared to their prizewinning coauthors.</p>
</sec>
<sec><h3>Research limitations</h3>
<p>This study ignored the rigorous mathematical deduction parts of DID, while focused on the practical parts.</p>
</sec>
<sec><h3>Practical implications</h3>
<p>This work gives experimental practice and potential guidelines to use DID method in science of science and bibliometrics studies.</p>
</sec>
<sec><h3>Originality/value</h3>
<p>This study gains insights into the usage of econometric tools in science of science.</p>
</sec>
</abstract>

Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Purpose</h3>
<p>Following the typical features of the grey-rhino event as predictability and profound influence, we attempt to find a special pattern called the grey-rhino in eminent technologies <italic>via</italic> patent analysis.</p>
</sec>
<sec><h3>Design/methodology/approach</h3>
<p>We propose to combine triadic patent families and technology life cycle to define the grey-rhino model. Firstly, we design the indicator rhino-index R<sub>h</sub> = ST/SP and descriptor sequence {R<sub>h</sub>}, where ST and SP are the accumulative number of triadic patent families and all patent families respectively for a specific technology. Secondly, according to the two typical features of the grey-rhino event, a grey-rhino is defined as a technology that meets both qualitative and quantitative conditions. Qualitatively, this technology has a profound influence. Quantitatively, in the emerging stage, R<sub>h</sub> ≥ Rae, where Rae is the average level of the proportion of triadic patent families. Finally, this model is verified in three datasets, namely Encyclopedia Britannica's list for the greatest inventions (EB technologies for short), MIT breakthrough technologies (MIT technologies) and Derwent Manual Code technologies (MAN technologies).</p>
</sec>
<sec><h3>Findings</h3>
<p>The result shows that there are 64.71% EB technologies and 50.00% MIT technologies meeting the quantitative standard of the grey-rhino model, but only 14.71% MAN technologies fit the quantitative standard. This falling trend indicates the quantitative standard of the grey-rhino model is reasonable. EB technologies and MIT technologies have profound influence on society, which means they satisfy the qualitative standard of the grey-rhino model. Hence, 64.71% EB technologies and 50.00% MIT technologies are grey-rhinos. In 14.71% MAN technologies meeting the quantitative standard, we make some qualitative judgments and deem U11-A01A, U12-A01A1A, and W01-A01A as grey-rhino technologies. In addition, grey-rhinos and non-grey-rhinos have some differences. R<sub>h</sub> values of grey-rhinos have a downward trend, while R<sub>h</sub> values of non-grey-rhinos have a contrary trend. R<sub>h</sub> values of grey-rhinos are scattered relatively in the early stage and centralize gradually, but non-grey-rhinos do not have this feature.</p>
</sec>
<sec><h3>Research limitations</h3>
<p>There are four main limitations. First, if a technology satisfies the quantitative standard of the model, it is likely to be a grey-rhino but expert judgments are necessary. Second, we don’t know why it will be eminent, which involves technical contents. Thirdly, we did not consider the China National Intellectual Property Administration (CNIPA) and the German Patent and Trademark Office (DPMA) which also play important roles in worldwide patents, so we hope to expand our study to the CNIPA and the DPMA. Furthermore, we did not compare the rhino-index with other patent indicators.</p>
</sec>
<sec><h3>Practical implications</h3>
<p>If a technology meets the quantitative standard, this can be seen as early warning signals and the technology may become a grey-rhino in the future, which can catch people's attention in the emerging stage and make people seize the technical opportunity early.</p>
</sec>
<sec><h3>Originality/value</h3>
<p>We define and verify a new pattern called the grey-rhino model in eminent technologies.</p>
</sec>
</abstract>

International Joint Informatics Laboratory

School of Cultural Heritage and Information Management

Identifying grey-rhino in eminent technologies via patent analysis

<abstract xmlns="http://www.w3.org/1999/xhtml">

<sec><h3>Purpose</h3>
<p>To supplement the quantitative portrait of Ukrainian Economics discipline with the results of gender and author ordering analysis at the level of individual authors, special methods of working with bibliographic data with a predominant share of non-English authors are used. The properties of gender mixing, the likelihood of male and female authors occupying the first position in the authorship list, as well as the arrangements of names are studied.</p>
</sec>
<sec><h3>Design/methodology/approach</h3>
<p>A data set containing bibliographic records related to Ukrainian journal publications in the field of Economics is constructed using Crossref metadata. Partial semi-automatic disambiguation of authors’ names is performed. First names, along with gender-specific ethnic surnames, are used for gender disambiguation required for further comparative gender analysis. Random reshuffling of data is used to determine the impact of gender correlations. To assess the level of alphabetization for our data set, both Latin and Cyrillic versions of names are taken into account.</p>
</sec>
<sec><h3>Findings</h3>
<p>The lack of well-structured metadata and the poor use of digital identifiers lead to numerous problems with automatization of bibliographic data pre-processing, especially in the case of publications by non-Western authors. The described stages for working with such specific data help to work at the level of authors and analyse, in particular, gender issues. Despite the larger number of female authors, gender equality is more likely to be reported at the individual level for the discipline of Ukrainian Economics. The tendencies towards collaborative or solo-publications and gender mixing patterns are found to be dependent on the journal: the differences for publications indexed in Scopus and/or Web of Science databases are found. It has also been found that Ukrainian Economics research is characterized by rather a non-alphabetical order of authors.</p>
</sec>
<sec><h3>Research limitations</h3>
<p>Only partial authors’ name disambiguation is performed in a semi-automatic way. Gender labels can be derived only for authors declared by full First names or gender-specific Last names.</p>
</sec>
<sec><h3>Practical implications</h3>
<p>The typical features of Ukrainian Economic discipline can be used to perform a comparison with other countries and disciplines, to develop an informed-based assessment procedure at the national level. The proposed way of processing publication data can be borrowed to enrich metadata about other research disciplines, especially for non-English speaking countries.</p>
</sec>
<sec><h3>Originality/value</h3>
<p>To our knowledge, this is the first large-scale quantitative study of Ukrainian Economic discipline. The results obtained are valuable not only at the national level, but also contribute to general knowledge about Economic research, gender issues, and authors’ names ordering. An example of the use of Crossref data is provided, while this data source is still less used due to a number of drawbacks. Here, for the first time, attention is drawn to the explicit use of the features of the Slavic authors’ names.</p>
</sec>
</abstract>

University of Social Sciences Spoleczna Akademia Nauk

Peculiarities of gender disambiguation and ordering of non-English authors’ names for Economic papers beyond core databases<xref ref-type="fn" rid="j_jdis-2023-0001_fn_001"><sup>①</sup></xref>

AHEAD OF PRINT

Volume 9 (2024): Issue 1 (February 2024)

Volume 8 (2023): Issue 4 (November 2023)

Volume 8 (2023): Issue 3 (June 2023)

Volume 8 (2023): Issue 2 (April 2023)

Volume 8 (2023): Issue 1 (February 2023)

Volume 7 (2022): Issue 4 (November 2022)

Volume 7 (2022): Issue 3 (August 2022)

Volume 7 (2022): Issue 2 (April 2022)

Volume 7 (2022): Issue 1 (February 2022)

Volume 6 (2021): Issue 4 (November 2021)

Volume 6 (2021): Issue 3 (June 2021)

Volume 6 (2021): Issue 2 (April 2021)

Volume 6 (2021): Issue 1 (February 2021)

Volume 5 (2020): Issue 4 (November 2020)

Volume 5 (2020): Issue 3 (August 2020)

Volume 5 (2020): Issue 2 (April 2020)

Volume 5 (2020): Issue 1 (February 2020)

Volume 4 (2019): Issue 4 (December 2019)

Volume 4 (2019): Issue 3 (August 2019)

Volume 4 (2019): Issue 2 (May 2019)

Volume 4 (2019): Issue 1 (February 2019)

Volume 3 (2018): Issue 4 (November 2018)

Volume 3 (2018): Issue 3 (August 2018)

Volume 3 (2018): Issue 2 (May 2018)

Volume 3 (2018): Issue 1 (February 2018)

Volume 2 (2017): Issue 4 (December 2017)

Volume 2 (2017): Issue 3 (August 2017)

Volume 2 (2017): Issue 2 (May 2017)

Volume 2 (2017): Issue 1 (February 2017)

Volume 1 (2016): Issue 4 (November 2016)

Volume 1 (2016): Issue 3 (August 2016)

Volume 1 (2016): Issue 2 (May 2016)

Volume 1 (2016): Issue 1 (February 2016)

Journal of Data and Information Science

Journal of Data and Information Science (JDIS, formerly Chinese Journal of Library and Information Science), sponsored by the Chinese Academy of Sciences (CAS) and published quarterly by the National Science Library of CAS, is the first internationally published English-language academic journal in Library and Information Science and related fields from China.  The Journal of Data and Information Science (JDIS) focuses on data-based research oriented toward the exploration of scientific research and innovation. The main areas of interest are science of science, evidence-based policymaking, research evaluation, computational social science, and scientometrics/bibliometrics/altmetrics/ informetrics. Emphasis is given to research that focuses on data, analytics, and knowledge discovery, and supports decision making and science policy. This includes modeling, innovation, data security, media and communications, and social development. Topics may include studies of metadata or full content data, text or non-textural data, structured or non-structural data, domain-specific or cross-domain data, and dynamic or interactive data.  Specific topic areas may include (but are not limited to):    Knowledge organization  Knowledge discovery and data mining  Knowledge integration and fusion  Semantic Web  Science of science  Bibliometrics and scientometrics  Analytic and diagnostic informetrics  Competitive intelligence  Predictive analysis  Social network analysis and metrics  Semantic and interactively analytic retrieval  Evidence-based policy analysis  Intelligent knowledge production  Knowledge-driven workflow management and decision-making  Knowledge-driven collaboration and its management  Domain knowledge infrastructure with knowledge fusion and analytics  Training for data &amp; information scientists  Development of data and information services    JDIS publishes theoretical and empirical work. Systematic reviews are welcome and applied research in development of advanced methods, services, and best practices is also an important part. But simple application of established informetrics on a specific research field or country is out of the scope.Welcome to submit your papers to JDIS.  Why subscribe and read  JDIS is the first and only English journal from China in Library and Information Science and related fields. With an aim to disseminate the cutting-edge research in these fields, it is devoted to the study and application of the theories, methods, techniques, services, and infrastructural facilities using big data to support knowledge discovery for decision and policy making. The basic emphasis is big data-based, analytics centered, knowledge discovery driven, and decision making supporting. JDIS has gathered a big body of high profile experts across the world who contribute their research to the journal. The international authors account for around 62% in its first publication year (2016).  Why submit  JDIS is the first and only English journal from China in Library and Information Science and related fields. It owns a number of world front-line scholars as editorial board members or reviewers. The turnaround time on average for a manuscript from submission to final decision is less than two and a half months.  Archiving  Sciendo archives the contents of this journal in Portico- digital long-term preservation service of scholarly books, journals and collections.  Plagiarism Policy  The editorial board is participating in a growing community of Similarity Check System's users in order to ensure that the content published is original and trustworthy. Similarity Check is a medium that allows for comprehensive manuscripts screening, aimed to eliminate plagiarism and provide a high standard and quality peer-review process. 

Journal of Data and Information Science

Volume 8 (2023): Issue 1 (February 2023)

Editorial

Large language models and scientific publishing

Perspective

How a systems perspective can help us with the interdisciplinarity puzzle

Research Note

Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021

Opinion

Causal inference using regression-based statistical control: Confusion in Econometrics

Research Paper

Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

Identifying grey-rhino in eminent technologies via patent analysis

Peculiarities of gender disambiguation and ordering of non-English authors’ names for Economic papers beyond core databases^①

Chercher

Tous les volumes et éditions dans cette revue

Journal of Data and Information Science

Volume 8 (2023): Issue 1 (February 2023)

Editorial

Large language models and scientific publishing

Perspective

How a systems perspective can help us with the interdisciplinarity puzzle

Research Note

Is big team research fair in national research assessments? The case of the UK Research Excellence Framework 2021

Opinion

Causal inference using regression-based statistical control: Confusion in Econometrics

Research Paper

Practical operation and theoretical basis of difference-in-difference regression in science of science: The comparative trial on the scientific performance of Nobel laureates versus their coauthors

Identifying grey-rhino in eminent technologies via patent analysis

Peculiarities of gender disambiguation and ordering of non-English authors’ names for Economic papers beyond core databases①

Chercher

Tous les volumes et éditions dans cette revue

Peculiarities of gender disambiguation and ordering of non-English authors’ names for Economic papers beyond core databases^①