Performance and efficiency evaluation is an essential but challenging task for managers in fields ranging from science to business. Therefore, in bibliometrics, several citation indicators, including intuitive indicators such as total and average citation counts and extended indicators such as the impact factor (IF) (Garfield, 1972) and
Within a researcher’s publication set, the rank distribution of citations should theoretically be a curve. The publication set is likely to include certain highly cited papers and many scarcely cited papers (Bornmann, Mutz, & Daniel, 2010), but the
Rank-citation curve with information on the number of publications. The area under the rank-citation curve is divided into four sections: the Figure 1
According to the definition of I3 (Leydesdorff & Bornmann (2011), an I3-type indicator can be formalized as
where
Similar to
in which the weighting scores for
The publication vector
When the
Because trace metrics summarize all the information in the citation curve, they can be applied for measuring the overall performance of a university, assignee, paper, or patent. The remainder of the paper is organized as follows. Section 2 provides a detailed explanation of how trace metrics were calculated and how data were chosen. Section 3 presents the results. Finally, Section 4 presents the discussions and conclusions.
We extended the performance matrix proposed by Ye and Leydesdorff (2014) to a primary matrix
where
where
The vectors
The three traces of matrices
Both
For a demonstration of trace metrics, we applied traces
In this research, we used full counts to assign credits of publications to organizations. Although some might debate that using full counts in bibliometrics would magnify the actual number of publications, full counting is the most intuitive and currently most widely-used counting method in bibliometrics. From the perspective of patents, only few patents have more than one assignee. Zheng et al. (2013) studied the influence of counting methods in patentometrics and found that the difference among different counting methods is slight. In this preliminary reseach of applying trace metrics in bibliometrics and patentometrics, we chose to compare the trace metric performance of universities and companies using a full counting method and to leave the author contribution-credit issue to future work.
We used the traces
For a patentometric test, we selected the top 30 assignees who owned the most patents in the National Bureau of Economic Research (NBER) computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. Similar to the procedure used for the bibliometric test, we selected the top 30 most cited US patents in the NBER computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. All patent data were obtained from the United States Patent and Trademark Office database.
The datasets covered group level (universities and companies) and individual level (paper and patent, individually as a single publication). For calculating the traces of a single document (a highly cited paper or patent), we followed Schubert’s (2009) method to construct a rank–citation graph of the single document by determining the number and citations of citing documents (i.e. documents that cite the document under consideration). Therefore, the
Applying trace metrics to a university enables assessing its academic performance. We call such metrics academic traces.
Figure 2 shows the values of academic traces
Academic traces Figure 2
Table 1 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among
Spearman and Pearson correlation coefficients among the Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level.Spearman Pearson 1 0.581 0.988 0.730 0.721 0.593 1 0.598 0.065 0.755 0.994 0.624 1 0.715 0.749 0.766 0.104 0.752 1 0.460 0.751 0.745 0.790 0.519 1
Several bibliometric indicators are used in patentometrics for estimating the performance of patents. Similar to the procedures performed for bibliometrics, a company can be evaluated according to the performance of its patents. When trace metrics are applied to a group level of patent, they are called assignee traces.
Figure 3 illustrates the values of assignee traces
Values of Figure 3
Table 2 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among
Spearman and Pearson correlation coefficients among the Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level.Spearman Pearson 1 −0.248 0.093 0.932 0.275 −0.507 1 0.655 −0.094 0.519 −0.482 0.881 1 0.275 0.799 0.303 −0.099 0.177 1 0.488 −0.151 0.646 0.750 −0.121 1
Spearman and Pearson correlation coefficients among the Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level.Spearman Pearson 1 0.868 0.994 0.926 0.922 0.820 1 0.838 0.746 0.907 0.992 0.828 1 0.926 0.915 0.901 0.627 0.880 1 0.867 0.793 0.897 0.827 0.706 1
At the group level, we observed that for both universities and companies, the difference between their average citation and
We determined that in contrast to the patentometric indicators, all bibliometric indicators showed significant correlations. This discrepancy means that bibliometric indicators as well as traces are generally applicable and that other factors such as market elements must be considered in patentometric indicators to ensure their applicability.
In addition to the universities, the trace metrics were applied to a single paper to evaluate its impact. We called these metrics impact traces.
Figure 4 shows the values of impact traces
Impact traces Figure 4
Table 3 lists Pearson (bottom left part of the table, without a background) and Spearman (top right part of the table, with a gray background) correlation coefficients among
Similar to our previous bibliometric analysis, the impact of a single patent was studied using trace metrics (subsequently denoted as patent traces).
Figure 5 illustrates the values of patent traces
Values of Figure 5
Spearman and Pearson correlation coefficients among the Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level. Significant correlation at 0.01 level.Spearman Pearson 1 0.830 0.988 0.963 0.892 0.276 1 0.815 0.839 0.747 0.989 0.363 1 0.964 0.905 0.962 0.410 0.975 1 0.959 0.976 0.347 0.975 0.985 1
Table 4 lists Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among
At the individual level, the differences in the average citation values and in the
Typically, an object receives trace metrics with a higher
We also determined that, at the individual level, all indicators showed significant correlations in both bibliometrics and patentometrics, demonstrating that all indicators, including traces, were effective indices for evaluation and cross-referencing.
When the performance matrix proposed by Ye and Leydesdorff (2014) is extended to a primary matrix, secondary matrix, and submatrix, the traces of the three performance matrices
Commonly used bibliometric indicators such as citation count and average citation are single point indicators, and they cannot accurately reflect variations in a rank-citation curve. Although the
We observed that the differences in trace metrics were greater than those in the average citation values and in the
For the trace metrics
If a university receives a negative
For patent owners, a negative trace metric value indicates imbalanced research and development distribution toward low-value patents. This might be tolerable for large enterprises because they might have sufficient capital to fabricate a long-term patent portfolio. However, for small businesses, it might indicate an impending financial failure to have such a negative value. By contrast, because the citing practice in patentometrics is different than in bibliometrics, and because certain patents receive low or zero citation despite being valuable, a negative value in
The meaning of the negative term
A recent popular topic in bibliometrics and university evaluation is field normalization. This issue is usually discussed in university evaluation, and more and more global university ranking systems have adopted field normalization to reduce the field bias of publications and citations of different research-oriented universities. In our bibliometric test, we have already chosen a field so we could basically bypass this issue. Moreover, if we look into the subfields of computer sciences, we find that most of them have similar numbers of publications and citations therefore the field normalization issue could also be disregarded.
For our patentometric test, as we used the NBER categories, in which the smallest division is the computer software and hardware, to select our patent data, it is impossible for us to do field normalization in our patentometric analysis. However, for future research dealing with other fields, especially for fields that have significant bibliometric differences among their subfields, field normalization might be considered when evaluating the trace metric performance.
Our analysis reveals that trace metrics, which consider zero citation as a negative contribution, provide a unique view on the impact efficiency of an organization. We also determined that trace metrics exhibit different indicating behaviors between hardware patents and software patents, whereas commonly used indicators such as average citation and