Performance and efficiency evaluation is an essential but challenging task for managers in fields ranging from science to business. Therefore, in bibliometrics, several citation indicators, including intuitive indicators such as total and average citation counts and extended indicators such as the impact factor (IF) (Garfield, 1972) and

Within a researcher’s publication set, the rank distribution of citations should theoretically be a curve. The publication set is likely to include certain highly cited papers and many scarcely cited papers (Bornmann, Mutz, & Daniel, 2010), but the

According to the definition of I3 (Leydesdorff & Bornmann (2011), an I3-type indicator can be formalized as

where _{i}_{i}_{i}_{i}_{i}

Similar to

in which the weighting scores for _{c}_{t}_{z}_{c}_{t}_{e}_{c}_{c}_{c}_{t}_{z}_{t}_{t}_{c}_{t}_{z}_{z}_{z}_{c}_{t}_{z}_{c}_{c}_{c}_{t}_{e}_{t}_{t}_{c}_{t}_{e}_{e}_{e}_{c}_{t}_{e}

The publication vector

When the _{1} = (^{T} and _{2} = (^{T} can be constructed. Accordingly, if an indicator is required for comparing or ranking scholarly individuals or groups, the traces of performance matrices that provide scalars that summarize academic performance, such as _{1} = Tr (_{1}) = _{1} + _{2} + _{3} and _{2} = Tr(_{2})= _{1} + _{2} + _{3}, can be computed. Therefore, multivariate information in the citation curve can be expressed in single measures.

Because trace metrics summarize all the information in the citation curve, they can be applied for measuring the overall performance of a university, assignee, paper, or patent. The remainder of the paper is organized as follows. Section 2 provides a detailed explanation of how trace metrics were calculated and how data were chosen. Section 3 presents the results. Finally, Section 4 presents the discussions and conclusions.

We extended the performance matrix proposed by Ye and Leydesdorff (2014) to a primary matrix _{1}, a secondary matrix _{2}, and a submatrix

where _{1}_{2}

where _{c}^{2}

_{t}

_{e}

_{h}_{h}_{c}_{e}

_{c}_{t}_{e}

_{c}

_{t}

_{z}

_{c}_{t}_{z}

The vectors _{1}_{2}, _{3}) and _{1}, _{2}, _{3}) are publication and citation vectors, respectively.

The three traces of matrices _{1}, _{2}, and _{1}, _{2}, and

Both _{1} and _{2} summarize the representative information distributed over the e-,

For a demonstration of trace metrics, we applied traces _{1}, _{2}, and

In this research, we used full counts to assign credits of publications to organizations. Although some might debate that using full counts in bibliometrics would magnify the actual number of publications, full counting is the most intuitive and currently most widely-used counting method in bibliometrics. From the perspective of patents, only few patents have more than one assignee. Zheng et al. (2013) studied the influence of counting methods in patentometrics and found that the difference among different counting methods is slight. In this preliminary reseach of applying trace metrics in bibliometrics and patentometrics, we chose to compare the trace metric performance of universities and companies using a full counting method and to leave the author contribution-credit issue to future work.

We used the traces _{1}, _{2}, and

For a patentometric test, we selected the top 30 assignees who owned the most patents in the National Bureau of Economic Research (NBER) computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. Similar to the procedure used for the bibliometric test, we selected the top 30 most cited US patents in the NBER computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. All patent data were obtained from the United States Patent and Trademark Office database.

The datasets covered group level (universities and companies) and individual level (paper and patent, individually as a single publication). For calculating the traces of a single document (a highly cited paper or patent), we followed Schubert’s (2009) method to construct a rank–citation graph of the single document by determining the number and citations of citing documents (i.e. documents that cite the document under consideration). Therefore, the

Applying trace metrics to a university enables assessing its academic performance. We call such metrics academic traces.

Figure 2 shows the values of academic traces _{1} (solid blue line with squares), _{2} (solid orange line with triangles), and _{1}, _{2}, and _{1}, _{2}, and _{1}, _{2}, _{2} and a drop in the _{2}. We carefully examined these four datasets and determined that, compared with other universities, Tsinghua University and CMU published more papers having numbers of citations that were lower than the _{t}_{t}_{2}). These _{2}-rise universities were compared with _{2}-drop-_{2}-drop-

Table 1 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among _{1}, _{2}, _{1} and _{2}, which had a low correlation coefficient with the average indicator _{2} and _{1}, _{2} and _{1},

Spearman and Pearson correlation coefficients among the _{1}, _{2}, and

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.581 Significant correlation at 0.01 level. | 0.988 Significant correlation at 0.01 level. | 0.730 Significant correlation at 0.01 level. | 0.721 Significant correlation at 0.01 level. | |

_{2} | 0.593 Significant correlation at 0.01 level. | 1 | 0.598 Significant correlation at 0.01 level. | 0.065 | 0.755 Significant correlation at 0.01 level. | |

0.994 Significant correlation at 0.01 level. | 0.624 Significant correlation at 0.01 level. | 1 | 0.715 Significant correlation at 0.01 level. | 0.749 Significant correlation at 0.01 level. | ||

0.766 Significant correlation at 0.01 level. | 0.104 | 0.752 Significant correlation at 0.01 level. | 1 | 0.460 Significant correlation at 0.01 level. | ||

0.751 Significant correlation at 0.01 level. | 0.745 Significant correlation at 0.01 level. | 0.790 Significant correlation at 0.01 level. | 0.519 Significant correlation at 0.01 level. | 1 |

Several bibliometric indicators are used in patentometrics for estimating the performance of patents. Similar to the procedures performed for bibliometrics, a company can be evaluated according to the performance of its patents. When trace metrics are applied to a group level of patent, they are called assignee traces.

Figure 3 illustrates the values of assignee traces _{1} (blue solid line with squares), _{2} (orange solid line with triangles), and _{1}, _{2}, and _{1}, _{2} and _{1} value. The reason for the drop in _{1} is that IBM has many zero-citation patents, leading to a considerably large value for

Table 2 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among _{1}, _{2}, _{1} negatively correlated with _{2}, _{e}_{z}_{1}; hence, the trend of _{1} was different from that of _{2},

Spearman and Pearson correlation coefficients among the _{1}, _{2}, and

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | −0.248 | 0.093 | 0.932 Significant correlation at 0.01 level. | 0.275 | |

_{2} | −0.507 Significant correlation at 0.01 level. | 1 | 0.655 Significant correlation at 0.01 level. | −0.094 | 0.519 Significant correlation at 0.01 level. | |

−0.482 Significant correlation at 0.01 level. | 0.881 Significant correlation at 0.01 level. | 1 | 0.275 | 0.799 Significant correlation at 0.01 level. | ||

0.303 | −0.099 | 0.177 | 1 | 0.488 Significant correlation at 0.01 level. | ||

−0.151 | 0.646 Significant correlation at 0.01 level. | 0.750 Significant correlation at 0.01 level. | −0.121 | 1 |

Spearman and Pearson correlation coefficients among the _{1}, _{2}, and

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.868 Significant correlation at 0.01 level. | 0.994 Significant correlation at 0.01 level. | 0.926 Significant correlation at 0.01 level. | 0.922 Significant correlation at 0.01 level. | |

0.820 Significant correlation at 0.01 level. | 1 | 0.838 Significant correlation at 0.01 level. | 0.746 Significant correlation at 0.01 level. | 0.907 Significant correlation at 0.01 level. | ||

0.992 Significant correlation at 0.01 level. | 0.828 Significant correlation at 0.01 level. | 1 | 0.926 Significant correlation at 0.01 level. | 0.915 Significant correlation at 0.01 level. | ||

0.901 Significant correlation at 0.01 level. | 0.627 Significant correlation at 0.01 level. | 0.880 Significant correlation at 0.01 level. | 1 | 0.867 Significant correlation at 0.01 level. | ||

0.793 Significant correlation at 0.01 level. | 0.897 Significant correlation at 0.01 level. | 0.827 Significant correlation at 0.01 level. | 0.706 Significant correlation at 0.01 level. | 1 |

At the group level, we observed that for both universities and companies, the difference between their average citation and _{1} value. This negative value can be considered a warning, rather than being perceived as indicating no market value; accordingly, patents’ potential market value should be investigated. Compared with an acceptable negative trace metric value in patentometrics, a negative trace metric value in bibliometrics indicates poor efficiency in conducting crucial research. Therefore, if a university receives a negative trace metric value, it should examine its research projects and consider adjusting them.

We determined that in contrast to the patentometric indicators, all bibliometric indicators showed significant correlations. This discrepancy means that bibliometric indicators as well as traces are generally applicable and that other factors such as market elements must be considered in patentometric indicators to ensure their applicability.

In addition to the universities, the trace metrics were applied to a single paper to evaluate its impact. We called these metrics impact traces.

Figure 4 shows the values of impact traces _{1} (solid blue line with squares), _{2} (solid orange line with triangles), and _{1}, _{2}, and _{1}, _{2}, and _{1} and _{1} and _{1}, _{2}, and

Table 3 lists Pearson (bottom left part of the table, without a background) and Spearman (top right part of the table, with a gray background) correlation coefficients among _{1}, _{2},

Similar to our previous bibliometric analysis, the impact of a single patent was studied using trace metrics (subsequently denoted as patent traces).

Figure 5 illustrates the values of patent traces _{1} (solid blue line with squares), _{2} (solid orange line with triangles), and _{1}, _{2}, and _{1}, _{2}, and _{2} value (Figure 5), and thus, a low Pearson correlation coefficient was observed between _{2} and other indicators (Table 4). All the top seven most cited patents had a relatively high _{e}_{1}, _{2}. After carefully examining these patents, we observed that the hardware patents P1–P7 had a relatively high _{1} and ^{4}, and _{2}, which was proportional to ^{2}.

Spearman and Pearson correlation coefficients among the _{1}, _{2}, and

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.830 Significant correlation at 0.01 level. | 0.988 Significant correlation at 0.01 level. | 0.963 Significant correlation at 0.01 level. | 0.892 Significant correlation at 0.01 level. | |

_{2} | 0.276 | 1 | 0.815 Significant correlation at 0.01 level. | 0.839 Significant correlation at 0.01 level. | 0.747 Significant correlation at 0.01 level. | |

0.989 Significant correlation at 0.01 level. | 0.363 Significant correlation at 0.01 level. | 1 | 0.964 Significant correlation at 0.01 level. | 0.905 Significant correlation at 0.01 level. | ||

0.962 Significant correlation at 0.01 level. | 0.410 Significant correlation at 0.01 level. | 0.975 Significant correlation at 0.01 level. | 1 | 0.959 Significant correlation at 0.01 level. | ||

0.976 Significant correlation at 0.01 level. | 0.347 Significant correlation at 0.01 level. | 0.975 Significant correlation at 0.01 level. | 0.985 Significant correlation at 0.01 level. | 1 |

Table 4 lists Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among _{1}, _{2}, _{2}.

At the individual level, the differences in the average citation values and in the _{1} and a lower value of _{2}, and P8 to P30 had approximately the similiar value of _{1}, and _{2}. The difference was due to the different citation types between the software and hardware patents.

Typically, an object receives trace metrics with a higher _{2}, a higher _{2}, and a lower _{1}. However, we observed that _{2} was the lowest trace metric value of the top 30 most cited patents. A lower _{2}, which considered the square of the citations in the tail part of the most cited patents, indicated that such citations were not comparable to those of their paper counterparts. However, although the average citation value in the tail part was lower than the _{2} value represented a steep rank-citation curve, which is acceptable in patentometrics but is a symbol of irrelevance in bibliometrics.

We also determined that, at the individual level, all indicators showed significant correlations in both bibliometrics and patentometrics, demonstrating that all indicators, including traces, were effective indices for evaluation and cross-referencing.

When the performance matrix proposed by Ye and Leydesdorff (2014) is extended to a primary matrix, secondary matrix, and submatrix, the traces of the three performance matrices _{1}, _{2}, and

Commonly used bibliometric indicators such as citation count and average citation are single point indicators, and they cannot accurately reflect variations in a rank-citation curve. Although the

We observed that the differences in trace metrics were greater than those in the average citation values and in the

For the trace metrics _{1} and _{2}, there was a negative term _{1} or _{2} value (e.g. Section 3.1 and Figure 3 show that IBM has a negative _{1} value of approximately −5000). Therefore, these two indicators can facilitate decision makers in examining the impact efficiency of their organization.

If a university receives a negative _{1} or _{2} value, which means that it has produced few high-impact papers (but numerous irrelevant papers), we suggest that the governors of the university examine their research policy. Perhaps they should combine several less-impact projects into a more influential large project to advance their impact. Furthermore, if trace metrics are used to evaluate universities, the negative effect engendered by having many irrelevant papers can encourage universities to conduct substantial research or to publish comprehensive works, instead of several short and separated papers that increase the number of publications.

For patent owners, a negative trace metric value indicates imbalanced research and development distribution toward low-value patents. This might be tolerable for large enterprises because they might have sufficient capital to fabricate a long-term patent portfolio. However, for small businesses, it might indicate an impending financial failure to have such a negative value. By contrast, because the citing practice in patentometrics is different than in bibliometrics, and because certain patents receive low or zero citation despite being valuable, a negative value in _{1} or _{2} might be acceptable. We suggest that company managers regularly review their own patents by using trace metrics. Because the maintenance fee for patents is a financial burden, managers can use trace metrics as a supplement to examine the value of their patents to determine which patents should be maintained.

The meaning of the negative term

A recent popular topic in bibliometrics and university evaluation is field normalization. This issue is usually discussed in university evaluation, and more and more global university ranking systems have adopted field normalization to reduce the field bias of publications and citations of different research-oriented universities. In our bibliometric test, we have already chosen a field so we could basically bypass this issue. Moreover, if we look into the subfields of computer sciences, we find that most of them have similar numbers of publications and citations therefore the field normalization issue could also be disregarded.

For our patentometric test, as we used the NBER categories, in which the smallest division is the computer software and hardware, to select our patent data, it is impossible for us to do field normalization in our patentometric analysis. However, for future research dealing with other fields, especially for fields that have significant bibliometric differences among their subfields, field normalization might be considered when evaluating the trace metric performance.

Our analysis reveals that trace metrics, which consider zero citation as a negative contribution, provide a unique view on the impact efficiency of an organization. We also determined that trace metrics exhibit different indicating behaviors between hardware patents and software patents, whereas commonly used indicators such as average citation and

#### Spearman and Pearson correlation coefficients among the C/P, h, T1, T2, and ST of the top 30 assignees.

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | −0.248 | 0.093 | 0.932 Significant correlation at 0.01 level. | 0.275 | |

_{2} | −0.507 Significant correlation at 0.01 level. | 1 | 0.655 Significant correlation at 0.01 level. | −0.094 | 0.519 Significant correlation at 0.01 level. | |

−0.482 Significant correlation at 0.01 level. | 0.881 Significant correlation at 0.01 level. | 1 | 0.275 | 0.799 Significant correlation at 0.01 level. | ||

0.303 | −0.099 | 0.177 | 1 | 0.488 Significant correlation at 0.01 level. | ||

−0.151 | 0.646 Significant correlation at 0.01 level. | 0.750 Significant correlation at 0.01 level. | −0.121 | 1 |

#### Spearman and Pearson correlation coefficients among the C, h, T1, T2, and ST of the top 30 highly cited patents.

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.830 Significant correlation at 0.01 level. | 0.988 Significant correlation at 0.01 level. | 0.963 Significant correlation at 0.01 level. | 0.892 Significant correlation at 0.01 level. | |

_{2} | 0.276 | 1 | 0.815 Significant correlation at 0.01 level. | 0.839 Significant correlation at 0.01 level. | 0.747 Significant correlation at 0.01 level. | |

0.989 Significant correlation at 0.01 level. | 0.363 Significant correlation at 0.01 level. | 1 | 0.964 Significant correlation at 0.01 level. | 0.905 Significant correlation at 0.01 level. | ||

0.962 Significant correlation at 0.01 level. | 0.410 Significant correlation at 0.01 level. | 0.975 Significant correlation at 0.01 level. | 1 | 0.959 Significant correlation at 0.01 level. | ||

0.976 Significant correlation at 0.01 level. | 0.347 Significant correlation at 0.01 level. | 0.975 Significant correlation at 0.01 level. | 0.985 Significant correlation at 0.01 level. | 1 |

#### Spearman and Pearson correlation coefficients among the C/P, h, T1, T2, and ST of the top 30 universities.

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.581 Significant correlation at 0.01 level. | 0.988 Significant correlation at 0.01 level. | 0.730 Significant correlation at 0.01 level. | 0.721 Significant correlation at 0.01 level. | |

_{2} | 0.593 Significant correlation at 0.01 level. | 1 | 0.598 Significant correlation at 0.01 level. | 0.065 | 0.755 Significant correlation at 0.01 level. | |

0.994 Significant correlation at 0.01 level. | 0.624 Significant correlation at 0.01 level. | 1 | 0.715 Significant correlation at 0.01 level. | 0.749 Significant correlation at 0.01 level. | ||

0.766 Significant correlation at 0.01 level. | 0.104 | 0.752 Significant correlation at 0.01 level. | 1 | 0.460 Significant correlation at 0.01 level. | ||

0.751 Significant correlation at 0.01 level. | 0.745 Significant correlation at 0.01 level. | 0.790 Significant correlation at 0.01 level. | 0.519 Significant correlation at 0.01 level. | 1 |

#### List of top 30 highly cited paper in the field “computer sciences” from ESI.

No. | Authors | Journal | Publication year |
---|---|---|---|

P1 | Robinson, M.D. et al. | 2010 | |

P2 | Li, H. & Durbin, R. | 2010 | |

P3 | Edgar, R.C. | 2010 | |

P4 | Quinlan, A.R. & Hall, I.M. | 2010 | |

P5 | Bullard, J.H. et al. | 2010 | |

P6 | Smoot, M.E. et al. | 2011 | |

P7 | Willer, C.J. et al. | 2010 | |

P8 | Wu, T.D. & Nacu, S. | 2010 | |

P9 | Wang, L.K. et al. | 2010 | |

P10 | Pruim, R.J. et al. | 2010 | |

P11 | Quince, C. et al. | 2011 | |

P12 | Milne, I. et al. | 2010 | |

P13 | Caporaso, J.G. et al. | 2010 | |

P14 | Hadfield, J.D. | 2010 | |

P15 | Edgar, R.C. et al. | 2011 | |

P16 | MacLean, B. et al. | 2010 | |

P17 | Friedman, J. et al. | 2010 | |

P18 | McLaren, W. et al. | 2010 | |

P19 | Hyatt, D. et al. | 2010 | |

P20 | Viechtbauer, W. | 2010 | |

P21 | Kembel, S.W. et al. | 2010 | |

P22 | Danecek, P. et al. | 2011 | |

P23 | Martin, D.P. et al. | 2010 | |

P24 | Huang, Y. et al. | 2010 | |

P25 | Pluskal, T. et al. | 2010 | |

P26 | Yu, N.Y. et al. | 2010 | |

P27 | O’Boyle, N.M. et al. | 2011 | |

P28 | Dweep, H. et al. | 2011 | |

P29 | Baraniuk, R.G. et al. | 2010 | |

P30 | Robin, X. et al. | 2011 |

#### Original records of top 30 cited patents in “computer software & hardware”.

Patent | Patent number | Year | Assignee |
---|---|---|---|

P1 | 7665051 | 2010 | Qimonda AG |

P2 | 7802219 | 2010 | Cadence Design Systems, Inc. |

P3 | 7917877 | 2011 | Cadence Design Systems, Inc. |

P4 | 7712056 | 2010 | Cadence Design Systems, Inc. |

P5 | 7962867 | 2011 | Cadence Design Systems, Inc. |

P6 | 7992122 | 2011 | GG Technology, Inc. |

P7 | 7971160 | 2011 | Fujitsu Semiconductor Limited |

P8 | 7738971 | 2010 | Ethicon Endo-Surgery, Inc. |

P9 | 8306853 | 2012 | Colts Laboratories |

P10 | 7693720 | 2010 | VoiceBox Technologies, Inc. |

P11 | 7949529 | 2011 | VoiceBox Technologies, Inc. |

P12 | 8301709 | 2012 | Google Inc. |

P13 | 7685126 | 2010 | Isilon Systems, Inc. |

P14 | 7716171 | 2010 | EMC Corporation |

P15 | 7650009 | 2010 | Digimarc Corporation |

P16 | 7809167 | 2010 | Bell_Matthew |

P17 | 8032409 | 2011 | Accenture Global Services Limited |

P18 | 7840537 | 2010 | CommVault Systems, Inc. |

P19 | 7827208 | 2010 | Facebook, Inc. |

P20 | 7643649 | 2010 | Digimarc Corporation |

P21 | 7647237 | 2010 | MiniMed, Inc. |

P22 | 7698160 | 2010 | VirtualAgility, Inc |

P23 | 7685254 | 2010 | Pandya_Ashish A. |

P24 | 7697719 | 2010 | Digimarc Corporation |

P25 | 7751596 | 2010 | Digimarc Corporation |

P26 | 7657849 | 2010 | Apple Inc. |

P27 | 7760905 | 2010 | Digimarc Corporation |

P28 | 7797204 | 2010 | Balent_Bruce F. |

P29 | 7653883 | 2010 | Apple Inc. |

P30 | 8200775 | 2012 | Newsilike Media Group, Inc |

#### Spearman and Pearson correlation coefficients among the C/P, h, T1, T2, and ST of the top 30 highly cited papers.

Spearman | _{1} | _{2} | ||||
---|---|---|---|---|---|---|

Pearson | ||||||

_{1} | 1 | 0.868 Significant correlation at 0.01 level. | 0.994 Significant correlation at 0.01 level. | 0.926 Significant correlation at 0.01 level. | 0.922 Significant correlation at 0.01 level. | |

0.820 Significant correlation at 0.01 level. | 1 | 0.838 Significant correlation at 0.01 level. | 0.746 Significant correlation at 0.01 level. | 0.907 Significant correlation at 0.01 level. | ||

0.992 Significant correlation at 0.01 level. | 0.828 Significant correlation at 0.01 level. | 1 | 0.926 Significant correlation at 0.01 level. | 0.915 Significant correlation at 0.01 level. | ||

0.901 Significant correlation at 0.01 level. | 0.627 Significant correlation at 0.01 level. | 0.880 Significant correlation at 0.01 level. | 1 | 0.867 Significant correlation at 0.01 level. | ||

0.793 Significant correlation at 0.01 level. | 0.897 Significant correlation at 0.01 level. | 0.827 Significant correlation at 0.01 level. | 0.706 Significant correlation at 0.01 level. | 1 |