Comparative Study of Trace Metrics between Bibliometrics and Patentometrics

We extended the performance matrix proposed by Ye and Leydesdorff (2014) to a primary matrix V₁, a secondary matrix V₂, and a submatrix SV (Huang et al., 2015), which consider the overall effects of citation distribution and publication distribution. V1=Y1Y2Y3X1X2X3Z1Z2Z3=YXZ,$$\begin{array}{} V_1=\left(\begin{array}{} Y_1 & Y_2 & Y_3\\ X_1 & X_2 & X_3\\ Z_1 & Z_2 & Z_3 \end{array}\right)=\left(\begin{array}{} Y\\ X\\ Z \end{array}\right), \end{array} $$(8)V2=X1X2X3Y1Y2Y3Z1Z2Z3=XYZ,$$\begin{array}{} V_2=\left(\begin{array}{} X_1 & X_2 & X_3\\ Y_1 & Y_2 & Y_3\\ Z_1 & Z_2 & Z_3 \end{array}\right)=\left(\begin{array}{} X\\ Y\\ Z \end{array}\right), \end{array} $$(9)SV=YhY2X1X2,$$\begin{array}{} SV=\left(\begin{array}{} Y_h & Y_2\\ X_1 & X_2 \end{array}\right), \end{array} $$(10)

where Xi=PiPiP$\begin{array}{} \displaystyle X_i=P_i\frac{P_i}{P} \end{array} $ is an I3-type score of publications and Yi=CiCiC$\begin{array}{} \displaystyle Y_i=C_i\frac{C_i}{C} \end{array} $ is an I3-type score of citations. For V₁ and V₂, X1=PcPcP,X2=PtPtP,$\begin{array}{} \displaystyle X_1=P_c\frac{P_c}{P},\,X_2=P_t\frac{P_t}{P}, \end{array} $ and X3=PzPzP,$\begin{array}{} \displaystyle X_3=P_z\frac{P_z}{P}, \end{array} $ whereas Y1=CcCcC,Y2=CtCtC,$\begin{array}{} \displaystyle Y_1=C_c\frac{C_c}{C},Y_2=C_t\frac{C_t}{C}, \end{array} $ and Y3=CeCeC.$\begin{array}{} \displaystyle Y_3=C_e\frac{C_e}{C}. \end{array} $ For SV, Yh=ChChC,$\begin{array}{} \displaystyle Y_h=C_h\frac{C_h}{C}, \end{array} $

where C_c is the number of citations in the h-core-area, equaling h²;

C_t is the number of citations in the t-area;

C_e is the number of citations in the e-area;

C_h is the number of citations of the h-area, C_h = C_c + C_e;

C is the total number of citations, equaling C_c + C_t + C_e;

P_c is the number of papers in the h-area, equaling h;

P_t is the number of papers in the t-area;

P_z is the number of papers having zero citations; and

P is the total number of papers, equaling P_c + P_t + P_z.

The vectors X = (X₁X₂, X₃) and Y = (Y₁, Y₂, Y₃) are publication and citation vectors, respectively.

The three traces of matrices V₁, V₂, and SV can then be used to obtain indicators T₁, T₂, and ST as follows: T1=Tr(V1)=Y1+X2+Z3=Cc2C+Pt2P+(Ce2C−<!-- - -->Pz2P),$$\begin{array}{} \displaystyle T_1={\rm Tr}(V_1)=Y_1+X_2+Z_3=\frac{C^2_c}{C}+\frac{P^2_t}{P}+(\frac{C^2_e}{C}-\frac{P^2_z}{P}), \end{array} $$(11)T2=Tr(V2)=X1+Y2+Z3=Pc2P+Ct2C+(Ce2C−<!-- - -->Pz2P),$$\begin{array}{} \displaystyle T_2={\rm Tr}(V_2)=X_1+Y_2+Z_3=\frac{P^2_c}{P}+\frac{C^2_t}{C}+(\frac{C^2_e}{C}-\frac{P^2_z}{P}), \end{array} $$(12)ST=Tr(SV)=Yh+X2=Ch2C+Pt2P.$$\begin{array}{} \displaystyle ST={\rm Tr}(SV)=Y_h+X_2=\frac{C^2_h}{C}+\frac{P^2_t}{P}. \end{array} $$(13)

Both T₁ and T₂ summarize the representative information distributed over the e-, h-, t-, and uncited areas in the rank–citation graph.

For a demonstration of trace metrics, we applied traces T₁, T₂, and ST to both bibliometrics and patentometrics to investigate the performance of an institution (e.g. a university or a company) and that of a single document (e.g. a paper or a patent). The traces in group level can be called academic traces (for universities or scientists) or assignee traces (for companies or assignees) and those in individual level can be called impact traces (for papers) or patent traces (for patents).

In this research, we used full counts to assign credits of publications to organizations. Although some might debate that using full counts in bibliometrics would magnify the actual number of publications, full counting is the most intuitive and currently most widely-used counting method in bibliometrics. From the perspective of patents, only few patents have more than one assignee. Zheng et al. (2013) studied the influence of counting methods in patentometrics and found that the difference among different counting methods is slight. In this preliminary reseach of applying trace metrics in bibliometrics and patentometrics, we chose to compare the trace metric performance of universities and companies using a full counting method and to leave the author contribution-credit issue to future work.

We used the traces T₁, T₂, and ST on both bibliometrics and patentometrics. For an informetric test, we applied traces to investigate the performance of the top 30 universities with respect to the computer sciences according to the 2014 Academic Ranking of World Universities (ARWU) – computer sciences. We also applied traces to the top 30 most cited papers from Essential Science Indicators (ESI) Highly Cited Papers – computer sciences, published in March 2015. The five year (i.e. from 2010/01/01 to 2014/12/31) bibliographic data of the top universities and the highly cited papers were collected from the Web of Science database updated on 2015/04/08, which means the citation counts were accumulated from 2010/01/01 to 2015/04/08. In order to compare the trace performances among the 30 universities in the computer sciences, we applied the ESI journal list to confine the bibliometric data we analyzed to the field of computer sciences.

For a patentometric test, we selected the top 30 assignees who owned the most patents in the National Bureau of Economic Research (NBER) computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. Similar to the procedure used for the bibliometric test, we selected the top 30 most cited US patents in the NBER computer hardware and software category that were issued from 2010/01/01 to 2014/12/31. All patent data were obtained from the United States Patent and Trademark Office database.

The datasets covered group level (universities and companies) and individual level (paper and patent, individually as a single publication). For calculating the traces of a single document (a highly cited paper or patent), we followed Schubert’s (2009) method to construct a rank–citation graph of the single document by determining the number and citations of citing documents (i.e. documents that cite the document under consideration). Therefore, the h-index of a single document could be determined.

Applying trace metrics to a university enables assessing its academic performance. We call such metrics academic traces.

Figure 2 shows the values of academic traces T₁ (solid blue line with squares), T₂ (solid orange line with triangles), and ST (solid green line with circles); this figure also shows the typical academic indicators of average citations per paper (C/P, brown dashed line with x) and the h-index (gray bar plot) of the top 30 computer science universities in the ARWU 2014 subject ranking. From left to right, the universities are listed in descending order according to total citations. Academic traces T₁, T₂, and ST share the same scale and are expressed along the left vertical axis, whereas C/P and the h-index were much lower than T₁, T₂, and ST, and they are expressed along the right vertical axis. Generally, T₁, T₂, ST, and h followed this descending trend, except for Tsinghua University and Carnegie Mellon University (CMU), which exhibited a rise in T₂ and a drop in the h-index. However, two universities, Taiwan University and Israel Institute of Technology (Techion), showed a drop in the h-index, but they did not show a rise in T₂. We carefully examined these four datasets and determined that, compared with other universities, Tsinghua University and CMU published more papers having numbers of citations that were lower than the h-index but higher than 0 (i.e. a higher P_t, resulting in a higher C_t and causing a square effect on T₂). These h-drop-T₂-rise universities were compared with T₂-drop-ST-rise universities such as the University of California, Berkeley, University of California, San Diego, University of Toronto, University of Michigan, and California Institute of Technology (CalTech), which had fewer total papers P and thus a higher Pt2/P$\begin{array}{} \displaystyle P^2_t/P \end{array} $ and Pz2/P.$\begin{array}{} \displaystyle P^2_z/P. \end{array} $ The T₂-drop-ST-rise universities can be explained as being low-publication but high-citation universities, which can also be proven by their above-average C/P values.

Figure 2

Table 1 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among T₁, T₂, ST, C/P, and the h-index. The three trace metrics can be divided into two groups: The first group contained T₁ and ST, which had high correlation coefficients with the commonly used bibliometric indicators C/P and h, whereas the second group contained T₂, which had a low correlation coefficient with the average indicator C/P, but it was still highly correlated with h. We found that, although both T₂ and C/P are highly correlated with T₁, ST and h, they do not show a correlation with each other. This means that T₂ and C/P may provide us with important information as T₁, ST and h do, but from differenct perspectives.

Table 1

Spearman and Pearson correlation coefficients among the C/P, h, T₁, T₂, and ST of the top 30 universities.

Several bibliometric indicators are used in patentometrics for estimating the performance of patents. Similar to the procedures performed for bibliometrics, a company can be evaluated according to the performance of its patents. When trace metrics are applied to a group level of patent, they are called assignee traces.

Figure 3 illustrates the values of assignee traces T₁ (blue solid line with squares), T₂ (orange solid line with triangles), and ST (green solid line with circles); this figure also shows the commonly used bibliometric indicators of average citations C/P (brown dashed line with x) and the h-index (gray bar plot) of the top 30 assignees in the NBER computer hardware and software category. The assignees are listed from left to right in descending order according to the total citations of their patents. T₁, T₂, and ST share the same scale and are expressed along the left vertical axis, whereas C/P and the h-index were much lower than T₁, T₂ and ST, and are expressed along the right vertical axis. In general, all indicators followed a descending trend, except for IBM, which had the most citations and a relatively high h-index but a considerably negative T₁ value. The reason for the drop in T₁ is that IBM has many zero-citation patents, leading to a considerably large value for Pz2P,$\begin{array}{} \displaystyle \frac{P^2_z}{P}, \end{array} $ subtracted from a considerably lower value Ce2C.$\begin{array}{} \displaystyle \frac{C^2_e}{C}. \end{array} $ In addition, software companies such as Microsoft, Google, Oracle, Amazon, Yahoo, and Digimarc achieved more satisfactory trace performance levels than hardware companies such as IBM, Apple, Sony, HP, and SAP did.

Figure 3

Table 2 shows Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among T₁, T₂, ST, C/P, and the h-index. In Pearson’s correlation analysis, T₁ negatively correlated with T₂, ST, and the h-index (Table 3). Compared with papers, most patents had relatively low citation values and thus, a low h-index and C_e and a high P_z, leading to a low T₁; hence, the trend of T₁ was different from that of T₂, ST, and the h-index.

Table 2

Spearman and Pearson correlation coefficients among the C/P, h, T₁, T₂, and ST of the top 30 assignees.

Table 3

Spearman and Pearson correlation coefficients among the C/P, h, T₁, T₂, and ST of the top 30 highly cited papers.

At the group level, we observed that for both universities and companies, the difference between their average citation and h-index values was small. The average citation value for the top 30 universities was approximately 5, whereas that for the top 30 companies was approximately 2. The h-index ranged from 15 to 40 for the universities, and it extended from 10 to 35 for the companies. The differences in trace metrics between universities and between companies were more significant. Most trace metrics varied from 0 to 2000 for the top 30 universities, and stretched from −1000 to 1500 for the top 30 companies. We considered zero citation as a negative contribution, and there were more zero-citation patents than zero-citation papers. Therefore, numerous companies had a negative T₁ value. This negative value can be considered a warning, rather than being perceived as indicating no market value; accordingly, patents’ potential market value should be investigated. Compared with an acceptable negative trace metric value in patentometrics, a negative trace metric value in bibliometrics indicates poor efficiency in conducting crucial research. Therefore, if a university receives a negative trace metric value, it should examine its research projects and consider adjusting them.

We determined that in contrast to the patentometric indicators, all bibliometric indicators showed significant correlations. This discrepancy means that bibliometric indicators as well as traces are generally applicable and that other factors such as market elements must be considered in patentometric indicators to ensure their applicability.

In addition to the universities, the trace metrics were applied to a single paper to evaluate its impact. We called these metrics impact traces.

Figure 4 shows the values of impact traces T₁ (solid blue line with squares), T₂ (solid orange line with triangles), and ST (solid green line with circles) as well as the commonly used academic indicators of average citations C/P (brown dashed line with x) and the h-index (gray bar plot) of the top 30 most cited computer science papers according to ESI data obtained in March 2015. Table A1 lists detailed information on these papers. The most cited papers were named according to their rank in citation; that is, the most cited paper was named P1, and the second most cited paper was named P2. Impact traces T₁, T₂, and ST share the same scale and are expressed along the left vertical axis, whereas C/P and the h-index were much lower than T₁, T₂, and ST, and are expressed along the right vertical axis. In general, all trace metrics followed a descending trend from left to right, except for P5, which exhibited a rise in T₁ and ST, and P3, P4, and P6, which exhibited a drop in T₁ and ST. These results may be attributable to P5 having a relatively low number of citing papers with zero citations and P3, P4, and P6 having a relatively high number of citing papers with zero citations. Compared with academic traces, impact traces had similar values but were less consistent among T₁, T₂, and ST.

Figure 4

Table 3 lists Pearson (bottom left part of the table, without a background) and Spearman (top right part of the table, with a gray background) correlation coefficients among T₁, T₂, ST, C/P, and the h-index. For highly cited papers, impact traces were highly correlated with average citations and the h-index.

Similar to our previous bibliometric analysis, the impact of a single patent was studied using trace metrics (subsequently denoted as patent traces).

Figure 5 illustrates the values of patent traces T₁ (solid blue line with squares), T₂ (solid orange line with triangles), and ST (solid green line with circles) as well as the commonly used indicators of average citations C/P (brown dashed line with x) and the h-index (gray bar plot) of the top 30 most cited patents in the NBER computer hardware and software category. Table A2 lists detailed information on these patents. Patent traces T₁, T₂, and ST share the same scale and are expressed along the left vertical axis, whereas C/P and the h-index were much lower than T₁, T₂, and ST and are expressed along the right vertical axis. These most cited patents are listed in descending order from left to right according to total citations. The top seven most cited patents exhibited a relatively low T₂ value (Figure 5), and thus, a low Pearson correlation coefficient was observed between T₂ and other indicators (Table 4). All the top seven most cited patents had a relatively high h-index, possibly indicating centrality in the h-core and thus a low C_e value. In contrast to the assignee traces, marked differences existed among patent traces T₁, ST, and T₂. After carefully examining these patents, we observed that the hardware patents P1–P7 had a relatively high h-index value and thus, exhibited considerable differences among T₁ and ST, which were dominated by h⁴, and T₂, which was proportional to h².

Figure 5

Table 4

Spearman and Pearson correlation coefficients among the C, h, T₁, T₂, and ST of the top 30 highly cited patents.

Table 4 lists Pearson (bottom left part of the table, with no background) and Spearman (top right part of the table, with a gray background) correlation coefficients among T₁, T₂, ST, C/P, and the h-index. Most of the indicators demonstrated a satisfactory correlation coefficient with other indicators, except for the Pearson correlation coefficient of T₂.

At the individual level, the differences in the average citation values and in the h-index values of the top 30 most cited papers were small. For the top 30 most cited patents, we determined that they could be divided to two groups: P1 to P7 had higher values of ST and T₁ and a lower value of T₂, and P8 to P30 had approximately the similiar value of ST, T₁, and T₂. The difference was due to the different citation types between the software and hardware patents.

Typically, an object receives trace metrics with a higher T₂, a higher ST that is lower than T₂, and a lower T₁. However, we observed that T₂ was the lowest trace metric value of the top 30 most cited patents. A lower T₂, which considered the square of the citations in the tail part of the most cited patents, indicated that such citations were not comparable to those of their paper counterparts. However, although the average citation value in the tail part was lower than the h-index value, the values of the total citations in the tail part were usually higher than those of the total citations in the h-region (the core and the accessed parts). The rank-citation curves of the most cited papers, universities, and companies were gradual and had thick and long tails, but the corresponding rank-citation curves of patents were steep, with most citations accumulated in the h-area, and they had thin and short tails. Therefore, a lower T₂ value represented a steep rank-citation curve, which is acceptable in patentometrics but is a symbol of irrelevance in bibliometrics.

We also determined that, at the individual level, all indicators showed significant correlations in both bibliometrics and patentometrics, demonstrating that all indicators, including traces, were effective indices for evaluation and cross-referencing.