Evaluating grant proposals: lessons from using metrics as screening device
, y | 07 jun 2023
Acerca de este artículo
Article Category: Research Paper
Publicado en línea: 07 jun 2023
Páginas: 66 - 92
Recibido: 19 dic 2022
Aceptado: 13 abr 2023
DOI: https://doi.org/10.2478/jdis-2023-0010
© 2023 Katerina Guba et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Figure 1.
Figure 2.
Figure 3.
Figure 4.
Project publication output by quartiles (only journal articles included)
| Physics | SSH | |||
|---|---|---|---|---|
| Journal quartiles | First wave (%) | Second wave (%) | First wave (%) | Second wave (%) |
| Q1 | 630 (49.8) | 579 (59) | 42 (8.4) | 48 (13) |
| Q2 | 298 (23.6) | 166 (16.9) | 93 (18.7) | 72 (19.5) |
| Q3 | 225 (17.8) | 137 (14) | 110 (22.1) | 141 (38.2) |
| Q4 | 112 (8.9) | 100 (10.2) | 253 (50.8) | 108 (29.3) |
| Total | 1265 (100) | 982 (100) | 498 (100) | 369 (100) |
Descriptive statistics of principal investigators’ research performance and project publication output
| Physics | SSH | |||
|---|---|---|---|---|
| 2014 |
2017 |
2014 |
2017 |
|
| Q1 papers (N) before | ||||
| Mean (SD) | 22.3 (58.4) | 17.5 (22.5) | 0.319 (0.845) | 1.22 (2.72) |
| Median [Min, Max] | 11 [0, 422] | 10.5 [0, 114] | 0 [0, 5.00] | 0 [0, 15] |
| Q1 papers (%) before | ||||
| Mean (SD) | 40.7 (24.1) | 48.4 (27.3) | 6.82 (18.4) | 16.2 (24.5) |
| Median [Min, Max] | 34 [0, 92.5] | 50.9 [0, 100] | 0 [0, 100] | 0 [0, 87.5] |
| Q1 papers (N) after | ||||
| Mean (SD) | 5.48 (5.77) | 7.06 (5.44) | 0.447 (0.875) | 1.17 (1.99) |
| Median [Min, Max] | 4 [0, 38] | 6 [0, 31] | 0 [0, 4] | 0 [0, 8] |
| Q1 papers (%) after |
||||
| Mean (SD) | 46.6 (29.9) | 55.9 (27.7) | 9.29 (21.0) | 11.4 (18.3) |
| Median [Min, Max] | 50 [0, 100] | 55.9 [0, 100] | 0 [0, 100] | 0 [0, 80] |
Principal investigators’ publications by journal quartiles (only journal articles included)
| Journal quartiles | Physics | Social science and humanities | ||
|---|---|---|---|---|
| First wave (%) | Second wave (%) | First wave (%) | Second wave (%) | |
| Q1 | 2563 (53.7) | 1437 (52.4) | 30 (12.3) | 50 (24.3) |
| Q2 | 710 (14.9) | 452 (16.5) | 13 (5.4) | 30 (14.6) |
| Q3 | 800 (16.7) | 600 (21.9) | 57 (23.4) | 65 (31.5) |
| Q4 | 703 (14.7) | 253 (9.2) | 144 (59) | 61 (29.6) |
| Total | 4776 (100) | 2742 (100) | 244 (100) | 206 (100) |
Descriptive statistics of principal investigators’ research performance in the five years before they received funding
| Physics | SSH | |||
|---|---|---|---|---|
| 2014 |
2017 |
2014 |
2017 |
|
| All Scopus publications in the 5 years before submitting (N) | ||||
| Mean (SD) | 41.5 (64.5) | 33.4 (31.1) | 2.6 (4.39) | 4.95 (4.68) |
| Median [Min, Max] | 25 [2, 466] | 25 [0, 153] | 1 [0, 29] | 4 [0, 27] |
| Journal papers indexed by Scopus (N) | ||||
| Mean (SD) | 35.3 (61.8) | 31.0 (29.2) | 2.12 (4.03) | 4.8 (4.7) |
| Median [Min, Max] | 21 [1, 450] | 22 [0, 150] | 0 [0, 28] | 4 [0, 27] |
| Journal papers indexed by Scopus (%) | ||||
| Mean (SD) | 82.2 (14.2) | 91 (17.3) | 39.3 (44.1) | 89.4 (27.1) |
| Median [Min, Max] | 84.6 [41.7, 100] | 96.3 [0, 100] | 0 [0, 100] | 100 [0, 100] |
| Conference papers indexed by Scopus (%) | ||||
| Mean (SD) | 14.6 (13.5) | 4.96 (9.16) | 4.02 (16.7) | 0.488 (3.12) |
| Median [Min, Max] | 12.5 [0, 52.9] | 0 [0, 56.3] | 0 [0, 100] | 0 [0, 20] |
| Other Scopus papers (%) | ||||
| Mean (SD) | 3.29 (6.02) | 1.59 (2.89) | 9.84 (22.8) | 2.79 (8.97) |
| Median [Min, Max] | 1.16 [0, 50] | 0 [0, 11.1] | 0 [0, 100] | 0 [0, 33.3] |
Descriptive statistics on projects’ publication output between the two waves of grant competitions
| Physics | SSH | |||
|---|---|---|---|---|
| 2014 |
2017 |
2014 |
2017 |
|
| All publications (N) | ||||
| Mean (SD) | 24.5 (17.4) | 21.9 (11.2) | 46.2 (27.1) | 56.2 (26.7) |
| Median [Min, Max] | 18 [3, 94] | 19 [5, 67] | 46 [5, 111] | 49 [14, 111] |
| Journal papers indexed by Scopus (N) | ||||
| Mean (SD) | 11 (7.83) | 12 (5.61) | 5.3 (4.46) | 9 (6.12) |
| Median [Min, Max] | 9 [0, 43] | 11 [3, 37] | 4 [0, 18] | 9 [0, 40] |
| Journal papers indexed by Scopus (%) |
||||
| Mean (SD) | 51.1 (24.8) | 61.5 (23.3) | 16.0 (17.4) | 19.8 (14.9) |
| Median [Min, Max] | 50 [0, 100] | 62.8 [9.09, 100] | 10.2 [0, 80] | 16.2 [0, 81.3] |
| Non-Russian Scopus journal papers (N) | ||||
| Mean (SD) | 9.24 (6.79) | 10.5 (5.66) | 2.22 (2.75) | 3.63 (3.09) |
| Median [Min, Max] | 8 [0, 41] | 10 [1, 37] | 1 [0, 14] | 3.00 [0, 11] |