A comprehensive review of existing corpora and methods for creating annotated corpora for event extraction tasks

To facilitate the rapid development of an annotated corpus, it is suggested to employ a hybrid approach as demonstrated by Li et al. (2022).

This involves partially annotating the texts manually, and then training ML algorithms to annotate the remaining data based on the trained model.

This strategy is faster than manually annotating all data. However, it is critical to measure the accuracy of the automatic annotations.

Develop a standardized annotation format that is universally accepted for annotating text corpus.

These formats should store all information required for common EE tasks.

Develop a universal text annotation converter for converting annotations between different formats (Figure 16).

Develop a complete annotation guideline and strictly adhere to it throughout the annotation process.

Utilize tools like Git version control to manage the version of annotation files.

The proposed method can handle large-scale text annotation tasks.

Provides a cost-effective alternative to annotate large amounts of text.

Annotation using LLMs slightly compromises the annotation accuracy.

LLMs alone cannot provide high-quality corpus annotations. The annotated corpus is not suitable for EE task.

The proposed approach improved result by 2%.

All models show improved performance with the GPT-4 + UD Turkish BOUN v2.11: 76.9% (best performance).

The model has been tested with data from UD English and Turkish Treebanks.

The authors use public data and verify the methodology complies with ethical standards.

The annotation outcome varies (inconsistent) depending on the user’s prompt.

The method is for entity annotation; thus output is not suitable for EE tasks.

gbert-large (P: 70.7%, R:97.9%, F1: 82.1%)

GottBERT-base (P: 80.0%, R: 89.9%, F1: 84.7%)

German-MedBERT (P: 72.7%, R: 81.8%, F1: 77.0%)

Solves limited corpus availability for non-English medical texts.

The proposed method shows a reliable performance.

The proposed method is computationally expensive.

The annotated corpus cannot be considered a gold-standard and requires more validation.

The method is for entity annotation, thus output is not suitable for EE tasks.

The annotation process is done together by humans and LLMs.

Provides a cost-effective alternative to annotate large amounts of text.

The study does not assess if LLM-generated annotations outperform human-annotated corpus.

The method is for entity annotation, thus output is not suitable for EE tasks.