Multimodal detection framework for financial fraud integrating LLMs and interpretable machine learning

We utilized the Doubao LLMs from ByteDance’s Volcano Ark platform to develop a semantic feature extraction framework for processing the full text of annual reports. The framework comprises four sequential steps (Fig. 2): text chunking, chunk summarization, full-text summarization, and embedding vector generation.

Figure 2.

Annual report semantic feature extraction framework.

Given the typical length of annual reports (exceeding 30,000 characters for MD&A in our corpus), the original text is first segmented into 1,024-character chunks to preserve contextual semantics. Each chunk is processed via the Doubao-pro-32k_v241215 model to generate localized summaries (prompt templates detailed in Table 1). These chunk summaries are then joined and input into the same model to produce a consolidated full-text summary, which distills the key content of the MD&A with logical coherence and linguistic fluency. Finally, semantic embeddings are derived from the full-text summary using the Doubao-embeddìng_v240715 model. Table 1 specifies the LLM configurations and prompt engineering strategies for the summarization and embedding task.

Textual sentiment features, defined as tone, are quantified through two indicators: full-text tone and MD&A-specific tone. Both metrics are sourced from the Annual Report Sentiment Database from the Chinese Research Data Services platform (CNRDS)^②. Following Zeng et al. (2018), the tone values provided by the database are computed using Equation (1), where POSword and NEGword represent the counts of positive and negative words, respectively, identified using the Loughran-McDonald (LM) Financial Sentiment Dictionary, adapted for Chinese-English financial contexts. Generally, a higher tone score indicates more optimistic management.1Tone =POSword - NEGword POSword + NEGwordTone = {{POSword - NEGword } \over {POSword + NEGword }}

In evaluating information, text readability often reflects its complexity. Drawing from financial research (Wang et al., 2018), we propose five readability metrics: 1)

Text length (Size) and Long sentence ratio (Ls). Extended text length and frequent long sentences reduce readability. Following established practices (Wang et al., 2018), Size (total character count) and Ls (proportion of sentences exceeding the average sentence length) serve as baseline readability indicators.

The study employed four algorithms—LR, XGBoost, LightGBM, and CatBoost—to construct corporate financial fraud detection models. LR is the baseline method due to its simplicity and strong interpretability, making it widely adopted in existing research. However, it exhibits limitations in addressing complex nonlinear problems. This study evaluates the performance of gradient-boosted decision tree (GBDT) algorithms in financial prediction tasks. According to the theoretical framework proposed by Chen and Guestrin (2016), GBDT minimizes loss functions by integrating weak learners through iterative optimization. Prior studies demonstrated that treebased ensemble methods outperform deep neural networks in predicting structured tabular data (Lundberg et al., 2020). These advantages primarily manifest in three aspects: (1) enhanced modeling capability for nonlinear relationships and complex feature interactions, (2) improved robustness during training, and (3) superior adaptability in parameter sensitivity, computational efficiency, and missing value handling. The three GBDT variants selected in this study exhibit distinct characteristics: XGBoost enhances generalization and computational efficiency through regularization terms and parallel computing techniques; LightGBM reduces model error via histogram-based algorithms and leaf-wise growth strategies; CatBoost excels in handling categorical features while maintaining high precision and stability.

The study utilizes ensemble tree algorithms to build fraud detection models and systematically applies SHAP to analyze feature importance and impact mechanisms in fraud identification. Rooted in Shapley value theory from game theory (Shapley, 1953), SHAP decomposes model predictions into weighted contributions from input features, thereby enhancing the interpretability of complex machine learning models. Under the SHAP framework, the prediction output y_i for a given sample x_i can be expressed as a linear combination of all feature contributions, as shown in Equation (2): 2yi=ybase+∑j=1Mf(xij){y_i} = {y_{base{\rm{ }}}} + \sum\limits_{j = 1}^M f \left( {{x_{ij}}} \right)

Here, y_base represents the global mean of the target variable (baseline value), x_ij denotes the j-th feature of sample x_i and f(x_ij) corresponds to the SHAP value of the feature, reflecting its marginal contribution to the prediction. A positive SHAP value indicates a promotive effect on fraud risk, while a negative value suggests an inhibitory effect. By quantifying these directional impacts, SHAP elucidates the model’s decision logic. As illustrated in Fig.3, the baseline value y_base serves as the initial reference (e.g., average predicted probability). Arrows depict deviations from this baseline, e.g. feature x_i1 exerts a positive contribution (f(x_i1) >0), elevating the prediction above the baseline, while x_i3 exhibits a negative contribution, reducing the prediction. The final prediction results from the algebraic sum of all feature contributions. SHAP provides post-hoc explanations applicable to various machine learning algorithms, particularly excelling in interpreting nonlinear decision processes of tree-based ensemble models.

Figure 3.

SHAP feature attribution diagram.

This study’s target variable is corporate fraud risk, and three predictors are financial indicators, corporate governance features, and annual report disclosures. SHAP analysis quantifies the marginal contributions of predictors to fraud risk, identifies early-warning indicators significantly associated with fraudulent behavior, and deepens understanding of fraud mechanisms.

For annual report analysis, this study employs the Doubao LLMs developed by the ByteDance Volcano Ark team^③. Two core components are utilized: the general-purpose model (Doubao-pro-32k_v241215) and the text embedding model (Doubao-embedding_v241215). Doubao-pro-32k_v241215 generates outlines of annual report content. Optimized on the transformer architecture, it supports a context window of up to 32,000 Chinese characters, making it particularly suitable for processing lengthy texts like corporate annual reports. On the Large Language Model Evaluation Benchmark released by the Beijing Academy of Artificial Intelligence (BAAI), Doubao-pro-32k ranks first in comprehensive performance among global open-source and commercial closed-source models^④. Through API integration, the AI-generated model successfully processed 5,304 listed companies’ MD&A sections, generating 1,024-word summaries per MD&A on average to distill critical corporate disclosures.

For feature extraction, the text embedding model Doubao-embedding_v241215 is deployed. Trained via a contrastive learning framework, it supports semantic vector representations up to 4,096 dimensions. We systematically tested four embedding dimensions (1,024, 512, 256, and 128) to obtain the best performance. The 256-dimensional representations were selected as the final features based on the validation set AUC metrics.

Following the processing pipeline illustrated in Fig. 3, we analyzed 5,304 annual reports. A representative example includes the MD&A section of Zangge Holdings’ 2019 annual report (Stock Code: 000408), where Doubao LLM condensed the original 15,659-character text into a 469-character semantic summary. This resulted in a 96.6% compression rate, while preserving critical fraud-related signals (Table 5). Importantly, this demonstrates the framework’s capacity to not just condense data, but to extract actionable insights from voluminous unstructured data, enhancing its practical value.

The AI-generated summary effectively distilled the MD&A content, organizing key themes such as revenue, cash flow, investments, and future outlooks. For example, the summary highlighted critical financial data: “ …with revenue of 2.018 billion yuan and net profit of 308 million yuan. The decline in performance was attributed to factors such as product sales volume, price, production volume, costs, and inventory depreciation.” Notably, these financial irregularities—declining performance, asset impairment, and investment losses—are strongly associated with elevated financial risk. This case was later confirmed by the China Securities Regulatory Commission (CSRC) as involving financial fraud, validating the predictive efficacy of the extracted features.

In subsequent experiments, we employed text embeddings generated by a large language model (LLM) to train a multilayer perceptron (MLP) classifier for detecting corporate financial fraud. Embeddings with four different dimensions (1024, 512, 256, and 128) were evaluated. As shown in Table 6, the 256-dimensional embeddings achieved the best performance, yielding an accuracy of 0.742, an F1 score of 0.733, and an AUC of 0.817. While theoretically, higher-dimensional embeddings such as the 1024-dimension may offer greater representational power, our results suggest that they are more prone to overfitting, leading to diminished performance.

To further assess model effectiveness, we included three widely-used open-source Chinese text embedding models—Text2Vec-based-Chinese, Embedding-Vl, and BGE-large-zh—alongside our chosen model, and applied all four to the fraud prediction task. As presented in Table 7, the commercial embedding model from Volcano Engine consistently outperformed the open-source alternatives on our dataset. Notably, it also overcomes the 512-token sequence length limitation inherent to conventional BERT models, thus better capturing the semantics of longer texts.

Based on these findings, we ultimately selected the 256-dimensional Doubao-embedding_v2407l5 model to represent the semantics of annual reports. This choice strikes an optimal balance between effectiveness and efficiency, and is particularly advantageous for the subsequent construction of ensemble tree models.

We constructed fraud detection models using four ML models: LR as the baseline classifier and three gradient-boosted tree methods—LightGBM, XGBoost, and CatBoost. The input features comprised 36 structured variables across four categories: financial metrics (19), corporate governance indicators (11), annual report tones (2), and readability metrics (4). Model training employed grid search with 5-fold cross-validation for hyperparameter optimization, evaluated using accuracy, AUC, and F1 score.

The experimental results (Table 8) reveal that the ensemble tree algorithms outperformed the baseline LR across all metrics, with AUC scores exceeding 0.85. Among them, XGBoost exhibited the highest performance, improving AUC by 12.1%, accuracy by 10.7%, and F1 score by 11.0% compared to LR. The superior performance of ensemble trees likely stems from their ability to capture nonlinear relationships among high-dimensional features and their inherent feature importance ranking, which automatically identifies key predictors. These findings align with the prior research in financial fraud detection, further validating the advantages of ensemble learning for complex financial classification tasks.

To investigate the contribution of individual feature categories, we further conducted an ablation study using XGBoost (the top-performing model). Starting with financial metrics as the baseline, we incrementally incorporated textual features derived from the annual report—tone (2 features), readability(4 features), and semantic (256-dimensional embedding vector)—to assess their predictive impact. The results are presented in Tables 9 and 10.

Table 9 demonstrates significant performance variations across feature combinations in the financial fraud prediction model. The baseline model using financial features achieved an AUC of 0.820, an accuracy of 0.740, and an F1 of 0.694. Tones enhanced performance, increasing AUC and F1 by 0.8% and 3.3%, validating sentiment’s role in fraud detection. Readability metrics improved AUC by 1.3%, underscoring text clarity’s discriminative value, while semantic features (high-dimensional embeddings from summaries) showed the greatest gain with a 3.9% AUC increase, reflecting their capacity to capture latent nuances. Integrating financial indicators with multidimensional textual features (tone, readability, semantics) produced a clear additive effect, leveraging annual reports’ informational richness to boost predictive accuracy and model robustness substantially.

The results (Table 10) revealed that multi-feature fusion significantly enhanced model performance. Using only financial features, the model achieved AUC, accuracy, and F1 of 0.820, 0.740, and 0.694, respectively. Incorporating textual features improved these metrics by 6.0%, 5.8%, and 8.6%. Further adding corporate governance features optimized performance (AUC = 0.894, accuracy = 0.812, F1 = 0.796). Although corporate governance features contributed modestly, their integration with textual features enhanced model robustness, suggesting they capture risk signals beyond annual reports. Thus, multi-feature fusion leverages complementary effects to substantially improve discriminative power.

Based on SHAP analysis, this study systematically evaluated the feature-importance of the optimal XGBoost model (AUC = 0.894, accuracy = 0.812). As illustrated in Fig. 4, financial indicators demonstrate the strongest predictive power among the 296 features. Within the top 10% of important features (Top30), financial indicators accounted for 10 entries. Specifically, the accounts receivable turnover ratio (Actrcbto, reflecting operational efficiency), return on equity (Roe_1, indicating profitability), and operating profit growth rate (Oirt, representing growth capacity) ranked 1st, 2nd, and 4th in the ranking list, highlighting the dominance of financial indicators in fraud prediction.

Figure 4.

Feature importance analysis (Top 10% Features).

Although fewer (4 entries within the top 10%), corporate governance metrics also exhibit notable predictive significance. Shareholding proportions of major shareholders (ShrHolder1 and ShrHolder3) ranked 3rd and 7th, while the management shareholding ratio (BShrRat) ranked 22nd, suggesting that ownership concentration and interest alignment mechanisms in governance structures significantly influence fraud risk. Textual features constitute the majority of Top30 features (16 entries), with 14 semantic features demonstrating predictive value. The f119, the highest-ranked one, placed 9th. Finance density and LM_tone2 ranked 12th and 16th, revealing risk signals embedded in report readability and narrative tone.

The feature importance distribution indicates that while structured financial data remains central to fraud detection, textual features also provide critical supplementary information. Financial metrics capture operational anomalies, governance features reflect institutional incentives for fraud, and textual features extract risk signals from management disclosures. The synergistic integration of multidimensional features offers novel insights for developing more accurate financial fraud early-warning systems.

SHAP analysis (Fig. 5) quantitatively elucidated the relationships between features and fraud risk. The beeswarm plot (Fig. 5) revealed predominantly negative associations between key features and fraud risk. For financial indicators, lower values of operational efficiency metrics (e.g., accounts receivable turnover ratio [Actrcbto] and total asset turnover ratio [Totastto]) correlated with higher risk (SHAP value > 0, blue dots area), while higher profitability metrics (e.g., return on equity [Roe_1]) concentrated in low-risk regions (SHAP value < 0, red dots area). In corporate governance, higher shareholding concentrations among major shareholders (ShrHolderl, ShrHolder3) exhibited significant risk-mitigating effects (SHAP value < 0, blue dots area). In comparison, textual features demonstrated that positive tone (LM_tone2) and higher financial terminology density (Finance_density)in management narratives correlated with reduced fraud risk. Among 14 semantic features, 9 showed risk-suppressing effects (SHAP > 0, red dots area), while 5 exhibited risk-enhancing effects (SHAP value >0, blue dots area).

Figure 5.

The relationship between Top10% important features and corporate fraud risk (Beeswarm plot).

The scatter plot (Fig. 6) further reveals distinct correlation patterns among key semantic features: f119 values exceeding the -0.0625 threshold demonstrate nonlinear risk amplification, while elevated f11 levels exhibit proportional risk mitigation. These results substantiate the operational dynamics of multimodal features in fraud prediction mechanisms.

Figure 6.

Relationship between annual report semantic features (f119, f11) and corporate fraud risk.