Development and initial validation of the Odense Child Trauma Screening: a story stem screening tool for preschool and young schoolchildren

The technique of a narrative story stem assessment tool involves the child being presented with the beginning of a story, i.e. a stem, which the test administrator acts out with affect using a standard set of toys, typically family or animal figures. The story stem consists of an everyday problem with a controlled degree of conflict or distress that is intended to induce emotional arousal in the child (10). When the administrator stops the story stem, the child is asked to continue and play out the story with the figures. When asked to continue the story, the child spontaneously draws on own impulse and mental scripts of social relations and behavior to finish the story (12-14). Story stem tools include several story stems, and the child’s play and narrative representation of feelings and behavior are coded for each story. Use of a story stem tool thus allows for a detailed examination of child mental representation of self, others, and relationships, which have been posited to have a mediating role in how early childhood adversity affects later development (13, 15).

Depending on the purpose of the assessment, different story stem tools with varying story stems and coding schemes can be used. Generally, story stem tools such as the Attachment Story Completion Task (ASCT; 16), the Manchester Child Attachment Story Task (MCAST; 17), the Attachment Doll Play Assessment (ADPA; 18), and the German Attachment Story Completion Procedure (GASCP; 19-21) have been developed to assess attachment in preschool and young school children, while the MacArthur Story Stem Battery (MSSB; 22) and the Story Stem Assessment Profile (SSAP; 5) in addition to attachment, also more broadly assess child mental representations of self, others, expectations of relationships, moral rules, moral emotions, and competences.

Story stem tools have been widely applied in research to examine typically developing children (14, 23) and clinical groups of children with externalizing and disruptive disorders (24-26), with anxiety and mood disorders (23, 27-29), and with children exposed to child maltreatment (5, 13), where the link between play and narrative representations in story stems and symptomatic behavior have been investigated.

Specifically, externalizing and disruptive behaviors have been found to be associated with less compliance and with negative narrative themes and representations (24) such as parental injury and role reversal (25) and disorganization (26). Furthermore, negative representations of self and others have been found to significantly predict internalized symptoms and anxiety symptoms (30); and symptoms of depression have been associated with negative adult representations (27) and narrative themes of shame (28).

In samples of children exposed to child maltreatment negative representations of parents and self, representations of a grandiose self (31), avoidant strategies, rejection, death (15), aggression (32), and role reversal (33) have been identified. Furthermore, children who have experienced neglect, but no abuse, have been found to show more passive parents in relation to child figure distress, whereas children exposed to abuse, especially physical abuse, showed narrative representations with anomalous acts (33).

Lastly, disorganized phenomena displayed within the setting of story stem measures of attachment are found more frequently in children with a trauma history of maltreatment compared to typically developing children (15, 34). Disorganized attachment is characterized by disturbed, conflicted, anomalous, or fearful behavior and a lack of a clear and cohesive, organized attachment strategy (35, 36). Disorganized attachment in children is assumed to develop because the caregiver, who should be a source of comfort and protection, is the source of fear, for instance because the caregiver is the source of maltreatment or because trauma-triggered adult behavior in a traumatized parent, is experienced as frightening and unpredictable by the child (37, 38). Disorganized attachment has shown to be a risk factor for psychopathology of both externalizing and internalizing problems, in childhood and adolescence (38). Specifically, MacDonald and colleagues (39) demonstrated in their longitudinal study that a disorganized attachment in children of 12 months of age significantly predicted a higher posttraumatic stress disorder (PTSD) symptom clusters of re-experience and of avoidance in the children at age 8.5 years but disorganized attachment was not associated with higher level of symptoms of anxiety disorders.

Inspired by the findings described above, the authors decided to develop the OCST (11); a story stem screening tool that, within a controlled and structured play-setting, could function as an indirect screening measure of traumatization for children aged 4-8 years, and a tool that would include the child as informant. The OCTS differs from other story stem tools as it is intended as a screening tool and was developed specifically to be utilized in clinical practice for a broad trauma population where the circumstances and nature of the trauma experience of the child for various reasons may be more or less unknown.

Other story stem tools such as the MSSB and the SSAP are characterized by being comprehensive story stem batteries with several story stems applicable for a thorough and detail-rich assessment. The MSSB has been used in various studies on typically developed children and specific clinical groups in which story stems and coding schemes have been developed to fit the aim of the study and the construct under examination (for a review, please see; 12). The SSAP was developed specifically for assessment of maltreated children (5, 40). The MCAST includes fewer story stems and is characterized by rigorous methodology and detailed scoring and classification of attachment, attachment behavior, and attachment strategies. However, in our experience, the use of comprehensive story stem batteries, the detailed level of coding, and the somewhat extensive and time-consuming labor connected to the existing story stem measures is an issue of concern when incorporating the research-based story stem approach into general clinical practice, where children with a broad range of mental health issues are referred.

The development and adaptation of the OCTS administration and coding scheme were therefore designed to include a relatively short story stem administration with five story stems, standardized rules of interaction and follow-up questions, and a relatively short coding scheme and scoring system to be used in cases of suspicion of child traumatization. The scoring was designed drawing on the vast empirical knowledge from other studies with story stem measures describing characteristics of narratives presented by children exposed to potential traumas. For a full review of the empirical findings on which the codes of the OCTS was derived, please see Eriksen & Elklit (41). The OCTS codes comprise categories of the child’s engagement and production of the narrative, nature of the narrative (e.g., narrative coherence), representations of adults and child in the narrative, and disorganized phenomena (42).

The present study aimed to provide initial examination of the reliability and validity of the OCTS as a screening tool for behavioural and play-based indicators of traumatization in children aged 4 to 8 years. Specifically, the study set out to examine internal consistency, interrater reliability, and test-retest reliability of the OCTS. Furthermore, the study examined multi-method convergent validity of the OCTS when compared to Danish validated measures, specifically modules of PTSD, major depressive disorder (MDD), and reactive attachment disorder (RAD) from the Diagnostic Infant and Preschool Assessment (DIPA) and the Strengths and Difficulties Questionnaire (SDQ) Total Difficulties Scale and subscales. These measures were chosen because PTSD and RAD are trauma related disorders, and symptoms of depression have been shown to be highly comorbid with PTSD (4, 43). Furthermore, exposure to multiple adverse or traumatic events have been shown to be associated with a broad range of internalizing and externalizing symptoms in children (4). Lastly, the ability of the OCTS to discriminate between a risk sample of children exposed to trauma and a community sample was explored.

A total of 49 children participated in the study. Parents and caregivers of 46 children participated in an interview about their child. The study included two samples. Inclusion criteria were Danish children age 4 to 8 years old. Parents or foster parents had to speak Danish or English at a functional level to take part in the interview. The children had to speak and understand Danish at a functional level to participate in the OCTS. For sample 1, the risk sample, the child had to have experienced at least one potentially traumatic event.

Sample 1: Participants were recruited from three child mental health clinics and the Southern Regional Children Center that assess children in cases of suspicion of physical or sexual abuse. Fifty-five children and their parents were invited. For 14 children, one or both parents with child custody declined to participate. Reasons for not participating were mainly the wish to limit further assessment of the child at a time with high level of family crisis, extensive legal case proceedings or ongoing psychological assessment. Parents of eight children agreed to participate but seven did not respond to several contact attempts to schedule interviews or withdrew due to lack of mental energy or resources, and one child did not want to be filmed for the OCTS and was therefore not included in the study. Thirty-three children and their parents and/or foster parents accepted to participate and were included in the study. Two children were excluded as coding was not possible due to one film being deleted by mistake and one child not engaging in sufficient symbolic play for reliable coding. Figure 1 illustrates flow of participant recruitment. The final sample comprised 31 children. Sample characteristics of the children are displayed in Table 1.

FIGURE 1.

Sample 2: was a community sample of children recruited from two elementary schools and a kindergarten. Parents were informed about the study on the schools’ and kindergarten’s intranet and invited to participate. Parents of 19 children consented for their child to participate. All 19 children were included. However, one child was excluded from the sample due to missing data from the OCTS retest due to holiday and missing data on the parent interview. Parents of three children did not participate in the parent interview because after several contact attempts the parents did not call back to schedule interviews. The final sample included 18 children and parents of 15 children.

Sample characteristics of participants are shown in Table 1. Of participants from the risk sample, 21 children were recruited from the Southern Regional Children Center and 10 from a mental health clinic.

Table 2 shows the number of children in each sample who had been exposed to the different types of traumas as reported in the DIPA. The children from the risk sample had experienced between 1 and 5 traumas (M = 2.77, SD = 1.2) and the children from the community sample between 0 and 3 traumas (M = 1.07, SD = .8). There was a significant difference in number of trauma between the two samples, t(41) = 4.98, p < .000. In addition to the first 11 trauma types in the DIPA, other traumas or distressing experiences was reported by caregivers for 61.3% of the children (n = 19) from the risk sample and for 26.7% of the children in the community sample (n = 4). In the risk sample other traumas included death of a caregiver or sibling, parental mental disorder, parental alcohol or substance abuse, severely problematic parental divorce, neglect, psychological violence, and placing in foster care or other acute separation from primary caregiver. Eighteen of the 19 children from the risk sample had also experienced at least one of the 11 first trauma types together with the 12^th ‘other trauma’. In the community sample, other traumas included severe chronic illness of a close family member and periods of acute separation from primary caregiver. Three out of four children from the community sample had only experienced the 12^th ‘other trauma’ while one had also experienced one of the first 11 trauma types categorized in the DIPA.

Scores on the OCTS, the DIPA, and the SDQ are depicted in Table 3. Thirteen out of 31 children (41.9%) from the risk group displayed symptoms to fulfill the diagnostic criteria for PTSD (i.e. the RDC-PA for 4-6-year olds or the DSM-IV for 7-8-year olds). Eight children (25.8%) fulfilled the diagnostic criteria (RDC-PA) for MDD, and two children (6.5%) fulfilled the criteria for RAD dishibited subtype. Of the ten children who fulfilled the diagnostic criteria for either MDD or RAD dishibited subtype, only one child with MDD did not fulfill the diagnostic criteria for PTSD. For the risk sample, scores on the SDQ Total Difficulties Scale was high. Scores on the subscale of emotional problems were slightly raised, and scores on the subscales of conduct problems, hyperactivity, and peer problems, were bordering “close to average” and “slightly raised” compared to Danish norms. For the community sample, all scores on the SDQ scales were within the normal range.

Five children from the risk sample was placed in foster care. No statistical difference was found between the children placed in foster care and the rest of the risk sample in regard to number of traumas experienced, nor on scores of the OCTS, the DIPA PTSD scales, MDD scale or the RAD scale. A Mann-Whitney test did however demonstrate that the SDQ Total Difficulties Scale was significantly higher for children placed in foster care (Mdn = 24.00) than for the rest of the risk sample (Mdn = 13.50), U = 19.50, z = -2.45, p < .05, r = -.68. Also, the SDQ subscale of hyperactivity was significantly higher for children in foster care (Mdn = 7.00) than for the rest of the risk sample (Mdn = 4.00), U = 26.50, z = -2.09, p < .05, r = -.58.

Cronbach’s Alpha coefficients for the OCTS are reported in Table 4. The total score for the OCTS demonstrated excellent internal consistency. Internal consistency of the partial score for each story ranged from acceptable to good.

Calculations of interrater reliability for the OCTS are based on 67 films. Results are shown in Table 4. Intraclass correlation coefficient was excellent for the total score of the OCTS and for all partial scores.

The calculation of test-retest reliability was based on a sample of 18 OCTS test-retests. Intraclass correlation for test-retest of the total scores of the OCTS was .67, 95% CI [.15, .87], p = .006.

Multi-method convergent validity of the OCTS as a screening measure of traumatization was examined by comparing the scores of the OCTS to continuous scores on the DIPA scales of PTSD, MDD, and RAD, and scales of the SDQ. Correlation coefficients between OCTS scales and DIPA and SDQ scales are shown in Table 5. Medium, positive significant correlations were found between the OCTS total score and the DIPA PTSD scale, MDD scale and the total RAD scale. Furthermore, there was a significant association between number of traumas and total score of the OCTS, rho = 0.48, p = .001. Medium, positive significant correlations were also found between the OCTS total score and the SDQ Total Difficulties Scale and the subscales of conduct problems, hyperactivity and peer problems.

The Mann-Whitney test demonstrated that the OCTS total score was significantly higher for the risk sample (Mdn = 4.33) than for the community sample (Mdn = 1.90), U = 96.00, z = -3.40, p < .001, r = -.54.

The aim of this study was to provide an initial examination of reliability and validity of the new story stem tool OCTS, that was developed to screen for indicators of traumatization in children aged 4-8 years based on child play behavior and narrative representations. The OCTS builds on the knowledge and methodology of existing story stem assessment tools such as the MSSB, the SSAP, and the MCAST but differs in its intended use as a screening measure for implementation in clinical practice for use with a broad trauma population of young children. The OCTS includes four known and widely applied story stems with a central conflict or distress, which we expected to work well with a broad child trauma population to elicit the child–s mental representations and strategies in different areas, and a shorter and more condensed coding scheme inspired by several coding systems from story stem studies.

The present validity study included two samples of Danish children; a risk sample of children exposed to traumatic experiences, where almost half of the children displayed symptoms consistent with a diagnosis of PTSD or MDD and a community sample, where none of the children displayed symptoms of neither PTSD nor MDD. The OCTS demonstrated good internal consistency, excellent interrater reliability, and acceptable test-retest reliability, which shows promise for the utility and implementation of the OCTS, a child-informant screening measure, into a broad range of Danish clinical practices. Furthermore, this study is, to our knowledge, the first to provide test-retest reliability of a story stem test. Other studies have included re-examination of children with story stem measures for other purposes, e.g. Green and colleagues (17) and Hodges and colleagues (15), but not to examine test-retest reliability. Our findings of test-retest reliability of the OCTS with a community sample was acceptable given the small number of children who were retested. To account for a small sample size, 95% CI–s were calculated, and estimated range of outcome were provided. Further exploration of test-rest reliability is therefore needed using a larger sample size. As for now, our findings illustrate that the OCTS can be re-administered to the same children and the results of repeated tests can be deemed reliable, despite the children being familiar with the content and arousal intensity of the story stems.

The study furthermore provided initial examination of the multi-method convergent validity of the OCTS as a screening measure of traumatization by comparing the scores of the OCTS to scores on the DIPA scales of PTSD, MDD, and RAD, and scales of the SDQ. The OCTS total score was significantly correlated with the total PTSD scale and the re-experiencing and hyperarousal subscales as well as the MDD scale and total scale of RAD. Additionally, significant correlations were found between the OCTS total score and the SDQ Total Difficulties Scale and SDQ subscales of conduct problems, hyperactivity, and peer problems. The associations between the OCTS and the DIPA PTSD scales and SDQ Total Difficulties scale were definite but moderate. However, the results are comparable to results of other studies where story stem measures have been compared to adult-reported child symptoms (e.g., 25-30 with correlation coefficients ranging between .12 and .41).

A possible explanation of the moderate association between the measures in the present study may be that the scores of the OCTS are compared to parent-reported symptoms and problems instead of another child-report measure. Indeed, low to moderate correlations between child- and parent-reports of child mental health have been found in various studies (9), and establishing convergent validity of child-informant based measures of traumatization and trauma symptomatology is generally challenged by the lack of age-appropriate self-report measures for this young age group. However, collectively the link between OCTS and DIPA PTSD, MDD, and RAD symptoms, SDQ scales, as well as the significant association between OCTS and number of trauma experiences provides initial evidence of the validity of the OCTS as a screening measure for young children. A high score on the OCTS then calls for further assessment of the child for the results to be confirmed, elaborated on or contradicted, potentially including differential diagnostics.

A note must also be made regarding the findings of non-significant associations between the OCTS and SDQ subscale of emotional problems. Based on findings from other story stem studies (27, 30) and the theoretical foundation of story stems measures in attachment theory, a significant albeit smaller correlation was expected. Possible explanations for the non-significant finding entails a small sample size in the present study, the composition of the sample with children exposed to traumas of primarily abuse or family violence or underreporting of child symptoms by primary caregivers. Futh and colleagues (26) did not find a significant association between disorganization as assessed on the MCAST and mother-reported SDQ emotional problems, and disorganized phenomena are one of the central categories of the OCTS coding system. Future studies on associations between the OCTS and measures of anxiety is warranted to elaborate on this relation.

Furthermore, the ability of the OCTS to differentiate between the trauma sample and the community sample adds to the initial evidence supporting the OCTS as a valid screening measure of play behavior and narrative representations indicative of traumatization in preschool and young school children. To strengthen this evidence further it would be relevant to replicate this finding in a larger sample. Also, the ability of the OCTS to discriminate, not just between a trauma sample and a community sample, but also between a sample of traumatized children and different clinical samples with oppositional defiant disorder, conduct disorder, or mood disorders must be examined. Furthermore, it is imperative to examine this issue in relation to the ability of the OCTS to differentiate between traumatized children and children with developmental disorders in which atypical behavior or narrative content must not be misinterpreted as indicators of the child being traumatized.

In addition to small sample size, other study limitations must be mentioned. Firstly, even though the children from the risk sample were recruited from several mental health facilities from different parts of the Southern Region of Denmark, most of the sample were Caucasian children who had primarily been exposed to abuse or family violence. A more diverse sample in terms of ethnicity and exposure to other non-interpersonal traumas would improve the generalizability of the results. However, the significance of this issue may not be alarming, as the study was a validity study of the OCTS with Danish children, and the majority (87%) of the population in Denmark are of Danish (i.e. Caucasian) ethnicity, and about 6% are immigrants from non-Western countries (56). Secondly, because the OCTS is filmed, we needed written informed consent from all parents with child custody of the children participating in the study as judged by the Regional Committee on Health Research Ethics for Southern Denmark and the Danish Ombudsman. Inevitably, this limited recruitment of children who had been maltreated by a parent with full or joint custody of the child or children for whom one parent with joint custody were unreachable. This can have caused selection bias despite considerable efforts to recruit children who for example were placed in foster care. When children who were placed in foster care were compared to the rest of the risk sample, no significant differences were found on trauma-specific scores from the OCTS and DIPA. Children in foster care did however score significantly higher on the SDQ indicating higher levels of overall problems compared to the rest of the risk group.

Furthermore, in the present study for legal and ethical reasons all administrations of the OCTS in the risk sample was done by psychologists. Administration of the OCTS in the community sample was done by one clinical psychologist and psychology students. The difference in clinical experience of administrators might have influenced our findings and future studies would benefit from having OCTS administrators with the same amount of experience of test administration when examining reliability and validity of the OCTS. Also, for this study all films were coded by psychology students to examine inter-rater reliability. Further examination of inter-rater reliability of coders with long clinical experience and student coders could strengthen a future study and use of the OCTS.

Lastly, potential age and gender differences in scores on the OCTS in were not explored due to the small sample sizes of the risk sample and the community sample. This should be addressed in future studies with larger samples. Indeed, Gloger-Tippelt and Kappler (21) illuminated on such variations related to gender and age within different samples in their study of a pooled analysis of 22 samples where 887 children were assessed with the German story stem measure GASCP. Gloger-Tippelt and Kappler (21) found that girls were 0.4 times less likely to display disorganized narratives compared to boys when controlling for risk status and age. Also, children from risk samples (i.e. children who experienced various types of maltreatment, relationship disruptions, or parental mental disorder) were 5.4 times more likely to display disorganized narratives compared to children from no-risk samples. The effect of risk was not moderated by age or gender.