Predicting Financial Risk Tolerance and Risk-Taking Behaviour: A Comparison of Questionnaires and Tests

Financial planners and advisors are tasked with accurately assessing the risk tolerance of clients before making investment or financial recommendations or implementing recommendations.¹ Although numerous assessment approaches are used in practice, nearly all financial planners and advisors use some type of risk tolerance questionnaire or test (Moore 2018). Questionnaires and tests can be generally classified as based either on economic or psychometric theory.

Those who advocate the use of measures derived from economic theory assert that ‘the only rigorous theoretical analyses relating risk tolerance to optimal portfolios are based on the economic concept of risk aversion’ (Hanna & Lindamood 2004, p. 27), which is premised on the notion of expected utility. In practice, economic theory approaches estimate risk aversion (the inverse of which is risk tolerance) using an investor’s responses to income, asset, or other gamble/lottery choice questions (Barsky et al. 1997) in an effort to derive an approximation of a person’s revealed-preference (Frey et al. 2017). Regardless of the type of question asked, or the manner in which a choice scenario is framed, nearly all economic theory measures rely on choice scenarios that require a person to choose between two options—one with a guaranteed outcome and the other with a 50% chance of success and a 50% chance of failure. Barsky et al. (1997) argued that if enough gamble/lottery questions are asked, it is possible to obtain an estimate of a person’s gamma coefficient, which can be used to construct a utility function where U is the utility function and c is permanent consumption or wealth. The risk aversion coefficient (λ) derived from a revealed-preference test represents the rate at which an individual will give up a higher expected income (or other asset) in exchange for less uncertainty. Theoretically, ‘[an] expected utility maximizer will choose the 50-50 gamble of doubling lifetime income as opposed to having it fall by the fraction 1 – λ if ½U(2c) + ½U(λc) > U(c)' (Barsky et al. 1997, p. 540).²

Estimates of risk aversion (risk tolerance) developed from measures based on economic theory— referred to as tests of revealed-preference—are thought to provide insights into the way people make risky choices. Those with low risk aversion (high risk tolerance) are known to be more likely to engage in sensation seeking activities like smoking, drinking heavily, and using illicit drugs. There is some evidence to suggest that low risk aversion is also associated with being self-employed and holding risky financial assets.

The economic-based approach to measuring risk aversion (risk tolerance) is not without its critics (see Dow & Werlang 1992). Hanna, Gutter, and Fan (2001) noted that some revealed-preference tests fail to provide enough context to capture a person’s true level of risk tolerance. Barsky et al. (1997) noted that the question response options imbedded in 50-50 gamble/lottery scenarios may be too complex for many people to answer accurately. In effect, choice scenarios may lead to random guessing.

Outside of economics, psychologists and behaviorists frame the measurement of risk attitudes differently. Instead of relying on choice scenarios in which the probabilities of choice outcomes are always based on predetermined probability estimates, psychometricians generally assume that choices are based on subjective probability estimates made by individuals at the time a decision is made. This assumption has led to the development of robust questionnaires based primarily on classical test theory. Outcome scores from such questionnaires are sometimes referred to as propensity measures (Frey et al. 2017).³ Classical test theory is premised on the notions of reliability and validity. A robust questionnaire based on psychometric principles is one in which random error is minimized across questions. To do this, a test developer focuses on asking appropriate questions that help uncover a person’s attitudes and future behavior. If developed properly, questionnaire items can be combined into a preference scale or index (Faff, Hallahan & McKenzie 2009).

Economists, behaviorists, and commercial software testing firms typically adopt one testing tradition when developing measurement tools. There is little agreement among proponents of one approach or another regarding test methodologies. Each assessment procedure offers users advantages and disadvantages. For example, the revealed-preference approach has the advantage that scores can be mapped to a mean variance optimized portfolio recommendation. A disadvantage, however, is that the questions used to estimate risk tolerance may not be valid with some test takers or populations. For example, traditional risk tolerance measurement techniques require a test taker to exhibit relatively high cognitive skills (Charness, Gneezy & Imas 2013). The psychometric approach has the advantage of providing users with statistically robust measures of reliability and validity. A disadvantage is that it is difficult to map psychometric risk tolerance scores to a portfolio or other financial recommendation.

To date, there have been few attempts to compare assessment methods. This study was undertaken to address this gap in the literature by addressing the following questions: (1) do measures based on economic theory and psychometric classical test theory exhibit concurrent validity;⁴ (2) do measures based on economic theory and psychometric classical test theory exhibit convergent validity;⁵ (3) do measures based on economic theory and psychometric classical test theory exhibit predictive validity; and (4) which measurement approach (i.e., revealed-preference testing or psychometric scaling) provides the clearest insight into investor risk-taking tendencies? As will be shown in this paper, while both approaches offer unique advantages and disadvantages, scores from a psychometric risk tolerance measure appear to be more valid in describing financial risktaking behavior. The remainder of this paper reviews relevant literature associated with the research questions. This is followed by an explanation of the research methodology, with a special emphasis on describing the different measures of financial risk tolerance evaluated in the study. The paper concludes with a report of findings and an applied discussion of results.

Among test developers, two traditions dominate the way financial risk tolerance assessments are created: measures of revealed-preference and psychometric assessments. Economists tend to utilize revealed-preference assessment techniques, whereas psychologists and those who construct tests using classical test theory utilize psychometric tools (Frey et al. 2017). Both revealed-preference and psychometric approaches stand in contrast to the use of stated preferences (see Adamowicz, Louviere & Williams 1994), which is a measurement technique often employed by financial planning practitioners when attempting to measure the extent to which someone is willing to take a financial risk in which the outcomes of the risk are potentially negative and unknown.⁶

The methodology used by a particular test developer tends to be driven by the person’s academic training. In order to understand the test development process in the context of financial risk tolerance, it is first useful to review the difference between risk and uncertainty because these two constructs often shape the manner in which tests are developed. According to Knight (1921) and described by Weber and Johnson (2009), ‘risk refers to situations where the decision-maker knows with certainty the mathematical probabilities of possible outcomes of choice alternatives. Uncertainty refers to situations where the likelihood of different outcomes cannot be expressed with any mathematical precision’ (p. 132). Those who advocate the use of revealed-preferences argue that ‘uncertain situations can be reduced to risky situations’ (Weber & Johnson 2009, p. 132). Ellsberg (1961) found fault with this argument by noting that people exhibit what he called ambiguity aversion, which makes it difficult to simplify uncertainty into a risk analysis.

The revealed-preference tradition, however, offers several unique advantages, not the least of which is the notion that revealed-preferences are easy to identify. Within the domain of financial risk tolerance, a person’s revealed-preference can be assessed by asking a test taker to choose between and among choice alternatives where monetary incentives (real or hypothetical) are provided (Eckel & Grossman 2008). The notion underlying this assessment approach is that what people say they will or will not do is often different from the choices made when faced with a choice that involves relatively little variance in monetary outcomes compared to a choice alternative with more variance (Samuelson 1948). Sometimes referred to as a behavioral score, revealed-preferences are generally assessed by documenting actual engagement in behavior (e.g., how often a person engages in or chooses a particular risky behavior) or as an evaluation of hypothetical monetary gambling/lottery choices. According to Frey et al. (2017), revealed-preference measures are generally designed to ‘capture specific cognitive processes, such as the integration of gains and losses or the role of learning and experience’ (p. 1). Proponents of the revealed-preference tradition in relation to financial risk tolerance assessment often argue that this measurement technique provides the most direct link to identifying a person’s utility function when investment choices are being made (Houthakker 1950; Richter 1966; Weber & Johnson 2009).

Objections have been raised about the wide use of revealed-preference methodologies. Frey et al. (2017) stated that measurements of revealed-preference may actually be capturing situational characteristics that help a person adapt to a particular situation (i.e., states) (Buss 1989) rather than traits, which can be thought of as preferences that exhibit consistency across time (Josef et al. 2016; Reynaud & Couture 2012). Others have noted that questions used in revealed-preference measures, particularly assessment techniques that rely on a test taker to choose between gambles or lotteries, demand a cognitive ability that falls outside the norm for generalized assessments (Dave et al 2010), which can create noisy data (Charness, Gneezy & Imas 2013). Corter and Chen (2006) reported that familiarity with concepts related to expected value and probability outcomes can create a familiarity bias that distorts the way some people answer choice dilemmas. In other words, answers derived from revealed-preference measures may not capture a decision-maker’s true willingness to take a financial risk. Another criticism is that to fully capture a revealed-preference, a test developer must assume that a test taker has access to full information and uses such information. As noted by Fischhoff et al. (1978), these assumptions are typically not observed and are unlikely in practice.

The primary alternative to a revealed-preference assessment is a psychometric test, which is sometimes referred to as a propensity measure. Psychometric tests are designed to assess a test taker’s attitudes in a way that uncovers an underlying trait.⁷ Psychometric tests are widely used to assess intelligence, personality, and other psychological constructs.⁸ An advantage associated with psychometric measurements is that a well-designed test can account for a test taker’s deeply held feelings of regret, fear, greed, and happiness associated with financial decision-making. Rather than a test taker’s score being derived primarily from cognitive appraisals, psychometric tests allow for the incorporation of emotional factors. The primary argument against the use of psychometric tests is that what a person states may not be related to the person’s ultimate behavior. Fischhoff et al. (1978) argued that this criticism is likely overstated. They noted that ‘attitudes elicited in surveys often correlate highly with behavior … Furthermore, they elicit present values rather than historical preferences (p. 130). Research conducted over the past decade has confirmed Fischhoff et al.’s findings. Several researchers have recently noted the superiority of psychometric tests in the domain of financial risk-taking. For example, Dohmen et al. (2011) compared self-reported attitudinal risk-taking questions to hypothetical gambling questions. Dohmen et al. reported that the self-rating items did a better job of predicting actual financial risk-taking behavior. A similar finding was reported by LÖnnqvist et al. (2015).

In addition to direct comparisons of measurement tools, some researchers have attempted to determine how well revealed-preference and psychometric questionnaires and tests correlate with each other. In general, and particularly in the context of financial decision-making, associations between measures have been weak. Consider the following conclusion from Frey et al. (2017): ‘… measures from the propensity and behavioral measurement traditions cannot be used interchangeably to capture risk preference’ (p. 8). This implies that those who rely on scores from risk assessments (e.g., financial planners, investment advisors, and other professionals who counsel individuals on day-to-day financial matters) cannot assume consistency across measurement techniques, nor can users of assessments presuppose that all assessment tools are equal. In effect, a decision must be made as to the validity of one measurement technique over the other. The findings from this study provide additional information to help guide the choice of financial risk tolerance assessment tools for practice.

Recruitment for participation in the study was conducted during late 2017. Given the exploratory nature of the project, recruitment flyers and emails were distributed in one college community in a southeastern US state. Based on institutional IRB restrictions, potential participants were screened by age. Specifically, in order to be eligible for participation, a participant needed to be at least 21 years of age at the time of the study. The recruitment period was open for six weeks. At the end of the recruitment period, 164 participants were selected for the study. Each participant was sent an email survey link using Qualtrics. Participants were offered a $US10 gift card upon completion of the survey and related tasks. A unique code was assigned to each participant who finished the survey.

The first research question asked whether the measures based on economic theory and psychometric classical test theory exhibited concurrent validity. A preliminary answer to this question can be found in Table 1. Table 1 shows the mean and standard deviation for each of the risk tolerance measures used in this study, as well as the non-parametric correlation coefficients among the measures. One would expect that each of the measures should be correlated. This was generally the case. The Barsky and H&L tests were statistically related. Both measures were also positively correlated with the propensity scale. Curiously, however, H&L test scores were not correlated with scores from the SCF risk item.

Results from tests undertaken to answer the first (concurrent validity) and second (convergent validity) research questions can be found in Table 2. Table 2 shows the mean and standard deviation for the gambling, gambling likelihood, financial experience, and investment knowledge questions, as well as the non-parametric correlation coefficients for each item linked to the Barsky, H&L, and propensity measures. Curiously, scores from the Barsky and H&L tests were not correlated with the casino gambling items. However, as suggested by Ahmad et al. (2011), Barsky and H&L test scores were correlated with investing knowledge, and for the Barsky test, a correlation with financial decisionmaking experience was noted. Scores on the propensity scale were correlated across the items.¹²

Table 3 provides a description of the portfolio allocation categories described by participants and the non-parametric correlation coefficients each asset class had with scores from the risk tolerance measures. Similar to the results shown in Table 2, neither the Barsky nor H&L tests were correlated to holdings in the asset classes. This is a curious finding in that theoretically the correlations should have been large and significant; however, given the relatively young average age of the sample, this may not be a surprise. It is possible participants did not understand the questions and/or held limited risky assets. The correlations were stronger for the propensity scale. Those who exhibited a higher tolerance for financial risk held less cash, more equities, and more hard assets (e.g., gold). No statistical significance was found between the propensity scale and fixed-income, business ownership, and real estate assets.

The last test of convergent validity is shown in Table 4. Scores on the Barsky, H&L, and propensity measures were not found to be correlated with gender, age, personal or household income, or education. While not necessarily surprising given the homogenous nature of the participant sample, what is curious is that the direction of the correlation coefficients (although not statistically significant) varied across measures.

Data in Tables 1 through 4 represent cross-sectional data used to answer the first two research questions. The third research question asked if measures based on economic theory and psychometric classical test theory exhibit predictive validity to observed risk-taking behavior. In order to answer this question, data from the risk-taking game were evaluated.

Table 5 shows how well scores on the three risk tolerance measures predicted engagement in the risk-taking game. Of the 40 participants, 27 indicated a willingness to play, whereas 13 opted out of the game immediately. Excluding significance levels, scores from each measure were useful in predicting who would participate in the game, with higher scores predicting participation. However, when statistical significance was estimated, only scores from the propensity scale were predictive of game participation.

Table 6 shows a more nuanced test of the predictive power of the risk tolerance scores. In this case, an analysis of variance test showed that propensity scale scores were useful in predicting three categories of participants: (1) those who opted out of the game, (2) those who opted in and chose the $US10 gamble, and (3) those who opted in and chose the $US20 gamble.¹³ A post-hoc analysis using the Tukey post-hoc criterion for significance indicated that those who opted out of the game were similar to those who selected the $US10 gamble. Propensity scale scores for those who selected the $US20 gamble (M = 27.69, SD = 4.90) were significantly greater than the scores of those who opted out of the game (M = 22.85, SD = 3.72).

Another analysis of variance test (results not shown) indicated that propensity scale scores offered insights into the profile of participants who maintained their bet, compared to those who changed their bet, after learning about the true odds associated with the game. Three categories of risktaking were examined: (1) those who opted out of the game, (2) those who retained their bet after hearing the odds, and (3) those who changed their bet after hearing the odds. A post-hoc analysis using the Tukey post-hoc criterion for significance indicated that those who opted out of the game were most similar to those who changed their bet and most dissimilar to those who retained their bet after hearing the true odds. In other words, participants with the highest risk tolerance scores on the propensity scale (M = 27.48, SD = 5.39) were more likely to participate in the game, and once a decision was made, they were the most likely to retain their choice. Neither Barsky nor H&L test scores were related to predicting the persistence of risk-taking choices.

Two robustness checks were made. First, a Kruskal-Wallis test was conducted to validate the risk-taking choice ANOVA findings (Table 6). Similar to an ANOVA analysis, the Kruskal-Wallis test is an omnibus test that indicates differences among groups. Because it is a non-parametric test, the analysis is based on median scores (Pett 2016). The results of the test matched the ANOVA findings. Specifically, neither Barsky nor H&L test scores were associated with risk choices. However, propensity scale scores were significant (^χ2K-W = 7.20, p = .03). Post-hoc analyses using the Dunn procedure indicated that those who chose the highest risk choice had the highest propensity scale scores, with the highest risk group exhibiting a significantly different choice compared to those who opted out of the game. Second, an analysis of variance test was made using the variable titled, ‘How knowledgeable are you about casino games’ as a covariate in the model comparing (1) those who opted out of the game, (2) those who opted in and chose the $US10 gamble, and (3) those who opted in and chose the $US20 gamble. It was hypothesized that a participant’s choice may have been influenced by the participant’s familiarity with casino games, with those with little experience opting out of the game immediately. Even when accounting for this possibility, the core findings reported above were confirmed; propensity scale scores were predictive of risk-taking behavior (F₃₆ = 3.99, p = .02), whereas Barsky and H&L test scores were statistically not significant.

Results from the t, ANOVA, and Kruskal-Wallis tests were used to address the fourth research question, which asked what measurement approach (i.e., those based on economic utility theory or assessments based on classical test theory) provides the clearest insight into risk-taking tendencies? Before addressing this question directly, it is important to reconsider the purpose and sample of this study. Specifically, this study was designed to explore the relationships among the Barsky, H&L, and propensity measures with the intent of comparing and contrasting each method’s predictive ability. In this regard, it was apparent that each measure offered unique advantages and disadvantages. The Barsky and H&L tests, for example, were very similar in terms of predictive power of participants’ risk-taking behavior. Both measures were positively correlated with the propensity scale. This implies that these tests, based on economic theory, provide some degree of convergent validity across measures.

Curiously, the associations among the economic theory-based tests and the other measures of risk tolerance (i.e., the SCF risk question or self-assessed risk tolerance) were inconsistent, whereas scores from the scale developed using classical test theory were correlated consistently across measures. Similarly, only the psychometric scale was correlated with knowledge of casino games and the likelihood of gambling. None of the measures were found to be associated with participant demographic characteristics, which may have been due to the homogenous nature of the sample.

Of critical importance were the findings showing patterns of predictive accuracy among the risk tolerance measures. The psychometric scale was the only measure to predict who was more likely to participate in a game in which the participant was required to make a wager when the outcomes of the game were unknown and potentially negative. Similarly, scores from the psychometric scale were predictive of the level and persistence of risk-taking. Those who exhibited the highest scores on the psychometric scale were more likely to take the greatest risk in the risk-taking task, and when given the opportunity to change their wager, those with a high psychometric scale score were more likely to retain their bet. Robustness checks confirmed these findings. In matching tests, neither of the revealed-preference tests exhibited predictive ability to the level of the psychometric measure.

The results from this study provide support for what has often been reported in the literature. As noted by Frey et al. (2017), revealed-preference tests may be measuring situational characteristics rather than trait attributes (Buss 1989). The notion of having financial decision-makers choose between gambles or lotteries with probability outcomes known prior to the decision may not correspond to the cognitive demands placed on someone when a decision must be made when outcome probabilities are unknown. At a minimum, such assessments are likely to, as Charness et al. (2013) noted, cause noisy data. At their weakest, revealed-preference tests may not accurately capture a decision-maker’s true preference for risk (i.e., risk appetite). Scores from a psychometric assessment appear to work more reliably because these tools are better able to account for a decision-maker’s emotions (e.g., feelings of regret, fear, greed, and happiness). These tools appear to do a relatively good job of predicting future behavior (Dohmen et al. 2011; LÖnnqvist et al. 2015). When viewed holistically, the findings from this study provide support to the following comment made by Frey et al. (2017): ‘… measures from the propensity and behavioral measurement traditions cannot be used interchangeably to capture risk preference’ (p. 8).

As such, a preliminary answer to the last research question—which measurement approach provides the clearest insight into risk-taking tendencies?—is that a questionnaire developed using principles from psychometric theory appears to offer greater validity when attempting to describe and predict financial risk-taking behavior, at least when compared across the measures examined in this study. The final choice of which assessment technique to use in practice should be based on the known strengths and weaknesses associated with each methodology. However, if the goal is to most accurately assess a decision-maker’s willingness to engage in a risky financial behavior in which the outcome of the decision is both unknown and potentially negative, a psychometric assessment will likely provide more insights than a revealed-preference test.

Further research is needed to confirm this and the other results presented in this paper. In this regard, several limitations associated with the current study need to be acknowledged and addressed in future work. To begin with, the sample used in this study suffered from a potential recruitment bias. It would be very useful to use a larger and more representative sample when replicating this study’s methodology. Moving beyond an exploratory sample will provide additional insights into the usefulness of risk tolerance assessment tools. Additionally, future studies should attempt to align the use of incentives more closely with the demographic characteristics of participants. In this study, the maximum incentive was $US30, which was deemed to be meaningful to those who participated in the study. However, it is possible that some participants viewed the incentive as “found money,” and as such, were willing to gamble even if this action was not in alignment with their stated or revealed risk tolerance. This possibility can be tested in future studies using larger incentives and categorizing future samples by income, net worth, financial knowledge, and similar characteristics.

To summarize, the results from this study are exploratory. This means that while the propensity scale, as a proxy for other questionnaires developed using principles from psychometric theory, showed the best concurrent, convergent, and predictive validity, this does not mean that tests based on economic theory are not valuable. It is possible that questionnaires based on economic theory work better at predicting behavior that does not involve a monetary risk (e.g., health, social, and physical risk-taking). Likewise, it is possible that economic theory tests and questionnaires, because of the types of questions asked, work best when administered to those with high cognitive ability. These and other potentialities need to be examined empirically before it will be possible to truly determine which measurement approach is the most valuable within the context of financial decision-making.