Presenting Amazon's Mechanical Turk as more than just a data sample – A study of MTurker experiences

For purposes of this research, ‘high work volition’ is defined as an MTurker's ability to choose the kind of work they wish to do (conventional jobs, MTurk work or other types of gig work like driving for Uber), their ability to have high level of control over their work of choice, and their ability to change their current work of choice if they wished to do so. In the same vein, ‘low work volition’ is defined as an MTurker's inability to choose the kind of work they wish to do, their inability to have high level of control over their work, and their inability to change their current work even if they wished to do so. An example high work volition MTurker is a college student who is mostly free from family/financial constraints and, thus, able to voluntarily perform MTurk work. A second example is an entrepreneur (single or married, but without children) who runs their own business to secure primary income and performs MTurk work during their spare time. A physically disabled individual who works on the MTurk platform from home because their disability precludes performing conventional full-time jobs outside the home serves as an example of a low work volition MTurker. An individual experiencing a break in their career because of layoffs and poor job prospects due to a poor economy is also an example of a low volition MTurker, because this individual performs MTurk work part-time alongside performing other gigs in order to maintain an income while actively looking for traditional, full-time work. The demarcation of MTurkers into high and low volition workers based on their personal and professional constraints is consistent with prior research findings, which found that about 30% of independent workers earn their main income from doing gig work due to lack of better employment opportunities (Manyika et al., 2016).

Manyika et al. (2016) surveyed 8,000 workers across the U.S. and Europe and found that 20 to 30% of the working age population in these economies engaged in some form of independent work, wherein workers either derived their income solely from gig work (digital or physical) or performed gig work for additional income alongside being employed at a steady-paying job. Further, Kuhn and Galloway (2019) studied the income sources of MTurkers and categorised them as either working full-time on this online platform or using MTurk to supplement their income from a regular job. Consistent with these prior studies, the current research also categorises MTurkers as belonging to two income categories - primary income and secondary income. Specifically, a ‘primary income MTurker’ spends the most amount of time in a day on the MTurk website completing HITs, thereby earning a larger portion of their income from doing MTurk work, whereas a ‘secondary income MTurker’ spends limited amount of time in a day completing MTurk HITs while spending a majority of their time on other jobs, thereby earning a smaller portion of their income from doing MTurk work.

Working individuals who possess high work volition tend to experience higher fulfillment, autonomy, and freedom to do the job (Allan et al., 2014; Spector, 1982). High volition also allows more freedom to make occupational choices despite constraints (Duffy et al., 2012). There is a strong correlation between work volition and job satisfaction (Duffy et al., 2013). Jabagi et al. (2019) offered that gig workers whose needs for volition and autonomy are met would experience high intrinsic motivation. By extension, we argue that MTurkers with high volition would experience satisfaction in their MTurk work.

With regards to life satisfaction, past research has consistently shown that work and non-work domains of an individual's life tend to spill over into another (Lambert, 1990; Adams et al., 1996). According to the PWT, work volition can affect both work and life-related outcomes and create a sense of control over job situations (Duffy et al., 2012) to increase life satisfaction in working adults (Duffy et al., 2013). Hence, high volition MTurkers would be expected to experience high life satisfaction.

Work stress arises from issues relating to job or role demands and job situations (Spielberger et al., 2003). One of the biggest reasons for heightened stress-levels in the (digital and physical) gig economy is the insecurities involved in performing these gigs. Fifty-two percent of workers surveyed agree that the gig economy lacks security while 27% claim that it can be exploitative (Business Reporter, 2017). Such uncertainties in a person's day-to-day work can lead to stress (Jacobs and Blustein, 2008). Research shows that working in an ambiguous employment market might feel significantly more stressful than being securely employed (Mantler et al., 2005). Furthermore, the constraints that low volition MTurkers have around personal finance and family also add to psychological stress, due to their low situational control (Spector, 1982; Lefcourt, 2013). Therefore,

Hypothesis 1: High work volition MTurkers will experience a) higher job satisfaction, b) higher life satisfaction, and c) lower stress than low work volition MTurkers.

Since research on MTurkers has largely ignored their income and associated stress, we derive the current understanding of MTurker stress from study findings based on comparable samples of physical gig workers. Deliveroo, a UK-based food delivery service provider, employs gig workers to make deliveries on bicycles. These individuals, who can often be found cycling through London city traffic even on rainy days, are paid a fixed hourly rate with the remaining part of their income dependent on the number of deliveries they complete (The Guardian, 2017). The uncertainties in this particular type of gig employment include uncertain working hours, and in effect, uncertain amounts of money earned at the end of each working day. Even the distance to be cycled to make deliveries is oftentimes unknown to workers as they are restricted from viewing the delivery address until they have accepted the notification in the app to make that delivery. Making food deliveries for Uber (Uber Eats) also involves similar pay-related insecurities. Consequently, an individual who derives their primary income in such a volatile work environment, where daily wages are unpredictable and physical exertion is high, is likely to be distressed and burned out as compared to an individual working in an office space earning a steady monthly income.

With regards to MTurkers, primary income MTurkers spend much of their working hours on the MTurk platform, where HITs like surveys are highly repetitive and compensation is earned as micropayments (often in cents per HIT) (Reddy et al., 2010). Gig work is known to have higher job demands and dropout rates than traditional employment (SherpaShare, 2015), which relate to increased stress (Mantler et al., 2005). Keith et al. (2017) reported that primary income MTurkers were more likely to be unemployed from traditional jobs and have less income than MTurkers who are less reliant on this platform for work. Therefore, while both primary and secondary income MTurkers experience stress, the former should experience higher levels because of the increased time spent in this environment and the lack of regular employment. Therefore,

Hypothesis 2: Primary income MTurkers will experience higher stress than secondary income MTurkers.

Alongside the direct effects of work volition and income category on satisfaction and stress, we contend that primary and secondary income reflect distinct conditions that can alter the relationship between work volition and these outcomes. In particular, we view income category as a moderator variable capable of weakening the relationship between volition and satisfaction. The PWT posits that work volition is influenced by constraints like limited income (Super, 1953; Duffy et al., 2012). Income acts a barrier to work volition because the lack of income and a poor economy may restrict people from being able to choose the job they really desire (London, 2014). For example, in the case of a married individual with children who supports their family on a single income, a job loss and subsequent loss of income would drastically limit their standard of living, forcing a potentially hasty search for a replacement employment opportunity, which may not necessarily be the job of their choice (Leana and Feldman, 1995).

It is known that MTurkers report lower income compared to the general working population and primary income MTurkers earn less income than secondary income MTurkers (Keith et al., 2017). Research suggests that when an individual's income is exceptionally low such that it is a means for survival, work orientation becomes more extrinsic than intrinsic (Blustein, 2006). Although work can be a source of meaning and intrinsic fulfillment for some individuals, for many people working meets their survival needs (Blustein et al., 2008; Maslow, 1968). Therefore, among individuals with high volition, work can be intrinsically satisfying whereas for those with low volition, working is more likely a means for surviving. Additionally, whether a person has an intrinsic or extrinsic orientation towards work is affected by the remuneration they receive for doing it (Caldwell et al., 1983). Malka and Chatman (2003) found that for individuals with high intrinsic orientation and thereby, increased work volition, a higher income improved their job satisfaction. This suggests that the relationship between work volition and satisfaction could be affected by varying levels of income. While a higher income level offers a favorable context that enhances the volition-satisfaction relation, this relationship is weakened at extremely low-income levels. Given that primary income MTurkers earn less than secondary income MTurkers, we posit,

Hypothesis 3: Income category moderates the relationship between a) work volition and job satisfaction and b) work volition and life satisfaction, such that the relationship is weaker for primary income MTurkers than for secondary income MTurkers.

We collected the data from a sample of voluntary Amazon MTurkers. First, we created a requester account on Amazon's MTurk platform (www.mturk.com) with an email address and by following the prompts on the website to complete the account registration (Amazon Web Services, Inc., 2020). Then, we published the link for the online survey (which was set up using Qualtrics) on the platform. Only those MTurkers who met the following three pre-screening parameters were given access to the link – i) be at least 18 years of age, ii) must reside in the U.S. (only workers from the U.S. were considered for this study), and iii) have at least 95% approval rating on the MTurk website. Qualifying MTurkers were able to view the survey link and could choose whether or not they wished to complete the survey. The compensation offered for completing this survey was set at US $0.85, which is consistent with the payment standard for MTurk survey takers (Buhrmester et al., 2011). A total of 396 responses from individual MTurkers were collected at two points in time. Of the 396, 220 responses were collected from MTurkers working evenings (6 pm to 10 pm) and 176 responses were collected from MTurkers working daytime (9 am to 2 pm). These time limits were applied in the MTurk platform to ensure that data from both full-time MTurkers (i.e., primary income) and part-time MTurkers (i.e., secondary income) were collected. Twenty responses were eliminated due to noncompletion, producing a final sample size of 373 MTurkers. The sample was 59.3% males and 40.7% females, with ages ranging from 21 to 69 years and an average of 46.2 years. About 2% of the MTurkers were over retirement age (66 years). Participants’ ethnicities were 74% Caucasian, 11% African American, 6% Asian, 4% Hispanic/Latino, and 5% other. Education-level of the sample was as follows: High School (23.1%), Associate's Degree (14.3%), Bachelor's Degree (45%), Master's Degree (15.2%), and Doctorate Degree (2.1%). Approximately, 62% worked full-time and 15.4% worked part-time outside of MTurk, while the remaining 22.3% only performed MTurk-related work. For a majority of MTurkers, annual personal salary (75.5%) and annual household salary (60.3%) was less than US $60,000. Based on type of HITs, 91% were survey takers, 6% did data entry, and 2% performed transcriptions.

Table 1 provides the descriptive statistics and correlations between the study variables.

Data were analysed using univariate analysis of variance and by performing mean contrasts. A univariate analysis of variance (ANOVA) was conducted to test the main effects in Hypotheses 1a, 1b, 1c, and 2. The moderation hypotheses, namely Hypotheses 3a and 3b, were tested using ANOVA by creating an interaction term for each model. The independent and moderator variables were mean centered to reduce multicollinearity (Aiken et al., 1991). Additionally, variance-inflation factor (VIF) for each interaction combination were calculated as 1.00 which was well below the cut-off of 10 indicating that multicollinearity would not hinder the interpretation of the results (Neter et al., 1996).

This study reveals that the effect of work volition was similar for both job satisfaction and life satisfaction. As hypothesised, high volition MTurkers had higher job satisfaction and life satisfaction. This finding is consistent with the PWT, which suggests that high work volition relates to higher levels of work fulfillment and overall well-being (Blustein, 2006; Duffy, Bott et al., 2013).

Next, with respect to the interaction between income category and work volition, life satisfaction, but not job satisfaction was subject to moderating effects. In particular, the relationship between work volition and life satisfaction was weaker (i.e., negative) for primary income MTurkers than for secondary income MTurkers. A possible explanation is that work volition is generally low among primary income MTurkers because this group might consist largely of individuals who are compelled to perform this work full-time due to their personal constraints. (For the present sample, 75% of primary income MTurkers had low work volition). We argue that because there is little difference in the low levels of volition for the majority of primary income MTurkers, the correlation between volition and life satisfaction for this group would be attenuated, thereby leading to the weaker relationship. An interesting finding regarding life satisfaction was that low volition MTurkers in the secondary income group had the lowest life satisfaction. Lachman and Weaver (1998) showed that a general sense of control is a predictor of life satisfaction among individuals at lower income levels (below US $50,000 per year). The average overall annual salary of these MTurkers was US $40,020, indicating that MTurkers not only experience a low sense of control in life but they are also navigating poor economic situations personally, all of which are contributing to a decrease in their life satisfaction.

For job satisfaction, although the interaction with income category did not yield statistical significance, primary income MTurkers did have lower job satisfaction than secondary income MTurkers. This dissatisfaction among primary income workers could be attributed to the amount of time that they spend in the uncertain work environment characterised by monotonous tasks and negligible pay. Prior studies of MTurkers have found that individuals who mainly did MTurk work for their income spent more than 40 hours a week on the platform (Ipeirotis, 2010). Another study of MTurker satisfaction revealed that MTurk requesters also play a crucial role in predicting worker satisfaction. Because requesters review MTurkers’ completed tasks and make a determination on whether to approve or deny pay to a particular worker, poor requester behaviours like offering unfair pay for completed tasks have been shown to negatively impact job satisfaction (Brawley and Pury, 2016). This impact may be stronger for low volition, primary income MTurkers because they rely heavily on their approval ratings of completed HITs.

As predicted, primary income MTurkers experienced more stress than secondary income MTurkers. These results suggest that individuals who spend most of their time in the MTurk environment to derive their primary income experience more stress than individuals with conventional jobs who do MTurk work on the side. Once again, the difference could be attributed to the nature of this work - highly repetitive and less predictability in terms of what HITs might be available each day resulting in erratic pay. Our sample had 91% survey takers; completing surveys is generally a monotonous task, wherein there is high likelihood of being bored. This leads to primary income MTurkers becoming more stressed and fatigued from laborious work. We also found that low volition MTurkers experienced higher stress than high volition MTurkers. This finding is supported by research on work stress, which suggests that people tend to be more stressed in situations where they feel they have less control (Houston, 1972). Low volition workers clearly have less control and freedom in their careers than do high volition workers.

Findings from this study of MTurkers is further validation of the PWT, which substantiates the psychological role that work plays in an individual's life while acknowledging the relevant social and economic forces that facilitate and/or constrain aspects of working (Blustein, 2013). While previous empirical research has integrated the PWT in work volition, these studies viewed ‘work’ from a traditional perspective using conventional samples such as full-time workers employed by firms (Zacher, 2014), health organisations (Duffy et al., 2013), and universities (Horton and Tucker, 2014). In contrast, this research examined decent work in a new and evolving work environment - Mechanical Turk, a form of digital gig work, and used a sample of workers from this unconventional workspace. As described in our research, digital gig work like MTurk, is different from conventional full-time work and the experiences of workers in this new employment platform are unique. Moreover, the PWT proposes that decent work can lead to fulfillment and well-being. In this regard, findings from our study extends the application of the PWT by identifying the unique implications of MTurk work for individuals. This research also complements recent theoretical work like that of Jabagi et al. (2019), which demonstrated practical applications of motivational theories for gig workers.

Understanding the degree to which MTurkers experience satisfaction and stress can inform practice and policy by offering gig employers and policymakers an opportunity to recognise the idiosyncrasies of these workers and implement improvements to their existing working conditions. The (digital and physical) gig economy has implications for various HR functions like compensation, training, and performance management (Aguinis and Lawal, 2013). There has been an increasing push by gig workers to compel ridesharing companies like Uber and Lyft to recognise their drivers as employees (as opposed to independent contractors), so that these workers could be eligible for higher-than-minimum wages and employment benefits (DePillis, 2018; Tolentino, 2017). Similar efforts would extend to Amazon, which currently does not view MTurkers as employees. At present, the entire responsibility of learning to do MTurk work falls on the MTurker. An opportunity exists for Amazon to assess whether offering trainings and pay-for-performance would help to improve MTurkers’ work performance and potentially resolve requesters’ concerns surrounding data quality and validity. For policy implications relating to external organizational factors, labour policymakers of countries where the gig economy is prevalent must hold companies and clients accountable for providing reasonable wages (HR.org, 2020) and analyze employment records and tax returns to identify evolving trends in work arrangements and human capital that contribute to their country's global competitiveness (Aguinis and Lawal, 2013).

Importantly, the influence of the PWT variables on job satisfaction and life satisfaction were not uniform. For instance, the link between volition and life satisfaction was moderated by income category (primary versus secondary), but this was not the case for job satisfaction. This inconsistency warrants further investigation. It is possible that the low levels of income that MTurkers earn per HIT in this workspace prevent them from viewing these wages as important enough to positively impact their satisfaction with (digital or physical) gig work. We believe the concept of ‘precarious work’ could be useful for addressing this inconsistency. Blustein et al. (2016) discuss the tremendous rise and increasing prevalence of precarious work in the past few years due to rapid technological advances, globalisation, and a changing labour force. MTurk is an illustration of said precarious work and all features attributed to precarious work are prevalent in this environment. With scholars calling for more studies anchored in the PWT on modern day precarious work (Rubery et al., 2018), we recognise that future avenues for studying MTurker experience remain. Other factors like the changing demographics of MTurkers must also be given consideration in order to learn more about these evolving forms of working.