CCi Mainstreaming and RHD Jams Outcomes Report

The challenge context and question for the CCi Mainstreaming Challenge were developed in close consultation with Centre management and Terry Cutler, Chair of the CCi Advisory Board, and were designed to reflect the fact that there exists considerable diversity amongst Centre participants in what constitutes „impact‟. The primary challenge question posed by the jam was thus kept deliberately broad:

The challenge question was accompanied by a short context-setting text to provide background and guidance on responding to the challenge question:

The challenge context and question were consciously positioned as widely and openly as possible in order to stimulate a broad range of responses and minimise the risk of disenfranchising particular cohorts within the participant group.

The RHD Challenge was structured around three key themes reflecting current areas of concern in the experience of Research Higher Degree students at the Centre. The wording of the themes was designed to be as meaningful as possible to the RHD participants in the event:

Participants were also provided with a „free space‟ area where any other ideas not captured in the above listed themes could be posted.

Of the 112 people invited to participate in the CCi Mainstreaming Challenge, 31.25% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard „ideation‟ events which rarely achieve participation rates of more than 20%.

Of those who did participate in the CCi Mainstreaming Challenge, just over ten percent contributed by vote only (i.e. did not post any ideas or comments). This highlights that the jam process provides two distinct ways for individuals to participate – through written contributions and through expressing opinions in the forms of votes. The CCi Mainstreaming community generated a vote-to-idea ratio that approximately matches the average of other jam events, voting at a rate of about 5 to 1

It can be noted that other jam events have generated a much higher vote-to-idea ratio, with a recent challenge achieving over 10 votes per idea on average. Further research is needed to conclusively identify the group characteristics that correlate with intensive voting patterns, however anecdotally it can be observed that the group involved is highly internally competitive and operates within a strongly hierarchical organisation compared to the collegiate structure of the CCi groups.

Figures 1 and 2 below show that participation over the duration of the event was strongly influenced by the communications programme designed to stimulate an increase in participation rates. Votes and comments in particular spiked significantly after the mid-point email reminder sent on 15-06-09.

Figure 1

Figure 2

Figure 1 also shows that posting of ideas started high at the commencement of the event but tailed off over time – this is a very typical pattern for idea contribution held over a multi-day period. However it is worth noting that in this case the communications programme was sufficiently effective to cause a small spike in new ideas posted on 15-06-09. It is typically very difficult to achieve this spike in idea contributions in the second week of a jam.

Of the 75 people invited to participate in the CCi RHD Challenge, 32% made contributions in the form of ideas, comments and or votes. This is an above average participation rate compared to standard „ideation‟ events which rarely achieve participation rates of more than 20%.

Of those who did participate in the Challenge, more than 45% contributed by vote or comment only, with 8% contributing by comment only and 37.5% contributing by vote only. This highlights that voting was an extremely important mechanism for expressing opinions for this particular community. However the CCi RHD community generated a vote-to-idea ratio that is slightly lower than the average of other jam events, voting at a rate of about 4.5 votes per idea on average.

Figures 3 and 4 below show that participation over the duration of this event was also strongly influenced by email reminders. Votes and comments spiked significantly after the mid-point email sent on 16-06-09, albeit with a delayed response of about 24 hours. It is also worth noting that the commenting rate in particular stayed high in the second week of this event, which may be attributed to the effectiveness of the reminder email or to the characteristic behaviours of the group involved

For example it may be tempting to infer that students (even at the RHD level) have a tendency to procrastinate, reflected in this case in the much higher voting and commenting rates observed in the second week of the event.

Figure 3

Figure 4

Figure 5 below shows the distribution of ideas, comments and votes contributed in the CCi RHD Challenge grouped by theme. (Note that this breakdown is not possible for the CCi Mainstreaming Challenge as it involved a single challenge question.)

Figure 5

The „future work opportunities and challenges‟ and „free space‟ themes clearly captured the most activity in terms of ideas, comments and votes. It is interesting to note that no ideas – and therefore no comments or votes – were posted against the theme related to the Centre‟s interdisciplinary research agenda. This indicates that this particular theme may have been mismatched with the actual concerns of the RHD community.

As shown in Figures 6, 7 and 8 below, the dominance of the „free space‟ or „miscellaneous‟ theme in attracting ideas, comments and votes indicates that the challenge themes were not particularly well matched to the types of concerns and opportunities perceived by the group. This may be due to the fact that this group had previously been engaged by the Centre via surveys, and some of the categorisation of „issues‟ identified through that mechanism may have inadvertently bled into the design of the jam process. This highlights that survey/questionnaire activities are fundamentally different to ideation initiatives, in that the former counts and categorises responses to known questions whereas the latter seeks potential solutions to problems or challenges broadly defined.

Figure 6

Figure 7

Figure 8

A recommendation arising from this outcome is that a single challenge (i.e. non-themed) jam design should be used unless there is strong and accurate knowledge of the topic categories that are relevant for the particular combination of challenge and participant group.

Contributions to both Challenges conformed to a “long tail” distribution (also known as a power law distribution). This means that a small number of people were responsible for the majority of the activity in each event. This pattern of contribution is seen in many online and interactive phenomena such as Wikipedia, social networking and internal firm collaboration processes.

In the CCi Mainstreaming Challenge, the top 3 individual contributors (in terms of the number of comments and ideas submitted) contributed just under 40% of the total ideas and comments generated by the jam. This is illustrated in Figure 9 below.

Figure 9

It may be the case that those contributors who are relatively prolific in an online context are much less likely to contribute in traditional face-to-face settings, so that their „voices‟ can be more clearly „heard‟ through a web-based process such as the jam. It is only possible to demonstrate this conclusively, however, by comparison to data on face-to-face contribution rates, which are not currently known for these groups.

It is possible however to map the source of the most popular ideas onto the long tail of contributions. For the CCi Mainstreaming Challenge this mapping is shown in Figure 10, with corresponding mappings shown for the RHD Challenge in Figure 11.

Figure 10

Figure 11

The concentration of popular ideas in the long tail of the distribution shows that the jam provides an effective mechanism for capturing valuable ideas from individuals who are relatively less prolific in their contributions compared to others. For example, the second most popular idea in the CCi Mainstreaming Challenge was submitted by one of the least prolific contributors in the event.

Votes cast during a jam can be negative (a thumbs-down or „Demote‟) or positive (a thumbs-up or „Promote‟ for the idea), resulting in an overall net score per idea. The top ideas by popularity (i.e. with the highest net score) for each Challenge are summarised in Tables 1 and 2 below.

Prizes were awarded to the contributor of the most popular idea in each event – a bottle of fine wine in the case of the CCi Mainstreaming Challenge; and a $1,000 travel bursary in the case of the RHD Challenge.

A key principle in making the most of a jam process is to recognise that the most popular ideas are not necessarily the most valuable to the organisation.

We can also rank ideas by the volume of overall activity they attracted in terms of both votes and comments. In Table 3 below, ideas are ranked by the number of votes they received, whether negative, positive or indifferent (i.e. „No Opinion‟), as shown in the column „All Votes‟, plus the number of comments generated, as shown in the column „Comments‟.

The jam voting system allows participants to express three types of opinion about an idea: positive („Promote‟), negative („Demote‟) or indifferent („No Opinion‟). The total number of votes received by an idea may not correlate with its popularity if the idea receives a relatively large number of negative or indifferent votes. This is demonstrated in the discrepancy between the values in the columns „All Votes‟ and „Net Score‟ in Table 3 above.

Table 4 below shows the three ideas that most polarised opinion during the CCi Mainstreaming Challenge.

The above breakdown of activity by idea emphasises that it is important to take a holistic view of each idea – and the qualitative and quantitative responses that each has generated – rather than focusing solely on relative popularity as given by the net voting score.

As discussed, the Mainstreaming Challenge used a single question design due to the wide divergence of views known to be held on the jam topic amongst the invited participants. While the three specific themes originally chosen for the CCi RHD Challenge were successful in eliciting a high response rate and strong voting levels, the relatively dominant use of the „free space‟ or „miscellaneous‟ theme indicates that the three original themes were a less than perfect match for the ideas and discussion subsequently generated – although all clearly fell under the umbrella theme of „improving the RHD experience at CCi‟.

In both cases there was a strong rationale for conducting a „meta-analysis‟ to identify emergent themes that collectively describe all of the ideas put forward during the events, and assist with deciding on specific actions going forward. These emergent themes are based on „bottom-up‟ analysis of the jam content itself rather than on the „top-down‟ themes that were originally conceived for structuring the discussion (in the case of the RHD Challenge) or the wide scope created by the deliberately broad challenge question set for the Mainstreaming jam.

To demonstrate this approach, six „emergent themes‟ from the Mainstreaming Challenge are identified:

All of the 22 ideas generated by the CCi Mainstreaming Challenge can be categorised under these six emergent themes, as tabled below:

Emergent themes such as these allow jam sponsors to organise action planning activities around the topics that have actually arisen from the jam process rather than those that may have been erroneously perceived as being relevant by management prior to the event.

This exhaustive approach to categorising ideas – i.e. providing a „home‟ for every proposal that is put forward - also highlights that it is crucial that all forms of response to ideas be taken into account when formulating action plans, and in particular that popularity votes as articulated through the net score per idea are balanced by the qualitative information in the discussion threads.

Finally, it is strongly recommended that the participant group be given clear information as to how future actions will be decided and implemented as a result of the jam, and that reporting back to the group on these initiatives is made a priority. Failure to do so will jeopardise the effectiveness of subsequent jam - or broader ideation - events.