The question of how to operationally define motivation is not trivial. Le, Casillas, Robbins and Langley (2005) developed the Student Readiness Inventory or SRI to provide metrics on the personality dimensions most relevant to post-secondary academic success. They found that conscientiousness, goal-focus, and academic self-confidence are the strongest components of academic motivation (Le, Casillas, Robbins, & Langley, 2005). Using another measure, the Big Five Inventory or BFI, Komarraju and Karau (2005) found that the BFI’s conscientiousness scale strongly correlated with academic motivation. In independent confirmation of this finding, Peterson, Casillas and Robbins (2006) found strong correlation between the Conscientious scale of the Big Five Inventory (BFI) and motivation as measured by the Student Readiness Inventory (SRI).

It a study of almost 4,000 students from 28 different institutions offering both two- and four-year programs, and including subjects from a wide variety of disciplines, Allen and Robbins (2010) tried to correlate “interest-major” congruence, motivation as measured by the SRI, and first-year performance as predictors of students’ completion of their studies within each program’s nominal duration, which they use as their operational definition of achievement. They found that only 12% of students in a two-year program completed their studies in the nominal period. Among students taking a two-year program, only motivation was significant and even then, only indirectly. While motivation did not directly correlate to achievement, it did correlate with first-year performance and first-year performance did correlate with achievement (Allen & Robbins, 2010).

In another study of why students underachieve, Balduf (2009) looked at individual study skills. She reported that nearly all students—even those with strong high school grades—reported that “high school did not require them to work hard enough and felt that they earned high grades without expending much effort” (2009, p. 284). Her research found that many students enter post-secondary programs without effective study skills or adequate time management abilities and that students were externally motivated (eg. looking for grades or parental rewards) rather than being internally motivated (eg. seeking understanding, understanding, or integration of various knowledge domains). She reports that students also found that the amount of freedom in post-secondary institutions was an additional challenge to their time management skills.

A more robust study of time management and student success used a combination of GPA results and a set of questionnaires administered to both the participants and a friend of each participant. This study found that time management skills, intelligence, time spent studying, waking up earlier, owning a computer, spending less time in passive leisure, and eating a healthy diet were all significant predictors of positive GPA results (George, Dixon, Stansal, Gelb, & Pheri, 2008).

Studies have also been conducted to determine if there is any relationship between students’ learning style and their success in learning particular programming languages (Thomas, Ratcliffe, Woodburry, & Jarman, 2002). Similar research on how students can best learn computer programming languages have focused on what are the best teaching styles to complement the various learning styles of students (Bayman & Mayer, 1988). Learning styles have been measured using the Myers-Briggs Type Indicator (MBTI), the Index of Learning Styles (ILS) Questionnaire (Soloman & Felder, 2009), and the Gregorc Style Delineator (GSD) (Gregorc, 1984).

Using the ILS, Thomas et al. (2002) found that reflective learners were better programmers than active learners, and verbal learners had higher scores than visual learners (2002). Davidson and Saveyne (1992), using the GSD found that abstract sequential mindsets correlate with a better ability to learn a programming language than other mindsets while those with abstract random mindsets have a negative correlation. There was no correlation among those students with a concrete learning style.

Another avenue of inquiry has recently started to look at the specific problems encountered by students in IT/CS programs. Students in these courses often encounter a level of abstraction in their subject matter unfamiliar to students in many other disciplines. Forming algorithms to solve computational problems often involves a number of different types of logical processing within the same problem space. Definitions in this knowledge domain often involve negation with conditional conjunction or disjunction. An example could be the definition of a “ball” in North American baseball: “A pitch at which the batter does not swing and which does not pass through the strike zone” (Goodwin & Johnson-Laird, 2010). For those familiar with the game, this definition seems self-evident, but for a person learning new skills (such as programming), the logic to conceptualize such an abstract concept can be daunting. Similarly, the definition that “x owns y” means that (in part), “it is permissible for x to use y, and not permissible for others to prevent x from using y” (Goodwin & Johnson-Laird, 2010).

Goodwin and Johnson-Laird hypothesize that people form mental models of these Boolean expressions that are not often correct in their logical representations. They found that people generally have a difficult time considering all the possibilities of such Boolean paradigms.

Stanovich (2002) argues that cognitive capacity (such as the ability to do well on an IQ test) is distinct from a person’s thinking disposition—a term that he uses to refer to the intersection of cognitive ability or intelligence, and personality traits such as will, desire and behaviour. Stanovich (2002) then suggests that “thinking dispositions can predict performance on reasoning and rational thinking tasks even after individual differences in measures of general cognitive ability have been partialled [sic] out” (p. 131). One type of thinking disposition, which Toplak and Stanovich (2002) call disjunctive reasoning or “as the tendency to consider all possible states of the world when deciding among options or when choosing a problem solution in a reasoning task” (p. 197), may be significant in identifying those people who can process complex logical problems. In a study involving nine disjunctive logic problems, a number of the problems tested generalized disjunctive thinking dispositions and these test items show a stronger association with thinking styles (or dispositions) than with cognitive ability and capacity, thus strengthening the argument in favour of a differentiation between the two constructs (Toplak & Stanovich, 2002).

A sub-type of complex reasoning problems known as “Analytical Reasoning” problems, have until recently been part of the Graduate Record Examination (GRE), an entrance test widely used for candidates to graduate schools throughout Europe and North America. They remain part of the Law School Admission Test (LSAT) and discussions are on-going about resurrecting their use in the GRE (Newstead, Bradon, Handley, Dennis, & Evans, 2006). In this class of problem, a narrative of the scenario contains the initial context and a list of the rules allowed in the problem space. The subject is then presented with a list of questions or problems. Successfully arriving at the solution involves a deep semantic understanding of the scenario and the ability to parse out the embedded rules. Only then can the subject start to solve the problem. Problem solving involves identifying which rule to apply first in order to reduce the number of possible options in the solution set as quickly as possible.

49 first year CIT students were enrolled at the start of the fall 2010 semester at Lethbridge College in Lethbridge, Alberta, Canada. Of these students, 43 agreed to participate in the study, an 87% response rate. By the time the first major or midterm exam was conducted in late October, 35 students remained enrolled in the course and wrote the exam. Only 27 students remained in the course to write the second midterm and final exam.

Of the students participating in this study, 2 were female; only 1 of which completed the course. The average age of the participants was 24 years, with the oldest being 50 years old and the youngest being 18.

The pre-test instrument included:

• The Big Five Inventory[1];
• The Index of Learning Styles (ILS)[2];
• 11 questions asking the student to rate their time management ability and practices;
• 11 questions asking the student to estimate the amount of time spent per week engaged in a number of recreational activities;
• 12 questions asking the student to solve logical problems that will involve disjunctive logic and analytical reasoning.

The BFI was used because it is a shorter instrument comprising only 44 questions, and is freely available for non-commercial use. Its use by Allen and Robbins has been shown to correlate with measures of student motivation as measured by the SRI.

The ILS was selected because it is developed from a composite of learning models, namely the Myers-Briggs Type Indicator (MBTI), Kolb’s Learning Style Model and the Herrmann Brain Dominance Instrument (Thomas, Ratcliffe, Woodburry, & Jarman, 2002). It is free of charge and is administered as a web application with immediate scoring.

A review of available literature found no suitable questionnaire or other instrument to measure the time management practices or recreational activities of post-secondary students. For this reason, a series of 22 questions were developed to determine if time management and the allocation of recreational time by students can provide any prediction of their academic success. These questions and the range of responses is listed in Appendix A.

Problem solving ability was measured using a series of questions proposed by Toplak and Stanovich (2002) and augmented by some less demanding questions proposed by Frederick (2005). One analytical reasoning question (Newstead, Bradon, Handley, Dennis, & Evans, 2006) was also included in an attempt to give this factor a wide spectrum for assessment. The text of these questions is found in Appendix B.

Following an introduction to the study and the solicitation of informed consent from participants, the pre-study instrument was provided to students in early September. Students completed the survey during a regular class period.

The evaluations were graded as follows:

• The BFI and ILS scores were calculated using the algorithms provided by their developers and authors.
• A time management scale was calculated from the first 11 time management questions by scoring each Likert item from 1 to 5. Questions numbered 2, 3, 7, 9 and 10 were reverse scored. The total of these values was then divided by the number of questions answered to provide an index against which to compare student responses.
• Time estimates for each recreational activity were created from questions 12-22 of the time management questions using the median value for each time interval’s range. Since the 6^th choice has no maximum value, it was arbitrarily assigned a value of 20 hours for this calculation. For example, if a student chose the 4^th option, “More than 5 hours and up to 10 hours”, that activity was given a value of 7.5 hours for that student. The total of all these values for each student was calculated as the total recreation time for the student.
• Each logical problem question was evaluated on a scale of 0 to 1.
• For scoring purposes, questions 1 and 8 were counted as a single question and justified by the principle that if a participant is going to choose that 1/3 of a population will live, then he or she must also be prepared to accept that 2/3 will die; or conversely if the student wants to play the odds that there is a one in three chance that everyone will be saved, then he or she must also accept that there is a two in three chance that no one will be saved.
• Problem 6 was scored so that each card was worth one quarter of the entire question. A full value of 1 point was earned only if the participant turned over every required card and left all unnecessary cards unturned.
• Problem 12 was similarly scored so that each choice was worth 0.20 of a point. Only by not choosing incorrect choices and choosing the correct choice could a participant earn a full point.
• All other problem solving questions were scored either 0 (if the participant made an incorrect choice) or 1.

As part of their regular course work, students completed three major exams during the semester; these exams and the final exam grade were then compared to the responses from the pre-study survey. Students who dropped the class prior to completing the first exam were excluded from the study. A cumulative performance score composed of the average of all exams completed by a student was also calculated.

Turning our attention to testing the proposed hypotheses, a Pearson Product Coefficient was calculated between each of the BFI indicators and academic performance (Table 4). Conscientiousness was the only scale to show a significance (p<0.05)—albeit a negative one—when correlated with the first midterm exam. However, this correlation was less significant when Conscientiousness was compared with academic performance over the entire semester (p@0.06).

In analyzing whether learning styles as measured by the ILS can predict academic performance, there appeared to be no significance at all (Table 5).

Table 4: Pearson product coefficients between BFI factors and performance on the first midterm and cumulative academic performance

Table 5: Pearson product coefficients comparing ILS styles with academic performance

Students’ self-assessment of time management practices and ability was also compared to academic performance. This analysis involved two phases. In the first phase, the participants’ assessment of their own time-management skills and abilities as measured from the first 11 questions were correlated to their academic performance. However, this relationship was not found to be significant (-.146, p=0.403). In the second phase, participants’ estimates of the time they spent on a variety of recreational activities were also collected. A frequency chart showing the number of students who reported engaging in the different activities is shown in Figure 1. This chart compares the median values of time spent per week for each activity with the number of students reporting that value.

Figure 1: Frequency of students participating in various recreational activities by hours spent per week in each activity.

The time spent by each student on each recreational activity was then correlated with academic performance (Table 6). A positive correlation was found between time spent playing computer games and academic performance; however, no other recreational activities appeared to correlate with exam results.

Table 6: Spearman rho correlations between participants' estimates of time spent on recreational activities and academic performance

Table 7: Spearman Rho Correlation Matrix between Problem Solving Questions and Academic Performance

*  Correlation is significant at the 0.05 level (1-tailed).

**  Correlation is significant at the 0.01 level (1-tailed).

Motivation is operationalized using the conscientiousness scale of the BFI. This scale “describes socially prescribed impulse control that facilitates task- and goal-directed behavior, such as thinking before acting, delaying gratification, following norms and rules, and planning, organizing and prioritizing tasks” (John, Naumann, & Soto, 2008, p. 120).

Intuitively, this seems like a critical quality in successful students in general. It is surprising then to discover that this scale does not correlate with academic performance in learning to write computer software and may even negatively correlate with success. This finding contradicts that of Allen and Robbins (2010) who found that motivation did correlate strongly with first-year performance. However, Allen and Robbins defined success as completing a program in the nominal period of the certificate or diploma of study. Using this definition, a student completing a two year diploma in three years would be deemed “unsuccessful.” A number of conjectures are possible to account for this finding: It may be that students who are less conscientious or motivated may still be successful at their course work but take fewer courses per semester of study. Thus their grades per course could still be high but they are delaying the completion of their program. Alternatively, motivation as a personality trait may not be a significant factor in successfully learning to develop computer software.

Comparing students’ preferred learning styles with their academic performance showed no significant correlations. This contradicts previous findings by Thomas et al. who found that reflective and verbal learners became better programmers (2002). That study used a sample of 107 computer science students at the University of Wales. However, this contradiction might be explainable. Academically oriented institutions such as universities and degree granting colleges often rely heavily on lecture based content delivery—the very sort of model that would appeal to students from an abstract, sequential, reflective or verbal oriented learner. Community colleges and institutes of applied learning such as Lethbridge College often utilize a wider variety of pedagogical methodologies and didactic techniques. These practices often use multimedia, group activities, project based learning and problem solving activities—in essence, the very strategies that might resonate with active, visual, random, or global learners—in addition to teaching approaches that may be more favourable to abstract, sequential, reflective or verbal learners.

Balduf reports that almost all students entering a post-secondary institution for the first time were not adequately prepared in terms of study skills, time management abilities and motivation (2009). This finding gave rise to the decision to retest this hypothesis in the current study. In comparing the sample used by Balduf and the current study a number of significant differences emerge: Balduf used as her population, 83 students who were on academic probation from their college, a small degree-granting institution. Only seven students, or 8.6% of this population, agreed to participate in her study. It is possible that, when she chose to interview these subjects, they were more interested in rationalizing their poor academic performance by attributing it to poor time management or lack of preparation by their high school environment. A better referential study is the one conducted by George et al. In this study of a random sample of undergraduate students, a number of personal behaviours and time did correlate with higher GPA.

The fact that the current sample showed no correlation between a self-assessment on time management ability and academic performance may have a number of attributions. It may be that students are not able to effectively assess whether they have good time management skills and strategies. The students in the current study were in the first two weeks of the first semester of their post-secondary program and may not have been in a position yet to assess their time management skills and abilities vis a vis the demands of a post-secondary environment.

If we turn our attention to how students spend their recreational time, we see that, except for volunteerism, the amount of time students spent in any recreational activity was negatively correlated with their sense of time management ability as measured by the 11 question survey. Volunteerism may be an aberration since 26 of the participants reported spending no time volunteering and 7 reported spending between 0 and 2 hours per week. The remaining 2 participants reported spending between 2 and 5 hours and between 10 and 20 hours per week respectively.

While time management is an important factor in academic success, it is a poor predictor of that success. The exception may be computer game playing. Time spent playing computer games correlated very strongly with performance on the first exam (p<0.01) and still showed a strong relationship with academic performance over the entire semester (p@0.062). 6 of the 35 participants reported spending more than 20 hours a week playing computer games. This finding is surprising; anecdotally, many professors can list at least one student who became so involved in computer games that their academic grades suffered. Perhaps those students most passionate about computer games may also be passionate about computer programming—the common thread here being passion around computers and technology. At the same time, it should be noted that time spent on social networking sites was not positively correlated with academic performance. Further work will be required to verify or understand these findings.

The most significant direction to pursue in predicting students’ ability to learn computer programming skills seems to come from the area of assessing problem solving ability. However, not every logical skill or ability is an equally reliable predictor. The pre-study survey included 12 questions related to logical problem solving and critical thinking ability. These tests can be grouped into a number of categories: