Reasons for using Andes

The primary reason for using Andes is that it helps students learn more from solving physics problems.  A secondary reason is that Andes acts as a homework grading service, thus reducing the burden on the course instructors.  Before discussing the evidence for these claims, a brief description of Andes is necessary.

A brief description of Andes

Andes helps students solve problems involve algebra, vectors and trigonometry.  Such problems are ubiquitous in college and high school AP physics courses.  A typical problem is:
A mountain climber of mass m hangs motionless from an old rope.   
If the climber is pulled upwards by rope with an acceleration a, 
what is the tension in the rope?
Andes has a graphical user interface that allows students to solve such a problem by entering steps, such as the following ones:
  1. Define the mountain climber as the body whose motion will be analyzed. 
  2. Define m as the mass of the climber. 
  3. Draw a vector, Fw, representing the weight force acting on the climber. 
  4. Draw a vector, Ft, representing the tension force acting on the climber. 
  5. Draw a vector, a, representing the acceleration of the climber. 
  6. Draw a coordinate system, with the y-axis pointing straight up. 
  7. Write the equation Fw_y + Ft_y = m*a_y. 
  8. Write the equation a= a_y
  9. Write the equation Fw_y = -m*g
  10. Write the equation Ft_y = Ft
  11. Solve the system of equations for Ft, yielding the equation Ft = m*a+m*g
After the student enters a step, such as the ones above, Andes immediately indicates whether the step is correct or incorrect.  If it is incorrect, then the student can either fix it or ask Andes for a hint about what's wrong with it.  If the student ever gets stuck and doesn't know a step to enter next, the student can ask Andes for a hint.  The important point is that Andes gives feedback and hints on each step. 

In contrast, most other homework grading services and physics tutoring systems have students enter only the answer to the problem.  In the case of the problem above, the student would enter only the formula, m*a+m*g.  The system then indicates whether the answer is correct or incorrect, and it may give a hint if the answer is incorrect.  These systems give feedback and hints on the answer, whereas Andes gives feedback and hints on every step leading up to the answer

Although Andes generally allows students the freedom to answer a problem with any sequence of steps they want, it does insist on certain problem solving conventions.  These conventions are intended to discourage a plug-and-chug style of problem solving and to encourage problem solving that is grounded in the physical meaning of the mathematical objects.  For instance, Andes requires that all variables be defined before they are used in an equation, and it requires that vector variables be defined by drawing the vectors.  It also awards more points to solutions that display direct applications of fundamental principles, such as step 7 above, which displays a direct application of Newton's second law.  In short, Andes encourages students to think of variables and equations semantically, paying attention to their meaning. 

The experimental evidence on student learning

Andes produces learning gains in two ways.  First, because Andes can grade every students' solution to every homework problem, Andes makes it feasible for instructors to include homework problem solving as part of the students' course grade, which may motivate students to actually do their homework.  This can cause increases in learning. 

The second way that Andes produces learning is by encouraging students to do semantically well-formed problem solving that leads to a correct solution.  As mentioned earlier, Andes requires certain practices (e.g., dimensional numbers in equations must have proper units) and it encourages others by giving extra points.  Moreover, its step-based feedback and hints virtually guarantee that all students can solve all problems correctly.  In order to test whether these features increase learning, we conducted a rigorous, four-year experiment at the United States Naval Academy, which will be described next.  For a complete description of the experiments, see VanLehn et al., 2005.

Method: Students from the introductory physics class were assigned to the Andes condition or the Control condition.  Students in the Andes condition did their homework problems on Andes.  Students in the Control condition did homework problems with pencil and paper.  We tried to keep all other instructional variables the same.  In particular, the same textbooks, labs and exams were used.  Most importantly, the pencil and paper homework was usually handed in and graded, and the homework scores of the Control students counted just as much as the homework scores of the Andes students.  That is, we provided the same incentives to both conditions' students to complete their homework.  Assuming our control of extraneous instructional variables was successful, the Andes students should learn more than the Control students only if there is a second source of benefit due to using Andes besides increasing the amount of homework done.

Bar chart of midterm results

Midterm grades at the USNA for students who used Andes for their homework versus students who did (mostly) graded pencil & paper homework.

Results: As seen in the chart, Andes students learned more than the Control students in each of the four years.  The results were statistically significant (the whiskers in the bar chart show the standard error of the means). 

However, results can be statistically reliable without being particularly large.  A common way to measure gains in education is effect size, which is defined by: [mean(experimental) - mean(control)] / standard-deviation(control).  By this measure, the effect size of Andes is 0.61.  By educational standards, this is considered moderately high.  For instance, peer tutoring programs generally have an effect size of 0.43 (Cohen, Kulik & Kulik, 1982) whereas the very best professional human tutors produce effect sizes as high as 2.0 (Bloom, 1984).  Thus, when compared to human tutors, Andes is somewhat better than a peer tutor (e.g., a student who got a high grade in the physics course last year) but well below a professional tutor.  At the US Naval Academy, the 0.61 effect size is a bit more than half a letter grade.  For instance, having a student do homework on Andes instead of paper should raise the exam grade from a C to a B-.

For comparison, let us consider physics homework grading services and tutors that accept only answers from students.  Like Andes, they can be used to motivate students to do their homework, and this can increase learning (Dufresne et al., 2002).  However, when compared to paper and pencil homework that is handed in and graded, the learning gains are the same (Bonham, et al., 2003; Pascarella, 2003). 

The bottom line is that even if an instructor's students already do all their homework, having them do it on Andes should increase their learning by a moderately large effect size.  If the students only do some of the homework, and the instructor is willing to use Andes to motivate them to do more of it, then the learning gains should be even larger.

Other considerations

Using Andes' user interface requires learning some non-obvious details, such as how to draw a zero-length vector.  Both students and instructors should study the introductory video before trying to solve problems. 

Andes does not replace a textbook or an instructor.  It is simply a large set of problems.  It is intended to be used as part of a college, high school or distance learning course.

Andes may be used with any introductory physics textbook that teaches problem solving using algebra and trigonometry.  Andes problems currently do not require writing integral or differential equations, but neither do many calculus-based physics textbooks. 

Andes has problems for all the major chapters in introductory textbooks, except for the chapters on thermodynamics and modern physics, which are still under development.  However, Andes currently does not have very many problems per chapter.  We are constantly adding problems, and welcome suggestions for which ones to add.

We also welcome suggestions for new features to add.  For instance, Andes currently does not allow vector equations, but we plan to add that capability soon because it is critically needed for some electricity and magnetism problems.

Andes is not a perfect tutor.  Although it seldom crashes and it is almost always accurate when labeling a step correct vs. incorrect, students sometimes do not understand its hints.  This is understandable if you pick up a student's partially solved physics problem, try to decide what the student is doing and what would be a good next step, and then give a hint on that step.  Even an expert's hints fail sometimes.  Moreover, students with particularly deep misconceptions may not understand some of Andes' hints.  Thus, the instructor should still expect and encourage students to ask questions about their homework.  Some students may also need help with the user interface, at least initially. 

In short, although Andes does increase student learning, and it currently costs no money to use it, it does cost some time from both instructors and students as they learn how to use it.  Although we do not have exact figures, it appears to take about an hour to become comfortable with Andes initially, and perhaps ten minutes every month thereafter to learn its hidden "features."

On the other hand, Andes probably saves more time than it requires.  On US Naval Academy questionnaires, most students report that, in their opinion, they spend less time doing their homework on Andes than they would if they had to do the same problems on paper.  Instructors probably save time by not having to grade papers or supervise the grading of papers.  Physics departments that have replaced human graders with homework grading services have reported considerable cost savings (Dufresne et al., 2002). 

References

Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16.

Bonham, S. W., Deardorff, D. L., & Beichner, R. J. (2003). Comparison of student performance using web and paper-based homework in college-level physics. Journal of Research in Science Teaching, 40(10), 1050-1071.

Cohen, P. A., Kulik, J. A., & Kulik, C.-L. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19(2), 237-248.

Dufresne, R. J., Mestre, J. P., Hart, D. M., & Rath, K. A. (2002). The effect of web-based homework on test performance in large enrollment introductory physics courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229-251.

Pascarella, A. M. (2002). CAPA (Computer-Assisted Personalized Assignments) in a Large University Setting. Unpublished Doctoral Dissertation, University of Colorado, Boulder, CO.

VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R. H., Taylor, L., et al. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), 147-204.  Also available as a pdf file.