Only with precise specifications is it possible to teach the fundamental engineering practice of implementation to specification. We rely on simple implementation exercises to teach and evaluate the students' ability to implement to specification. To facilitate the grading and to emphasize the role of automated testing, we use the PGMGEN testing tool [7] to automatically grade the execution behavior of the implementations.
In some cases we provide a specification and the students must develop the implementation, but often we give the students a specification and an implementation, and ask them to modify the implementation to use a different algorithm. For these exercises, 50% of the grade is based on the source code (style, simplicity, and efficiency) and the other 50% on the execution behavior. To determine the grade for the execution behavior, we use a PGMGEN test suite developed by the instructor. These test suites contain hundreds---sometimes thousands---of test cases and are not available to the students until after they have handed in their implementations. We evaluate the student solutions at zero-tolerance: the implementation must pass all test cases to get any credit for execution behavior. The zero-tolerance approach, combined with a large number of test cases that are executed automatically, reinforces implementation to specification: students realize that even ``minor'' errors in the implementation will be exposed.
Figure 2 shows the students' scores for an implementation-to-specification assignment. The students were given a specification and an array-based implementation of a symbol table module. They were asked to change the implementation to use a hash table. They were given a simple test suite and were instructed to carefully inspect their implementations. Over half of the students passed the assignment, indicating that their implementation passed all the test cases. Many of the students that did not pass all the test cases also lost marks on the source code.
Figure 2: Scores for the implementation-to-specification exercise
In the instructor's solution, only 15 lines were changed from or added to the original implementation. Many of the students perceived this exercise as hard, primarily because they found it hard to exploit the existing implementation; they developed their implementations from scratch, completely ignoring the array-based implementation.