The most significant issues bedeviling the use of WeBWorK as an on-line Gateway test are (1) its configuration as a homework problem administering system, rather than a testing system, (2) the resulting manner in which problem sets are created, and the variation intrinsic to those, (3) its ability or lack thereof to force some sort of proctored login, and (4) its lack of a feature to return problem solutions to students. We explore these issues here and look at some possible solutions or work arounds below.
Test administration: The assumption in WeBWorK is that students should have the ability to work and revise their answers after seeing if they got the solutions correct (though this can be restricted; see below). Thus if a problem set is taken multiple times only the final, passing, score is retained in the grade database. Use as a practice test: The structure that makes WeBWorK difficult to use as a test is actually quite good for a practice test. In this case we are mostly concerned with determining whether a student has done any practices at all -- not necessarily the results on all of them. Thus having a single record that a practice was completed is quite sufficient.
Problem set variation: Another problem --from the point of view of a Gateway test -- with WeBWorK's "world view" is that it is predicated on generating a single problem set (albeit unique) per student. By contrast, for a gateway test we want to generate multiple, unique, problems sets for each student.
Proctoring capability: WeBWorK has no inherent ability to force a proctored login before beginning a Gateway test, which makes it more difficult to run a testing environment with strong authentication. There are, however, a number of work-arounds for this.
Problem solutions: Consistent with its design as a homework system, WeBWorK does not give students the correct solutions to the problems that they miss. It does tell them which problems they missed, however. It is very easy for students to download and print a copy of the test/problem set on which they have been working, and if solutions are made available these can be included in the downloaded copy. However, there appears to be no easy way for the solution to be made available only after the student has taken the test/problem set (and not before).
Some of the issues indicated above have work-arounds. As with almost any work-around, there are different ways of doing each, and we do not try and delimit all of them in the following.
Test administration: There are two possible ways of getting around the problems with test administration in WeBWorK. First, we could alter the WeBWorK codebase. This would allow the disabling of a student's ability to reload and resubmit a given gateway attempt. The level of difficulty in doing this is not insignificant but by no means insurmountable. Second, we could restrict the number of attempts using WeBWorK's ability to impose a maximum number of tries on a given problem. By setting this to "1," the student is prevented from resubmitting the problem to get a better grade. This runs into the difficulty of requiring a new problem (set) every time the student takes the gateway. This issue is addressed below.
Problem set variation: This is arguably the most difficult issue to resolve before being able to use WeBWorK for a Gateway exam. Because the problem sets are generated by student, rather than by attempt, the only way within WeBWorK of introducing variation between attempts is to give students a different login name for each attempt. This is, in itself, not so large a difficulty as it may appear, and could be accomplished in a couple of ways. First, we could alter the WeBWorK codebase to introduce a "login iterator" module. In the overview of the student interface, this would be equivalent to replacing the login process shown in figure 1 to the right with the one that is shown in figure 2 below. The "login iterator" would map the student login to a different login, say "username1" for the first attempt at the gateway, "username2" for the second, and so forth, and therefore onto a different problem set for each attempt.
The second way this could be done is by simply requiring different student logins for each attempt. This would require the students to do the work of the "login iterator", trying a different username with each attempt.
Note that in either case by requiring a different login for each student-attempt (rather than each student), data is retained on each time the student takes a test as well. This would allow analysis of the number of times students practice, the number of times they take the Gateway, which problems they missed, etc.
Proctoring capabilities: Again, there are two possible solutions to this. First would be to have two WeBWorK servers, one for general access (practice problems), and the other being a restricted access server used for the proctored exams. By requiring a proctor login to access the latter, some measure of restriction could be introduced. The second would be to have two WeBWorK classes, one of which would be the practice class and the second a proctored class. For the latter, the students' passwords would be the concatenation of their password and a proctor's password, thus requiring the presence of the proctor for them to be able to login.
Problem solutions: I haven't come up with a good work-around for this. If we are willing to give the students information on what they missed but not what the correct solution is, then WeBWorK needs no further modification.
The one place where WeBWorK will have to be massaged to allow its use for the UM Gateway is to address the issue of problem generation by student rather than by attempt. The crude work-around of having the students log in with successive usernames would work, but is probably sufficiently undesirable as to demand some alternative, of which the login iterator is probably the best. An alternative, less invasive, solution would be to have a module that preceded the login module shown in figures 1 and 2 above which generated a new login for the student -- that is, the student would enter their username and password, and be told a new username (with an appended number) for the following session. They could then log in as usual.
Because of the equivalence between student logins and tests, a test (problem or problem set) is created for each student login. Therefore, at the beginning of the term there must be enough logins (or, enough tests) for all student Gateway attempts -- or a means by which additional logins/tests may be generated "on the fly." It is probably simplest to assume that a sufficient number of tests (problem sets) are generated at the beginning of the term (or beginning of the time period in which the Gateway may be taken). The most simplistic (but least efficient) way of doing this is to cap the number of attempts a student is allowed -- or pick an initial number after which additional tests must be generated -- and generate problem sets for that number of logins per student. A more sophisticated solution would be to generate a sufficient number of tests, regardless of login, and then have either the "login iterator" (discussed in workarounds) or pre-login module (discussed in required changes) do the mapping between the existing test bank and new student logins.
Note that because the grading of problem sets in WeBWorK is done on a problem-by-problem basis, it is probably easiest to have a single problem on a problem set be the entire Gateway test. This means that in the student interface to WeBWorK, the intermediate step of selecting which problem to do in the problem set is not needed. Getting rid of it would, however, require changing the WeBWorK code base.
As discussed in the context of proctoring in the work-arounds section above, it is probably necessary to have two classes defined for each class-section of the course using the Gateway tests. These are for the proctored and unproctored test modes, and could exist on the same or separate servers.
The issue of proctoring the gateways has been addressed in the work-arounds section above. The generation of data for the classes is then done for either the practice or the proctored class, by the instructor of the class. It may be desirable or possible to develop tools to assist them in doing this.
WeBWorK is not designed to be a testing system, and this makes it difficult to easily apply for the Gateway exam. It should be possible to do this, but will require some changes. Once this is done, the ease of use for the students will be quite good. Maintentance of the system from the system administrator or instructors' standpoint is less straightforward.
Potential additional uses for the gateway testing software include, among other things, (1) on-line placement testing or calculus readiness testing, (2) calculus or precalculus tutorials providing self-paced or supporting materials for students with weakness in either of these subjects, and (3) integrated readiness tests and tutorials, which would route students from a test to tutorials on missed material.
WeBWorK is not a testing system. Thus for any testing scenario such as (1) the same types of difficulties, as outlined above, are confronted. It is, however, a homework system, which allows its extension to tutorial-type applications such as (2) quite easily. It is fairly easy to imagine a system whereby a student would work though a series of annotated problems that would themselves comprise the tutorial. Extending this to (3), integrated tests and tutorials, would require somewhat more thought but appears to fall within the capabilities of the software.
Implementing a Gateway testing system with WeBWorK is possible but non-trivial. The technical issues involved in configuring the authentication and student log-in names, as well as nailing down a good proctoring system, would require some working out. There may be some system issues of server capability if hundreds or thousands of students are taking a test which is CGI/Perl based as well, though this, too, is not insurmountable. WeBWorK's extension into on-line tutorials and homework-type assignments would be much easier to accomplish, as this is the role for which it was developed.