BACKGROUND OF THE INVENTION

For many years standardized tests have been administered to examinees for various reasons such as for educational testing or for evaluating particular skills. For instance, academic skills tests (e.g. SATs, LSATs, GMATs, etc.) are typically administered to a large number of students. Results of these tests are used by colleges, universities and other educational institutions as a factor in determining whether an examinee should be admitted to study at that educational institution. Other standardized testing is carried out to determine whether or not an individual has attained a specified level of knowledge, or mastery, of a given subject. Such testing is referred to as mastery testing (e.g. achievement tests offered to students in a variety of subjects and the results being used for college entrance decisions).

FIG. 1 depicts a sample question and sample direction which might be given on a standardized test. The stem 4, the stimulus 5, responses 6 and directions 7 for responding to the stem 4 are collectively referred to as an item. The stem 4 refers to a test question or statement to which an examinee is to respond, e.g., question 13. The stimulus 5 is the text and/or graphical information (e.g., a map, scale, graph, or reading passage) to which a stem may refer. Often the same stimulus is used with more than one stem. Some items do not have a stimulus. Items having a common stimulus are defined as a set. In FIG. 1, questions 13 and 14 refer to stimulus 5 and therefore form a set. Items sharing common directions are defined as a group. Thus, questions 8-27 form a group. Only questions 8-14, however, are shown in FIG. 1.

A typical standardized answer sheet for a multiple choice exam is shown in FIG. 2. The examinee is required to select one of the responses according to the directions provided with each item and fill in the appropriate circle on the answer sheet. For instance, the correct answer to the question stated by stem 1 is choice B of the responses 3. Thus, the circle designated 8 in FIG. 2 corresponding to choice (b) is the correct answer to this item, i.e. question 13 should be filled in by the examinee as shown.

Generally, examinees register to take a particular test, by filling out a registration form and sending it to a test processing center such as Educational Testing Service, Princeton, N.J. by a specified registration date. A registration form usually requires that an examinee provide information such as the examinee's name and address, test to be taken and some related biographical information. After all of the registration forms have been received by the test administration center, the examinee information such as name, address, some recipients background questions, etc., is processed. Each examinee is scheduled to take the test by assigning to that examinee a place and time at which the test can be administered to that examinee. Typically, a number of examinees are scheduled to take the test at the same time and same place to conserve on administrative costs. One or more test administrators will be present at the locations where the test is scheduled to be taken.

Test administrators are generally responsible for distributing the test material, providing instructions to the examinees, monitoring any timing constraints required by the particular test and collecting the test material when the testing time has ended or when the examinee has finished taking the test. After collecting the examinees' responses and other test material, the administrator either directly or indirectly sends them back to the test processing facility, for scoring and evaluation.

After all of the examinees' tests are graded, statistical and other processing may be provided for various reasons. For instance, to assess one examinee's score, it is necessary to compare his or her score to those of other examinees taking the same test. Another important reason to evaluate the test results for statistical purposes is to create and update an informations bank containing the performance statistics of each item used or created for previous tests. This information may then be used for the creation of future tests.

A goal of standardized testing is to construct a test efficiently for the purpose of measuring a skill, ability, etc. Therefore, each test is constructed to conform to a test specification which defines the rules and/or constraints for selecting the items. In constructing a test, test developers select items from a pool of items so that the combination of selected items satisfy the test specification.

A test is typically divided into sections of questions. The test specification generally defines the number of items to be presented in the test, the number of test sections, the number of questions in each section, the time for taking the test, and the allotted time for responding to all the items in each test section. The test specification also specifies criteria for item selection. These are based on at least four item characteristics which include: (1) item content, e.g., mathematical questions relating to arithmetic, algebra, or geometry; (2) cross-information among items, e.g., more than one item testing the same point; (3) number of items/set, i.e. a identification of a subset of items of a larger set; and (4) statistical properties of items derived from pretesting, e.g. difficulty of the selected items.

In recent years, these methods for creating, delivering, administering, and scoring tests have been determined to be inadequate. Due to the number of examinees taking standardized tests, the demand for developing new and more diverse tests and a need to provide more flexibility in scheduling tests without sacrificing administration costs and security have increased.

One solution to these demands would be to automate the entire testing process. However, up until now only a few attempts have been made to automate only portions of the testing process. Furthermore, these attempts are limited in their ability to generate a variety of item types. They are not modular in their design to allow independent replacement of software or hardware, nor do they provide security and integrity features required for a standardized testing environment.

There have been attempts to develop computerized tools for instructional purposes. These products, although primarily geared to delivering instructional systems, often contain testing components as well. Some examples of instructional programs are available from Computer Curriculum Corp., Computer Networking Specialists Inc., Computer Systems Research, DEGEM, Ideal Learning, Josten's Learning Corp., New Century Education, Plato Educational Services--TRO Inc., Unisys--ICOPN System, Wasatch Education System, and Wicat Systems. Wasatch Courseware, for instance, provides on-line tools, such as a notebook, a pop-up calculator, word processor, graphics tool, glossary, and a database embedded into the lessons. Josten's Learning Corp. provides some flexibility in the hardware and software available for executing lessons such as networked or non-networked systems, the use of third party software, and the ability to operate its instructional system from a remote site. Ideal Learning has a management system which is also capable of accommodating third party software, and its test scoring system can score tests which are generated by a number of test developers including standardized tests. The DEGEM System is a networked system which is capable of providing statistical data on student or class progress. Therefore, although some of these instructional programs incorporate some features which could be utilized in an automated standardized computer-based testing system, none of them provides a flexible and integrated system for developing, generating, delivering, administering and processing computerized standardized tests.

There are also a number of systems for computerizing parts of the test construction process. (See e.g., a review by Hsu and Sadock (1985)). Perhaps the most comprehensive of these testing programs is the MicroCAT System developed by Assessment Systems Corporation. The MicroCAT System comprises four primary subsystems, one for each of development, examination, assessment, and conventional testing.

Although MicroCAT has been noted for its comprehensiveness, it has been criticized for a number of limitations. For instance, development of a test having a specification which does not match one of its predefined templates requires a detailed understanding of MicroCAT's programming language. Its graphics tools are very limited, and other commercial drawing packages such as PC Paint cannot be substituted for MicroCat's graphics. Furthermore, there is no on-line help available from either the test development system or from the examination system. Without an on-line help facility, a system such as MicroCAT could not practically be used to deliver and administer standardized tests to thousands of examinees each year. To use the MicroCAT assessment System, the test data must have been based on tests which were generated only by MicroCAT's specifications. Furthermore, MicroCAT does not provide security for examinee performance files nor does it provide integrity features to guard against power interruptions and the like.

To accommodate standardized tests in computer based testing, there is a need for a comprehensive computer based testing system which provides flexible test development and production, test administration and test delivery, as well as preprocessing and postprocessing of item statistics and examinee performance. Such a system should incorporate data integrity features, including system failure recovery and data security features. The design should be modular and extensible so that substantially every hardware and software component can be modified or replaced without affecting the functioning of the remainder of the system.

SUMMARY OF THE INVENTION

The present invention fulfills the above-described need by providing a computer-based testing system comprising a test development system and a test delivery system. The test development system comprises a test document creation system for specifying the test contents, an item preparation system for computerizing each of the items in the test, a test preparation system for preparing a computerized test, and a test packaging system for combining all of the items and test components into a computerized test package. The computerized test package is then delivered to authorized examinees on a workstation by the test delivery system.

The computer-based testing system in a preferred embodiment further comprises an administrative system for initiating and terminating the delivery of the computerized test to an examinee. Preferably, the administrative system also provides security to prevent access by unauthorized persons.

In a further preferred embodiment, the test development system generates, and the test delivery system presents to the examinee one or more of the following: tutorials instructing examinees how to interact with the workstation, directions for taking the test, a help facility selectable by an examinee for additional instructions, and a review facility selectable by an examinee for providing a list of questions in the test so that the examinee can select a specific question to be presented.

The present invention also provides a method of computerized testing comprising the steps of producing a computerized test, delivering the computerized test on a workstation to an examinee, and recording the examinee's responses to the questions presented. In a preferred embodiment, the method further comprises administering the test to an examinee and securing the test from access by unauthorized persons.

In a further preferred embodiment, user controls are provided to the examinee to provide navigational control and to access functions of the computerized test.

The present invention also provides a method of administering a computerized test on a computer workstation. According to the method of the present invention, the computerized test is installed on the workstation and the delivery of the computerized test to an authorized examinee is initiated. In a preferred embodiment, the method checks the installed computerized test for data integrity and viruses and protects the installed computerized test from access by unauthorized persons. In a further preferred embodiment, the method also records security-related information in log records and forms a security log file of those records.

The present invention additionally provides a method for delivering a computerized test for standardized testing. According to the method, a standardized test is created and then an electronic form of the test is prepared to produced the computerized test. A workstation is provided having a display for presenting items of the computerized test to an examinee and having a storage device. The examinees responses to the items are accepted and then recorded.

The present invention also provides a data distribution system. In a preferred embodiment, the data distribution system receives examinee performance files, security log files, and error log files all of which are preferably generated by the computer based testing system according to the present invention. The data distribution system comprises a file processing component and preferably an examinee performance database, security log database and a computer based testing network database. The file processing component processes the received files and updates the databases with information related to the delivery of the computerized tests to examinees on workstations in test centers. In further preferred embodiments, the data distribution system comprises a format post processing component for formatting information in the examinee performance database according to a definition file provided by a post processing system that maintains information about individuals who take a particular test. In an additional preferred embodiment, a report processing component is provided to retrieve data from one or more of the databases to generate any of the following reports: activity, audit trail, daily processing control, exception, security/event log, and essay.


BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood, and its numerous objects and advantages will become apparent by reference to the following detailed description of the invention when taken in conjunction with the following drawings.

FIG. 1 is an example of written test questions and related directions.

FIG. 2 is a sample answer sheet used for paper and pencil tests.

FIG. 3 is a general overview of computer based testing facilities.

FIG. 4 is an interface diagram depicting the interfaces of each of the CBT (Computer Based Testing) systems according to the present invention.

FIG. 5 is an interface diagram showing the subsystems of the test development system according to the present invention.

FIG. 6 is an interface diagram showing the administrative system and test delivery system interfaces according to the present invention.

FIG. 7 is an interface diagram showing the subsystems of the NDDS (Network Data Distribution System) according to the present invention.

FIG. 8 is an information flow diagram for the "TD/DC" system.

FIG. 9 is a functional flow diagram of test production according to the present invention.

FIG. 10 shows an example of a dialog box presented by the IPT (Item Preparation Tool).

FIG. 11 shows an example of some item components as viewed from the item preparation tool.

FIG. 12 shows the scrolling capability utilized by the item preparation tool.

FIG. 13 provides an example of selecting text in a stimulus by reverse video.

FIG. 14 shows dialog boxes presented by the IPT which prompt for response parameters.

FIG. 15 shows a dialog box presented by the IPT which prompts for key information.

FIG. 16 shows an example of an item having a stem referencing a demarcated portion of a stimulus.

FIG. 17 depicts an appropriate response to the item presented and shown in FIG. 16.

FIG. 18 shows a second example of an item having a stem referencing the item's stimulus.

FIG. 19 depicts an appropriate response to the item presented and shown in FIG. 18.

FIGS. 20 and 21 show an example of a reference file replete with custom and interaction codes according to the present invention.

FIG. 22 shows the flow of items and keys from test development to test production services.

FIG. 23 shows a functional block diagram of the test preparation process according to the present invention.

FIG. 24 shows the relationship between the session script, test scripts, and units.

FIG. 25 provides a functional flow diagram of the test packaging process according to the present invention.

FIG. 26 is a block diagram showing the components of a test package.

FIG. 27 is a block diagram showing some item level components included in the presentation BLOB (Binary Large Object).

FIG. 28 is a block diagram showing some test level components included in the presentation BLOB.

FIG. 29 is a block diagram showing some table components included in a SKM BLOB (Scoring Key Management).

FIG. 30 provides a functional block diagram of the test packaging process.

FIG. 31 shows a functional block diagram of the test delivery system according to present invention.

FIG. 32 is a functional flow diagram of the test delivery process according to the present invention.

FIG. 33 shows some primary screen components of each screen presented during the test delivery.

FIGS. 34(a)-(g) show examples of pane arrangements supported by the present invention.

FIG. 35 is an example of a directions screen.

FIGS. 36(a)-(l) show examples of message screens supported by the present invention.

FIGS. 37-39 provide some examples of Help Screens.

FIG. 40 provides an example of a review screen.

FIG. 41 provides a few examples of selectable testing controls supported by the present invention.

FIGS. 42(a)-(b), 43, 44(a)-(c), 45-47 provide some examples of examinee interaction with items having various response types.

FIG. 48 shows the data format of an examinee performance file according to the present invention.

FIG. 49 shows possible data fields associated with the "Start Session" log event.

FIG. 50 shows possible data fields associated with the "End Item" log event.

FIG. 51 shows a functional flow diagram of the administrative process according to the present invention.

FIG. 52 shows some of the files on a workstation hard disk after it has been configured for computer based testing.

FIG. 53 is a functional flow diagram describing the Close-of-Day Procedure.

FIG. 54 shows the data format of a log record according to the present invention.

FIG. 55 is a flowchart of the Main-- Procedure of the Administrative Application.

FIG. 56A and 56B are flowcharts of the Start-- System-- Procedure of the Administrative Application.

FIG. 57 is an example of a system status screen.

FIG. 58 is a flowchart of the Logon-- Procedure of the Administrative Application.

FIG. 59 is an example of a sign-in screen provided by the Administrative Application.

FIG. 60 is a flowchart of the Process-- State-- Procedure of the Administrative Application.

FIG. 61 is a flowchart of the State-- Null-- Procedure of the Administrative Application.

FIG. 62 is a flowchart of the Change-- Password-- Procedure of the Administrative Application.

FIG. 63 is a flowchart of the State-- Admin-- Procedure of the Administrative Application.

FIG. 64 is an example of a screen showing the main menu of the Administrative System.

FIGS. 65A and 65B are flowcharts of the State-- Close-- Procedure of the Administrative Application.

FIG. 66 is an example of a screen provided by the Administrative Application allowing an administrator to enter test counts.

FIG. 67 is a flowchart of the State-- Maint-- Procedure of the Administrative Application.

FIG. 68 is a flowchart of the State-- TDS-- Procedure of the Administrative Application.

FIG. 69 is an example of a screen provided by the Administrative Application when the testing session is complete.

FIG. 70 is a flowchart of the State-- Exit-- Procedure of the Administrative Application.

FIG. 71 is a flowchart of the Menu-- OpTest-- Procedure of the Administrative Application.

FIG. 72 is a flowchart of the Menu-- DemoTest-- Procedure of the Administrative Application.

FIGS. 73A and 73B are a flowchart of the Menu-- TestCommon-- Procedure of the Administrative Application.

FIG. 74 is an example of a screen provided by the Administrative application allowing an administrator to enter examinee identification information.

FIG. 75 is an example of a screen provided by the Administrative Application allowing an administrator to administer a test.

FIG. 76 is an example of an examinee confirmation screen.

FIG. 77 is an example of a screen provided by the Administrative Application allowing an administrator to terminate the testing session or to edit the examinee information.

FIG. 78 is a flowchart of the Menu-- RestartTest-- Procedure of the Administrative Application.

FIG. 79 is an example of a screen provided by the Administrative Application listing the available testing programs.

FIG. 80 is an example of a screen provided by the Administrative Application listing the sessions that are available for restart.

FIG. 81 is a flowchart of the Menu-- CloseDay-- Procedure of the Administrative Application.

FIG. 82 is a flowchart of the Menu-- Exit-- Procedure of the Administrative Application.

FIGS. 83A and 83B are flowcharts of the Menu-- LogonMaint-- Procedure of the Administrative Application.

FIG. 84 is a flowchart of the Menu-- CHGPassword-- Procedure of the Administrative Application.

FIG. 85 is a flowchart of the Stop-- System-- Procedure of the Administrative Application.

FIG. 86 is a flowchart of the Menu-- About-- Procedure of the Administrative Application.

FIG. 87 is a block diagram showing the inputs and outputs of the Network Data Distribution System (NDDS) according to a preferred embodiment of the present invention.

FIG. 88 is an example of the Main Menu Screen for the NDDS.

FIG. 89 is a preferred directory structure of the NDDS.

FIGS. 90A through 90C is a flowchart of a preferred procedure for processing CBT transmission files.

FIG. 91 is an example of a screen provided by the Administrative Application prompting the administrator to enter the data disk.

FIGS. 92A and 92B is a flowchart of a preferred procedure for processing the CBT Data Disks.

FIG. 93 is an example of a screen provided by the Administrative Application prompting the administrator to insert the backup disks.

FIGS. 94A and 94B is a flowchart of a preferred procedure processing the CBT Backup Disks.

FIG. 95 is an example of the screen provided by the NDDS of an activity report selection menu.

FIG. 96 is a flowchart of a preferred procedure for implementing the activity reporting process according to the present invention.

FIG. 97 is an example of a screen provided by the NDDS of an audit trail report selection menu.

FIG. 98 is a flowchart of a preferred procedure for implementing the audit trail reporting process.

FIG. 99 is an example of a screen provided by the NDDS of a Security/Event Log Report Selection Menu.

FIGS. 100A through 100F are examples of screens provided by the NDDS for carrying out Computer Based Testing Network (CBTN) processing according to a preferred embodiment of the present invention.


DETAILED DESCRIPTION

I. Computer Based Testing (CBT) System Overview

Referring to the drawings wherein like numerals represent like elements, there is illustrated in FIG. 3 a general overview of computer based testing. Computerized tests are developed at a central processing site 1. Development of a computerized test includes the creation of various data files by application specific software. The computerized tests are packaged at the central processing site 1 and delivered to one or more test centers 2. Each test center 2 provides at least one workstation 3 on which a computerized test is administered to an examinee. In a preferred embodiment, the workstation 3 is a personal computer equipped with a mouse.

A test center 2 may for example be located at a school or a dedicated test site. Generally, a test administrator located at the test center 2 loads the computerized test, data files and application software developed at the central processing site 1 onto the hard disk of each workstation 3 at the test center 2. The administrator initiates the delivery of a computerized test to an examinee who is scheduled to take the test. The examinee's responses to questions presented by the test are preferably stored on the hard disk on each workstation 3 and are later preferably backed-up by the administrator and transferred to the central processing site 1 for scoring and evaluation.

In FIG. 3, one central processing site 1, three test centers 2, and 9 workstations 3 apportioned among the test centers 2 are shown. However, it should be understood, that any number test centers 2 and workstations 3 may be used by the CBT system.

A block diagram of the CBT system is shown in FIG. 4. The lines in the diagram demarcating each system represent interfaces that pass information between the various systems which comprise the CBT system. The double line separates those systems that reside at the test centers from those that reside at the central processing site. Those systems shown within the double lines are the systems residing at the test centers, and those systems shown outside the double lines are the systems residing at the central processing site.

Still referring to FIG. 4, the CBT system comprises six separate systems. The Administrative System 14 provides substantially all administrative and control functions at a test center. The Test Delivery System 12 actually presents questions and information to the examinee at a workstation. The Administrative System 14 initiates the delivery of a test to an authorized examinee and secures the test by prohibiting access by any unauthorized person. Communication with the Test Delivery System 12 occurs through the Administrative System 14. The Network Data Distribution System (NDDS) 18 receives data from the test centers 2 and distributes returned data to other systems at the central processing site. The Test Development System 10 provides substantially all functions necessary to create test questions and package computerized tests. The Preprocessing System 20 provides functions performed prior to the testing session, such as registration. Such Preprocessing systems are typically custom designed for a particular test and are provided commercially by various testing support companies such as Educational Testing Service, Psychological Corporation, and American College Testing Service. Like the Preprocessing System 20, Postprocessing systems are typically customized for each type of test and are provided commercially through Educational Testing Service, Psychological Corporation, and American College Testing Service. The Postprocessing System 16 provides functions performed after the testing session, such as issuing official score reports or archiving examinee records.

A detailed block diagram of the Test Development System 10 is shown in FIG. 5. Five primary functions are performed within the Test Development System 10, including test development/document creation ("TD/DC") 62, an item preparation system 60, scoring and test key management (SKM) 66, computerized test preparation system 57 and test packaging 58. The Test Development System 10 permits test developers to develop items and a Test Production Staff (TPS) to computerize the items. It also supports the creation of test scripts. A test script is the electronic form of a test. It provides option settings and configuration data necessary for delivering the test on a workstation. The Test Development System 10 also supports the creation of test keys, i.e., correct response to each item, and the packaging of all components into one test.

Items are preferably written and created using the "TD/DC" (Test Development/Document Creation) system 62. The "TD/DC" system 62 interfaces with the Item Preparation system 60 and with the scoring and key management system 66 to computerize the item content and key respectively. The Item Preparation System 60 is used to computerize the items for presentation by the Test Delivery System 12 and enter the computerized version of the item key which differs depending on the item type. The Item Preparation System 60 prepares data for the scoring and key management system 66 to communicate the computerized key information. The Item Preparation System 60 interfaces with the Test Preparation System 57 to prepare the test scripts. The Test Packaging System 58 interfaces with the Item Preparation System 60, the scoring and key management system 66, and the Test Preparation System 57 to obtain all of the item and test components packaged into a computerized test. The NDDS Interface 56 transmits the computerized test from the Test Packaging System 58 to the NDDS 18 as shown in FIG. 4, for delivery to a test center 2 as shown in FIG. 3. After an examinee has taken a computerized test, an postprocessing interface 64 with the Postprocessing System 16 provides information about item keys to the "TD/DC" System 62 which uses this information to alter items or add new items for use in subsequent tests.

FIG. 6 shows a block diagram of the Administrative System 14 and Test Delivery System 12 and their respective interfaces as shown in FIG. 4. The Administrative System 14 permits test center administrators to control delivery of tests, transmit results to a central processing site, and perform administrative functions such as backup of data, item and software maintenance, and reporting. The Administrative System 14 further prohibits access to the computerized test by unauthorized persons.

An interface with the Network Data Distribution System (NDDS) 18 at the central processing site enables the Administrative System 14 at a test center to send packaged examinee data and reports to a central processing site. Data and software are sent from the central processing site to the test centers on diskettes.

Still referring to FIG. 6, the Test Delivery application Interface 26 is shown as having three specific interface functions. First, the Administration system 14 can initiate a delivery of a computerized test and pass the necessary information to the Test Delivery System 12. The Administration System 14 may also interact with the Test Delivery System 12 for purposes such as terminating the testing session, processing examinee breaks, and timing essays, as appropriate.

After termination of the test delivery, the Test Delivery System 12 transfers information such as examinee performance data, return codes, and other processing data as appropriate to the Administration System 14 through the Test Delivery Application Interface 26. The Administration System 14 then regains control of the workstation.

The Administrative Application 30 provides test center administrators withthe ability to perform various functions including: controlling access to computerized tests and related data through levels of authorization and password protection; entering and editing examinee identification information prior to the testing session; selecting the test to administer; terminating tests; backing up examinee and administrative data; transmitting data to the central processing site; changing passwords and adding or deleting administrator logon IDs; and reporting irregularity and activity data to the central processing site.

A detailed block diagram of the Network Data Distribution System (NDDS) 18 with its interfaces is shown in FIG. 7. The Network Data Distribution System (NDDS) 18 provides substantially all necessary support functions for the CBT system to control the network of test centers. The Test Center Administrative Application Interface 36 permits the transfer of applications and computerized tests to the test centers and examinee records and reporting information (data related to system errors and test/workstation security) from the test centers to the central processing site. The Test Development Interface 42 provides the means by which new or revised computerized tests are sent from the Test Development System 10 to the NDDS 18. The NDDS 18 uses a Test Center Information Database 40 to determine which test centers should be sent any new/revised tests, and to create reports from data received from the test centers. The NDDS Processing component 44 receives data from test centers 2 via Administrative Application Interface 36, checks it, sorts it, and processes it according to its type (program data, security data and reporting data). Reporting data is used to create the necessary reports. Program data such as examinee records, are processed to consolidate and reformat the information in a form suitable for postprocessing. The Distribution Interface 38 then distributes the processed data to the Postprocessing System 16.

II. Test Development System

A. Test Development/Document Creation

In a preferred embodiment, test developers create tests at the central processing site. In computer based testing, the tests are created by the test developers (TD) and are further processed and packaged by test production staff (TPS). The "TD/DC" System which is developed by Educational Testing Service is preferably used to create the test forms. It should be understood, however, that other test document creation systems could likewise be used to create test forms for computer based testing. Therefore, although a detailed explanation of "TD/DC" is not required for a description of the test development system, an understanding of its functional and procedural operation will aid in understanding the test development system.

The "TD/DC" System 62 (FIG. 5) is a fully automated system in and of itself. It consists of a central item database and local personal computer based workstations. The central item database stores substantially all items previously used on standardized exams as well as other items that have been created but not previously selected for inclusion in an exam. Associated with each item stored in the central item database, is data stored in fields related to the item's answer key, revisions it has undergone, a list of the test forms in which the item has previously appeared, and statistical data indicating how the item performed at each previous administration. Other descriptive data fields include information related to the item type (i.e., multiple choice or fill in the blank), the item's author, copyright information, and both content and cognitive information specific to each testing program (e.g., SAT, GRE, etc.).

Every item is assigned a unique number called an accession number that identifies the item and all of its associated data. The "TD/DC" system software allows items to be located by means of any of the data fields associated with the item.

The central item database software allows test developers to access item information within the central item database. This software supports the downloading of items and associated data to local workstations. Pools of items may also be selecting and then downloaded. This software also enables test developers to bank, edit, and classify features of the items stored in the central item database. Additionally, statistical information related to the use of an item in an administration of a test is received by the central database software. Through a statistical feedback system, this statistical data is added to the data stored in the central item database.

New items may be written by test developers at the local workstations. Software provided on the local workstations supports classifying, banking, and viewing these new items. With respect to the downloaded items, this software also allows test developers to view those items and their associated data and permits the test developers to enter and revise the statistical data.

Test developers also assemble draft tests on local workstations. The local workstations provide software supporting item selection based on a number of criteria so that tests may be assembled to satisfy substantially any test specification. When all of the items are selected to satisfy the test specification, these items and associated data are assembled into what is referred to as a Worksheet.

Test production for computer based testing, requires certain inputs from test developers and/or a test document creation system. FIG. 8 depicts the inputs provided by the test developer and the outputs generated by "TD/DC" for use by TPS. For instance, if the "TD/DC" system is used, the information shown as offload files 74 and workfolder 76 is preferably provided to TPS. As described above, worksheets 72 are created by TD by downloading the selected items and associated data. An offload program may then be invoked to copy the offload files, such as the item component text (stem, stimulus, response, and directions), the response type, (e.g., multiple choice) response class (e.g., single response answer), the answer key and the item's accession number, onto a diskette. In preferred embodiments, the TD also prepare a workfolder 76 containing information related to the computerized presentation of the items, and graphics to be prepared as a computerized image and incorporated into the item presentation.

B. Test Production

1. Overview of Test Production

Test production comprises at least three primary functions, item preparation, test preparation and test packaging. The Item Preparation System 60 as shown in FIG. 5 is used by TPS to create an "on screen" version of the items prepared. As described above, the test questions are prepared beforehand by test developers preferably using the "TD/DC" System 62 or equivalent system. Item text is edited until the content is satisfactory to the test developers. An offload program is used by the test developers to copy the selected items to a diskette which is sent with a work folder of batch-related documents to the test production staff. The test production staff then makes a computer deliverable image of the items in the form of files and prepares a test script for implementing the test. The test packaging system 58 combines the item files with the test script to form the computerized test.

In a preferred embodiment, the Item Preparation system 60 comprises seven programs providing the functions shown in the flow diagram of FIG. 8. Table 1 below itemizes the programs used by the Item Preparation System 60. The table lists the purpose of each program, the program name, the operating environment in which the program preferably is executed and the user during the test development process. "WORD FOR WINDOWS" and "PAINTBRUSH" are commercially available from Microsoft Corporation.
TABLE 1
______________________________________
PROGRAM COMPONENTS OF
THE ITEM PREPARATION SUBSYSTEM
Purpose Program Name Environment User
______________________________________
Item Offload
CBTOFF DOS TD*
Item Element
IEG DOS TPS**
Generator
Word "WORD" "WINDOWS" TPS
Processing
Translation
XLATE DOS SYS***
Graphics Prep
"PAINTBRUSH" "WINDOWS" TPS
Item IPT "WINDOWS" TPS
Preparation
Item Review
IVT "WINDOWS" TD/TPS
______________________________________
*TD test developers
**TPS test production staff
***SYS Systems

A functional flow diagram of the Test Production Process is shown in FIG. 9. Assuming for purposes of description that the system used to create the test document is the "TD/DC" system, items are first offloaded at 210 by test developers using the item offload function of the "TD/DC" system and are stored on a diskette. The offloaded items and a corresponding workfolder are then transferred to the TPS. If graphics are to be shown with the item as determined at 212, a computerized image of the graphics are prepared at 214 by TPS. TPS reads the items from the diskette and separate them into component parts shown at 216 using the Item Element Generator (IEG). The text of the item components can then be edited using a word processor at 218 such as "WORD" for example. Information may be added to each component regarding, for instance, point size, font, leading, column arrangement, etc. during word processing 218. Then, the items may be prepared at 220 using the Item Preparation Tool (IPT) to arrange the components on the computer display and specify characteristics about each item (e.g., multiple choice, response type, etc.). Finally, an electronic "proofing" copy of the items is returned to the test developers for review at 224. The test developers may then provide corrections or final approval. The proofing copy is preferably reviewable by test developers via a modified form of the Item Preparation Tool known as the Item View Tool (IVT).

TPS may iterate between word processing 218 and item preparation at 220 until satisfied with the results. Likewise, the entire process from item offload 210 through item review 224 may be repeated. Simple corrections can be made by sending marked-up prints from the test developers to test production staff so that the most current version of the test can be called to the screen again and edited according to the comments produced by the test developers.

Once final approval has been received from the test developers, a test is prepared as shown at 226 by a test preparation tool based on information provided in the workfolder. Finally, TPAK packages all of the item files and test scripts into a computerized test at 227.

2. Item Preparation

a. Item Offload Program

The Item Offload function 210 is executed by test developers on the "TD/DC" system using the item offload program. This function extracts data from the "TD/DC" system to be used as input for test production which ultimately results in the creation of a computerized test. In the "TD/DC" system, a file which is known as the Worksheet and described previously in Section II (A), lists the accession numbers of items. Each Worksheet has a unique name which is assigned by a test developer. Item offload 210 creates a single file with all of the offloadable information for all items in the Worksheet. The offload file's name is the same as the Worksheet name. Item offload 210 opens the Worksheet, reads each item pointed to by the accession number, and writes to an ASCII file the item's accession number, classification codes, rationale and item text.

Test production requires specific classification information. Thus, information contained in the Worksheet is preferably categorized by the test developers. One category may include item information. For instance, test developers should provide codes that indicate how each part of the item's text is used for production purposes, e.g., the code indicating that particular text is the stem, the response, or the directions associated with each item.

Another category of information which may be provided to test production is rationale information. Again, test developers may insert codes and text related to the key description, number of responses required to properly answer the question posed by the stem, a paraphrase summarizing the content of the item, and general remarks regarding the appearance of each item.

A third category of information which could be provided to test production is classification profiles of the items included in the Worksheet. The classification profiles may include codes identifying the item class, item type, item structure, and information describing how the item should be presented to an examinee. The item class refers to its response class and is indicative of whether a single response, multiple response, or free response is required by the item. The item type code specifies its response type and is indicative of the type of response required to answer the question posed by the item. For instance, the item type codes specifies selection of a value on a scale, selection of a response with an ellipse or a box, insert text, or select a choice from a table. These and many other item types will be described in detail below.

The item presentation code specifies a predefined screen template. The screen templates indicate how many panes the screen should be divided into for each item and the location of the panes. Additional codes are used to specify which item information, i.e., stem, response, directions or stimulus, is to be placed in each pane and its position within the pane. Similarly, the item presentation codes will be described in more detail below.

A fourth category of information which may be provided to test production is the stimulus formation for a set of items. The stimulus formation code specifies the beginning of a stimulus to be shared by a set of items and the presentation of the stimulus. When a stimulus is referenced by only a single item, the item is called a discrete item. In the case of a discrete item, the stimulus information codes are preferably provided with the item information codes rather than the stimulus information codes.

Prior to executing item offload 210, the test developer using the "TD/DC" system may insert tags in the item and rationale text which are used by the Item Element Generator (IEG) program to separate the text into smaller components. Tags in the item text are used to delineate the directions, stem and response components. Tags in the rationale text are used to supply information about the key, number of required responses and a paraphrase summarizing the item, as well as the rationale text.

b. Graphics

The work folder indicates whether graphics are required as shown at 212 in FIG. 9. TPS may generate the graphics using "PAINTBRUSH" as one example. The graphics files are named, for example, by adding an extension, .Gnn, to the accession number of the item which contains the graphic to be presented. The "nn" is representative of a number so that up to 99 graphics in this example may be associated with an item.

c. Item Element Generator (IEG)

The Item Element Generator (IEG) is a DOS program used by the TPS. The IEG is preferably written in Microsoft C version 5.1. The IEG separates the ASCII file created by the Item Offload program as shown at 216. The individual items in the Offload file are separated from each other, then the items themselves are broken into components and stored in files.

The component files are preferably named by tacking an extension onto an item accession number. The extension specifies which piece of the item (i.e., stem, stimulus, response, directions, etc.) the file contains. Since the accession number is used for the base name of a file, it is easy to locate all of an item components. Table 2 below lists each component file with a predetermined extension.
TABLE 2
______________________________________
COMPONENT FILES
EXTENSION CROSS REFERENCE
File Name Contents
______________________________________
acc.STE Stem component
acc.REF Stimulus component
acc.DIR Directions component
acc.RSP Response component
acc.CTL Control component
______________________________________

A control file is also generated by the IEG for each item. A control file is a master repository for an item's information. Specifically, the control file defines how these components are to appear as an item on a display during test delivery. Minimally, a control file and a stem file are generated for each item. A directions file and a response file may be generated if the item contains directions and response components. A reference file contains the stimulus material and is generated only for items which belong to a set. Additionally, the control files of set members contain the accession number of the stimulus material.

The IEG also creates two log files during its execution and records errors in an error file. The error file and log files are named by adding an extension to the base name of the input file, that is, the file generated during item offload 210. Thus, for example, if the input file is named TEST, log or error files are created by adding a unique extension to TEST.

One of the log files is the Item Accession List File, which contains the accession numbers of discrete items and members of sets (excluding stimulus material) for which component files were produced. This file is named using an input file name and .IAL extension such as inputfile. IAL. Accession numbers are preferably written to the file in ASCII format, one per line, in the order in which they were processed.

The second log file created by IEG is the Batch History File, which contains the accession numbers of all items in the offload file. This file is named using an input file name and .BHF extension such as inputfile.BHF. Accession numbers are preferably written to the file in ASCII format, one per line, in the order in which they were processed. Accession numbers of the set may be written to the file one after the other, starting with the stimulus material.

The Error Log file is an ASCII file that logs errors encountered during IEG execution. This file is named inputfile.ERR.

The .IAL and .BHF are preferably always generated. The .ERR file is generated if errors are reported. Individual component files are generated depending upon the item type and tags embedded in the item by test developers.

In a preferred embodiment, IEG uses a code conversion table to convert "TD/DC" classification codes to Item Preparation codes. By using a separate file for this table, the IEG does not have to be recompiled and linked if the codes are extended or altered in any way.

d. Item Preparation Tool

In a preferred embodiment, the Item Preparation Tool (IPT) is a WINDOWS-based application used by the TPS. The IPT is also preferably written in Microsoft C version 5.1 along with the WINDOWS Software Development Kit. It is the electronic analog of tools with which to prepare the item image. Using the Item Preparation Tool, TPS can process items received after the component parts are separated by the IEG.

After item preparation is started at 220, test production staff can scan a list of items available for processing from the item accession list file which is created by the IEG at the step 216. The list is generated by compiling all files with the .STE extension in the item accession list file.

Each item having a .STE file extension is then edited and processed by test production staff to create a computer deliverable form of the item which is developed by the test developers. In combination with the item preparation process 220, test production staff can edit any of the item text components by word processing 218. After editing the text of the desired component, test production staff may return to item preparation 220. The revised text may then be displayed by the IPT.

The IPT is preferably a menu-driven application. The lowest level menu options have dialog boxes. Dialog boxes are typically used in a WINDOWS environment for prompting a user to input the data. Most dialog boxes typically have two selectable buttons. One button labelled "OK" is selected by a user to exit the dialog box when he or she has finished entering the data. The second button "CANCEL" allows the user to exit the dialog box without entering the data.

Table 3 below describes each of the menu options and the content of the dialog box presented to a user, i.e. test production staff, after invoking the menu option in a preferred embodiment. It is well known in menu-driven applications to layer the menus. For instance, IPT provides eight main menu options in capital letters in Table 3 below, e.g. FILE, VIEW, PRESENTATION, etc. One or more lower level menus may be invoked when one of these main menu options is selected by the user. For instance, the menu selection listed in Table 3 such as "PRESENTATION/Components/Directions" indicates that the user had selected the main menu option PRESENTATION. A first lower level menu was then provided, and the user selected "Components." Subsequently, a second lower level menu was provided, and the user selected the "Directions" option.
TABLE 3
______________________________________
MENU STRUCTURE AND DIALOG BOX CONTENT
MENU OPTION DIALOG BOX CONTENT
______________________________________
FILE/Open This dialog box allows the user to either
type in the accession number of an item or
select the accession number from a list.
The user may also be permitted to change
drives and directories. When an item is
opened, it is read into memory, formatted
and then displayed. It is displayed using
the current values found in the control
file. If critical information is
missing from the control file, default
values may be used. If an item is
currently being displayed when
FILE/open on a new item is
invoked, the current item may be visually
checked for integrity and the user will be
prompted whether or not to save the item.
FILE/Save The currently displayed item is saved.
Preferably no dialog box or buttons are
displayed.
FILE/Integrity
The currently displayed item is checked for
check integrity. If anything is wrong with the item,
the user is informed. The user may then
correct the problem(s). Preferably no dialog
box is displayed unless a warning message
is shown to the user. This warning box
may contain an OK button.
FILE/Print The currently displayed screen "item" is
Screen printed. Preferably no dialog box or buttons
are shown.
FILE/Exit The application is terminated. The currently
displayed item will be checked for integrity.
If the currently displayed item has not been
saved, the user will be prompted to save
it. A confirmation box may be displayed.
FILE/About Software ID, version number and copyright
information about the IPT are displayed.
An OK button is preferably displayed.
VIEW/Directions
The text of the directions associated with the
currently displayed item is shown. An OK
button is preferably displayed.
VIEW/Summary
A summary of the currently displayed item,
including all component accession IDs, the
item's presentation on the display, response
information, etc. are shown. An OK button
is preferably displayed.
PRESENTATION/
The user is permitted to select a template
Template with one or more presentation panes for the
currently displayed item, including the pane
in which each component will be displayed
and the ordering of components within
each pane.
PRESENTATION/
The user is permitted to associate a file
Components/ name containing the text of the directions
Directions to the item.
PRESENTATION/
The user is permitted to associate a file
Components/ name containing the text of the stimulus
stimulus to the item.
PRESENTATION/
The user is permitted to type the text of the
Paraphrase paraphrase. The paraphrase is used on the
review screen of the test delivery
application.
PRESENTATION/
The user is permitted to specify an area of a
Positioning reading passage that will be centered and
highlighted when an examinee is presented
with the currently displayed item. The
beginning and ending character positions
to be centered and highlighted are specified
in this dialog box.
RESPONSE The user is permitted to select the response
class and type. All additional response
parameters that may be required to describe
that response class and type can be specified
by clicking on the parameters button.
Single-select
A parameter is preferably available
Multiple Choice
to permit the user to choose whether
or not to display an indicator with
each choice.
Multiple-select
A parameter is preferably available to
Multiple Choice
permit the user to choose whether or not
to display an indicator with each choice.
Scale The user specifies the shape of the scale, the
width (or height or radius depending on the
shape of the scale), label orientation, labels,
and tic mark sizes. The defined scales may
include, for example, horizontal and vertical
time lines, circles and semi-circles.
KEYS The user is permitted to specify the answer
key(s). A list of possible keys may be
displayed so that the user selects the choices
for the key(s).
SCORE The user is permitted to specify whether or
not the item is scored.
REPRESS It performs the same function as File Open
using the accession number of the currently
displayed item. This is available so that
when changes are made to the item outside
of the IPT (i.e. WORD), the user can easily
reread and display it. Preferably no dialog
box or buttons are displayed.
SELF It provides help in using the IPT. The type
of help available includes how to use
the various dialog boxes, system menus, and
general item preparation procedures.
______________________________________

Detailed flowcharts and corresponding pseudo code of the IPT application is provided in Appendix A. However, the following example will be provided for a more complete understanding of the use of the item preparation tool.

FIG. 10 shows a dialog box 230 generated by the Item Preparation Tool when a user selects by clicking the menu option FILE/Open. The FILE menu option then lists several other menu options including an "Open" option. The user then selects "Open" and the dialog box 230 as shown in FIG. 9 is opened and prompts for the opening of a particular item.

"Open:" 228 shows which item is selected to be opened. "Path:" 229 shows where on the hard drive these items are stored. "Files" 231 provide a selectable list of all the available items in that directory. "Directories" 233 provide a selectable list of file directories on the hard drive. The "Open" 223 and "Cancel" 225 buttons may be executed after the appropriate item is selected or to exit the "Open" option.

Together FIGS. 11 and 12 present a screen-print of a simple reading passage. The top line 232 in FIG. 11 reveals the name of the item identified by the item accession number. The second line 234 is a menu bar from which item preparation functions can be initiated to construct the item identified in line 232. The left-hand box is a pane and contains the reading passage 236. The reading passage 236 is the stimulus and is contained in the reference file for the item, MH000001.REF. The right-hand pane contains the directions 238 and the stem 240 which are stored in separate files, but designed to be displayed in the same pane. The stem 240 in this instance indicates that the answer should be entered by interacting with the reference file or reading passage 236. By inserting specific types of custom codes referred to as interaction codes into the text of the reference file, an examinee can respond by selecting a sentence in the reference file. The sentence will become highlighted on the screen as shown in FIG. 13 after it has been selected. Line reference numbers 242 are found directly to the left of the text of the passage. They are preferably included in graphic files and are not actual text.

A reading passage is often longer than the screen size allows. FIG. 12 shows the scrolling feature of the Item Preparation tool that allows the user to move through the passage by using the scroll mechanism 244 lying between the panes 246 and 248. The use and implementation of a scrolling feature are well known.

FIG. 14 shows a dialog box 250 that prompts for response parameters after invoking the RESPONSE menu option. These parameters set up the nature and functions of the response area. The "Number of Req. Responses;" field 247 indicates how many responses are necessary in order to answer the item. The "Response Class" box 249 indicates the general category of the response, i.e., single choice, multiple choice, or free response. The "Response Type" box 251 indicates the specific form of the response. A description of the different response types is provided in Section III below.

Further demarcation of the item is accomplished by entering information in a series of nested dialog boxes as shown in FIG. 15. For instance, if "Multiple Choice" box 239 is selected from the "Response Type" box 251, then a "Multiple Choice" box 252 is opened. The "Number of Choices" 241 may be computed by the Item Preparation Tool. For example, in FIG. 15, the "Number of Choices" 241 is set at eight. This number is based upon the amount of specific interaction codes included in the response area. This number is important in error checking for codes because it should reflect the exact number of available responses available, i.e., whether the number is less than the intended number of responses. The Item Preparation Tool may indicate an error in coding. "Block Set" 243 specifies which set of interaction codes are to be read by the software for a specific item. "Indicator" 245 prompts for different response designs. Examples of indicators are also described in Section III below. "Invert Choices" 237 indicates whether the response option should be highlighted by reverse video when selected. The component 253 permits test production staff to enter which of the options is the correct choice. It should be understood that different response types require different parameters than the example shown in FIGS. 12 and 13. The data entered by test production staff in the dialog boxes shown in FIGS. 14 and 15 are stored in the item control file.

A different item is shown in FIG. 16, although the same reading passage 236 in the reference file is used. In this instance, part of the reading passage 236 is highlighted in reverse video 257 when the item is presented. Note again the directions 254, stem 256, and response 258 are included in one pane. However, different response parameters have been set for this item, i.e., there are four options instead of eight. Interaction is set to the response file and not to the reference file, ellipses are included rather than invert choices. FIG. 17 shows the selection of a response option 255 from the responses 258.

FIG. 18 shows yet a third item which refers to the same reading passage 236. The stem 259 asks for a word to be selected in the second paragraph. Although the item parameter dialogue box is not shown, item parameters are set up in a third way so that there are 42 options (as many words as in the second paragraph). Interaction has been switched back to the reading passage 236 or reference file, and the choices selected by an examinee will be inverted. FIG. 19 shows the correct answer at 261 after being selected.

e. Word Processing

In a preferred embodiment, the word processor is "WORD FOR WINDOWS," which is available from Microsoft Corporation. It is used by test production staff to edit component files produced by the IEG or to create completely new items. Component files are edited for two purposes. The first purpose is to effect the appearance of the item text by adding fonts, point size, bolding, etc. The second purpose is to insert "Custom Codes" before and after sections of text.

Component files are initially stored in ASCII by the IEG as described above, but they are converted to Microsoft's RTF format when they are saved in "WORD FOR WINDOWS." Even though "WORD FOR WINDOWS" does the conversions, the test production staff is responsible for selecting the correct format. After editing, text is written in RTF format back to the component file from which it came. Thus, a component file may contain either ASCII or RTF formatted data, depending upon whether the file has been edited by "WORD FOR WINDOWS."

Now turning to the second reason for editing a component file, namely to add Custom Codes, literal strings are inserted into the text and thus allow computerized features to be added to the test. In a preferred embodiment, Custom Codes always start with a ".linevert split.."

Custom Codes belong to one of three classes: 1) codes that stand alone, 2) codes followed by a parameter, and 3) codes followed by additional data. "Stand alone" codes appear in the text by themselves. Their very presence is all the information conveyed by the code. "Parameterized" codes are distinguished from other classes of codes in that they are followed by a parameter enclosed in square brackets ("[" and "]"). The parameter immediately follows the code without any intervening characters or white space. "Data" codes are followed by other data. The data is arbitrary text. To prevent conflicts with parameterized codes, a single white space character is used to separate the code from the user-supplied data.

Table 4 below summarizes some examples of Custom Codes and their use. Optional elements of parameters are enclosed in curly braces ("{" and "}"). These codes are used to include a graphic in the component. The parameter "nn" is used to form part of the graphic file extension. Graphic files are named accession. Gnn, where accession is the same as the base name of the file in which the graphic code appears. Thus, for example, if the graphic appears in a stimulus component whose name is TD-00081.REF, the graphic file name may be TD-00081.Gnn.
TABLE 4
______________________________________
CUSTOM CODES
Long/Short Name Use
______________________________________
.linevert split.ACCESSION
Provides an accession number
.linevert split.ACC[nnnnnnnn]
for the item.
.linevert split.BLOCKSTART
Marks the beginning of the
.linevert split.BKS
item's distractor for a
multiple choice item.
.linevert split.BLOCKEND
Marks the end of the
.linevert split.BKE
distractor.
.linevert split.COMMENTS
Adds a comment to the
.linevert split.COM . . . .
component to provide a way
for TPS to annotate a
component.
.linevert split.END
This Code indicates the end of
the accession component.
.linevert split.GRAPHICRIGHT[nn]
This code indicates the graphic
.linevert split.GRR [nn]
is to appear on the right side
of the component text. Text
will flow around the graphic.
The graphic base line will be
aligned to the text base line.
.linevert split.GRAPHICLEFT[nn]
This code indicates the graphic
.linevert split.GRL [nn]
is to appear on the left side
of the component text. Text
will flow around the graphic.
The graphic base line will be
aligned to the text base line.
.linevert split.GRAPHICUP[nn]
This code indicates that the
.linevert split.GRU [nn]
graphic is to appear in line
with graphic base line aligned
to the text base line. The
graphic is positioned
horizontally as though it were
a giant character.
.linevert split.GRAPHICDOWN[nn]
This code indicates that the
.linevert split.GRD [nn]
graphic is to appear in line
with the graphic top aligned to
the text base line. The
graphic is positioned
horizontally as though it were
a giant character.
.linevert split.GRAPHICPLACED[nn{-}h]
This code indicates that the
.linevert split.GRP[nn{-}h]
graphic is to appear in line
with the graphic base line
displaced above or below the
text base line according to the
parameter "h," which specifies
the displacement in pixels.
.linevert split.HORZLINE
This code draws a horizontal
.linevert split.HOR
line across the component
window. The code should
appear on a line by itself--with
hard returns preceding and
following it. Otherwise,
the line will appear to
cut through the text
that follows.
.linevert split.PLACERESPONSE
This code marks the position
.linevert split.PRS
within a component, generally
the response component, to
place the response object.
______________________________________

FIG. 20 is an example of the actual reference file in "WORD FOR WINDOWS" where it is manipulated and replete with formatting and interaction codes. The control of the reference file shown in FIG. 20 is based on the sample items shown in FIGS. 10 through 17. The name of the file is included in the first line of text. In this instance, ".linevert split.ACCESSION[MH000001]" is the file name. The next custom code on the first line is an interaction code, ".linevert split.PMC" which is a response code and indicates a "Place Multiple Choice" to be included in the passage. Since this code is not item specific, it can be used once and be referenced by any number of items (in this instance, by items 1 and 3). The next element, ".linevert split.GRL[01]," is a code that calls in a graphical image which exists in a separate file. This code indicates that at this point in the file, a graphic should be placed to the left margin before any more text is included. The graphic in this instance is the line reference numbers 5 through 20. The graphical numbers 25 through 30 lay within a separate graphical file, included after the word "us" in the third paragraph of the text of the passage.

One can find the ".linevert split.GRL[02]" code in FIG. 21 in the fourth line of text. The small arrow 260 after the graphic code is a "WORD FOR WINDOWS" formatting command. It is a tab marker which specifies the paragraph indent.

The next interaction code, ".linevert split.BKS[1]," is a "Block Start" code; it will be followed by a ".linevert split.BKE[1]," "Block End" code. These codes set the boundary around a specified option for a specific item. At this point, it is helpful to refer back to FIG. 15 and note that a "Block Set" number is identified for each item. Where there is more than one item referring to the same area of text, separate Block codes may be included. In the passage there are also block codes ".linevert split.BKS[2]" and ".linevert split.BKE[2]" indicating option boundaries for selecting a response in the third example item as shown in FIG. 18 that uses the same area of text. Thus, only those portions of the reading passage 236 which are "blocked" by interaction codes which are related to a specified item will be active when that item is presented.

In the fourth and seventh lines of text in FIG. 18 there is a .linevert split.HCS[2]-.linevert split.HCE[2] code set, indicating a highlight and center. This interaction code highlights the demarcated area of the reading passage 236 and centers it when the item is presented by the item preparation tool, as shown in FIG. 13.

Finally, in FIG. 21, there is a ".linevert split.HOR" code that produces a horizontal line at this point in the text when read by the item preparation tool and an ".linevert split.END" code, indicating the end of the field of custom codes.

f. Translation

XLATE is one of the item preparation programs listed in Table 1 and compiles RTF format documents created by "WORD" into a binary equivalent. Binary conversion speeds up the execution of the text display modules embedded in the Item Preparation Tool and also results in a storage savings (mostly memory savings).

XLATE is a system program which is generally not executed by test production staff from a command line, menu pick, icon, etc. The Item Preparation Tool runs XLATE whenever it is determined that a component file has been changed within "WORD FOR WINDOWS." This is detected by comparing the date/time stamp of the .STE, .RSP, and .DIR files of the item currently displayed by Item Preparation Tool with that of their binary equivalent the .STB, .RSB, and .DIB files. The binary file name is created from the component file name extension by changing the last letter of the extension to "B." For example, if the stem component's name is accession. STE, the binary file outputted by XLATE will be accession. STB. If the binary file is older, its source equivalent must have been edited while the user was in the "WORD FOR WINDOWS". Thus, The Item Preparation Tool runs XLATE to update the binary file by translating the source code.

During conversion, errors are preferably written to an error log file in ASCII format. The error file name is created from the template XLATE???.ERR by substituting the component file extension for the question marks. Thus, for example, if the component file is named accession. STE, the error file name will be XLATESTE.ERR.

g. Data Interfaces and Flow between Test

Development and Test Production

FIG. 22 presents a flow diagram of items and keys between TD 650 and TPS 660. As previously stated in Section II.B., the test developers create and select items for a particular test preferably using the "TD/DC" system. However, test items may be created and selected by any test document creation system or prepared by hand as long as TPS 660 is provided with the information described in section II.B.2.a. related to item offload. Three methods of providing the information to TPS 660 are shown in FIG. 22.

The three methods are enumerated in FIG. 22, by the numerals 1, 2 and 3 located along the path lines. The use of the "TD/DC" System 652 is enumerated as path 1. If the key descriptions are prepared by a method other than "TD/DC" 652, they may be written on paper and provided to TPS 660 via path 2. Additionally, the item text, presentation information and classification information which are collectively referred to as the item description, may also be determined by test developers who do not use the "TD/DC" system, but prepare this information on paper as shown by path 3.

Test developers create and select the items to be included in a test as shown at 650. If the test developers use the "TD/DC" system, they execute item Offload to produce a diskette having the files containing the item description and respective keys as shown at 652 and 658. The diskette is then sent to TPS as shown at 660. If the files containing the key information are not generated by the test developers using the "TD/DC" system, but rather by another method, the test developers may provide a written key description to TPS 660 as shown at 654. Similarly, test developers may prepare a written form of the item description shown at 656 and provide the written description to TPS at 660. If graphics are included in the written description at 656, a CBT artist will prepare computerized graphics files at 662 and 664. These item graphics component files are also provided to TPS as shown at 660.

If the item and key description are provided to TPS 660 via diskette created during item offload, TPS invokes the item element generator at 666. The component files are separated as described above and filed in the an Item Preparation (IP) database at 670. If the item description or key description is presented to TPS 660 on paper, TPS must manually enter the information using the IPT and word processing at 668. Then, the component files created by TPS 660 are stored in the IP database 670. Once all of the necessary component files are stored in the IP database 670, TPS uses the IPT (Item Preparation Tool) and word processing to prepare the computerized version of each item at 668. Component files may be replaced in the IP database after being edited as shown at 670 and further processed at 668 using IPT or word processing.

When TPS is satisfied with the computerized version of the item, the test developers may view the items as they will be presented to an examinee as shown at 672 and 674 using the IVT. If the test developers desire changes to the items as presented, they can provide revision information to TPS Staff via paths 2 or 3 and the whole cycle may be repeated until items are completed.

3. Test Preparation

a. Overview

FIG. 23 shows the functions performed by TD and TPS and the software which is used to perform these functions for preparing a computerized test. Test developers assemble the test as shown at 682. As shown at 686, item selection is preferably automated (AIS) using the "TD/DC" system or an equivalent test document creation system. Using "TD/DC", test developers enter the test specifications into the "TD/DC" system. Based on these specifications, "TD/DC" searches its central database for items which satisfy the test specification, e.g., 50 math questions, 25 of which are algebra problems and 25 which are geometry problems. Then, the test developers review the items selected by "TD/DC" for sensitivity and overlap constraints described in the background section. If the test developer decides that the sensitivity or overlap constraints are not satisfied by the current selection of items, certain items may be designated to be replaced by another item from the database. In addition, test developers provide a test description specifying the directions, messages, timing of sections, number of sections of the test, etc. as shown at 692. If a computer adaptive test (CAT) is to be run, test developers may run a computer adaptive test simulation at 684 which are known to skilled test developers.

Using the Test Preparation Tool (TPT) and TOOLBOOK 696, TPS prepares the test level components as shown at 700. TOOLBOOK is commercially available from Asymetrix Corporation. The test level components include scripts 716, item table block sets 706, general information screens 708, direction screens 710, message screens 712, and tutorial units 714. Each of the test components will be described in detail below.

As the components are prepared, the TPT stores them in a TPS network directory 702. Then, the components are entered into the TPS Production database 704. The components stored in the TPS Production database 704 will be retrieved during test packaging which is described below.

A script consists of a series of files and further specifies the option settings and configuration data which the Test Delivery Application (TDA) needs for operation. Option settings are specified for the Test Delivery System to determine whether a certain feature is enabled for the test. Most option settings are simple yes/no declarations, but some offer a limited set of choices (e.g., mouse speed: slow, medium, fast). Configuration data is highly variable information such as the section name to be displayed in the Title Line during a test session or the list of items to be displayed.

b. Scripts

During test preparation, scripts 716 are prepared and combined with the items prepared during item preparation. Scripts control the sequence of events during a testing session. Two types of scripts 716 are preferably used: the session script 718 and one or more test scripts 720. The session script 718 controls the order in which units within the testing session are presented. Units provide specific services to the examinee, such as delivering a test or presenting a score report. Just as the session script controls the session, the test script controls what is presented to the examinee during the testing unit. Each testing unit may include one or more delivery units, which are separately timed and scored subdivisions of a test. The system can dynamically select, or spiral, scripts and other test components so that examinees are given what appear to be different tests. FIG. 24 shows the relationship among session scripts 718, test scripts 720, and units.

Some examples of units supported by the system are described in Table 5 below:
TABLE 5
______________________________________
Unit Description
______________________________________
General The general information screen unit
Information Screen
allows the incorporation of a single
(GIS) Unit screen of information in the session.
The screen is used to display
information that the examinee does
not interact with, such as copyright
notices, rules of conduct, or pauses.
Multiple GIS units can occur within a
session.
Tutorial Unit The tutorial unit presents test
familiarization materials to the
examinee. Multiple tutorial units
can occur within a session.
Break Unit The break unit is used to implement a
scheduled break within a session.
Multiple break units can occur within
the session.
Testing Unit The testing unit presents a test. It
is controlled by a test script. Any
of the other units, except a scoring
& reporting unit or another testing
unit, can be embedded within the test
script. This permits a testing
program to organize a test as several
sections, with tutorials, breaks, and
general information screens between
them. In addition, a special type of
unit, a delivery unit, can appear
only within a test script. The
delivery unit is equivalent to a test
or section of a test and presents
items to examinees. Multiple testing
units can occur within a session.
Examinee Data The data collection unit is used to
Collection Unit
obtain additional information from
the examinee, such as demographic or
debriefing information. This unit is
basically a special type of testing
unit which is not scored. Multiple
data collection units can occur
within a session.
Scoring & The scoring & reporting unit provides
Reporting Unit
for field scoring of one or more
testing units delivered in a session.
It must follow all testing units that
are to be scored. Only one scoring
reporting unit can be included in a
session, although it can be omitted
if field scoring is not required.
______________________________________

Testing programs control the behavior and appearance of their tests through options that are enforced by the Test Delivery Application. Table 6 below indicates the options that are available and the levels at which a testing program can specify each option. The options, which are explained in the text following the table, are selectable at one or more of the following levels:
test package (i.e., all instances of a particular test, such as GRE General),
session,
testing unit,
delivery unit (section).
TABLE 6
______________________________________
Test Testing
Deliv.
Program Option
Package Session Unit Unit
______________________________________
Primary Controls
Y Y Y Y
Explicit Prompting
Y Y Y Y
Must Answer Y Y Y Y
Timing Y Y Y Y
Threshold Interval
Y Y Y Y
Last Threshold
Y Y Y Y
Disable Time Y Y Y Y
Count During/After
Y Y Y Y
Directions
Delivery Mode Y Y Y Y
Score Unit Y Y Y Y
Examinee Quit Restart
Y Y Y Y
Display Scores
Y Y Y Y
Cancel Scores Y Y Y Y
______________________________________

At the package level, options that are in effect for the entire package are defined in a Package Profile file. Some examples include the program name, which may appear on the title line of the test. Another option may indicate whether the program permits administrators to restart sessions after an examiner terminates the test. Additionally, options may list the session scripts to deliver the test.

The session script is the second-level component of the testing package. It performs two primary functions: First, it specifies the Session Control Information, which defines the default options that are in effect for the entire examinee testing session. Second, it controls the order in which units within the testing session are presented and the options used to present them. The units that can be presented within a session script are: General information screen units, Tutorial units, Break units, Data collection units, Scoring and reporting units, and Testing units.

The session control information contains the default options in effect for the entire session. Control information can be provided at multiple levels within the testing session. Thus, the control information provided at the session level can be overridden by information that occurs later in the session. The information provided at the session level would generally include the following: Name--the session script name to be used by administrators in selecting a specific session script from Administrative Application menus; Input device--the input device to be used during the session (e.g., mouse or keyboard); Color--the colors to be used during the session; Messages--program-specific messages to override default messages during the session; Demo Script--indicates whether the script presents a demonstration or operational test; Research Indicator--indicates whether the script presents a research pilot test; Special Timing--indicates whether the script is standard or specially timed version.

The GIS unit allows the incorporation of a single screen of information in the session. It may contain, for example, the following information: reference to the actual text and graphics that will be presented on the examinee's screen; the type of dismissal: automatic or manual; the time after which dismissal should occur.

The tutorial unit presents test familiarization materials to the examinee. It may contain, for example, the following information: reference to a tutorial, which is the familiarization information that will be presented on the examinee's screen, e.g., "How to Use a Mouse", "How to Scroll", "How to Use the Testing Tools", "How to Answer"; information that controls the content of the tutorial--the information varies depending on the tutorial selected. For example, for the "How to Use the Testing Tools" and "How to Answer" tutorials, the specific tools and item types to be covered must be defined. The mouse and scrolling tutorials are generic--no content information is required when those tutorials are selected; an indicator for first occurrence--this indicator applies only to the Testing Tools and How to Answer tutorials. Introductory information is presented during the first occurrence of these tutorials and is omitted if they occur again later in the session.

The break unit is used to implement a scheduled break within a session. It contains the following information: reference to a break procedure, which is the actual text and graphics that will be presented on the examinee's screen during a break, along with the program that controls their presentation. It should be understood that multiple procedures could be supported; the length of the break.

The data collection unit is used to obtain additional information from the examinee, such as demographic or debriefing information. It is preferably implemented as a special instance of the testing unit, in which no scoring is done. Like the testing unit, a data collection unit references a test script, which control as the sequence and options of the unit.

The scoring and reporting unit provides for scoring, and optionally reporting, the results of one or more testing units delivered in a session. If the testing program selects a Display Scores option, it preferably displays all traditional score types including raw, percent correct, converted and composite scores. If testing programs select the Cancel Scores option, the examinee will be given the option of cancelling the scores.

The scoring and reporting unit preferably invokes the Educational Testing Services SKM (Scoring and Key Management) routines to return the following information: the score name for insertion into the score report, such as "Reading" or "Antonyms"; the score type for insertion into the score report, such as "number right," "percentile," or "converted score"; the score value, such as "650" or "passed". It should be understood that any automated scoring system which provides this information can be used or the information may be provided directly by a user.

The information needed to display a score report is preferably identical to that required for a message screen: reference to the actual text and graphics to be presented on the examinee's screen.

The testing unit presents a test, based on the contents of a test script that may have been selected at runtime. The following units can be included within a testing unit: general information screen unit; tutorial unit; break unit; delivery unit, which delivers items to the examinee. This permits testing programs to interleave general information screens, tutorials, and breaks with sections of a test. The testing unit contains the following information: script selection mode indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used, the reference is to a set of test scripts.

Like the session script, the test script performs two primary functions. First, it specifies the test and delivery unit control information. Test control information defines the options that are in effect for the testing unit. Delivery unit control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the order in which units are presented within the testing unit and the options used to present them. The rules for presentation of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included within a test script.

At least three delivery modes are preferably supported within one or more testing units: linear, adaptive, and essay. The test script references different components depending on the delivery mode of the test, but in all cases the end result is a reference to a specific item or essay topic to present to the examinee. Multiple delivery units can be used to organize the testing unit into sections, and each delivery unit can present a different mode of test.

The test control information includes the following information, which is in effect for the testing unit: Logical Name--the name used to associate the testing unit with a scoring specification provided by the SKM system or an equivalent thereof; Directions--a reference to the text and/or graphics to be presented as general directions; Sections--the number of sections within the test; create score data--whether scoring data for online scoring is to be created for this testing unit; Message Overrides--any program-specific messages that are to replace the default messages within the testing unit. The test control information can temporarily override the options specified in the session control information.

In addition to the test control information, a test script can also contain delivery unit control information. The delivery unit control information can be specified to change the options in effect for the duration of the section. The following information is preferably included in the delivery unit control information:
Type Indicator--whether this delivery unit delivers a test or a section; Directions--if the delivery unit is a section, reference to the test and graphics of the section directions; Title Line Test--defines the name that will be used in the title line;
Mode--the delivery mode of the test or section, e.g., linear, adaptive, or essay;
Timing--timing options in effect for the test or section, including whether the test is untimed, timed in its entirety, or timed by section, and whether timing begins when directions for the test or section are displayed or when the first screen after the directions is displayed;
Scoring--whether scoring data for online scoring is to be created for the delivery unit;
Messages--program-specific messages to be used to replace the defaults during the test or section;
Item Information--specifies how the times in the test are organized and where to find them. The contents of the item information varies with the mode of the test being offered. For example, the item information for linear tests is generally a reference to an item-by-item listing of the items to be presented, called an item table. Essay tests may reference an essay procedure and a topic pool if more than one essay topic is provided for selection by the examinee. Whether more than one essay is to be stored or only one is chosen for scoring is also defined. For adaptive tests, reference to an adaptive algorithm and an item pool should be specified.
Testing Tools--the testing tools available during the test or section. Examples of testing tools are described in detail in Section III(E);
Explicit Prompting--whether or not explicit prompting is to be used to makesure the examinee supplies exactly the required number of responses;
Must Answer--whether or not examinees should be allowed to move off an item without providing a response.

Detailed flowcharts and corresponding pseudo code for the TPT application are provided in Appendix B.

4. Test Packaging

After all of the items have been constructed for computer delivery by the test production staff and approved by test developers and the scripts and tutorials have been created, the test production staff packages all of the relevant files together using the Test Packaging Tool (TPAK) and the Score Key Management (SKM) system. In a preferred embodiment, this process requires three steps. First, the test components are combined into a draft test package so that the flow and presentation of the test may be reviewed. After the draft test package has been reviewed, the test components are formed into a blue-line test package. After the blue-line test package undergoes successful quality assurance tests, locks are applied so that data cannot be altered in the approved test packages. The final tests are then distributed to test centers.

A flowchart depicting the steps executed by TPAK and SKM to package a computerized test is shown in FIG. 25. First, TPAK is used to create a delivery package at 601. This step involves creating a presentation database which incorporates the presentation information from the test scripts, creating a presentation parts lists which lists an ID for each component used to create the presentation database and creating other files subsequently used by SKM. An SKM database is created at 603 from the files generated by TPAK. These files preferably contain the item table described above and item scoring information. The SKM database and the presentation database are then combined by TPAK at 605 to produce installation files for distribution. After all quality assurance procedures have been performed and satisfied, TPAK preferably applies a lock on all of the installation files as depicted at 607. After this lock is applied a new test version should be created if changes to the test package are required.

FIG. 26 shows the components of a final test package. Four primary groups of files are packaged together to form the final test package. These four groups of files are the Profile and Index files 801, the Presentation Binary Large Object (BLOB) 802, the SKM BLOB 803, and the Problem Item Notification (PIN BLOB) 804.

Although the SKM BLOB 803 and PIN BLOB 804 are shown in FIG. 26, they are not necessary components of the test package. However, they are preferably included in the test package. The SKM BLOB 803 should be included if the examinee responses are to be scored at the testing center after the examinee has completed the test. If the SKM BLOB 803 is not included in the test package, the examinee responses are scored at the central processing site by a program specific (e.g., SAT, GRE) postprocessing system. The PIN BLOB 804 is used to identify items which had been included in the test but which are later determined either not to be scored or administered to the examinee. Thus, although the PIN designations provide a preferable feature, its inclusion in the final test package is not necessary. The details of the SKM and PIN BLOBs will be described below.

The profile and index files 801 include the Package Parts List File (PPL) 805, the package profile file (PP) 806, and the BLOB Index Files 807. The PPL 805 contains a list of identification codes and version numbers, each ID code and version number being associated with a component in the test package. The PP file 806 contains an identification of each test included in the test package. Although only one test is actually included in a test package, multiple scripts may be provided for dynamic selection (spiralling) or for special conditions (e.g., untimed versions for examinees with disabilities). Spiralling is a technique in which test components (e.g., item substitution) are varied so that examinees taking the same test appear to be taking different tests. Spiralling inhibits cheating among examinees whose workstations are in close physical proximity because it increases the likelihood that each examinee will be interacting with a visibly different test. The BLOB index files 807 provide an index for the Presentation BLOB 802, SKM BLOB 803, and PIN BLOB 804 which are packaged into the computerized test. These indices function as a guide for locating specific data within each of these BLOBs.

The presentation BLOB 802 includes item level components 808 and test level components 809. Examples of the item level components 808 are shown in FIG. 27 while examples of the test level components are shown in FIG. 27. Referring to FIG. 26, the item level components 808 may include the item stimulus 820, the item directions 821, the item stem 822, the item response 823, and the item graphics 824. Each of these item level components has been described in detail in conjunction with item preparation in Section II.B.

Referring to FIG. 27, examples of the test level components 809 are the test scripts 830, general information screens 831, test level directions 832, message screens 833, and tutorial units 834. Each of the test level components 809 has been described in detail in conjunction with test preparation or will be described in detail in conjunction with test delivery in Section III.

Returning to FIG. 27, The SKM BLOB 803 is preferably created by the scoring and key management application and incorporated into the final test package. The SKM BLOB 803 preferably includes the Answer Keys 810, scoring tables 811, and scoring specifications 812. The answer keys 810 provide the correct response or responses to each item included in the test form. The scoring specification 812 provides information relating to how each item is to be scored.

FIG. 29 shows some examples of scoring tables which may be included in the SKM BLOB 803. The item and custom item tables 840, the item and custom item blocks 841, conversion tables 842, Item Response Theory (IRT) parameters 843, Theta estimation parameter tables 844, K-factor tables 845, and Item weight tables 846 are preferably included in the SKM BLOB 803. Since all of these tables are well known to those in the testing industry, a description of these tables will not be provided here.

Returning again to FIG. 26, the PIN BLOB 804 includes pointers to pinned items 813 and the Do Not Score (DNS) and Do Not Administer (DNA) flags 814. Some time after the computerized test has been prepared by TPS, it may be determined that one or more items included in the test do not perform as intended, e.g., more than one response would be correct although only one was intended. Thus, TD may designate these items and indicate whether the item should not be scored or not be administered. SKM then creates the PIN BLOB 804 with pointers and flags for each of the items that has been found to misperform. When the test delivery system delivers the computerized test, it will not present items flagged with a DNA and those items flagged with a DNS will not be scored.

FIG. 30 shows the functions performed by TD, TPS, and SKM staff toimplement the steps described above with reference to FIG. 25. Turning now to FIG. 30, as a part of test preparation, TD specifies the scoring specifications and conversion tables at 776. In a preferred embodiment, SKM staff utilizes the automated SKM system developed by Educational Testing Service as shown in FIG. 30. For instance, if the ETS SKM system or an equivalent automated SKM system is utilized, the answer keys and tables may be retrieved from an SKM database at 770 and 756. The scoring specifications and conversion tables are added to the data retrieved from the SKM database as shown at 758, and an SKM BLOB 762 can then be created. TPS combines the test components, item components and BLOBs to create a draft test package at 736 and 738 respectively. TD reviews the draft test package at 772 as it is delivered by the Test Delivery Application at 778. If certain items do not perform as they should, TD identifies those items for the SKM staff to create a PIN BLOB 764 based on the information provided by TD. TPS adds the PIN BLOB to the test package and makes any other revisions the TD identified after reviewing the draft test package. When TD has authorized the test package at 780, TPS prepares a blue line test package 740 and sets a level 2 lock on the test and item level components. A level 2 lock prevents modification to the locked components by unauthorized persons. After the blue line test has been finally authorized by TD at 780, TPS creates a set of data distribution disks at 742 and applies a level 3 lock at 748. A level 3 lock virtually eliminates the potential for any changes to be made to the computerized test.

Like the IPT and the TPT, TPAK is preferably a menu-driven application. Detailed flowcharts and corresponding pseudo code of the TPAK application are provided in Appendix C.

In a preferred embodiment, a modified version of TPAK called ETPAK (Encrypted TPAK) is executed. In addition to verifying the presence of each of the required test files, ETPAK encrypts at least the item files (e.g., .STE, .REF, .RSP, and .CTL). After the required files have been packaged together, they may be transferred to a test center at which the computerized test is administered and delivered to an examinee.

III. The Test Delivery System

A. Overview

A block diagram of the Test Delivery System 12 is shown in FIG. 31. A test delivery application (TDA) 510 controls the test session, as directed by the test program 514, CBT files 516, and test delivery application data (TDA data) 512. The test program 514 and CBT files 516 are administration system files and are preferably stored on the work station or server (if workstations are networked via local area network) hard disk prior to delivery of a computerized test to an examinee. Other files and applications such as the HELP facility 526 and the REVIEW facility 528 are also preferably stored on the hard disk in advance. An examinee performance file 522 is created during each test session to record an examinee's responses and other activity during the test session.

Although the Administration system will be described in detail below, a brief description is provided here as it is relevant to the implementation of the Test Delivery system.

The center administrator uses a combination of manual and computer procedures to control operations and deliver tests to examinees at the testing center. All of the files and applications shown in FIG. 31 are sent to the testing center in electronic form. All of these files can be loaded, on the workstations using standard set up procedures. The administrative application 511 of the Administrative system permits the administrator to initialize each workstation at the start of the day, to sign an examinee onto a workstation, to start a testing session, and to close each station at the end of the day. The center administrator is also responsible for performing backup procedures and transmitting the examinee performance files 522 to the central processing site for scoring and evaluation.

FIG. 32 provides a high level flow diagram of the test delivery procedure. In a preferred embodiment, when the administrator completes the procedure to sign an examinee onto a workstation and selects a test at 500, the Test Delivery Application is invoked. The Test Delivery Application reads the session script and executes the units it prescribes. When the end of the session script is reached, the Test Delivery Application returns control to the Administrative Application.

B. Test Delivery Application Data Flow

Scripts define the sequence of tasks to be performed by the TDA 510 as well as the information necessary to complete each task. The scripts define option settings, files containing program-specific text, and the items to be displayed.

Although the following section provides a detailed description of the operation and use of the Test Delivery System and the presentation of various screens by the Test Delivery system, detailed flowcharts and corresponding pseudo code of the TDA are provided in Appendix D.

C. Title Line

The screen format during a delivery unit is preferably divided into three main sections; a title line 2250, an item presentation area 2252, and a primary control area 2254 as illustrated in FIG. 33. The title line 2250 is preferably presented as one solid gray bar at the top of the screen. It should be understood that numerous other color combinations are possible. The title line 2250 is capable of displaying various information relating to the test or taking the test. For instance, the title line 2250 may include the time remaining in the test or test section. Preferably, time is displayed automatically although an examinee may optionally turn it off. In a preferred embodiment, remaining time is displayed left justified on the title line 2250 in HH:MM format until the last few minutes of the test. At that point, the display format changes to MM:SS and flashes for three seconds so that the examinee is alerted that the time remaining for taking the test is nearly over.

Other information in the title line 2250 may include the name of the computerized test and program-specified text pertinent to what is being presented (e.g., section name). Information to help orient the examinee is also preferably displayed in title line 2250. For instance, when an item or tutorial screen is displayed in the presentation area 2252, the notation "xx of yy" or "xx" appears in title line 2250. The "xx" refers to the item number within the test or section, or the screen within the tutorial. The "yy" indicates the total number of items in the test or section, or screens within the tutorial. Additional orientation information to be provided to the examinee in the title line 2250 may include the descriptive word such as "HELP" "REVIEW" and "DIRECTION" to indicate a currently displayed screen. The "HELP," "REVIEW" and "DIRECTIONS" screens will be described below.

D. Presentation Area

The presentation area 2252 of the screen is used to display items screen such as the text and graphics and non-item screens such as direction screens, message screens, HELP screens and REVIEW screens. Direction screens are used to display directions for the test, section, and others. Message screens display information and examinee options at transition points during the test session to control the flow of the test. Transition points indicate where a new section or new item is to be displayed or when the test delivery application moves from an item screen to a non-item screen. HELP screens and REVIEW screens respectively enable an examinee to interact with the HELP and REVIEW facilities. These non-item screens will be described in more detail below.

1. Screens

a. Item Screens

Item screens are used to display items. Items can be mapped into the presentation area using one of a predetermined number of standard templates. (See Table 3, Presentation/Template Menu in the description of the Item Preparation System). In a preferred embodiment, the templates provide different combinations of one-, two-, or three-pane arrangements resulting in seven possible templates such as those shown in FIGS. 34(a)-(g). The panes are placed in the presentation area 252 like tiles which butt against each other but do not overlap. These arrangements provide test developers with some flexibility in designing item layouts.

In preferred embodiments, the text/graphics within a pane automatically becomes vertically scrollable if the volume of information is larger than the pane size. Horizontal scrolling can also be supported although it is not required for operation of the invention. A scrolling pane has a vertical (industry standard) scroll bar on the pane's right side. In addition, a status bar may be placed at the top of the pane. The phrase "Beginning," "More Available," or "End" is placed in the status bar and indicates the position of the currently displayed text within the pane as the examinee interacts with the scroll bar. "Beginning" is displayed if the topmost information is visible in the pane. "End" is displayed if the bottommost information is visible. "More Available" is displayed if neither the topmost nor bottommost information is visible.

In preferred embodiments, the default size of a pane is 50% of the presentation area. Thus, a two pane arrangement cuts the presentation area into two equally sized panes--either horizontally or vertically (see templates shown in 34(b) and 34(e)). A three-pane arrangement is produced by simply cutting one of the panes that result from a two-pane cut into two equally sized panes (see templates shown in FIGS. 34(c), (d), (f) and (g)). It should be understood that the default size and ratios can be adjusted to create templates having panes of substantial size.

The components of an item can be mapped into any pane of a template, but preferably panes are not empty. Additionally, components can share panes and can be placed in any order. Thus, for example, the stem and response can be assigned to the same pane.

If a stimulus component exists, the test developer can elect to have a stimulus status bar placed at the top of the pane containing the stimulus component. The status bar displays the status as the stimulus pane scrolls. The stimulus status bar contains the phrase "Questions xx to yy" flushed left, where "xx" is the item number of the first item to which the stimulus applies and "yy" is the last item to which it applies.

b. Direction Screens

Directions screens are typically used to display test, section, group and set directions. The directions may contain text and/or graphics which are specified by the test script (i.e., declarations in test, section, group or set configuration files). Scrolling is used to navigate through directions.

There are a number of ways to map the item directions component, i.e., accession. DIR, into the item's presentation area 252. For example, directions may be embedded above or below another component pane which may contain stem, response, or stimulus. Alternatively, directions may be placed in a special directions pane that is inserted below the title line 250 and above the template. Preferably, the directions pane fills the entire width of the presentation area 252, and the vertical height of the directions pane is adjustable to fit in the presentation area along with any of the template arrangements of FIG. 34 (a)-(g). Vertical scrolling is preferably supported if there are more directions than the pane can display. In a preferred embodiment, the size of the directions pane is specified and devoted to the template. Template sizing rules are then applied as described above. Still further, item directions can be displayed immediately before the item is displayed for the first time in a standard directions screen. Generally, test and section directions will only be displayed in a single directions screen.

FIG. 35 is an example of a section directions screen. A single button, DISMISS DIRECTIONS 265 dismisses the directions screen when the examinee selects it. Once the examinee dismisses the directions, they are preferably accessible through the HELP facility. If an examinee tries to dismiss the directions before scrolling to the end, a warning message is preferably displayed. The message as shown in FIG. 36(a) notifies the examinee that the directions should be read completely and that they can later be retrieved through HELP.

In preferred embodiments, test directions are displayed as the first screen of the delivery unit. Test directions notify the examinee, for ex