(PICTURE)(PICTURE) (PICTURE)Europäis hes Patentamt
(1 9) (PICTURE) European Patent Offi_e
(PICTURE) Offi_e euroPéen des brevets (1 1 ) E _ O 664 O4_ B_
(1 2) EUROPEAN PATENT _PECl FICATlON
(45) Date of pUbliCation and mention (51 ) _nt C_.6_. _O9B _JOO, _OgB _/O2,
of the grant of the patent_. _ogB _/o__
O8.12.1999 Bulletin 1999l49
(2 ) (86) International application number_.
1 AppliCatiOn nUmber'. 93923861 _4 p__lu_g3logi6i
(22) Date of filing_. 12.1 O.1993 (8_) _nternat__ona_ pub___Cat__on number..
(PICTURE)WO 94lO9466 (28.O4.1994 Gazette 1994l1 O)
(54) _Y_TEM AND METHOD FOR COMPUTER BA_ED TE_TING
SYSTEM UND VERFAH REN ZUR COMPUTERUNTERSTUETZTEN PRUEFUNG
(PICTURE)(PICTURE)
SYSTEME ET PROCEDE D' EXAMEN IN FORMATISE
__
__
O
_
_ N w. h. . h f h b_. . f h . f h f h E .
_ Ote'_ It In nIne mOnt S rOm t e PU ICatIOn O t e mentIOn O t e 9rant O t e UrOPean Patent_ anY PerSOn maY 9IVe
o nOtiCe tO the EUrOpean Patent OffiCe Of OppOSitiOn tO the EUrOpean patent granted. NOtiCe Of OppOSitiOn Shall be filed in
ii a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art.
w (PICTURE)99(1 ) European Patent Convention). Printed by Jouve, 75OO1 PARIS (FR)
(PICTURE)
EP O 664 O41 B1
Des_ription
5 [OOO1] For many years standardized tests have been administered to examinees for various reasons such as for
educational testing or for evaluating particular skills. For instance, academic skills tests (e.g. SATs, LSATs, GMATs,
etc.) are typically administered to a large number of students. Results of these tests are used by colleges, universities
and other educational institutions as a factor in determining whether an examinee should be admitted to study at that
educational institution. Other standardized testing is carried out to determine whether or not an individual has attained
1O a specified level of knowledge, or mastery, of a given subject. Such testing is referred to as mastery testing (e.g.
achievement tests offered to students in avariety of subjects and the results being used for college entrance decisions).
[OOO2] Figure 1 depicts a sample question and sample direction which might be given on a standardized test. The
stem 4, the stimulus 5, responses 6 and directions 7 for responding to the stem 4 are collectively referred to as an
item. The stem 4 refers to a test question or statement to which an examinee is to respond, e.g., question 1 3. The
15 stimulus 5 is the text and/or graphical information (e.g., a map, scale, graph, or reading passage) to which a stem may
refer. Often the same stimulus is used with more than one stem. Some items do not have a stimulus. Items having a
common stimulus are defined as a set. In Figure 1 , questions 1 3 and 1 4 refer to stimulus 5 and therefore form a set.
Items sharing common directions are defined as a group. Thus, questions 8-27 form a group. Only questions 8-1 4,
however, are shown in Figure 1 .
2O [OOO3] Atypical standardized answer sheet for a multiple choice exam is shown in Figure 2. The examinee is required
to select one of the responses according to the directions provided with each item and fill in the appropriate circle on
the answer sheet. For instance, the correct answer to the question stated by stem 1 is choice B of the responses 3.
Thus, the circle designated 8 in Figure 2 corresponding to choice (b) is the correct answer to this item, i.e. question
1 3 should be filled in by the examinee as shown.
25 [OOO4] Generally, examinees register to take a particular test, by filling out a registration form and sending it to a test
processing center such as Educational Testing Service, Princeton, NJ by a specified registration date. A registration
form usually requires that an examinee provide information such as the examinee's name and address, test to be taken
and some related biographical information. After all of the registration forms have been received by the test adminis-
tration center, the examinee information such as name, address, some recipients background questions, etc., is proc-
3O essed. Each examinee is scheduled to take the test by assigning to that examinee a place and time at which the test
can be administered to that examinee. Typically, a number of examinees are scheduled to take the test at the same
time and same place to conserve on administrative costs. One or more test administrators will be present atthe locations
where the test is scheduled to be taken.
[OOO5] Test administrators are generally responsible for distributing the test material, providing instructions to the
35 examinees, monitoring any timing constraints required by the particular test and collecting the test material when the
testing time has ended or when the examinee has finished taking the test. After collecting the examinees' responses
and other test material, the administrator either directly or indirectly sends them back to the test processing facility, for
scoring and evaluation.
[OOO6] After all of the examinees' tests are graded, statistical and other processing may be provided for various
4O reasons. For instance, to assess one examinee's score, it is necessary to compare his or her score to those of other
examinees taking the same test. Another important reason to evaluate the test results for statistical purposes is to
create and update an informations bank containing the performance statistics of each item used or created for previous
tests. This information may then be used for the creation of future tests.
[OOO7] A goal of standardized testing is to construct a test efficiently for the purpose of measuring a skill, ability, etc.
45 Therefore, each test is constructed to conform to a test specification which defines the rules and/or constraints for
selecting the items. In constructing a test, test developers select items from pool of items so that the combination of
selected items satisfy the test specification.
[OOO8] A test is typically divided into sections of questions. The test specification generally defines the number of
items to be presented in the test, the number of test sections, the number of questions in each section, the time for
5O taking the test, and the allotted time for responding to all the items in each test section. The test specification also
specifies criteria for item selection. These are based on at least four item characteristics which include_. (1 ) item content,
e.g., mathematical questions relating to arithmetic, algebra, or geometry; (2) cross-information among items, e.g., more
than one item testing the same point; (3) number of items/set, i.e. a identification of a subset of items of a larger set;
and (4) statistical properties of items derived from pretesting, e.g. difficulty of the selected items.
55 [OOO9] In recent years, these methods for creating, delivering, administering, and scoring tests have been determined
to be inadequate. Due to the number of examinees taking standardized tests, the demand for developing new and
more diverse tests and a need to provide more flexibility in scheduling tests without sacrificing administration costs
and security have increased. 2
(PICTURE)
EP O 664 O41 B1
[OO1 O] One solution to these demands would be to automate the entire testing process. However, up until now only
a few attempts have been made to automate only portions of the testing process. Furthermore, these attempts are
limited in their ability to generate a variety of item types. They are not modular in their design to allow independent
replacement of software or hardware, nor do they provide security and integrity features required for a standardized
5 testing environment.
[OO1 1] There have been attempts to develop computerized tools for instructional purposes. These products, although
primarily geared to delivering instructional systems, often contain testing components as well. Some examples of in-
structional programs are available from Computer Curriculum Corp. , Computer Networking Specialists Inc. , Computer
Systems Research, DEGE M, Ideal Learning, Josten's Learning Corp. , New Century Education, Plato Educational Serv-
1O ices - TRO Inc. , Unisys - ICOPN System, Wasatch Education System, and Wicat Systems. Wasatch Courseware, for
instance, provides on-line tools, such as a notebook, a pop-up calculator, word processor, graphics tool, glossary, and
a database embedded into the lessons. Josten's Learning Corp. provides some flexibility in the hardware and software
available for executing lessons such as networked or non-networked systems, the use of third party software, and the
ability to operate its instructional system from a remote site. Ideal Learning has a management system which is also
15 capable of accommodating third party software, and its test scoring system can score tests which are generated by a
number of test developers including standardized tests. The DEGEM System is a networked system which is capable
of providing statistical data on student or class progress. Therefore, although some of these instructional programs
incorporate some features which could be utilized in an automated standardized computer-based testing system, none
of them provides a flexible and integrated system for developing, generating, delivering, administering and processing
2O computerized standardized tests.
[OO1 2] There are also a number of systems for computerizing parts of the test construction process. (See e.g. , a
review by Hsu and Sadock (1 985)). Perhaps the most comprehensive of these testing programs is the MicroCAT
System developed by Assessment Systems Corporation. The MicroCAT System comprises four primary subsystems,
one for each of development, examination, assessment, and conventional testing.
25 [OO1 3] Although MicroCAT has been noted for its comprehensiveness, it has been criticized for a number of limita-
tions. For instance, development of a test having a specification which does not match one of its predefined templates
requires a detailed understanding of MicroCAT's programming language. Its graphics tools are vey limited, and other
commercial drawing packages such as PC Paint cannot be substituted for MicroCat's graphics. Furthermore, there is
no on-line help available from either the test development system or from the examination system. Without an on-line
3O help facility, a system such as MicroCAT could not practically be used to deliver and administer standardized tests to
thousands of examinees each year. To use the MicroCAT assessment System, the test data must have been based
on tests which were generated only by MicroCAT's specifications. Furthermore, MicroCAT does not provide security
for examinee performance files nor does it provide integrity features to guard against power interruptions and the like.
[OO1 4] To accommodate standardized tests in computer based testing, there is a need for a comprehensive computer
35 based testing system which provides flexible test development and production, test administration and test delivery,
as well as preprocessing and postprocessing of item statistics and examinee performance. Such a system should
incorporate data integrity features, including system failure recovery and data security features. The design should be
modular and extensible so that substantially every hardware and software component can be modified or replaced
without affecting the functioning of the remainder of the system.
4O [OO1 5] US-A-5 OO2 491 discloses a computerized interactive instruction system in which a teacher is able to select
questions to be asked and monitor the students' responses in order to judge the effectiveness of the teaching.
[OO1 6] US-A-4 671 772 discloses a computerized management appraisal and feedback system in which users re-
sponses are used to compile a selection of video clips indicating a subject's strengths and weaknesses as a manager.
[OO1 7] US-A-5,O59, 1 27 discloses a computerized mastey testing system for the computerized implementation of
45 sequential testing in order to reduce test length without sacrificing mastery classification accuracy.
[OO1 8] Various aspects of the present invention are set out in the accompanying claims.
5O [OO1 9] The present invention will be better understood, and its numerous objects and advantages will become ap-
parent by reference to the following detailed description of the invention when taken in conjunction with the following
drawings.
[OO2O] Figure 1 is an example of written test questions and related directions.
[OO21] Figure 2 is a sample answer sheet used for paper and pencil tests.
55 [OO22] Figure 3 is a general overview of computer based testing facilities.
[OO23] Figure 4 is an interface diagram depicting the interfaces of each of the CBT (Computer Based Testing) systems
according to the present invention.
[OO24] Figure 5 is an interface diagram showing the subsystems of the test development system according to the
3
EP O 664 O41 B1
present invention.
[OO25] Figure 6 is an interface diagram showing the administrative system and test delivery system interfaces ac-
cording to the present invention.
[OO26] Figure 7 is an interface diagram showing the subsystems of the N DDS (Network Data Distribution System)
5 according to the present invention.
[OO27] Figure 8 is an information flow diagram for the ''TD/DC'' system.
[OO28] Figure 9 is a functional flow diagram of test production according to the present invention.
[OO29] Figure 1 O shows an example of a dialog box presented by the l PT (ltem Preparation Tool).
[OO3O] Figure 1 1 shows an example of some item components as viewed from the item preparation tool.
1O [OO31] Figure 1 2 shows the scrolling capability utilized by the item preparation tool.
[OO32] Figure 1 3 provides an example of selecting text in a stimulus by reverse video.
[OO33] Figure 1 4 shows dialog boxes presented by the l PT which prompt for response parameters.
[OO34] Figure 1 5 shows a dialog box presented by the l PT which prompts for key information.
[OO35] Figure 1 6 shows an example of an item having a stem referencing a demarcated portion of a stimulus.
15 [OO36] Figure 1 7 depicts an appropriate response to the item presented and shown in Figure 1 6.
[OO37] Figure 1 8 shows a second example of an item having a stem referencing the item's stimulus.
[OO38] Figure 1 9 depicts an appropriate response to the item presented and shown in Figure 1 8.
[OO39] Figures 2O and 21 show an example of a reference file replete with custom and interaction codes according
to the present invention.
2O [OO4O] Figure 22 shows the flow of items and keys from test development to test production services.
[OO41] Figure 23 shows a functional block diagram of the test preparation process according to the present invention.
[OO42] Figure 24 shows the relationship between the session script, test scripts, and units.
[OO43] Figure 25 provides a functional flow diagram of the test packaging process according to the present invention.
[OO44] Figure 26 is a block diagram showing the components of a test package.
25 [OO45] Figure 27 is a block diagram showing some item level components included in the presentation BLOB (Binay
Large Object).
[OO46] Figure 28 is a block diagram showing some test level components included in the presentation BLOB.
[OO47] Figure 29 is a block diagram showing some table components included in a SKM BLOB (Scoring Key Man-
agement).
3O [OO48] Figure 3O provides a functional block diagram of the test packaging process.
[OO49] Figure 31 shows a functional block diagram of the test delivey system according to present invention.
[OO5O] Figure 32 is a functional flow diagram of the test delivery process according to the present invention.
[OO51] Figure 33 shows some primary screen components of each screen presented during the test delivery.
[OO52] Figures 34(a)-(g) show examples of pane arrangements supported by the present invention.
35 [OO53] Figure 35 is an example of a directions screen.
[OO54] Figures 36(a)-(l) show examples of message screens supported by the present invention.
[OO55] Figures 37-39 provide some examples of Help Screens.
[OO56] Figure 4O provides an example of a review screen.
[OO57] Figure 41 provides a few examples of selectable testing controls supported by the present invention.
4O [OO58] Figures 42(a)-(b), 43, 44(a)-(c), 45-47 provide some examples of examinee interaction with items having
various response types.
[OO59] Figure 48 shows the data format of an examinee performance file according to the present invention.
[OO6O] Figure 49 shows possible data fields associated with the ''Start Session'' log event.
[OO61] Figure 5O shows possible data fields associated with the ''End Item'' log event.
45 [OO62] Figure 51 shows a functional flow diagram of the administrative process according to the present invention.
[OO63] Figure 52 shows some of the files on a workstation hard disk after it has been configured for computer based
testing.
[OO64] Figure 53 is a functional flow diagram describing the Close-of-Day Procedure.
[OO65] Figure 54 shows the data format of a log record according to the present invention.
5O [OO66] Figure 55 is a flowchart of the Main-Procedure of the Administrative Application.
[OO67] Figure 56A and 56B are flowcharts of the Start-System-Procedure of the Administrative Application.
[OO68] Figure 57 is an example of a system status screen.
[OO69] Figure 58 is a flowchart of the Logon-Procedure of the Administrative Application.
[OO7O] Figure 59 is an example of a sign-in screen provided by the Administrative Application.
55 [OO71] Figure 6O is a flowchart of the Process-State-Procedure of the Administrative Application.
[OO72] Figure 61 is a flowchart of the State-Null-Procedure of the Administrative Application.
[OO73] Figure 62 is a flowchart of the Change-Password-Procedure of the Administrative Application.
[OO74] Figure 63 is a flowchart of the State-Admin-Procedure of the Administrative Application.
4
EP O 664 O41 B1
[OO75] Figure 64 is an example of a screen showing the main menu of the Administrative System.
[OO76] Figures 65A and 65B are flowcharts of the State-Close-Procedure of the Administrative Application.
[OO77] Figure 66 is an example of a screen provided by the Administrative Application allowing an administrator to
enter test counts.
5 [OO78] Figure 67 is a flowchart of the State-Maint-Procedure of the Administrative Application.
[OO79] Figure 68 is a flowchart of the State-TDS-Procedure of the Administrative Application.
[OO8O] Figure 69 is an example of a screen provided by the Administrative Application when the testing session is
complete.
[OO81] Figure 7O is a flowchart of the State-Exit-Procedure of the Administrative Application.
1O [OO82] Figure 71 is a flowchart of the Menu-OpTest-Procedure of the Administrative Application.
[OO83] Figure 72 is a flowchart of the Menu-DemoTest-Procedure of the Administrative Application.
[OO84] Figures 73A and 73B are a flowchart of the Menu-TestCommon-Procedure of the Administrative Application.
[OO85] Figure 74 is an example of a screen provided by the Administrative application allowing an administrator to
enter examinee identification information.
15 [OO86] Figure 75 is an example of a screen provided by the Administrative Application allowing an administrator to
administer a test.
[OO87] Figure 76 is an example of an examinee confirmation screen.
[OO88] Figure 77 is an example of a screen provided by the Administrative Application allowing an administrator to
terminate the testing session or to edit the examinee information.
2O [OO89] Figure 78 is a flowchart of the Menu-RestartTest-Procedure of the Administrative Application.
[OO9O] Figure 79 is an example of a screen provided by the Administrative Application listing the available testing
programs.
[OO91] Figure 8O is an example of a screen provided by the Administrative Application listing the sessions that are
available for restart.
25 [OO92] Figure 81 is a flowchart of the Menu-CloseDay-Procedure of the Administrative Application.
[OO93] Figure 82 is a flowchart of the Menu-Exit-Procedure of the Administrative Application.
[OO94] Figures 83A and 83B are flowcharts of the Menu-Logon Maint-Procedure of the Administrative Application.
[OO95] Figure 84 is a flowchart of the Menu-CHG Password-Procedure of the Administrative Application.
[OO96] Figure 85 is a flowchart of the Stop-System-Procedure of the Administrative Application.
3O [OO97] Figure 86 is a flowchart of the Menu-About-Procedure of the Administrative Application.
[OO98] Figure 87 is a block diagram showing the inputs and outputs of the Network Data Distribution System (NDDS)
according to a preferred embodiment of the present invention. Figure 88 is an example of the Main Menu Screen for
the NDDS.
[OO99] Figure 89 is a preferred directoy structure of the NDDS.
35 [O1 OO] Figures 9OA through 9OC is a flowchart of a preferred procedure for processing CBT transmission files.
[O1 O1] Figure 91 is an example of a screen provided by the Administrative Application prompting the administrator
to enter the data disk.
[O1 O2] Figures 92A and 92B is a flowchart of a preferred procedure for processing the CBT Data Disks.
[O1 O3] Figure 93 is an example of a screen provided by the Administrative Application prompting the administrator
4O to insert the backup disks.
[O1 O4] Figures 94A and 94B is a flowchart of a preferred procedure for processing the CBT Backup Disks.
[O1 O5] Figure 95 is an example of the screen provided by the NDDS of an activity report selection menu.
[O1 O6] Figure 96 is a flowchart of a preferred procedure for implementing the activity reporting process according to
the present invention.
45 [O1 O7] Figure 97 is an example of a screen provided by the N DDS of an audit trail report selection menu.
[O1 O8] Figure 98 is a flowchart of a preferred procedure for implementing the audit trail reporting process.
[O1 O9] Figure 99 is an example of a screen provided by the NDDS of a Security/Event Log Report Selection Menu.
[O1 1 O] Figures 1 OOA through 1 OOF are examples of screens provided by the NDDS for carying out Computer Based
Testing Network (CBTN) processing according to a preferred embodiment of the present invention.
5O (PICTURE)
l. Computer Based Testing (CBT) System Overview
55 [O1 1 1] Referring to the drawings wherein like numerals represent like elements, there is illustrated in Figure 3 a
general overview of computer based testing. Computerized tests are developed at a central processing site 1 . Devel-
opment of a computerized test includes the creation of various data files by application specific software. The compu-
terized tests are packaged at the central processing site 1 and delivered to one or more test centers 2. Each test center
5
EP O 664 O41 B1
2 provides at least one workstation 3 on which a computerized test is administered to an examinee. In a preferred
embodiment, the workstation 3 is a personal computer equipped with a mouse.
[O112] A test center 2 may for example be located at a school or a dedicated test site. Generally, a test administrator
Iocated at the test center 2 loads the computerized test, data files and application software developed at the central
5 processing site 1 onto the hard disk of each workstation 3 at the test center 2. The administrator initiates the delivery
of a computerized test to an examinee who is scheduled to take the test. The examinee's responses to questions
presented by the test are preferably stored on the hard disk on each workstation 3 and are later preferably backed-up
by the administrator and transferred to the central processing site 1 for scoring and evaluation.
[O113] In Figure 3, one central processing site 1 , three test centers 2, and 9 workstations 3 apportioned among the
1O test centers 2 are shown. However, it should be understood, that any number test centers 2 and workstations 3 may
be used by the CBT system.
[O114] A block diagram of the CBT system is shown in Figure 4. The lines in the diagram demarcating each system
represent interfaces that pass information between the various systems which comprise the CBT system. The double
line separates those systems that reside at the test centers from those that reside at the central processing site. Those
15 systems shown within the double lines are the systems residing at the test centers, and those systems shown outside
the double lines are the systems residing at the central processing site.
[O115] Still referring to Figure 4, the CBT system comprises six separate systems. The Administrative System 1 4
provides substantially all administrative and control functions at a test center. The Test Delivery System 1 2 actually
presents questions and information tothe examinee at aworkstation. The Administrative System 1 4 initiates the delivery
2O of a test to an authorized examinee and secures the test by prohibiting access by any unauthorized person. Commu-
nication with the Test Delivery System 1 2 occurs through the Administrative System 1 4. The Network Data Distribution
System (NDDS) 1 8 receives data from the test centers 2 and distributes returned data to other systems at the central
processing site. The Test Development System lo provides substantially all functions necessaryto create test questions
and package computerized tests. The Preprocessing System 2O provides functions performed prior to the testing ses-
25 sion, such as registration. Such Preprocessing systems are typically custom designed for a particular test and are
provided commercially by various testing support companies such as Educational Testing Service, Psychological Cor-
poration, and American College Testing Service. Like the Preprocessing System 2O, Postprocessing systems are typ-
ically customized for each type of test and are provided commercially through Educational Testing Service, Psycho-
Iogical Corporation, and American College Testing Service. The Postprocessing System 1 6 provides functions per-
3O formed after the testing session, such as issuing official score reports or archiving examinee records.
[O116] A detailed block diagram of the Test Development System 1 O is shown in Figure 5. Five primary functions are
performed within the Test Development System 1 O, including test development/document creation (''TD/DC'') 62, an
item preparation system 6O, scoring and test key management (SKM) 66, computerized test preparation system 57
and test packaging 58. The Test Development System 1 O permits test developers to develop items and a Test Produc-
35 tion Staff (TPS) to computerize the items. It also supports the creation of test scripts. A test script is the electronic form
of a test. It provides option settings and configuration data necessary for delivering the test on a workstation. The Test
Development System 1 O also supports the creation of test keys, i.e., correct response to each item, and the packaging
of all components into one test.
[O117] Items are preferably written and created using the ''TD/DC'' (Test Development/Document Creation) system
4O 62. The ''TD/DC'' system 62 interfaces with the Item Preparation system 6O and with the scoring and key management
system 66 to computerizethe item content and key respectively. The Item Preparation System 6O is usedto computerize
the items for presentation by the Test Delivery System 1 2 and enter the computerized version of the item key which
differs depending on the item type. The Item Preparation System 6O prepares dataforthe scoring and key management
system 66 to communicate the computerized key information. The Item Preparation System 6O interfaces with the Test
45 Preparation System 57 to prepare the test scripts. The Test Packaging System 58 interfaces with the ltem Preparation
System 6O, the scoring and key management system 66, and the Test Preparation System 57 to obtain all of the item
and test components packaged into a computerized test. The NDDS Interface 56 transmits the computerized test from
the Test Packaging System 58 to the NDDS 1 8 as shown in Figure 4, for delivey to a test center 2 as shown in Figure
3. After an examinee has taken a computerized test, an postprocessing interface 64 with the Postprocessing System
5O 1 6 provides information about item keys to the ''TD/DC'' System 62 which uses this information to alter items or add
new items for use in subsequent tests.
[O118] Figure 6 shows a block diagram of the Administrative System 1 4 and Test Delivery System 1 2 and their
respective interfaces as shown in Figure 4. The Administrative System 1 4 permits test center administrators to control
delivey of tests, transmit results to a central processing site, and perform administrative functions such as backup of
55 data, item and software maintenance, and reporting. The Administrative System 1 4 fu_her prohibits access to the
computerized test by unauthorized persons.
[O119] An interface with the Network Data Distribution System (NDDS) 1 8 at the central processing site enables the
Administrative System 1 4 at a test center to send packaged examinee data and reports to a central processing site.
6
EP O 664 O41 B1
Data and software are sent from the central processing site to the test centers on diskettes.
[O12O] Still referring to Figure 6, the Test Delivey application Interface 26 is shown as having three specific interface
functions. First, the Administration system 1 4 can initiate a delivery of a computerized test and pass the necessary
information to the Test Delivery System 1 2. The Administration System 1 4 may also interact with the Test Delivey
5 System 1 2 for purposes such as terminating the testing session, processing examinee breaks, and timing essays, as
appropriate.
[O121] After termination of the test delivery, the Test Delivey System 1 2 transfers information such as examinee
performance data, return codes, and other processing data as appropriate to the Administration System 1 4 through
the Test Delivery Application Interface 26. The Administration System 1 4 then regains control of the workstation.
1O [O122] The Administrative Application 3O provides test center administrators with the ability to perform various func-
tions including_. controlling access to computerized tests and related data through levels of authorization and password
protection; entering and editing examinee identification information prior to the testing session; selecting the test to
administer; terminating tests; backing up examinee and administrative data; transmitting data to the central processing
site; changing passwords and adding or deleting administrator logon IDs; and reporting irregularity and activity data
15 to the central processing site.
[O123] A detailed block diagram of the Network Data Distribution System (NDDS) 1 8 with its interfaces is shown in
Figure 7. The Network Data Distribution System (NDDS) 1 8 provides substantially all necessary support functions for
the CBT system to control the network of test centers. The Test Center Administrative Application Interface 36 permits
the transfer of applications and computerized tests to the test centers and examinee records and reporting information
2O (data related to system errors and tesUworkstation security) from the test centers to the central processing site. The
Test Development Interface 42 provides the means by which new or revised computerized tests are sent from the Test
Development System 1 O to the NDDS 1 8. The NDDS 1 8 uses a Test Center Information Database 4O to determine
which test centers should be sent any new/revised tests, and to create reports from data received from the test centers.
The NDDS Processing component 44 receives data from test centers 2 via Administrative Application Interface 36,
25 checks it, sorts it, and processes it according to its type (program data, security data and reporting data). Reporting
data is used to create the necessary reports. Program data such as examinee records, are processed to consolidate
and reformat the information in a form suitable for postprocessing. The Distribution Interface 38 then distributes the
processed data to the Postprocessing System 1 6.
3O Il. Test Development System
A. Test DevelopmenUDo_ument Creation
[O124] In a preferred embodiment, test developers create tests at the central processing site. In computer based
35 testing, the tests are created by the test developers (TD) and are further processed and packaged by test production
staff (TPS). The ''TD/DC'' System which is developed by Educational Testing Service is preferably used to create the
test forms. It should be understood, however, that other test document creation systems could likewise be used to
create test forms for computer based testing. Therefore, although a detailed explanation of ''TD/DC'' is not required for
a description of the test development system, an understanding of its functional and procedural operation will aid in
4O understanding the test development system.
[O125] The ''TD/DC'' System 62 (Figure 5) is a fully automated system in and of itself. It consists of a central item
database and local personal computer based workstations. The central item database stores substantially all items
previously used on standardized exams as well as other items that have been created but not previously selected for
inclusion in an exam. Associated with each item stored in the central item database, is data stored in fields related to
45 the item's answer key, revisions it has undergone, a list of the test forms in which the item has previously appeared,
and statistical data indicating how the item performed at each previous administration. Other descriptive data fields
include information related to the item type (i.e., multiple choice or fill in the blank), the item's author, copyright infor-
mation, and both content and cognitive information specific to each testing program (e.g., SAT, GRE, etc.).
[O126] Every item is assigned a unique number called an accession number that identifies the item and all of its
5O associated data. The ''TD/DC'' system software allows items to be located by means of any of the datafields associated
with the item.
[O127] The central item database software allows test developers to access item information within the central item
database. This software supports the downloading of items and associated data to local workstations. Pools of items
may also be selecting and then downloaded. This software also enables test developers to bank, edit, and classify
55 features of the items stored in the central item database. Additionally, statistical information related to the use of an
item in an administration of a test is received by the central database software. Through a statistical feedback system,
this statistical data is added to the data stored in the central item database.
[O128] New items may be written by test developers at the local workstations. Software provided on the local work-
7
EP O 664 O41 B1
stations supports classifying, banking, and viewing these new items. With respect to the downloaded items, this soft-
ware also allows test developers to viewthose items and their associated data and permits the test developers to enter
and revise the statistical data.
[O129] Test developers also assemble draft tests on local workstations. The local workstations provide software
5 supporting item selection based on a number of criteria so that tests may be assembled to satisfy substantially any
test specification. When all of the items are selected to satisfy the test specification, these items and associated data
are assembled into what is referred to as a Worksheet.
[O13O] Test production for computer based testing, requires certain inputs from test developers and/or a test docu-
ment creation system. Figure 8 depicts the inputs provided by the test developer and the outputs generated by ''TD/
1O DC'' for use by TPS. For instance, ifthe ''TD/DC'' system is used, the information shown as offload files 74 and workfolder
76 is preferably provided to TPS. As described above, worksheets 72 are created by TD by downloading the selected
items and associated data. An offload program may then be invoked to copy the offload files, such as the item com-
ponent text (stem, stimulus, response, and directions), the response type, (e.g., multiple choice) response class (e.g.,
single response answer), the answer key and the item's accession number, onto a diskette. In preferred embodiments,
15 the TD also prepare a workfolder 76 containing information related to the computerized presentation of the items, and
graphics to be prepared as a computerized image and incorporated into the item presentation.
B. Test Produ_tion
2O 1 . Overview of Tegt Produ_tion
[O131] Test production comprises at least three primary functions, item preparation, test preparation and test pack-
aging. The Item Preparation System 6O as shown in Figure 5 is used by TPS to create an ''on screen'' version of the
items prepared. As described above, the test questions are prepared beforehand by test developers preferably using
25 the ''TD/DC'' System 62 or equivalent system. ltem text is edited until the content is satisfactoy to the test developers.
An offload program is used by the test developers to copy the selected items to a diskette which is sent with a work
folder of batch-related documents to the test production staff. The test production staff then makes a computer deliv-
erable image of the items in the form of files and prepares a test script for implementing the test. The test packaging
system 58 combines the item files with the test script to form the computerized test.
3O [O132] In a preferred embodiment, the Item Preparation system 6O comprises seven programs providing the functions
shown in the flow diagram of Figure 8. Table 1 below itemizes the programs used by the Item Preparation System 6O.
The table lists the purpose of each program, the program name, the operating environment in which the program
preferably is executed and the user during the test development process. ''WORD FOR Wl NDOWS'' and ''PAINT-
BRUSH'' are commercially available from Microsoft Corporation.
35 (PICTURE)
TABLE 1
4O
45 kTD - test developers
kkTPS - test production staff
5O kkkSYS - SysteMs
[O133] A functional flow diagram of the Test Production Process is shown in Figure 9. Assuming for purposes of
description that the system used to create the test document is the ''TD/DC'' system, items are first offloaded at 21 O
by test developers using the item offload function of the ''TD/DC'' system and are stored on a diskette. The offloaded
55 items and a corresponding workfolder are then transferred to the TPS. If graphics are to be shown with the item as
determined at 21 2, a computerized image of the graphics are prepared at 21 4 by TPS. TPS reads the items from the
diskette and separate them into component parts shown at 21 6 using the Item Element Generator (l EG). The text of
the item components can then be edited using a word processor at 21 8 such as ''WORD'' for example. Information
8
EP O 664 O41 B1
may be added to each component regarding, for instance, point size, font, leading, column arrangement, etc. during
word processing 21 8. Then, the items may be prepared at 22O using the Item Preparation Tool (IPT) to arrange the
components on the computer display and specify characteristics about each item (e.g., multiple choice, response type,
etc.). Finally, an electronic ''proofing'' copy of the items is returned to the test developers for review at 224. The test
5 developers maythen provide corrections orfinal approval. The proofing copy is preferably reviewable bytest developers
via a modified form of the Item Preparation Tool known as the Item View Tool (IVT).
[O134] TPS may iterate between word processing 21 8 and item preparation at 22O until satisfied with the results.
Likewise, the entire process from item offload 21 O through item review 224 may be repeated. Simple corrections can
be made by sending marked-up prints from the test developers to test production staff so that the most current version
1O of the test can be called to the screen again and edited according to the comments produced by the test developers.
[O135] Once final approval has been received from the test developers, a test is prepared as shown at 226 by a test
preparation tool based on information provided in the workfolder. Finally, TPAK packages all of the item files and test
scripts into a computerized test at 227.
15 2. ltem Preparatio_
a. Item Offload Program
[O136] The Item Offload function 21 O is executed by test developers on the ''TD/DC'' system using the item offload
2O program. This function e_racts data from the ''TD/DC'' system to be used as input for test production which ultimately
results in the creation of a computerized test. In the ''TD/DC'' system, a file which is known as the Worksheet and
described previously in Section Il (A), lists the accession numbers of items. Each Worksheet has a unique name which
is assigned by a test developer. Item offload 21 O creates a single file with all of the offloadable information for all items
in the Worksheet. The offload file's name is the same as the Worksheet name. Item offload 21 O opens the Worksheet,
25 reads each item pointed to by the accession number, and writes to an ASCll file the item's accession number, classi-
fication codes, rationale and item text.
[O137] Test production requires specific classification information. Thus, information contained in the Worksheet is
preferably categorized bythe test developers. One category may include item information. For instance, test developers
should provide codes that indicate how each part of the item's text is used for production purposes, e.g., the code
3O indicating that particular text is the stem, the response, or the directions associated with each item.
[O138] Another category of information which may be provided to test production is rationale information. Again, test
developers may insert codes and text related to the key description, number of responses required to properly answer
the question posed by the stem, a paraphrase summarizing the content of the item, and general remarks regarding
the appearance of each item.
35 [O139] A third category of information which could be provided to test production is classification profiles of the items
included in the Worksheet. The classification profiles may include codes identifying the item class, item type, item
structure, and information describing how the item should be presented to an examinee. The item class refers to its
response class and is indicative of whether a single response, multiple response, or free response is required by the
item. The item type code specifies its response type and is indicative of the type of response required to answer the
4O question posed by the item. For instance, the item type codes specifies selection of a value on a scale, selection of a
response with an ellipse or a box, insert text, or select a choice from a table. These and many other item types will be
described in detail below.
[O14O] The item presentation code specifies a predefined screen template. The screen templates indicate how many
panes the screen should be divided into for each item and the location of the panes. Additional codes are used to
45 specify which item information, i.e., stem, response, directions or stimulus, is to be placed in each pane and its position
within the pane. Similarly, the item presentation codes will be described in more detail below.
[O141] A fourth categoy of information which may be provided to test production is the stimulus formation for a set
of items. The stimulus formation code specifies the beginning of a stimulus to be shared by a set of items and the
presentation of the stimulus. When a stimulus is referenced by only a single item, the item is called a discrete item. In
5O the case of a discrete item, the stimulus information codes are preferably provided with the item information codes
rather than the stimulus information codes.
[O142] Prior to executing item offload 21 O, the test developer using the ''TD/DC'' system may insert tags in the item
and rationale text which are used by the Item Element Generator (IEG) program to separate the text into smaller
components. Tags in the item text are used to delineate the directions, stem and response components. Tags in the
55 rationale text are used to supply information about the key, number of required responses and a paraphrase summa-
rizing the item, as well as the rationale text. 9
EP O 664 O41 B1
b. Graphi_s
[O143] The work folder indicates whether graphics are required as shown at 21 2 in Figure 9. TPS may generate the
graphics using ''PAINTBRUSH'' as one example. The graphics files are named, for example, by adding an extension, .
5 Gnn, to the accession number of the item which contains the graphic to be presented. The ''nn'' is representative of a
number so that up to 99 graphics in this example may be associated with an item.
_. Item Element Generator (IEG)
1O [O144] The Item Element Generator (IEG) is a DOS program used by the TPS. The IEG is preferably written in
Microsoft C version 5.1 . The l EG separates the ASCll file created by the Item Offload program as shown at 21 6. The
individual items in the Offload file are separated from each other, then the items themselves are broken into components
and stored in files.
[O145] The component files are preferably named by tacking an extension onto an item accession number. The
15 extension specifies which piece of the item (i.e., stem, stimulus, response, directions, etc.) the file contains. Since the
accession number is used for the base name of a file, it is easy to locate all of an item components. Table 2 below lists
each component file with a predetermined extension. TABLE 2
2O (PICTURE)
25
3o [O146] A COntrOl file iS aISO generated by the IEG fOr eaCh item. A COntrOl file iS a maSter repOSitOry fOr an item'S
information. Specifically, the control file defines how these components are to appear as an item on a display during
test delivey. Minimally, a control file and a stem file are generated for each item. A directions file and a response file
may be generated if the item contains directions and response components. A reference file contains the stimulus
material and is generated only for items which belong to a set. Additionally, the control files of set members contain
35 the aCCeSSiOn nUmber Of the StimUIUS material.
[O147] The l EG also creates two log files during its execution and records errors in an error file. The error file and
Iog files are named by adding an extension to the base name of the input file, that is, the file generated during item
offload 21 O. Thus, for example, if the input file is named TEST, log or errorfiles are created by adding a unique extension
to TEST.
4o [O148] One Of the lOg fileS iS the Item ACCeSSiOn LiSt File, WhiCh COntainS the aCCeSSiOn nUmberS Of diSCrete itemS
and members of sets (excluding stimulus material) for which component files were produced. This file is named using
an input file name and .IAL extension such as inputfile.IAL. Accession numbers are preferably written to the file in
ASCll format, one per line, in the order in which they were processed.
[O149] The second log file created by l EG is the Batch Histoy File, which contains the accession numbers of all
45 itemS in the OfflOad file. ThiS file iS named USing an inpUt file name and .BHF eXtenSiOn SUCh aS inpUtfile.BHF. ACCeSSiOn
numbers are preferably written to the file in ASCll format, one per line, in the order in which they were processed.
Accession numbers of the set may be written to the file one after the other, starting with the stimulus material.
[O15O] The Error Log file is an ASCll file that logs errors encountered during l EG execution. This file is named inputfile.
ERR.
5o [O151] The .IAL and . BHF are preferably aIWayS generated. The . ERR file iS generated if errOrS are repOrted. Indi-
vidual component files are generated depending upon the item type and tags embedded in the item by test developers.
[O152] In a preferred embodiment, IEG uses a code conversion table to convert ''TD/DC'' classification codes to Item
Preparation codes. By using a separate file for this table, the l EG does not have to be recompiled and linked if the
codes are extended or altered in any way.
55 d. Item Preparation Tool
[O153] In a preferred embodiment, the Item Preparation Tool (l PT) is a Wl NDOWS-based application used by the
1 O
EP O 664 O41 B1
TPS. The IPT is also preferably written in Microsoft C version 5. 1 along with the Wl NDOWS Software Development
Kit. It is the electronic analog of tools with which to prepare the item image. Using the Item Preparation Tool, TPS can
process items received after the component parts are separated by the l EG.
[O154] After item preparation is started at 22O, test production staff can scan a list of items available for processing
5 from the item accession list file which is created by the l EG at the step 21 6. The list is generated by compiling all files
with the .STE extension in the item accession list file.
[O155] Each item having a .STE file extension is then edited and processed by test production staff to create a
computer deliverable form of the item which is developed by the test developers. In combination with the item prepa-
ration process 22O, test production staff can edit any of the item text components by word processing 21 8. After editing
1O the text of the desired component, test production staff may return to item preparation 22O. The revised text may then
be displayed by the l PT.
[O156] The IPT is preferably a menu-driven application. The lowest level menu options have dialog boxes. Dialog
boxes are typically used in a Wl NDOWS environment for prompting a user to input the data. Most dialog boxes typically
have two selectable buttons. One button labelled ''OK'' is selected by a user to exit the dialog box when he or she has
15 finished entering the data. The second button ''CANCEL'' allows the userto exit the dialog box without entering the data.
[O157] Table 3 below describes each of the menu options and the content of the dialog box presented to a user, i.e.
test production staff, after invoking the menu option in a preferred embodiment. It is well known in menu-driven appli-
cations to layer the menus. For instance, IPT provides eight main menu options in capital letters in Table 3 below, e.
g. Fl LE, Vl EW, PRESENTATlON, etc. One or more lower level menus may be invoked when one of these main menu
2O options is selected by the user. For instance, the menu selection listed in Table 3 such as ''PRESENTATlON/Compo-
nents/ Directions'' indicates that the user had selected the main menu option PRESENTATlON. Afirst lower level menu
was then provided, and the user selected ''Components.'' Subsequently, a second lower level menu was provided, and
the user selected the ''Directions'' option.
25 (PICTURE)
TABLE 3
3O
35
4O
45
5O
55
EP O 664 O41 B1
(PICTURE)
TABLE 3 (continued)
5
1O
15
2O
25
3O
35
4O
45
5O
55 1 2
EP O 664 O41 B1
(PICTURE)
TABLE 3 (continued)
5
1O
15 Detailed flowcharts and corresponding pseudo code of the l PT application is provided in Appendix A. However, the
following example will be provided for a more complete understanding of the use of the item preparation tool.
_o [O158] Figure 1 O shows a dialog box 23O generated by the Item Preparation Tool when a user selects by clicking the
menu option Fl LE/Open. The Fl LE menu option then lists several other menu options including an ''Open'' option. The
user then selects ''Open'' and the dialog box 23O as shown in Figure 9 is opened and prompts for the opening of a
particular item.
[O159] ''Open_.'' 228 shows which item is selected to be opened. ''Path_.'' 229 shows where on the hard drive these
_5 items are stored. ''Files'' 231 provide a selectable list of all the available items in that directory. ''Directories'' 233 provide
a selectable list of file directories on the hard drive. The ''Open'' 223 and ''Cancel'' 225 buttons may be executed after
the appropriate item is selected or to exit the ''Open'' option.
[O16O] Together Figures 1 1 and 1 2 present a screen-print of a simple reading passage. The top line 232 in Figure
1 1 reveals the name of the item identified by the item accession number. The second line 234 is a menu bar from which
3o item preparation functions can be initiated to construct the item identified in line 232. The left-hand box is a pane and
contains the reading passage 236. The reading passage 236 is the stimulus and is contained in the reference file for
the item, MHOOOOO1 . REF. The right-hand pane contains the directions 238 and the stem 24O which are stored in
separate files, but designed to be displayed in the same pane. The stem 24O in this instance indicates that the answer
should be entered by interacting with the reference file or reading passage 236. By inserting specific types of custom
35 codes referred to as interaction codes into the text of the reference file, an examinee can respond by selecting a
sentence in the reference file. The sentence will become highlighted on the screen as shown in Fig. 1 3 after it has
been selected. Line reference numbers 242 are found directly to the left of the text of the passage. They are preferably
included in graphic files and are not actual text.
[O161] A reading passage is often longer than the screen size allows. Figure 1 2 shows the scrolling feature of the
4o Item Preparation tool that allowsthe userto move through the passage by using the scroll mechanism 244 Iying between
the panes 246 and 248. The use and implementation of a scrolling feature are well known.
[O162] Figure 1 4 shows a dialog box 25O that prompts for response parameters after invoking the RESPONSE menu
option. These parameters set up the nature and functions of the response area. The ''Number of Req. Responses;''
field 247 indicates how many responses are necessary in order to answer the item. The ''Response Class'' box 249
45 indicates the general categoy of the response, i.e., single choice, multiple choice, or free response. The ''Response
Type'' box 251 indicates the specific form of the response. A description of the different response types is provided in
Section Ill below.
[O163] Further demarcation of the item is accomplished by entering information in a series of nested dialog boxes
as shown in Figure 1 5. For instance, if ''Multiple Choice'' box 239 is selected from the ''Response Type'' box 251 , then
5o a ''Multiple Choice'' box 252 is opened. The ''Number of Choices'' 241 may be computed by the Item Preparation Tool.
For example, in Figure 1 5, the ''Number of Choices'' 241 is set at eight. This number is based upon the amount of
specific interaction codes included in the response area. This number is important in error checking for codes because
it should reflect the exact number of available responses available, i.e., whether the number is less than the intended
number of responses, The Item Preparation Tool may indicate an error in coding. ''Block Set'' 243 specifies which set
55 of interaction codes are to be read by the software for a specific item. ''Indicator'' 245 prompts for different response
designs. Examples of indicators are also described in Section Ill below. ''Invert Choices'' 237 indicates whether the
response option should be highlighted by reverse video when selected. The component 253 permits test production
staff to enter which of the options is the correct choice. It should be understood that different response types require
13
EP O 664 O41 B1
different parameters than the example shown in Figures 1 2 and 1 3. The data entered by test production staff in the
dialog boxes shown in Figures 1 4 and 1 5 are stored in the item control file.
[O164] A different item is shown in Figure 1 6, although the same reading passage 236 in the reference file is used.
In this instance, part of the reading passage 236 is highlighted in reverse video 257 when the item is presented. Note
5 again the directions 254, stem 256, and response 258 are included in one pane. However, different response param-
eters have been set for this item, i.e., there are four options instead of eight. Interaction is set to the response file and
not to the reference file, ellipses are included rather than invert choices. Figure 1 7 shows the selection of a response
option 255 from the responses 258.
[O165] Figure 1 8 shows yet a third item which refers to the same reading passage 236. The stem 259 asks for a
1O word to be selected in the second paragraph. Although the item parameter dialogue box is not shown, item parameters
are set up in a third way so that there are 42 options (as many words as in the second paragraph). Interaction has
been switched back to the reading passage 236 or reference file, and the choices selected by an examinee will be
inverted. Figure 1 9 shows the correct answer at 261 after being selected.
15 e. Wo_d __o_eggi_g
[O166] In a preferred embodiment, the word processor is ''WORD FOR Wl NDOWS,'' which is available from Microsoft
Corporation. It is used by test production staff to edit component files produced by the l EG or to create completely new
items. Component files are edited for two purposes. The first purpose is to effect the appearance of the item text by
2O adding fonts, point size, bolding, etc. The second purpose is to insert ''Custom Codes'' before and after sections of text.
[O167] Component files are initially stored in ASCll by the l EG as described above, but they are converted to Micro-
soft's RTF format when they are saved in ''WORD FOR Wl NDOWS.'' Even though ''WORD FOR Wl NDOWS'' does the
conversions, the test production staff is responsible for selecting the correct format. After editing, text is written in RTF
format backtothe component file from which it came. Thus, a componentfile may contain either ASCll or RTF formatted
25 data, depending upon whether the file has been edited by ''WORD FOR Wl NDOWS.''
[O168] Now turning to the second reason for editing a component file, namely to add Custom Codes, literal strings
are inserted into the text and thus allow computerized features to be added to the test. In a preferred embodiment,
CUStom CodeS aIWayS Start With a '''i.''
[O169] Custom Codes belong to one of three classes_. 1 ) codes that stand alone, 2) codes followed by a parameter,
3O and 3) codes followed by additional data. ''Stand alone'' codes appear in the text by themselves. Their vey presence
is all the information conveyed by the code.
''Parameterized'' codes are distinguished from other classes of codes in that they are followed by a parameter enclosed
in square brackets (''['' and '']''). The parameter immediately follows the code without any intervening characters or
white space. ''Data'' codes are followed by other data. The data is arbitray text. To prevent conflicts with parameterized
35 codes, a single white space character is used to separate the code from the user-supplied data.
[O17O] Table 4 below summarizes some examples of Custom Codes and their use. Optional elements of parameters
are enclosed in curly braces (''('' and '')''). These codes are used to include a graphic in the component. The parameter
''nn'' is used to form part of the graphic file extension. Graphic files are named accession.Gnn, where accession is the
same as the base name of the file in which the graphic code appears. Thus, for example, if the graphic appears in a
4O stimulus component whose name is TD-OOO81 . REF, the graphic file name may be TD-OOO81 .Gnn.
(PICTURE)
TABLE 4
45
5O
55 14
EP O 664 O41 B1
(PICTURE)
TABLE 4 (continued)
5
1O
15
2O
25
3o [O171] Figure 2O is an example of the actual reference file in ''WORD FOR WINDOWS'' where it is manipulated and
replete with formatting and interaction codes. The control of the reference file shown in Figure 2O is based on the
sample items shown in Figures 1 O through 1 7. The name of the file is included in the first line of text. In this instance,
'''iACCESSlON[MHOOOOO1 ]'' iS the file name. The neXt CUStom Code on the firSt line iS an interaCtion Code, '''iPMC'' WhiCh
is a response code and indicates a ''Place Multiple Choice'' to be included in the passage. Since this code is not item
35 specific, it can be used once and be referenced by any number of items (in this instance, by items 1 and 3). The next
element, '''iGRL[O1 ],'' iS a Code that CalIS in a graphiCal image WhiCh eXiStS in a Separate file. ThiS Code indiCateS that
at this point in the file, a graphic should be placed to the left margin before any more text is included. The graphic in
this instance is the line reference numbers 5 through 2O. The graphical numbers 25 through 3O lay within a separate
graphical file, included after the word ''us'' in the third paragraph of the text of the passage.
4o [O172] One Can find the '''iGRL[O2]'' Code in FigUre 21 in the foUrth line of teXt. The Small arroW 26O after the graphiC
code is a ''WORD FOR WINDOWS'' formatting command. It is a tab marker which specifies the paragraph indent.
[O173] The neXt interaCtion Code, '''iBKS[1 ],'' iS a ''BloCk Start'' Code; it Will be folloWed by a '''iBKE[1 ],'' ''BloCk End''
code. These codes set the boundary around a specified option for a specific item. At this point, it is helpful to refer
back to Figure 1 5 and note that a ''Block Set'' number is identified for each item. Where there is more than one item
45 referring to the same area of text, separate Block codes may be included. In the passage there are also block codes
'''iBKS[2]'' and '''iBKE[2]'' indiCating option boUndarieS for SeleCting a reSponSe in the third eXample item aS ShoWn in
Figure 1 8 that uses the same area of text. Thus, only those portions of the reading passage 236 which are ''blocked''
by interaction codes which are related to a specified item will be active when that item is presented.
[O174] In the fourth and seventh lines of text in Figure 1 8 there is a iHCS[2]- _CE[2] code set, indicating a highlight
5o and center. This interaction code highlights the demarcated area of the reading passage 236 and centers it when the
item is presented by the item preparation tool, as shown in Figure 1 3.
[O175] Finally, in FigUre 21 , there iS a '''iHOR'' Code that prodUCeS a horiZontal line at thiS point in the teXt When read
by the item preparation tool and an '''i END'' Code, indiCating the end of the field of CUStom CodeS.
55 f. Translation
[O176] XLATE is one of the item preparation programs listed in Table 1 and compiles RTF format documents created
by ''WORD'' into a binay equivalent. Binary conversion speeds up the execution of the text display modules embedded
15
EP O 664 O41 B1
in the Item Preparation Tool and also results in a storage savings (mostly memoy savings).
[O177] XLATE is a system program which is generally not executed by test production staff from a command line,
menu pick, icon, etc. The Item Preparation Tool runs XLATE whenever it is determined that a component file has been
changed within ''WORD FOR WINDOWS.'' This is detected by comparing the date/time stamp of the .STE, . RSP, and .
5 DIR files of the item currently displayed by Item Preparation Tool with that of their binary equivalent the .STB, . RSB,
and . Dl B files. The binary file name is created from the component file name extension by changing the last letter of
the extension to ''B.'' For example, if the stem component's name is accession.STE, the binary file outputted by XLATE
will be accession.STB. If the binary file is older, its source equivalent must have been edited while the user was in the
''WORD FOR WINDOWS''. Thus, The Item Preparation Tool runs XLATE to update the binay file by translating the
1O source code.
[O178] During conversion, errors are preferably written to an error log file in ASCll format. The error file name is
created from the template XLATE???.ERR by substituting the component file extension for the question marks. Thus,
for example, if the component file is named accession.STE, the error file name will be XLATESTE.ERR.
15 g. Data _nterfa_eg and F_ow between Tegt Deve_op_ent and Tegt Produ_tion
[O179] Figure 22 presents a flow diagram of items and keys between TD 65O and TPS 66O. As previously stated in
Section Il. B., the test developers create and select items for a particular test preferably using the ''TD/DC'' system.
However, test items may be created and selected by any test document creation system or prepared by hand as long
2O as TPS 66O is provided with the information described in section ll.B.2.a. related to item offload. Three methods of
providing the information to TPS 66O are shown in Figure 22.
[O18O] The three methods are enumerated in Figure 22, by the numerals 1 , 2 and 3 located along the path lines.
The use of the ''TD/DC'' System 652 is enumerated as path 1 . If the key descriptions are prepared by a method other
than ''TD/DC'' 652, they may be written on paper and provided to TPS 66O via path 2. Additionally, the item text,
25 presentation information and classification information which are collectively referred to as the item description, may
also be determined by test developers who do not use the ''TD/DC'' system, but prepare this information on paper as
shown by path 3.
[O181] Test developers create and select the items to be included in a test as shown at 65O. If the test developers
use the ''TD/DC'' system, they execute item Offload to produce a diskette having the files containing the item description
3O and respective keys as shown at 652 and 658. The diskette is then sent to TPS as shown at 66O. If the files containing
the key information are not generated by the test developers using the ''TD/DC'' system, but rather by another method,
the test developers may provide a written key description to TPS 66O as shown at 654. Similarly, test developers may
prepare a written form of the item description shown at 656 and provide the written description to TPS at 66O. If graphics
are included in the written description at 656, a CBT artist will prepare computerized graphics files at 662 and 664.
35 These item graphics component files are also provided to TPS as shown at 66O.
[O182] If the item and key description are provided to TPS 66O via diskette created during item offload, TPS invokes
the item element generator at 666. The component files are separated as described above and filed in the an Item
Preparation (IP) database at 67O. If the item description or key description is presented to TPS 66O on paper, TPS
must manually enter the information using the IPT and word processing at 668. Then, the component files created by
4O TPS 66O are stored in the IP database 67O. Once all of the necessay component files are stored in the IP database
67O, TPS uses the IPT (ltem Preparation Tool) and word processing to prepare the computerized version of each item
at 668. Component files may be replaced in the l P database after being edited as shown at 67O and further processed
at 668 using IPT or word processing.
[O183] When TPS is satisfied with the computerized version of the item, the test developers may view the items as
45 they will be presented to an examinee as shown at 672 and 674 using the lVT. lf the test developers desire changes
to the items as presented, they can provide revision information to TPS Staff via paths 2 or 3 and the whole cycle may
be repeated until items are completed.
3. Test Preparation
5O a. Overview
[O184] Figure 23 shows the functions performed by TD and TPS and the software which is used to perform these
functions for preparing a computerized test. Test developers assemble the test as shown at 682. As shown at 686,
55 item selection is preferably automated (AIS) using the ''TD/DC'' system or an equivalent test document creation system.
Using ''TD/DC'', test developers enter the test specifications into the ''TD/DC'' system. Based on these specifications,
''TD/DC'' searches its central database for items which satisfy the test specification, e.g., 5O math questions, 25 of
which are algebra problems and 25 which are geomety problems. Then, the test developers review the items selected
16
EP O 664 O41 B1
by ''TD/DC'' for sensitivity and overlap constraints described in the background section. If the test developer decides
that the sensitivity or overlap constraints are not satisfied by the current selection of items, certain items may be des-
ignated to be replaced by another item from the database. In addition, test developers provide a test description spec-
ifying the directions, messages, timing of sections, number of sections of the test, etc. as shown at 692. If a computer
5 adaptive test (CAT) is to be run, test developers may run a computer adaptive test simulation at 684 which are known
to skilled test developers.
[O185] Using the Test Preparation Tool (TPT) and TOOLBOOK 696, TPS prepares the test level components as
shown at 7OO. TOOLBOOK is commercially available from Asymetrix Corporation. The test level components include
scripts 71 6, item table block sets 7O6, general information screens 7O8, direction screens 71 O, message screens 71 2,
1O and tutorial units 71 4. Each of the test components will be described in detail below.
[O186] As the components are prepared, the TPT stores them in a TPS networkdirectory 7O2. Then, the components
are entered into the TPS Production database 7O4. The components stored in the TPS Production database 7O4 will
be retrieved during test packaging which is described below.
[O187] A script consists of a series of files and further specifies the option settings and configuration data which the
15 Test Delivery Application (TDA) needs for operation. Option settings are specified for the Test Delivey System to
determine whether a certain feature is enabled for the test. Most option settings are simple yes/no declarations, but
some offer a limited set of choices (e.g., mouse speed_. slow, medium, fast). Configuration data is highly variable in-
formation such as the section name to be displayed in the Title Line during a test session or the list of items to be
displayed.
2O b. S_ripts
[O188] During test preparation, scripts 71 6 are prepared and combined with the items prepared during item prepa-
ration. Scripts control the sequence of events during a testing session. Two types of scripts 71 6 are preferably used_.
25 the session script 71 8 and one or more test scripts 72O. The session script 71 8 controls the order in which units within
the testing session are presented. Units provide specific services to the examinee, such as delivering a test or pre-
senting a score report. Just as the session script controls the session, the test script controls what is presented to the
examinee during the testing unit. Each testing unit may include one or more delivey units, which are separately timed
and scored subdivisions of a test. The system can dynamically select, or spiral, scripts and other test components so
3O that examinees are given what appear to be different tests. Figure 24 shows the relationship among session scripts
71 8, test scripts 72O, and units.
[O189] Some examples of units supported by the system are described in Table 5 below_.
TABLE 5
35 (PICTURE)
4O
45
5O
55 17
EP O 664 O41 B1
(PICTURE)
TABLE 5 (continued)
5
1O [O1 9O] Testing programs control the behavior and appearance of their tests through options that are enforced by the
15 Test Delivery Application. Table 6 below indicates the options that are available and the levels at which a testing program
can specify each option. The options, which are explained in the text following the table, are selectable at one or more
of the following levels_.
- test package (i.e., all instances of a particular test, such as GRE General),
_o - SeSSlOn,
- testing unit,
- delivery unit (section).
_5 (PICTURE)
TABLE 6
3O
35
4O
45 [O1 91] At the package level, options that are in effect for the entire package are defined in a Package Profile file.
Some examples include the program name, which may appear on the title line of the test. Another option may indicate
whether the program permits administrators to restart sessions after an examiner terminates the test. Additionally,
5o options may list the session scripts to deliver the test.
[O1 92] The session script is the second-level component of the testing package. It performs two primary functions_.
First, it specifies the Session Control Information, which defines the default options that are in effect for the entire
examinee testing session. Second, it controls the order in which units within the testing session are presented and the
options used to present them. The units that can be presented within a session script are_. General information screen
55 units, Tutorial units, Break units, Data collection units, Scoring and reporting units, and Testing units.
[O1 93] The session control information contains the default options in effect for the entire session. Control information
can be provided at multiple levels within the testing session. Thus, the control information provided at the session level
can be overridden by information that occurs later in the session. The information provided at the session level would
1 8
EP O 664 O41 B1
generally include the following_. Name - the session script name to be used by administrators in selecting a specific
session script from Administrative Application menus; Input device - the input device to be used during the session (e.
g. , mouse or keyboard); Color - the colors to be used during the session; Messages - program-specific messages to
override default messages during the session; Demo Script - indicates whether the script presents a demonstration or
5 operational test; Research Indicator -indicates whether the script presents a research pilot test; Special Timing - indi-
cates whether the script is standard or specially timed version.
[O1 94] The Gl S unit allows the incorporation of a single screen of information in the session. It may contain, for
example, the following information_. reference to the actual text and graphics that will be presented on the examinee's
screen; the type of dismissal_. automatic or manual; the time after which dismissal should occur.
1O [O1 95] The tutorial unit presents test familiarization materials to the examinee. It may contain, for example, the fol-
Iowing information_. reference to a tutorial, which is the familiarization information that will be presented on the exam-
inee's screen, e.g., ''Howto Use a Mouse'', ''Howto Scroll'', ''Howto Use the Testing Tools'', ''How to Answer''; information
that controls the content of the tutorial--the information varies depending on the tutorial selected. For example, for the
''How to Use the Testing Tools'' and ''How to Answer'' tutorials, the specific tools and item types to be covered must be
15 defined. The mouse and scrolling tutorials are generic--no content information is required when those tutorials are
selected; an indicator for first occurrence--this indicator applies only to the Testing Tools and How to Answer tutorials.
Introductory information is presented during the first occurrence of these tutorials and is omitted if they occur again
Iater in the session.
[O1 96] The break unit is used to implement a scheduled break within a session. It contains the following information_.
2O reference to a break procedure, which is the actual text and graphics that will be presented on the examinee's screen
during a break, along with the program that controls their presentation. It should be understood that multiple procedures
could be supported; the length of the break.
[O1 97] The data collection unit is used to obtain additional information from the examinee, such as demographic or
debriefing information. It is preferably implemented as a special instance of the testing unit, in which no scoring is done.
25 Like the testing unit, a data collection unit references a test script, which control as the sequence and options of the unit.
[O1 98] The scoring and reporting unit provides for scoring, and optionally reporting, the results of one or more testing
units delivered in a session. If the testing program selects a Display Scores option, it preferably displays all traditional
score types including raw, percent correct, converted and composite scores. If testing programs select the Cancel
Scores option, the examinee will be given the option of cancelling the scores.
3O [O1 99] The scoring and reporting unit preferably invokes the Educational Testing Services SKM (Scoring and Key
Management) routines to return the following information_. the score name for insertion into the score report, such as
''Reading'' or ''Antonyms''; the score type for insertion into the score report, such as ''number right,'' ''percentile, '' or
''converted score''; the score value, such as ''65O'' or ''passed''. It should be understood that any automated scoring
system which provides this information can be used or the information may be provided directly by a user.
35 [O2OO] The information needed to display a score report is preferably identical to that required for a message screen_.
reference to the actual text and graphics to be presented on the examinee's screen.
[O2O1] The testing unit presents a test, based on the contents of a test script that may have been selected at runtime.
The following units can be included within a testing unit_. general information screen unit; tutorial unit; break unit; delivey
unit, which delivers items to the examinee. This permits testing programs to interleave general information screens,
4O tutorials, and breaks with sections of a test. The testing unit contains the following information_. script selection mode
indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which
controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used,
the reference is to a set of test scripts.
[O2O2] Like the session script, the test script performs two primary functions. First, it specifies the test and delivery
45 unit control information. Test control information defines the options that are in effect for the testing unit. Delivery unit
control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the
order in which units are presented within the testing unit and the options used to present them. The rules for presentation
of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included
within a test script.
5O [O2O3] At least three delivey modes are preferably supported within one or more testing units_. linear, adaptive, and
essay. The test script references different components depending on the delivery mode of the test, but in all cases the
end result is a reference to a specific item or essay topic to present to the examinee. Multiple delivery units can be
used to organize the testing unit into sections, and each delivey unit can present a different mode of test.
[O2O4] The test control information includes the following information, which is in effect for the testing unit_. Logical
55 Name - the name used to associate the testing unit with a scoring specification provided by the SKM system or an
equivalent thereof; Directions - a reference to the text and/or graphics to be presented as general directions; Sections
- the number of sections within the test; create score data - whether scoring data for online scoring is to be created for
this testing unit; Message Overrides - any program-specific messages that are to replace the default messages within
1 9
EP O 664 O41 B1
the testing unit. The test control information can temporarily override the options specified in the session control infor-
mation.
[O2O5] In addition to the test control information, a test script can also contain delivey unit control information. The
delivey unit control information can be specified to change the options in effect for the duration of the section. The
5 following information is preferably included in the delivery unit control information_.
Type Indicator - whether this delivey unit delivers a test or a section; Directions - if the delivey unit is a section,
reference to the test and graphics of the section directions; Title Line Test - defines the name that will be used in
the title line_,
1O Mode - the delivery mode of the test or section, e.g., linear, adaptive, or essay;
Timing - timing options in effect for the test or section, including whether the test is untimed, timed in its entirety,
or timed by section, and whether timing begins when directions for the test or section are displayed or when the
first screen after the directions is displayed;
Scoring - whether scoring data for online scoring is to be created for the delivery unit;
15 Messages - program-specific messages to be used to replace the defaults during the test or section_,
Item Information - specifies how the times in the test are organized and where to find them. The contents of the
item information varies with the mode of the test being offered. For example, the item information for linear tests
is generally a reference to an item-by-item listing of the items to be presented, called an item table. Essay tests
may reference an essay procedure and a topic pool if more than one essay topic is provided for selection by the
2O examinee. Whether more than one essay is to be stored or only one is chosen for scoring is also defined. For
adaptive tests, reference to an adaptive algorithm and an item pool should be specified.
Testing Tools - the testing tools available during the test or section. Examples of testing tools are described in
detail in Section Ill(E);
Explicit Prompting - whether or not explicit prompting is to be used to make sure the examinee supplies exactly
25 the required number of responses_,
Must Answer - whether or not examinees should be allowed to move off an item without providing a response.
[O2O6] Detailed flowcharts and corresponding pseudo code for the TPT application are provided in Appendix B.
3O 4. Test Pa_kaging
[O2O7] After all of the items have been constructed for computer delivery by the test production staff and approved
by test developers and the scripts and tutorials have been created, the test production staff packages all of the relevant
files together using the Test Packaging Tool (TPAK) and the Score Key Management (SKM) system. In a preferred
35 embodiment, this process requires three steps. First, the test components are combined into a draft test package so
that the flow and presentation of the test may be reviewed. After the draft test package has been reviewed, the test
components are formed into a blue-line test package. After the blue-line test package undergoes successful quality
assurance tests, locks are applied so that data cannot be altered in the approved test packages. The final tests are
then distributed to test centers.
4O [O2O8] Aflowchart depicting the steps executed by TPAK and SKM to package a computerized test is shown in Figure
25. First, TPAK is used to create a delivery package at 6O1 . This step involves creating a presentation database which
incorporates the presentation information from the test scripts, creating a presentation parts lists which lists an ID for
each component used to create the presentation database and creating other files subsequently used by SKM. An
SKM database is created at 6O3 from the files generated by TPAK. These files preferably contain the item table de-
45 scribed above and item scoring information. The SKM database and the presentation database are then combined by
TPAK at 6O5 to produce installation files for distribution. After all quality assurance procedures have been performed
and satisfied, TPAK preferably applies a lock on all of the installation files as depicted at 6O7. After this lock is applied
a new test version should be created if changes to the test package are required.
[O2O9] Figure 26 shows the components of a final test package. Four primay groups of files are packaged together
5O to form the final test package. These four groups of files are the Profile and lndex files 8O1 , the Presentation Binary
Large Object (BLOB) 8O2, the SKM BLOB 8O3, and the Problem Item Notification (Pl N BLOB) 8O4.
[O21 O] Although the SKM BLOB 8O3 and Pl N BLOB 8O4 are shown in Figure 26, they are not necessary components
of the test package. However, they are preferably included in the test package. The SKM BLOB 8O3 should be included
if the examinee responses are to be scored at the testing center after the examinee has completed the test. If the SKM
55 BLOB 8O3 is not included in the test package, the examinee responses are scored at the central processing site by a
program specific (e.g., SAT, GRE) postprocessing system. The Pl N BLOB 8O4 is used to identify items which had been
included in the test but which are later determined either not to be scored or administered to the examinee. Thus,
although the Pl N designations provide a preferable feature, its inclusion in the final test package is not necessary. The
2O
EP O 664 O41 B1
details of the SKM and Pl N BLOBs will be described below.
[O211] The profile and index files 8O1 include the Package Parts List File (PPL) 8O5, the package profile file (PP)
8O6, and the BLOB Index Files 8O7. The PPL 8O5 contains a list of identification codes and version numbers, each l D
code and version number being associated with a component in the test package. The PP file 8O6 contains an identi-
5 fication of each test included in the test package. Although only one test is actually included in a test package, multiple
scripts may be provided for dynamic selection (spiralling) orforspecial conditions (e.g., untimed versions for examinees
with disabilities). Spiralling is atechnique in which test components (e.g., item substitution) are varied sothat examinees
taking the same test appear to be taking different tests. Spiralling inhibits cheating among examinees whose worksta-
tions are in close physical proximity because it increases the likelihood that each examinee will be interacting with a
1O visibly different test. The BLOB index files 8O7 provide an index for the Presentation BLOB 8O2, SKM BLOB 8O3, and
PIN BLOB 8O4 which are packaged into the computerized test. These indices function as a guide for locating specific
data within each of these BLOBs.
[O212] The presentation BLOB 8O2 includes item level components 8O8 and test level components 8O9. Examples
of the item level components 8O8 are shown in Figure 27 while examples of the test level components are shown in
15 Figure 27. Referring to Figure 26, the item level components 8O8 may include the item stimulus 82O, the item directions
821 , the item stem 822, the item response 823, and the item graphics 824. Each of these item level components has
been described in detail in conjunction with item preparation in Section Il.B.
[O213] Referring to Figure 27, examples of the test level components 8O9 are the test scripts 83O, general information
screens 831 , test level directions 832, message screens 833, and tutorial units 834. Each of the test level components
2O 8O9 has been described in detail in conjunction with test preparation or will be described in detail in conjunction with
test delivery in Section Ill.
[O214] Returning to Figure 27, The SKM BLOB 8O3 is preferably created by the scoring and key management ap-
plication and incorporated into the final test package. The SKM BLOB 8O3 preferably includes the Answer Keys 81 O,
scoring tables 81 1 , and scoring specifications 81 2. The answer keys 81 O provide the correct response or responses
25 to each item included in the test form. The scoring specification 81 2 provides information relating to how each item is
to be scored.
[O215] Figure 29 shows some examples of scoring tables which may be included in the SKM BLOB 8O3. The item
and custom item tables 84O, the item and custom item blocks 841 , conversion tables 842, Item Response Theory (IRT)
parameters 843, Theta estimation parameter tables 844, K-factortables 845, and Item weight tables 846 are preferably
3O included in the SKM BLOB 8O3. Since all of these tables are well known to those in the testing industry, a description
of these tables will not be provided here.
[O216] Returning again to Figure 26, the Pl N BLOB 8O4 includes pointers to pinned items 81 3 and the Do Not Score
(DNS) and Do Not Administer (DNA) flags 81 4. Some time after the computerized test has been prepared by TPS, it
may be determined that one or more items included in the test do not perform as intended, e.g., more than one response
35 would be correct although only one was intended. Thus, TD may designate these items and indicate whether the item
should not be scored or not be administered. SKM then creates the Pl N BLOB 8O4 with pointers and flags for each of
the items that has been found to misperform. When the test delivey system delivers the computerized test, it will not
present items flagged with a DNA and those items flagged with a DNS will not be scored.
[O217] Figure 3O shows the functions performed by TD, TPS, and SKM staff to implement the steps described above
4O with reference to Figure 25. Turning nowto Fig. 3O, as a part of test preparation, TD specifies the scoring specifications
and conversion tables at 776. In a preferred embodiment, SKM staff utilizes the automated SKM system developed
by Educational Testing Service as shown in Figure 3O. For instance, if the ETS SKM system or an equivalent automated
SKM system is utilized, the answer keys and tables may be retrieved from an SKM database at 77O and 756. The
scoring specifications and conversion tables are added to the data retrieved from the SKM database as shown at 758,
45 and an SKM BLOB 762 can then be created. TPS combines the test components, item components and BLOBs to
create a draft test package at 736 and 738 respectively. TD reviews the draft test package at 772 as it is delivered by
the Test Delivery Application at 778. If certain items do not perform as they should, TD identifies those items for the
SKM staff to create a Pl N BLOB 764 based on the information provided by TD. TPS adds the Pl N BLOB to the test
package and makes any other revisions the TD identified after reviewing the draft test package. When TD has author-
5O ized the test package at 78O, TPS prepares a blue line test package 74O and sets a level 2 lock on the test and item
Ievel components. A level 2 lock prevents modification to the locked components by unauthorized persons. After the
blue line test has been finally authorized by TD at 78O, TPS creates a set of data distribution disks at 742 and applies
a level 3 lockat 748. A level 3 lockvirtually eliminates the potential for any changes to be made to the computerized test.
[O218] Like the l PT and the TPT, TPAK is preferably a menu-driven application. Detailed flowcharts and correspond-
55 ing pseudo code of the TPAK application are provided in Appendix C.
[O219] In a preferred embodiment, a modified version of TPAK called ETPAK (Encypted TPAK) is executed. In
addition to verifying the presence of each of the required test files, ETPAK encypts at least the item files (e.g., .STE, .
REF, . RSP, and .CTL). After the required files have been packaged together, they may be transferred to a test center
21
EP O 664 O41 B1
at which the computerized test is administered and delivered to an examinee.
Ill. The Test Delivery System
5 A. Overview
[o22O] A block diagram of the Test Delivey System 1 2 is shown in Figure 31 . A test delivery application (TDA) 51 O
controls the test session, as directed by the test program 51 4, CBT files 51 6, and test delivery application data (TDA
data) 51 2. The test program 51 4 and CBT files 51 6 are administration system files and are preferably stored on the
1o work station or server (if workstations are networked via local area network) hard disk priorto delivery of a computerized
test to an examinee. Other files and applications such as the HELP facility 526 and the REVl EW facility 528 are also
preferably stored on the hard disk in advance. An examinee performance file 522 is created during each test session
to record an examinee's responses and other activity during the test session.
[O221] Although the Administration system will be described in detail below, a brief description is provided here as
15 it is relevant to the implementation of the Test Delivery system.
[o222] The center administrator uses a combination of manual and computer procedures to control operations and
deliver tests to examinees at the testing center. All of the files and applications shown in Figure 31 are sent to the
testing center in electronic form. All of these files can be loaded, on the workstations using standard set up procedureS.
The administrative application 51 1 of the Administrative system permits the administrator to initialize each workstation
2O at the start of the day, to sign an examinee onto a workstation, to start a testing session, and to close each station at
the end of the day. The center administrator is also responsible for performing backup procedures and transmitting the
examinee performance files 522 to the central processing site for scoring and evaluation.
[O223] Figure 32 provides a high level flow diagram of the test delivey procedure. In a preferred embodiment, when
the administrator completes the procedure to sign an examinee onto a workstation and selects a test at 5OO, the TeSt
25 Delivey Application is invoked. The Test Delivery Application reads the session script and executes the units it pre-
scribes. When the end of the session script is reached, the Test Delivery Application returns control tothe Administrative
Application.
B. Test Delivery Appli_ation Data Flow
3O [O224] Scripts define the sequence of tasks to be performed by the TDA 51 O as well as the information necessaY
to complete each task. The scripts define option settings, files containing program-specific text, and the items to be
displayed.
[O225] Although the following section provides a detailed description of the operation and use of the Test DeliveY
35 System and the presentation of various screens by the Test Delivery system, detailed flowcharts and correspondin9
pseudo code of the TDA are provided in Appendix D.
C. Title Line
4o [O226] The screen format during a delivery unit is preferably divided into three main sections; a title line 225O, an
item presentation area 2252, and a primary control area 2254 as illustrated in Figure 33. The title line 225O is preferably
presented as one solid gray barat the top of the screen. It should be understood that numerous other color combinationS
are possible. The title line 225O is capable of displaying various information relating to the test or taking the test. For
instance, the title line 225O may include the time remaining in the test or test section. Preferably, time is displayed
45 automatically although an examinee may optionally turn it off. In a preferred embodiment, remaining time is displayed
Ieft justified on the title line 225O in HH_.MM format until the last few minutes of the test. At that point, the display format
changes to MM_.SS and flashes for three seconds so that the examinee is alerted that the time remaining for taking
the test is nearly over.
[o227] Other information in the title line 225O may include the name of the computerized test and program-specified
5O text pertinent to what is being presented (e.g., section name). Information to help orient the examinee is also preferably
displayed in title line 225O. For instance, when an item or tutorial screen is displayed in the presentation area 2252,
the notation ''xx of yy'' or ''xx'' appears in title line 225O. The ''xx'' refers to the item number within the test or section,
or the screen within the tutorial. The ''yy'' indicates the total number of items in the test or section, or screens within
the tutorial. Additional orientation information to be provided to the examinee in the title line 225O may include the
55 descriptive word such as ''HELP'' ''REVl EW'' and ''Dl RECTlON'' to indicate a currently displayed screen. The ''HELP,''
''REVIEW'' and ''DIRECTlONS'' screens will be described below.
22
EP O 664 O41 B1
D. Presentation Area
[O228] The presentation area 2252 of the screen is used to display items screen such as the text and graphics and
non-item screens such as direction screens, message screens, HELP screens and REVl EW screens. Direction screens
5 are used to display directions for the test, section, and others. Message screens display information and examinee
options at transition points during the test session to control the flow of the test. Transition points indicate where a new
section or new item is to be displayed or when the test delivery application moves from an item screen to a non-item
screen. HELP screens and REVIEW screens respectively enable an examinee to interact with the HELP and REVIEW
facilities. These non-item screens will be described in more detail below.
1O 1 . S_reens
a. Item S_reens
15 [O229] Item screens are used to display items. Items can be mapped into the presentation area using one of a
predetermined number of standard templates. (See Table 3, Presentation/Template Menu in the description of the Item
Preparation System). In a preferred embodiment, the templates provide different combinations of one-, two-, or three-
pane arrangements resulting in seven possible templates such as those shown in Figures 34(a)-(g). The panes are
placed in the presentation area 252 like tiles which butt against each other but do not overlap. These arrangements
2O provide test developers with some flexibility in designing item layouts.
[O23O] In preferred embodiments, the text/graphics within a pane automatically becomes vertically scrollable if the
volume of information is larger than the pane size. Horizontal scrolling can also be supported although it is not required
for operation of the invention. A scrolling pane has a vertical (industry standard) scroll bar on the pane's right side. In
addition, a status bar may be placed at the top of the pane. The phrase ''Beginning,'' ''More Available,'' or ''End'' is
25 placed in the status bar and indicates the position of the currently displayed text within the pane as the examinee
interacts with the scroll bar. ''Beginning'' is displayed if the topmost information is visible in the pane. ''End'' is displayed
if the bottommost information is visible. ''More Available'' is displayed if neitherthe topmost nor bottommost information
is visible.
[O231] In preferred embodiments, the default size of a pane is 5OOXo of the presentation area. Thus, a two pane
3O arrangement cuts the presentation area into two equally sized panes -- either horizontally or vertically (see templates
shown in 34(b) and 34(e)). A three-pane arrangement is produced by simply cutting one of the panes that result from
a two-pane cut into two equally sized panes (see templates shown in Figures 34(c), (d), (f) and (g)). It should be
understood that the default size and ratios can be adjusted to create templates having panes of substantial size.
[O232] The components of an item can be mapped into any pane of a template, but preferably panes are not empty.
35 Additionally, components can share panes and can be placed in any order. Thus, for example, the stem and response
can be assigned to the same pane.
[O233] If a stimulus component exists, the test developer can elect to have a stimulus status bar placed at the top
of the pane containing the stimulus component. The status bar displays the status as the stimulus pane scrolls. The
stimulus status bar contains the phrase ''Questions xx to yy'' flushed left, where ''xx'' is the item number of the first item
4O to which the stimulus applies and ''yy'' is the last item to which it applies.
b. Dire_tion S_reens
[O234] Directions screens are typically used to display test, section, group and set directions. The directions may
45 contain text and/or graphics which are specified by the test script (i.e., declarations in test, section, group or set con-
figuration files). Scrolling is used to navigate through directions.
[O235] There are a number of ways to map the item directions component, i.e., accession. Dl R, into the item's pres-
entation area 252. For example, directions may be embedded above or below another component pane which may
contain stem, response, or stimulus. Alternatively, directions may be placed in a special directions pane that is inserted
5O below the title line 25O and above the template. Preferably, the directions pane fills the entire width of the presentation
area 252, and the vertical height of the directions pane is adjustable to fit in the presentation area along with any of
the template arrangements of Figure 34 (a)-(g). Vertical scrolling is preferably supported if there are more directions
than the pane can display. In a preferred embodiment, the size of the directions pane is specified and devoted to the
template. Template sizing rules are then applied as described above. Still further, item directions can be displayed
55 immediately before the item is displayed for the first time in a standard directions screen. _enerally, test and section
directions will only be displayed in a single directions screen.
[O236] Figure 35 is an example of a section directions screen. A single button, DISMISS Dl RECTlONS 265 dismisses
the directions screen when the examinee selects it. Once the examinee dismisses the directions, they are preferably
23
EP O 664 O41 B1
accessible through the HELP facility. If an examinee tries to dismiss the directions before scrolling to the end, a warning
message is preferably displayed. The message as shown in Figure 36(a) notifies the examinee that the directions
should be read completely and that they can later be retrieved through HELP.
[O237] In preferred embodiments, test directions are displayed as the first screen of the delivey unit. Test directions
5 notify the examinee, for example, the number of sections, misconduct notification, test administration instructions, and
break policy during the test session.
[O238] If the test contains sections, one set of section directions should be provided for each section. Section direc-
tions are preferably delivered at the start of each section. They include the number of items in the section, the time
allowed for answering the questions presented in each section, and the reference aids that will be required throughout
1O the section.
[O239] Group directions are used to introduce items of a like type; for example, analogies or antonyms. Group di-
rections are automatically displayed in a preferred embodiment upon displaying the first item of the group by the test
delivey application. Group directions are typically displayed once and eliminated thereafter. An optional paraphrase
may be associated with the group directions. The paraphrase is used to emphasize items on the REVl EW screen as
15 will be described below.
[O24O] Set directions are used to introduce a set of items that share the same stimulus material, e.g., an illustration
or reading passage. Set directions are also tied to specific items sharing the same stimulus material. Set directions
are preferably displayed when the first item of the set is to be displayed. Set directions are typically displayed once
and eliminated thereafter. An optional paraphrase may also be associated with the set directions. The paraphrase is
2O used to emphasize items on the REVl EW screen as described below.
_. Message S_reens
[O241] Message screens appear automatically in preferred embodiments at transition points and contain one or more
25 option boxes. The examinee should not continue interacting with the test until dismissing the message box by choosing
one of the options. When a message screen is displayed, clicking in any other location on the screen should be ignored.
[O242] Message screens consist of at least the message title line, text/graphics, and button icons. The title and icons
are fixed, but message files may specify the text and/or graphics that appear on message screens. Two types of
message screens are provided. One type pops up and overlays the center of the current screen. The second type
3O covers the entire display monitor. Examples of some possible message screens are shown in Figures 36 (a)-(l) and
described in Table 7 below.
(PICTURE)
TABLE 7
35
4O
45
5O
55 24
EP O 664 O41 B1
(PICTURE)
TABLE 7 (continued)
5
1O
15 d. HELP S_reens
2O [O243] The HELP screen is mapped intothe item presentation area wheneverthe examinee selects the HELP facility.
Figure 37 shows the format of a HELP Screen. Help buttons 28O are available from the HELP screen. The examinee
can display directions, scrolling instructions, etc. in the pane 281 by clicking on one of the buttons. The directions
include test, section, group, set, and item directions. Examinees use scrolling to navigate through HELP screens. For
Example, when the examinee selects the TEST Dl RECTlONS button 282, information will appear in pane 281 as
25 shown in Figure 37 and the selected TEST Dl RECTlONS button 282 will be grayed to indicate the current selection.
The selection of the grayed button will have no effect.
[O244] Referring to Figure 38, when the examinee selects the ''TESTING TOOLS'' button 283, a menu screen displays
all available testing tools for that particular section which are defined by the section configuration file of the test script.
When the examinee selects one of the tools, information about that tool is presented in pane 281 . For instance, Figure
3O 3g shows a sample screen which appears when an examinee requests help on the calculator testing tool.
[O245] In a preferred embodiment, the Help facility is context sensitive. If an examinee invokes HELP from a directions
screen, a message is displayed to further instruct the examinee as to how to proceed. If HELP is invoked from an item
screen and the following display items exist, the Help pane 281 will first display the group directions for that item, then
the set directions, and lastly the pop-up directions. If no directions exist for the item, a message informing the examinee
35 that the item screen includes all the necessay information will be displayed.
[O246] To exit the Help facility, an examinee selects the ''RETURN TO WHERE l WAS'' button 285 shown in Figure 39.
e. REVIEW S_reens
4O [O24_] The Review screen is mapped into the presentation area wheneverthe examinee selects the REVl EWfacility.
Figure 4O shows that the screen presentation REVl EW buttons 286 are available to the examinee from each Review
screen. The top pane 287 contains directions on how to conduct the review. The Review pane 288 is used to display
group, set, and item paraphrases (if provided), plus item status. Item status consists of a phrase and possibly a check
mark. In a preferred embodiment, the status phrase can specify ''Not Answered,'' ''Answered,'' or ''Not Seen.'' The check
45 mark indicates whether the item was marked for review by the examinee during the test session.
[O248] Scrolling is preferably supported in the Review pane 288 if the information cannot be presented on a single
screen. Upon invoking the Review facility, the item from which REVl EW was invoked is highlighted. The examinee can
highlight a different item and then select the ''GO TO QUESTlON'' button 289 to review that item. At any point, the
examinee can select the ''RETURN TO WHERE l WAS'' button 29O to return to the point from which the REVl EW
5O function was invoked.
[O249] In a further preferred embodiment, when the examinee clicks on one of the paraphrases and if the examinee
has not previously seen the group or set directions, they are displayed. Thereafter, or if the examinee has already seen
the group or set directions, clicking on the paraphrase brings up the first item of the group or set. If the REVl EWfunction
is invoked from a directions screen, the examinee is returned to the directions screen by selecting the ''RETURN TO
55 WHERE l WAS'' button 2gO. However, when the examinee moves to an item, the directions screen from which REVl EW
is invoked is considered to have been dismissed. Once directions are dismissed, they are still preferably accessible
through the HELP facility. 25
EP O 664 O41 B1
E. Control Area
[O25O] The primay control area 2254 preferably provides testing tools for giving the examinee a degree of control
over the testing session. In a preferred embodiment, there are ten testing tools (also referred to as ''primary controls'').
5 Each tool has its own icon. Icons are pictorial representation of a function available to a user which can be activated
by selecting that icon. The icons for each of these testing tools are shown in Figure 41 .
[O251] Referring to Figure 41 , the NEXT icon 2255, the PREV icon 2256, the REVIEW icon 226O and HELP icon
2257 can be used by the examinee to move from one screen to another screen. When the NEXT icon 2255 is selected,
the examinee can move on to the next screen. Selecting the PREV icon 2256 enables the examinee to move back to
1O the previous screen. The HELP icon 2257 can be selected by the examinee to invoke the HELP facility. When HELP
is invoked, the examinee moves to a HELP screen to retrieve previously presented direction and information about
topics covered in the tutorials. The examinee is returned to the screen from which HELP was invoked when the Help
screen is exited.
[O252] The MARK icon 2259 enables the examinee to mark an item for review. In a preferred embodiment, both
15 answered and unanswered items can be marked. A marked item is indicated on an item screen by displaying a check-
mark in the MARK icon 2259. The checkmark may also appear next to the marked item in the Review screen when
the REVl EW facility is invoked after an item has been marked. The examinee can unmark an item by clicking on the
MARK icon 2259 a second time. However, preferably the examinee need not unmark items in order to leave a section.
[O253] The REVl EW icon 226O, when selected, presents the review screen to the examinee listing the items in the
2O section in the order they were presented to the examinee, along with any group or set paraphrase associated with item
and an indication of whether the item has been marked by the examinee from the item screen. The examinee then has
the ability to go directly to any item in a test section by clicking, as described above.
[O254] Preferably, the examinee can invoke the REVl EW facility from any item screen, and the REVl EW screen will
display the status of all the items in the section regardless of whether all of the items had been presented. In a further
25 preferred embodiment, the examinee may skip some items by advancing to a subsequent item.
[O255] The ERASE icon 2258 enables the examinee to reset all selected choices for the current item to their original
state. The Tl ME icon 2261 allows the examinee to turn on and off the remaining time display in the title line 225O. The
EXIT icon 2262 allows the examinee to leave the current section of the test. The QUIT icon 2263 allows the examinee
to quit the test. The CALC icon 2264 allows the examinee to use on-screen calculator.
3O [O256] A testing tool is said to exist if it appears on every screen. The existence of each of the depicted icons; PREV,
CALC, QUIT, EXIT, Tl ME, REVIEW, MARK and ERASE is specified by each test script or the section configuration file.
The NEXT and HELP tools preferably exist in all tests. Testing tools that do not exist should not appear on the screen,
and in a preferred embodiment, the location of the remaining tools is adjusted to close any gaps left by non-existent
tools.
35 [O257] Test scripts can define the existence of tools to limit the ways in which examinees can navigate through the
test. For example, a program can define a forward-progression-only test by eliminating the PREV and REVl EW tools.
[O258] A testing tool is said to be available if it exists and can be used. Preferably, a testing tool is displayed in black
if available and in gray when it is not. For instance, in preferred embodiments, the NEXT icon 2255, the PREV icon
2256, ERASE icon 2258, MARK icon 2259, and the CALC icon 2264 are available only from item screens. However,
4O the HELP icon 2257 is available from all screens. The REVl EW icon 226O is available from item screens and group or
set directions screens while the Tl ME icon 2261 may or may not be available from directions screens depending upon
whether the section configuration file indicates timing is to start before or after the presentation of directions.
F. E_aminee Intera_tion
45 [O259] An item response is defined by its class and type as provided by the test developers and implemented by the
test production staff as described above with reference to Figures 1 3 and 1 5. The following will describe how an ex-
aminee can interact with the workstation to respond to each item according to its response class and type.
[O26O] As described previously, there are three response classes; single selection, multiple selection and free re-
5O sponse. Single selection requires that the exam