Automated Scoring of Short-Answer Questions: A Progress Report

1.1. Automated scoring of free-test responses

Much of the early work on automated scoring of constructed responses focused on essays. Ellis Page’s work in this area dates back more than a half-century (Page, Citation1966, Citation1967). Page scored essays based on surface features that correlated with human ratings of those essays. The approach used surrogates (e.g., word count, use of punctuation) rather than more direct measures of the quality of the writing. This general approach has remained a part of some subsequent systems for automated scoring of free text, but has obvious limitations when scoring is intended to reflect the content rather than the form of the writing.

Recognizing that the use of constructed-response items was hampered by cost and the difficulties in producing reliable scores, researchers for Educational Testing Service (in collaboration with others) examined applications of what they referred to as expert systems in the 1990s. These systems were designed to score constructed responses to computer programming and mathematics problems. Braun et al. (Citation1990) showed that their best-performing system could analyze 93–95% of responses (produced by a sample of students who took the Advanced Placement Computer Science Examination) and produce correlations with human scores that were in the .83–.88 range for some problems. For other problems included in the study, the correlations were precipitously lower. Similarly, Bennett and Sebrechts (Citation1996) examined the potential for an expert system to support qualitative feedback to test takers responding to math problems on the Graduate Record Examination. On average, the proportion of agreement between the expert system and human judges was .91 when determining whether the answer was correct or incorrect. The system was much less useful for identifying the specific errors required to support feedback, although this may have been driven significantly by lower agreement between the judges on these errors.

In a review of approaches to automated scoring, Martinez and Bennett (Citation1992) note that “Perhaps the most difficult constructed response to score is one involving natural language” (p. 164). During the following decades, considerable attention was given to that problem. As we have already noted, Page’s system was based on specific quantifiable characteristics of the response. Although subsequent systems incorporated this approach, two important alternative approaches emerged as more sophisticated methods for scoring text responses. The first of these produced a measure that represented the similarity between a new response and responses that had already received human scores. This general approach was initially referred to as latent semantic analysis (e.g., Landauer et al., Citation1998).

To implement latent semantic analysis, a large corpus of relevant text is identified. The corpus is used to create a multidimensional semantic space in which every word, sentence, paragraph, or document in the corpus can be represented by a vector of numbers. The corpus will typically include hundreds of thousands of paragraphs of text, and the semantic space will have hundreds of dimensions. Previously scored essays can then be located in this semantic space and the associated vectors for new essays can then be compared to those of previously scored essays by using a similarity metric such as the cosine of the angle between the vector representing the new essay and that representing any previously scored essay. The score for the previously scored essay that represents the closest match, or the average score for some set of similar essays, then can be assigned to the new essay. This general approach was shown to be useful for scoring a range of written responses (Burstein et al., Citation2001, Landauer et al., Citation2000; Streeter et al., Citation2011; Swygert et al., Citation2003). The one application in which latent semantic analysis has been less effective is in scoring short written responses where the answer key defines highly specific concepts (LaVoie et al., Citation2020; Willis, Citation2015).

For scoring short responses in which content is critical, a second approach has been developed in which the string of words in the response is explicitly matched to concepts delineated in a key. Numerous studies have reported on this general approach (Cook et al., Citation2012; Jani et al., Citation2020; Liu et al., Citation2014; Sam et al., Citation2016; Sarker et al., Citation2019; Sukkarieh & Blackmore, Citation2009; Willis, Citation2015; Yamamoto et al., Citation2017). These studies represent a range of variations on the same basic theme. The simplest systems use automation to increase the efficiency of human judges. The approach uses a key (sometimes called a dictionary) and each time the human judges designate a response as correct or incorrect, the response is added to the key. If a subsequent test taker enters a response that matches any existing response in the key, the system scores that response without showing it to a human judge. When the total number of responses is substantially greater than the number of unique responses, this general approach can be highly efficient (Willis, Citation2015; Yamamoto et al., Citation2017). For example, Sam et al. (Citation2016) used this approach to score 15 short answer questions presented to 266 medical students and reported that each question was scored for all students in less than two minutes.

Other approaches that match to a key or dictionary rely on fuzzy matching using natural language processing techniques. For example, INCITE (Harik et al., Citation2023; Sarker et al., Citation2019) is a system that matches examinee responses to dictionaries developed by content experts using two types of fuzzy matching. The first of these is based on the Levenshtein Ratio. The Levenshtein Ratio is defined as the sum of the length of the two strings being evaluated for a potential match minus the distance between the strings divided by the sum of the length of the strings:

$\mathit{Levenshtein}\mathit{Ratio}=\frac{\mathit{Length}-\mathit{Distance}}{\mathit{Length}}.$

In this context, the “distance” between two strings equals the number of deletions, insertions, or substitutions required to transform one string to the other. This approach can identify matches when the response has been misspelled. Without fuzzy matching, each acceptable misspelling must be entered into the dictionary separately. The second form of fuzzy matching used in the INCITE system is referred to as bag of words. With this approach, matching is based on the number of words in one sample that also appear in the second sample without regard to word order. As Harik et al. note, the INCITE system was developed to score patient notes produced as part of a clinical skills assessment for medical licensure. Scoring was based on identifying key concepts in the note. The system was sufficiently effective that operational implementation was scheduled to begin within weeks of when the examination was suspended because of the COVID-19 outbreak (Harik et al., Citation2023). (Although these patient notes were much longer than the response to an SAQ, the individual concepts were similar in length and complexity to SAQ responses).

So far, we have presented three conceptually different approaches for scoring text: 1) scoring based on quantifiable aspects of the response, 2) similarity-based procedures in which responses are represented as a vector of numbers, and 3) matching to a keyed response. It is not uncommon, however, for a system to combine aspects of more than one of these approaches. For example, the e-rater system – which was developed at Educational Testing Service and deployed in combination with human raters for scoring the writing sample component of the Graduate Management Admissions Test (Burstein et al., Citation2001) – scores quantifiable aspects of a response and also includes a module that employs a similarity-based procedure. Similarly, c-rater – a system also developed by Educational Testing Service for scoring short content-based responses – uses both matching and similarity-based procedures (Leacock & Chodorow, Citation2003).

1.2. Recent advances in NLP technology

The success of the approaches described in the previous section notwithstanding, recent advances in artificial intelligence and natural language processing have provided new tools with the potential to substantially increase the accuracy of automated scoring. With these approaches, sometimes described as large language models, words, phrases, or sentences are represented by vectors in multidimensional space referred to as embeddings. These models are much more complex than those used previously (for example, those used with latent semantic analysis); they are trained on massive amounts of text and may have billions of parameters. Unlike one-hot encodings, where each word is represented by a binary vector with a single “1” indicating its presence in a specific position and “0”s elsewhere (e.g., [0,0,0,1,0,…0,0,0,1]), embeddings are dense vectors that encode information about the context and meaning of words based on their distributional patterns in the training data (e.g., [0.2,−0.5,0.1,…,0.3,−0.4]). This means that similar words tend to have similar vector representations, allowing models to generalize better across different contexts and tasks. Among other things, the increased complexity allows the systems to interpret words within a context so that a word with multiple meanings (e.g., “river bank” and “savings bank”) is interpreted correctly. For an introduction to these models for a measurement audience, we refer the reader to Yaneva et al. (Citation2023).

BERT (Bidirectional Encoder Representations from Transformers; Devlin et al., Citation2018), is one of the most widely used of these models. The advances represented by BERT raised questions regarding whether similarity-based methods utilizing this type of powerful neural architecture might outperform matching procedures for scoring SAQs. Bexte et al. (Citation2022) examined the performance of BERT for scoring short responses using a publicly available data set. The average quadradic weighted kappa across prompts was .75, suggesting the approach is promising. However, BERT has several disadvantages for scoring tasks. Firstly, while BERT does routinely output embeddings, these embeddings are for each token, which means the tensors for two given sentences (or phrases) will not necessarily be the same size. This requires token embeddings to be combined in some way (e.g., mean pooling) such that each response is expressed by the same sized vector. The results of this pooling are often unsatisfactory (Reimers & Gurevych, Citation2019).

One alternative is to use BERT to directly predict whether or not a given pair of responses belong to the same class (e.g., correct or incorrect). These predictions then can be used to quantify the similarity of responses. This computation is typically done by inputting two sentences (or in this case, responses) into the BERT model and deriving a joint embedding for the pair. The output is passed to a simple regression function to derive the final label. This label prediction approach generally outperforms the abovementioned approach using BERT embeddings but does so at a cost. First, this approach is highly inefficient, which can increase the computational time required. Second, no independent response embeddings are computed. Without embeddings that correspond to individual responses, some potentially useful analyses cannot be done such as a principal components analysis.

To address these shortcomings, Reimers and Gurevych (Citation2019) introduced Sentence-BERT (S-BERT) as a more efficient embeddings-based solution. S-BERT produces fixed-sized vectors for input sentences (in this case responses), which can then be efficiently compared using similarity measures like cosine similarity or Euclidean distance. Sentence embeddings produced by S-BERT have been shown to perform well in tasks related to semantic textual similarity, thanks to the use of siamese network architecture. Bexte et al. (Citation2022)Footnote¹ evaluated a finetuned version of S-BERT and the results closely approximated those produced with the full BERT model – the average quadradic weighted kappa dropped by only .01.

More recently, Suen et al. (Citation2023) extended the work of Bexte et al. (Citation2022) by incorporating a comparative learning procedure into a system using S-BERT to improve the similarity matching. They provided a direct comparison of this new system known as ACTA (Analysis of Clinical Text for Assessment) and the INCITE system that uses direct and fuzzy matching and found that ACTA substantially outperformed INCITE. The level of agreement with human judges ranged from .88 to .90 for INCITE and from .97 to .99 for ACTA. This makes a strong case that with current technology, similarity-based procedures may be the preferred approach for scoring short content-based responses. (A more complete description of this system is included in the method section.)

In the present study we apply the ACTA system to score responses collected as part of a high-stakes assessment of internal medicine administered to medical students. The motivation for the data collection was to evaluate the potential to use SAQs in that context, but again, the results reflect on the potential for the currently available NLP models to support the use of SAQs across the spectrum of educational measurement.

2.1. The Data Set

As we noted, the data for this study were collected as part of operational administration of the NBME Internal Medicine Subject Exam. This examination is usually administered at the end of the test taker’s internal medicine clerkship. Conditions for administration are standardized, and the score from the test typically contributes to the examinee’s clerkship grade or pass/fail decision.

To evaluate the automated scoring system, we used 71 SAQs. Each of these items had been used previously in MCQ format on the United States Medical Licensing Examination (USMLE). These items, which were part of a larger group of items selected for subsequent use on the NBME Internal Medicine Subject Exam, had been identified as appropriate for conversion to the SAQ format. The MCQs were converted to SAQs by removing the option list and appropriately adjusting the lead-in. For example, “Which of the following is the most likely diagnosis?” would be changed to “What is the most likely diagnosis?” (The item stems were unchanged.) Each examinee testing during the data collection period was presented with two of these study items. This resulted in an average of 526 (SD = 20) responses per item.

2.2. Human Scoring

Each of the examinee responses was scored by a group of content experts using the following procedure:

• Three physicians reviewed and scored a sample of responses for each item. Based on this activity they developed a scoring rubric for each item.

• Two nurse practitioners and two physician’s assistants then used the rubrics to independently score the remaining responses. (Each unique response was presented once to each of the four raters). If all four raters agreed that a response was correct (or incorrect), that judgment was used as the final score. If there was disagreement, the response was presented to the physicians, and they made the final decision.

These human scores provide the standard used both to train and evaluate the automated scoring system (For additional information about the item set and human scoring, see Mee et al. (Citation2023).

2.3. Automated Scoring

Before the ACTA system can be used to score responses, it must be trained using the human scores just described. Then, each new response i can be compared to each of the already-scored responses in the system’s training set. The response j from the training set that is most similar to response i is identified and response i is assigned the same score as response j.

Identifying the response j from the training set that is most similar to a new response i is straightforward once each response is represented by an embedding – the system simply computes the cosine similarity between each i-j pair and selects the pair with the highest similarity index. However, the success of this process will depend on the quality of the embeddings. For this reason, embeddings were optimized for this particular task by further training – or finetuning—S-BERT using the responses in the training set, which have already been scored by content experts. By incorporating these expert scores into the system at the time of training, embeddings will be encoded not only with the semantic relationships between responses but also with information on their correctness or incorrectness for a given item.

To fine-tune the model, each expert-scored response for a given item is paired with every other expert-scored response and each pair is assigned a label of 1 if both responses share the same score (i.e., both are correct or both are incorrect) and 0 otherwise. Then, the pairs are passed into S-BERT and the model is trained to minimize a contrastive loss function. This contrastive loss is minimized when responses that share the same label are pushed closer together and responses that have different labels are pushed further apart (Hadsell et al., Citation2006). Here, “closer” and “further” are measured by the cosine distance between the embeddings for each pair of responses. Cosine distance is defined as the complement of cosine similarity:

$\mathit{similarity}\left(e_1,e_2\right)=\frac{e_1^T{e}_2}{\left|\left|e_1\right|\right|\left|\left|e_2\right|\right|}$

$\mathit{distance}\left(e_1,e_2\right)=1-\mathit{similarity}\left(e_1,e_2\right),$

where e₁ and e₂ are the embeddings for a given pair of responses in the training set. The sum of the squared distance for responses with the same label and the squared difference between some margin and the distance for responses with different labels is then defined as the contrastive loss:

$\mathcal{L}\left(e_1,e_2,\mathit{Y}\right)=\left(\mathit{Y}\right){\left(d\mathrm{istance}\left(e_1,e_2\right)\right)}^2+\left(1-\mathit{Y}\right)\mathit{max}{\left(0,m-d\mathrm{istance}\left(e_1,e_2\right)\right)}^2,$

where Y is the label (1, if both responses have the same score; 0, if they differ) and m is the margin – a hyperparameter that defines the smallest distance between responses with different scores that is not penalized by the loss function. In other words, the contrastive loss function minimizes the distance between responses with the same score while trying to ensure that responses with different scores are a minimum distance, m, apart.

Once the model has been finetuned using the procedure just described, the scoring process outlined above can be applied: a new response i is passed into the system to generate its embedding and it is then compared to the embedding for each response in the training set for a given item and is assigned the score (i.e., correct or incorrect) of the most similar known response j as measured by cosine similarity provided that that similarity exceeds a defined threshold. (The importance of this threshold is discussed below.)

2.4. Evaluation of the Automated Scoring System

There are several variables that must be considered in evaluating the usefulness of an automated scoring system. The importance of each of these variables will be affected by the specifics of the assessment context. Some reflect on the validity of score interpretations and others on issues of practicality. The accuracy of the scores is clearly central, but in the context of a system like the one evaluated in this study, a threshold is used to determine if a new response will be matched to a previously scored response. With higher thresholds, the classifications will become increasingly accurate, but this may occur at the expense of leaving more responses unmatched (and, therefore, unscored by the automated system). If these unmatched responses require manual scoring by humans, the efficiency of an automated scoring system can be defined as the proportion of responses that do not require human scoring – i.e., the proportion that can be matched and scored automatically by the scoring system.

In general, the more responses that are human scored prior to training the model (i.e., the larger the training set), the fewer unmatched responses and the greater the accuracy. In this way, the training set represents the amount of information that contributes to accuracy and efficiency whereas the threshold represents the balance of information – whatever its amount – between the competing goals of accuracy and efficiency. Therefore, the size of training set and the threshold level are two related but distinct design considerations. In this study, both are varied as study conditions.

The amount of training data was manipulated by varying the proportion of available responses used to train the system. We trained a model for each of the 71 items multiple times using 20%, 40%, 60%, or 80% of the responses. (We use percentages instead of specific numbers of responses because the number of responses varied across items.) For each of these training conditions, the results were cross validated so that all responses were used to evaluate the scoring system at least once and all responses were used for training at least once. For example, in the case of the 80% condition, the first 20% of the responses would be held out for evaluation and training would be carried out on the remaining 80%. The process would then be repeated with the second 20% of responses being held out; again, training would be carried out using the remaining 80%. This process was repeated until each of the responses had been included in the evaluation sample once and – for the 80% condition – in the training set four times. With respect to the thresholds, these were varied as a study condition and included values of .98, .95, .90, .85, .80, .75, and .65. A value of .95 is typically considered reasonably conservative.

The scoring system was evaluated by assessing its accuracy and efficiency for each condition. Accuracy was measured by the level of agreement between the human scores and the machine scores for each response. To this end, we present two measures: proportion agreement and F1 scores. The former provides an intuitive measure of the extent to which the two scores are equivalent. It was calculated using the following formula:

$\mathit{Proportion}\mathit{Agreement}=\frac{\mathit{True}\mathit{Positives}+\mathit{True}\mathit{Negatives}}{\mathit{True}\mathit{Positives}+\mathit{True}\mathit{Negatives}+\mathit{False}\mathit{Positive}+\mathit{False}\mathit{Negatives}}.$

The latter measure is a commonly used index of accuracy in machine learning (Han et al., Citation2012). It represents the harmonic mean of the precision and recall:

$\mathit{F}1=\frac{2}{\frac{1}{\mathit{Recall}}\times \frac{1}{\mathit{Precision}}}.$

The value can also be expressed as

$\mathit{F}1=\frac{\mathit{True}\mathit{Positive}}{\mathit{True}\mathit{Positive}+.5\left(\mathit{False}\mathit{Positive}+\mathit{False}\mathit{Negative}\right)}.$

Both measures take on a value of one when the proportion of non-identified correct answers (false negatives) and wrongly identified incorrect answers (false positives) are zero, and a value of zero when no answers are scored correctly. Efficiency was measured by the proportion of unmatched responses for each condition. This was calculated as

$\mathit{Proportion}\mathit{Unscored}=\frac{\mathit{Number}\mathit{of}\mathit{Unscored}\mathit{Responses}}{\mathit{Number}\mathit{of}\mathit{Unscored}\mathit{Responses}+\mathit{Number}\mathit{of}\mathit{Scored}\mathit{Responses}}.$

All three of these measures were calculated using all responses for each of five threshold conditions crossed with each of four training set conditions.

In what follows, we provide aggregate results for the accuracy measures as well as the proportion of unscored responses as a function of the training set size (percentage) and the threshold. This provides a high-level assessment of the performance of the scoring system. We then report the F1 scores and the proportion of unmatched responses at the item level. This provides a sense of how the performance of the scoring system varies across items.

We then compare item difficulty (as measured by the proportion of responses scored as correct) for scores based on the automated system with those based on human judgments. This provides a first look at the extent to which the two score scales can be viewed as interchangeable. Finally, to better understand the unmatched responses, we report the proportion of unmatched responses that were judged to be correct by human judges. This shows the extent to which unmatched responses can be viewed as missing at random.