A Cost-Benefit Analysis of Automatic Item Generation

Automatic item generation (AIG) – a means of leveraging technology to create large quantities of items – requires a minimum number of items to offset the sizable upfront investment (i.e., model development and technology deployment) in order to achieve cost savings. In this cost-benefit analysis, we estimated the cost of each step of AIG and manual item writing and applied cost-benefit formulas to calculate the number of items that would have to be produced before the upfront costs of AIG outweigh manual item writing costs in the context of K-12 mathematics items. Results indicated that AIG is more cost-effective than manual item writing when developing, at a minimum, 173 to 247 items within one fine-grained content area (e.g., fourth- through seventh-grade area of figures). The article concludes with a discussion of implications for test developers and the nonmonetary tradeoffs involved in AIG.