From ’F’ to ’A’ on the N.Y. Regents Science Exams: An Overview of the Aristo Project | AI2

AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy!, but the rich variety of standardized exams has remained a landmark challenge. Even as recently as 2016, the best AI system could achieve merely 59.3% on an 8th Grade science exam. This talk reports success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90% on the exam’s non-diagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83% on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern Natural Language Processing (NLP) methods can result in mastery on this task. While not a full solution to general question-answering (the questions are limited to 8th Grade multiple-choice science) it represents a significant milesto