Researchers develop AI that solves a matrix-based visual cognitive test

Multiple choice tests provide test-takers the ability to compare answers to eliminate choices (or guess the correct one). Each choice can be compared with the question to infer patterns that might have been missed; it’s arguably the ability to narrow down the right answer from sets of answers that’s the test of true comprehension.

Inspired by this, researchers at Tel Aviv University and Facebook developed a machine learning model that generates answers to the Raven Progressive Matrix (RPM), a type of intelligence test where the goal is to complete the location in a grid of abstract images. The coauthors claim that their algorithm is not only able to generate a plausible set of answers competitive with state-of-the-art methods, but that it could be used to build an automatic tutoring system that adjusts to the proficiencies of individual students.

RPM is a nonverbal test typically used in educational settings like schools. It’s usually a 60-item exam given to measure abstract reasoning, which is regarded as a nonverbal estimate of fluid intelligence (i.e., the ability to solve novel reasoning problems). Each question — a single problemz– consists of eight images placed
on a grid of size 3 x 3. The task is to generate the missing ninth image on the third row of the third column such that it matches the patterns of the rows and columns of the grid.

RPM combines what the researchers describe as pathways: reconstruction, recognition, and generation. The reconstruction pathway provides supervision so that each image is encoded into a numerical representation and aggregated along rows and columns. The recognition pathway shapes the representations in a way that makes the semantic information more explicit. As for the generation pathway, it relies on embedding the visual representation from the first pathway and the semantic embedding obtained with the assistance of the second to map the semantic representation of a given question to an image.

In an experiment involving a dataset of matrices problems called RAVEN-FAIR, the researchers report that their model attained 60.8% accuracy overall. “Our method presents very convincing generation results. The state of the art recognition methods regard the generated answer as the right one in a probability that approaches that of the ground truth answer,” they wrote. “This is despite the non-deterministic nature of the problem, which means that the generated answer is often completely different … from the ground truth image. In addition, we demonstrate that the generation capability captures most rules, with little neglect of specific ones.”

Beyond potential applications in education, the researchers assert that the shift from selecting an answer from a closed set to generating an answer could lead to more interpretable machine learning methods. Because the generated output may reveal information about the underlying inference process, models like theirs, they say, could be useful in validating machine logic through the implementation of AI systems.