Sber’s neural network, GigaChat, has demonstrated sufficient knowledge to compete with an MSc graduate in a qualifying exam in mathematics and computer science. During the test at Skoltech, the commission graded GigaChat’s performance as a 4 (“good”).

The exam was conducted in two steps. First, the model was tested using 500 unique, closed-format MMLU questions prepared by Skoltech professors and covering nine subject areas in computer science: algorithms and data structures, databases, cybersecurity, software, programming, networks and hardware, specialized software, management, and artificial intelligence. GigaChat correctly answered 76.9% of the questions across the subject domain.

The second step was an in-person exam conducted by a commission of eight doctors of technical and physical-mathematical sciences. To evaluate GigaChat’s knowledge, the experts selected three tickets from a pool of 45. Each ticket included a theoretical question to test fundamental knowledge of the subject area and an applied task. The tickets were created specifically for the test and were not available in open databases, ensuring that GigaChat applied real-world knowledge rather than searching for ready answers. To ensure an unbiased evaluation, the experts reviewed the answers, asking for clarifications and explanations. They also analyzed how well the model coped with unusual wording.

The assessment was made using a subject-based approach. For instance, when evaluating GigaChat’s performance in programming, the expert in that field had the deciding vote. The commission used weighting factors to ensure the utmost objectivity. The model scored 7 out of 10 points on the in-person exam, corresponding to a grade of 4 on the Skoltech grading scale.

GigaChat has previously passed the Unified State Exam in social studies, as well as specialized exams in medicine, finance, economics, musicology, and other fields, all of which were taken at top Russian universities.