T/SAIAS 0013.4-2023 Active

Scope: This document specifies the evaluation framework, evaluation indicators, evaluation dimensions, and evaluation methods for general-purpose artificial intelligence (AI) large models in education. This document is applicable to the evaluation of the basic capabilities and teaching functions of general-purpose AI large models. Main technical content: This document defines a series of evaluation indicators used to assess the comprehensive capabilities of large models, including the natural language processing capabilities of the model itself, its application in various educational scenarios, its values, security, and data compliance. For each of the above dimensions, high-quality and dynamically updated evaluation data are required. 1. The evaluation framework for general-purpose AI large models includes five parts: evaluation indicators, evaluation dimensions, evaluation datasets, evaluation of the model's basic capabilities, and evaluation of security risks. 2. This document defines three first-level evaluation indicators and twelve second-level evaluation indicators, four first-level evaluation dimensions, 19 second-level evaluation dimensions, and 76 third-level evaluation dimensions. Each third-level evaluation dimension requires corresponding test datasets for the evaluation of its corresponding indicators. 3. All evaluation dimensions can be tested separately, or selected subsets of evaluation dimensions can be combined to form a test subset, or the full set can be used for combined testing. The evaluation dimensions of the model's basic capabilities and security risks can refer to existing group, industry, national, or international standards.