T/CBA 220-2021 Remote banking artificial intelligence customer service evaluation index specification

Scope

Main Technical Content: 4.1 Evaluation Indicators 4.1.1 Performance Indicators, used to measure the AI customer service’s voice recognition and semantic understanding capabilities, including voice recognition accuracy, question recognition rate, and intent understanding accuracy. 4.1.2 Operational Effect Indicators, used to evaluate the AI customer service’s ability to improve customer experience and streamline service processes, including interaction accuracy, problem-solving rate, and intelligent routing rate. 4.2 Methods of Evaluating Indicators 4.2.1 Sampling Method: A method of estimating the AI customer service evaluation indicators based on labeled interaction samples. According to GB/T 21664-2008, when the incident occurrence rate is 50% and the absolute error is 1%, the required sample size is 10,000. After completing the sample extraction, the results determined by the labeling personnel for the unit’s business processes are compared with the actual results. If they match, they are considered correct. 4.2.2 Reporting Method: A method of calculating the AI customer service evaluation indicators based on system operational data. Typical system operational data includes call volume, interaction count, customer evaluation count, and transferred call volume. 4.3 Content of Evaluation Indicators 4.3.1 Voice Recognition Accuracy 4.3.1.1 Indicator Interpretation: The proportion of correctly recognized words in the AI customer service’s transcription of valid interactions compared to the total number of words. This indicator is primarily applicable to voice interaction AI. 4.3.1.2 Calculation Approach 4.3.1.2.1 Applicable languages: primarily the national common language and characters of the People’s Republic of China (Mandarin and standard Chinese), including Arabic numerals and commonly used English. When calculating, Arabic numerals, single English letters (e.g., e-payment, ETC), and a single digit or letter are counted as one character; E

Development Information

Related Standards