a) that Recommendations ITU-R BS.1116, ITU-R BT.500, ITU-R BS.1284, ITU-R BT.710, ITU-R BT.1128 and ITU R BT.1129 have established some methods for assessing the subjective quality of audio and video systems;
b) that subjective listening tests permit assessment of the degree of annoyance caused to the listener by any impairment of the wanted signal during its transmission between the source and the listener;
c) that classical objective methods may not be adequate for assessing advanced audio coding schemes and that perceptual objective assessment methods are being developed for testing the sound quality of sound systems;
e) that the introduction of new advanced digital audio systems exploiting psycho-acoustic properties, especially with small impairments, requires advancements in subjective assessment methods;
f) that Recommendation ITU-R BS.1116 has established a method for the evaluation of the sound quality of sound systems which introduce small impairments;
g) that the detailed procedures of Recommendation ITU-R BS.1116 can be time-consuming and comparatively expensive to carry out and therefore a pre-selection method for rejecting systems which introduce large impairments is desirable,
1 that the testing, evaluation and reporting procedures given in Annex 1 be used as a means of distinguishing between those audio systems which introduce large impairments and other systems.
This Recommendation is based on Recommendation ITU-R BS.1116. It differs mainly in the section referring to test method.
Recommendation ITU-R BS.1116 – Methods for the subjective assessment of small impairments in audio systems including multichannel sound systems – is intended for the assessment of audio systems which introduce impairments that can be hard to detect. For that purpose, it is necessary to control as many of the test parameters as possible and to carry out a sufficient number of assessments to produce statistically reliable estimates of the impairments. The requirements of Recommendation ITU R BS.1116 are consequently stringent. The procedure is costly in time and effort and is unnecessary if the systems under test produce significant impairments.
In order to avoid carrying out such tests unnecessarily, a pre-selection methodology that can reliably reject systems introducing large impairments is desirable. This reduction in the extent of the test method may reduce its sensitivity, but the method must still be capable of distinguishing between systems which introduce large impairments and those that do not.
2 Experimental design
In designing the tests, the considerations of Recommendation ITU R BS.1116, § 2, should be taken into account.
3 Selection of the listening panel
For the selection of the listening panel, the considerations of Recommendation ITU R BS.1116, § 3, should be taken into account.
However, to maximize the sensitivity of the test and minimize the number of trials required to obtain reliable results, the expert listeners should be thoroughly familiar with the types of impairment expected. That is, they should preferably be specialists in the assessment of the types of coding systems under test.
4 Test method
For the details of the test method, the general considerations of Recommendation ITU R BS.1116, § 4, should be taken into account. As the test procedure is the most time-consuming aspect of Recommendation ITU-R BS.1116, it is by reduction of the test methodology that most savings in time and effort can be made. At present, the following methods can be recommended.
If it can be shown that the systems under test can be properly assessed using headphone listening, then the assessments may be carried out by more than one subject at the same time. This excludes systems which inherently require the use of loudspeakers, any systems or tests which involve the acoustics of the listening environment and systems involving any spatial effects which cannot be properly reproduced by headphones.
Using subjects in groups, individual switching is not practical and it is not possible to randomize the order of presentation of stimuli for each test subject within a group. Between groups, different orders of presentation should be used. The principle of double-blind testing should be maintained.
The data analysis should be carried out according to Recommendation ITU R BS.1116, § 9, as far as is appropriate. The analysis should take into account the sub-division of the test subjects into groups.
4.2 Assessment using small numbers of expert listeners
The time taken by the test procedure can be reduced greatly by reducing the number of listeners. This is only practicable with expert listeners, who have previously demonstrated high levels of accuracy and consistency in similar tests.
The limited amount of data which might result from such tests may make useful statistical analysis impossible.
The test procedure may be further simplified by using a two-way comparison. In that way, the test subjects are required to make only one assessment between two signals, one of which is known to be the reference signal. The reference signal may be impaired and it may then be appropriate to use the seven-point comparison scale (see Recommendation ITU-R BS.1284 – Methods for the subjective assessment of sound quality – General requirements). It should be noted that such a method can give only a limited indication of the reliability of the subjects.
Recommendation ITU R BS.1116, § 5, includes descriptions of the audio quality attributes that might be assessed, for monophonic, two-channel and multichannel stereophonic systems.
For the chosen reproduction devices, the considerations of Recommendation ITU R BS.1116, § 7, should be followed.
8 Listening conditions
For the relevant listening conditions, the considerations of Recommendation ITU R BS.1116, § 8, should be followed.
9 Statistical analysis
For the statistical analysis of the data, Recommendation ITU R BS.1116, § 9, should be followed as far as is appropriate.
It is important to note that, due to the methodology employed, the results from the pre-selection tests can only be used to determine whether or not to include or reject audio systems for subsequent formal tests. Only a statement about whether a system has “passed” or “failed” can be made. No further conclusions regarding the relative performances of the systems may be drawn from the results of the pre-selection tests.
The decision threshold should be set so that marginal systems are included in the “pass” category.
10 Presentation of results
For the presentation of the results, the considerations of Recommendation ITU R BS.1116, § 10, should be taken into account as far as is appropriate.
The objective of the tests is simply to distinguish between systems that produce large impairments and those that do not. The final result will be simply a statement of whether a system has “passed” or “failed”.
11 Contents of test report
The contents of the test report should follow the considerations of Recommendation ITU R BS.1116, § 11, as far as is appropriate.
* Radiocommunication Study Group 6 made editorial amendments to this Recommendation in 2002 in accordance with Resolution ITU-R 44.