Background: Advice on Cochrane Diagnostic Test Accuracy Reviews (DTARs) highlights the importance of comparative test accuracy studies and encourages their inclusion. These studies measure the results of two index tests and a reference standard in the same population. The theoretical advantage is that, unlike indirect comparisons, the difference between test accuracies is not confounded by differences in study methods, populations and reference standards. There may therefore be implications for the assessment of methodological study quality. Objective: To explore problems arising in the methodological quality assessment of direct comparison studies. Methods: We will use the existing suggested approach to quality assessment in DTARs, based on QUADAS, in at least two systematic reviews containing comparative test accuracy studies and identify problems, including issues not addressed. We will propose solutions and seek wider consensus on these amongst groups like the UK and the Continental Europe Support Units for Cochrane DTARs. Results: We have already conducted methodological quality assessment in one relevant systematic review on PET and PET/CT for the recurrence of breast cancer. Although the sources and extent of bias in within-study comparisons are not fully known, in this case they appear to be less than those associated with absolute estimates of test accuracy. However, there was an additional need to consider the time interval between the two index tests as well as between the index test and the reference standard, and there may be potential issues relating to the order of index tests. The possibility of selection bias emanating from the requirement to receive two sets of index tests has been previously noted. Conclusions: The issues of methodological study quality affecting direct comparison studies do appear to differ in several important respects from those affecting single arm test accuracy assessments. They may actually be simpler to assess, but this requires further investigation.