Institutional prevalence of malignancy of indeterminate thyroid cytology is necessary but insufficient to accurately interpret molecular marker tests

Introduction

Thyroid nodules with indeterminate cytology frequently undergo diagnostic surgery, which confirms malignancy in approximately one-third of cases. For these individuals, a second surgical procedure is often necessary to complete total thyroidectomy (1, 2); whereas for the other two-thirds, surgery might have been unnecessary exposing the patient to the risk of complications (3). The rapid increase in the number of thyroid biopsies being performed in the United States (seven-fold in the last decade) (4) has led to a parallel dramatic increase in the number of thyroid nodules with indeterminate cytology (1). In this context, several commercial tests involving the use of molecular markers have been developed over the past several years in an effort to refine the diagnosis of these nodules preoperatively. The long-term clinical and economic implications of their implementation in routine clinical practice are unknown. However, these tests are already in widespread use in the United States and have established a place in the recently revised version of the American Thyroid Association (ATA) guidelines and in the National Comprehensive Cancer Network (NCCN) Guidelines released in the past several years (5, 6).

The clinical utility of these tests relies on the negative and positive predictive values (NPV and PPV) attained, which are determined both by the intrinsic sensitivity and specificity of the test, and by the prevalence of malignancy (PoM) within the tested indeterminate categories. As the actual PoM for the indeterminate categories defined in the Bethesda system for reporting thyroid cytopathology (Bethesda) is highly variable between institutions, the performance of molecular markers tests needs to be individualized (2, 7). Because molecular marker tests are already integrated in our institution’s thyroid nodule clinical pathway, we conducted this retrospective study to analyze and optimize their use at our academic cancer center, following the recommendations of the ATA. We also describe our initial experience with the test marketed as miRInform by Asuragen Inc. (Austin, TX, USA).

Subjects and methods

In this retrospective, institutional review board-approved study, we reviewed all the thyroid cytology evaluated at our institution from October 2008 to May 2014. Cytology reports in which the biopsy was performed on thyroid tissue remnants after thyroidectomy or in an organ other than the thyroid gland (lymph node, parathyroid gland, or other structure) were excluded (Fig. 1). Cytology results at our institution have been classified according to the Bethesda system since October 2008. In nodules with multiple biopsies, only the most recent cytology report was used for the study. Final histological diagnosis was obtained from review of histology reports. Only the histology of the biopsied nodule was used to assess the PoM in each Bethesda category; other nodules within the thyroid (including incidental malignancies in nodules that had not been biopsied) were not included in the denominator or the numerator of the calculation. In four nodules included in the study, focal areas of microcarcinoma were detected within an otherwise benign appearing nodule. These nodules were included in the malignant group, but the pathological size of the malignant foci was used as the pathological size of the tumor. Because the true PoM of each Bethesda category cannot be determined unless all thyroid nodules with indeterminate cytology are resected, which is not clinically indicated in many cases, we calculated the bounds within which the ‘true’ PoM must lie for each group using three different approaches: First, we calculated the higher limit of our institutional PoM using only resected nodules for which there was a clear histological correlation. Because nodules with a cytologic diagnosis other than malignant may be more likely resected if they exhibit other high risk features, this approach may falsely raise the calculated PoM. To limit this potential bias, we then calculated the lower limit of our institutional PoM by considering as benign all nonresected nodules. Finally, we developed an intermediate PoM, considering as benign all non-resected nodules with at least one sonographic follow-up (i.e. excluding those without follow-up after the biopsy). The estimated NPV and PPV for Afirma, miRInform, and ThyroSeq v2 were calculated based on the sensitivity and specificity reported in extant publications, using Bayes’ theorem, whereas 95% confidence intervals were developed using the Student’s t-distribution. Specifically, to assess the predicted performance of the Afirma assay (Veracyte Inc., South San Francisco, CA, USA), we used the data published by Alexander and coworkers (8); for miRInform (Asuragen Inc), we used the data published by Beaudenon-Huibregtse and coworkers (9) (sensitivity and specificity for Bethesda III and IV categories were calculated from their Table 2, because they were not provided in the text); and for ThyroSeq v2, we used the data published by Nikiforov and coworkers (10, 11)

Study cohort selection process.

Citation: European Journal of Endocrinology 174, 5; 10.1530/EJE-15-1163

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Study cohort selection process.

Citation: European Journal of Endocrinology 174, 5; 10.1530/EJE-15-1163

• Download Figure
• Download figure as PowerPoint slide

Study cohort selection process.

Citation: European Journal of Endocrinology 174, 5; 10.1530/EJE-15-1163

• Download Figure
• Download figure as PowerPoint slide

We then reviewed our own institutional experience with miRInform, which was in use between July 2012 and October 2014. Because this was a retrospective study, we recognized that the pathologists reading the histology were aware of both the cytological diagnosis and the oncogene panel result at the time of histological interpretation. Two nodules were tested twice with miRInform during the study period, but, for analysis purposes, each specimen was included only once: no mutation was identified in these two nodules, on either occasion. The actual performance of this test and 95% confidence intervals were calculated and compared to the predicted performance.

Statistical analysis was performed using SAS version 9.4 (Cary, NC, USA).

Results

We analyzed a total of 2829 thyroid nodules in 2175 patients (Fig. 1). Sixty-three percent (n = 1777) of these nodules were biopsied at our institution, whereas the remaining 37% (n = 1052) were biopsied elsewhere but had the cytology slides reviewed by our pathology department. Baseline characteristics of both the patients and the nodules are shown in Table 1. The PoM for each Bethesda category was estimated on 1103 of the 1132 resected nodules after excluding 29 cases (2.6%) with unclear histological correlation. In these 29 cases, at least one focus of cancer was identified on histological evaluation, but the presence of an adjacent nonmalignant nodule and the lack of information provided in the pathology report did not permit confirmation with confidence that the malignancy represented the biopsied nodule. Surgery was performed at our cancer center for 952 nodules (86%) and elsewhere in the remaining 151 nodules; of those performed elsewhere, 108 (72%) histological specimens were available for review by our Pathology department. Overall, our Pathology group issued the histological diagnosis in 1060 (96%) of 1103 nodules included in the study. The histological diagnosis of the other 43 (4%) nodules was extracted from outside pathology reports.

The overall PoM in the resected nodules was 50%. The distribution among Bethesda categories of all the specimens included in the study, as well as the percentage of resection, the percentage of nodules with sonographic follow-up, and the PoM, is shown in Table 2. The rate of resection was 55% (188/340) in the Bethesda III category and 72% (282/390) in the Bethesda IV category. The PoM within resected nodules (the upper limit of our institutional PoM) was 30% in Bethesda III specimens and 33% in Bethesda IV specimens. If we consider “benign” all nonresected thyroid nodules (the lower limit of our institutional PoM), the PoM would be 16 and 23% for the Bethesda III and IV, respectively. Excluding nodules without ultrasound follow-up after the biopsy (our institutional intermediate PoM), the PoM would be 21 and 28% for Bethesda III and IV, respectively. The calculated NPV and PPV for the Bethesda III and IV categories for Afirma, miRInform, and ThyroSeq v2, based on these prevalence figures and published sensitivity and specificity data for each test, are shown in Table 3.

Between July 2012 and October 2014, 116 nodules were tested with miRInform. All cytological and histological specimens in this group were collected at our institution, and the diagnoses were issued by our Anatomic Pathology department in all but two nodules, in a single patient operated elsewhere. Baseline characteristics of these patients and nodules are shown in Table 1. Cytology was indeterminate in 94% (n = 109) of the nodules tested with miRInform (Bethesda III, IV, and V categories). Histological correlation was available in 63 nodules (54%), and malignancy was confirmed in 41% (n = 26) of the resected nodules. There was partial sample deficit in eight specimens (7%): RNA failure in six (rearrangements were not evaluated); partial DNA failure in one (RAS mutations were not evaluated); and RNA failure and partial DNA failure in another (rearrangements and KRAS mutations were not evaluated). A mutation was identified in 16% of the nodules (n = 19), and histological correlation was available in 84% of them (n = 16). Benign histology was confirmed in 19% of resected nodules with a mutation (false positives, n = 3), and a mutation was not identified in 50% of the cancers in this series (false negatives, n = 13). In two nodules, there was partial sample deficit (RNA failure), and therefore, not all of the mutations in the panel could be tested. Correlation between cytology, histology, and molecular results is shown in Table 4. The general characteristics of the true-positive, false-positive, and false-negative nodules are provided in Table 5. The general performance of the test was calculated for the combined indeterminate group (Bethesda III, IV, and V categories; NPV 74% and PPV 79%) and for the Bethesda III (NPV 84%) and IV (NPV 75% and PPV 77%) categories individually (Table 6). The PPV could not be calculated for the Bethesda III category because no mutations were identified in any of these specimens.

Discussion

This study reports a large thyroid nodule cytology–histology correlation, using the Bethesda system, to calculate the theoretical performance at our institution of the three most widely used molecular tests. All cytological and almost all histological diagnoses (96%) included in the study were issued by the Pathology department at our institution, which is important to ensure the internal validation of the results. However, given the retrospective design of the study, patients were managed according to the current standard of care, and therefore, histological confirmation was available in only 40% of all nodules in this series. Not having the final pathology of 60% of the nodules is a limitation of this study, but this is driven in particular by observation of most nodules with benign cytology. The rate of resection increased progressively along with the presumed risk of malignancy of each category (15, 55, 72, 90, and 86% for Bethesda II, III, IV, V, and VI specimens, respectively). These rates of resection are similar to, or slightly higher than, those reported in a recent meta-analysis on >25 000 biopsies (1).

The results of the cytology–histology correlation in this study are similar to other series (1, 2), though with several unique characteristics. We observed a higher overall PoM (50%) compared to the estimated average in a recent meta-analysis (34%) (1). This difference could be the consequence of several factors: First, many patients with an indeterminate or malignant diagnosis on cytology are referred by their physicians or self-referred to undergo surgery at our specialized cancer center. Second, many of our patients have another established malignancy, which increases the probability of second primary malignancies (12, 13, 14). Third, incidental thyroid nodules found during a 18F-fluorodeoxyglucose positron emission tomography scan are a relatively common presentation in our patient population, which also increases the probability of malignancy (15). All of these factors increase the pretest probability of cancer and may limit the generalizability of our data to the broader community. The PoM for our Bethesda V (suspicious-for-malignancy) group (90%) is one of the highest reported, reflecting the high specificity that our pathologists have for this category. Due to the high PoM observed in this category, no additional benefit from any molecular marker test is expected at our institution, and we have therefore elected not to apply molecular markers to this group of nodules and have excluded them from the analysis of test performance.

We used the mid-point sensitivity and specificity reported in previous clinical validation studies to calculate the theoretical NPV and PPV of the commercially available molecular marker tests on our population. It is important to note that these predictive values are calculated with the estimated sensitivity and specificity of single studies, each of them with limited sample size and wide confidence intervals, which could differ significantly from the true performance. This might be especially relevant in the calculations made for ThyroSeq v2 because the test metrics used for these calculations are derived from a single institution, nonblinded study, whereas for Afirma and miRInform, these parameters are derived from multicenter and blinded studies (8, 9, 10, 11). For example, the original single-center, nonblinded study of the seven-gene oncogene panel (upon which miRInform was based) performed rather better than in the more recent multicenter, blinded study, emphasizing the fact that single-center results lack external validation (9, 16).

Acknowledging the possible limitations of these calculations, theoretically, the highest NPV and PPV should be achieved by ThyroSeq v2 in our cohort. ThyroSeq v2 has been commercialized as a test able to achieve high NPV and PPV in the setting of a broad range of pretest PoM, making it useful for both “rule-in” and “rule-out” cancer for Bethesda III and IV specimens. However, the false-negative rate of ThyroSeq v2 for the Bethesda III specimens may be underestimated because most (84%) mutation-negative nodules have not been resected in the only published study to date, and therefore, the NPV might be actually lower than reported (11). Afirma, marketed as a “rule-out” test, would also reach the threshold of NPV of 95% or higher proposed by the NCCN guidelines to favor observation in lieu of surgery on the Bethesda III specimens, if we considered the calculated ‘intermediate’ or ‘lowest’ institutional PoM (6). Based on this, Afirma could also reach an NPV close to 95% in Bethesda IV specimens at our institution. Whether this rather higher residual risk of malignancy justifies the observation in lieu of surgery can only be determined on a case-by-case basis. Finally, miRInform, as a ‘rule-in’ test, would perform similarly to ThyroSeq v2 in the Bethesda IV category. However, its performance on Bethesda III specimens would be disappointingly low and similar to that achieved by Afirma. Because the original study of the seven-gene oncogene panel in which a posttest PoM of 6% for Bethesda III mutation-negative specimens was reported (16), observation has been considered acceptable in this subgroup of patients (7). However, our results, in agreement with a more recent, multicenter, blinded study using this panel, show a lower NPV that would made observation inappropriate in this setting (9).

Between July 2012 and January 2014, miRInform was used sporadically at our institution. In February 2014, it was included in the standard-of-care work-up for thyroid nodules with indeterminate cytology on our institutional clinical pathway. We present in this report the first independent study published on the performance of this commercially available test. Our observations may be impacted by the fact that nodules with a mutation identified were more likely resected (84 vs 48%). Although this strengthens the calculation of the PPV, it may also overestimate the NPV because some false negatives may not yet have been uncovered.

The most striking finding is that none of the 47 samples classified in the Bethesda III category at our institution had a mutation identified by the test, but three cancers were diagnosed in the 19 resected nodules in this group, a false-negative rate of 16%. One of these samples had a partial sample deficit (RNA failure) but proved to be an intrathyroidal parathyroid carcinoma in which none of the mutations studied would be likely to occur. The other two cancers were a follicular thyroid carcinoma (FTC) and a follicular variant of papillary thyroid carcinoma (FVPTC), without any mutation or rearrangement identified, despite confirmed sample integrity. The rate of mutations in Bethesda III specimens was lower than in the Bethesda IV category in single-center publications using either the seven-gene oncogene panel or the more sensitive oncogene panel ThyroSeq v2, in part due to the lower PoM of this category (11, 16). In fact, in the most recent publication using ThyroSeq v2, a mutation was identified in less than 7% of the 465 nodules tested with a Bethesda III diagnosis (11). However, the PoM of both categories at our institution is similar (30 and 33% for resected nodules), consistent with the heterogeneity of this cytological category in reported studies (2). This heterogeneity would be expected to impact not only the PoM but also the histological findings at the time of surgery probably causing significant differences on test performance among institutions. If comparable performance of these tests is desired among centers, the inclusion and exclusion criteria of the Bethesda III category need to be better defined and probably need to be subcategorized to improve risk stratification (17, 18, 19).

In our study, a single mutation was detected among the 18 nodules with cytologic diagnosis of Hürthle cell neoplasm (HCN). In most publications, these nodules are analyzed together with the follicular neoplasms, because they are grouped under the same Bethesda category. However, in our opinion, this differentiation is fundamental for the application of molecular markers. It is known that the most common mutations present in thyroid cancer (evaluated by miRInform) are not usually present in Hürthle cell carcinoma (HCC) (20). In the initial report of ThyroSeq, a mutation was identified in 39% of the HCCs (21), suggesting that the broader panel might detect some of these malignancies. However, this finding needs further independent and prospective validation, and these results probably would be insufficient to avoid surgery if a mutation is not identified. However, despite having a lower risk of malignancy than other follicular neoplasms, 65–90% of HCNs are classified as suspicious by Afirma (22, 23). Although the NPV in benign HCN by Afirma seems to be preserved, the cost-effectiveness in this setting is significantly challenged.

It is notable that we tested five nodules with benign cytology and found a mutation in two of them (NRAS Q61K in both). Both had a prior biopsy read as indeterminate that had prompted the repeat biopsy with sample collection for the oncogene panel. One patient had surgery that confirmed an encapsulated, noninvasive FVPTC. The other is being followed, and the nodule has not grown significantly after a follow-up of >2 years. One of the other three nodules with benign cytology, without an identified mutation, also had a prior indeterminate cytology, whereas in the remaining two nodules, it was unclear why the test was performed. We also found a surprisingly low rate of mutations in the suspicious-for-malignancy and diagnostic-for-malignancy categories, albeit with only small numbers tested (2/6, 33%). Interestingly, one of the nodules classified as malignant by the cytology (called PTC on cytology) was actually classified as FTC on histology. The cytology was reviewed after the histological diagnosis and the classification remained the same, because there were numerous oncocytic cells with prominent nucleoli, grooves, and rare pseudoinclusions identified, which made it difficult to rule out a PTC on smears alone. However, cells on the cytological specimens were identical to the FTC cells noted on the resection specimen, which was best classified as an encapsulated, minimally invasive FTC. The other five tumors were FVPTC, one of which had a mutation identified.

Despite being the most common mutation among thyroid cancers, no BRAF-V600E (BRAF) mutations were detected in this series. This is in part because most specimens submitted for molecular testing were classified as Bethesda III and IV, in which the most prevalent cancers are the FTC and the FVPTC. In fact, they represent 88.5% of the cancers in our series (n = 4 and n = 19, respectively). BRAF is not present in FTC and infrequent in the FVPTC (around 25% exhibit a BRAF mutation), and therefore, BRAF is a mutation rarely found in thyroid nodules with indeterminate cytology (24, 25). This also reflects the sensitivity that our pathologists have for the typical nuclear features of PTC, which are more frequently present in tumors with a mutated BRAF, and therefore classified as malignant by cytology (26). In our series, 58% of the FVPTC (n = 11) had a PAX/PPARG rearrangement or a RAS mutation (one of them in combination with a RET/PTC1 rearrangement), whereas no mutation was detected in the other 42% (n = 8). A RAS mutation was also found in two FTC and three benign nodules (one follicular adenoma, one Hürthle cell adenoma, and one adenomatoid nodule). Three nodules with a RAS mutation have not yet been resected. Concordant with prior reports, 79% (11/14) of the resected nodules with a RAS mutation proved to be malignant on final histology, most of them being low-risk follicular pattern cancers (9, 16, 27, 28).

In this study, we determined our institutional average PoM for each cytological category to optimize the use and interpretation of molecular marker tests. However, this is merely the first step toward an accurate interpretation, because the PoM for a given cytological category can only be accurately measured if all nodules in that category are resected. Moreover, the PoM may change over time in response to practice changes and availability of molecular marker testing. It is also critical to recognize that the NPV and PPV should really be individualized for each nodule as other factors such as the sonographic appearance or the cytological features might also impact the pretest PoM (17, 18, 19). Although our results with the seven-gene oncogene panel might not be applicable to other institutions, miRInform performed better on follicular neoplasms than on HCN or Bethesda III samples, for which it might have limited utility.

Despite all the study limitations described previously, our results remain relevant, because they highlight the need to tailor the use of these molecular marker tests to the outcomes of each institution’s cytology reporting and not only to the average PoM. In our opinion, studies like this should be conducted in any center where molecular markers are being considered for use, or where these tests are already part of routine clinical practice to guide clinicians on the selection of the most appropriate test and the interpretation of the results. Until those data are available, molecular markers should be used with caution, although patients who avoid surgery on the basis of negative molecular testing should be offered more careful and more prolonged surveillance than would be typical for patients with truly benign cytology.

Declaration of interest

Dr McIver reports receiving payments for scientific consulting services from Veracyte, South San Francisco, CA, USA, Asuragen, Austin, TX, USA, and CBLPath, Rye Brook, NY, USA (manufacturers or sellers of Afirma, miRInform, and Thyroseq V2, respectively) between 2011 and 2014. No other conflicts of interest exist. Dr Leon has received sponsored research funding from Rosetta Genomics. The remaining authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the research reported.

Funding

This research did not receive any specific grant from any funding agency in the public, commercial, or not-for-profit sector.