Development and validation of a nomogram to predict coronary heart disease in patients with rheumatoid arthritis in northern China

Clinical characteristics

In this study, we analyzed a cohort of RA patients with and without CHD who were admitted to the Department of Rheumatology and Immunology of the First Affiliated Hospital of China Medical University from 2011 to 2017. We divided the total cohort of patients into a training cohort (n=1012) and a validation cohort (n=274). We then assessed the clinical characteristics and serological statuses of patients in the training and validation cohorts, and performed univariate analysis to examine the differences between RA patients and RA+CHD patients in each cohort, including a chi-square tests and t tests (Table 1). There were no significant differences in mean age and sex between RA patients and RA+CHD patients in either the training cohort or the validation cohort. In the training cohort, the age distribution, hypertension, anti-CCP antibody positivity, rheumatoid factor positivity, erythrocyte sedimentation rate (ESR), C-reactive protein (CRP) levels, and dyslipidemia of low-density lipoprotein cholesterol (LDL-c), total cholesterol (TC), triglycerides and high-density lipoprotein cholesterol (HDL-c) differed significantly between the RA and RA+CHD groups (all P<0.10). In the validation cohort, the age distribution, hypertension, anti-CCP antibody positivity, rheumatoid factor positivity, ESR, and dyslipidemia of LDL-c, TC, triglycerides and HDL-c differed significantly between the RA and RA+CHD groups (all P<0.10), while CRP levels did not differ between the two groups in this cohort.

Model evaluation

In order to select a high-performance model for clinical data prediction, we evaluated the F1-scores of six machine learning models: gradient boosting decision tree (GBDT), k-nearest-neighbors (KNN), logistic regression (LR), random forest (RF), XGradient-boosting (XGB) and support vector machine (SVM). Figure 1 displays the F1-score for each model in the training and validation cohorts. The LR model performed at the same level as the other machine learning models, based on its F1-score and area under the curve (AUC), indicating that the LR algorithm was an effective prediction tool for the current data (Supplementary Table 1). In the training cohort, the KNN algorithm displayed better prediction performance than the other algorithms, which may have been due to the algorithm itself, and there was an over-fitting phenomenon. However, in the validation group, the KNN algorithm exhibited poor prediction effects, indicating that this algorithm was unstable. The SVM and LR algorithms had more stable performance, and the LR algorithm could be used to construct a nomogram. Therefore, the LR algorithm was used for clinical data prediction.

Figure 1. Model evaluation (F1-score) results based on the number of features across 6 models in training group and validation group. Abbreviation: GBDT: gradient boosting decision tree; KNN: k-nearest-neighbors; LR: logistic regression; RF: random forest; XGB: xgradient-boosting; SVM: support vector machine.

In addition, we constructed a simple model based on the training and validation cohorts. We also constructed a complex model of the RA and RA+CHD patients in the training and validation cohorts by adjusting the simple model for age and sex. Based on the F1-scores and AUCs in the training and validation cohorts, the average performance of the complex model was better than that of the simple model, indicating that sex and age are associated with the risk of CHD in patients with RA. Thus, the inclusion of these factors could improve the accuracy of CHD risk prediction.

Risk factors for the development of CHD in RA patients

To further explore the independent risk factors for CHD in RA patients, we carried out a non-conditional LR analysis based on the results of our univariate analysis (Supplementary Table 2). First, we constructed a simple model of RA patients and RA+CHD patients in the training cohort. The independent risk factors for the development of CHD among RA patients in the training cohort were hypertension (OR=3.06, 95% CI: 2.26-4.15, P<0.001), rheumatoid factor positivity (OR=1.48, 95% CI: 1.04-2.11, P=0.031), high LDL-c levels (OR=1.85, 95% CI: 1.25-2.72, P=0.002), low HDL-c levels (OR=2.17, 95% CI: 1.26-3.76, P=0.006), high triglyceride levels (OR=1.41, 95% CI: 1.14-1.74, P=0.002) and a high ESR (OR=1.01, 95% CI: 1.00-1.01, P=0.001).

We also constructed a complex model of RA patients and RA+CHD patients in the training cohort by adjusting the simple model for age and sex. We found that age (OR=1.05, 95% CI: 1.04-1.07, P<0.001), hypertension (OR=2.95, 95% CI: 2.15-4.05, P<0.001), anti-CCP antibody positivity (OR=1.41, 95% CI: 1.00-1.97, P=0.047), high LDL-c levels (OR=1.95, 95% CI: 1.31-2.91, P=0.001), low HDL-c levels (OR=2.04, 95% CI: 1.16-3.61, P=0.013), high triglyceride levels (OR=1.47, 95% CI: 1.19-1.81, P<0.001) and a high ESR (OR=1.01, 95% CI: 1.00-1.01, P=0.009) were independent risk factors for the development of CHD among RA patients in the training cohort.

Development of an individualized prediction model

Our univariate analysis identified age, hypertension, anti-CCP antibody positivity, rheumatoid factor positivity, a high ESR, high CRP levels, and dyslipidemia of LDL-c, TC, triglycerides and HDL-c as independent predictors of CHD (Table 1). It is worth noting that in univariate analysis, the relationships between certain independent and dependent variables may be masked by confounding factors, so variables with P values <0.100 were entered into a multi-factor model analysis. Complicated models combined with the above independent predictors were developed into nomograms Complicated models combined with the above independent predictors were developed into nomograms (Figure 2). The figure displays the score associated with the level of each influencing factor, the personal total score (the sum of the scores for all the factors) and the predicted risk of CHD for the RA patient. For example, an RA patient who was 65 years old (48.5 points) and male (2.5 points), with hypertension (23.0 points), 2.0 mmol/L LDL-c (17.5 points), 2.5 mmol/L triglycerides (16.5 points), 2.4 mmol/L HDL-c (25.0 points), 80 mm/h ESR (12.5 points) and anti-CCP antibody positivity (10.0 points) would have a total points value of 144.5, resulting in an estimated probability of 58% for CHD.

Figure 2. RA patients developed to CHD’s nomogram. The CHD nomogram was developed in the training cohort, with serum lipids, inflammatory markers, and serological status in RA patients. Abbreviation: LDL, low-density lipoprotein cholesterol; TC, total cholesterol; TG, triglycerides; HDL, high-density lipoprotein cholesterol; RF+, positive rheumatoid factor; CRP, C-reactive protein; Anti-CCP-positive, positive anti-cyclic citrullinated peptide antibody; ESR, erythrocyte sedimentation rate.

Validation of the nomogram

The calibration curves of the nomogram for CHD development in RA patients indicated that there was good agreement between the predicted and observed outcomes in the training cohort (Figure 3A and 3B). The concordance index (C-index) of the prediction nomogram was 0.73 when the simple model was used and 0.77 when the complex model was used. Similarly, there was good agreement between the predicted and observed outcomes in the validation cohort (Figure 3C and 3D). The C-index of the nomogram for the prediction of CHD was 0.76 with the simple model and 0.73 with the complex model.

Figure 3. Calibration curves of the CHD and the model with the addition of sex and age prediction in each cohort. Abbreviation: (A) Calibration curve of the CHD in the simple model of the development cohort. (B) Calibration curve of the complex model with addition of adjusted sex and age in the development cohort. (C) Calibration curve of the CHD in the simple model of the validation cohort. (D) Calibration curve of the complex model with addition of adjusted sex and age in the validation cohort.

Sensitivity and specificity of the FRS and our prediction models

We next constructed receiver operating characteristic curves for the simple model, the complex model and the FRS (Figure 4). Based on the AUCs, the complex prediction model (AUC: 0.77, 95% CI: 0.74-0.80, P<0.000) had a better diagnostic value than the simple prediction model (AUC: 0.73, 95% CI: 0.69-0.76, P<0.000) and the FRS (AUC: 0.66, 95% CI: 0.62-0.70, P<0.000) in predicting the development of CHD in RA patients. The sensitivity of the simple prediction model (73.8%) was higher than those of the complex prediction model (63.9%) and the FRS (71.8%). However, the specificity of the complex prediction model (77.2%) was higher than those of the simple prediction model (59.2%) and the FRS (52.4%) (Supplementary Table 3).

Figure 4. Diagnostic value for FRS and the clinical prediction model to predict CHD. Abbreviation: FRS, diagnostic value for FRS to predict CHD; Pre1, diagnostic value for simple clinical prediction model to predict CHD; Pre2, diagnostic value for complex clinical prediction model to predict CHD.

Clinical use

Next, we performed a decision curve analysis for the nomogram, as presented in Figure 5. The decision curve demonstrated that if the threshold probability of a patient or doctor was <19%, using the nomogram to predict CHD was more beneficial than either the treat-all-patients scheme or the treat-none scheme. Within this range, the net benefits of the simple and complex models were comparable, with several overlaps. However, when the threshold value was >19%, the net benefit of the complex model (blue line) was higher than that of the simple model (red line) in predicting the risk of CHD in RA patients.

Figure 5. Decision curve analysis for serum lipids, inflammatory markers, and serological status in RA and CHD patients of the simple and complex model in the training cohorts. The y-axis represents the net benefit, the x-axis represents the high-risk threshold of CHD in RA patients. The red line represents the nomogram of predictors in simple model. The blue line represents the complex model with addition of sex and age. The gray line represents the assumption that all patients have CHD.

Clinical impact curve

A clinical impact curve can be used to demonstrate the clinical effectiveness of a model by predicting the risk stratification of 1000 people with the bootstrap technique. We drew clinical impact curves for the simple and complex models to determine their cost-benefit ratios, as shown in Supplementary Figure 2. The figure displays the number of high-risk patients (the number of positive cases predicted by the model) and the number of high-risk patients with events (the number of true-positive cases). In this example, the dotted vertical line illustrates a tentative cut-off point (18% risk of CHD) at which 400 of 1000 men (40%) would be filtered by the simple model, with about 160 of these (40%) being true CHD cases. At this same cut-off point, 380 of 1000 men (38%) would be filtered by the complex model, with about 160 of these (42%) being true CHD cases.

Subjects

This retrospective study was based on the EMRs of patients admitted to the inpatient Department of Rheumatology and Immunology of the First Affiliated Hospital of China Medical University between January 2011 and December 2017. The EMRs were classified and coded in accordance with the 10^th edition of the International Classification of Diseases. The code for RA is M05.x-06.x, and the code for CHD is I25.x.

All patients fulfilled the American College of Rheumatology 1987/2010 criteria for RA and/or the American Heart Association guidelines for CHD. RA+CHD patients met the following inclusion criteria: (1) conformity to the above RA diagnostic criteria; (2) conformity to the above CHD diagnostic criteria; and (3) over 18 years old. The onset times of RA and CHD were judged based on the durations of RA and CHD recorded in the EMRs, as well as the epidemiological investigation of the patient’s disease history by telephone follow-up and professional clinicians’ assessment. If the duration of CHD was shorter than the duration of RA, we inferred that the patient had developed CHD after being diagnosed with RA, and included the patient in the RA+CHD group. RA patients without CHD met the following inclusion criteria: (1) conformity to the above RA diagnostic criteria; (2) over 18 years old. Patients were excluded if they met the following criteria: (1) presence of diabetes; (2) presence of connective tissue diseases, including systemic lupus erythematosus, scleroderma and dry syndrome; and (3) lack of laboratory assessment data such as the levels of lipid markers, inflammatory markers or antibodies associated with RA.

Based on the inclusion and exclusion criteria, 364 RA+CHD patients and 922 RA patients were included in our study. The screening flow chart is shown in Supplementary Figure 1. We divided the study cohort into a training cohort and a validation cohort, and used 10-fold cross-validation to obtain a reliable and stable model. The training cohort, a random sample of 80% of the total cohort, was used to construct the prediction model of CHD in RA patients. The remaining 20% of the total cohort was designated as the internal validation cohort and used to determine the calibration and discrimination of the risk score. Ultimately, there were 1012 patients in the training cohort (294 RA+CHD patients and 718 RA patients) and 274 patients in the validation cohort (70 RA+CHD patients and 204 RA patients).

Data collection

From the original EMRs, we collected data on demographics and health behaviors, including age, sex, disease course, menstruation (for women), body mass index, smoking habits and comorbidities. Subjects were categorized as having smoking habits (that is, long-term regular smoking of one cigarette per day) or hardly smoking (that is, never smoking or having quit smoking for one year). It is worth noting that missing data such as height, weight and menopausal history were obtained by telephone follow-up. The results of all clinical analyses of anti-CCP antibody positivity, rheumatoid factor positivity, complement component 3 levels, complement component 4 levels, the ESR, CRP levels and serum lipid measures (LDL-c, HDL-c, TC and triglyceride levels) were also obtained from the EMRs.

Framingham risk score

The FRS, which incorporates age, gender, smoking status, blood pressure, TC and HDL-c levels, was used for CHD risk assessment. We excluded the presence of CHD risk equivalents (diabetes) identified in the National Cholesterol Education Program Adult Treatment Panel III guidelines [28].

Simple model and complex model

Women are more likely to develop RA than men, possibly due to the higher estrogen levels in women. Therefore, sex is an important variable in the study of risk factors for RA. In addition, CHD is an age-related disease, so the variable of age could not be ignored in this study. Therefore, two models were constructed: a simple model that did not include the variables of sex and age, and a complex model that did. By comparing the two models, we could assess the contribution of age and gender to CHD in patients with RA.

Statistical analysis

After assembling the study cohort, we performed a univariate analysis to compare the baseline characteristics between the RA and RA+CHD groups in both the training cohort and the validation cohort. Normality tests and descriptive statistical analyses (mean, percentage, maximum, minimum, etc.) were performed on the serological indicators, blood lipids, inflammatory markers and RA antibodies, and the differences between the two groups were analyzed by the chi-square test, t test or non-parametric test.

Next, a machine learning experiment was performed based on the scikit-learn open source algorithm library. The machine learning evaluation models adopt Neusoft's automatic medical analysis platform, and the parameters are selected according to the best parameters automatically given by the platform. A Bayesian Optimization method was used for algorithm tuning, and cross-validation was used to evaluate the algorithm. In the evaluation, we set the N-folds value to 5, selected LR, SVM, RF, XGB, GBDT and KNN for algorithm comparison, used a single-experiment design for each algorithm for 30 minutes, and performed a total of three experiments. Data from the training cohort and the validation cohort were evaluated in each experiment. The evaluation indicators included the AUC, F1-score, accuracy, precision, recall rate and balance error (ber), and the final result of the experiment was three secondary experimental averages.

Based on the results of the univariate analysis and machine learning experiment, LR analysis was used to analyze the risk factors of CHD in RA patients in the simple and complex models of the training cohort. We selected prediction indicators to build the prediction models based on our univariate analysis. To provide clinicians with a quantitative tool to predict an individual RA patient’s probability of developing CHD, we built a nomogram based on the LR analysis of the training cohort. The potential associations of the prediction indicators were first assessed in the training cohort and then validated in the validation cohort. A calibration curve was drawn to evaluate the calibration of the nomogram, and the Hosmer Lemeshow test was performed. The recognition performance of the nomogram was quantified based on the C-index, which was measured by the bootstrap method with 1000 resamplings. A decision curve analysis was conducted to determine the clinical usefulness of the nomogram by quantifying its net benefits at different threshold probabilities in the validation cohort.

Using the FRS, we evaluated the risk of incident CVD events over the subsequent 10 years for all subjects based on the factors of age, sex, current smoking status, blood pressure, TC levels and HDL-c levels. Finally, we compared our clinical prediction model based on LR analysis with the FRS in terms of its sensitivity, specificity and AUC. A receiver operating characteristic curve was drawn to evaluate the clinical value of the prediction model in RA patients. All data were analyzed with SPSS 23.0 software and R software.