Overview
Executive Summary
This project was developed as the final submission for BMIN 5030: Data Science for Biomedical Informatics at the University of Pennsylvania. It examines three binary predictors (insurance coverage, family savings, and education level) in relation to reported HPV vaccine receipt using data from the NHANES 2021-2023 cycle.
Logistic regression and XGBoost were implemented in R as complementary classification approaches. SMOTE was applied to address the class imbalance between vaccinated and non-vaccinated respondents. Feature importance was assessed using XGBoost gain scores and logistic regression coefficients. The analysis was evaluated on a post-SMOTE split, not on an independent held-out sample.
This case study documents the original analytical workflow and its methodological limitations transparently. Because SMOTE was applied before the data split, the reported evaluation metrics should not be treated as externally validated performance estimates. A dedicated section describes what a more rigorous rebuild would require.
Central Question
Research Question
“Which selected access and socioeconomic variables (insurance coverage, family savings, and education level) were associated with reported HPV vaccine receipt in the analyzed NHANES 2021-2023 sample?”
Background
Project Context
Motivation
HPV vaccination rates in the United States remain below recommended targets, with disparities documented across socioeconomic, educational, and insurance-related dimensions. This BMIN 5030: Data Science for Biomedical Informatics final project applied classification modeling to NHANES survey data to explore whether three selected binary variables showed measurable associations with reported vaccine receipt, and to practice applying machine learning techniques to an observational public-health dataset.
Background
The National Health and Nutrition Examination Survey (NHANES) combines interview and physical examination data from a nationally representative sample of the United States civilian non-institutionalized population. The 2021-2023 data collection used the questionnaire module IMQ_L for immunization status and additional modules for insurance, income-related measures, and demographics. Because the HPV vaccination question (IMQ060) is directed at female participants and the education variable (DMDEDUC2) is collected only for participants aged 20 and older, the complete-case analytical dataset used in this project effectively represents female NHANES 2021-2023 respondents in the 20-to-49 age range with non-missing values on all four included variables. The original survey uses a complex sample design with clustering and stratification; this analysis did not apply survey design weights or design variables, which means the results cannot be generalized as nationally representative estimates.
Contributions
My Role
BMIN 5030: Data Science for Biomedical Informatics final course project. Responsible for the full analytical workflow: variable selection, dataset construction, recoding, class balancing, model training, evaluation, and interpretation. Faculty consulted during development: Dr. Fuchiang Tsui (variable selection and data complexity) and Dr. Jesse Hsu (statistical modeling approaches), as acknowledged in the project documentation.
Identify and load four NHANES 2021-2023 data modules using the nhanesA R package
Merge modules on the NHANES sequence number (SEQN) and apply a complete-case filter
Define the binary outcome: reported HPV vaccine receipt (IMQ060) coded as vaccinated or not vaccinated
Recode insurance (HIQ011), family savings (IND310), and education (DMDEDUC2) as binary predictors
Apply SMOTE class balancing using the smotefamily package
Implement an XGBoost classifier and a logistic regression model using caret and glm
Evaluate both models using confusion matrices, accuracy, precision, recall, and F1 via caret
Visualize descriptive distributions for the outcome and all three predictors
Generate feature importance outputs using XGBoost gain scores and logistic regression coefficients
Author the written interpretation, recommendations, and references for the course submission
Inputs
Data Source
Outcome Classes
2
Predictors Used
3
Models Evaluated
2
Survey Cycle
2021-2023
National Health and Nutrition Examination Survey (2021-2023)
Period
2021-2023 data collection cycle (L cycle)
Scale
Four NHANES modules: IMQ_L (immunization), HIQ_L (health insurance), INQ_L (income and savings), DEMO_L (demographics). Merged on SEQN; complete-case filter applied across all four included variables. Final analytical sample: 829 respondents (323 reported receiving an HPV vaccine; 506 reported not receiving one). Confirmed from the rendered table(merged_data$HPV_vaccine) output in the original course submission.
The HPV vaccination question (IMQ060) was directed at female participants ages 9 to 49. The education variable (DMDEDUC2) is collected only for NHANES participants aged 20 and older. Because the complete-case filter required non-missing education data, the analytical sample was effectively restricted to female respondents ages 20 to 49. The exact final analytical sample size (N=829: 323 vaccinated, 506 not vaccinated) is confirmed from the rendered table output in the original course HTML submission, not inferred visually. The family savings variable (IND310) asks about total savings or cash assets for the family and is used here as an economic proxy; it does not represent an official poverty ratio or income-to-poverty ratio. NHANES uses a complex sample design with clustering and stratification; this analysis did not apply survey design weights, so results cannot be extrapolated as nationally representative estimates.
Pipeline
Data Engineering Workflow
The HPV vaccination question (IMQ060) was directed at female participants ages 9 to 49. The education variable (DMDEDUC2) is collected only for NHANES participants aged 20 and older. Because the complete-case filter required non-missing education data, the analytical sample was effectively restricted to female respondents ages 20 to 49. The exact final analytical sample size (N=829: 323 vaccinated, 506 not vaccinated) is confirmed from the rendered table output in the original course HTML submission, not inferred visually. The family savings variable (IND310) asks about total savings or cash assets for the family and is used here as an economic proxy; it does not represent an official poverty ratio or income-to-poverty ratio. NHANES uses a complex sample design with clustering and stratification; this analysis did not apply survey design weights, so results cannot be extrapolated as nationally representative estimates.
Loaded four NHANES 2021-2023 (L cycle) data modules using the nhanesA R package: IMQ_L (immunization), HIQ_L (health insurance), INQ_L (income and savings), and DEMO_L (demographics including education).
Selected four variables: IMQ060 (HPV vaccine receipt, female participants 9-49), HIQ011 (health insurance coverage), IND310 (total family savings or cash assets), and DMDEDUC2 (education level, participants 20 and older). Renamed to HPV_vaccine, Insurance, Poverty, and Education respectively.
Merged the four data modules using left join on SEQN (the NHANES sequence number). Filtered to retain only rows with non-missing values on all four included variables. Because DMDEDUC2 collects education data only for participants aged 20 and older, the complete-case filter effectively restricted the analytical sample to female respondents ages 20 to 49. Final pre-SMOTE count: 829 respondents (323 vaccinated, 506 not vaccinated), confirmed from the rendered course output.
Recoded all four variables to binary 1/2 factor levels. HPV vaccine: 1 = Yes (ever received), 2 = No. Insurance: 1 = Covered, 2 = Not Covered. Family savings: 1 = Less than $3,000, 2 = $3,000 or more. Education: 1 = Below college (less than some college), 2 = College or above. Refused and don't-know responses were treated as missing and excluded by the complete-case filter.
Applied SMOTE (Synthetic Minority Oversampling Technique) using the smotefamily package to address the class imbalance between vaccinated and non-vaccinated respondents. Parameters: K=5 nearest neighbors, dup_size=2. SMOTE was applied to the full analytical dataset before the train-test split. This is a known methodological limitation: synthetic observations may appear in both the training and evaluation subsets, creating information leakage risk and potentially inflating recall estimates.
Split the SMOTE-balanced dataset 70/30 into training and evaluation subsets using set.seed(123). Trained two models on the training subset: (1) XGBoost with binary logistic objective, max_depth=3, eta=0.1, nrounds=100, using a sparse model matrix; and (2) logistic regression using R's glm function with binomial family. Both models used Insurance, Family Savings, and Education as predictors.
Evaluated both models on the post-SMOTE 30% evaluation split using caret's confusionMatrix with the positive class set to vaccinated. Extracted XGBoost feature importance by gain. Visualized logistic regression feature importance as a coefficient bar chart using ggplot2. Both models used the same three predictors: Insurance, Family Savings, and Education.
Methods
Analytical Methodology
Expand each method for description and purpose.
HPV Vaccination Analytics modeling workflow. Three binary predictors were used: insurance coverage (NHANES variable HIQ011, coded as covered or not covered), family savings (NHANES variable IND310, coded as less than three thousand dollars or three thousand dollars or more, used as an economic proxy), and education level (NHANES variable DMDEDUC2, coded as below college or college and above). These were merged on the NHANES sequence number and filtered to complete cases. All four variables were recoded to binary one-or-two factor levels. SMOTE class balancing was then applied to the full analytical dataset using K equals five nearest neighbors and a duplication size of two. Caution: SMOTE was applied before the data was split, which introduces potential information leakage and likely inflates recall estimates. The balanced data was then split approximately seventy percent training and thirty percent evaluation. Two classification models were trained: XGBoost using a binary logistic objective with maximum tree depth of three, learning rate of zero point one, and one hundred rounds; and logistic regression using R glm with a binomial family. Both models were evaluated using a confusion matrix, accuracy, precision, recall, and F1 score via the caret package. Feature importance was assessed using XGBoost gain scores and logistic regression coefficients. The binary outcome was reported HPV vaccine receipt: vaccinated versus not vaccinated.
Analysis Outputs
Selected Output Previews
All six charts below are native interactive visualizations built from the original course analysis data. Count values for descriptive charts (A through D) are confirmed from the rendered table outputs in the original BMIN 5030 course submission. Feature importance rankings (E) reflect the XGBoost gain ordering reported in the course output; exact gain values are approximate. Logistic regression coefficient directions (F) are qualitative representations based on the feature importance order and expected predictor relationships; exact values were not extracted from the original submission. All reported evaluation metrics should be interpreted with the methodological limitations described below.
Fewer than 4 in 10 respondents reported receiving an HPV vaccine. This imbalance between vaccinated (323) and not vaccinated (506) was why SMOTE oversampling was applied before model training. Hover a bar to see the percentage breakdown.
Most respondents had health insurance, yet overall vaccination remained below 40%. This suggests that insurance coverage alone does not fully explain vaccination behavior — education and savings were also included as predictors in the models.
More than 70% of respondents reported family savings below $3,000. This variable is derived from the NHANES IND310 question and used here as a broad economic proxy. Lower household savings may reflect financial pressures that deprioritize preventive care like vaccination.
About two-thirds of respondents had at least some college education. Education was the most influential predictor in both the XGBoost and logistic regression models, consistent with research linking higher educational attainment to greater health literacy and uptake of preventive services.
Ranking confirmed from original XGBoost output. Exact gain values are approximate.
XGBoost assigns each feature a gain score reflecting how much it improved the model at each decision split. Education (college versus below college) was the most important predictor by a wide margin, followed by insurance coverage and family savings. Gain measures usefulness in splitting — not the direction or magnitude of the effect on vaccination likelihood.
Direction qualitative. Magnitudes approximate — exact values not extracted from course output.
Bars extending right indicate a positive association with HPV vaccination; bars extending left indicate a negative association. College education was most strongly associated with higher vaccination likelihood, while lacking insurance was associated with lower odds. Note: the logistic model classified every respondent as vaccinated (see Limitations), so these directions reflect coefficient signs rather than discriminative performance.
Confirmed Outcomes
Documented Analytical Outputs
A binary HPV vaccine receipt outcome was defined from NHANES variable IMQ060 (Yes: ever received an HPV vaccine / No: never received) for female respondents; Refused and Don't Know responses were treated as missing and excluded
Three binary predictors were constructed from NHANES variables HIQ011 (insurance: covered / not covered), IND310 (family savings: less than $3,000 / $3,000 or more, used as an economic proxy), and DMDEDUC2 (education: below college / college or above)
SMOTE class balancing was applied to the merged complete-case dataset using K=5 nearest neighbors; the balanced data was then split 70/30 into training and evaluation subsets
XGBoost (binary:logistic, max_depth=3, eta=0.1, nrounds=100) and logistic regression (glm, binomial) were trained and evaluated on the same post-SMOTE split; evaluation metrics are documented in the Limitations section with methodological context
Original descriptive plots and feature importance visualizations were generated for the course submission; the XGBoost gain-based importance plot and the HPV vaccination status distribution are included as original course project outputs
This case study documents the original BMIN503/EPID600 course analysis and its methodological limitations. Because SMOTE was applied before the data split, the reported evaluation metrics should not be interpreted as validated out-of-sample performance. The predictors showed associations with reported HPV vaccine receipt within the analyzed sample, but these patterns cannot be generalized as nationally representative estimates because survey design weights were not applied.
Rigor
Data Quality & Methodological Safeguards
Complete-case filtering applied: all respondents with missing values on HPV_vaccine, Insurance, Poverty, or Education were excluded from the analytical dataset
All four variables explicitly recoded from original NHANES survey codes to binary 1/2 factor levels before modeling, with refused and don't-know responses treated as missing
Both models used set.seed(123) to enable replication of the train-test split
Two modeling approaches (XGBoost and logistic regression) provided cross-method comparison for the same predictors and outcome
Feature importance was assessed separately for each model using method-appropriate metrics: gain for XGBoost and coefficient magnitude for logistic regression
Descriptive frequency distributions were generated for all four variables and inspected before modeling
Responsible Interpretation
Limitations
Understanding these limitations is essential for correctly interpreting results.
SMOTE applied before the train-test split
In the original course workflow, SMOTE was applied to the full analytical dataset before the data was split into training and evaluation subsets. This means synthetic observations generated from vaccinated-class respondents may have appeared in both the training and evaluation subsets. As a consequence, the evaluation subset was not a fully independent hold-out of original observations.
The reported evaluation metrics (see Original Course Evaluation in this section) should be interpreted as internal evaluation results on a post-SMOTE split rather than as externally validated performance estimates. Recall and F1 scores are likely optimistic. A section below describes what a methodologically sound rebuild would require.
Logistic regression collapsed to the positive class
In the original course analysis, the logistic regression model predicted every observation in the evaluation split as vaccinated. Recall was 100% and true negatives were zero. This indicates the decision threshold of 0.5 on the probability output placed all predictions on the positive side, likely because SMOTE inflated the vaccinated class to the point where the model learned to assign higher probability to vaccinated status for most observations.
The logistic regression recall and F1 figures do not indicate useful class discrimination. This result is documented transparently here rather than omitted. It illustrates a common consequence of applying oversampling before evaluation: the model may appear to perform well on one metric while providing no useful separation on the other class.
Original Course Evaluation Metrics
The following evaluation metrics were reported in the original rendered project output (BMIN503/EPID600 final submission). They are reproduced here for transparency. Both models were evaluated on the same post-SMOTE 30% evaluation split. These metrics cannot be interpreted as externally validated estimates of performance on unseen data. XGBoost: Accuracy 76.32% · Precision 76.74% · Recall 99.14% · F1 86.52% Logistic Regression: Accuracy 76.64% · Precision 76.64% · Recall 100.00% · F1 86.78% The logistic regression predicted every evaluated observation as vaccinated (confusion matrix: 233 true positives, 71 false positives, 0 true negatives, 0 false negatives). Its recall of 100% and F1 of 86.78% therefore do not indicate that the model successfully discriminated between classes. Neither model's figures should be described as strong performance, high predictive accuracy, or successful validation.
These metrics are presented as a historical record of the original course analysis, not as performance benchmarks. Evaluation on an independent pre-SMOTE hold-out set would be required before any assessment of real-world predictive utility.
Binary outcome without dose-count information
The outcome variable (IMQ060) captures whether the respondent reported ever receiving an HPV vaccine, coded as Yes or No. It does not capture the number of doses received, whether the respondent completed the recommended series, or when vaccination occurred. The analysis cannot distinguish between respondents who received one dose and those who completed the full recommended schedule.
All portfolio copy uses 'reported HPV vaccine receipt' to reflect the binary nature of the outcome. Dose count, series completion, and timing are not claimed.
Female-only questionnaire scope
The HPV vaccination question IMQ060 in NHANES L was directed at female participants ages 9 to 49. Male participants were not included in this variable's collection scope in the data modules used. The analytical sample therefore represents female respondents only.
All population descriptions use 'female participants' or 'female respondents' to reflect the actual scope of the questionnaire item.
Effective age restriction from DMDEDUC2
The education variable (DMDEDUC2) is collected only for NHANES participants aged 20 and older. Because the complete-case filter required non-missing education data, the analytical sample was effectively restricted to female respondents aged 20 to 49, even though IMQ060 covers ages 9 to 49. Respondents aged 9 to 19 would have missing DMDEDUC2 values and were excluded.
The population is described as 'female NHANES respondents ages 20 to 49 with complete data on all four modeled variables.' The 20-year lower bound is derived from the DMDEDUC2 variable scope (collected only for participants 20 and older), not estimated visually.
Family savings measure as a limited economic proxy
The economic variable used in this analysis (IND310) asks about total savings or cash assets for the family. It was recoded to a binary variable: less than $3,000 versus $3,000 or more. This measure is a limited economic proxy: it captures savings holdings at one point in time, not income, income-to-poverty ratio, or persistent economic hardship. The source project titled this variable 'poverty level' in some plots, but IND310 is not an official poverty ratio measure.
All portfolio copy uses 'family savings' or 'family savings indicator' to accurately describe IND310. Descriptive plots from the original project that use the label 'poverty levels' are noted as original historical outputs with clarification that the underlying variable is IND310.
Survey design weights not applied
NHANES uses a complex probability sample with unequal selection probabilities, stratification, and clustering. Analyses that do not incorporate the provided sampling weights and design variables produce estimates that describe the analytical sample but cannot be generalized as nationally representative estimates of the United States population.
No nationally representative conclusions are drawn from this analysis. Results are described as patterns observed in the analyzed NHANES sample.
Cross-sectional observational design
NHANES is a cross-sectional survey. All variables were measured at or close to a single point in time, so temporal sequences between predictors and the outcome cannot be established. The analysis is associative, not causal. Observed associations between insurance, savings, education, and vaccine receipt do not establish that those factors caused or prevented vaccination.
The research question and all portfolio copy use associative language. The word 'causes' and similar causal framings are not used.
Self-reported vaccine receipt
IMQ060 relies on respondent self-report of whether they ever received an HPV vaccine. Self-reported vaccination status may be subject to recall error or social desirability bias. Verification against vaccination records was not part of this analysis.
The outcome is described as 'reported HPV vaccine receipt' throughout to reflect the self-reported nature of the measure.
Small and selected predictor set
The final models included only three predictors: insurance coverage, family savings, and education level. Factors documented in the broader literature on HPV vaccination (including age, race, ethnicity, geographic region, provider recommendation, and parental attitudes) were not included in the model. The analytical dataset also contained no sex or age variables in the final model specification.
Results are framed as patterns associated with the three selected variables, not as comprehensive explanations of HPV vaccine receipt. Unmeasured confounding is acknowledged.
No external or temporal validation
The analysis was evaluated only on a post-SMOTE internal split. No external dataset, later NHANES cycle, or independent sample was used to validate the model's generalizability. Performance on a different population or time period is unknown.
External validation is identified as a required step before any practical application of these models, as described in the What I Would Change section.
Changes in HPV vaccine recommendations over time
HPV vaccine recommendations in the United States have changed since the vaccine's introduction, including expansions to male recipients, catch-up age limits, and schedule modifications. The 2021-2023 data reflect a specific period in the evolution of those guidelines, and patterns observed may not hold for different cycles.
Survey cycle is reported precisely as 2021-2023 and no cross-cycle comparisons are made.
What I Would Change in a Rebuild
This reflection documents the methodological improvements that would be applied in a more rigorous version of this analysis: 1. Split the original complete-case observations first, before any oversampling, to preserve an untouched evaluation set of real survey respondents. 2. Apply SMOTE only to the training subset, not to the full dataset. 3. Prefer applying oversampling within each training resample or cross-validation fold rather than once before the loop. 4. Preserve the held-out evaluation set as strictly unseen data for final reporting. 5. Use stratified resampling to maintain class proportions across folds. 6. Report class-specific precision, recall, specificity, and full confusion matrices for both models rather than only accuracy and F1. 7. Evaluate ROC-AUC and precision-recall AUC as threshold-independent performance summaries. 8. Examine probability calibration to assess whether predicted probabilities correspond to observed frequencies. 9. Compare both models against a simple baseline, for example predicting the majority class for every observation. 10. Incorporate NHANES survey weights and design variables to produce estimates that are appropriate for population-level inference. 11. Include age as a predictor or perform age-stratified analyses, given its known association with vaccination likelihood. 12. Explore a broader predictor set including race, ethnicity, geographic region, and provider access measures. 13. Perform external or temporal validation before any practical application.
This analysis was completed as a course project with the scope and time constraints of an academic submission. The reflection above is presented as evidence of methodological growth, not as a criticism of the original submission's educational value.
Stack
Technology Stack
Core Environment
Modeling and Evaluation
Visualization
Reflection
Lessons Learned
The ordering of preprocessing steps (specifically whether oversampling precedes or follows the train-test split) determines whether evaluation metrics describe model performance on unseen data or on data partially derived from the training distribution. This distinction has a large effect on recall and must be documented explicitly in any analysis that uses synthetic oversampling.
A model that reports high recall but zero true negatives is not performing well on the classification task; it has collapsed to the positive class. Examining only F1 or accuracy without looking at the full confusion matrix would hide this failure mode.
A small predictor set with high collinearity or shared variance can produce unstable feature importance rankings across model runs. Publishing a single-run importance chart as if it were a stable finding overstates what the analysis demonstrated.
The variable label in a survey codebook often differs from how a derived variable should be described publicly. IND310 is titled 'Total savings/cash assets for the family,' not a poverty index. Writing accurate public copy required reading the source codebook, not just the variable name that appeared in the analysis.
Framing an academic course analysis publicly requires an extra layer of precision: distinguishing what the original project claimed from what can be said about it with hindsight and methodological awareness. That reframing is itself a form of professional communication skill.
Forward
Next Steps
Apply the rebuild methodology described in the What I Would Change section: split first, oversample only the training subset, and evaluate on a held-out set of original observations
Expand the predictor set to include age, self-reported race and ethnicity, geographic region, and provider recommendation if available in the same NHANES cycle
Incorporate NHANES survey weights and design variables to produce survey-weighted prevalence estimates appropriate for population-level inference
Evaluate model performance using ROC-AUC and precision-recall AUC in addition to accuracy and F1, and compare both models against a majority-class baseline
Examine whether the association patterns differ by age group by performing age-stratified analyses, given the effective restriction of the analytic sample to females aged 20 to 49
Validate the rebuilt models on a different NHANES cycle to test temporal stability of the observed associations