Journal of Preventive Medicine and Public Health

The Korean Society for Preventive Medicine (대한예방의학회)
권호 표지
DOI QR코드

DOI QR Code

Inappropriate Survey Design Analysis of the Korean National Health and Nutrition Examination Survey May Produce Biased Results

  • Kim, Yangho (Department of Occupational and Environmental Medicine, Ulsan University Hospital, University of Ulsan College of Medicine) ;
  • Park, Sunmin (Department of Food and Nutrition, Hoseo University) ;
  • Kim, Nam-Soo (Institue of Environmental & Occupational Medicine, Soonchunhyang University) ;
  • Lee, Byung-Kook (Korea Industrial Health Association)
  • Received : 2012.11.13
  • Accepted : 2013.03.15
  • Published : 2013.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

Objectives: The inherent nature of the Korean National Health and Nutrition Examination Survey (KNHANES) design requires special analysis by incorporating sample weights, stratification, and clustering not used in ordinary statistical procedures. Methods: This study investigated the proportion of research papers that have used an appropriate statistical methodology out of the research papers analyzing the KNHANES cited in the PubMed online system from 2007 to 2012. We also compared differences in mean and regression estimates between the ordinary statistical data analyses without sampling weight and design-based data analyses using the KNHANES 2008 to 2010. Results: Of the 247 research articles cited in PubMed, only 19.8% of all articles used survey design analysis, compared with 80.2% of articles that used ordinary statistical analysis, treating KNHANES data as if it were collected using a simple random sampling method. Means and standard errors differed between the ordinary statistical data analyses and design-based analyses, and the standard errors in the design-based analyses tended to be larger than those in the ordinary statistical data analyses. Conclusions: Ignoring complex survey design can result in biased estimates and overstated significance levels. Sample weights, stratification, and clustering of the design must be incorporated into analyses to ensure the development of appropriate estimates and standard errors of these estimates.

Keywords

References

  1. Korea Centers for Disease Control and Prevention. Second workshop for the data analysis of the Korea National Health & Nutrition Examination Survey and online survey of adolescent health behavior. Cheongwon: Korea Centers for Disease Control and Prevention; 2012, p. 1-60 (Korean).
  2. Jacobson AF, Hamilton P, Galloway J. Obtaining and evaluating data sets for secondary analysis in nursing research. West J Nurs Res 1993;15(4):483-494. https://doi.org/10.1177/019394599301500407
  3. Korn EL, Graubard BI. Analysis of large health surveys: accounting for the sampling design. J R Stat Soc Ser A Stat Soc 1995;158(2):263-295. https://doi.org/10.2307/2983292
  4. Magee T, Lee SM, Giuliano KK, Munro B. Generating new knowledge from existing data: the use of large data sets for nursing research. Nurs Res 2006;55(2 Suppl):S50-S56. https://doi.org/10.1097/00006199-200603001-00009
  5. Levy PS, Lemeshow S. Sampling of populations: methods and applications. 3rd ed. New York: Wiley-Interscience Publication; 1999, p. 333-363.
  6. Lee ES, Forthofer RN. Analyzing complex survey data. 2nd ed. London: SAGA Publications; 2006, p. 11-20.
  7. Wheeler DC, VanHorn JE, Paskett ED. A comparison of designbased and model based analysis of sample survey in geography. Atlanta: Rollings School of Public Health, Emory University; 2007, p. 1-34.
  8. Lohr SL. Sampling: design and analysis. 2nd ed. Boston: Brooks/Cole, Cengage Learning; 2010, p. 281-320.
  9. Korn EL, Graubard BI. Analysis of health surveys. New York: Wiley-Interscience Publication; 1999, p. 8-50.
  10. National Center for Health Statistics. The National Health and Nutrition Examination Survey (NHANES): analytic and reporting guidelines. Atlanta: Centers for Disease Control and Prevention; 2012, p. 1-14.
  11. Korea Centers for Disease Control and Prevention. Guideline for the evaluation of the fourth Korea National Health and Nutrition Examination Survey. Cheonwon: Korea Centers for Disease Control Prevention; 2009, p. 19-34 (Korean)
  12. Lee BK, Kim Y. Iron deficiency is associated with increased levels of blood cadmium in the Korean general population: analysis of 2008-2009 Korean National Health and Nutrition Examination Survey data. Environ Res 2012;112:155-163. https://doi.org/10.1016/j.envres.2011.10.013
  13. Park S, Lee BK. Vitamin D deficiency is an independent risk factor for cardiovascular disease in Koreans aged ≥50 years: results from the Korean National Health and Nutrition Examination Survey. Nutr Res Pract 2012;6(2):162-168. https://doi.org/10.4162/nrp.2012.6.2.162
  14. Hong S, Oh HJ, Choi H, Kim JG, Lim SK, Kim EK, et al. Characteristics of body fat, body fat percentage and other body composition for Koreans from KNHANES IV. J Korean Med Sci 2011;26(12):1599-1605. https://doi.org/10.3346/jkms.2011.26.12.1599
  15. Hahs-Vaughn DL. A primer for using and understanding weights with national datasets. J Exp Educ 2005;73(3):221-248. https://doi.org/10.3200/JEXE.73.3.221-248
  16. National Research Council. Methods for analysis of complex surveys. In: National Research Council. Understanding American agriculture: challenges for the agricultural resource management survey. Washington, DC: National Academies Press; 2007, p. 127-148.
  17. Burden S, Probst Y, Steel DG, Tapsell LC. The impact of complex survey design on prevalence estimates of intakes of food groups in the Australian National Children's Nutrition and Physical Activity Survey. New South Wales: Centre for Statistical and Survey Methodology, University of Wollongong; 2010, p. 1-21.
  18. Kalton G. Models in the practice of survey sampling (revisited). J Off Stat 2002;18(2):129-154.
  19. Rodgers-Farmer AY, Davis D. Analyzing complex survey data. Soc Work Res 2001;25(3):185-192. https://doi.org/10.1093/swr/25.3.185
  20. Brogan D. Software for sample survey data, misuse of standard packages. In: Armitage P, Colton T, editors. Encyclopedia of biostatistics. 2nd ed. Estados Unidos: John Wiley & Sons; 2005, p. 5057-5064.
  21. West BT. Statistical and methodological issues in the analysis of complex sample survey data: practical guidance for trauma researchers. J Trauma Stress 2008;21(5):440-447. https://doi.org/10.1002/jts.20356
  22. Lemeshow S, Letenneur L, Dartigues JF, Lafont S, Orgogozo JM, Commenges D. Illustration of analysis taking into account complex survey considerations: the association between wine consumption and dementia in the PAQUID study. Am J Epidemiol 1998;148(3):298-306. https://doi.org/10.1093/oxfordjournals.aje.a009639
  23. Korn EL, Graubard BI. Examples of differing weighted and unweighted estimates from a sample survey. Am Stat 1995;49(3): 291-295.
  24. Cho GJ, Park HT, Shin JH, Hur JY, Kim SH, Lee KW, et al. The relationship between blood mercury level and osteoporosis in postmenopausal women. Menopause 2012;19(5):576-581. https://doi.org/10.1097/gme.0b013e3182377294
  25. Heeringa SG, West BT, Berglund PA. Applied survey data analysis. London: CRC Press; 2005, p. 13-50.
  26. Park I, Lee H. Design effects for the weighted mean and total estimators under complex survey sampling. Surv Methodol 2004;30(2):183-193.
  27. Centers for Disease Control and Prevention. Specifying weighting parameters; 2011 [cited 2013 Mar 1]. Available from: http://www.cdc.gov/nchs/tutorials/nhanes/SurveyDesign/ Weighting/intro.htm.
  28. Centers for Disease Control and Prevention. National Health and Nutrition Examination Survey, 2007-2008; 2012 [cited 2013 Mar 1]. Available from: http://www.cdc.gov/nchs/data/ nhanes/nhanes_07_08/overviewbrochure_0708.pdf.
  29. Lee BK, Kim Y. Proper sample-weighted data analysis is required to confirm the association between blood cadmium concentration and lipid profile in Korean adults: re: Kim K. Blood cadmium concentration and lipid profile in Korean adults. Environ. Res. 2011. doi:10.1016/j.envres.2011.12.008. Environ Res 2012;115:66-67.
  30. Lee BK. Without the application of appropriate statistical methods, it is difficult to reach the proper conclusion. Menopause 2012;19(6):710.
  31. Platt RW, Harper SB. Survey data with sampling weights: is there a "best" approach? Environ Res 2013;120:143-144. https://doi.org/10.1016/j.envres.2012.08.006
  32. Hahs-Vaughn DL, McWayne CM, Bulotsky-Shearer RJ, Wen X, Faria AM. Methodological considerations in using complex survey data: an applied example with the Head Start Family and Child Experiences Survey. Eval Rev 2011;35(3):269-303. https://doi.org/10.1177/0193841X11412071

Cited by

  1. Prevalence and Risk Factors for Age-Related Macular Degeneration: Korean National Health and Nutrition Examination Survey 2008-2011 vol.39, pp.12, 2014, https://doi.org/10.3109/02713683.2014.907431
  2. Inverse Association between High Blood 25-Hydroxyvitamin D Levels and Diabetic Retinopathy in a Representative Korean Population vol.9, pp.12, 2014, https://doi.org/10.1371/journal.pone.0115199
  3. Relationship between Olfactory Dysfunction and Suicidal Ideation: The Korea National Health and Nutrition Examination Survey vol.29, pp.4, 2013, https://doi.org/10.2500/ajra.2015.29.4194
  4. The relationship between chronic obstructive pulmonary disease and comorbidities: A cross-sectional study using data from KNHANES 2010–2012 vol.109, pp.1, 2013, https://doi.org/10.1016/j.rmed.2014.10.015
  5. Association of Hearing Loss and Tinnitus with Health-Related Quality of Life: The Korea National Health and Nutrition Examination Survey vol.10, pp.6, 2013, https://doi.org/10.1371/journal.pone.0131247
  6. Association between Blood Lead Levels and Age-Related Macular Degeneration vol.10, pp.8, 2013, https://doi.org/10.1371/journal.pone.0134338
  7. Association between Obesity and Chronic Laryngitis in Women – Data from the Korea National Health and Nutrition Examination Survey vol.8, pp.4, 2015, https://doi.org/10.1159/000437004
  8. Dietary vitamin C intake protects against COPD: the Korea National Health and Nutrition Examination Survey in 2012 vol.11, pp.None, 2013, https://doi.org/10.2147/copd.s119448
  9. Gender Difference in the Epidemiological Association between Metabolic Syndrome and Olfactory Dysfunction: The Korea National Health and Nutrition Examination Survey vol.11, pp.2, 2013, https://doi.org/10.1371/journal.pone.0148813
  10. Serum 25-Hydroxyvitamin D Levels and Dry Eye Syndrome: Differential Effects of Vitamin D on Ocular Diseases vol.11, pp.2, 2013, https://doi.org/10.1371/journal.pone.0149294
  11. Association between Grapes Intake and Diabetic Retinopathy: Inhibitory Effect of Resveratol on Diabetic Retinopathy vol.57, pp.2, 2013, https://doi.org/10.3341/jkos.2016.57.2.276
  12. Age- and Gender-Related Mean Hearing Threshold in a Highly-Screened Population: The Korean National Health and Nutrition Examination Survey 2010–2012 vol.11, pp.3, 2016, https://doi.org/10.1371/journal.pone.0150783
  13. The Association Between Smoking Tobacco After a Diagnosis of Diabetes and the Prevalence of Diabetic Nephropathy in the Korean Male Population vol.49, pp.2, 2013, https://doi.org/10.3961/jpmph.15.062
  14. Inverse relationship between sleep duration and myopia vol.94, pp.3, 2013, https://doi.org/10.1111/aos.12776
  15. Relationship between tinnitus and suicidal behaviour in Korean men and women: a cross‐sectional study vol.41, pp.3, 2016, https://doi.org/10.1111/coa.12500
  16. The Association of Metabolic Syndrome with Diabetic Retinopathy: The Korean National Health and Nutrition Examination Survey 2008–2012 vol.11, pp.6, 2016, https://doi.org/10.1371/journal.pone.0157006
  17. Positive Association between Blood 25-Hydroxyvitamin D Levels and Pterygium after Control for Sunlight Exposure vol.11, pp.6, 2013, https://doi.org/10.1371/journal.pone.0157501
  18. Relationship between total fruit and vegetable intake and self-reported knee pain in older adults vol.21, pp.7, 2013, https://doi.org/10.1007/s12603-016-0842-7
  19. High carbohydrate diets are positively associated with the risk of metabolic syndrome irrespective to fatty acid composition in women: the KNHANES 2007-2014 vol.68, pp.4, 2013, https://doi.org/10.1080/09637486.2016.1252318
  20. Epidemiological association of olfactory dysfunction with hearing loss and dysphonia in the Korean population : A cross-sectional study vol.96, pp.47, 2013, https://doi.org/10.1097/md.0000000000008890
  21. The Prevalence of “Drinking and Biking” and Associated Risk Factors: The Korea National Health and Nutrition Examination Survey vol.32, pp.9, 2013, https://doi.org/10.3346/jkms.2017.32.9.1396
  22. Prevalence of Self-reported Allergic Diseases and IgE Levels: A 2010 KNHANES Analysis vol.9, pp.4, 2017, https://doi.org/10.4168/aair.2017.9.4.329
  23. Predictors of cardiorespiratory fitness in female and male adults with different body mass index: National Health and Nutrition Examination Survey 1999-2004 dataset vol.49, pp.1, 2013, https://doi.org/10.1080/07853890.2016.1252056
  24. Associations between Intraocular Pressure and Systemic Parameters according to the KNHNES 2008-2011 vol.58, pp.4, 2013, https://doi.org/10.3341/jkos.2017.58.4.430
  25. Chronic obstructive pulmonary disease is independently associated with hypertension in men : A survey design analysis using nationwide survey data vol.96, pp.19, 2013, https://doi.org/10.1097/md.0000000000006826
  26. The association between periodontitis and sleep duration vol.44, pp.5, 2013, https://doi.org/10.1111/jcpe.12713
  27. Epiretinal Membrane: Prevalence and Risk Factors from the Korea National Health and Nutrition Examination Survey, 2008 through 2012 vol.31, pp.6, 2013, https://doi.org/10.3341/kjo.2016.0098
  28. Patient Awareness of Cataract and Age-related Macular Degeneration among the Korean Elderly: A Population-based Study vol.31, pp.6, 2017, https://doi.org/10.3341/kjo.2017.0025
  29. Prevalences and Management of Diabetes and Pre-diabetes among Korean Teenagers and Young Adults: Results from the Korea National Health and Nutrition Examination Survey 2005–2014 vol.32, pp.12, 2013, https://doi.org/10.3346/jkms.2017.32.12.1984
  30. The positive association of total protein intake with femoral neck strength (KNHANES IV) vol.29, pp.6, 2018, https://doi.org/10.1007/s00198-018-4451-7
  31. Association between secondhand smoke exposure and hypertension in never smokers: a cross-sectional survey using data from Korean National Health and Nutritional Examination Survey V, 2010–2012 vol.8, pp.5, 2013, https://doi.org/10.1136/bmjopen-2017-021217
  32. Prevalence and risk factors of elevated alanine aminotransferase among Korean adolescents: 2001-2014 vol.18, pp.1, 2013, https://doi.org/10.1186/s12889-018-5548-9
  33. Association between body mass index and health-related quality of life among Chinese elderly—evidence from a community-based study vol.18, pp.1, 2013, https://doi.org/10.1186/s12889-018-6086-1
  34. Korean survey data reveals an association of chronic laryngitis with tinnitus in men vol.13, pp.1, 2018, https://doi.org/10.1371/journal.pone.0191148
  35. Determinants of the utilization of postpartum family visits: Evidence from rural areas of Eastern China vol.13, pp.3, 2013, https://doi.org/10.1371/journal.pone.0194061
  36. Inverse relationship between serum hsCRP concentration and hand grip strength in older adults: a nationwide population-based study vol.10, pp.8, 2013, https://doi.org/10.18632/aging.101529
  37. Association of Serum TSH With Handgrip Strength in Community-Dwelling Euthyroid Elderly vol.103, pp.11, 2013, https://doi.org/10.1210/jc.2018-01095
  38. Comparison of the Marginal Utility and Disease Burden of Hearing Loss and Other Chronic Diseases vol.39, pp.6, 2013, https://doi.org/10.4082/kjfm.17.0033
  39. Impact of Physical Activity and Sleep Duration on Depressive Symptoms in Hypertensive Patients: Results from a Nationally Representative Korean Sample vol.15, pp.12, 2013, https://doi.org/10.3390/ijerph15122611
  40. Income and Education are Independently Associated with Visual Impairment: The Korean National Health and Nutrition Examination Survey 2010-2012 vol.34, pp.3, 2013, https://doi.org/10.1080/08820538.2019.1597133
  41. Lack of Association Between Vitamin D and Hand Grip Strength in Asians: A Nationwide Population-Based Study vol.104, pp.2, 2013, https://doi.org/10.1007/s00223-018-0480-7
  42. The Relationship Between Housing Types and Metabolic and Weight Phenotypes: A Nationwide Survey vol.17, pp.3, 2013, https://doi.org/10.1089/met.2018.0109
  43. Lower hand grip strength in older adults with non-alcoholic fatty liver disease: a nationwide population-based study vol.11, pp.13, 2013, https://doi.org/10.18632/aging.102068
  44. Associations between Retinal Nerve Fiber Layer Defect and Systemic Indices According to the KNHNES 2010-2012 vol.60, pp.11, 2013, https://doi.org/10.3341/jkos.2019.60.11.1089
  45. How to use replicate weights in health survey analysis using the National Nutrition and Physical Activity Survey as an example vol.22, pp.18, 2019, https://doi.org/10.1017/s1368980019001927
  46. Association between serum 25-hydroxyvitamin D levels and myopia in general Korean adults vol.68, pp.1, 2013, https://doi.org/10.4103/ijo.ijo_760_19
  47. Age- and sex-based evaluation of the association between refractive error and age-related macular degeneration in the Korean population vol.15, pp.1, 2013, https://doi.org/10.1371/journal.pone.0228468
  48. Association Between Dietary Nutrient Intake and Chronic Obstructive Pulmonary Disease Severity: A Nationwide Population-Based Representative Sample vol.17, pp.1, 2020, https://doi.org/10.1080/15412555.2019.1698530
  49. Positive Effects of the National Cigarette Price Increase Policy on Smoking Cessation in South Korea vol.83, pp.1, 2013, https://doi.org/10.4046/trd.2019.0011
  50. Hormone‐related events and periodontitis in women vol.47, pp.4, 2013, https://doi.org/10.1111/jcpe.13248
  51. Sleep duration and health-related quality of life in Korean adults: 2007-2015 Korea National Health and Nutrition Examination Survey vol.24, pp.2, 2013, https://doi.org/10.1007/s11325-019-01972-7
  52. Increased Overall Heart Rate Irregularity Risk by Hyperuricemia in the General Population: Results from the Korean National Health and Nutrition Examination Survey vol.56, pp.10, 2013, https://doi.org/10.3390/medicina56100501
  53. Association between Relative Handgrip Strength and Osteoporosis in Older Women: The Korea National Health and Nutrition Examination Survey 2014–2018 vol.24, pp.4, 2013, https://doi.org/10.4235/agmr.20.0076
  54. Association of Participation in Disease Management Programs and Health-Related Quality of Life in Korean Population: Results from the Korea National Health and Nutrition Examination Survey IV and V (2 vol.12, pp.None, 2013, https://doi.org/10.1177/21501327211012185
  55. Reliability of Portable Spirometry Performed in the Korea National Health and Nutrition Examination Survey Compared to Conventional Spirometry vol.84, pp.4, 2021, https://doi.org/10.4046/trd.2021.0016
  56. Effects of education, income, and occupation on prevalence and symptoms of knee osteoarthritis vol.11, pp.1, 2013, https://doi.org/10.1038/s41598-021-93394-3
  57. Association between dual use of e-cigarette and cigarette and chronic obstructive pulmonary disease: an analysis of a nationwide representative sample from 2013 to 2018 vol.21, pp.1, 2013, https://doi.org/10.1186/s12890-021-01590-8
  58. Association between blood cadmium levels and the risk of osteopenia and osteoporosis in Korean post-menopausal women vol.16, pp.1, 2013, https://doi.org/10.1007/s11657-021-00887-9