ADVERTISEMENT

Issue Navigator

Volume 14 No. 10
Earn CME
Accepted Papers





Scientific Investigations

Accuracy and Reliability of Internet Resources Providing Information on Obstructive Sleep Apnea

Jongmin Lee, MD1; Woo Ho Ban, MD2; Hyun Kyung Park, MS3; Eunbyeol Na, BS3; Sug Kyung Kim, MS3; Hyeon Hui Kang, MD4; Sang Haak Lee, MD, PhD4
1Division of Pulmonary, Allergy and Critical Care Medicine, Department of Internal Medicine, Seoul St Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea; 2Division of Pulmonary, Allergy and Critical Care Medicine, Department of Internal Medicine, Incheon St Mary's Hospital, College of Medicine, The Catholic University of Korea, Incheon, Republic of Korea; 3CC&I Research, Seoul, Republic of Korea; 4Division of Pulmonary, Critical Care and Sleep Medicine, Department of Internal Medicine, St Paul's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea

ABSTRACT

Study Objectives:

To evaluate the readability, content, and quality of internet information about obstructive sleep apnea (OSA).

Methods:

Three major search engines (Google, Yahoo, and Bing) were used to retrieve the first 200 hits for the term “sleep apnea.” The website contents were assessed using established guidelines, while the website quality was evaluated using the validated DISCERN instrument and Journal of the American Medical Association benchmarks.

Results:

In total, 243 websites met the inclusion criteria for this study. The websites were divided into five categories: scientific resource, foundation, news/ media report, commercial website, and personal commentary. The mean Flesch-Kincaid grade level, which indicated the readability of the websites, was 7.5. The quality of most websites was poor, with news/media reports and commercial websites exhibiting poorer quality than the others. The Health On the Net (HON) code, which certifies the reliability and credibility of online content regarding human health, was applied only to 14.3% websites; the reliability of all these websites was poor. The content score for the diagnosis and management of OSA was lower for news/media reports and commercial websites than for scientific resources. News/media reports exhibited the worst content scores.

Conclusions:

Our findings suggest that most online health resources regarding OSA are not reliable and are inaccurate. Health care providers should be aware of the high variability in online information concerning OSA.

Commentary:

A commentary on this article appears in this issue on page 1643.

Citation:

Lee J, Ban WH, Park HK, Na E, Kim SK, Kang HH, Lee SH. Accuracy and reliability of internet resources providing information on obstructive sleep apnea. J Clin Sleep Med. 2018;14(10):1717–1723.


BRIEF SUMMARY

Current Knowledge/Study Rationale: Patients generally use the internet as a resource for health information. However, no study has evaluated the content and quality of online information on obstructive sleep apnea (OSA).

Study Impact: The results of this study demonstrate that websites related to OSA are often incomplete and inaccurate, and the low quality of online information about this condition is a significant barrier to patient education. This study highlights the need to ensure the accuracy and quality of online health information concerning OSA.

INTRODUCTION

The internet has become a common source for health-related information and medical queries.1 In 2014, 84% of the United States population used the internet, with 72% of that population searching for health information.2 In total, 35% adults in the United States attempt to obtain a diagnosis of a medical condition for themselves or others by seeking health information on the internet.2 The internet is an attractive resource for patients because it provides information on almost all health-related topics immediately. However, there are concerns about the accuracy and reliability of online health information, and there is a possibility that the patients may be misinformed.3,4

Obstructive sleep apnea (OSA) is a common breathing disorder characterized by narrowing of the upper airway that impairs normal ventilation during sleep.5 Patients with untreated OSA may be at an increased risk of motor vehicle accidents, metabolic dysregulation, and cardiovascular diseases such as hypertension, coronary artery disease, congestive heart failure, arrhythmias, and stroke.6,7 Therefore, the recognition and treatment of OSA is important for improving the quality of life and lowering the mortality and morbidity of patients with OSA. Although clinical guidelines for the diagnosis and management of OSA were published by the American Academy of Sleep Medicine in 2009 and updated in 2017,5,8 there is a lot of misleading information about OSA on the internet, which can result in serious consequences for patients.

Thus far, few studies have examined the quality of online health information about OSA. Therefore, the objective of the current study was to analyze the readability and quality of online health information concerning OSA.

METHODS

Data Sources

We used three well-known internet search engines for our analysis of online health information about OSA. Google, Yahoo, and Bing (http://google.com, http://yahoo.com, http://bing.com) searches for “sleep apnea” were performed on September 1, 2016. After deleting the history and cookies of the web browser, searches were conducted using the United States version of each search engine.

Website Selection

The first 200 English language websites providing information on OSA in each search engine were collected. Websites that required registration or enrollment fees to access information, duplicate websites, websites that only include videos, and scientific journal articles for medical professionals were excluded.

Evaluation of the Readability and Accuracy of Websites

The website contents were crawled using Python software 2.7.12 (Python Software Foundation, Beaverton, Oregon, United States). All websites were classified into five categories: scientific resource, foundation, news/media report, commercial website, and personal commentary. The readability of websites was evaluated using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid grade level,9,10 while the quality of information provided by each website was assessed using publishing benchmarks laid by the Journal of the American Medical Association (JAMA) and the DISCERN instrument. The JAMA benchmarks provide a means for the qualitative assessment of websites for the presence of authorship, references (attribution), date of last update (currency), and disclosure of conflicts of interest (disclosure).11 The DISCERN instrument is a validated tool designed to judge the quality of written information about treatment choices.12 The instrument comprises 16 questions rated from 1 (low quality/ not addressed) to 5 (high quality/fully addressed). Questions 1 through 8 evaluate the reliability, questions 9 through 15 evaluate the quality of information regarding treatment options, and question 16 evaluates the overall publication.12 In addition, each website was evaluated for the Health on the Net Foundation (HON) code. HON is a nonprofit organization that reviews health information websites for eight points of conduct: authoritative, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy.13 The content score was created by two authors (J Lee and WH Ban) using a predefined scoring system based on clinical guidelines (Table 1). To avoid discrepancies in findings, the DISCERN and content scores were rechecked by two independent reviewers.

Criteria for scoring the content of websites providing information on obstructive sleep apnea.

jcsm.14.10.1717.t01.jpg

table icon
Table 1

Criteria for scoring the content of websites providing information on obstructive sleep apnea.

(more ...)

Statistical Analysis

Statistically significant differences between groups were analyzed using chi-square tests, analysis of variance, Bonferroni (Dunn) t tests, Kruskal-Wallis tests, or Fisher exact tests as appropriate. Data are expressed as mean ± standard deviations or numbers and percentages. A two-tailed value of P < .05 was considered statistically significant. All statistical analyses were performed using SAS software, version 9.4 (SAS Institute Inc., Cary, North Carolina, United States), and a graph was plotted using the R 3.4.1 version (R Foundation, Vienna, Austria).

RESULTS

Characteristics of Websites

After the exclusion of duplicates and websites that did not meet the inclusion criteria, 243 websites were considered relevant for this study (Figure 1). Table 2 shows the characteristics of websites classified according to the search engine. Commercial websites were the majority in Yahoo and Google, whereas news/media reports were the majority in Bing. FRES and the Flesch-Kincaid grade level were 57.1 ± 13.7 and 7.5 ± 2.5, respectively. There were no significant differences according to the search engine. The mean DISCERN total score, mean JAMA benchmark score, and mean content score were 49 ± 14, 1.9 ± 1.3, and 10.1 ± 4.1, respectively (Table 3), with significantly higher values for Google websites than for the other websites (P = .0002, P < .0001, and P = .0486, respectively). Most websites did not have HON certification (85.19%).

Characteristics of websites providing information on obstructive sleep apnea.

jcsm.14.10.1717.t02.jpg

table icon
Table 2

Characteristics of websites providing information on obstructive sleep apnea.

(more ...)

Reliability and quality of the content of websites providing information on obstructive sleep apnea.

jcsm.14.10.1717.t03.jpg

table icon
Table 3

Reliability and quality of the content of websites providing information on obstructive sleep apnea.

(more ...)

Search results and selection of websites providing information on obstructive sleep apnea.

The first 200 English language websites providing information on obstructive sleep apnea in each search engine were collected. Websites that required registration or enrollment fees to access information, duplicate websites, websites that only include videos, and scientific journal articles for medical professionals were excluded. After the exclusion of duplicates and websites that did not meet the inclusion criteria, 243 websites were considered relevant for this study.

jcsm.14.10.1717a.jpg

jcsm.14.10.1717a.jpg
Figure 1

Search results and selection of websites providing information on obstructive sleep apnea.

(more ...)

Reliability and Quality of Websites

The mean DISCERN total score for scientific resources was significantly higher than that for news/media reports and commercial websites (P < .0001), whereas individual scores for reliability (questions 1–8) and treatment choices (questions 9–15) were significantly higher for scientific resources than for commercial websites (P < .0001). The overall quality (question 16) of scientific reports was higher than that of news/media reports, commercial websites, and personal commentaries (P < .0001). There was no significant difference between DISCERN scores for scientific resources and those for foundations. The mean JAMA benchmark score was significantly lower for commercial websites than for scientific reports, foundations, and news/media reports (P = .0016), with no significant difference between commercial websites and personal commentaries.

Relationship Between the Presence of an HON Code and the Website Quality

As mentioned previously, approximately 14% of websites were HON accredited. Table 4 shows the relationship between the presence of an HON code and the website quality. There were no significant differences between the HON-positive and HON-negative websites in each website category. The DISCERN total score, JAMA benchmark score, and content score were significantly higher for HON-negative websites than for HON-positive websites (P = .0009, P < .0001, and P = .01, respectively).

Characteristics of websites providing information on obstructive sleep apnea according to the HON code.

jcsm.14.10.1717.t04.jpg

table icon
Table 4

Characteristics of websites providing information on obstructive sleep apnea according to the HON code.

(more ...)

Content Scores

Table 5 shows the content scores for websites in each category. The mean total content score was the highest for scientific resources (12.3 ± 2.9) and the lowest for news/ media reports (7.2 ± 4.1; Figure 2), and it was significantly higher for scientific resources than for news/media reports and commercial websites (P < .0001). Scores for definitions and symptoms were significantly lower for news/media reports than for all other categories except personal commentary (P < .0001). News/media reports and commercial websites exhibited significantly lower scores for diagnosis and management than did scientific resources (P = .0005 and P = .0002, respectively). A list of the websites containing higher content scores (> 14) is provided in Table S1 in the supplemental material.

Content scores by the category of websites providing information on obstructive sleep apnea.

jcsm.14.10.1717.t05.jpg

table icon
Table 5

Content scores by the category of websites providing information on obstructive sleep apnea.

(more ...)

Content scores according to the category of websites providing information on obstructive sleep apnea.

The mean total content score was the highest for scientific resources and the lowest for news/media reports, and it was significantly higher for scientific resources than for news/media reports and commercial websites (P < .0001). ANOVA = analysis of variance.

jcsm.14.10.1717b.jpg

jcsm.14.10.1717b.jpg
Figure 2

Content scores according to the category of websites providing information on obstructive sleep apnea.

(more ...)

Website Characteristics Based on the Search Results Ranking

Table S2 in the supplemental material shows the type, readability, and quality of websites according to their place in the search results, and whether they appeared earlier (group 1) or later (group 2). The type of website was significantly different between the two groups (P < .0001); scientific resources were the majority in group 1, whereas commercial websites were the majority in group 2. There was no significant difference between readability scores (FRES and the Flesch-Kincaid grade level) for group 1 and those for group 2. The mean total DISCERN and JAMA score of group 1 was significantly higher than that of group 2 (P = .001 for DISCERN, P = .018 for JAMA, respectively).

DISCUSSION

In the current study, we evaluated the readability, content, and quality of internet information about OSA and found that most online health resources regarding OSA are not reliable and are inaccurate.

Of late, with the widespread provision of internet services and increased popularity of online medical information, the number of patients accessing and applying this information has increased.14 However, the reliability and quality of currently available medical information concerning OSA has not been evaluated. To the best of our knowledge, the current study is the first to investigate the readability and quality of health information about OSA on the internet.

The current study included 243 websites providing health information about OSA. Commercial websites were the most common (37.0%), followed by scientific resources (22.2%). According to the JAMA benchmark and DISCERN total scores, the Yahoo websites were the most reliable, whereas the Bing websites were the least reliable. This could be attributed to differences in the proportion of scientific resources and foundations, which deliver relatively accurate and reliable information, between Yahoo and Bing.

The FRES is based on a ranking scale of 0 to 100, and the higher score represents content that can be more easily understood. Scores between 60 and 80 should generally be comprehensible for 12 to 15 year olds. The F-K grade level refers to the American school grade needed to comprehend the material on the website. In this study, the overall means for the FRES score and F-K grade level of the websites were 57.1 and 7.5, respectively. The National Institutes of Health recommends that the reading grade level for written health material should be between the seventh and eighth grade levels for the average adult.15 Accordingly, the readability of websites providing information about OSA was satisfactory. In other words, the information was easy to comprehend by average consumers.

We identified 14 content items corresponding to established OSA features and treatments. The content score was higher for scientific resources and foundations than for news/media reports, commercial websites, and personal commentaries. Medical consumers most frequently encounter commercial websites, which transmit medical information for a profit; therefore, these websites are very likely to provide industry-directed and inadequate information to medical consumers. In particular, we found that content scores for diagnosis and management, which are important for the readers to decide a treatment plan by themselves, were low for commercial websites. Several treatment options, including continuous positive airway pressure, oral appliances, lifestyle modifications, and surgery, should be considered for the successful management of OSA, and personalized and combination approaches should be evaluated according to the individual patient's characteristics.16 Therefore, websites should provide comprehensive medical information, including various treatment options and data that allow for the self-estimation of anatomical and economical characteristics. In addition, for the readers' convenience, we provide a list of the websites that contains higher content scores (> 14) in Table S1 in the supplemental material.

In the current study, not only the quantity of content but also the reliability and quality of information were poorer in commercial websites than in scientific resources and foundations. In particular, DISCERN scores for questions 9 through 15, which are associated with the treatment choice, were significantly lower for commercial websites than for scientific resources. Therefore, medical consumers may receive unreliable information concerning the diagnosis and treatment of OSA through commonly exposed commercial websites, resulting in inefficient medical use.

We found that only 14.8% of websites about OSA were certified by HON. HON is an independent organization that assesses whether websites provide understandable, accessible, and trustworthy information. Previous studies have established HON certification as a standard for the reliability of information.17 However, DISCERN, JAMA benchmark, and content scores for websites with HON certification were significantly lower than those for websites without HON certification in the current study. This unexpected finding suggests that it is very difficult for patients to obtain accurate and reliable information from the internet. Medical consumers should be aware that even certified websites may not contain accurate medical information.

Although we used United States versions of each search engine in this study, there could be a concern that our results were influenced by our location (Seoul, Republic of Korea). To address that concern, we asked our colleague in Philadelphia, Pennsylvania to search “sleep apnea” in the three search engines in the same manner as this study. Although it was impossible to compare the result statistically, the concordance rate between Korea and the United States was relatively high (Google: 83.2%, Bing: 93.1%, Yahoo: 87.2%).

As Table S2 in the supplemental material shows, the type, reliability, and quality of the websites were significantly different according to their ranking in the search results. It is well known that search engines report sites in an order based on the particular wording and other aspects of the websites. A search engine results page is generated based on a series of metrics that determines their relevance to the searched term.18 Webpages that score well on a search engine's algorithmic test are displayed higher in the list. These algorithms are generally based on factors such as the content of a webpage, the trustworthiness of the website, and other external factors.19 Furthermore, every major search engine accepts paid listings, and this guarantees that a paid site will appear in the top results for a related keyword. Therefore, medical consumers should be careful when they search health information online.

This research has some limitations. First, potential selection bias may have played a role during website selection and observer bias may have occurred during the assessment of the websites for readability and reliability. Second, an appropriate method for assessing the quality of content pertaining to OSA was not used. Third, we limited our research to online documents. However, other information sources such as YouTube, television, radio, and word of mouth are also significant. Therefore, future studies should focus on minimizing these biases and developing appropriate scoring systems for measuring the quality of information available through the internet. Moreover, additional studies need to be carried out in order to evaluate information from the other types of sources previously mentioned.

Although the quality of internet resources is inconsistent, the internet is still one of the easiest ways for patients to procure information about OSA. Therefore, it is important to develop appropriate standards for certifying such websites, so that patients can obtain reliable health-related information from the internet. In addition, health care providers should attempt to explain the diagnosis, prognosis, risk factors, and treatment of OSA in a manner that meets the patient's expectation levels.

CONCLUSIONS

In conclusion, our results suggest that most online health resources regarding OSA are not reliable and are inaccurate. Although medical information about OSA provided by websites is vast and readable, it is not very reliable, with most sites providing insufficient and biased medical information. As a result, patients may face difficulty in procuring appropriate and precise information from the internet, leading to adverse economic and individual health consequences. Therefore, web providers should give more comprehensive and sufficient information to medical consumers and establish a well-organized certification system to ensure the provision of good-quality and reliable medical information about OSA.

DISCLOSURE STATEMENT

Work for this study was performed at St Paul's Hospital, College of Medicine, The Catholic University of Korea. All authors have seen and approved this manuscript. The authors report no conflicts of interest.

ABBREVIATIONS

FRES

Flesch Reading Ease Score

HON

Health on the Net Foundation

JAMA

Journal of the American Medical Association

OSA

obstructive sleep apnea

ACKNOWLEDGMENTS

The authors thank JB Park at University of Pennsylvania, for supporting our study.

REFERENCES

1 

Carroll AE, Zimmerman FJ, Rivara FP, Ebel BE, Christakis DA. Perceptions about computers and the internet in a pediatric clinic population. Ambul Pediatr. 2005;5(2):122–126. [PubMed]

2 

Susannah F, Maeve D. Health Online 2013. http://www.pewinternet.org/2013/01/15/health-online-2013. Published January 15, 2013. Accessed August 28, 2016.

3 

Fisher JH, O'Connor D, Flexman AM, Shapera S, Ryerson CJ. Accuracy and reliability of internet resources for information on idiopathic pulmonary fibrosis. Am J Respir Crit Care Med. 2016;194(2):218–225. [PubMed]

4 

Grewal P, Williams B, Alagaratnam S, Neffendorf J, Soobrah R. Quality of vascular surgery Web sites on the Internet. J Vasc Surg. 2012;56(5):1461–1467. [PubMed]

5 

Kapur VK, Auckley DH, Chowdhuri S, et al. Clinical practice guideline for diagnostic testing for adult obstructive sleep apnea: an American Academy of Sleep Medicine clinical practice guideline. J Clin Sleep Med. 2017;13(3):479–504. [PubMed Central][PubMed]

6 

Budhiraja R, Budhiraja P, Quan SF. Sleep-disordered breathing and cardiovascular disorders. Respir Care. 2010;55(10):1322–1332; discussion 1330-1322. [PubMed Central][PubMed]

7 

Tregear S, Reston J, Schoelles K, Phillips B. Obstructive sleep apnea and risk of motor vehicle crash: systematic review and meta-analysis. J Clin Sleep Med. 2009;5(6):573–581. [PubMed Central][PubMed]

8 

Epstein LJ, Kristo D, Strollo PJ Jr, et al. Clinical guideline for the evaluation, management and long-term care of obstructive sleep apnea in adults. J Clin Sleep Med. 2009;5(3):263–276. [PubMed Central][PubMed]

9 

Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221–233. [PubMed]

10 

Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation Of New Readability Formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) For Navy Enlisted Personnel. Millington, TN: Institute for Simulation and Training, University of Central Florida; 1975.

11 

Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA. 1997;277(15):1244–1245. [PubMed]

12 

Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105–111. [PubMed Central][PubMed]

13 

Barker S, Charlton NP, Holstege CP. Accuracy of internet recommendations for prehospital care of venomous snake bites. Wilderness Environ Med. 2010;21(4):298–302. [PubMed]

14 

Hu X, Bell RA, Kravitz RL, Orrange S. The prepared patient: information seeking of online support group members before their medical appointments. J Health Commun. 2012;17(8):960–978. [PubMed]

15 

How to write easy to read health materials. https://medlineplus.gov/etr.html. Accessed August 28, 2016.

16 

Carberry JC, Amatoury J, Eckert DJ. Personalized management approach for OSA. Chest. 2018;153(3):744–755. [PubMed]

17 

Ball MJ. Twenty years of health on the net: committed to reliable information. Stud Health Technol Inform. 2016;225:738–740. [PubMed]

18 

Jansen BJ, Spink A. Investigating customer click through behaviour with integrated sponsored and nonsponsored results. International Journal of Internet Marketing and Advertising. 2009;5(1-2):74–94

19 

Juon C, Greiling D, Buerkle C. Internet Marketing Start to Finish: Drive Measurable, Repeatable Online Sales with Search Marketing, Usability, CRM, and Analytics. Indianapolis, IN: Que Publishing; 2012.


Supplemental Material

Supplemental Material

(.pdf | 35 KB)