Automating Content Extraction of HTML documents

Suhit Gupta, Gail E. Kaiser, Peter Grimm, Michael Chiang, Justin Starren

Research output: Contribution to journalArticle

63 Citations (Scopus)

Abstract

Web pages often contain clutter (such as unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting," which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction." We have developed a framework that employs an easily extensible set of techniques. It incorporates advantages of previous work on content extraction. Our key insight is to work with DOM trees, a W3C specified interface that allows programs to dynamically access document structure, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages. This proxy can be used both centrally, administered for groups of users, as well as by individuals for personal browsers. We have also, after receiving feedback from users about the proxy, created a revised version with improved performance and accessibility in mind.

Original languageEnglish (US)
Pages (from-to)179-224
Number of pages46
JournalWorld Wide Web
Volume8
Issue number2
DOIs
StatePublished - Jun 2005
Externally publishedYes

Fingerprint

HTML
Websites
Personal digital assistants
World Wide Web
Feedback

Keywords

  • Accessibility
  • Content extraction
  • DOM trees
  • HTML documents
  • Reformatting
  • Speech rendering
  • Text summarization

ASJC Scopus subject areas

  • Computer Networks and Communications

Cite this

Gupta, S., Kaiser, G. E., Grimm, P., Chiang, M., & Starren, J. (2005). Automating Content Extraction of HTML documents. World Wide Web, 8(2), 179-224. https://doi.org/10.1007/s11280-004-4873-3

Automating Content Extraction of HTML documents. / Gupta, Suhit; Kaiser, Gail E.; Grimm, Peter; Chiang, Michael; Starren, Justin.

In: World Wide Web, Vol. 8, No. 2, 06.2005, p. 179-224.

Research output: Contribution to journalArticle

Gupta, S, Kaiser, GE, Grimm, P, Chiang, M & Starren, J 2005, 'Automating Content Extraction of HTML documents', World Wide Web, vol. 8, no. 2, pp. 179-224. https://doi.org/10.1007/s11280-004-4873-3
Gupta, Suhit ; Kaiser, Gail E. ; Grimm, Peter ; Chiang, Michael ; Starren, Justin. / Automating Content Extraction of HTML documents. In: World Wide Web. 2005 ; Vol. 8, No. 2. pp. 179-224.
@article{be07a699523c440d9bb2c504d85b5809,
title = "Automating Content Extraction of HTML documents",
abstract = "Web pages often contain clutter (such as unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of {"}useful and relevant{"} content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike {"}Content Reformatting,{"} which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses {"}Content Extraction.{"} We have developed a framework that employs an easily extensible set of techniques. It incorporates advantages of previous work on content extraction. Our key insight is to work with DOM trees, a W3C specified interface that allows programs to dynamically access document structure, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages. This proxy can be used both centrally, administered for groups of users, as well as by individuals for personal browsers. We have also, after receiving feedback from users about the proxy, created a revised version with improved performance and accessibility in mind.",
keywords = "Accessibility, Content extraction, DOM trees, HTML documents, Reformatting, Speech rendering, Text summarization",
author = "Suhit Gupta and Kaiser, {Gail E.} and Peter Grimm and Michael Chiang and Justin Starren",
year = "2005",
month = "6",
doi = "10.1007/s11280-004-4873-3",
language = "English (US)",
volume = "8",
pages = "179--224",
journal = "World Wide Web",
issn = "1386-145X",
publisher = "Springer New York",
number = "2",

}

TY - JOUR

T1 - Automating Content Extraction of HTML documents

AU - Gupta, Suhit

AU - Kaiser, Gail E.

AU - Grimm, Peter

AU - Chiang, Michael

AU - Starren, Justin

PY - 2005/6

Y1 - 2005/6

N2 - Web pages often contain clutter (such as unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting," which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction." We have developed a framework that employs an easily extensible set of techniques. It incorporates advantages of previous work on content extraction. Our key insight is to work with DOM trees, a W3C specified interface that allows programs to dynamically access document structure, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages. This proxy can be used both centrally, administered for groups of users, as well as by individuals for personal browsers. We have also, after receiving feedback from users about the proxy, created a revised version with improved performance and accessibility in mind.

AB - Web pages often contain clutter (such as unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting," which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction." We have developed a framework that employs an easily extensible set of techniques. It incorporates advantages of previous work on content extraction. Our key insight is to work with DOM trees, a W3C specified interface that allows programs to dynamically access document structure, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages. This proxy can be used both centrally, administered for groups of users, as well as by individuals for personal browsers. We have also, after receiving feedback from users about the proxy, created a revised version with improved performance and accessibility in mind.

KW - Accessibility

KW - Content extraction

KW - DOM trees

KW - HTML documents

KW - Reformatting

KW - Speech rendering

KW - Text summarization

UR - http://www.scopus.com/inward/record.url?scp=14844363192&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=14844363192&partnerID=8YFLogxK

U2 - 10.1007/s11280-004-4873-3

DO - 10.1007/s11280-004-4873-3

M3 - Article

AN - SCOPUS:14844363192

VL - 8

SP - 179

EP - 224

JO - World Wide Web

JF - World Wide Web

SN - 1386-145X

IS - 2

ER -