Automating Content Extraction of HTML documents

Suhit Gupta, Gail E. Kaiser, Peter Grimm, Michael F. Chiang, Justin Starren

Research output: Contribution to journalArticle

64 Scopus citations

Abstract

Web pages often contain clutter (such as unnecessary images and extraneous links) around the body of an article that distracts a user from actual content. Extraction of "useful and relevant" content from web pages has many applications, including cell phone and PDA browsing, speech rendering for the visually impaired, and text summarization. Most approaches to making content more readable involve changing font size or removing HTML and data components such as images, which takes away from a webpage's inherent look and feel. Unlike "Content Reformatting," which aims to reproduce the entire webpage in a more convenient form, our solution directly addresses "Content Extraction." We have developed a framework that employs an easily extensible set of techniques. It incorporates advantages of previous work on content extraction. Our key insight is to work with DOM trees, a W3C specified interface that allows programs to dynamically access document structure, rather than with raw HTML markup. We have implemented our approach in a publicly available Web proxy to extract content from HTML web pages. This proxy can be used both centrally, administered for groups of users, as well as by individuals for personal browsers. We have also, after receiving feedback from users about the proxy, created a revised version with improved performance and accessibility in mind.

Original languageEnglish (US)
Pages (from-to)179-224
Number of pages46
JournalWorld Wide Web
Volume8
Issue number2
DOIs
StatePublished - Jun 1 2005

Keywords

  • Accessibility
  • Content extraction
  • DOM trees
  • HTML documents
  • Reformatting
  • Speech rendering
  • Text summarization

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Automating Content Extraction of HTML documents'. Together they form a unique fingerprint.

  • Cite this

    Gupta, S., Kaiser, G. E., Grimm, P., Chiang, M. F., & Starren, J. (2005). Automating Content Extraction of HTML documents. World Wide Web, 8(2), 179-224. https://doi.org/10.1007/s11280-004-4873-3