Assessing the Feasibility of Large-Scale Natural Language Processing in a Corpus of Ordinary Medical Records

A Lexical Analysis

William (Bill) Hersh, Emily M. Campbell, Susan E. Malveau

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

Objective: Identify the lexical content of a large corpus of ordinary medical records to assess the feasibility of large-scale natural language processing. Methods: A corpus of 560 megabytes of medical record text from an academic medical center was broken into individual words and compared with the words in six medical vocabularies, a common word list, and a database of patient names. Unrecognized words were assessed for algorithmic and contextual approaches to identifying more words, while the remainder were analyzed for spelling correctness. Results: About 60% of the words occurred in the medical vocabularies, common word list, or names database. Of the remainder, one-third were recognizable by other means. Of the remaining unrecognizable words, over three-fourths represented correctly spelled real words and the rest were misspellings. Conclusions: Large-scale generalized natural language processing methods for the medical record will require expansion of existing vocabularies, spelling error correction, and other algorithmic approaches to map words into those from clinical vocabularies.

Original languageEnglish (US)
Pages (from-to)580-584
Number of pages5
JournalJournal of the American Medical Informatics Association
Volume4
Issue numberSUPPL.
StatePublished - 1997
Externally publishedYes

Fingerprint

Natural Language Processing
Vocabulary
Medical Records
Names
Databases

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Assessing the Feasibility of Large-Scale Natural Language Processing in a Corpus of Ordinary Medical Records : A Lexical Analysis. / Hersh, William (Bill); Campbell, Emily M.; Malveau, Susan E.

In: Journal of the American Medical Informatics Association, Vol. 4, No. SUPPL., 1997, p. 580-584.

Research output: Contribution to journalArticle

@article{e63f5967e3e04188ba9340bb7d340213,
title = "Assessing the Feasibility of Large-Scale Natural Language Processing in a Corpus of Ordinary Medical Records: A Lexical Analysis",
abstract = "Objective: Identify the lexical content of a large corpus of ordinary medical records to assess the feasibility of large-scale natural language processing. Methods: A corpus of 560 megabytes of medical record text from an academic medical center was broken into individual words and compared with the words in six medical vocabularies, a common word list, and a database of patient names. Unrecognized words were assessed for algorithmic and contextual approaches to identifying more words, while the remainder were analyzed for spelling correctness. Results: About 60{\%} of the words occurred in the medical vocabularies, common word list, or names database. Of the remainder, one-third were recognizable by other means. Of the remaining unrecognizable words, over three-fourths represented correctly spelled real words and the rest were misspellings. Conclusions: Large-scale generalized natural language processing methods for the medical record will require expansion of existing vocabularies, spelling error correction, and other algorithmic approaches to map words into those from clinical vocabularies.",
author = "Hersh, {William (Bill)} and Campbell, {Emily M.} and Malveau, {Susan E.}",
year = "1997",
language = "English (US)",
volume = "4",
pages = "580--584",
journal = "Journal of the American Medical Informatics Association",
issn = "1067-5027",
publisher = "Oxford University Press",
number = "SUPPL.",

}

TY - JOUR

T1 - Assessing the Feasibility of Large-Scale Natural Language Processing in a Corpus of Ordinary Medical Records

T2 - A Lexical Analysis

AU - Hersh, William (Bill)

AU - Campbell, Emily M.

AU - Malveau, Susan E.

PY - 1997

Y1 - 1997

N2 - Objective: Identify the lexical content of a large corpus of ordinary medical records to assess the feasibility of large-scale natural language processing. Methods: A corpus of 560 megabytes of medical record text from an academic medical center was broken into individual words and compared with the words in six medical vocabularies, a common word list, and a database of patient names. Unrecognized words were assessed for algorithmic and contextual approaches to identifying more words, while the remainder were analyzed for spelling correctness. Results: About 60% of the words occurred in the medical vocabularies, common word list, or names database. Of the remainder, one-third were recognizable by other means. Of the remaining unrecognizable words, over three-fourths represented correctly spelled real words and the rest were misspellings. Conclusions: Large-scale generalized natural language processing methods for the medical record will require expansion of existing vocabularies, spelling error correction, and other algorithmic approaches to map words into those from clinical vocabularies.

AB - Objective: Identify the lexical content of a large corpus of ordinary medical records to assess the feasibility of large-scale natural language processing. Methods: A corpus of 560 megabytes of medical record text from an academic medical center was broken into individual words and compared with the words in six medical vocabularies, a common word list, and a database of patient names. Unrecognized words were assessed for algorithmic and contextual approaches to identifying more words, while the remainder were analyzed for spelling correctness. Results: About 60% of the words occurred in the medical vocabularies, common word list, or names database. Of the remainder, one-third were recognizable by other means. Of the remaining unrecognizable words, over three-fourths represented correctly spelled real words and the rest were misspellings. Conclusions: Large-scale generalized natural language processing methods for the medical record will require expansion of existing vocabularies, spelling error correction, and other algorithmic approaches to map words into those from clinical vocabularies.

UR - http://www.scopus.com/inward/record.url?scp=0347210453&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0347210453&partnerID=8YFLogxK

M3 - Article

VL - 4

SP - 580

EP - 584

JO - Journal of the American Medical Informatics Association

JF - Journal of the American Medical Informatics Association

SN - 1067-5027

IS - SUPPL.

ER -