Universal compression of memoryless sources over unknown alphabets

Alon Orlitsky, Narayana P. Santhanam, Junan Zhang

Research output: Contribution to journalArticle

94 Citations (Scopus)

Abstract

It has long been known that the compression redundancy of independent and identically distributed (i.i.d.) strings increases to infinity as the alphabet size grows. It is also apparent that any string can be described by separately conveying its symbols, and its pattern - the order in which the symbols appear. Concentrating on the latter, we show that the patterns of i.i.d. strings over all, including infinite and even unknown, alphabets, can be compressed with diminishing redundancy, both in block and sequentially, and that the compression can be performed in linear time. To establish these results, we show that the number of patterns is the Bell number, that the number of patterns with a given number of symbols is the Stirling number of the second kind, and that the redundancy of patterns can be bounded using results of Hardy and Ramanujan on the number of integer partitions. The results also imply an asymptotically optimal solution for the Good-Turing probability-estimation problem.

Original languageEnglish (US)
Pages (from-to)1469-1481
Number of pages13
JournalIEEE Transactions on Information Theory
Volume50
Issue number7
DOIs
StatePublished - Jul 2004
Externally publishedYes

Fingerprint

redundancy
Redundancy
symbol
Conveying

Keywords

  • Large and unknown alphabets
  • Patterns
  • Set and integer partitions
  • Universal compression

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Information Systems

Cite this

Universal compression of memoryless sources over unknown alphabets. / Orlitsky, Alon; Santhanam, Narayana P.; Zhang, Junan.

In: IEEE Transactions on Information Theory, Vol. 50, No. 7, 07.2004, p. 1469-1481.

Research output: Contribution to journalArticle

Orlitsky, Alon ; Santhanam, Narayana P. ; Zhang, Junan. / Universal compression of memoryless sources over unknown alphabets. In: IEEE Transactions on Information Theory. 2004 ; Vol. 50, No. 7. pp. 1469-1481.
@article{ba2d87252fd141429eb98acca334c54b,
title = "Universal compression of memoryless sources over unknown alphabets",
abstract = "It has long been known that the compression redundancy of independent and identically distributed (i.i.d.) strings increases to infinity as the alphabet size grows. It is also apparent that any string can be described by separately conveying its symbols, and its pattern - the order in which the symbols appear. Concentrating on the latter, we show that the patterns of i.i.d. strings over all, including infinite and even unknown, alphabets, can be compressed with diminishing redundancy, both in block and sequentially, and that the compression can be performed in linear time. To establish these results, we show that the number of patterns is the Bell number, that the number of patterns with a given number of symbols is the Stirling number of the second kind, and that the redundancy of patterns can be bounded using results of Hardy and Ramanujan on the number of integer partitions. The results also imply an asymptotically optimal solution for the Good-Turing probability-estimation problem.",
keywords = "Large and unknown alphabets, Patterns, Set and integer partitions, Universal compression",
author = "Alon Orlitsky and Santhanam, {Narayana P.} and Junan Zhang",
year = "2004",
month = "7",
doi = "10.1109/TIT.2004.830761",
language = "English (US)",
volume = "50",
pages = "1469--1481",
journal = "IEEE Transactions on Information Theory",
issn = "0018-9448",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "7",

}

TY - JOUR

T1 - Universal compression of memoryless sources over unknown alphabets

AU - Orlitsky, Alon

AU - Santhanam, Narayana P.

AU - Zhang, Junan

PY - 2004/7

Y1 - 2004/7

N2 - It has long been known that the compression redundancy of independent and identically distributed (i.i.d.) strings increases to infinity as the alphabet size grows. It is also apparent that any string can be described by separately conveying its symbols, and its pattern - the order in which the symbols appear. Concentrating on the latter, we show that the patterns of i.i.d. strings over all, including infinite and even unknown, alphabets, can be compressed with diminishing redundancy, both in block and sequentially, and that the compression can be performed in linear time. To establish these results, we show that the number of patterns is the Bell number, that the number of patterns with a given number of symbols is the Stirling number of the second kind, and that the redundancy of patterns can be bounded using results of Hardy and Ramanujan on the number of integer partitions. The results also imply an asymptotically optimal solution for the Good-Turing probability-estimation problem.

AB - It has long been known that the compression redundancy of independent and identically distributed (i.i.d.) strings increases to infinity as the alphabet size grows. It is also apparent that any string can be described by separately conveying its symbols, and its pattern - the order in which the symbols appear. Concentrating on the latter, we show that the patterns of i.i.d. strings over all, including infinite and even unknown, alphabets, can be compressed with diminishing redundancy, both in block and sequentially, and that the compression can be performed in linear time. To establish these results, we show that the number of patterns is the Bell number, that the number of patterns with a given number of symbols is the Stirling number of the second kind, and that the redundancy of patterns can be bounded using results of Hardy and Ramanujan on the number of integer partitions. The results also imply an asymptotically optimal solution for the Good-Turing probability-estimation problem.

KW - Large and unknown alphabets

KW - Patterns

KW - Set and integer partitions

KW - Universal compression

UR - http://www.scopus.com/inward/record.url?scp=3042606358&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=3042606358&partnerID=8YFLogxK

U2 - 10.1109/TIT.2004.830761

DO - 10.1109/TIT.2004.830761

M3 - Article

AN - SCOPUS:3042606358

VL - 50

SP - 1469

EP - 1481

JO - IEEE Transactions on Information Theory

JF - IEEE Transactions on Information Theory

SN - 0018-9448

IS - 7

ER -