Bayesian supervised dimensionality reduction

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Dimensionality reduction is commonly used as a preprocessing step before training a supervised learner. However, coupled training of dimensionality reduction and supervised learning steps may improve the prediction performance. In this paper, we introduce a simple and novel Bayesian supervised dimensionality reduction method that combines linear dimensionality reduction and linear supervised learning in a principled way. We present both Gibbs sampling and variational approximation approaches to learn the proposed probabilistic model for multiclass classification. We also extend our formulation toward model selection using automatic relevance determination in order to find the intrinsic dimensionality. Classification experiments on three benchmark data sets show that the new model significantly outperforms seven baseline linear dimensionality reduction algorithms on very low dimensions in terms of generalization performance on test data. The proposed model also obtains the best results on an image recognition task in terms of classification and retrieval performances.

Original languageEnglish (US)
Article number6475174
Pages (from-to)2179-2189
Number of pages11
JournalIEEE Transactions on Cybernetics
Volume43
Issue number6
DOIs
StatePublished - Dec 2013
Externally publishedYes

Fingerprint

Supervised learning
Image recognition
Sampling
Experiments
Statistical Models

Keywords

  • Dimensionality reduction
  • Gibbs sampling
  • Handwritten digit recognition
  • Image recognition
  • Image retrieval
  • Multiclass classification
  • Subspace learning
  • Variational approximation

ASJC Scopus subject areas

  • Computer Science Applications
  • Human-Computer Interaction
  • Information Systems
  • Software
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this

Bayesian supervised dimensionality reduction. / Gonen, Mehmet.

In: IEEE Transactions on Cybernetics, Vol. 43, No. 6, 6475174, 12.2013, p. 2179-2189.

Research output: Contribution to journalArticle

@article{b2c60b21de664d97935a7ff34accffe4,
title = "Bayesian supervised dimensionality reduction",
abstract = "Dimensionality reduction is commonly used as a preprocessing step before training a supervised learner. However, coupled training of dimensionality reduction and supervised learning steps may improve the prediction performance. In this paper, we introduce a simple and novel Bayesian supervised dimensionality reduction method that combines linear dimensionality reduction and linear supervised learning in a principled way. We present both Gibbs sampling and variational approximation approaches to learn the proposed probabilistic model for multiclass classification. We also extend our formulation toward model selection using automatic relevance determination in order to find the intrinsic dimensionality. Classification experiments on three benchmark data sets show that the new model significantly outperforms seven baseline linear dimensionality reduction algorithms on very low dimensions in terms of generalization performance on test data. The proposed model also obtains the best results on an image recognition task in terms of classification and retrieval performances.",
keywords = "Dimensionality reduction, Gibbs sampling, Handwritten digit recognition, Image recognition, Image retrieval, Multiclass classification, Subspace learning, Variational approximation",
author = "Mehmet Gonen",
year = "2013",
month = "12",
doi = "10.1109/TCYB.2013.2245321",
language = "English (US)",
volume = "43",
pages = "2179--2189",
journal = "IEEE Transactions on Cybernetics",
issn = "2168-2267",
publisher = "IEEE Advancing Technology for Humanity",
number = "6",

}

TY - JOUR

T1 - Bayesian supervised dimensionality reduction

AU - Gonen, Mehmet

PY - 2013/12

Y1 - 2013/12

N2 - Dimensionality reduction is commonly used as a preprocessing step before training a supervised learner. However, coupled training of dimensionality reduction and supervised learning steps may improve the prediction performance. In this paper, we introduce a simple and novel Bayesian supervised dimensionality reduction method that combines linear dimensionality reduction and linear supervised learning in a principled way. We present both Gibbs sampling and variational approximation approaches to learn the proposed probabilistic model for multiclass classification. We also extend our formulation toward model selection using automatic relevance determination in order to find the intrinsic dimensionality. Classification experiments on three benchmark data sets show that the new model significantly outperforms seven baseline linear dimensionality reduction algorithms on very low dimensions in terms of generalization performance on test data. The proposed model also obtains the best results on an image recognition task in terms of classification and retrieval performances.

AB - Dimensionality reduction is commonly used as a preprocessing step before training a supervised learner. However, coupled training of dimensionality reduction and supervised learning steps may improve the prediction performance. In this paper, we introduce a simple and novel Bayesian supervised dimensionality reduction method that combines linear dimensionality reduction and linear supervised learning in a principled way. We present both Gibbs sampling and variational approximation approaches to learn the proposed probabilistic model for multiclass classification. We also extend our formulation toward model selection using automatic relevance determination in order to find the intrinsic dimensionality. Classification experiments on three benchmark data sets show that the new model significantly outperforms seven baseline linear dimensionality reduction algorithms on very low dimensions in terms of generalization performance on test data. The proposed model also obtains the best results on an image recognition task in terms of classification and retrieval performances.

KW - Dimensionality reduction

KW - Gibbs sampling

KW - Handwritten digit recognition

KW - Image recognition

KW - Image retrieval

KW - Multiclass classification

KW - Subspace learning

KW - Variational approximation

UR - http://www.scopus.com/inward/record.url?scp=84890110725&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84890110725&partnerID=8YFLogxK

U2 - 10.1109/TCYB.2013.2245321

DO - 10.1109/TCYB.2013.2245321

M3 - Article

C2 - 23757527

AN - SCOPUS:84890110725

VL - 43

SP - 2179

EP - 2189

JO - IEEE Transactions on Cybernetics

JF - IEEE Transactions on Cybernetics

SN - 2168-2267

IS - 6

M1 - 6475174

ER -