World Library  
Flag as Inappropriate
Email this Article

Algorithmic learning theory

Article Id: WHEBN0000383480
Reproduction Date:

Title: Algorithmic learning theory  
Author: World Heritage Encyclopedia
Language: English
Subject: Computational learning theory, Computational epistemology, CN2 algorithm, Learning theory (education), Sample exclusion dimension
Collection: Computational Learning Theory, Formal Languages, Learning Theory (Education)
Publisher: World Heritage Encyclopedia

Algorithmic learning theory

Algorithmic learning theory is a mathematical framework for analyzing machine learning problems and algorithms. Synonyms include formal learning theory and algorithmic inductive inference. Algorithmic learning theory is different from statistical learning theory in that it does not make use of statistical assumptions and analysis. Both algorithmic and statistical learning theory are concerned with machine learning and can thus be viewed as branches of computational learning theory.


  • Distinguishing Characteristics 1
  • Learning in the limit 2
  • Other Identification Criteria 3
  • See also 4
  • References 5
  • External links 6

Distinguishing Characteristics

Unlike statistical learning theory and most statistical theory in general, algorithmic learning theory does not assume that data are random samples, that is, that data points are independent of each other. This makes the theory suitable for domains where observations are (relatively) noise-free but not random, such as language learning [1] and automated scientific discovery.[2][3]

The fundamental concept of algorithmic learning theory is learning in the limit: as the number of data points increases, a learning algorithm should converge to a correct hypothesis on every possible data sequence consistent with the problem space. This is a non-probabilistic version of statistical consistency, which also requires convergence to a correct model in the limit, but allows a learner to fail on data sequences with probability measure 0.

Algorithmic learning theory investigates the learning power of Turing machines. Other frameworks consider a much more restricted class of learning algorithms than Turing machines, for example learners that compute hypotheses more quickly, for instance in polynomial time. An example of such a framework is probably approximately correct learning.

Learning in the limit

The concept was introduced in E. Mark Gold's seminal paper "Language identification in the limit".[4] The objective of language identification is for a machine running one program to be capable of developing another program by which any given sentence can be tested to determine whether it is "grammatical" or "ungrammatical". The language being learned need not be English or any other natural language - in fact the definition of "grammatical" can be absolutely anything known to the tester.

In Gold's learning model, the tester gives the learner an example sentence at each step, and the learner responds with a hypothesis, which is a suggested program to determine grammatical correctness. It is required of the tester that every possible sentence (grammatical or not) appears in the list eventually, but no particular order is required. It is required of the learner that at each step the hypothesis must be correct for all the sentences so far.

A particular learner is said to be able to "learn a language in the limit" if there is a certain number of steps beyond which its hypothesis no longer changes. At this point it has indeed learned the language, because every possible sentence appears somewhere in the sequence of inputs (past or future), and the hypothesis is correct for all inputs (past or future), so the hypothesis is correct for every sentence. The learner is not required to be able to tell when it has reached a correct hypothesis, all that is required is that it be true.

Gold showed that any language which is defined by a Turing machine program can be learned in the limit by another Turing-complete machine using enumeration. This is done by the learner testing all possible Turing machine programs in turn until one is found which is correct so far - this forms the hypothesis for the current step. Eventually, the correct program will be reached, after which the hypothesis will never change again (but note that the learner does not know that it won't need to change).

Gold also showed that if the learner is given only positive examples (that is, only grammatical sentences appear in the input, not ungrammatical sentences), then the language can only be guaranteed to be learned in the limit if there are only a finite number of possible sentences in the language (this is possible if, for example, sentences are known to be of limited length).

Language identification in the limit is a highly abstract model. It does not allow for limits of runtime or computer memory which can occur in practice, and the enumeration method may fail if there are errors in the input. However the framework is very powerful, because if these strict conditions are maintained, it allows the learning of any program known to be computable. This is because a Turing machine program can be written to mimic any program in any conventional programming language. See Church-Turing thesis.

Other Identification Criteria

Learning theorists have investigated other learning criteria,[5] such as the following.

  • Efficiency: minimizing the number of data points required before convergence to a correct hypothesis.
  • Mind Changes: minimizing the number of hypothesis changes that occur before convergence.[6]

Mind change bounds are closely related to mistake bounds that are studied in statistical learning theory.[7] Kevin Kelly has suggested that minimizing mind changes is closely related to choosing maximally simple hypotheses in the sense of Occam’s Razor.[8]

See also


  1. ^ Jain, S. et al (1999): Systems That Learn, 2nd ed. Cambridge, MA: MIT Press.
  2. ^ Langley, P.; Simon, H.; Bradshaw, G. & Zytkow, J. (1987), Scientific Discovery: Computational Explorations of the Creative Processes, MIT Press, Cambridge
  3. ^ Schulte, O. (2009), Simultaneous Discovery of Conservation Laws and Hidden Particles With Smith Matrix Decomposition, in Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), pp. 1481-1487
  4. ^ Gold, E. Mark (1967), Language Identification in the Limit (PDF) 10,   The original paper.
  5. ^ Jain, S. et al (1999): Systems That Learn, 2nd ed. Cambridge, MA: MIT Press.
  6. ^ Luo, W. & Schulte, O. (2005), Mind Change Efficient Learning, in Peter Auer & Ron Meir, ed., Proceedings of the Conference on Learning Theory (COLT), pp. 398-412
  7. ^ Jain, S. and Sharma, A. (1999), On a generalized notion of mistake bounds, Proceedings of the Conference on Learning Theory (COLT), pp.249-256.
  8. ^ Kevin T. Kelly (2007), Ockham’s Razor, Empirical Complexity, and Truth-finding Efficiency, Theoretical Computer Science, 383: 270-289.

External links

  • Learning Theory in Computer Science.
  • The Stanford Encyclopaedia of Philosophy provides a highly accessible introduction to key concepts in algorithmic learning theory, especially as they apply to the philosophical problems of inductive inference.
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.

Copyright © World Library Foundation. All rights reserved. eBooks from World eBook Library are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.