Introduction

TheanoLM is a recurrent neural network language modeling tool implemented using the Python library Theano. Theano allows the user to customize and extend the neural network very conveniently, still generating highly efficient code that can utilize multiple GPUs or CPUs for parallel computation. TheanoLM allows the user to specify an arbitrary network architecture. New layer types and optimization methods can be easily implemented.

TheanoLM can be used for rescoring n-best lists and Kaldi lattices, decoding HTK word lattices, and generating text. It can be called from command line or from a Python script.

Implementations of many currently popular layer types are provided, such as long short-term memory (LSTM), gated recurrent units (GRU), bidirectional recurrent networks, gated linear units (GLU), and highway networks are provided. Several different Stochastic Gradient Descent (SGD) based optimizers are implemented, including RMSProp, AdaGrad, ADADELTA, and Adam.

There are several features that are especially useful with very large vocabularies. The effective vocabulary size can be reduced by using a class model. TheanoLM supports also subword vocabularies create e.g. using Morfessor. In addition to the standard cross-entropy cost, one can use sampling based noise-contrastive estimation (NCE) or BlackOut.

Publications

Seppo Enarvi and Mikko Kurimo (2016), TheanoLM — An Extensible Toolkit for Neural Network Language Modeling. In Proceedings of the 17th Annual Conference of the International Speech Communication Association (INTERSPEECH).

Seppo Enarvi, Peter Smit, Sami Virpioja, and Mikko Kurimo (2017), Automatic Speech Recognition with Very Large Conversational Finnish and Estonian Vocabularies. In IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Peter Smit, Siva Gangireddy, Seppo Enarvi, Sami Virpioja, and Mikko Kurimo (2017), Aalto System for the 2017 Arabic Multigenre Broadcast Challenge. In Proceedings of the 2017 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).

License

TheanoLM is licensed under the Apache License, Version 2.0.