Debian Science Linguistics packages

An open-source shallow-transfer machine translation engine, Apertium is initially aimed at related-language pairs.

It uses finite-state transducers for lexical processing, hidden Markov models for part-of-speech tagging, and finite-state based chunking for structural transfer.

The system is largely based upon systems already developed by the Transducens group at the Universitat d'Alacant, such as interNOSTRUM (Spanish-Catalan, http://www.internostrum.com/welcome.php) and Traductor Universia (Spanish-Portuguese, http://traductor.universia.net).

It will be possible to use Apertium to build machine translation systems for a variety of related-language pairs simply providing the linguistic data needed in the right format.

This package contails Perl scripts to evaluate Apertium-based machine translation output against reference: WER, PER, TER, BLEU.

Module for compiling lexical selection rules and processing them in the pipeline.

Artha is a off-line English thesaurus with distinct features like:

• hot-key press word look-up (select text on any window and press a preset hot-key for look-up)
• regular expressions based search (broaden search using wild-cards like *, ?, etc.)
• passive desktop notifications (of word definitions for uninterrupted work-flow)
• spelling suggestions (when the exact spelling is vague/not known)

Once launched, it monitors for a preset hot-key combination. When some text is selected on any window and the hot-key is pressed, it pops-up with the word looked-up. Should the user prefer passive notifications, this can be done by enabling the notifications option.

When the term looked for is vague/not known, then either the search can be broadened with the use of regular expressions (*, ?, etc.) in the search string or spelling suggestions when a term is incorrect.

For regular expressions based search to work, wordnet-sense-index package is required.

Constraint Grammar compiler and applicator for the 3rd edition of CG that is developed and maintained by VISL SDU and GrammarSoft ApS.

CG-3 can be used for disambiguation of morphology, syntax, semantics, etc; dependency markup, target language lemma choice for MT, QA systems, and much more. The core idea is that you choose what to do based on the whole available context, as opposed to n-grams.

See https://visl.sdu.dk/cg3.html for more documentation

Collatinus can be used to lemmatise latin texts, i.e. extract words and make a lexicon which indicates for each word its canonic form, and how the form actually found in the text was derived from it, for instance by declining it. Example : rosam gives : rosa-rosae -- acc. sing. Collatinus provides a nice graphic front-end to each operation.

Collatinus-nouus (stands for Collatinus, new generation) replaces every previous version of Collatinus.

This package provides a documentation in HTML format.

Dimbl is a wrapper around the k-nearest neighbor classifier in TiMBL, offering parallel classification on multi-CPU machines. Dimbl splits the original training set, builds separate TiMBL classifiers per training subset, and merges their nearest-neighbor sets per classified instance

Dimbl's features are:

• Wraps neatly around TiMBL, retaining all command line options;
• Knows what to do with your multiple, duo, or quad cores;
• Makes use of the OpenMP specification for parallel programming;
• Can attain superlinear speed gains compared to standard TiMBL.

Dimbl is a product of the ILK Research Group (Tilburg University, The Netherlands).

If you do scientific research in Natural Language Processing using the Memory-Based Learning technique, Dimbl will likely be of use to you.

fastText is a library for efficient learning of word representations and sentence classification, which refers subword information to enrich word vectors.

Memory-Based Learning (MBL) is a machine-learning method applicable to a wide range of tasks in Natural Language Processing (NLP).

Frog is a modular system integrating a morphosyntactic tagger, lemmatizer, morphological analyzer, and dependency parser for natural languages. It is based upon it's predecessor TADPOLE (TAgger, Dependency Parser, and mOrphoLogical analyzEr). Using Memory-Based Learning techniques, frog tokenizes, tags, lemmatizes, and morphologically segments word tokens in incoming UTF-8 text files, and assigns a dependency graph to each sentence. Frog is particularly targeted at the increasing need for fast, automatic NLP systems applicable to very large (multi-million to billion word) document collections that are becoming available due to the progressive digitization of both new and old textual data. Up to now, frog has only been tested and used using corpora of Dutch natural language (see the frogdata package for samples).

Frog is a product of the Centre of Language and Speech Technology at Radboud University Nijmegen, it subsumes previous work by the ILK Research Group (Tilburg University, The Netherlands) and the CLiPS Research Centre (University of Antwerp, Belgium). It is currently maintained at the KNAW Humanities Cluster.

If you do scientific research in NLP, Frog will likely be of use to you.

Data package providing Giellatekno language resources for North Saami.

The Helsinki Finite-State Transducer software is intended for the implementation of morphological analysers and other tools which are based on weighted and unweighted finite-state transducer technology.

Minimal HFST optimized lookup format based spell checker library and a demonstrational implementation of command line based spell checker.

The IRST Language Modeling Toolkit can be used to learn a language model from data. The generated n-gram models should be usable on any system supporting ARPA language model format.

This package provides the command line tools.

Detects over 80 languages in UTF-8 text, based largely on groups of four letters. Also tables for 160+ language version.

This is the development package.

In Sleator, D. and Temperley, D. "Parsing English with a Link Grammar" (1991), the authors defined a new formal grammatical system called a "link grammar". A sequence of words is in the language of a link grammar if there is a way to draw "links" between words in such a way that the local requirements of each word are satisfied, the links do not cross, and the words form a connected graph. The authors encoded English grammar into such a system, and wrote this program to parse English using this grammar.

link-grammar can be used for linguistic parsing for information retrieval or extraction from natural language documents. It can also be used as a grammar checker.

This package contains the user-executable binary.

The lttoolbox contains the augmented letter transducer tools for natural language processing used by Apertium, a platform for building rule-based and hybrid machine translation systems. The software is also useful for making morphological analysers and generators for natural language processing applications.

MBT is a memory-based tagger-generator and tagger in one. The tagger-generator part can generate a sequence tagger on the basis of a training set of tagged sequences; the tagger part can tag new sequences. MBT can, for instance, be used to generate part-of-speech taggers or chunkers for natural language processing. Features:

• Tagger generation: tagged text in, tagger out,
• Optional feedback loop: feed previous tag decision back to input of next decision,
• Easily customizable feature representation; can incorporate user-provided features,
• Automatic generation of separate sub-taggers for known words and unknown words,
• Can make use of full algorithmic parameters of TiMBL.

MBT is a product of the Centre of Language and Speech Technology (Radboud University Nijmegen, The Netherlands), the ILK Research Group (Tilburg University, The Netherlands) and the CLiPS Research Centre (University of Antwerp, Belgium).

If you do scientific research in natural language processing, MBT will likely be of use to you.

MbtServer extends Mbt with a server layer, running as a TCP server. Mbt is a memory-based tagger-generator and tagger for natural language processing. MbtServer provides the possibility to access a trained tagger from multiple sessions. It also allows one to run and access different taggers in parallel.

MbtServer is a product of the Centre for Language and Speech Technology (Radboud University, Nijmegen, The Netherlands), the ILK Research Group (Tilburg University, The Netherlands) and the CLiPS Research Centre (University of Antwerp, Belgium).

If you do scientific research in natural language processing, MbtServer will likely be of use to you.

The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution. These tasks are usually required to build more advanced text processing services. OpenNLP also included maximum entropy and perceptron based machine learning.

This package contains the command line wrapper.

PyNLPl, pronounced as 'pineapple', is a Python library for Natural Language Processing. It contains various modules useful for common, and less common, NLP tasks. PyNLPl can be used for basic tasks such as the extraction of n-grams and frequency lists, and to build simple language models. It also contains complex data types and algorithms. Moreover, it includes parsers for file formats common in NLP (e.g. FoLiA/Giza/Moses/ARPA/Timbl/CQL) and clients to interface with various NLP specific servers. PyNLPl most notably features a very extensive library for working with FoLiA XML (Format for Linguistic Annotation).

This is the Python 3 version.

Thinc is the machine learning library powering spaCy https://spacy.io. It features a battle-tested linear model designed for large sparse learning problems, and a flexible neural network model under development for spaCy v2.0 https://spacy.io/usage/v2.

Thinc is a practical toolkit for implementing models that follow the "Embed, encode, attend, predict" architecture. It's designed to be easy to install, efficient for CPU usage and optimised for NLP and deep learning with text – in particular, hierarchically structured input and variable-length sequences.

An R implementation of the LexRank algorithm implementing stochastic graph-based method for computing relative importance of textual units for Natural Language Processing. The technique on the problem of Text Summarization (TS) is tested. Extractive TS relies on the concept of sentence salience to identify the most important sentences in a document or set of documents. Salience is typically defined in terms of the presence of particular important words or in terms of similarity to a centroid pseudo-sentence.

An R interface to the C libstemmer library that implements Porter's word stemming algorithm for collapsing words to a common root to aid comparison of vocabulary. Currently supported languages are Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish.

SentencePiece is an unsupervised text tokenizer/detokenizer mainly designed for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training.

Memory-Based Learning (MBL) is a machine-learning method applicable to a wide range of tasks in Natural Language Processing (NLP).

The Tilburg Memory Based Learner, TiMBL, is a tool for NLP research, and for many other domains where classification tasks are learned from examples. It is an efficient implementation of k-nearest neighbor classifier.

TiMBL's features are:

• Fast, decision-tree-based implementation of k-nearest neighbor classification;

• Implementations of IB1 and IB2, IGTree, TRIBL, and TRIBL2 algorithms;

• Similarity metrics: Overlap, MVDM, Jeffrey Divergence, Dot product, Cosine;

• Feature weighting metrics: information gain, gain ratio, chi squared, shared variance;

• Distance weighting metrics: inverse, inverse linear, exponential decay;

• Extensive verbosity options to inspect nearest neighbor sets;
• Server functionality and extensive API;
• Fast leave-one-out testing and internal cross-validation;
• and Handles user-defined example weighting.

TiMBL is a product of the Centre of Language and Speech Technology (Radboud University, Nijmegen, The Netherlands), the ILK Research Group (Tilburg University, The Netherlands) and the CLiPS Research Centre (University of Antwerp, Belgium).

If you do scientific research in NLP, timbl will likely be of use to you.

timblserver is a TiMBL wrapper; it adds server functionality to TiMBL. It allows TiMBL to run multiple experiments as a TCP server, optionally via HTTP.

The Tilburg Memory Based Learner, TiMBL, is a tool for Natural Language Processing research, and for many other domains where classification tasks are learned from examples.

TimblServer is a product of the ILK Research Group (Tilburg University, The Netherlands) and the CLiPS Research Centre (University of Antwerp, Belgium).

If you do scientific research in NLP, TimblServer will likely be of use to you.

Ucto can tokenize UTF-8 encoded text files (i.e. separate words from punctuation, split sentences, generate n-grams), and offers several other basic preprocessing steps that make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation.

This package provides the command-line tool itself.

Ucto was written by Maarten van Gompel and Ko van der Sloot. Work on Ucto was funded by NWO, the Netherlands Organisation for Scientific Research, under the Implicit Linguistics project, the CLARIN-NL program, and the CLARIAH project.

Ucto is a product of the Centre of Language and Speech Technology (Radboud University Nijmegen), and previously the ILK Research Group (Tilburg University, The Netherlands).

If you are interested in machine parsing of UTF-8 encoded text files, e.g. to do scientific research in natural language processing, ucto will likely be of use to you.

Ucto can tokenize UTF-8 encoded text files (i.e. separate words from punctuation, split sentences, generate n-grams), and offers several other basic preprocessing steps that make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation.

This package provides necessary language-specific datafiles for running Ucto.

Ucto was written by Maarten van Gompel and Ko van der Sloot. Work on Ucto was funded by NWO, the Netherlands Organisation for Scientific Research, under the Implicit Linguistics project, the CLARIN-NL program, and the CLARIAH project.

Ucto is a product of the Centre of Language and Speech Technology (Radboud University Nijmegen), and previously the ILK Research Group (Tilburg University, The Netherlands).

WordNet(C) is an on-line lexical reference system whose design is inspired by current psycholinguistic theories of human lexical memory. English nouns, verbs, adjectives and adverbs are organized into synonym sets, each representing one underlying lexical concept. Different relations link the synonym sets.

WordNet was developed by the Cognitive Science Laboratory at Princeton University under the direction of Professor George A. Miller (Principal Investigator).

WordNet is considered to be the most important resource available to researchers in computational linguistics, text analysis, and many related areas.

Binary and manpages of WordNet as well as general manpages.

This is a transitional dummy package. It can safely be removed.

This package contains Apertium APY which is simple Apertium API written in Python 3 meant as a drop-in replacement for ScaleMT.

Data package providing Apertium language resources for Aragonese

Data package providing Apertium language resources for translating between the Aragonese and Catalan languages.

Data package providing Apertium language resources for Belarusian.

Data package providing Apertium language resources for translating between the Belarusian and Russian languages.

This is a linguistic package for the Apertium shallow-transfer machine translation system. The package can be used to translate between Breton and French.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for Catalan.

Data package providing Apertium language resources for translating between the Catalan and Sardinian languages.

Data package providing Apertium language resources for Crimean Tatar

Data package providing Apertium language resources for translating between the Crimean Tatar and Turkish languages.

Data package providing Apertium language resources for translating between the Welsh and English languages.

Data package providing Apertium language resources for Danish.

Data package providing Apertium language resources for translating from the Danish to the Norwegian Nynorsk/Norwegian Bokmål variants and from Danish to Norwegian Nynorsk.

This is a transitional dummy package. It can safely be removed.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for translating between the English and Galician languages.

Data package providing Apertium language resources for translating between the Esperanto and Catalan languages.

This is a linguistic package for the Apertium shallow-transfer machine translation system. The package can be used to translate between Esperanto and English.

Data package providing Apertium language resources for translating between the Esperanto and Spanish languages.

Data package providing Apertium language resources for translating between the Esperanto and French languages.

This is a transitional dummy package. It can safely be removed.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for translating between the Spanish and Galician languages.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for translating between the Spanish and Portuguese languages.

Data package providing Apertium language resources for translating between the Spanish and Romanian languages.

Data package providing Apertium language resources for translating between the Basque and English languages.

Data package providing Apertium language resources for translating between the Basque and Spanish languages.

Data package providing Apertium language resources for translating between the French and Spanish languages.

Data package providing Apertium language resources for French.

Data package providing Apertium language resources for translating between the French and Catalan languages.

Data package providing Apertium language resources for Serbo-Croatian.

Data package providing Apertium language resources for translating between the Serbo-Croatian and English languages.

Data package providing Apertium language resources for translating between the Serbo-Croatian and Macedonian languages.

Data package providing Apertium language resources for translating between the Serbo-Croatian and Slovenian languages.

Data package providing Apertium language resources for Hindi.

This is a transitional dummy package. It can safely be removed.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for Icelandic.

Data package providing Apertium language resources for translating between the Icelandic and English languages.

Data package providing Apertium language resources for Italian.

Data package providing Apertium language resources for Kazakh

Data package providing Apertium language resources for translating between the Kazakh and Tatar languages.

This is a transitional dummy package. It can safely be removed.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for translating between the Maltese and Arabic languages.

Data package providing Apertium language resources for Norwegian Nynorsk.

Data package providing Apertium language resources for translating between the Norwegian Nynorsk and Norwegian Bokmål languages.

Data package providing Apertium language resources for Norwegian Bokmål.

Data package providing Apertium language resources for translating between the Occitan and Catalan languages.

Data package providing Apertium language resources for translating between the Occitan and Spanish languages.

Data package providing Apertium language resources for Occitan.

Data package providing Apertium language resources for Polish.

Data package providing Apertium language resources for translating between the Polish and Silesian languages.

This is a transitional dummy package. It can safely be removed.

Data package providing Apertium language resources for translating between the Portuguese and Galician languages.

Data package providing Apertium language resources for Russian

Apertium module for reordering separable/discontiguous multiwords.

Data package providing Apertium language resources for translating between the Northern Sami and Norwegian Bokmål languages.

Data package providing Apertium language resources for Spanish.

Data package providing Apertium language resources for translating between the Spanish and Aragonese languages.

Data package providing Apertium language resources for Sardinian.

Data package providing Apertium language resources for translating between the Sardinian and Italian languages.

Data package providing Apertium language resources for Swedish.

Data package providing Apertium language resources for translating between the Swedish and Danish languages.