Linguistics: Corpora

Subset of the Switchboard-1 (LDC93S7) corpus, was used in NISTs 1996 Speaker Recognition Evaluation.

Contains the complete set of English, Arabic and Chinese training data for the 2004 Automatic Content Extraction (ACE) technology evaluation.

Contains approximately 1,800 files of mixed genre text in English, Arabic, and Chinese annotated for entities, relations, and events.

Compilation of American English nicknames to given name mappings based on information in US government records, public web profiles and financial and property reports. Note the User Agreement below:

Adds automatically-generated syntactic and discourse structure annotation to English Gigaword (5th ed.) and also contains an API and tools for reading the datasets XML files.

Flight information for 46 cities and 52 airports. Includes over 774 scenarios completed by 137 subjects, yielding a total of over 7,300 utterances. All utterances are transcribed and 2,900 of them have been categorized and annotated with canonical reference answers.

The corpus includes speech from seven (four male, three female) FM radio news announcers associated with WBUR, a public radio station. The main radio news portion of the corpus consists of over seven hours of news stories recorded in the WBUR radio studio during broadcasts over a two year period.

Originally recorded in 1978-79 as part of a British Home Office study into speaker identification techniques, it was subsequently realized that a large body of unconstrained conversational material might be of interest to researchers working in other speech processing fields.

Consists primarily of three Arabic-English lexicon files: prefixes (299 entries), suffixes (618 entries), and stems (82,158 entries representing 38,600 lemmas).

Consists of 120 unscripted 30-minute telephone conversations between native speakers of English.

Time-aligned transcripts and documentation files for 120 unscripted telephone conversations between native speakers of English.

Contains contiguous five or ten minute segments taken from 120 unscripted telephone conversations between native speakers of Egyptian Colloquial Arabic (ECA), the spoken variety of Arabic found in Egypt.

Contains ASCII versions of the CELEX lexical databases of English (Version 2.5), Dutch (Version 3.1) and German (Version 2.0).

Contains approximately 342 hours of English speech and transcripts from noisy environments and 50 hours of noisy environment audio.

2005 ed. of the Chinese Treebank.

Approximately two million words of annotated and parsed text from Chinese newswire, government documents, magazine articles, various broadcast news and broadcast conversation programs, web newsgroups, weblogs, discussion forums, chat messages and transcribed conversational telephone speech.

Contains a moderately broad coverage English lexicon (with about 38,000 lemmas).

Adds multiple kinds and instances of automatically-generated syntactic, semantic and coreference annotations to The New York Times Annotated Corpus

The corpus contains more than 450 million words of text and is equally divided among spoken, fiction, popular magazines, newspapers, and academic texts. It includes 20 million words each year from 1990-2012 and the corpus is also updated regularly (the most recent texts are from Summer 2012). Because of its design, it is perhaps the only corpus of English that is suitable for looking at current, ongoing changes in the language

Contains more than 400 million words of text from the 1810s-2000s, organized by genre and decade.

Consists of continuous speech in English by native speakers of 22 different languages. The corpus contains 4925 telephone-quality utterances, information about the speakers' linguistic backgrounds and perceptual judgments about the accents in the utterances.

Contains sense-tagged word occurrences for 121 nouns and 70 verbs which are among the most frequently occurring and ambiguous words in English.

The first release of the European Corpus Initiative, the Multilingual Corpus 1 (ECI/MCI), has 46 subcorpora in 27 (mainly European) languages. The total size of these is roughly 92 million (lexical) words.

Contains audio recordings and corresponding transcripts, collected over an eight month period in 2000-2001 and designed to support research in emotional prosody. The recordings consist of professional actors reading a series of semantically neutral utterances (dates and numbers) spanning fourteen distinct emotional categories.

A comprehensive archive of newswire text data that has been acquired over several years by the LDC at the University of Pennsylvania. The fifth edition includes all of the contents in English Gigaword Fourth Edition (LDC2009T13) plus new data covering the 24-month period of January 2009 through December 2010.

Provides access to ten corpora created by Mark Davies, retired professor of linguistics at Brigham Young University.

Consists of a combination of automated and manual revisions of the Penn Treebank annotation of Wall Street Journal (WSJ) stories.

Comprised of 12,100 English essays written by speakers of 11 non-English native languages as part of an international test of academic English proficiency, TOEFL (Test of English as a Foreign Language).

Consists of approximately 3,000 English and 1,500 Spanish news articles about civil unrest events annotated with temporal tags.

Composed of 1.9 billion words from 1.8 million web pages in 20 different English-speaking countries.

Parallel texts in English and Canadian French, drawn from official records of the proceedings of the Canadian Parliament (mid-1970's through 1988).

Obtained during January 1999 from the bilingual website of the Department of Justice of the Hong Kong Special Administrative Region (HKSAR) of the Peoples Republic of China.

Re-recording of a subset of the TIMIT corpus through different telephone handsets.

Contains approximately 215 hours of Cantonese conversational and scripted telephone speech collected in 2011 along with corresponding transcripts.

Digital archive of the most influential sources of information of Russia as well as a range of analytical services for mass media and social networks monitoring.

Contains 7,849 Mandarin Chinese "utterances" and their phonetic segmentation and tone labels separated into training and test sets.

Comprised of approximately 192 hours of Mandarin-English code-switching speech from 156 speakers with associated transcripts.

The NLTK has built-in support for dozens of corpora and trained models.

Contains over 1.8 million articles written and published by the New York Times between January 1, 1987 and June 19, 2007.

Transcripts for the 1997 TREC Spoken Document Retrieval Track. Consists of the 50 hours (87 transcripts) of 1996 Broadcast News radio and television recordings.

Telephone conversations and broadcast news in the following languages: Mesopotamian Arabic, South Levantine Arabic, North Levantine Arabic, Moroccan Arabic, Bengali, Czech, Dari, English, Persian, Hindi, Lao, Mandarin Chinese, Panjabi, Pushto, Polish, Russian, Slovak, Spanish, Tamil, Thai, Turkish, Ukrainian, Urdu, Standard Arabic

Collection of English news text from the Los Angeles Times, Washington Post, New York Times, Reuters and the Wall Street Journal.

6.64 billion words of data from web-based newspapers and magazines from 2010 to the present time. More importantly, the corpus grows by about 100-120 million words of data each month (from 200,000 - 250,000 new articles), or about 1.2 billion words each year.

Large corpus comprising various genres of text (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, talk shows) in three languages (English, Chinese, and Arabic) with structural information (syntax and predicate argument structure) and shallow semantics (word sense linked to an ontology and coreference).

Annotates the Wall Street Journal (WSJ) section of Treebank-2 (LDC95T7) with discourse relations.

Consists of coreference, bridging and event-event relations (temporal, causal, subevent and reporting relations) annotations over 95 English newswire, discussion forum and narrative text documents.

Comprises eight sociolinguistic interviews with a total of 9 speakers, conducted in the 1960s and 70s.

Collection of about 2,400 two-sided telephone conversations among 543 speakers (302 male, 241 female) from all areas of the United States.

Transcripts of approximately 24 hours of English telephone conversations collected by LDC between 1999-2000.

Contains training and evaluation data produced in support of the TAC KBP Entity Discovery and Linking (EDL) tasks in 2016 and 2017.

Large-scale relation extraction dataset with 106,264 examples built over English newswire and web text used in the NIST TAC KBP English slot filling evaluations during the period 2009-2014.

Contains 183 news articles that have been annotated with temporal information, adding events, times and temporal links between events and times.

Version of the TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) has all the waveform files formatted with ms-wav / RIFF headers, to make the corpus more accessible to a wider audience.

Contains the complete TIMIT acoustic-phonetic speech corpus, a total of 6300 sentences, 10 sentences spoken by each of 630 speakers from 8 major dialect regions of the United States.

Documents in the test collection are varied in style, size and subject domain. The first disk contains material from the Wall Street Journal, (1986, 1987, 1988, 1989), the AP Newswire (1989), the Federal Register (1989), information from Computer Select disks (Ziff-Davis Publishing) and short abstracts from the Department of Energy. The second disk contains information from the same sources, but from different years. The third disk contains more information from the Computer Select disks, plus material from the San Jose Mercury News (1991), more AP newswire (1990) and about 250 megabytes of formatted U.S. Patents.

Contains over 1.6 million words of hand-parsed material from the Dow Jones News Service, plus an additional one million words tagged for part-of-speech. This material is a subset of the language model corpus for the DARPA CSR large-vocabulary speech recognition project.

Consists of 1 million words of 1989 Wall Street Journal material annotated in Treebank II style, a small sample of ATIS-3 material annotated in Treebank II style, and a fully tagged version of the Brown corpus.

Comprised of approximately 34 hours of English speech and orthographic transcripts. This corpus was designed to sample variability in speaking within individual speakers and across a large number of speakers.

Continuation of the above.

The documents come from the Office of Conference Services at the UN in New York and are drawn from archives that span the period between 1988 and 1993.

Contains English word n-grams and their observed frequency counts.

Contains what is probably the most accurate word frequency data for English. The data is based on the one billion word Corpus of Contemporary American English (COCA). Data is for personal research purposes only and cannot be shared with others via any medium nor used for commercial purposes.

A British English Speech Corpus for Large Vocabulary Continuous Speech Recognition (The Cambridge University Version of the ARPA CSR Corpus WSJ0).

Database contains a large scale, high-quality speech corpus to support text-dependent speaker authentication research, such as is used in secure access technology.