Executive Summary
The University of Illinois has conducted collaborative research with the
Korea Advanced Institute of Science & Technology (KAIST) during the period 1/1/2022-12/31/2026.
This work is focused on fair & inclusive automatic speech recognition
for the automatic recognition and synthesis of speech in
under-resourced languages, for people with speech disabilities that
affect intelligibility, and for code-switched conversations in which
people switch between two or more languages in order to communicated
desired meanings.
Background
Automatic speech recognition (ASR) requires acoustic models: heavily
parameterized computable functions (e.g., deep neural networks) that
map from acoustic signals to sequences of text characters, phones,
words, or sub-word units. In 2025, traditional stand-alone
neural-network based automatic speech recognizers were surpassed in
accuracy by multimodal large language models (MLLMs), which use
adapter modules to convert audio frames into a representation usable
by the MLLM. It is not yet clear, however, whether the stand-alone
ASR is the best adapter for an MLLM, or whether some other
architecture is superior. All of these architectures require large
quantities of data to train: An ASR, in particular, is traditionally
trained using labeled speech corpora, in which the correct text
transcription of each speech waveform is known. The requirement of
labeled training data limits the generalizability of ASR to new
languages, and to low-data scenarios such as the recognition of
dysarthric speech, children’s speech, and code-switched speech.
Dysarthria is a disorder in which speech intelligibility is impaired
by atypical articulation, phonation, respiration, prosody or a
combination thereof. The ability to communicate effectively, such as
via speech is essential to safety, independence, social engagement,
and quality of life. Unfortunately, this communication method is lost
to persons with dysarthria because family, caregivers, peers and
assistive devices, such as smart speakers or voice dialing often fail
to comprehend. Automatic speech recognition (ASR) and human listeners
may be trained to understand dysarthric speech, but training takes
time and practice for both: a human listener or an ASR require hours
of exposure to a PwD before learning their speech patterns. Novel
speech technologies have the potential to greatly improve the speed
and effectiveness of ASR adaptation to people with dysarthria,
and have the potential for consequently large impact.
Children’s speech is difficult to recognize because it differs from
adult speech acoustically, and because the text content is quite
different from that of adult speech. Very young children tend to
speak in very short sentences, and their sentences may contain a
number of non-words or grammatical errors that fool ASR.
Code-switching is a natural mode of communication in which two or more
conversants, each of whom is fluent in the same pair of languages,
switch between languages in order to communicate a desired meaning or
nuance. Multilingual ASR fails to correctly transcribe code-switched
conversation because it has been trained using data that is primarily
monolingual, hence it assumes that all words in a given sentence will
be drawn from the same language.
Unsupervised ASR (ASR-U) is a two-step pre-training process that can
create ASR in a language for which there are text corpora and speech
corpora, but there is no transcribed speech corpus. (1) First, an
acoustic encoder is trained using unlabeled audio data, with an
unsupervised training criterion such as masked hidden-unit language
modeling. (2) Second, a neural translator learns to translate the
encoded speech vectors into characters whose sequences resemble text
data in the same language. Since it was first proposed in 2017, ASR-U
has focused on the development of automatic phone recognition in an
unknown language, and has never yet, prior to the work we propose
here, been extended to the study of automatic word recognition in an
under-resourced language.
Proposed Work
Research at the University of Illinois is focused on few-shot
ASR using in-context learning, dysarthric speech recognition, and
unsupervised speech recognition.
Speech in-context learning
Speech foundation models have recently demonstrated the ability to
perform Speech In-Context Learning (SICL). Selecting effective
in-context examples is crucial for SICL performance, yet selection
methodologies remain under-explored. In this work, we propose
Text-Embedding KNN for SICL (TICL), a simple pipeline that uses
semantic context to enhance off-the-shelf large multimodal models’
speech recognition abilities without fine-tuning. Across challenging
automatic speech recognition tasks, including accented English,
multilingual speech, and children’s speech, our method will enable
models to surpass the zero-shot performance of current ASR.
Dysarthric Speech Recognition
While the last decade has witnessed significant advancements in
Automatic Speech Recognition (ASR) systems, performance of these
systems for individuals with speech disabilities remains inadequate,
partly due to limited public training data. To bridge this gap, we
will run the Second Annual Speech Accessibility Project Challenge
(SAPC2), utilizing over 1400 hours of SAP data collected and
transcribed from more than 1000 individuals with diverse speech
disabilities. Hosted on CodeAI and leveraging the remote evaluation
pipeline, the SAP Challenge will evaluate submissions based on
Character Error Rate of systems with unlimited computational
complexity (track 1) and based on a tradeoff between Character Error
Rate and Transcription Latency (track 2). In our own research, we
will continue to study methods of data augmentation and voice
conversion for improved dysarthric ASR.
Unsupervised speech recognition
n our 2023 paper “A Theory of Unsupervised Speech Recognition”
(Wang, Hasegawa-Johnson and Yoo, ACL 2023), we demonstrated that it is
possible to train an ASR with zero recognition error rate from
untranscribed speech data. Our theory has advanced little since 2023
because the cryptographic methods proposed in that paper fail if the
Bayes error rate of the ASR is too large: Bayes error rates larger
than only a few percentage points can disrupt the methods we proposed.
Research in 2026 will explore search-based methods for learning an
unsupervised ASR, first proposed by Hermann Ney and his student in 2016. Professor Ney and his student demonstrated that their method
also learns an ASR with zero error rate if the problem has a Bayes
error rate of zero, and they furthermore proved that the computational
complexity can be minimized using A* search, though they sacrificed
theoretical optimality for a method that gave dramatically better
computational performance. We propose to explore non-zero-Bayes-error
extensions of their theory. In the case of non-zero Bayes error rate,
their theory becomes equivalent to finding the minimum in a
cross-entropy space. Their theory therefore guarantees that the
optimal ASR can be computed without labeled speech data, but warns
that a theoretically guaranteed solution may be NP-complete. We will
explore exact solutions, and if those fail, approximate solutions to
the minimization problem proposed by Ney.
I