View on GitHub

Unet-TTS

One-shot TTS with Improved Unseen Speaker and Style Transfer

Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning

Email: rayn.li@cloudminds.com

Our proposed algorithm has powerful speaker and style transfer capabilities, especially excellent imitation of out-of-domain emotions.

Code

Colab notebook

Paper link

Abstract

One-shot voice cloning aims to transform speaker voice and speaking style in speech synthesized from a text-to-speech (TTS) system, where only a shot recording from the target speech can be used. Out-of-domain transfer is still a challenging task, and one important aspect that impacts the accuracy and similarity of synthetic speech is the conditional representations carrying speaker or style cues extracted from the limited references. In this paper, we present a novel one-shot voice cloning algorithm called Unet-TTS that has good generalization ability for unseen speakers and styles. Based on a skip-connected U-net structure, the new model can efficiently discover speaker-level and utterance-level spectral feature details from the reference audio, enabling accurate inference of complex acoustic characteristics as well as imitation of speaking styles into the synthetic speech. According to both subjective and objective evaluations of similarity, the new model outperforms both speaker embedding and unsupervised style modeling (GST) approaches on an unseen emotional corpus.

Demo (One-shot Unseen Emotion Transfer)

Model Description:
Unet-TTS - Our proposed model
GST - Tacotron with unsupervised style modeling of GST
SpkEmbed - Tacotron with speaker embedding

These reference emotion speech to be transferred are unseen in the training process.

1. Same Text as Reference

Neutral

Unet-TTS GST SpkEmbed
Unet-TTS GST SpkEmbed

Angry

Unet-TTS GST SpkEmbed
Unet-TTS GST SpkEmbed

Surprise

Unet-TTS GST SpkEmbed
Unet-TTS GST SpkEmbed

Happy

Unet-TTS GST SpkEmbed
Unet-TTS GST SpkEmbed

Sad

Unet-TTS GST SpkEmbed
Unet-TTS GST SpkEmbed

2. Arbitrary Text (AT)

(The text of the reference is the same as above)

Neutral

Reference Unet-TTS GST

Angry

Reference Unet-TTS GST

Surprise

Reference Unet-TTS GST

Happy

Reference Unet-TTS GST

Sad

Reference Unet-TTS GST

3. Other

Description Reference Unet-TTS
Pause style
Stress style
PC Recording
Phone recording