PDF to Speech using FastSpeech 2

In this article, we will explore how to convert PDF documents into speech using FastSpeech 2, a powerful text-to-speech (TTS) model. FastSpeech 2 leverages advancements in deep learning and natural language processing (NLP) to generate high-quality and expressive speech from written text. Whether you want to listen to your favorite book, convert educational material into audio format, or create an audiobook, FastSpeech 2 can be a valuable tool in your arsenal.

The process involves extracting the text from a PDF document and then converting it into an audio file. We will walk through the installation of the required libraries, demonstrate how to extract text from a PDF, convert the text into speech using FastSpeech 2, and export the generated audio file. Let's get started!

Installation

Before we begin, we need to install the necessary libraries. Open your terminal or command prompt and run the following commands:

pip install pdfplumber PyPDF2

This command installs pdfplumber and PyPDF2, which are libraries used for extracting text from PDF documents.

Next, we need to install additional libraries required for text-to-speech conversion using FastSpeech 2:

pip install fairseq transformers g2p_en torchvision

These libraries include fairseq, transformers, g2p_en, and torchvision, which are essential for loading and utilizing the FastSpeech 2 model and performing text-to-speech conversion.

Please note that if you are using Google Colab, you will need to add an exclamation mark in front of these commands and restart the runtime after running the above commands to ensure that the libraries are correctly imported.

Extracting text from pdf

First, we need to extract the text from the PDF document. Follow the steps below to accomplish this:

Import Libraries

We start by importing the necessary libraries for extracting text from PDFs.

import pdfplumber
import PyPDF2

Setup File Object

Next, we set up the file object for the PDF file we want to extract text from. Replace "/content/Atomic Habits.pdf" with the path to your PDF file.

file = "/content/Atomic Habits.pdf" # replace with path to your pdf file

pdfFileObj = open(file, 'rb')
pdfReader = PyPDF2().PdfReader(pdfFileObj)

Extract text from pdf object

We can now extract the text from the PDF object using the pdfplumber library.

text = "" # define empty text string

with pdfplumber.open(file) as pdf:
  page = pdf.pages[7] # page index to extract text from
  text += page.extract_text() # add the extracted text to the text string

In this code snippet, we open the PDF file using pdfplumber and access a specific page (in this case, page 7). We extract the text from that page and append it to the text string variable.

Converting text to audio

Now that we have extracted the text from the PDF, we can proceed with converting it to audio using FastSpeech 2. We will walk through the process of installing the required libraries, importing the pretrained model, and converting the text to speech.

Import Libraries

Let's start by importing the required libraries.

from fairseq.checkpoint_utils import load_model_ensemble_and_task_from_hf_hub
from fairseq.models.text_to_speech.hub_interface import TTSHubInterface
import torch
import IPython.display as ipd

Import Pretrained model

Next, we import the pretrained FastSpeech 2 model from the Hugging Face model hub. We also override the default arguments to use the HiFi-GAN vocoder and disable fp16.

models, cfg, task = load_model_ensemble_and_task_from_hf_hub(
    "facebook/fastspeech2-en-ljspeech",
    arg_overrides={"vocoder": "hifigan", "fp16": False}
)

model = models[0]

In this code block, we load the FastSpeech 2 model and its configuration from the Hugging Face model hub. We set the model name to "facebook/fastspeech2-en-ljspeech" and other parameters like the vocoder and fp16 mode. We assign the loaded model to the model variable.

Text to Speech Conversion

We will now convert the extracted text to speech using the FastSpeech 2 model. We start by updating the model configuration with the data configuration. Next, we build the generator using the updated configuration and the loaded model. We then get the model input using the text we extracted from the PDF. Finally, we convert the text to speech using the model, generator, and model input.

TTSHubInterface.update_cfg_with_data_cfg(cfg, task.data_cfg)
generator = task.build_generator(models, cfg)

These lines of code above update the FastSpeech 2 configuration with the appropriate data configuration and build the generator for speech synthesis.

sample = TTSHubInterface.get_model_input(task, text)

We then generate the model input sample using the get_model_input function, passing in the extracted text.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
sample['net_input']['src_tokens'] = sample['net_input']['src_tokens'].to(device)

for key, value in sample.items():
    if isinstance(value, torch.Tensor):
        sample[key] = value.to(device)

model.to(device)

'The above code block ensures that the model and input tensors are moved to the appropriate device (GPU if available, otherwise CPU) for faster computation. This is an optional step, but it is recommended if you have a GPU available. If you are using Google Colab, you can enable GPU acceleration by going to Runtime > Change Runtime Type and selecting GPU as the hardware accelerator. You can then run the above code block to move the model and input tensors to the GPU.

wav, rate = TTSHubInterface.get_prediction(task, model, generator, sample)

We obtain the waveform and sample rate of the generated audio using the get_prediction function. We pass in the task, model, generator, and model input sample as arguments. The generated audio is stored in the wav variable, while the sample rate is stored in the rate variable. We can now play the audio using the IPython display library. We first convert the wav tensor to a numpy array and then pass it to the Audio function. We also pass in the sample rate as an argument.

wav_cpu = wav.cpu()
wav_np = wav_cpu.numpy()
ipd.Audio(wav_np, rate=rate)

Export Audio

Finally, if you want to save the audio as a file, you can use the following code:

import scipy.io.wavfile as wavfile

wavfile.write("/content/trial1.wav", rate, wav_np)

This code snippet saves the generated audio as a WAV file named "trial1.wav" in the specified path ("/content/"). Replace the path and file name with your desired location and name.

Now you can enjoy the synthesized speech from the PDF document or share it with others in audio format!

That concludes our tutorial on converting PDF to speech using FastSpeech 2. Feel free to explore further customization options and experiment with different PDFs to generate high-quality audio content.