Speech to Text and Custom Speech

Azure Speech-to-Text (STT) is a cloud-based API that converts spoken audio into text. It is part of the Azure Speech service, which also includes Text-to-Speech, Speech Translation, and Speaker Recognition. The STT API uses deep neural networks trained on massive datasets of diverse speech to recognize words in real-time or batch. It supports:

The recognition process involves several stages:

Azure provides a prebuilt base model that works well for general-purpose transcription (e.g., meeting transcription, voice commands). However, it may struggle with:

Custom Speech allows you to train a custom model by providing additional data:

The training process follows these steps:

Using the Azure CLI or REST API, you can manage Custom Speech models. Example CLI commands:

# Create a Speech resource
az cognitiveservices account create --name mySpeech --resource-group myRG --kind SpeechServices --sku S0 --location eastus

# List base models (via REST API)
curl -X GET "https://<region>.cognitiveservices.azure.com/speechtotext/v3.1/models/base" -H "Ocp-Apim-Subscription-Key: <key>"

# Start a training job
curl -X POST "https://<region>.cognitiveservices.azure.com/speechtotext/v3.1/models" -H "Ocp-Apim-Subscription-Key: <key>" -H "Content-Type: application/json" -d '{
  "baseModel": {"self": "https://<region>.cognitiveservices.azure.com/speechtotext/v3.1/models/base/<base-model-id>"},
  "datasets": [{"self": "https://<region>.cognitiveservices.azure.com/speechtotext/v3.1/datasets/<dataset-id>"}],
  "properties": {"purpose": "LanguageModel"}
}'

Common errors: - 401 Unauthorized: Invalid or expired subscription key. - 400 Bad Request: Audio format not supported or duration too long (max 60s per request for real-time). - 429 Rate limit exceeded: Too many requests; implement exponential backoff.

A hospital wants to transcribe doctor-patient conversations in real time to update electronic health records (EHRs). The default STT model often fails on medical terminology like 'pneumothorax' or 'echocardiogram'. The hospital uses Custom Speech by uploading 20 hours of de-identified audio from previous consultations with transcripts. They also upload a text corpus of medical textbooks and a pronunciation file for drug names. After training, the WER drops from 25% to 8%. The custom model is deployed to an endpoint integrated with the EHR system. Common misconfiguration: using audio from a different department (e.g., radiology vs. cardiology) can hurt accuracy, so they create separate models per specialty.

A large insurance company wants to analyze customer calls to detect sentiment and compliance issues. They use batch transcription to process thousands of hours of recorded calls. The default model misinterprets industry terms like 'deductible' and 'co-pay' and struggles with various regional accents. They create a Custom Speech model using 50 hours of call audio with transcripts and a text corpus of insurance policy documents. The model is trained once and used for all batch jobs. A key consideration: audio quality varies (cell phones, speakerphones); they use noise adaptation data to improve robustness. When misconfigured (e.g., using only clean studio audio), the model fails on real noisy calls.

A manufacturing company adds voice commands to a robotic arm for hands-free operation in a noisy factory. The default STT model has high WER due to background noise (90 dB machinery) and specialized commands like 'grip 45 degrees'. They record 10 hours of commands in the actual factory environment with the same microphone that will be used in production. They also upload a text file of all possible commands. The custom model achieves 95% accuracy. A common mistake: training on clean audio and expecting it to work in noise – the model must be trained on representative noise conditions.