Amazon Polly, Rekognition, and AI Services

Amazon Polly Overview

Amazon Polly is a cloud service that converts text into lifelike speech. It uses advanced deep learning technologies to synthesize speech that sounds natural. Polly supports multiple languages and voices, including both standard (concatenative) and Neural (NTTS) voices. Neural voices produce higher quality, more natural speech and are recommended for conversational applications.

How Polly Works

You send text input to Polly via the AWS SDK, CLI, or Management Console. Polly returns an audio stream of the spoken text. You can specify output format (MP3, OGG Vorbis, PCM, etc.), sample rate, and voice. Optionally, you can use Speech Synthesis Markup Language (SSML) to control pronunciation, pitch, speed, and volume. Polly also supports lexicons (custom dictionaries) to specify how certain words are pronounced, e.g., acronyms or foreign names.

Key Features and Limits

Amazon Rekognition Overview

Amazon Rekognition is a service for image and video analysis. It can detect objects, scenes, faces, text, and activities. It also provides face comparison and face search capabilities. Rekognition uses pre-trained deep learning models that require no ML expertise.

How Rekognition Works

You provide an image or video to Rekognition via API operations. For images, you can pass the image bytes directly (max 5 MB) or reference an S3 object. For videos, you must store the video in S3 and use the StartLabelDetection, StartFaceDetection, etc., APIs to start asynchronous analysis. Rekognition returns results in JSON format.

Key Features and Limits

- Image Analysis: - DetectLabels: Returns labels (objects, scenes, concepts) with confidence scores. Max 10 MB image size (5 MB if base64). - DetectFaces: Detects faces and returns attributes like age range, gender, emotions, landmarks, pose. Up to 100 faces per image. - CompareFaces: Compares a source face with faces in a target image. Returns similarity score. - DetectText: Detects text in images (e.g., street signs, documents). Returns bounding boxes and text content. - DetectModerationLabels: Detects adult/offensive content. Returns confidence scores for categories like Explicit Nudity, Violence, etc. - RecognizeCelebrities: Detects celebrities in images. - Video Analysis: - Asynchronous operations: StartLabelDetection, StartFaceDetection, StartContentModeration, StartPersonTracking, StartCelebrityRecognition, StartTextDetection. - You receive results via SNS topic or by calling GetXxx APIs. The job status can be checked with GetXxx. - Video input must be in S3 bucket (same region as Rekognition). Supported formats: MP4, MOV, AVI, etc. Max video length: 30 minutes for most operations. - Face Search: - Requires creating a collection (max 20 million faces per collection). You index faces using IndexFaces. Then search with SearchFaces (by face ID) or SearchFacesByImage (by image). - Collection ID must be unique per account per region. - Limits: - For DetectLabels and DetectModerationLabels, the image must be in JPEG or PNG format. - For DetectText, the image can be JPEG or PNG, max 5 MB. - Video analysis jobs have a maximum duration of 30 minutes. If the video is longer, you must split it. - Throttling: 5 requests per second (RPS) for most image APIs, 2 RPS for video APIs (varies by region).

Other AI Services

Integration Patterns

These AI services integrate via SDK, CLI, and through AWS services like Lambda, Step Functions, and S3. Common patterns:

Security and Encryption

Scenario 1: E-Learning Platform with Text-to-Speech

An e-learning startup wants to provide audio versions of course materials. They use Amazon Polly to convert lesson text into MP3 files. They store the generated audio in S3 and serve it via CloudFront. They use SSML to adjust pronunciation of technical terms and control pacing. They cache audio files to avoid re-synthesizing the same text. Challenges: handling long texts (use async synthesis), managing costs (neural voices are more expensive), and ensuring low latency for real-time previews. They set up a Lambda function triggered by S3 uploads to automatically generate audio for new lessons. They monitor CloudWatch metrics for errors and throttling.

Scenario 2: Social Media Moderation with Rekognition

A social media company needs to automatically moderate user-uploaded images. They use S3 event notifications to trigger a Lambda function that calls Rekognition's DetectModerationLabels on each image. If the confidence score for explicit content exceeds a threshold (e.g., 70%), the image is flagged for manual review or automatically rejected. They also use DetectText to catch offensive text in images. They batch process videos using StartContentModeration. Performance considerations: Rekognition has rate limits (5 RPS for image APIs); they implement a queue (SQS) to decouple uploads from processing. They also use Rekognition's face search to identify banned users from a collection.

Scenario 3: Multilingual Customer Support with Transcribe and Translate

A global e-commerce company wants to transcribe and translate customer support calls in real-time. They use Amazon Transcribe's streaming API to transcribe audio from the call center. The transcription is streamed to a Lambda function that calls Amazon Translate to translate the text into the agent's preferred language. The translated text is displayed on the agent's screen. For post-call analytics, they store transcriptions in S3 and use Comprehend to extract sentiment and key phrases. Challenges: handling multiple languages, ensuring low latency (streaming requires sub-second processing), and dealing with accents and background noise. They use custom language models with Transcribe for domain-specific terminology.