A team is fine-tuning a large language model using reinforcement learning from human feedback (RLHF) in SageMaker. Which THREE components are essential for the RLHF pipeline? (Select THREE.)
Trap 1: Value network
Incorrect. While some RL algorithms use a value network, the standard RLHF pipeline for LLMs requires a reference model instead. A value network is not essential.
Trap 2: Feature store
Incorrect. A feature store is a repository for storing and managing features for machine learning, but it is not a core component of the RLHF pipeline.
Trap 3: Hyperparameter tuner
Incorrect. A hyperparameter tuner helps optimize training parameters but is not essential to the RLHF process itself.
- A
Policy network (the LLM being fine-tuned)
Correct. The policy network is the LLM being fine-tuned, which learns to generate responses that maximize rewards.
- B
Value network
Why wrong: Incorrect. While some RL algorithms use a value network, the standard RLHF pipeline for LLMs requires a reference model instead. A value network is not essential.
- C
Feature store
Why wrong: Incorrect. A feature store is a repository for storing and managing features for machine learning, but it is not a core component of the RLHF pipeline.
- D
Hyperparameter tuner
Why wrong: Incorrect. A hyperparameter tuner helps optimize training parameters but is not essential to the RLHF process itself.
- E
Reward model
Correct. The reward model provides the reward signal based on human feedback, guiding the policy network's updates.