Feature Overview

 
This Feature card is for the work to support pairwise data and contrastive loss. This capability enables pairwise data and contrastive loss for preference tuning. By training a model with pairs of options where one is explicitly preferred over the other, we can use a contrastive loss function to guide the model in outputting higher scores for the preferred option while simultaneously lowering scores for the non-preferred option. This allows the model to learn to discriminate between preferred and non-preferred choices based on pairwise comparisons.

 
Goals

• The primary user type for this feature is data scientists and machine learning engineers who work with preference-tuning either using datasets for RLHF or providing a constitution for RLAIF.
• This feature expands upon existing supported dataset formats by adding support for pairwise data and contrastive loss.

Requirements

To consider this Feature complete, the following requirements must be met:

1. The system should be able to accept and process pairwise data.
2. The system should be able to calculate contrastive loss for each pair of options.
3. InstructLab should allow the model to be trained using the pairwise data and contrastive loss.
4. The model should accurately discriminate between preferred and non-preferred choices based on the pairwise comparisons.

Background

Pairwise data and contrastive loss are essential for preference tuning, as they allow models to learn from explicit preferences and make accurate predictions based on those preferences. However, currently, our system does not support these features, which limits its applicability in certain use cases.

Done

1. The system should be able to accept and process pairwise data.
2. The system should be able to calculate contrastive loss for each pair of options.
3. InstructLab should allow the model to be trained using the pairwise data and contrastive loss.
4. The model should be able to accurately discriminate between preferred and non-preferred choices based on the pairwise comparisons.

 Questions to Answer

1. How will the pairwise data be structured and formatted, maintaining backward compatibility with the current dataset format?
2. What metrics will be used to evaluate the model's performance in discriminating between preferred and non-preferred choices?

Out of Scope

1. Implementing other types of loss functions or data structures.
2. Optimizing the system for specific hardware or infrastructure.
3. Integrating with external APIs or services.

Customer Considerations

When designing and delivering this Feature, the following customer-specific considerations must be made:

1. The customer should be hinted (in RLAIF) or involved (RLHF) in the data preparation process to ensure that the pairwise data is accurate and relevant.
2. The customer should be provided with clear documentation on how to use the new functionality and interpret the results.
3. The customer should be notified of any changes to the system that may impact their existing workflows or processes.