Skip to content

Rapidata leaderboard

RapidataLeaderboard #

RapidataLeaderboard(
    name: str,
    instruction: str,
    show_prompt: bool,
    id: str,
    openapi_service: OpenAPIService,
)

An instance of a Rapidata leaderboard.

Used to interact with a specific leaderboard in the Rapidata system, such as retrieving prompts and evaluating models.

Parameters:

Name Type Description Default
name str

The name that will be used to identify the leaderboard on the overview.

 required
instruction str

The instruction that will determine what how the models will be evaluated.

 required
show_prompt bool

Whether to show the prompt to the users.

 required
id str

The ID of the leaderboard.

 required
openapi_service OpenAPIService

The OpenAPIService instance for API interaction.

 required
Source code in src/rapidata/rapidata_client/leaderboard/rapidata_leaderboard.py 
24
25
26
27
28
29
30
def __init__(self, name: str, instruction: str, show_prompt: bool, id: str, openapi_service: OpenAPIService):
    self.__openapi_service = openapi_service
    self.name = name
    self.instruction = instruction
    self.show_prompt = show_prompt
    self._prompts: list[str] = []
    self.id = id

prompts property #

prompts: list[str]

Returns the prompts that are registered for the leaderboard.

evaluate_model #

evaluate_model(
    name: str, media: list[str], prompts: list[str]
) -> None

Evaluates a model on the leaderboard.

Parameters:

Name Type Description Default
name str

The name of the model.

 required
media list[str]

The generated images/videos that will be used to evaluate the model.

 required
prompts list[str]

The prompts that correspond to the media. The order of the prompts must match the order of the media. The prompts that are used must be registered for the leaderboard. To see the registered prompts, use the prompts property.

 required
Source code in src/rapidata/rapidata_client/leaderboard/rapidata_leaderboard.py 
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def evaluate_model(self, name: str, media: list[str], prompts: list[str]) -> None:
    """
    Evaluates a model on the leaderboard.

    Args:
        name: The name of the model.
        media: The generated images/videos that will be used to evaluate the model.
        prompts: The prompts that correspond to the media. The order of the prompts must match the order of the media.
            The prompts that are used must be registered for the leaderboard. To see the registered prompts, use the prompts property.
    """
    if not media:
        raise ValueError("Media must be a non-empty list of strings")

    if len(media) != len(prompts):
        raise ValueError("Media and prompts must have the same length")

    if not all(prompt in self.prompts for prompt in prompts):
        raise ValueError("All prompts must be in the registered prompts list. To see the registered prompts, use the prompts property.")

    # happens before the creation of the participant to ensure all media paths are valid
    assets = []
    prompts_metadata: list[list[PromptMetadata]] = []
    for media_path, prompt in zip(media, prompts):
        assets.append(MediaAsset(media_path))
        prompts_metadata.append([PromptMetadata(prompt)])

    participant_result = self.__openapi_service.leaderboard_api.leaderboard_leaderboard_id_participants_post(
        leaderboard_id=self.id,
        create_leaderboard_participant_model=CreateLeaderboardParticipantModel(
            name=name,
        )
    )
    dataset = RapidataDataset(participant_result.dataset_id, self.__openapi_service)

    dataset._add_datapoints(assets, prompts_metadata)

    self.__openapi_service.leaderboard_api.leaderboard_leaderboard_id_participants_participant_id_submit_post(
        leaderboard_id=self.id,
        participant_id=participant_result.participant_id
    )