Validation set manager
ValidationSetManager #
Responsible for everything related to validation sets. From creation to retrieval.
Attributes:
Name | Type | Description |
---|---|---|
rapid |
RapidsManager
|
The RapidsManager instance. |
Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_classification_set #
create_classification_set(
name: str,
instruction: str,
answer_options: list[str],
datapoints: list[str],
truths: list[list[str]],
data_type: str = MEDIA,
contexts: list[str] | None = None,
media_contexts: list[str] | None = None,
explanations: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a classification validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction by which the labeler will answer. |
required |
answer_options
|
list[str]
|
The options to choose from when answering. |
required |
datapoints
|
list[str]
|
The datapoints that will be used for validation. |
required |
truths
|
list[list[str]]
|
The truths for each datapoint. Outer list is for each datapoint, inner list is for each truth. example: options: ["yes", "no", "maybe"] datapoints: ["datapoint1", "datapoint2"] truths: [["yes"], ["no", "maybe"]] -> first datapoint correct answer is "yes", second datapoint is "no" or "maybe" |
required |
data_type
|
str
|
The type of data. Defaults to RapidataDataTypes.MEDIA. Other option: RapidataDataTypes.TEXT ("text"). |
MEDIA
|
contexts
|
list[str]
|
The contexts for each datapoint. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction and answer options. (Therefore will be different for each datapoint) Will be match up with the datapoints using the list index. |
None
|
media_contexts
|
list[str]
|
The list of media contexts i.e. links to the images / videos for the comparison. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction. (Therefore will be different for each datapoint) Will be matched up with the datapoints using the list index. |
None
|
explanations
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first datapoint correct answer is "yes", second datapoint is "no" or "maybe"Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_compare_set #
create_compare_set(
name: str,
instruction: str,
datapoints: list[list[str]],
truths: list[str],
data_type: str = MEDIA,
contexts: list[str] | None = None,
media_contexts: list[str] | None = None,
explanation: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a comparison validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction to compare against. |
required |
truths
|
list[str]
|
The truths for each comparison. List is for each comparison. example: instruction: "Which image has a cat?" datapoints = [["image1.jpg", "image2.jpg"], ["image3.jpg", "image4.jpg"]] truths: ["image1.jpg", "image4.jpg"] -> first comparison image1.jpg has a cat, second comparison image4.jpg has a cat |
required |
datapoints
|
list[list[str]]
|
The compare datapoints to create the validation set with. Outer list is for each comparison, inner list the two images/texts that will be compared. |
required |
data_type
|
str
|
The type of data. Defaults to RapidataDataTypes.MEDIA. Other option: RapidataDataTypes.TEXT ("text"). |
MEDIA
|
contexts
|
list[str]
|
The contexts for each datapoint. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction and truth. (Therefore will be different for each datapoint) Will be match up with the datapoints using the list index. |
None
|
media_contexts
|
list[str]
|
The list of media contexts i.e. links to the images / videos for the comparison. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction. (Therefore will be different for each datapoint) Will be matched up with the datapoints using the list index. |
None
|
explanation
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first comparison image1.jpg has a cat, second comparison image4.jpg has a catSource code in src/rapidata/rapidata_client/validation/validation_set_manager.py
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 |
|
create_select_words_set #
create_select_words_set(
name: str,
instruction: str,
truths: list[list[int]],
datapoints: list[str],
sentences: list[str],
required_precision: float = 1.0,
required_completeness: float = 1.0,
explanation: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a select words validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction to show to the labeler. |
required |
truths
|
list[list[int]]
|
The truths for each datapoint. Outer list is for each datapoint, inner list is for each truth. example: datapoints: ["datapoint1", "datapoint2"] sentences: ["this example 1", "this example with another text"] truths: [[0, 1], [2]] -> first datapoint correct words are "this" and "example", second datapoint is "with" |
required |
datapoints
|
list[str]
|
The datapoints that will be used for validation. |
required |
sentences
|
list[str]
|
The sentences that will be used for validation. The sentece will be split up by spaces to be selected by the labeler. Must be the same length as datapoints. |
required |
required_precision
|
float
|
The required precision for the labeler to get the rapid correct (minimum ratio of the words selected that need to be correct). Defaults to 1.0 (no wrong word can be selected). |
1.0
|
required_completeness
|
float
|
The required completeness for the labeler to get the rapid correct (miminum ratio of total correct words selected). Defaults to 1.0 (all correct words need to be selected). |
1.0
|
explanation
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first datapoint the correct words are "this" and "example", second datapoint is "with"Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_locate_set #
create_locate_set(
name: str,
instruction: str,
truths: list[list[Box]],
datapoints: list[str],
contexts: list[str] | None = None,
media_contexts: list[str] | None = None,
explanation: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a locate validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction to show to the labeler. |
required |
truths
|
list[list[Box]]
|
The truths for each datapoint. Outer list is for each datapoint, inner list is for each truth. example: datapoints: ["datapoint1", "datapoint2"] truths: [[Box(0, 0, 100, 100)], [Box(50, 50, 150, 150)]] -> first datapoint the object is in the top left corner, second datapoint the object is in the center |
required |
datapoints
|
list[str]
|
The datapoints that will be used for validation. |
required |
contexts
|
list[str]
|
The contexts for each datapoint. Defaults to None. |
None
|
media_contexts
|
list[str]
|
The list of media contexts i.e. links to the images / videos for the comparison. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction. (Therefore will be different for each datapoint) Will be matched up with the datapoints using the list index. |
None
|
explanation
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first datapoint the object is in the top left corner, second datapoint the object is in the centerSource code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_draw_set #
create_draw_set(
name: str,
instruction: str,
truths: list[list[Box]],
datapoints: list[str],
contexts: list[str] | None = None,
media_contexts: list[str] | None = None,
explanation: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a draw validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction to show to the labeler. |
required |
truths
|
list[list[Box]]
|
The truths for each datapoint. Outer list is for each datapoint, inner list is for each truth. example: datapoints: ["datapoint1", "datapoint2"] truths: [[Box(0, 0, 100, 100)], [Box(50, 50, 150, 150)]] -> first datapoint the object is in the top left corner, second datapoint the object is in the center |
required |
datapoints
|
list[str]
|
The datapoints that will be used for validation. |
required |
contexts
|
list[str]
|
The contexts for each datapoint. Defaults to None. |
None
|
media_contexts
|
list[str]
|
The list of media contexts i.e. links to the images / videos for the comparison. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction. (Therefore will be different for each datapoint) Will be matched up with the datapoints using the list index. |
None
|
explanation
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first datapoint the object is in the top left corner, second datapoint the object is in the centerSource code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_timestamp_set #
create_timestamp_set(
name: str,
instruction: str,
truths: list[list[tuple[int, int]]],
datapoints: list[str],
contexts: list[str] | None = None,
media_contexts: list[str] | None = None,
explanation: list[str | None] | None = None,
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a timestamp validation set.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
instruction
|
str
|
The instruction to show to the labeler. |
required |
truths
|
list[list[tuple[int, int]]]
|
The truths for each datapoint defined as start and endpoint based on miliseconds. Outer list is for each datapoint, inner list is for each truth. example: datapoints: ["datapoint1", "datapoint2"] truths: [[(0, 10)], [(20, 30)]] -> first datapoint the correct interval is from 0 to 10, second datapoint the correct interval is from 20 to 30 |
required |
datapoints
|
list[str]
|
The datapoints that will be used for validation. |
required |
contexts
|
list[str]
|
The contexts for each datapoint. Defaults to None. |
None
|
media_contexts
|
list[str]
|
The list of media contexts i.e. links to the images / videos for the comparison. Defaults to None. If provided has to be the same length as datapoints and will be shown in addition to the instruction. (Therefore will be different for each datapoint) Will be matched up with the datapoints using the list index. |
None
|
explanation
|
list[str | None]
|
The explanations for each datapoint. Will be given to the annotators in case the answer is wrong. Defaults to None. |
None
|
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Example
This would mean: first datapoint the correct interval is from 0 to 10, second datapoint the correct interval is from 20 to 30Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
create_mixed_set #
create_mixed_set(
name: str,
rapids: list[Rapid],
dimensions: list[str] = [],
) -> RapidataValidationSet
Create a validation set with a list of rapids.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name of the validation set. (will not be shown to the labeler) |
required |
rapids
|
list[Rapid]
|
The list of rapids to add to the validation set. |
required |
dimensions
|
list[str]
|
The dimensions to add to the validation set accross which users will be tracked. Defaults to [] which is the default dimension. |
[]
|
Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
get_validation_set_by_id #
get_validation_set_by_id(
validation_set_id: str,
) -> RapidataValidationSet
Get a validation set by ID.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
validation_set_id
|
str
|
The ID of the validation set. |
required |
Returns:
Name | Type | Description |
---|---|---|
RapidataValidationSet |
RapidataValidationSet
|
The ValidationSet instance. |
Source code in src/rapidata/rapidata_client/validation/validation_set_manager.py
find_validation_sets #
find_validation_sets(
name: str = "", amount: int = 1
) -> list[RapidataValidationSet]
Find validation sets by name.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
The name to search for. Defaults to "" to match with any set. |
''
|
amount
|
int
|
The amount of validation sets to return. Defaults to 1. |
1
|
Returns:
Type | Description |
---|---|
list[RapidataValidationSet]
|
list[RapidataValidationSet]: The list of validation sets. |