Example Classify Order

To learn about the basics of creating an order, please refer to the quickstart guide.

BasicAdvanced

With this order, we want to rate different images based on a Likert scale to better understand how well the model generated the images we intended. We have various images that we want to evaluate, and we will assess how accurately they represent the desired concepts. When you run this with your own examples, you may use local paths to your images instead of the URLs.

The NoShuffle setting is used to ensure that the answer options remain in a fixed order, because the answer options are ordered.

from rapidata import RapidataClient, NoShuffle

# List of image URLs representing different emotions
IMAGE_URLS: list[str] = [
    "https://assets.rapidata.ai/tshirt-4o.png",   # Related T-Shirt with the text "Running on caffeine & dreams"
    "https://assets.rapidata.ai/tshirt-aurora.jpg",   # Related T-shirt with the text "Running on caffeine & dreams"
    "https://assets.rapidata.ai/teamleader-aurora.jpg",   # Unrelated image
]

CONTEXTS: list[str] = [
    "A t-shirt with the text 'Running on caffeine & dreams'"
] * len(IMAGE_URLS) # Each image will have the same context to be rated by

if __name__ == "__main__":
    rapi = RapidataClient()

    # Create a classification order for emotions based on the images
    order = rapi.order.create_classification_order(
        name="Likert Scale Example",  
        instruction="How well does the image match the description?",
        answer_options=["1: Not at all", 
                        "2: A little", 
                        "3: Moderately", 
                        "4: Very well", 
                        "5: Perfectly"], 
        contexts=CONTEXTS,
        datapoints=IMAGE_URLS,
        responses_per_datapoint=25,
        settings=[NoShuffle()]  # Do not shuffle the answer options
    ).run()  # Execute the order

    # Display a progress bar for the order
    order.display_progress_bar()

    # Retrieve and print the results of the classification
    results = order.get_results()
    print(results)

To preview the order and see what the annotators see, you can run the following code:

order.preview()

To open the order in the browser, you can run the following code:

order.view()

In the advanced example we will first create a validation set to give the annotators a reference how they should rate the images.

To get a better understanding of validation sets, please refer to the Improve Quality guide.

from rapidata import RapidataClient, NoShuffle

# ===== TASK CONFIGURATION =====
# Likert scale options (from lowest to highest agreement)
ANSWER_OPTIONS: list[str] = [
    "1: Not at all",
    "2: A little",
    "3: Moderately", 
    "4: Very well",
    "5: Perfectly"
]

INSTRUCTION: str = "How well does the image match the description?"

# ===== VALIDATION DATA =====
# This validation set helps ensure quality responses by providing known examples
VALIDATION_IMAGE_URLS: list[str] = [
    "https://assets.rapidata.ai/email-4o.png",
    "https://assets.rapidata.ai/email-aurora.jpg",
    "https://assets.rapidata.ai/teacher-aurora.jpg",
]

VALIDATION_CONTEXT: str = "A laptop screen with clearly readable text, addressed to the marketing team about an upcoming meeting"

VALIDATION_CONTEXTS: list[str] = [VALIDATION_CONTEXT] * len(VALIDATION_IMAGE_URLS)

# Expected correct answers for each validation image (multiple acceptable answers possible)
VALIDATION_TRUTHS: list[list[str]] = [
    ["5: Perfectly", "4: Very well"],  # First image matches very well
    ["3: Moderately"],                 # Second image matches moderately
    ["1: Not at all"]                  # Third image doesn't match at all
]

# ===== REAL TASK DATA =====
# Images to be classified
IMAGE_URLS: list[str] = [
    "https://assets.rapidata.ai/tshirt-4o.png",       # Related T-Shirt with text
    "https://assets.rapidata.ai/tshirt-aurora.jpg",   # Related T-shirt with text
    "https://assets.rapidata.ai/teamleader-aurora.jpg", # Unrelated image
]

# Description that workers will compare against the images
T_SHIRT_DESCRIPTION: str = "A t-shirt with the text 'Running on caffeine & dreams'"
CONTEXTS: list[str] = [T_SHIRT_DESCRIPTION] * len(IMAGE_URLS)

def create_validation_set(client: RapidataClient) -> str:
    """
    Create a validation set to ensure quality responses.

    Args:
        client: The RapidataClient instance

    Returns:
        The validation set ID
    """
    validation_set = client.validation.create_classification_set(
        name="Example Likert Scale Validation Set",
        instruction=INSTRUCTION,
        answer_options=ANSWER_OPTIONS,
        contexts=VALIDATION_CONTEXTS,
        datapoints=VALIDATION_IMAGE_URLS,
        truths=VALIDATION_TRUTHS
    )
    return validation_set.id


def main():
    """Run the complete example workflow"""
    # Initialize the client
    client = RapidataClient()

    # Step 1: Create validation set
    validation_set_id = create_validation_set(client)
    print(f"Created validation set with ID: {validation_set_id}")

    # Step 2: Create and run the classification order
    print("Creating and running classification order...")
    order = client.order.create_classification_order(
        name="Likert Scale Example",  
        instruction=INSTRUCTION,
        answer_options=ANSWER_OPTIONS, 
        contexts=CONTEXTS,
        datapoints=IMAGE_URLS,
        validation_set_id=validation_set_id,  # Using our validation set
        responses_per_datapoint=25,
        settings=[NoShuffle()]               # Keep Likert scale in order
    ).run()  # Start the order

    order.display_progress_bar()

    results = order.get_results()
    print(results)

if __name__ == "__main__":
    main()

To preview the order and see what the annotators see, you can run the following code:

order.preview()

To open the order in the browser, you can run the following code:

order.view()