Upload config

CompressionConfig

Bases: BaseModel

Per-upload override for the asset service's compression behaviour.

Any field left as None falls back to the value the asset service has configured globally. Set enabled to True (or False) to force the behaviour for this client regardless of the server default. Quality is expected in the 1..100 range and max_dimension must be at least 1; both are validated server-side.

enabled governs both image and video compression: enabled=False preserves the original image and the original video (resolution and bitrate), which is the way to guarantee an uploaded 1080p clip reaches annotators untouched. quality and max_dimension only affect images; videos have no equivalent knob.

Applies to single-asset uploads (/asset/file and /asset/url) and to batched URL uploads via the orchestrator's /asset/batch-upload path.

Attributes:

Name	Type	Description
enabled	bool | None	Force compression on or off for both images and videos. None to defer to the server default.
quality	int | None	WebP quality (1..100) to use when image compression runs. Images only.
max_dimension	int | None	Maximum width or height in pixels when image compression runs. Images only.

is_set

is_set() -> bool

Whether any field has been overridden from its default of None. enabled=False counts as set — it is the explicit "force compression off" request, distinct from "defer to server default".

Source code in src/rapidata/rapidata_client/config/upload_config.py

def is_set(self) -> bool:
    """
    Whether any field has been overridden from its default of ``None``.
    ``enabled=False`` counts as set — it is the explicit "force compression
    off" request, distinct from "defer to server default".
    """
    return any(
        v is not None for v in (self.enabled, self.quality, self.max_dimension)
    )

cache_suffix

cache_suffix() -> str

Stable string used as part of the asset upload cache key so that the same source asset uploaded under different compression settings does not collide on a single cache entry.

The separator characters |, / and = are reserved — none of the existing field types (bool, int) can produce them, so the suffix round-trips unambiguously. Revisit this if a free-form string field is ever added.

Source code in src/rapidata/rapidata_client/config/upload_config.py

def cache_suffix(self) -> str:
    """
    Stable string used as part of the asset upload cache key so that
    the same source asset uploaded under different compression settings
    does not collide on a single cache entry.

    The separator characters ``|``, ``/`` and ``=`` are reserved — none of
    the existing field types (``bool``, ``int``) can produce them, so the
    suffix round-trips unambiguously. Revisit this if a free-form string
    field is ever added.
    """
    if not self.is_set():
        return ""
    return f"|c={self.enabled}/{self.quality}/{self.max_dimension}"

UploadConfig

UploadConfig(**kwargs)

Bases: BaseModel

Holds the configuration for the upload process.

Attributes:

Name	Type	Description
maxWorkers	int	The maximum number of worker threads for concurrent uploads. Defaults to 25.
maxRetries	int	The maximum number of retries for failed uploads. Defaults to 3.
cacheToDisk	bool	Enable disk-based caching for file uploads. If False, uses in-memory cache only. Defaults to True. Note: URL assets are always cached in-memory regardless of this setting. Caching cannot be disabled entirely as it's required for the two-step upload flow.
cacheTimeout	float	Cache operation timeout in seconds. Defaults to 0.1.
cacheLocation	Path	Directory for cache storage. Defaults to ~/.cache/rapidata/upload_cache. This is immutable. Only used for file uploads when cacheToDisk=True.
cacheShards	int	Number of disk-cache shards for concurrent file-cache access. Defaults to 32. Each shard is a separate on-disk store that holds open file handles, so a higher value raises the process's file-descriptor count — which can exceed a low ulimit -n and surface as "Too many open files". 32 comfortably covers the default maxWorkers of 25. Must be positive. Immutable at runtime — set it via the RAPIDATA_cacheShards environment variable. Only used for file uploads when cacheToDisk=True.
enableBatchUpload	bool	Enable batch URL uploading (two-step process). Defaults to True.
batchSize	int	Number of URLs per batch (100-5000). Defaults to 1000.
batchPollInterval	float	Polling interval in seconds. Defaults to 0.5.
compression	CompressionConfig | None	Per-upload override for the asset service's image-compression behaviour. Defaults to None (use server-side defaults).
contextShortening	bool	When True, every datapoint context is shortened for the order/job instruction before upload, keeping only the part relevant to the question. Defaults to False. Independent of this setting, a context longer than the backend's maximum length is always shortened so the backend accepts it, with a warning; that cannot be turned off.
failureTolerance	float	The fraction of a job's datapoints allowed to fail while still creating the job definition (0.0-1.0). 0.0 (default) is strict: any failed upload aborts creation so no incomplete definition is left behind, and the failed datapoints can be retried into the same dataset. 1.0 creates the definition regardless of how many datapoints failed, as long as at least one datapoint uploads successfully (a definition over an empty dataset is never created). Overridable per call via the failure_tolerance argument on create_*_job_definition. Defaults to 0.0.
checkForExplicitContent	bool | None	Opt in or out of Rapidata's server-side explicit-content check, applied when a job is assigned to an audience. None (default) uses the account's default. True forces the check on. False requests skipping it — honored only when the account is permitted to skip; otherwise the check still runs and a warning is logged. Defaults to None.

Source code in src/rapidata/rapidata_client/config/upload_config.py

def __init__(self, **kwargs):
    super().__init__(**kwargs)
    self._migrate_cache()