Skip to content

batchify

The batchify function is the one and only entrypoint for the batchling library, everything starts from there.

It can be used either through the Python SDK or CLI depending on your needs.

It comes with a bunch of parameters that you can customize to alter how batching is performed:

batchify 

batchify(
    batch_size=50,
    batch_window_seconds=2.0,
    batch_poll_interval_seconds=10.0,
    dry_run=False,
    cache=True,
)

Context manager used to activate batching for a scoped context.
Requests are accumulated within this context and will be batched and sent to the provider when the batch size or window is reached.
Batches are accumulated in different queues based on the provider/endpoint/model triplet.

Parameters:

Name Type Description Default
batch_size int

Submit a batch when this many requests are queued for a given provider/endpoint/model triplet.

 50
batch_window_seconds float

Submit a provider batch after this many seconds from the moment the first request is queued, even if size not reached.

 2.0
batch_poll_interval_seconds float

Poll active batches for results every this many seconds.

 10.0
dry_run bool

If True, intercept and batch requests without sending provider batches.
Use it to debug or before sending big jobs.
Batched requests resolve to synthetic responses.

 False
cache bool

If True, enable persistent request cache lookups.
This parameter allows to skip the batch submission and go straight to the polling phase for requests that have already been sent.

 True

Returns:

Type Description
BatchingContext

Context manager that yields None.
The context manager is only used to scope the batching context and is not used to yield any target.
Source code in src/batchling/api.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
def batchify(
    batch_size: int = 50,
    batch_window_seconds: float = 2.0,
    batch_poll_interval_seconds: float = 10.0,
    dry_run: bool = False,
    cache: bool = True,
) -> BatchingContext:
    """
    Context manager used to activate batching for a scoped context.<br>
    Requests are accumulated within this context and will be batched and sent to the provider when the batch size or window is reached.<br>
    Batches are accumulated in different queues based on the provider/endpoint/model triplet.

    Parameters
    ----------
    batch_size : int, optional
        Submit a batch when this many requests are queued for a given provider/endpoint/model triplet.
    batch_window_seconds : float, optional
        Submit a provider batch after this many seconds from the moment the first request is queued, even if size not reached.
    batch_poll_interval_seconds : float, optional
        Poll active batches for results every this many seconds.
    dry_run : bool, optional
        If ``True``, intercept and batch requests without sending provider batches.<br>
        Use it to debug or before sending big jobs.<br>
        Batched requests resolve to synthetic responses.
    cache : bool, optional
        If ``True``, enable persistent request cache lookups.<br>
        This parameter allows to skip the batch submission and go straight to the polling phase for requests that have already been sent.

    Returns
    -------
    BatchingContext
        Context manager that yields ``None``.<br>
        The context manager is only used to scope the batching context and is not used to yield any target.
    """
    # 1. Install hooks globally (idempotent)
    install_hooks()

    # 2. Create Batcher instance with provided configuration
    batcher = Batcher(
        batch_size=batch_size,
        batch_window_seconds=batch_window_seconds,
        batch_poll_interval_seconds=batch_poll_interval_seconds,
        dry_run=dry_run,
        cache=cache,
    )

    # 3. Return BatchingContext with no yielded target.
    return BatchingContext(
        batcher=batcher,
    )

Next steps

Now that you have more information about the batchify function, you can: