lifelong_methods.buffer package

Submodules

lifelong_methods.buffer.buffer module

class lifelong_methods.buffer.buffer.BufferBase(config: Dict, buffer_dir: Optional[str] = None, map_size: int = 1000000000.0, essential_transforms_fn: Optional[Callable[[PIL.Image.Image], torch.Tensor]] = None, augmentation_transforms_fn: Optional[Callable[[PIL.Image.Image], torch.Tensor]] = None)

Bases: abc.ABC, torch.utils.data.dataset.Dataset

A buffer that saves memories from current task to replay them during later tasks

Parameters

  • config (Dict) –

    A dictionary that has the following key value pairs: n_memories_per_class (int): Number of memories/samples to save per class, set to -1 to use total_n_mems total_n_mems (int): The total number of memories to save (mutually exclusive with n_memories_per_class) max_mems_pool_per_class (int): The pool size per class to sample randomly the memories from which the buffer

    chooses what memories to keep, set to -1 to choose memories from all the class samples

  • buffer_dir (Optional[str]) – The directory where the buffer data will be kept (None for keeping the buffer data in memory) (default: None)

  • map_size (int) – Th estimated size of the buffer lmdb database, in bytes (defalt: 1e9)

  • essential_transforms_fn (Optional[Callable[[Image.Image], torch.Tensor]]) – A function that contains the essential transforms (for example, converting a pillow image to a tensor) that should be applied to each image. This function is applied only when the augmentation_transforms_fn is set to None (as in the case of a test set) or inside the disable_augmentations context (default: None)

  • augmentation_transforms_fn – (Optional[Callable[[Image.Image], torch.Tensor]]): A function that contains the essential transforms (for example, converting a pillow image to a tensor) and augmentation transforms (for example, applying random cropping) that should be applied to each image. When this function is provided, essential_transforms_fn is not used except inside the disable_augmentations context (default: None)

buffer_state_dict() → Dict

This function returns a dict that contains the current state of the buffer

Returns

a dictionary with all the attributes (key is attribute name) and their values, except the attributes in the self.non_savable_attributes

Return type

Dict

load_buffer_state_dict(state_dict: Dict) → None

This function loads the object attributes with the values in state_dict

Parameters

state_dict (Dict) – a dictionary with the attribute names as keys and their values

get_image_indices_by_class(class_name: str) → numpy.ndarray

get the indices of the samples of class “class_name”

Parameters

class_name (str) – The class name

Returns

The indices of the samples of class “class_name”

Return type

np.ndarray

begin_adding_samples_to_lmdb() → None
A function that needs to be called before adding samples to the buffer, in case of using an lmdb buffer, so that

a transaction is created.

end_adding_samples_to_lmdb() → None
A function that needs to be called after adding samples to the buffer is done, in case of using an lmdb buffer,

so that the transaction is committed.

reset_lmdb_database() → None
A function that needs to be called after each epoch, in case of using an lmdb dataset, to close the environment

and open a new one to kill active readers

add_sample(class_label: str, image: PIL.Image.Image, labels: Tuple[str, str], rank: int = 0) → None

Add a sample to the buffer.

Parameters

  • class_label (str) – The class label of the image, and in case the image has multiple labels, the class label for which the sample should be associated with in the buffer

  • image (Image.Image) – The image to be added

  • labels (Tuple[str, str]) – The labels of the image (including the class_label), in case the image has only one label, provide the second label as NO_LABEL_PLACEHOLDER

  • rank (int) – The rank of the current gpu, in case of using multiple gpus

remove_samples(class_label: str, n: int) → None

Remove a number (n) of the samples associated with class “class_label”.

Parameters

  • class_label (str) – The class label of which the sample is associated with in the buffer

  • n (int) – The number of samples to remove

update_buffer_new_task(new_task_data: iirc.lifelong_dataset.torch_dataset.Dataset, dist_args: Optional[Dict] = None, **kwargs) → None
Update the buffer by adding samples of classes of a new task, after removing samples associated with the older

classes in case the buffer has a fixed size (self.fixed_n_mems_per_cla is set to False)

Parameters

  • new_task_data (Dataset) – The new task data

  • dist_args (Optional[Dict]) – a dictionary of the distributed processing values in case of multiple gpu (ex:

  • of the device) (default (rank) – None)

  • **kwargs – arguments associated with each method

disable_augmentations()

A context where only the essential transformations are applied

class lifelong_methods.buffer.buffer.TaskDataMergedWithBuffer(buffer: lifelong_methods.buffer.buffer.BufferBase, task_data: iirc.lifelong_dataset.torch_dataset.Dataset, buffer_sampling_multiplier: float = 1.0)

Bases: torch.utils.data.dataset.Dataset

A torch dataset object that merges the task data and the buffer with the specified options

Parameters

  • buffer (BufferBase) – A buffer object that includes the memories from previous classes

  • task_data (data.Dataset) – A dataset object that contains the new task data

  • buffer_sampling_multiplier (float) – A multiplier for sampling from the buffer more/less times than the size of the buffer (for example a multiplier of 2 samples from the buffer (with replacement) twice its size per epoch, a multiplier of 1 ensures that all the buffer samples will be retrieved once”)

Module contents