Storage Module

The Storage module provides different storage implementations for data management, including hot, cold, and mixed storage strategies.

Base Storage

Base classes and utilities for storage implementations.

class true_storage.storage.base.StoragePolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Storage policies for data management.

STRICT = 'strict'
LENIENT = 'lenient'
LAZY = 'lazy'
class true_storage.storage.base.StorageMetrics(hits=0, misses=0, total_operations=0, total_errors=0, avg_response_time=0.0, storage_size=0, last_cleanup_time=0.0)[source]

Bases: object

Storage metrics for monitoring.

hits: int = 0
misses: int = 0
total_operations: int = 0
total_errors: int = 0
avg_response_time: float = 0.0
storage_size: int = 0
last_cleanup_time: float = 0.0
__init__(hits=0, misses=0, total_operations=0, total_errors=0, avg_response_time=0.0, storage_size=0, last_cleanup_time=0.0)
class true_storage.storage.base.StorageEvent(storage_id, event_type, data=None)[source]

Bases: object

Base class for storage events.

__init__(storage_id, event_type, data=None)[source]
class true_storage.storage.base.StorageEventHandler[source]

Bases: ABC

Abstract base class for storage event handlers.

abstract handle_event(event)[source]

Handle a storage event.

Return type:

None

class true_storage.storage.base.StorageOptimizer[source]

Bases: ABC

Abstract base class for storage optimizers.

abstract optimize(data)[source]

Optimize data for storage.

Return type:

Any

abstract deoptimize(data)[source]

Restore data from optimized form.

Return type:

Any

class true_storage.storage.base.StorageStrategy[source]

Bases: ABC, Generic[T]

Abstract base class for storage strategies.

abstract store(key, value)[source]

Store a value.

Return type:

None

abstract retrieve(key)[source]

Retrieve a value.

Return type:

Optional[TypeVar(T)]

abstract delete(key)[source]

Delete a value.

Return type:

None

class true_storage.storage.base.BaseStorageManager(storage_id, policy=StoragePolicy.STRICT, event_handlers=None, optimizer=None)[source]

Bases: object

Base class for storage management.

__init__(storage_id, policy=StoragePolicy.STRICT, event_handlers=None, optimizer=None)[source]
add_event_handler(handler)[source]

Add an event handler.

Return type:

None

emit_event(event_type, data=None)[source]

Emit a storage event.

Return type:

None

register_callback(event_type, callback)[source]

Register a callback for an event type.

Return type:

None

_trigger_callbacks(event_type, *args, **kwargs)[source]

Trigger callbacks for an event type.

Return type:

None

update_metrics(operation, success, response_time)[source]

Update storage metrics.

Return type:

None

get_metrics()[source]

Get current storage metrics.

Return type:

StorageMetrics

optimize_data(data)[source]

Optimize data for storage.

Return type:

Any

deoptimize_data(data)[source]

Restore data from optimized form.

Return type:

Any

Hot Storage

Hot storage module providing fast in-memory caching with LRU eviction strategy.

This module implements a high-performance hot storage system using LRU (Least Recently Used) caching mechanism with features like expiration, optimization, and event handling.

Classes:

HotStorage: Main class implementing hot storage functionality. StoragePolicy: Enumeration of possible storage policies. (you can import it from base.py)

Functions:

None (all functionality is encapsulated in classes)

Types:

None

Exceptions:

StorageError: Raised when storage operations fail

Key Features:

  • LRU (Least Recently Used) caching strategy

  • Thread-safe operations

  • Automatic item expiration

  • Data optimization support

  • Event emission system

  • Performance metrics tracking

  • Configurable storage policies

  • Callback system for storage events

class true_storage.storage.hot.HotStorage(storage_id='hot_storage', max_size=100, expiration_time=300, policy=StoragePolicy.STRICT)[source]

Bases: BaseStorageManager

Hot storage implementation with LRU cache and advanced features.

__init__(storage_id='hot_storage', max_size=100, expiration_time=300, policy=StoragePolicy.STRICT)[source]
_is_expired(key)[source]

Check if a key has expired.

Return type:

bool

_remove_expired_items()[source]

Remove expired items from storage.

Return type:

None

store(key, value)[source]

Store a value in hot storage.

Return type:

None

retrieve(key)[source]

Retrieve a value from hot storage.

Return type:

Optional[Any]

delete(key)[source]

Delete a value from hot storage.

Return type:

None

clear()[source]

Clear all items from hot storage.

Return type:

None

get_size()[source]

Get the current size of hot storage.

Return type:

int

get_keys()[source]

Get all non-expired keys in hot storage.

Return type:

list[str]

get_stats()[source]

Get storage statistics.

Return type:

Dict[str, Any]

class true_storage.storage.hot.StoragePolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Storage policies for data management.

STRICT = 'strict'
LENIENT = 'lenient'
LAZY = 'lazy'
exception true_storage.storage.hot.StorageError[source]

Bases: Exception

Base exception for storage-related errors.

Cold Storage

Cold storage module providing persistent disk-based storage with compression.

This module implements a disk-based storage system with data compression and metadata tracking for efficient long-term data storage.

Classes:

ColdStorage: Main class implementing cold storage functionality. StoragePolicy: Enumeration of possible storage policies. (imported from base.py)

Functions:

None (all functionality is encapsulated in classes)

Types:

None

Exceptions:

StorageError: Raised when storage operations fail

Key Features:

  • File system based persistent storage

  • Data compression using zlib

  • Metadata tracking for stored items

  • Thread-safe operations

  • Performance metrics tracking

  • Event emission system

  • Configurable storage policies

  • Automatic directory management

  • Size and statistics tracking

class true_storage.storage.cold.ColdStorage(storage_id='cold_storage', storage_directory='cold_storage', compression_level=6, policy=StoragePolicy.STRICT)[source]

Bases: BaseStorageManager

Cold storage implementation with compression and advanced features.

__init__(storage_id='cold_storage', storage_directory='cold_storage', compression_level=6, policy=StoragePolicy.STRICT)[source]
_load_metadata()[source]

Load metadata from JSON file.

Return type:

None

_save_metadata()[source]

Save metadata to JSON file.

Return type:

None

store(key, value)[source]

Store a value in cold storage.

Return type:

None

retrieve(key)[source]

Retrieve a value from cold storage.

Return type:

Optional[Any]

delete(key)[source]

Delete a value from cold storage.

Return type:

None

clear()[source]

Clear all items from cold storage.

Return type:

None

get_size()[source]

Get the total size of cold storage in bytes.

Return type:

int

get_keys()[source]

Get all keys in cold storage.

Return type:

list[str]

get_stats()[source]

Get storage statistics.

Return type:

Dict[str, Any]

get_item_metadata(key)[source]

Get metadata for a specific item.

Return type:

Optional[Dict]

class true_storage.storage.cold.StoragePolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Storage policies for data management.

STRICT = 'strict'
LENIENT = 'lenient'
LAZY = 'lazy'
exception true_storage.storage.cold.StorageError[source]

Bases: Exception

Base exception for storage-related errors.

Mixed Storage

Mixed storage module combining hot and cold storage capabilities.

This module implements a hybrid storage system that leverages both in-memory (hot) and disk-based (cold) storage for optimal performance and data persistence.

Classes:

MixedStorage: Main class implementing hybrid storage functionality. StoragePolicy: Enumeration of possible storage policies. (imported from base.py)

Functions:

None (all functionality is encapsulated in classes)

Types:

None

Exceptions:

StorageError: Raised when storage operations fail

Key Features:

  • Hybrid storage combining hot and cold storage benefits

  • Automatic data synchronization between storage layers

  • Intelligent data retrieval strategy

  • Performance metrics for both storage layers

  • Storage optimization capabilities

  • Hot storage warm-up functionality

  • Thread-safe operations

  • Event emission system

  • Configurable storage policies

class true_storage.storage.mixed.MixedStorage(storage_id='mixed_storage', max_size=100, expiration_time=300, storage_directory='cold_storage', compression_level=6, policy=StoragePolicy.STRICT)[source]

Bases: BaseStorageManager

Mixed storage implementation combining hot and cold storage.

__init__(storage_id='mixed_storage', max_size=100, expiration_time=300, storage_directory='cold_storage', compression_level=6, policy=StoragePolicy.STRICT)[source]
store(key, value)[source]

Store a value in mixed storage.

Return type:

None

retrieve(key)[source]

Retrieve a value from mixed storage.

Return type:

Optional[Any]

delete(key)[source]

Delete a value from mixed storage.

Return type:

None

clear()[source]

Clear all items from mixed storage.

Return type:

None

get_hot_stats()[source]

Get hot storage statistics.

Return type:

Dict[str, Any]

get_cold_stats()[source]

Get cold storage statistics.

Return type:

Dict[str, Any]

get_stats()[source]

Get combined storage statistics.

Return type:

Dict[str, Any]

get_keys()[source]

Get all keys from both hot and cold storage.

Return type:

list[str]

optimize_hot_storage()[source]

Optimize hot storage by removing least accessed items.

Return type:

None

warm_up_hot_storage(keys)[source]

Pre-load specified keys into hot storage.

Return type:

None

class true_storage.storage.mixed.StoragePolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Storage policies for data management.

STRICT = 'strict'
LENIENT = 'lenient'
LAZY = 'lazy'
exception true_storage.storage.mixed.StorageError[source]

Bases: Exception

Base exception for storage-related errors.