app.domain.helpers
Submodules
app.domain.helpers.enums
Module with classes that inherit from python’s builtin Enum class.
-
class
HttpCodes(value)[source] Bases:
enum.EnumEnumerator class used to represent HTTP response codes.
The following codes are considered:
DUMMY: Dummy value. Use when no valid HTTP code exists;
OK: Callee accepts the sent file;
BAD_REQUEST: Callee refuses the integrity of sent file;
NOT_FOUND: Callee is not a member of the network;
NOT_ACCEPTABLE: Callee already has a file with same Id;
TIME_OUT: Message lost in translation;
SERVER_DOWN: Metadata server is offline;
-
BAD_REQUEST: int = 400
-
DUMMY: int = -1
-
NOT_ACCEPTABLE: int = 406
-
NOT_FOUND: int = 404
-
OK: int = 200
-
SERVER_DOWN: int = 521
-
TIME_OUT: int = 408
app.domain.helpers.exceptions
Module with classes that inherit from Python’s builtin Exception class and represent domain specific errors.
-
exception
DistributionShapeError(value='')[source] Bases:
ExceptionRaised when a vector shape is expected to match a matrix’s column or row shape and does not.
-
__init__(value='')[source] Initialize self. See help(type(self)) for accurate signature.
-
-
exception
IllegalArgumentError(value='')[source] Bases:
ValueErrorGeneric error used to indicate a parameter is not valid or expected.
-
__init__(value='')[source] Initialize self. See help(type(self)) for accurate signature.
-
-
exception
MatlabEngineContainerError(value='')[source] Bases:
ExceptionRaised an AttributeError or EngineError is found in matlab container modules.
-
__init__(value='')[source] Initialize self. See help(type(self)) for accurate signature.
-
app.domain.helpers.matlab_utils
Module with Matlab related classes.
-
class
MatlabEngineContainer[source] Bases:
objectSingleton class wrapper containing thread safe access to a MatlabEngine.
This class provides a thread-safe way to access one singleton matlab engine object when running simulations in threaded mode. Having one single engine is important, since starting up an engine takes approximately 12s (machine dependant), not including the time matlab scripts are executing and data convertions between python and matlab and back.
-
eng A matlab engine instance, which can be used for example for matrix and vector optimization operations throughout the simulations.
Note
MatlabEngine objects are not thread safe, thus it is recommended that you utilize the a wrapper function that obtains
_LOCK, before you send any requests toeng.
-
__init__()[source] Instantiates a new MatlabEngineContainer object.
Note
Do not directly invoke constructor, use
get_instance()instead.- Return type
-
static
get_instance()[source] Used to obtain a singleton instance of
MatlabEngineContainer.If one instance already exists that instance is returned, otherwise a new one is created and returned.
- Returns
A reference to the existing
MatlabEngineContainerinstanceor None if matlab python engine is not properly installed.- Return type
-
matrix_global_opt(a, v_)[source] Constructs an optimized transition matrix using the matlab engine.
Constructs an optimized transition matrix using linear programming relaxations and convex envelope approximations for the specified steady state
v_, this is done by invoke the matlabscript matrixGlobalOpt.m located insideMATLAB_DIR.- Parameters
a (numpy.ndarray) – A non-optimized symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic steady state distribution vector.
- Returns
Markov Matrix with
v_as steady state distribution and the respectivemixing rateorNone.- Raises
EngineError – If you do not have a valid MatLab license.
- Return type
Any
-
_LOCK= <unlocked _thread.RLock object owner=0 count=0> A re-entrant lock used to make eng shareable by multiple threads.
-
_instance: app.domain.helpers.matlab_utils.MatlabEngineContainer = None A reference to the instance of MatlabEngineContainer or None.
-
app.domain.helpers.matrices
Module used by domain.cluster_groups.Cluster to create transition
matrices for the simulation.
You should implement your own metropolis-hastings or alternative algorithms as well as any steady-state or transition matrix optimization algorithms in this module.
-
_adjency_matrix_sdp_optimization(a)[source] Optimizes a symmetric adjacency matrix using Semidefinite Programming.
The optimization is done with respect to the uniform stochastic vector with the the same length as the inputed symmetric matrix.
Note
This function tries to use Mosek Solver, if a valid license is not found, it uses SCS Solver instead.
- Parameters
a (numpy.ndarray) – Any symmetric adjacency matrix.
- Returns
The optimal matrix or None if the problem is unfeasible.
- Return type
Optional[Tuple[cvxpy.problems.problem.Problem, cvxpy.expressions.variable.Variable]]
-
_construct_random_walk_matrix(a)[source] Builds a random walk matrix over the given adjacency matrix
- Parameters
a (numpy.ndarray) – Any adjacency matrix.
- Returns
A matrix representing the performed random walk.
- Return type
-
_construct_rejection_matrix(rw, v_)[source] Builds a matrix of rejection probabilities for a given random walk.
- Parameters
rw (numpy.ndarray) – a random_walk over an adjacency matrix
v_ (numpy.ndarray) – a stochastic desired distribution vector
- Returns
A matrix whose entries are acceptance probabilities for
rw.- Return type
-
_get_diagonal_entry_probability_v1(rw, r, i)[source] Helper function used during the metropolis-hastings algorithm.
Calculates the value that should be assigned to the entry
(i, i)of the transition matrix being calculated by the metropolis hastings algorithm by considering the rejection probability over the random walk that was performed on an adjacency matrix.Note
This method does considers element-wise rejection probabilities for random walk matrices. If you wish to implement a modification of the metropolis-hastings algorithm and you do not utilize rejection matrices use
_get_diagonal_entry_probability_v2()instead.- Parameters
rw (numpy.ndarray) – A random walk over an adjacency matrix.
r (numpy.ndarray) – A matrix whose entries contain acceptance probabilities for
rw.i (int) – The diagonal-index of
rwwhere summation needs to be performed on. E.g.:rw[i, i].
- Returns
A probability to be inserted at entry
(i, i)of the transition matrix outputed by the_metropolis_hastings().- Return type
numpy.float64
-
_get_diagonal_entry_probability_v2(m, i)[source] Helper function used during the metropolis-hastings algorithm.
Calculates the value that should be assigned to the entry
(i, i)of the transition matrix being calculated by the metropolis hastings algorithm by considering the rejection probability over the random walk that was performed on an adjacency matrix.Note
This method does not consider element-wise rejection probabilities for random walk matrices. If you wish to implement a modification of the metropolis-hastings algorithm and you utilize rejection matrices use
_get_diagonal_entry_probability_v1()instead.- Parameters
m (numpy.ndarray) – The matrix to receive the diagonal entry value.
i (int) – The diagonal entry index. E.g.:
m[i, i].
- Returns
A probability to be inserted at entry
(i, i)of the transition matrix outputed by the_metropolis_hastings().- Return type
numpy.float64
-
_metropolis_hastings(a, v_, column_major_out=True, version=2)[source] Constructs a transition matrix using metropolis-hastings algorithm.
Note
The input Matrix hould have no transient states/absorbent nodes, but this is not enforced or verified.
- Parameters
a (numpy.ndarray) – A symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic vector that is the steady state of the resulting transition matrix.
column_major_out (bool) – Indicates whether to return transition_matrix output is in row or column major form.
version (int) – Indicates which version of the algorith should be used (default is version 2).
- Returns
An unlabeled transition matrix with steady state
v_.- Raises
DistributionShapeError – When the length of
v_is not the same as the matrix a.MatrixNotSquareError – When matrix a is not a square matrix.
- Return type
-
get_mixing_rate(m)[source] Calculats the fast mixing rate the input matrix.
The fast mixing rate of matrix
mis the highest eigenvalue that is smaller than one. If returned value is1.0than the matrix has transient states or absorbent nodes and as a result is not a markov matrix.- Parameters
m (numpy.ndarray) – A matrix.
- Returns
The highest eigenvalue of
mthat is smaller than one or one.- Return type
-
is_connected(m, directed=False)[source] Checks if a matrix is connected by counting the number of connected components.
- Parameters
m (numpy.ndarray) – The matrix to be verified.
directed (bool) – If
medges are directed, i.e., ifmis an adjency matrix in which the edges bidirectional.Falsemeans they are.Truemeans they are not.
- Returns
Trueif the matrix is a connected graph, elseFalse.- Return type
-
is_symmetric(m, tol=1e-08)[source] Checks if a matrix is symmetric by performing element-wise equality comparison on entries of
mandm.T.- Parameters
m (numpy.ndarray) – The matrix to be verified.
tol (float) – The tolerance used to verify the entries of the
m(default is 1e-8).
- Returns
Trueif themis symmetric, elseFalse.- Return type
-
make_connected(m)[source] Turns a matrix into a connected matrix that could represent a connected graph.
- Parameters
m (numpy.ndarray) – The matrix to be made connected.
- Returns
A connected matrix. If
mwas symmetric the modified matrix will also be symmetric.- Return type
-
new_go_transition_matrix(a, v_)[source] Constructs a transition matrix using global optimization techniques.
Constructs an optimized markov matrix using linear programming relaxations and convex envelope approximations for the specified steady state
v. Result is only trully optimal if \(normal(Mopt - (1 / len(v)), 2)\) is equal to the highest Markov Matrix eigenvalue that is smaller than one.- Parameters
a (numpy.ndarray) – A non-optimized symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic steady state distribution vector.
- Returns
Markov Matrix with
v_as steady state distribution and the respective mixing rate.- Return type
Tuple[Optional[numpy.ndarray], float]
-
new_mgo_transition_matrix(a, v_)[source] Constructs an optimized transition matrix using the matlab engine.
Constructs an optimized transition matrix using linear programming relaxations and convex envelope approximations for the specified steady state
v. Result is only trully optimal if \(normal(Mopt - (1 / len(v)), 2)\) is equal to the highest Markov Matrix eigenvalue that is smaller than one.Note
This function’s code runs inside a matlab engine because it provides a non-convex SDP solver BMIBNB. If you do not have valid matlab license the output of this function is always
(None, float('inf').- Parameters
a (numpy.ndarray) – A non-optimized symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic steady state distribution vector.
- Returns
Markov Matrix with
v_as steady state distribution and the respective mixing rate.- Return type
Tuple[Optional[numpy.ndarray], float]
-
new_mh_transition_matrix(a, v_)[source] Constructs a transition matrix using metropolis-hastings.
Constructs a transition matrix using metropolis-hastings algorithm for the specified steady state
v.Note
The input Matrix hould have no transient states or absorbent nodes, but this is not enforced or verified.
- Parameters
a (numpy.ndarray) – A symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic steady state distribution vector.
- Returns
Markov Matrix with
v_as steady state distribution and the respective mixing rate orNone, float('inf')if the problem is infeasible.- Return type
Tuple[numpy.ndarray, float]
-
new_sdp_mh_transition_matrix(a, v_)[source] Constructs a transition matrix using semi-definite programming techniques.
Constructs a transition matrix using metropolis-hastings algorithm for the specified steady state
v. The provided adjacency matrix A is first optimized with semi-definite programming techniques for the uniform distribution vector.- Parameters
a (numpy.ndarray) – A non-optimized symmetric adjency matrix.
v_ (numpy.ndarray) – A stochastic steady state distribution vector.
- Returns
Markov Matrix with
v_as steady state distribution and the respective mixing rate orNone, float('inf')if the problem is infeasible.- Return type
Tuple[Optional[numpy.ndarray], float]
-
new_symmetric_connected_matrix(size, allow_sloops=True, force_sloops=True)[source] Generates a random symmetric matrix which is also connected.
See
new_symmetric_matrix()andmake_connected().- Parameters
size (int) – The length of the square matrix.
allow_sloops (bool) – See
new_symmetric_matrix()for clarifications.force_sloops (bool) – See
new_symmetric_matrix()for clarifications.
- Returns
A matrix that represents an adjacency matrix that is also connected.
- Return type
-
new_symmetric_matrix(size, allow_sloops=True, force_sloops=True)[source] Generates a random symmetric matrix.
The generated adjacency matrix does not have transient state sets or absorbent nodes and can effectively represent a network topology with bidirectional connections between
network nodes.- Parameters
size (int) – The length of the square matrix.
allow_sloops (bool) – Indicates if the generated adjacency matrix allows diagonal entries representing self-loops. If
False, then, all diagonal entries must be zeros. Otherwise, they can be zeros or ones.force_sloops (bool) – Indicates if the diagonal of the generated matrix should be filled with ones. If
Falsevalid diagonal entries are decided byallow_self_loopsparam. Otherwise, diagonal entries are filled with ones. Ifallow_self_loopsisFalseandenforce_loopsisTrue, an error is raised.
- Returns
The adjency matrix representing the connections between a groups of
network nodes.- Raises
IllegalArgumentError – When
allow_self_loops(False) conflicts withenforce_loops(True).- Return type
app.domain.helpers.smart_dataclasses
Module with classes that help to avoid domain class polution by encapsulating attribute and method behaviour.
-
class
FileBlockData(cluster_id, name, number, data)[source] Bases:
objectWrapping class for the contents of a file block.
Among other responsabilities FileBlockData helps managing simulation parameters, e.g., replica control such or file block integrity.
-
cluster_id Unique identifier of the cluster that manages the file block.
-
name The name of the file the file block belongs to.
-
number The number that uniquely identifies the file block.
-
id Concatenation the the name and number.
-
references Tracks how many references exist to the file block in the simulation environment. When it reaches 0 the file block ceases to exist and the simulation fails.
-
replication_epoch When a reference to the file block is lost, i.e., decremented, a replication epoch that simulates time to copy blocks from one node to another is assigned to this attribute. Until a loss occurs and after a loss is recovered, recovery_epoch is set to positive infinity.
-
data A base64-encoded string representation of the file block bytes.
-
sha256 The hash value of data resulting from a SHA256 digest.
-
__init__(cluster_id, name, number, data)[source] Creates an instance of FileBlockData.
-
__str__()[source] Overrides default string representation of FileBlockData instances.
- Returns
A dictionary representation of the object.
-
can_replicate(epoch)[source] Informs the calling network node if file block needs replication.
-
decrement_and_get_references()[source] Decreases by one and gets the number of file block references.
- Returns
The number of file block references existing in the simulation environment.
-
set_replication_epoch(epoch)[source] Sets the epoch in which replication levels should be restored.
This method tries to assign a new epoch, in the future, at which recovery should be performed. If the proposition is sooner than the previous proposition then assignment is accepted, else, it’s rejected.
Note
This method of calculating the replication_epoch may seem controversial, but the justification lies in the assumption that if there are more network nodes monitoring file parts, than failure detections should be in theory, faster, unless complex consensus algorithms are being used between volatile peers, which is not our case. We assume peers only report their suspicions to a small number of trusted of monitors who then decide if the reported network node is disconnected, consequently losing the instance of FileBlockData and possibly others.
-
update_epochs_to_recover(epoch)[source] Update the replication_epoch after a recovery attempt was carried out.
If the recovery attempt performed by some network node successfully managed to restore the replication levels to the original target, then, replication_epoch is set to positive infinity, otherwise, another attempt will be done in the next epoch.
-
-
class
FileData(name, sim_id=0, origin='')[source] Bases:
objectHolds essential simulation data concerning files being persisted.
FileData is a helper class which has responsabilities such as tracking how many file block replicas currently existing in a
cluster groupbut also keeping simulation events logged in RAM until the simulation ends, at which point the logs are written to disk.-
name The name of the original file.
- Type
-
existing_replicas The number of file parts including blocks that exist for the named file that exist in the simulation. Updated every epoch.
- Type
-
logger Object that stores captured simulation data. Stored data can be post-processed using user defined scripts to create items such has graphs and figures.
- Type
-
out_file File output stream to where captured data is written in append mode and to which
loggerwill be written to at the end of the simulation.
-
__init__(name, sim_id=0, origin='')[source] Creates an instance of
FileData.- Parameters
name (str) – Name of the file to be referenced by the
FileDataobject.sim_id (int) – Identifier that generates unique output file names, thus guaranteeing that different simulation instances do not overwrite previous
output files.origin (str) – The name of the simulation file name that started the simulation process. See
Masterandhive_simulation. In addition to the previous, the origin should somehow include the cluster class name being run, to differentiate simulations’ output files being executed by different distributed storage system implementations.
- Return type
-
fclose(msg=None)[source] Closes the output stream controlled by the
FileDatainstance.
-
fwrite(msg)[source] Appends a message to the output stream of
FileData.The method automatically adds a new line character to
msg.
-
jwrite(cluster, origin, epoch)[source] Appends a json string to the output stream of
FileData.The logged data are all attributes belonging to
logger.- Parameters
cluster (domain.cluster_groups.Cluster) – The
Clusterobject that manages the simulated persistence of thenamed file.origin (str) – The name of the simulation file name that started the simulation process. See
Masterandhive_simulation.epoch (int) – The epoch at which the
loggerwas appended toout_file.
- Return type
-
-
class
LoggingData[source] Bases:
objectLogger object that stores simulation events and other data.
Note
Some attributes might not be documented, but should be straight forward to understand after inspecting their usage in the source code.
-
cswc Indicates how many consecutive steps a file as been in convergence. Once convergence is not verified by
equal_distributions()this attribute is reset to zero.- Type
-
largest_convergence_window Stores the largest convergence window that occurred throughout the simulation, i.e., it stores the highest verified
cswc.- Type
-
convergence_set Set of consecutive epochs in which convergence was verified. This list only stores the most up to date convergence set and like
cswcis cleared once convergence is not verified, after being appended toconvergence_sets.- Type
List[int]
-
convergence_sets Stores all but the most recent
convergence_set. If simulation terminates andconvergence_setis not an empty list, that list will be appended to this one.- Type
List[List[int]]
-
terminated Indicates the epoch at which the simulation was terminated.
- Type
-
terminated_messages Set of at least one error message that led to the failure of the simulation or one success message, at
termination epoch.- Type
List[str]
-
successfull When the simulation is
terminated, this value is set toTrueif no errors or failures occurred, i.e., if the simulation managed to persist the file throughout the entiresimulation epochs.- Type
-
blocks_corrupted The number of
file block replicaslost at each simulation epoch due to disk errors.- Type
List[int]
-
blocks_existing The number of existing
file block replicasinside thecluster groupmembers’ storage disks at each epoch.- Type
List[int]
-
blocks_lost The number of
file block replicasthat were lost at each epoch due tonetwork nodesgoing offline.- Type
List[int]
-
blocks_moved The number of messages containing
file block replicasthat were transmited, including those that were not delivered or acknowledged, at each epoch.- Type
List[int]
-
cluster_size_bm The number of
network nodesregistered at acluster group's members list, before themaintenance stepof the epoch.- Type
List[int]
-
cluster_size_am The number of
network nodesregistered at acluster group's members list, after themaintenance stepof the epoch.- Type
List[int]
-
cluster_status_bm Strings describing the health of the
cluster groupat each epoch, before themaintenance stepof the epoch.- Type
List[str]
-
cluster_status_am Strings describing the health of the
cluster groupat each epoch, after themaintenance stepof the epoch.- Type
List[str]
-
delay_replication Log of the average time it took to recover one or more lost
file block replicas, at each epoch.- Type
List[float]
-
delay_suspects_detection Log of the time it took for each suspicious
network nodeto be evicted from the hiscluster groupafter having hisstatuschanged from online to offline or suspicious.
-
initial_spread Records the strategy used distribute file blocks in the beggining of the simulation. See
spread_files().- Type
-
matrices_nodes_degrees Stores the
in-degreeandout-degreeof eachnetwork nodein thecluster group. One dictionary is kept in the list for each transition matrix used throughout the simulation. The integral part of the float value is the in-degree, the decimal part is the out-degree.
-
off_node_count The number of
network nodeswhose status changed to offline or suspicious, at each epoch.- Type
List[int]
-
topologies_goal_achieved Stores if a boolean indicating if each topology achieved the desired density distribution, on average.
- Type
List[bool]
-
topologies_goal_distance Stores magnitude difference between the desired density distribution and the topologies’ average density distribution.
- Type
List[float]
-
transmissions_failed The number of message transmissions that were lost in the overlay network of a
cluster group, at each epoch.- Type
List[int]
-
_recursive_len(item)[source] Recusively sums the length of all lists in
convergence_sets.- Parameters
item (Any) – A sub list of
convergence_setsthat needs that as not yet been counted.- Returns
The number of epochs that were registered at the inputed sub list.
- Return type
-
log_bandwidth_units(n, epoch)[source] Logs the amount of moved file blocks moved at an epoch.
-
log_corrupted_file_blocks(n, epoch)[source] Logs the amount of corrupted file block blocks at an epoch.
-
log_existing_file_blocks(n, epoch)[source] Logs the amount of existing file blocks in the simulation environment at an epoch.
-
log_fail(epoch, message='')[source] Logs the epoch at which a simulation terminated due to a failure.
Note
This method should only be called when simulation terminates due to a failure such as a the loss of all blocks of a file block or the simultaneous disconnection of all network nodes in the cluster.
-
log_lost_file_blocks(n, epoch)[source] Logs the amount of permanently lost file block blocks at an epoch.
-
log_lost_messages(n, epoch)[source] Logs the amount of failed message transmissions at an epoch.
-
log_maintenance(size_bm, size_am, status_bm, status_am, epoch)[source] Logs cluster membership status and size at an epoch.
- Parameters
size_bm (int) – The number of network nodes in the cluster before maintenance.
size_am (int) – The number of network nodes in the cluster after maintenance.
status_bm (str) – A string that describes the status of the cluster before maintenance.
status_am (str) – A string that describes the status of the cluster after maintenance.
epoch (int) – A simulation epoch at which termination occurred.
- Return type
-
log_matrices_degrees(nodes_degrees)[source] Logs the degree of all nodes in a Markov Matrix overlay, at the time of its creation, before any faults on the overlay occurs.
- Parameters
nodes_degrees (Dict[str, str]) – A dictionary mapping the
node identifiersto theirin-degreeandout-degreeseperated by the delimiteri#o.
-
log_off_nodes(n, epoch)[source] Logs the amount of disconnected network_nodes at an epoch.
-
log_replication_delay(delay, calls, epoch)[source] Logs the expected delay_replication at epoch at an epoch.
-
log_suspicous_node_detection_delay(node_id, delay)[source] Logs the expected delay_replication at epoch at an epoch.
- Parameters
delay (int) – The time it took until the specified node was evicted from a
Clusterafter it was known to be offline by the perfect failure detector.node_id (str) – A unique
Network Nodeidentifier.
- Return type
-
log_topology_goal_performance(achieved_goal, distance_magnitude)[source] Logs wether or not the topology reached it’s goals, on average.
-
register_convergence(epoch)[source] Increments
cswcby one and tries to update theconvergence_setChecks if the counter for consecutive epoch convergence is bigger than
MIN_CONVERGENCE_THRESHOLDand if it is, it appends theepochto the most recentconvergence_set.
-
save_sets_and_reset()[source] Resets all convergence variables
Tries to update
largest_convergence_windowandconvergence_setswhenconvergence_setis not an empty list.- Return type
-