hazimp.parallel

Functions that haven’t found a proper module.

Module Contents

Classes

Parallel

Parallelise to run on a cluster.

Functions

scatter_dict(whole)

Broadcast and recieve a dictionary where the values are 1d arrays

gather_dict(subdict, indexes)

Recieve a dictionary from the children where the values are 1d arrays

csv2dict(filename, use_parallel=True)

Read a csv file in and return the information as a dictionary

Attributes

STATE

class hazimp.parallel.Parallel

Bases: object

Parallelise to run on a cluster.

Parameters:

  • rank – What is the id of this node in the cluster.

  • size – How many processors are there in the cluster.

  • node – name of the cluster node.

  • is_parallel – True if parallel is operational

  • file_tag – A string that can be added to files to identify who wrote the file.

_not_parallel(self)

Set the attributes if there is only one node.

hazimp.parallel.STATE
hazimp.parallel.scatter_dict(whole)

Broadcast and recieve a dictionary where the values are 1d arrays and the arrays are chunked for the workers. Only rank 0 needs the whole dictionary.

Parameters:

whole – The dictionary of 1d arrays to subdict.

Returns:

(chunk of dictionary of 1d arrays, indexes of whole array)

hazimp.parallel.gather_dict(subdict, indexes)

Recieve a dictionary from the children where the values are 1d arrays and the arrays are chunks of the whole dictionary.

Parameters:

  • indexes – The indexes into the whole array.

  • subdict – The dictionary of 1d arrays to subset.

Returns:

whole array

hazimp.parallel.csv2dict(filename, use_parallel=True)

Read a csv file in and return the information as a dictionary where the key is the column names and the values are column arrays.

This dictionary will be chunked and sent to all processors.

Parameters:

filename – The csv file path string.

Returns:

subsection of the array