4. Architecture¶
- COMPUTE
- computation
The definition & execution of networked operation is splitted in 1+2 phases:
… it is constrained by these IO data-structures:
… populates these low-level data-structures:
- network graph (COMPOSE time)
- execution dag (COMPILE time)
- execution steps (COMPILE time)
- solution (EXECUTE time)
… and utilizes these main classes:
graphtik.op.FunctionalOperation
graphtik.netop.NetworkOperation
graphtik.network.Network
graphtik.network.ExecutionPlan
graphtik.network.Solution
- compose
- COMPOSITION
The phase where operations are constructed and grouped into netops and corresponding networks.
Tip
- Use
operation()
builder class to constructFunctionalOperation
instances. - Use
compose()
factory to prepare the net internally, and buildNetworkOperation
instances.
- Use
- compile
- COMPILATION
- The phase where the
Network
creates a new execution plan by pruning all graph nodes into a subgraph dag, and derriving the execution steps. - execute
- EXECUTION
- sequential
- EXECUTION
The phase where the
ExecutionPlan
calls the underlying functions of all operations contained in execution steps, with inputs/outputs taken from the solution.Currently there are 2 ways to execute:
- sequential
- parallel, with a
multiprocessing.ProcessPool
Plans may abort their execution by setting the abort run global flag.
- parallel
- parallel execution
- execution pool
- task
- parallel execution
Execute operations in parallel, with a thread pool or process pool (instead of sequential). Operations and netop are marked as such on construction, or enabled globally from configurations.
Note that sideffects are not expected to function with process pools, certainly not when marshalling is enabled.
- process pool
When the
multiprocessing.Pool()
class is used for parallel execution, the tasks must be communicated to/from the worker process, which requires pickling, and that may fail. With pickling failures you may try marshalling with dill library, and see if that helps.Note that sideffects are not expected to function at all. certainly not when marshalling is enabled.
- thread pool
- When the
multiprocessing.dummy.Pool()
class for parallel execution, the tasks are run in process, so no marshalling is needed. - marshalling
Pickling parallel operations and their inputs/outputs using the
dill
module. It is configured either globally withset_marshal_tasks()
or set with a flag on each operation / netop.Note that sideffects do not work when this is enabled.
- configurations
The functions controlling compile & execution globally are defined in
config
module; they undelying global data are stored incontextvars.ContextVar
instances, to allow for nested control.All boolean configuration flags are tri-state (
None, False, True
), allowing to “force” all operations, when they are not set to theNone
value. All of them default toNone
(false).- graph
- network graph
The
Network.graph
(currently a DAG) contains allFunctionalOperation
and_DataNode
nodes of some netop.They are layed out and connected by repeated calls of
Network._append_operation()
by Network constructor.This graph is then pruned to extract the dag, and the execution steps are calculated, all ingridents for a new
ExecutionPlan
.- dag
- execution dag
- solution dag
- execution dag
There are 2 directed-acyclic-graphs instances used:
- the
ExecutionPlan.dag
, in the execution plan, which contains the pruned nodes, used to decide the execution steps; - the
Solution.dag
in the solution, which derives the canceled operations due to rescheduled/failed operations upstreams.
- the
- steps
- execution steps
The
ExecutionPlan.steps
contains a list of the operation-nodes only from the dag, topologically sorted, and interspersed with instruction steps needed to compute the asked outputs from the given inputs.It is built by
Network._build_execution_steps()
based on the subgraph dag.The only instruction step is for performing evictions.
- evictions
- The
_EvictInstruction
steps erase items from solution as soon as they are not needed further down the dag, to reduce memory footprint while computing. - solution
A
Solution
instance created internally byNetworkOperation.compute()
to hold the values both inputs & outputs, and the status of executed operations. It is based on acollections.ChainMap
, to keep one dictionary for each operation executed +1 for inputs.The results of the last operation executed “wins” in the final outputs produced, BUT while executing, the needs of each operation receive the solution values in reversed order, that is, the 1st operation result (or given input) wins for some needs name.
Rational:
During execution we want stability (the same input value used by all operations), and that is most important when consuming input values - otherwise, we would use (possibly overwritten and thus changing)) intermediate ones.
But at the end we want to affect the calculation results by adding operations into some netop - furthermore, it wouldn’t be very usefull to get back the given inputs in case of overwrites.
- overwrites
- Values in the solution that have been written by more than one operations,
accessed by
Solution.overwrites
: - net
- network
- the
Network
contains a graph of operations and can compile an execution plan or prune a cloned network for given inputs/outputs/node predicate. - plan
- execution plan
Class
ExecutionPlan
perform the execution phase which contains the dag and the steps.Compileed execution plans are cached in
Network._cached_plans
across runs with (inputs, outputs, predicate) as key.- inputs
- a dictionary of named input values given to a single operation, or to
a netop, fed into
Operation.compute()
method. - outputs
A dictionary of computed values returned by a single operation or a netop when method
Operation.compute()
is called, or the actual (partial or complete) provides returned by someFunctionalOperation
.All computed values are retained in it when no specific outputs requested, to
NetworkOperation.compute()
, that is, no evictions happens.A function may return partial outputs.
- returns dictionary
- When an operation is marked with this flag, the underlying function is not expected to treturn a sequence but a dictionary; hence, no “zipping” of outputs/provides takes place.
- operation
- Either the abstract notion of an action with specified needs and provides,
or the concrete wraper
FunctionalOperation
for arbitrary functions (anycallable
). - netop
- network operation
- The
NetworkOperation
class holding a network of operations. - needs
- A list of names of the compulsory/optional values or sideffects an operation’s underlying callable requires to execute.
- provides
- A list of names of the values produced when the operation’s underlying callable executes.
- sideffects
- Fictive needs or provides not consumed/produced by the underlying function
of an operation, annotated with
sideffect
. A sideffect participates in the solution of the graph but is never given/asked to/from functions. - prune
- pruning
A subphase of compilation performed by method
Network._prune_graph()
, which extracts a subgraph dag that does not contain any unsatisfied operations.It topologically sorts the graph, and prunes based on given inputs, asked outputs, node predicate and operation needs & provides.
- unsatisfied operation
The core of pruning & rescheduling, performed by
network._unsatisfied_operations()
function, which collects all operations that fall into any of these 2 cases:- reschedule
- rescheduling
- partial outputs
- partial operation
- canceled operation
- rescheduling
The partial pruning of the solution’s dag during execution. It happens when any of these 2 conditions apply:
- an operation is marked with the
FunctionalOperation.rescheduled
attribute, which means that its underlying callable may produce only a subset of its provides (partial outputs); - endurance is enabled, either globally (in the configurations), or for a specific operation.
the solution must then reschedule the remaining operations downstreams, and possibly cancel some of those ( assigned in
Solution.canceled
).- an operation is marked with the
- endurance
Keep executing as many operations as possible, even if some of them fail. Endurance for an operation is enabled if
set_endure_operations()
is true globally in the configurations or ifFunctionalOperation.endurance
is true.You may interrogate
Solution.executed
to discover the status of each executed operations or callscream_if_incomplete()
.- predicate
- node predicate
- A callable(op, node-data) that should return true for nodes to be included in graph during compilation.
- abort run
A global configurations flag that when set with
abort_run()
function, it halts the execution of all currently or future plans.It is reset automatically on every call of
NetworkOperation.compute()
(after a succesfull intermediate compilation), or manually, by callingreset_abort()
.