4. Architecture

COMPUTE
computation

%3 graphtik-v4.1.0 flowchart cluster_compute compute operations operations compose compose operations->compose network network compose->network compile compile network->compile inputs input names inputs->compile outputs output names outputs->compile predicate node predicate predicate->compile plan execution plan compile->plan execute execute plan->execute solution solution execute->solution values input values values->execute The definition & execution of networked operation is splitted in 1+2 phases:

… it is constrained by these IO data-structures:

… populates these low-level data-structures:

… and utilizes these main classes:

graphtik.op.FunctionalOperation
graphtik.netop.NetworkOperation
graphtik.network.Network
graphtik.network.ExecutionPlan
graphtik.network.Solution
compose
COMPOSITION

The phase where operations are constructed and grouped into netops and corresponding networks.

Tip

compile
COMPILATION
The phase where the Network creates a new execution plan by pruning all graph nodes into a subgraph dag, and derriving the execution steps.
execute
EXECUTION
sequential

The phase where the ExecutionPlan calls the underlying functions of all operations contained in execution steps, with inputs/outputs taken from the solution.

Currently there are 2 ways to execute:

  • sequential
  • parallel, with a multiprocessing.ProcessPool
parallel
parallel execution
process pool
Execute operation in parallel, with a thread/process pool (instead of sequential). When a process pool is used, data & operations must be *pickled* to/from the worker process, and that may fail. You may marshal them with dill library to fix them.
marshal
Pickling parallel operations and their inputs/outputs using the dill module.
configurations

A global _execution_configs affecting execution stored in a contextvars.ContextVar.

Tip

Instead of directly modifying _execution_configs, prefer the special set_...() & is_...() methods exposed from the graptik package:

graph
network graph

The Network.graph (currently a DAG) contains all FunctionalOperation and _DataNode nodes of some netop.

They are layed out and connected by repeated calls of Network._append_operation() by Network constructor.

This graph is then pruned to extract the dag, and the execution steps are calculated, all ingridents for a new ExecutionPlan.

dag
execution dag

There are 2 directed-acyclic-graphs instances used:

steps
execution steps

The ExecutionPlan.steps contains a list of the operation-nodes only from the dag, topologically sorted, and interspersed with instruction steps needed to compute the asked outputs from the given inputs.

It is built by Network._build_execution_steps() based on the subgraph dag.

The only instruction step is for performing eviction.

evict
eviction
The _EvictInstruction steps erase items from solution as soon as they are not needed further down the dag, to reduce memory footprint while computing.
solution

A Solution instance created internally by NetworkOperation.compute() to hold the values both inputs & outputs, and the status of executed operations. It is based on a collections.ChainMap, to keep one dictionary for each operation executed +1 for inputs.

The results of the last operation executed “wins” in the final outputs produced, BUT while executing, the needs of each operation receive the solution values in reversed order, that is, the 1st operation result (or given input) wins for some needs name.

Rational:

During execution we want stability (the same input value used by all operations), and that is most important when consuming input values - otherwise, we would use (possibly overwritten and thus changing)) intermediate ones.

But at the end we want to affect the calculation results by adding operations into some netop - furthermore, it wouldn’t be very usefull to get back the given inputs in case of overwrites.

overwrites
Values in the solution that have been written by more than one operations, accessed by Solution.overwrites:
net
network
the Network contains a graph of operations and can compile an execution plan or prune a cloned network for given inputs/outputs/node predicate.
plan
execution plan

Class ExecutionPlan perform the execution phase which contains the dag and the steps.

Compileed execution plans are cached in Network._cached_plans across runs with (inputs, outputs, predicate) as key.

inputs
a dictionary of named input values given to a single operation, or to a netop, fed into Operation.compute() method.
outputs

A dictionary of computed values returned by a single operation or a netop when method Operation.compute() is called, or the actual (partial or complete) provides returned by some FunctionalOperation.

All computed values are retained in it when no specific outputs requested, to NetworkOperation.compute(), that is, no data-eviction happens.

operation
Either the abstract notion of an action with specified needs and provides, or the concrete wraper FunctionalOperation for arbitrary functions (any callable).
netop
network operation
The NetworkOperation class holding a network of operations.
needs
A list of names of the compulsory/optional values or sideffects an operation’s underlying callable requires to execute.
provides
A list of names of the values produced when the operation’s underlying callable executes.
sideffects
Fictive needs or provides not consumed/produced by the underlying function of an operation, annotated with sideffect. A sideffect participates in the solution of the graph but is never given/asked to/from functions.
prune
pruning

A subphase of compilation performed by method Network._prune_graph(), which extracts a subgraph dag that does not contain any unsatisfied operations.

It topologically sorts the graph, and prunes based on given inputs, asked outputs, node predicate and operation needs & provides.

unsatisfied operation

The core of pruning & rescheduling, performed by method network._unsatisfied_operations(), which collects all operations that fall into any of these 2 cases:

  • they have needs that do not correspond to any of the given inputs or the intermediately computed outputs of the solution;
  • all threir provides are NOT needed by any other operation, nor are asked as outputs.
reschedule
rescheduling
partial outputs
canceled operation

The partial pruning of the solution’s dag during execution. It happens when any of these 2 conditions apply:

  • an operation is marked with the FunctionalOperation.reschedule attribute, which means that its underlying callable may produce only a subset of its provides (partial outputs);
  • endurance is enabled, either globally (in the configurations), or for a specific operation.

the solution must then reschedule the remaining operations downstreams, and possibly cancel some of those ( assigned in Solution.canceled).

endurance

Keep executing as many operations as possible, even if some of them fail. Endurance for an operation is enabled if set_endure_execution() is true globally in the configurations or if FunctionalOperation.endurance is true.

You may interogate Solution operations to discover whether they have been:

predicate
node predicate
A callable(op, node-data) that should return true for nodes to be included in graph during compilation.