5. API Reference¶

`graphtik`	Lightweight computation graphs for Python.
`graphtik.op`	About operation nodes (but not net-ops to break cycle).
`graphtik.modifiers`	A modifier change dependency behavior during compilation or execution.
`graphtik.netop`	About network operations (those based on graphs)
`graphtik.network`	compile & execute network graphs of operations.
`graphtik.plot`	Plotting of graph graphs handled by active plotter (see also `base`).
`graphtik.config`	configurations for network execution, and utilities on them.
`graphtik.base`	Generic utilities, or `plot` API but without polluting imports.
`graphtik.sphinxext`	Extends Sphinx with `graphtik` directive rendering plots from doctest code.

Module: op¶

About operation nodes (but not net-ops to break cycle).

class graphtik.op.FunctionalOperation(fn: Callable = None, name=None, needs: Union[Collection, str, None] = None, provides: Union[Collection, str, None] = None, aliases: Mapping = None, *, parents: Tuple = None, rescheduled=None, endured=None, parallel=None, marshalled=None, returns_dict=None, node_props: Mapping = None)[source]¶

An operation performing a callable (ie a function, a method, a lambda).

Tip

Use operation() factory to build instances of this class instead.
Call withset() on existing instances to re-configure new clones.

__abstractmethods__ = frozenset({})[source]¶

__call__(*args, **kwargs)[source]¶: Like dict args, delegates to compute().

__eq__(other)[source]¶: Operation identity is based on name and parents.

__hash__()[source]¶: Operation identity is based on name and parents.

__init__(fn: Callable = None, name=None, needs: Union[Collection, str, None] = None, provides: Union[Collection, str, None] = None, aliases: Mapping = None, *, parents: Tuple = None, rescheduled=None, endured=None, parallel=None, marshalled=None, returns_dict=None, node_props: Mapping = None)[source]¶

Build a new operation out of some function and its requirements.

See operation() for the full documentation of parameters, study the code for attributes (or read them from rendered sphinx site).

__module__ = 'graphtik.op'¶

__repr__()[source]¶: Display more informative names for the Operation class

_abc_impl = <_abc_data object>¶

_fn_needs = None[source]¶: Value names the underlying function requires (dupes preserved, without sideffects, with stripped sideffected dependencies).

_fn_provides = None[source]¶: Value names the underlying function produces (dupes preserved, without aliases & sideffects, with stripped sideffected dependencies).

_match_inputs_with_fn_needs(named_inputs) → Tuple[list, list, dict][source]¶

_prepare_match_inputs_error(exceptions: List[Tuple[Any, Exception]], missing: List, varargs_bad: List, named_inputs: Mapping) → ValueError [source]¶

_zip_results_with_provides(results) → dict [source]¶: Zip results with expected “real” (without sideffects) provides.

aliases = None[source]¶

an optional mapping of fn_provides to additional ones, together comprising this operations op_provides.

You cannot alias an alias.

compute(named_inputs=None, outputs: Union[Collection, str, None] = None) → dict [source]¶

Compute (optional) asked outputs for the given named_inputs.

It is called by Network. End-users should simply call the operation with named_inputs as kwargs.

Parameters: named_inputs – the input values with which to feed the computation.
Returns list: Should return a list values representing the results of running the feed-forward computation on inputs.

property deps¶

All dependency names, including op_ & internal _fn_.

if not DEBUG, all deps are converted into lists, ready to be printed.

endured = None[source]¶: If true, even if callable fails, solution will reschedule; ignored if endurance enabled globally.

fn = None[source]¶: The operation’s underlying function.

marshalled = None[source]¶: If true, operation will be marshalled while computed, along with its inputs & outputs. (usefull when run in parallel with a process pool).

name = None[source]¶: a name for the operation (e.g. ‘conv1’, ‘sum’, etc..); it will be prefixed by parents.

needs = None[source]¶: The needs almost as given by the user (which may contain MULTI-sideffecteds and dupes), roughly morphed into _fn_provides + sideffects (dupes preserved, with sideffects & SINGULARIZED sideffecteds). It is stored for builder functionality to work.

node_props = None[source]¶: Added as-is into NetworkX graph, and you may filter operations by NetworkOperation.withset(). Also plot-rendering affected if they match Graphviz properties, unless they start with underscore(_).

op_needs = None[source]¶: Value names ready to lay the graph for pruning (NO dupes, WITH aliases & sideffects, and SINGULAR sideffecteds).

op_provides = None[source]¶: Value names ready to lay the graph for pruning (NO dupes, WITH aliases & sideffects, and SINGULAR sideffecteds).

parallel = None[source]¶: execute in parallel

parents = None[source]¶: a tuple wth the names of the parents, prefixing name, but also kept for equality/hash check.

prepare_plot_args(plot_args: graphtik.base.PlotArgs) → graphtik.base.PlotArgs [source]¶: Delegate to a provisional network with a single op .

provides = None[source]¶: The provides almost as given by the user (which may contain MULTI-sideffecteds and dupes), roughly morphed into _fn_provides + sideffects (dupes preserved, without aliases, with sideffects & SINGULARIZED sideffecteds). It is stored for builder functionality to work.

rescheduled = None[source]¶: If true, underlying callable may produce a subset of provides, and the plan must then reschedule after the operation has executed. In that case, it makes more sense for the callable to returns_dict.

returns_dict = None[source]¶

If true, it means the underlying function returns dictionary , and no further processing is done on its results, i.e. the returned output-values are not zipped with provides.

It does not have to return any alias outputs.

Can be changed amidst execution by the operation’s function, but it is easier for that function to simply call set_results_by_name().

withset(fn: Callable = None, **kw) → graphtik.op.FunctionalOperation [source]¶: Make a clone with the some values replaced.

class graphtik.op.Operation[source]¶

An abstract class representing an action with compute().

__abstractmethods__ = frozenset({'compute'})¶

__dict__ = mappingproxy({'__module__': 'graphtik.op', '__doc__': 'An abstract class representing an action with :meth:`.compute()`.', '__name__': <property object>, 'compute': <function Operation.compute>, '__dict__': <attribute '__dict__' of 'Operation' objects>, '__weakref__': <attribute '__weakref__' of 'Operation' objects>, '__abstractmethods__': frozenset({'compute'}), '_abc_impl': <_abc_data object>})¶

__module__ = 'graphtik.op'¶

__name__ = 'Operation'¶

__weakref__¶: list of weak references to the object (if defined)

_abc_impl = <_abc_data object>¶

abstract compute(named_inputs, outputs=None)[source]¶

Compute (optional) asked outputs for the given named_inputs.

It is called by Network. End-users should simply call the operation with named_inputs as kwargs.

Parameters: named_inputs – the input values with which to feed the computation.
Returns list: Should return a list values representing the results of running the feed-forward computation on inputs.

graphtik.op._dict_without(kw, *to_skip)[source]¶

graphtik.op._spread_sideffects(deps: collections.abc.Collection) → Tuple[collections.abc.Collection, collections.abc.Collection][source]¶

Build fn/op dependencies from user ones by stripping or singularizing any sideffects.

Returns

the given deps duplicated as (fn_deps, op_deps), where any instances of sideffect() and sideffected() are processed like this:

fn_deps

sideffected() are replaced by the pure _Modifier.sideffected dependency consumed/produced by underlying functions, in the order it is first met (the rest duplicate sideffected are discarded).
sideffect() are simply dropped;

op_deps

sideffected() are replaced by a sequence of “singular” sideffected instances, one for each item in their _Modifier.sideffects attribute, in the order they are first met (any duplicates are discarded, order is irrelevant, since they don’t reach the function);

graphtik.op.as_renames(i, argname)[source]¶

Parses a list of (source–>destination) from dict, list-of-2-items, single 2-tuple.

Returns: a (possibly empty)list-of-pairs

Note

The same source may be repeatedly renamed to multiple destinations.

graphtik.op.operation(fn: Callable = None, name=None, needs: Union[Collection, str, None] = None, provides: Union[Collection, str, None] = None, aliases: Mapping = None, *, rescheduled=None, endured=None, parallel=None, marshalled=None, returns_dict=None, node_props: Mapping = None)[source]¶

An operation factory that can function as a decorator.

Parameters

fn –
The callable underlying this operation. If given, it builds the operation right away (along with any other arguments).

If not given, it returns a “fancy decorator” that still supports all arguments here AND the withset() method.

Hint

This is a twisted way for “fancy decorators”.

After all that, you can always call FunctionalOperation.withset() on existing operation, to obtain a re-configured clone.
name (str) – The name of the operation in the computation graph. If not given, deduce from any fn given.
needs –
the list of (positionally ordered) names of the data needed by the operation to receive as inputs, roughly corresponding to the arguments of the underlying fn (plus any sideffects).

It can be a single string, in which case a 1-element iterable is assumed.
See also
provides –
the list of (positionally ordered) output data this operation provides, which must, roughly, correspond to the returned values of the fn (plus any sideffects & aliases).

It can be a single string, in which case a 1-element iterable is assumed.

If they are more than one, the underlying function must return an iterable with same number of elements, unless param returns_dict is true, in which case must return a dictionary that containing (at least) those named elements.
See also
aliases – an optional mapping of provides to additional ones
rescheduled – If true, underlying callable may produce a subset of provides, and the plan must then reschedule after the operation has executed. In that case, it makes more sense for the callable to returns_dict.
endured – If true, even if callable fails, solution will reschedule. ignored if endurance enabled globally.
parallel – execute in parallel
marshalled – If true, operation will be marshalled while computed, along with its inputs & outputs. (usefull when run in parallel with a process pool).
returns_dict – if true, it means the fn returns dictionary with all provides, and no further processing is done on them (i.e. the returned output-values are not zipped with provides)
node_props – Added as-is into NetworkX graph, and you may filter operations by NetworkOperation.withset(). Also plot-rendering affected if they match Graphviz properties., unless they start with underscore(_)

Returns

when called with fn, it returns a FunctionalOperation, otherwise it returns a decorator function that accepts fn as the 1st argument.

Note

Actually the returned decorator is the FunctionalOperation.withset() method and accepts all arguments, monkeypatched to support calling a virtual withset() method on it, not to interrupt the builder-pattern, but only that - besides that trick, it is just a bound method.

Example:

This is an example of its use, based on the “builder pattern”:

>>> from graphtik import operation, varargs

>>> op = operation()
>>> op
<function FunctionalOperation.withset at ...

That’s a “fancy decorator”.

>>> op = op.withset(needs=['a', 'b'])
>>> op
FunctionalOperation(name=None, needs=['a', 'b'], provides=[], fn=None)

If you call an operation with fn un-initialized, it will scream:

>>> op.compute({"a":1, "b": 2})
Traceback (most recent call last):
ValueError: Operation was not yet provided with a callable `fn`!

You may keep calling withset() until a valid operation instance is returned, and compute it:

>>> op = op.withset(needs=['a', 'b'],
...                 provides='SUM', fn=lambda a, b: a + b)
>>> op
FunctionalOperation(name='<lambda>', needs=['a', 'b'], provides=['SUM'], fn='<lambda>')
>>> op.compute({"a":1, "b": 2})
{'SUM': 3}

>>> op.withset(fn=lambda a, b: a * b).compute({'a': 2, 'b': 5})
{'SUM': 10}

graphtik.op.reparse_operation_data(name, needs, provides) → Tuple[collections.abc.Hashable, collections.abc.Collection, collections.abc.Collection][source]¶

Validate & reparse operation data as lists.

Returns: name, needs, provides,

As a separate function to be reused by client building operations, to detect errors early.

Module: modifiers¶

A modifier change dependency behavior during compilation or execution.

The needs and provides annotated with modifiers designate, for instance, optional function arguments, or “ghost” sideffects.

class graphtik.modifiers._Modifier[source]¶

Annotate a dependency with a combination of modifier.

Private, in the sense that users should use the factory functions mapped(), optional() etc, and is_optional() predicates.

_repr¶: pre-calculated representation

fn_kwarg¶: Map my name in needs into this kw-argument of the function.

optional¶: required is None, regular optional or varargish?

sideffected¶: An existing dependency in solution that sustain (must have sustained) the sideffects by(for) the underlying function.

sideffects¶: At least one name(s) denoting the sideffects modification(s) on the sideffected, performed/required by the operation. If it is an empty tuple`, it is an abstract sideffect.

withset(**kw)[source]¶

Make a new modifier with kwargs: name(or sideffected), fn_kwarg, optional, sideffects

Parameters: optional – either a bool or an _Optionals enum, as taken from optional from another modifier instance

class graphtik.modifiers._Optionals[source]¶

An enumeration.

property varargish¶: True if the optionals is a varargish.

graphtik.modifiers.is_mapped(dep) → Optional[str][source]¶: Check if a dependency is mapped (and get it).

graphtik.modifiers.is_optional(dep) → bool [source]¶: Check (and get) if a dependency is optional (varargish/sideffects included).

graphtik.modifiers.is_pure_sideffect(dep) → bool [source]¶: Check if it is sideffects but not a sideffected.

graphtik.modifiers.is_sideffect(dep) → bool [source]¶: Check if a dependency is sideffects or sideffected.

graphtik.modifiers.is_sideffected(dep) → bool [source]¶: Check if it is sideffected (and get it).

graphtik.modifiers.is_vararg(dep) → bool [source]¶: Check if an optionals dependency is vararg.

graphtik.modifiers.is_varargish(dep) → bool [source]¶: Check if an optionals dependency is varargish.

graphtik.modifiers.is_varargs(dep) → bool [source]¶: Check if an optionals dependency is varargs.

graphtik.modifiers.mapped(name: str, fn_kwarg: str)[source]¶

Annotate a needs that (optionally) map inputs name –> argument-name.

Parameters

fn_kwarg –

The argument-name corresponding to this named-input. If not given, a regular string is returned.

Note

This extra mapping argument is needed either for optionals (but not varargish), or for functions with keywords-only arguments (like def func(*, foo, bar): ...), since inputs are normally fed into functions by-position, not by-name.

Example:

In case the name of the function arguments is different from the name in the inputs (or just because the name in the inputs is not a valid argument-name), you may map it with the 2nd argument of mapped():

>>> from graphtik import operation, compose, mapped

>>> @operation(needs=['a', mapped("name-in-inputs", "b")], provides="sum")
... def myadd(a, *, b):
...    return a + b
>>> myadd
FunctionalOperation(name='myadd',
                    needs=['a', mapped('name-in-inputs'-->'b')],
                    provides=['sum'],
                    fn='myadd')

>>> graph = compose('mygraph', myadd)
>>> graph
NetworkOperation('mygraph', needs=['a', 'name-in-inputs'], provides=['sum'], x1 ops: myadd)

>>> sol = graph.compute({"a": 5, "name-in-inputs": 4})['sum']
>>> sol
9

graphtik.modifiers.optional(name: str, fn_kwarg: str = None)[source]¶

Annotate optionals needs corresponding to defaulted op-function arguments, …

received only if present in the inputs (when operation is invoked). The value of an optional is passed as a keyword argument to the underlying function.

Example:

>>> from graphtik import operation, compose, optional

>>> @operation(name='myadd',
...            needs=["a", optional("b")],
...            provides="sum")
... def myadd(a, b=0):
...    return a + b

Notice the default value 0 to the b annotated as optional argument:

>>> graph = compose('mygraph', myadd)
>>> graph
NetworkOperation('mygraph',
                 needs=['a', optional('b')],
                 provides=['sum'],
                 x1 ops: myadd)

The graph works both with and without c provided in the inputs:

>>> graph(a=5, b=4)['sum']
9
>>> graph(a=5)
{'a': 5, 'sum': 5}

Like mapped() you may map input-name to a different function-argument:

>>> operation(needs=['a', optional("quasi-real", "b")],
...           provides="sum"
... )(myadd.fn)  # Cannot wrap an operation, its `fn` only.
FunctionalOperation(name='myadd',
                    needs=['a', optional('quasi-real'-->'b')],
                    provides=['sum'],
                    fn='myadd')

graphtik.modifiers.sideffect(name, optional: bool = None)[source]¶

sideffects denoting modifications beyond the scope of the solution.

Both needs & provides may be designated as sideffects using this modifier. They work as usual while solving the graph (compilation) but they have a limited interaction with the operation’s underlying function; specifically:

input sideffects must exist in the solution as inputs for an operation depending on it to kick-in, when the computation starts - but this is not necessary for intermediate sideffects in the solution during execution;
input sideffects are NOT fed into underlying functions;
output sideffects are not expected from underlying functions, unless a rescheduled operation with partial outputs designates a sideffected as canceled by returning it with a falsy value (operation must returns dictionary).

Hint

If modifications involve some input/output, prefer the sideffected() modifier.

You may still convey this relationships by including the dependency name in the string - in the end, it’s just a string - but no enforcement of any kind will happen from graphtik, like:

>>> from graphtik import sideffect

>>> sideffect("price[sales_df]")
sideffect: 'price[sales_df]'

Example:

A typical use-case is to signify changes in some “global” context, outside solution:

>>> from graphtik import operation, compose, sideffect

>>> @operation(provides=sideffect("lights off"))  # sideffect names can be anything
... def close_the_lights():
...    pass

>>> graph = compose('strip ease',
...     close_the_lights,
...     operation(
...         name='undress',
...         needs=[sideffect("lights off")],
...         provides="body")(lambda: "TaDa!")
... )
>>> graph
NetworkOperation('strip ease', needs=[sideffect: 'lights off'],
                 provides=[sideffect: 'lights off', 'body'],
                 x2 ops: close_the_lights, undress)

>>> sol = graph()
>>> sol
{'body': 'TaDa!'}

sideffect¶

Note

Something has to provide a sideffect for a function needing it to execute - this could be another operation, like above, or the user-inputs; just specify some dummy value for the sideffect:

>>> sol = graph.compute({sideffect("lights off"): True})

graphtik.modifiers.sideffected(dependency: str, sideffect0: str, *sideffects: str, optional: bool = None, fn_kwarg: str = None)[source]¶

Annotates a sideffected dependency in the solution sustaining side-effects.

Like sideffect() but annotating a real dependency in the solution, allowing that dependency to be present both in needs and provides of the same function.

Example:

A typical use-case is to signify columns required to produce new ones in pandas dataframes (emulated with dictionaries):

>>> from graphtik import operation, compose, sideffected

>>> @operation(needs="order_items",
...            provides=sideffected("ORDER", "Items", "Prices"))
... def new_order(items: list) -> "pd.DataFrame":
...     order = {"items": items}
...     # Pretend we get the prices from sales.
...     order['prices'] = list(range(1, len(order['items']) + 1))
...     return order

>>> @operation(
...     needs=[sideffected("ORDER", "Items"), "vat rate"],
...     provides=sideffected("ORDER", "VAT")
... )
... def fill_in_vat(order: "pd.DataFrame", vat: float):
...     order['VAT'] = [i * vat for i in order['prices']]
...     return order

>>> @operation(
...     needs=[sideffected("ORDER", "Prices", "VAT")],
...     provides=sideffected("ORDER", "Totals")
... )
... def finalize_prices(order: "pd.DataFrame"):
...     order['totals'] = [p + v for p, v in zip(order['prices'], order['VAT'])]
...     return order

To view all internal dependencies, enable DEBUG in configurations:

>>> from graphtik import debug_enabled

>>> with debug_enabled(True):
...     finalize_prices
FunctionalOperation(name='finalize_prices',
                    needs=[sideffected('ORDER'<--'Prices'),
                           sideffected('ORDER'<--'VAT')],
                    op_needs=[sideffected('ORDER'<--'Prices'),
                              sideffected('ORDER'<--'VAT')],
                    fn_needs=['ORDER'],
                    provides=[sideffected('ORDER'<--'Totals')],
                    op_provides=[sideffected('ORDER'<--'Totals')],
                    fn_provides=['ORDER'], fn='finalize_prices')

Notice that declaring a single sideffected with multiple sideffects, expands into multiple “singular” sideffected dependencies in the network (check needs & op_needs above).

>>> proc_order = compose('process order', new_order, fill_in_vat, finalize_prices)
>>> sol = proc_order.compute({
...      "order_items": ["toilet-paper", "soap"],
...      "vat rate": 0.18,
... })
>>> sol
{'order_items': ['toilet-paper', 'soap'],
 'vat rate': 0.18,
 'ORDER': {'items': ['toilet-paper', 'soap'],
           'prices': [1, 2],
           'VAT': [0.18, 0.36],
           'totals': [1.18, 2.36]}}

sideffecteds¶

Notice that although many functions consume & produce the same ORDER dependency (check fn_needs & fn_provides, above), something that would have formed cycles, the wrapping operations need and provide different sideffected instances, breaking the cycles.

graphtik.modifiers.vararg(name: str)[source]¶

Annotate a varargish needs to be fed as function’s *args.

Module: netop¶

About network operations (those based on graphs)

class graphtik.netop.NetworkOperation(operations, name, *, outputs=None, predicate: Callable[[Any, Mapping], bool] = None, rescheduled=None, endured=None, parallel=None, marshalled=None, merge=None, node_props=None)[source]¶

An operation that can compute a network-graph of operations.

Tip

Use compose() factory to prepare the net and build instances of this class.

compile(inputs=None, outputs=<UNSET>, predicate: Callable[[Any, Mapping], bool] = <UNSET>) → graphtik.network.ExecutionPlan [source]¶

Produce a plan for the given args or outputs/predicate narrowed earlier.

Parameters

named_inputs – a string or a list of strings that should be fed to the needs of all operations.
outputs – A string or a list of strings with all data asked to compute. If None, all possible intermediate outputs will be kept. If not given, those set by a previous call to withset() or cstor are used.
predicate – Will be stored and applied on the next compute() or compile(). If not given, those set by a previous call to withset() or cstor are used.

Returns

the execution plan satisfying the given inputs, outputs & predicate

Raises

ValueError –

If outputs asked do not exist in network, with msg:

Unknown output nodes: …
If solution does not contain any operations, with msg:

Unsolvable graph: …
If given inputs mismatched plan’s needs, with msg:

Plan needs more inputs…
If net cannot produce asked outputs, with msg:

Unreachable outputs…

compute(named_inputs: Mapping = <UNSET>, outputs: Union[Collection, str, None] = <UNSET>, predicate: Callable[[Any, Mapping], bool] = <UNSET>) → graphtik.network.Solution [source]¶

Compile a plan & execute the graph, sequentially or parallel.

Attention

If intermediate compilation is successful, the “global abort run flag is reset before the execution starts.

Parameters

named_inputs – A mapping of names –> values that will be fed to the needs of all operations. Cloned, not modified.
outputs – A string or a list of strings with all data asked to compute. If None, all intermediate data will be kept.

Returns

The solution which contains the results of each operation executed +1 for inputs in separate dictionaries.

Raises

ValueError –

If outputs asked do not exist in network, with msg:

Unknown output nodes: …
If plan does not contain any operations, with msg:

Unsolvable graph: …
If given inputs mismatched plan’s needs, with msg:

Plan needs more inputs…
If net cannot produce asked outputs, with msg:

Unreachable outputs…

Module: network¶

compile & execute network graphs of operations.

exception graphtik.network.AbortedException[source]¶

Raised from Network when abort_run() is called, and contains the solution …

with any values populated so far.

__module__ = 'graphtik.network'¶

__weakref__¶: list of weak references to the object (if defined)

class graphtik.network.ExecutionPlan[source]¶

A pre-compiled list of operation steps that can execute for the given inputs/outputs.

It is the result of the network’s compilation phase.

Note the execution plan’s attributes are on purpose immutable tuples.

net¶: The parent Network

needs¶: An IndexedSet with the input names needed to exist in order to produce all provides.

provides¶: An IndexedSet with the outputs names produces when all inputs are given.

dag¶: The regular (not broken) pruned subgraph of net-graph.

steps¶: The tuple of operation-nodes & instructions needed to evaluate the given inputs & asked outputs, free memory and avoid overwriting any given intermediate inputs.

asked_outs¶: When true, evictions may kick in (unless disabled by configurations), otherwise, evictions (along with prefect-evictions check) are skipped.

__abstractmethods__ = frozenset({})[source]¶

__dict__ = mappingproxy({'__module__': 'graphtik.network', '__doc__': "\n A pre-compiled list of operation steps that can :term:`execute` for the given inputs/outputs.\n\n It is the result of the network's :term:`compilation` phase.\n\n Note the execution plan's attributes are on purpose immutable tuples.\n\n .. attribute:: net\n\n The parent :class:`Network`\n .. attribute:: needs\n\n An :class:`.IndexedSet` with the input names needed to exist in order to produce all `provides`.\n .. attribute:: provides\n\n An :class:`.IndexedSet` with the outputs names produces when all `inputs` are given.\n .. attribute:: dag\n\n The regular (not broken) *pruned* subgraph of net-graph.\n .. attribute:: steps\n\n The tuple of operation-nodes & *instructions* needed to evaluate\n the given inputs & asked outputs, free memory and avoid overwriting\n any given intermediate inputs.\n .. attribute:: asked_outs\n\n When true, :term:`evictions` may kick in (unless disabled by :term:`configurations`),\n otherwise, *evictions* (along with prefect-evictions check) are skipped.\n ", 'prepare_plot_args': <function ExecutionPlan.prepare_plot_args>, '__repr__': <function ExecutionPlan.__repr__>, 'validate': <function ExecutionPlan.validate>, '_check_if_aborted': <function ExecutionPlan._check_if_aborted>, '_prepare_tasks': <function ExecutionPlan._prepare_tasks>, '_handle_task': <function ExecutionPlan._handle_task>, '_execute_thread_pool_barrier_method': <function ExecutionPlan._execute_thread_pool_barrier_method>, '_execute_sequential_method': <function ExecutionPlan._execute_sequential_method>, 'execute': <function ExecutionPlan.execute>, '__dict__': <attribute '__dict__' of 'ExecutionPlan' objects>, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object>})¶

__module__ = 'graphtik.network'¶

__repr__()[source]¶: Return a nicely formatted representation string

_abc_impl = <_abc_data object>¶

_check_if_aborted(solution)[source]¶

_execute_sequential_method(solution: graphtik.network.Solution)[source]¶

This method runs the graph one operation at a time in a single thread

Parameters: solution – must contain the input values only, gets modified

_execute_thread_pool_barrier_method(solution: graphtik.network.Solution)[source]¶

This method runs the graph using a parallel pool of thread executors. You may achieve lower total latency if your graph is sufficiently sub divided into operations using this method.

Parameters: solution – must contain the input values only, gets modified

_handle_task(future, op, solution) → None [source]¶: Un-dill parallel task results (if marshalled), and update solution / handle failure.

_prepare_tasks(operations, solution, pool, global_parallel, global_marshal) → Union[Future, graphtik.network._OpTask, bytes][source]¶: Combine ops+inputs, apply marshalling, and submit to execution pool (or not) …

based on global/pre-op configs.

execute(named_inputs, outputs=None, *, name='') → graphtik.network.Solution [source]¶

Parameters

named_inputs – A mapping of names –> values that must contain at least the compulsory inputs that were specified when the plan was built (but cannot enforce that!). Cloned, not modified.
outputs – If not None, they are just checked if possible, based on provides, and scream if not.

Returns

The solution which contains the results of each operation executed +1 for inputs in separate dictionaries.

Raises

ValueError –

If plan does not contain any operations, with msg:

Unsolvable graph: …
If given inputs mismatched plan’s needs, with msg:

Plan needs more inputs…
If net cannot produce asked outputs, with msg:

Unreachable outputs…

prepare_plot_args(plot_args: graphtik.base.PlotArgs) → graphtik.base.PlotArgs [source]¶

Called by plot() to create the nx-graph and other plot-args, e.g. solution.

Clone the graph or merge it with the one in the plot_args (see PlotArgs.clone_or_merge_graph().
For the rest args, prefer PlotArgs.with_defaults() over _replace(), not to override user args.

validate(inputs: Union[Collection, str, None], outputs: Union[Collection, str, None])[source]¶

Scream on invalid inputs, outputs or no operations in graph.

Raises

ValueError –

If cannot produce any outputs from the given inputs, with msg:

Unsolvable graph: …
If given inputs mismatched plan’s needs, with msg:

Plan needs more inputs…
If net cannot produce asked outputs, with msg:

Unreachable outputs…

exception graphtik.network.IncompleteExecutionError[source]¶

Error report when any netop operations were canceled/failed.

The exception contains 3 arguments:

the causal errors and conditions (1st arg),
the list of collected exceptions (2nd arg), and
the solution instance (3rd argument), to interrogate for more.

Returned by check_if_incomplete() or raised by scream_if_incomplete().

__module__ = 'graphtik.network'¶

__str__()[source]¶: Return str(self).

__weakref__¶: list of weak references to the object (if defined)

class graphtik.network.Network(*operations, graph=None)[source]¶

A graph of operations that can compile an execution plan.

needs[source]¶: the “base”, all data-nodes that are not produced by some operation

provides[source]¶: the “base”, all data-nodes produced by some operation

__abstractmethods__ = frozenset({})[source]¶

__init__(*operations, graph=None)[source]¶

Parameters

operations – to be added in the graph
graph – if None, create a new.

Raises

ValueError –

if dupe operation, with msg:

Operations may only be added once, …

__module__ = 'graphtik.network'¶

__repr__()[source]¶: Return repr(self).

_abc_impl = <_abc_data object>¶

_append_operation(graph, operation: graphtik.op.Operation)[source]¶

Adds the given operation and its data requirements to the network graph.

Invoked during constructor only (immutability).
Identities are based on the name of the operation, the names of the operation’s needs, and the names of the data it provides.
Adds needs, operation & provides, in that order.

Parameters

graph – the networkx graph to append to
operation – operation instance to append

_apply_graph_predicate(graph, predicate)[source]¶

_build_execution_steps(pruned_dag, inputs: Collection, outputs: Optional[Collection]) → List[source]¶

Create the list of operation-nodes & instructions evaluating all

operations & instructions needed a) to free memory and b) avoid overwriting given intermediate inputs.

Parameters

pruned_dag – The original dag, pruned; not broken.
outputs – outp-names to decide whether to add (and which) evict-instructions

Instances of _EvictInstructions are inserted in steps between operation nodes to reduce the memory footprint of solutions while the computation is running. An evict-instruction is inserted whenever a need is not used by any other operation further down the DAG.

_cached_plans = None[source]¶: Speed up compile() call and avoid a multithreading issue(?) that is occurring when accessing the dag in networkx.

_prune_graph(inputs: Union[Collection, str, None], outputs: Union[Collection, str, None], predicate: Callable[[Any, Mapping], bool] = None) → Tuple[networkx.classes.digraph.DiGraph, Collection, Collection, Collection][source]¶

Determines what graph steps need to run to get to the requested outputs from the provided inputs: - Eliminate steps that are not on a path arriving to requested outputs; - Eliminate unsatisfied operations: partial inputs or no outputs needed; - consolidate the list of needs & provides.

Parameters

inputs – The names of all given inputs.
outputs – The desired output names. This can also be None, in which case the necessary steps are all graph nodes that are reachable from the provided inputs.
predicate – the node predicate is a 2-argument callable(op, node-data) that should return true for nodes to include; if None, all nodes included.

Returns

a 3-tuple with the pruned_dag & the needs/provides resolved based on the given inputs/outputs (which might be a subset of all needs/outputs of the returned graph).

Use the returned needs/provides to build a new plan.

Raises

ValueError –

if outputs asked do not exist in network, with msg:

Unknown output nodes: …

_topo_sort_nodes(dag) → List[source]¶: Topo-sort dag respecting operation-insertion order to break ties.

compile(inputs: Union[Collection, str, None] = None, outputs: Union[Collection, str, None] = None, predicate=None) → graphtik.network.ExecutionPlan [source]¶

Create or get from cache an execution-plan for the given inputs/outputs.

See _prune_graph() and _build_execution_steps() for detailed description.

Parameters

inputs – A collection with the names of all the given inputs. If None`, all inputs that lead to given outputs are assumed. If string, it is converted to a single-element collection.
outputs – A collection or the name of the output name(s). If None`, all reachable nodes from the given inputs are assumed. If string, it is converted to a single-element collection.
predicate – the node predicate is a 2-argument callable(op, node-data) that should return true for nodes to include; if None, all nodes included.

Returns

the cached or fresh new execution plan

Raises

ValueError –

If outputs asked do not exist in network, with msg:

Unknown output nodes: …
If solution does not contain any operations, with msg:

Unsolvable graph: …
If given inputs mismatched plan’s needs, with msg:

Plan needs more inputs…
If net cannot produce asked outputs, with msg:

Unreachable outputs…

find_op_by_name(name) → Optional[graphtik.op.Operation][source]¶: Fetch the 1st operation named with the given name.

find_ops(predicate) → List[graphtik.op.Operation][source]¶

Scan operation nodes and fetch those satisfying predicate.

Parameters: predicate – the node predicate is a 2-argument callable(op, node-data) that should return true for nodes to include.

graph = None[source]¶: The networkx (Di)Graph containing all operations and dependencies, prior to compilation.

prepare_plot_args(plot_args: graphtik.base.PlotArgs) → graphtik.base.PlotArgs [source]¶

Called by plot() to create the nx-graph and other plot-args, e.g. solution.

Clone the graph or merge it with the one in the plot_args (see PlotArgs.clone_or_merge_graph().
For the rest args, prefer PlotArgs.with_defaults() over _replace(), not to override user args.

class graphtik.network.Solution(plan, input_values: dict)[source]¶

A chain-map collecting solution outputs and execution state (eg overwrites)

__abstractmethods__ = frozenset({})[source]¶

__copy__()[source]¶: New ChainMap or subclass with a new copy of maps[0] and refs to maps[1:]

__delitem__(key)[source]¶

__init__(plan, input_values: dict)[source]¶: Initialize a ChainMap by setting maps to the given mappings. If no mappings are provided, a single empty dictionary is used.

__module__ = 'graphtik.network'¶

__repr__()[source]¶: Return repr(self).

_abc_impl = <_abc_data object>¶

_layers = None[source]¶: An ordered mapping of plan-operations to their results (initially empty dicts). The result dictionaries pre-populate this (self) chainmap, with the 1st map (wins all reads) the last operation, the last one the input_values dict.

_reschedule(dag, nbunch_to_break, op)[source]¶: Update dag/canceled/executed ops and return newly-canceled ops.

canceled = None[source]¶: A sorted set of canceled operation\s due to upstream failures.

check_if_incomplete() → Optional[graphtik.network.IncompleteExecutionError][source]¶: Return a IncompleteExecutionError if netop operations failed/canceled.

dag = None[source]¶

Cloned from plan will be modified, by removing the downstream edges of:

any partial outputs not provided, or
all provides of failed operations.

FIXME: SPURIOUS dag reversals on multi-threaded runs (see below next assertion)!

debugstr()[source]¶

executed = None[source]¶

A dictionary with keys the operations executed, and values their status:

no key: not executed yet
value None: execution ok
value Exception: execution failed
value Collection: canceled provides

finalize()[source]¶: invoked only once, after all ops have been executed

finalized = None[source]¶: a flag controlled by plan (by invoking finalized() is invoked) which becomes True when this instance has finished accepting results.

is_failed(op)[source]¶

operation_executed(op, outputs)[source]¶

Invoked once per operation, with its results.

It will update executed with the operation status and if outputs were partials, it will update canceled with the unsatisfied ops downstream of op.

Parameters

op – the operation that completed ok
outputs – The named values the op` actually produced, which may be a subset of its provides. Sideffects are not considered.

operation_failed(op, ex)[source]¶

Invoked once per operation, with its results.

It will update executed with the operation status and the canceled with the unsatisfied ops downstream of op.

property overwrites¶

The data in the solution that exist more than once.

A “virtual” property to a dictionary with keys the names of values that exist more than once, and values, all those values in a list, ordered:

before finalized(), as computed;
after finalized(), in reverse.

plan = None[source]¶: the plan that produced this solution

prepare_plot_args(plot_args: graphtik.base.PlotArgs) → graphtik.base.PlotArgs [source]¶: delegate to plan, with solution

scream_if_incomplete()[source]¶: Raise a IncompleteExecutionError if netop operations failed/canceled.

solid = None[source]¶: A unique identifier to distinguish separate flows in execution logs.

class graphtik.network._EvictInstruction[source]¶

A step in the ExecutionPlan to evict a computed value from the solution.

It’s a step in ExecutionPlan.steps for the data-node str that frees its data-value from solution after it is no longer needed, to reduce memory footprint while computing the graph.

__module__ = 'graphtik.network'¶

__repr__()[source]¶: Return repr(self).

__slots__ = ()[source]¶

class graphtik.network._OpTask(op, sol, solid)[source]¶

Mimic concurrent.futures.Future for sequential execution.

This intermediate class is needed to solve pickling issue with process executor.

__call__()[source]¶: Call self as a function.

__init__(op, sol, solid)[source]¶: Initialize self. See help(type(self)) for accurate signature.

__module__ = 'graphtik.network'¶

__repr__()[source]¶: Return repr(self).

__slots__ = ('op', 'sol', 'result', 'solid')¶

get()[source]¶: Call self as a function.

logname = 'graphtik.network'¶

marshalled()[source]¶

op¶

result¶

sol¶

solid¶

graphtik.network._do_task(task)[source]¶

Un-dill the simpler _OpTask & Dill the results, to pass through pool-processes.

See https://stackoverflow.com/a/24673524/548792

graphtik.network._isDebugLogging()[source]¶

graphtik.network._optionalized(graph, data)[source]¶: Retain optionality of a data node based on all needs edges.

graphtik.network._unsatisfied_operations(dag, inputs: Collection) → List[source]¶

Traverse topologically sorted dag to collect un-satisfied operations.

Unsatisfied operations are those suffering from ANY of the following:

They are missing at least one compulsory need-input.
Since the dag is ordered, as soon as we’re on an operation, all its needs have been accounted, so we can get its satisfaction.
Their provided outputs are not linked to any data in the dag.
An operation might not have any output link when _prune_graph() has broken them, due to given intermediate inputs.

Parameters

dag – a graph with broken edges those arriving to existing inputs
inputs – an iterable of the names of the input values

Returns

a list of unsatisfied operations to prune

graphtik.network._yield_datanodes(nodes)[source]¶: May scan dag nodes.

graphtik.network.collect_requirements(graph) → Tuple[boltons.setutils.IndexedSet, boltons.setutils.IndexedSet][source]¶: Collect & split datanodes in (possibly overlapping) needs/provides.

graphtik.network.log = <Logger graphtik.network (WARNING)>¶: If this logger is eventually DEBUG-enabled, the string-representation of network-objects (network, plan, solution) is augmented with children’s details.

graphtik.network.yield_node_names(nodes)[source]¶: Yield either op.name or str(node).

graphtik.network.yield_ops(nodes)[source]¶: May scan (preferably) plan.steps or dag nodes.

Module: plot¶

Plotting of graph graphs handled by active plotter (see also base).

Separate from graphtik.base to avoid too many imports too early.

class graphtik.plot.Plotter(theme: graphtik.plot.Theme = None, **styles_kw)[source]¶

a plotter renders diagram images of plottables.

default_theme[source]¶: The customizable Theme instance controlling theme values & dictionaries for plots.

build_pydot(plot_args: graphtik.base.PlotArgs) → pydot.Dot[source]¶

Build a pydot.Dot out of a Network graph/steps/inputs/outputs and return it

to be fed into Graphviz to render.

See Plottable.plot() for the arguments, sample code, and the legend of the plots.

legend(filename=None, jupyter_render: Mapping = None, theme: graphtik.plot.Theme = None)[source]¶

Generate a legend for all plots (see Plottable.plot() for args)

See Plotter.render_pydot() for the rest arguments.

plot(plot_args: graphtik.base.PlotArgs)[source]¶

render_pydot(dot: pydot.Dot, filename=None, jupyter_render: str = None)[source]¶

Render a pydot.Dot instance with Graphviz in a file and/or in a matplotlib window.

Parameters

dot – the pre-built pydot.Dot instance
filename (str) –
Write a file or open a matplotlib window.
- If it is a string or file, the diagram is written into the file-path
  
  Common extensions are .png .dot .jpg .jpeg .pdf .svg call plot.supported_plot_formats() for more.
- If it IS True, opens the diagram in a matplotlib window (requires matplotlib package to be installed).
- If it equals -1, it mat-plots but does not open the window.
- Otherwise, just return the pydot.Dot instance.
seealso

PlotArgs.filename
jupyter_render –
a nested dictionary controlling the rendering of graph-plots in Jupyter cells. If None, defaults to default_jupyter_render; you may modify those in place and they will apply for all future calls (see Jupyter notebooks).

You may increase the height of the SVG cell output with something like this:
```
netop.plot(jupyter_render={"svg_element_styles": "height: 600px; width: 100%"})
```

Returns

the matplotlib image if filename=-1, or the given dot annotated with any jupyter-rendering configurations given in jupyter_render parameter.

See Plottable.plot() for sample code.

with_styles(**kw) → graphtik.plot.Plotter [source]¶

Returns a cloned plotter with a deep-copied theme modified as given.

Module: config¶

configurations for network execution, and utilities on them.

Module: base¶

Generic utilities, or plot API but without polluting imports.

exception graphtik.base.MultiValueError[source]¶

graphtik.base.NO_RESULT = <NO_RESULT>¶: When an operation function returns this special value, it implies operation has no result at all, (otherwise, it would have been a single result, None). Usefull for partial outputs who want to cancel their single result witout being delcared as returns dictionary.

class graphtik.base.PlotArgs[source]¶

All the args of a Plottable.plot() call,

check this method for a more detailed explanation of its attributes.

clone_or_merge_graph(base_graph) → graphtik.base.PlotArgs [source]¶

Overlay graph over base_graph, or clone base_graph, if no attribute.

Returns: the updated plot_args

property clustered¶: Collect the actual clustered dot_nodes among the given nodes.

property clusters¶

Either a mapping of node-names to dot(.)-separated cluster-names, or false/true to enable plotter’s default clustering of nodes based on their dot-separated name parts.

Note that if it’s None (default), the plotter will cluster based on node-names, BUT the Plan may replace the None with a dictionary with the “pruned” cluster (when its dag differs from network’s graph); to suppress the pruned-cluster, pass a truthy, NON-dictionary value.

property dot¶: Where to add graphviz nodes & stuff.

property dot_item¶: The pydot-node/edge created

property filename¶: where to write image or show in a matplotlib window

property graph¶: what to plot (or the “overlay” when calling Plottable.plot())

property inputs¶: the list of input names .

property jupyter_render¶: jupyter configuration overrides

property kw_render_pydot¶

property name¶: The name of the graph in the dot-file (important for cmaps).

property nx_attrs¶: Attributes gotten from nx-graph for the given graph/node/edge. They are NOT a clone, so any modifications affect the nx graph.

property nx_item¶: The node (data(str) or Operation) or edge as gotten from nx-graph.

property outputs¶: the list of output names .

property plottable¶: who is the caller

property plotter¶: If given, overrides :active plotter`.

property solution¶: Contains the computed results, which might be different from plottable.

property steps¶: the list of execution plan steps.

property theme¶: If given, overrides plot theme plotter will use.

with_defaults(*args, **kw) → graphtik.base.PlotArgs [source]¶: Replace only fields with None values.

class graphtik.base.Plottable[source]¶

Classes wishing to plot their graphs should inherit this and …

implement property plot to return a “partial” callable that somehow ends up calling plot.render_pydot() with the graph or any other args bound appropriately. The purpose is to avoid copying this function & documentation here around.

plot(filename: Union[str, bool, int] = None, show=None, *, plotter: graphtik.plot.Plotter = None, theme: graphtik.plot.Theme = None, graph: networkx.Graph = None, name=None, steps=None, inputs=None, outputs=None, solution: graphtik.network.Solution = None, clusters: Mapping = None, jupyter_render: Union[None, Mapping, str] = None) → pydot.Dot[source]¶

Entry-point for plotting ready made operation graphs.

Parameters

filename (str) –
Write a file or open a matplotlib window.
- If it is a string or file, the diagram is written into the file-path
  
  Common extensions are .png .dot .jpg .jpeg .pdf .svg call plot.supported_plot_formats() for more.
- If it IS True, opens the diagram in a matplotlib window (requires matplotlib package to be installed).
- If it equals -1, it mat-plots but does not open the window.
- Otherwise, just return the pydot.Dot instance.
seealso

PlotArgs.filename, Plotter.render_pydot()
plottable –
the plottable that ordered the plotting. Automatically set downstreams to one of:
```
op | netop | net | plan | solution | <missing>
```
seealso

PlotArgs.plottable
plotter –
the plotter to handle plotting; if none, the active plotter is used by default.

seealso

PlotArgs.plotter
theme –
Any plot theme overrides; if none, the Plotter.default_theme of the active plotter is used.

seealso

PlotArgs.theme
name –
if not given, dot-lang graph would is named “G”; necessary to be unique when referring to generated CMAPs. No need to quote it, handled by the plotter, downstream.

seealso

PlotArgs.name
graph (str) –
(optional) A nx.Digraph with overrides to merge with the graph provided by underlying plottables (translated by the active plotter).

It may contain graph, node & edge attributes for any usage, but these conventions apply:

'graphviz.xxx' (graph/node/edge attributes)
Any “user-overrides” with this prefix are sent verbatim a Graphviz attributes.

Note

Remember to escape those values as Graphviz HTML-Like strings (use plot.graphviz_html_string()).

no_plot (node/edge attribute)
element skipped from plotting (see “Examples:” section, below)

seealso

PlotArgs.graph
inputs –
an optional name list, any nodes in there are plotted as a “house”

seealso

PlotArgs.inputs
outputs –
an optional name list, any nodes in there are plotted as an “inverted-house”

seealso

PlotArgs.outputs
solution –
an optional dict with values to annotate nodes, drawn “filled” (currently content not shown, but node drawn as “filled”). It extracts more infos from a Solution instance, such as, if solution has an executed attribute, operations contained in it are drawn as “filled”.

seealso

PlotArgs.solution
clusters –
Either a mapping, or false/true to enable plotter’s default clustering of nodes base on their dot-separated name parts.

Note that if it’s None (default), the plotter will cluster based on node-names, BUT the Plan may replace the None with a dictionary with the “pruned” cluster (when its dag differs from network’s graph); to suppress the pruned-cluster, pass a truthy, NON-dictionary value.

Practically, when it is a:
- dictionary of node-names –> dot(.)-separated cluster-names, it is respected, even if empty;
- truthy: cluster based on dot(.)-separated node-name parts;
- falsy: don’t cluster at all.
seealso

PlotArgs.clusters
jupyter_render –
a nested dictionary controlling the rendering of graph-plots in Jupyter cells, if None, defaults to jupyter_render; you may modify it in place and apply for all future calls (see Jupyter notebooks).

seealso

PlotArgs.jupyter_render
show –

Deprecated since version v6.1.1: Merged with filename param (filename takes precedence).

Returns

a pydot.Dot instance (for reference to as similar API to pydot.Dot instance, visit: https://pydotplus.readthedocs.io/reference.html#pydotplus.graphviz.Dot)

The pydot.Dot instance returned is rendered directly in Jupyter/IPython notebooks as SVG images (see Jupyter notebooks).

Note that the graph argument is absent - Each Plottable provides its own graph internally; use directly render_pydot() to provide a different graph.

NODES:

oval: function
egg: subgraph operation
house: given input
inversed-house: asked output
polygon: given both as input & asked as output (what?)
square: intermediate data, neither given nor asked.
red frame: evict-instruction, to free up memory.
filled: data node has a value in solution OR function has been executed.
thick frame: function/data node in execution steps.

ARROWS

solid black arrows: dependencies (source-data need-ed by target-operations, sources-operations provides target-data)
dashed black arrows: optional needs
blue arrows: sideffect needs/provides
wheat arrows: broken dependency (provide) during pruning
green-dotted arrows: execution steps labeled in succession

To generate the legend, see legend().

Examples:

>>> from graphtik import compose, operation
>>> from graphtik.modifiers import optional
>>> from operator import add

>>> netop = compose("netop",
...     operation(name="add", needs=["a", "b1"], provides=["ab1"])(add),
...     operation(name="sub", needs=["a", optional("b2")], provides=["ab2"])(lambda a, b=1: a-b),
...     operation(name="abb", needs=["ab1", "ab2"], provides=["asked"])(add),
... )

>>> netop.plot(True);                       # plot just the graph in a matplotlib window # doctest: +SKIP
>>> inputs = {'a': 1, 'b1': 2}
>>> solution = netop(**inputs)              # now plots will include the execution-plan

>>> netop.plot('plot1.svg', inputs=inputs, outputs=['asked', 'b1'], solution=solution);           # doctest: +SKIP
>>> dot = netop.plot(solution=solution);    # just get the `pydot.Dot` object, renderable in Jupyter
>>> print(dot)
digraph netop {
fontname=italic;
label=<netop>;
<a> [fillcolor=wheat, margin="0.04,0.02", shape=invhouse, style=filled, tooltip="(int) 1"];
...

You may use the PlotArgs.graph overlay to skip certain nodes (or edges) from the plots:

>>> import networkx as nx

>>> g = nx.DiGraph()  # the overlay
>>> to_hide = netop.net.find_op_by_name("sub")
>>> g.add_node(to_hide, no_plot=True)
>>> dot = netop.plot(graph=g)
>>> assert "<sub>" not in str(dot), str(dot)

abstract prepare_plot_args(plot_args: graphtik.base.PlotArgs) → graphtik.base.PlotArgs [source]¶

Called by plot() to create the nx-graph and other plot-args, e.g. solution.

Clone the graph or merge it with the one in the plot_args (see PlotArgs.clone_or_merge_graph().
For the rest args, prefer PlotArgs.with_defaults() over _replace(), not to override user args.

class graphtik.base.Token(*args)[source]¶

Guarantee equality, not(!) identity, across processes.

hashid¶

graphtik.base.aslist(i, argname, allowed_types=<class 'list'>)[source]¶: Utility to accept singular strings as lists, and None –> [].

graphtik.base.astuple(i, argname, allowed_types=<class 'tuple'>)[source]¶

graphtik.base.func_name(fn, default=Ellipsis, mod=None, fqdn=None, human=None, partials=None) → Optional[str][source]¶

FQDN of fn, descending into partials to print their args.

Parameters

default – What to return if it fails; by default it raises.
mod – when true, prepend module like module.name.fn_name
fqdn – when true, use __qualname__ (instead of __name__) which differs mostly on methods, where it contains class(es), and locals, respectively (PEP 3155). Sphinx uses fqdn=True for generating IDs.
human – when true, explain built-ins, and assume partials=True (if that was None)
partials – when true (or omitted & human true), partials denote their args like fn({"a": 1}, ...)

Returns

a (possibly dot-separated) string, or default (unless this is ...`).

Raises

Only if default is ..., otherwise, errors debug-logged.

Examples

>>> func_name(func_name)
'func_name'
>>> func_name(func_name, mod=1)
'graphtik.base.func_name'
>>> func_name(MultiValueError.mro, fqdn=0)
'mro'
>>> func_name(MultiValueError.mro, fqdn=1)
'MultiValueError.mro'

Even functions defined in docstrings are reported:

>>> def f():
...     def inner():
...         pass
...     return inner

>>> func_name(f, mod=1, fqdn=1)
'graphtik.base.f'
>>> func_name(f(), fqdn=1)
'f.<locals>.inner'

On failures, arg default controls the outcomes:

TBD

graphtik.base.func_source(fn, default=Ellipsis, human=None) → Optional[Tuple[str, int]][source]¶

Like inspect.getsource() supporting partials.

Parameters

default – If given, better be a 2-tuple respecting types, or ..., to raise.
human – when true, denote builtins like python does

graphtik.base.func_sourcelines(fn, default=Ellipsis, human=None) → Optional[Tuple[str, int]][source]¶

Like inspect.getsourcelines() supporting partials.

Parameters: default – If given, better be a 2-tuple respecting types, or ..., to raise.

graphtik.base.jetsam(ex, locs, *salvage_vars: str, annotation='jetsam', **salvage_mappings)[source]¶

Annotate exception with salvaged values from locals() and raise!

Parameters

ex – the exception to annotate
locs –
locals() from the context-manager’s block containing vars to be salvaged in case of exception

ATTENTION: wrapped function must finally call locals(), because locals dictionary only reflects local-var changes after call.
annotation – the name of the attribute to attach on the exception
salvage_vars – local variable names to save as is in the salvaged annotations dictionary.
salvage_mappings – a mapping of destination-annotation-keys –> source-locals-keys; if a source is callable, the value to salvage is retrieved by calling value(locs). They take precedence over`salvage_vars`.

Raises

any exception raised by the wrapped function, annotated with values assigned as attributes on this context-manager

Any attributes attached on this manager are attached as a new dict on the raised exception as new jetsam attribute with a dict as value.
If the exception is already annotated, any new items are inserted, but existing ones are preserved.

Example:

Call it with managed-block’s locals() and tell which of them to salvage in case of errors:

try:
    a = 1
    b = 2
    raise Exception()
exception Exception as ex:
    jetsam(ex, locals(), "a", b="salvaged_b", c_var="c")
    raise

And then from a REPL:

import sys
sys.last_value.jetsam
{'a': 1, 'salvaged_b': 2, "c_var": None}

** Reason:**

Graphs may become arbitrary deep. Debugging such graphs is notoriously hard.

The purpose is not to require a debugger-session to inspect the root-causes (without precluding one).

Naively salvaging values with a simple try/except block around each function, blocks the debugger from landing on the real cause of the error - it would land on that block; and that could be many nested levels above it.

Module: sphinxext¶

Extends Sphinx with graphtik directive rendering plots from doctest code.

class graphtik.sphinxext.DocFilesPurgatory[source]¶

Keeps 2-way associations of docs <–> abs-files, to purge them.

doc_fpaths: Dict[str, Set[Path]] = None[source]¶: Multi-map: docnames -> abs-file-paths

purge_doc(docname: str)[source]¶: Remove doc-files not used by any other doc.

register_doc_fpath(docname: str, fpath: pathlib.Path)[source]¶: Must be absolute, for purging to work.

class graphtik.sphinxext.GraphtikDoctestDirective(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine)[source]¶: Embeds plots from doctest code (see graphtik).

class graphtik.sphinxext.GraphtikTestoutputDirective(name, arguments, options, content, lineno, content_offset, block_text, state, state_machine)[source]¶: Like graphtik directive, but emulates doctest testoutput blocks.

class graphtik.sphinxext.dynaimage(rawsource='', *children, **attributes)[source]¶: Writes a tag in tag attr (<img> for PNGs, <object> for SVGs/PDFs).

class graphtik.sphinxext.graphtik_node(rawsource='', *children, **attributes)[source]¶

Non-writtable node wrapping (literal-block + figure(dynaimage)) nodes.

The figure gets the :name: & :align: options, the contained dynaimage gets the :height”, :width:, :scale:, :classes: options.