Graphtik¶
(src: 10.5.0, git: v10.5.0 , Apr 25, 2023)
It’s a DAG all the way down!
Computation graphs for Python & Pandas¶
Graphtik is a library to compose, solve, execute & plot graphs of python functions (a.k.a pipelines) that consume and populate named data (a.k.a dependencies), whose names may be nested (such as, pandas dataframe columns), based on whether values for those dependencies exist in the inputs or have been calculated earlier.
In mathematical terms, given:
a partially populated data tree, and
a set of functions operating (consuming/producing) on branches of the data tree,
graphtik collects a subset of functions in a graph that when executed consume & produce as values as possible in the data-tree.
Its primary use case is building flexible algorithms for data science/machine learning projects.
It should be extendable to implement the following:
an IoC dependency resolver (e.g. Java Spring, Google Guice);
an executor of interdependent tasks based on files (e.g. GNU Make);
a custom ETL engine;
a spreadsheet calculation engine.
Graphtik sprang from Graphkit (summer 2019, v1.2.2) to experiment with Python 3.6+ features, but has diverged significantly with enhancements ever since.
Table of Contents
- 1. Operations
- 2. Pipelines
- 3. Plotting and Debugging
- 4. Architecture
- 5. API Reference
- Package: graphtik
- Module: fnop
FnOp
FnOp.__abstractmethods__
FnOp.__call__()
FnOp.__init__()
FnOp.__module__
FnOp.__repr__()
FnOp._abc_impl
FnOp._fn_needs
FnOp._fn_provides
FnOp._match_inputs_with_fn_needs()
FnOp._prepare_match_inputs_error()
FnOp._user_needs
FnOp._user_provides
FnOp._zip_results_plain()
FnOp._zip_results_returns_dict()
FnOp._zip_results_with_provides()
FnOp.aliases
FnOp.compute()
FnOp.deps
FnOp.endured
FnOp.fn
FnOp.marshalled
FnOp.name
FnOp.needs
FnOp.node_props
FnOp.parallel
FnOp.prepare_plot_args()
FnOp.provides
FnOp.rescheduled
FnOp.returns_dict
FnOp.validate_fn_name()
FnOp.withset()
NO_RESULT
NO_RESULT_BUT_SFX
_process_dependencies()
as_renames()
identity_fn()
jsonp_ize_all()
operation()
prefixed()
reparse_operation_data()
- Module: pipeline
Pipeline
Pipeline.__abstractmethods__
Pipeline.__call__()
Pipeline.__init__()
Pipeline.__module__
Pipeline.__name__
Pipeline.__qualname__
Pipeline.__repr__()
Pipeline._abc_impl
Pipeline.compile()
Pipeline.compute()
Pipeline.graph
Pipeline.name
Pipeline.needs
Pipeline.outputs
Pipeline.predicate
Pipeline.prepare_plot_args()
Pipeline.provides
Pipeline.withset()
_id_bool()
_id_tristate_bool()
build_network()
compose()
nest_any_node()
- Module: modifier
Accessor
HCatAcc()
JsonpAcc()
VCatAcc()
_Modifier
_Optionals
_modifier()
_modifier_cstor_matrix
acc_contains()
acc_delitem()
acc_getitem()
acc_setitem()
dep_renamed()
dep_singularized()
dep_stripped()
dependency()
get_accessor()
get_jsonp()
get_keyword()
hcat()
implicit()
is_implicit()
is_optional()
is_pure_sfx()
is_sfx()
is_sfxed()
is_vararg()
is_varargish()
is_varargs()
jsonp_ize()
keyword()
modifier_withset()
modify()
optional()
sfx()
sfxed()
sfxed_vararg()
sfxed_varargs()
vararg()
varargs()
vcat()
- Module: planning
Network
Network.needs
Network.provides
Network.__abstractmethods__
Network.__init__()
Network.__module__
Network.__repr__()
Network._abc_impl
Network._append_operation()
Network._apply_graph_predicate()
Network._build_execution_steps()
Network._cached_plans
Network._deps_tuplized()
Network._prune_graph()
Network.compile()
Network.graph
Network.prepare_plot_args()
_format_cycle()
_optionalized()
_topo_sort_nodes()
_yield_also_chained_docs()
_yield_chained_docs()
clone_graph_with_stripped_sfxed()
collect_requirements()
inputs_for_recompute()
log
root_doc()
unsatisfied_operations()
yield_also_chaindocs()
yield_also_subdocs()
yield_also_superdocs()
yield_chaindocs()
yield_datanodes()
yield_node_names()
yield_ops()
yield_subdocs()
yield_superdocs()
- Module: execution
ExecutionPlan
ExecutionPlan.net
ExecutionPlan.needs
ExecutionPlan.provides
ExecutionPlan.dag
ExecutionPlan.steps
ExecutionPlan.asked_outs
ExecutionPlan.comments
ExecutionPlan.__abstractmethods__
ExecutionPlan.__dict__
ExecutionPlan.__module__
ExecutionPlan.__repr__()
ExecutionPlan._abc_impl
ExecutionPlan._check_if_aborted()
ExecutionPlan._execute_sequential_method()
ExecutionPlan._execute_thread_pool_barrier_method()
ExecutionPlan._handle_task()
ExecutionPlan._prepare_tasks()
ExecutionPlan.execute()
ExecutionPlan.graph
ExecutionPlan.prepare_plot_args()
ExecutionPlan.validate()
OpTask
Solution
Solution.__abstractmethods__
Solution.__annotations__
Solution.__contains__()
Solution.__copy__()
Solution.__delitem__()
Solution.__getitem__()
Solution.__init__()
Solution.__module__
Solution.__repr__()
Solution.__setitem__()
Solution._abc_impl
Solution._initial_inputs
Solution._overwrites_cache
Solution._populate_op_layer_with_outputs()
Solution._reschedule()
Solution.broken
Solution.canceled
Solution.check_if_incomplete()
Solution.copy()
Solution.dag
Solution.debugstr()
Solution.elapsed_ms
Solution.executed
Solution.graph
Solution.is_failed()
Solution.is_layered
Solution.layers
Solution.operation_executed()
Solution.operation_failed()
Solution.overwrites
Solution.plan
Solution.prepare_plot_args()
Solution.scream_if_incomplete()
Solution.solid
Solution.update()
_do_task()
_isDebugLogging()
log
task_context
- Module: plot
Plotter
Ref
StylesStack
Theme
Theme.arch_url
Theme.broken_color
Theme.canceled_color
Theme.data_bad_html_label_keys
Theme.data_template
Theme.edge_defaults
Theme.evicted_color
Theme.failed_color
Theme.fill_color
Theme.kw_data
Theme.kw_data_evicted
Theme.kw_data_in_solution
Theme.kw_data_in_solution_null
Theme.kw_data_inp
Theme.kw_data_inp_only
Theme.kw_data_io
Theme.kw_data_label
Theme.kw_data_missing
Theme.kw_data_out
Theme.kw_data_out_only
Theme.kw_data_overwritten
Theme.kw_data_pruned
Theme.kw_data_sideffect
Theme.kw_data_sideffected
Theme.kw_data_to_evict
Theme.kw_edge
Theme.kw_edge_alias
Theme.kw_edge_broken
Theme.kw_edge_endured
Theme.kw_edge_head_op
Theme.kw_edge_implicit
Theme.kw_edge_mapping_keyword
Theme.kw_edge_null_result
Theme.kw_edge_optional
Theme.kw_edge_pruned
Theme.kw_edge_rescheduled
Theme.kw_edge_sideffect
Theme.kw_edge_subdoc
Theme.kw_edge_tail_op
Theme.kw_graph
Theme.kw_graph_plottable_type
Theme.kw_graph_plottable_type_unknown
Theme.kw_legend
Theme.kw_op
Theme.kw_op_canceled
Theme.kw_op_endured
Theme.kw_op_executed
Theme.kw_op_failed
Theme.kw_op_label
Theme.kw_op_label2
Theme.kw_op_marshalled
Theme.kw_op_parallel
Theme.kw_op_prune_comment
Theme.kw_op_pruned
Theme.kw_op_rescheduled
Theme.kw_op_returns_dict
Theme.kw_step
Theme.kw_step_badge
Theme.node_defaults
Theme.null_color
Theme.op_bad_html_label_keys
Theme.op_badge_styles
Theme.op_template
Theme.overwrite_color
Theme.pruned_color
Theme.resched_thickness
Theme.show_chaindocs
Theme.show_steps
Theme.sideffect_color
Theme.steps_color
Theme.subdoc_color
Theme.theme_attributes()
Theme.truncate_args
Theme.vector_color
Theme.withset()
USER_STYLE_PREFFIX
active_plotter_plugged()
as_identifier()
default_jupyter_render
get_active_plotter()
get_node_name()
graphviz_html_string()
is_empty_array()
is_empty_frame()
is_nx_node_dependent()
legend()
make_data_value_tooltip()
make_fn_tooltip()
make_op_prune_comment()
make_op_tooltip()
make_overwrite_tooltip()
make_template()
quote_html_tooltips()
quote_node_id()
remerge()
save_plot_file_by_sha1()
set_active_plotter()
supported_plot_formats()
- Module: config
abort_run()
debug_enabled()
evictions_skipped()
execution_pool_plugged()
get_execution_pool()
is_abort()
is_debug()
is_endure_operations()
is_layered_solution()
is_marshal_tasks()
is_parallel_tasks()
is_reschedule_operations()
is_skip_evictions()
operations_endured()
operations_reschedullled()
reset_abort()
set_debug()
set_endure_operations()
set_execution_pool()
set_layered_solution()
set_marshal_tasks()
set_parallel_tasks()
set_reschedule_operations()
set_skip_evictions()
solution_layered()
tasks_in_parallel()
tasks_marshalled()
- Module: base
AbortedException
IncompleteExecutionError
Operation
PlotArgs
PlotArgs.clone_or_merge_graph()
PlotArgs.clustered
PlotArgs.clusters
PlotArgs.dot
PlotArgs.dot_item
PlotArgs.filename
PlotArgs.graph
PlotArgs.inputs
PlotArgs.jupyter_render
PlotArgs.kw_render_pydot
PlotArgs.name
PlotArgs.nx_attrs
PlotArgs.nx_item
PlotArgs.outputs
PlotArgs.plottable
PlotArgs.plotter
PlotArgs.solution
PlotArgs.steps
PlotArgs.theme
PlotArgs.with_defaults()
Plottable
RenArgs
Token
aslist()
asset()
astuple()
first_solid()
func_name()
func_source()
func_sourcelines()
- Module: jetsam
- Module: jsonpointer
- Package: sphinxext
- 6. Changes
- TODOs
- Changelog
- v10.5.0 (25 Apr 2023, @ankostis): REVIVE project, Bump DEPS
- v10.4.0 (9 Oct 2020, @ankostis): CWD, callbacks non-marshalled, preserve concat index-names
- v10.3.0 (21 Sep 2020, @ankostis): CONCAT pandas, Hierarchical overwrites, implicit(), post-cb
- v10.2.1 (18 Sep 2020, @ankostis): plot sol bugfix
- v10.2.0 (16 Sep 2020, @ankostis): RECOMPUTE, pre-callback, drop op_xxx, ops-eq-op.name, drop NULL_OP
- v10.1.0 (5 Aug 2020, @ankostis): rename return-dict outs; step number badges
- v10.0.0 (19 Jul 2020, @ankostis): Implicits; modify(); auto-name pipelines; plot data as overspilled
- v9.3.0 (8 Jul 2020, @ankostis): Sphinx AutoDocumenter; fix plot sfx-nodes
- v9.1.0 (4 Jul 2020, @ankostis): Bugfix, panda-polite, privatize modifier fields
- v9.0.0 (30 Jun 2020, @ankostis): JSONP; net, evictions & sfxed fixes; conveyor fn; rename modules
- v8.4.0 (15 May 2020, @ankostis): subclass-able Op, plot edges from south–>north of nodes
- v8.3.1 (14 May 2020, @ankostis): plot edges from south–>north of nodes
- v8.3.0 (12 May 2020, @ankostis): mapped–>keyword, drop sol-finalize
- v8.2.0 (11 May 2020, @ankostis): custom Solutions, Task-context
- v8.1.0 (11 May 2020, @ankostis): drop last plan, Rename/Nest, Netop–>Pipeline, purify modules
- v8.0.2 (7 May 2020, @ankostis): re-MODULE; sideffect –> sfx; all DIACRITIC Modifiers; invert “merge” meaning
- v7.1.2 (6 May 2020, @ankostis): minor reschedule fixes and refactoring
- v7.1.0 (4 May 2020, @ankostis): Cancelable sideffects, theme-ize & expand everything
- v7.0.0 (28 Apr 2020, @ankostis): In-solution sideffects, unified OpBuilder, plot badges
- v6.2.0 (19 Apr 2020, @ankostis): plotting fixes & more styles, net find util methods
- v6.1.0 (14 Apr 2020, @ankostis): config plugs & fix styles
- v6.0.0 (13 Apr 2020, @ankostis): New Plotting Device…
- v5.7.1 (7 Apr 2020, @ankostis): Plot job, fix RTD deps
- v5.7.0 (6 Apr 2020, @ankostis): FIX +SphinxExt in Wheel
- v5.2.2 (03 Mar 2020, @ankostis): stuck in PARALLEL, fix Impossible Outs, plot quoting, legend node
- v5.2.1 (28 Feb 2020, @ankostis): fix plan cache on skip-evictions, PY3.8 TCs, docs
- v5.2.0 (27 Feb 2020, @ankostis): Map needs inputs –> args, SPELLCHECK
- v5.1.0 (22 Jan 2020, @ankostis): accept named-tuples/objects provides
- v5.0.0 (31 Dec 2019, @ankostis): Method–>Parallel, all configs now per op flags; Screaming Solutions on fails/partials
- v4.4.1 (22 Dec 2019, @ankostis): bugfix debug print
- v4.4.0 (21 Dec 2019, @ankostis): RESCHEDULE for PARTIAL Outputs, on a per op basis
- v4.3.0 (16 Dec 2019, @ankostis): Aliases
- v4.2.0 (16 Dec 2019, @ankostis): ENDURED Execution
- v4.1.0 (13 Dec 2019, @ankostis): ChainMap Solution for Rewrites, stable TOPOLOGICAL sort
- v4.0.1 (12 Dec 2019, @ankostis): bugfix
- v4.0.0 (11 Dec 2019, @ankostis): NESTED merge, revert v3.x Unvarying, immutable OPs, “color” nodes
- v3.1.0 (6 Dec 2019, @ankostis): cooler
prune()
- v3.0.0 (2 Dec 2019, @ankostis): UNVARYING NetOperations, narrowed, API refact
- v2.3.0 (24 Nov 2019, @ankostis): Zoomable SVGs & more op jobs
- v2.2.0 (20 Nov 2019, @ankostis): enhance OPERATIONS & restruct their modules
- v2.1.1 (12 Nov 2019, @ankostis): global configs
- v2.1.0 (20 Oct 2019, @ankostis): DROP BW-compatible, Restruct modules/API, Plan perfect evictions
- v2.0.0b1 (15 Oct 2019, @ankostis): Rebranded as Graphtik for Python 3.6+
- v1.3.0 (Oct 2019, @ankostis): NEVER RELEASED: new DAG solver, better plotting & “sideffect”
- v1.2.4 (Mar 7, 2018)
- 1.2.2 (Mar 7, 2018, @huyng): Fixed versioning
- 1.2.1 (Feb 23, 2018, @huyng): Fixed multi-threading bug and faster compute through caching of find_necessary_steps
- 1.2.0 (Feb 13, 2018, @huyng)
- 1.1.0 (Nov 9, 2017, @huyng)
- 1.0.4 (Nov 3, 2017, @huyng): Networkx 2.0 compatibility
- 1.0.3 (Jan 31, 2017, @huyng): Make plotting dependencies optional
- 1.0.2 (Sep 29, 2016, @pumpikano): Merge pull request yahoo#5 from yahoo/remove-packaging-dep
- 1.0.1 (Aug 24, 2016)
- 1.0 (Aug 2, 2016, @robwhess)
- v10.5.0 (25 Apr 2023, @ankostis): REVIVE project, Bump DEPS
- Index
Features¶
Deterministic pre-decided execution plan (unless partial-outputs or endured operations defined, see below).
Can assemble existing functions without modifications into pipelines.
dependency resolution can bypass calculation cycles based on data given and asked.
Support functions with partial outputs; keep working even if certain endured operations fail.
Facilitate trivial conveyor operations and alias on provides.
Support cycles, by annotating repeated updates of dependency values as sideffects, (e.g. to add columns into
pandas.DataFrame
s).Hierarchical dependencies may access data values deep in solution with json pointer path expressions.
Hierarchical dependencies annotated as implicit imply which subdoc dependency the function reads or writes in the parent-doc.
Early eviction of intermediate results from solution, to optimize memory footprint.
Solution tracks all intermediate overwritten values for the same dependency.
Elaborate Graphviz plotting with configurable plot themes.
Integration with Sphinx sites with the new
graphtik
directive.Authored with debugging in mind.
Parallel execution (but underdeveloped & DEPRECATED).
Anti-features¶
It’s not meant to follow complex conditional logic based on dependency values (though it does support that to a limited degree).
It’s not an orchestrator for long-running tasks, nor a calendar scheduler - Apache Airflow, Dagster or Luigi may help for that.
It’s not really a parallelizing optimizer, neither a map-reduce framework - look additionally at Dask, IpyParallel, Celery, Hive, Pig, Hadoop, etc.
It’s not a stream/batch processor, like Spark, Storm, Fink, Kinesis, because it pertains function-call semantics, calling only once each function to process data-items.
Differences with schedula¶
schedula is a powerful library written roughly for the same purpose, and works differently along these lines (ie features below refer to schedula):
terminology (<graphtik> := <schedula>):
pipeline := dispatcher
plan := workflow
solution := solution
Dijkstra planning runs while calling operations:
Powerful & flexible (ie all operations are dynamic, domains are possible, etc).
Supports weights.
Cannot pre-calculate & cache execution plans (slow).
Calculated values are stored inside a graph (mimicking the structure of the functions):
graph visualizations absolutely needed to inspect & debug its solutions.
- graphs imply complex pre/post processing & traversal algos
(vs constructing/traversing data-trees).
Reactive plotted diagrams, web-server runs behind the scenes.
Operation graphs are stackable:
plotted nested-graphs support drill-down.
graphtik emulates that with data/operation names (operation nesting), but always a unified graph is solved at once, bc it is impossible to dress nesting-funcs as a python-funcs and pre-solve plan (schedula does not pre-solve plan, Dijkstra runs all the time). See TODO about plotting such nested graphs.
Schedula does not calculate all possible values (ie no overwrites).
Schedula computes precedence based on weights and lexicographical order of function name.
Re-inserting operation does not overrides its current function - must remove it first.
graphtik precedence based insertion order during composition.
Virtual start and end data-nodes needed for Dijkstra to solve the dag.
No domains (execute-time conditionals deciding whether a function must run).
Probably Re-computations is more straightforward in graphtik.
TODO: more differences with schedula exist.
Quick start¶
Here’s how to install:
pip install graphtik
OR with dependencies for plotting support (and you need to install Graphviz program separately with your OS tools):
pip install graphtik[plot]
Let’s build a graphtik computation pipeline that produces the following
x3 outputs out of x2 inputs (α
and β
):
>>> from graphtik import compose, operation
>>> from operator import mul, sub
>>> @operation(name="abs qubed",
... needs=["α-α×β"],
... provides=["|α-α×β|³"])
... def abs_qubed(a):
... return abs(a) ** 3
Hint
Notice that graphtik has not problem working in unicode chars for dependency names.
Compose the abspow
function along with mul
& sub
built-ins
into a computation graph:
>>> graphop = compose("graphop",
... operation(mul, needs=["α", "β"], provides=["α×β"]),
... operation(sub, needs=["α", "α×β"], provides=["α-α×β"]),
... abs_qubed,
... )
>>> graphop
Pipeline('graphop', needs=['α', 'β', 'α×β', 'α-α×β'],
provides=['α×β', 'α-α×β', '|α-α×β|³'],
x3 ops: mul, sub, abs qubed)
You may plot the function graph in a file like this (if in jupyter, no need to specify the file, see Jupyter notebooks):
>>> graphop.plot('graphop.svg') # doctest: +SKIP
As you can see, any function can be used as an operation in Graphtik, even ones imported from system modules.
Run the graph-operation and request all of the outputs:
>>> sol = graphop(**{'α': 2, 'β': 5})
>>> sol
{'α': 2, 'β': 5, 'α×β': 10, 'α-α×β': -8, '|α-α×β|³': 512}
Solutions are plottable as well:
>>> solution.plot('solution.svg') # doctest: +SKIP
Run the graph-operation and request a subset of the outputs:
>>> solution = graphop.compute({'α': 2, 'β': 5}, outputs=["α-α×β"])
>>> solution
{'α-α×β': -8}
… where the (interactive) legend is this:
>>> from graphtik.plot import legend
>>> l = legend()