Graphtik

Latest version in PyPI Latest release in GitHub (src: 10.4.0, git: v10.4.0-8-gcc43a29 , Apr 13, 2021) Supported Python versions of latest release in PyPi Development Status Travis continuous integration testing ok? (Linux) ReadTheDocs ok? cover-status Code Style Apache License, version 2.0

Github watchers Github stargazers Github forks Issues count

It’s a DAG all the way down!

solution_x9_nodes quarantine quarantine get_out_or_stay_home OP: get_out_or_stay_home 0 ? } FN: get_out_or_stay_home quarantine:s->get_out_or_stay_home:n space space get_out_or_stay_home:s->space:n time time get_out_or_stay_home:s->time:n exercise OP: exercise 1 FN: exercise space:s->exercise:n read_book OP: read_book 2 FN: read_book time:s->read_book:n fun fun exercise:s->fun:n body body exercise:s->body:n read_book:s->fun:n brain brain read_book:s->brain:n legend legend

Computation graphs for Python & Pandas

Graphtik is a library to compose, solve, execute & plot graphs of python functions (a.k.a pipelines) that consume and populate named data (a.k.a dependencies), whose names may be nested (such as, pandas dataframe columns), based on whether values for those dependencies exist in the inputs or have been calculated earlier.

In mathematical terms, given:

graphtik finds and executes the subset of functions producing as many values as possible in the tree.

Usage overview of graphtik library

  • Its primary use case is building flexible algorithms for data science/machine learning projects.

  • It should be extendable to implement the following:

    • an IoC dependency resolver (e.g. Java Spring);

    • an executor of interdependent tasks based on files (e.g. GNU Make);

    • a custom ETL engine;

    • a spreadsheet calculation engine.

Graphtik sprang from Graphkit (summer 2019, v1.2.2) to experiment with Python 3.6+ features, but has diverged significantly with enhancements ever since.

Table of Contents

Features

Anti-features

  • It’s not an orchestrator for long-running tasks, nor a calendar scheduler - Apache Airflow, Dagster or Luigi may help for that.

  • It’s not really a parallelizing optimizer, neither a map-reduce framework - look additionally at Dask, IpyParallel, Celery, Hive, Pig, Spark, Hadoop, etc.

  • It’s not meant to follow complex conditional logic based on dependency values (though it does support that to a limited degree).

Quick start

Here’s how to install:

pip install graphtik

OR with dependencies for plotting support (and you need to install Graphviz program separately with your OS tools):

pip install graphtik[plot]

Let’s build a graphtik computation pipeline that produces the following x3 outputs out of x2 inputs (α and β):

(1)\[ \begin{align}\begin{aligned}α \times β\\α - α \times β\\|α - α \times β| ^ 3\end{aligned}\end{align} \]
>>> from graphtik import compose, operation
>>> from operator import mul, sub
>>> @operation(name="abs qubed",
...            needs=["α-α×β"],
...            provides=["|α-α×β|³"])
... def abs_qubed(a):
...    return abs(a) ** 3

Hint

Notice that graphtik has not problem working in unicode chars for dependency names.

Compose the abspow function along with mul & sub built-ins into a computation graph:

>>> graphop = compose("graphop",
...    operation(mul, needs=["α", "β"], provides=["α×β"]),
...    operation(sub, needs=["α", "α×β"], provides=["α-α×β"]),
...    abs_qubed,
... )
>>> graphop
Pipeline('graphop', needs=['α', 'β', 'α×β', 'α-α×β'],
         provides=['α×β', 'α-α×β', '|α-α×β|³'],
         x3 ops: mul, sub, abs qubed)

You may plot the function graph in a file like this (if in jupyter, no need to specify the file, see Jupyter notebooks):

>>> graphop.plot('graphop.svg')      # doctest: +SKIP

As you can see, any function can be used as an operation in Graphtik, even ones imported from system modules.

Run the graph-operation and request all of the outputs:

>>> sol = graphop(**{'α': 2, 'β': 5})
>>> sol
{'α': 2, 'β': 5, 'α×β': 10, 'α-α×β': -8, '|α-α×β|³': 512}

Solutions are plottable as well:

>>> solution.plot('solution.svg')      # doctest: +SKIP

Run the graph-operation and request a subset of the outputs:

>>> solution = graphop.compute({'α': 2, 'β': 5}, outputs=["α-α×β"])
>>> solution
{'α-α×β': -8}

… where the (interactive) legend is this:

>>> from graphtik.plot import legend
>>> l = legend()

legend