6. Graphtik Changelog¶
GitHub Releases¶
Changelog¶
v5.2.1 (28 Feb 2020, @ankostis): fix plan cache on skip-evictions, PY3.8 TCs, docs¶
- FIX(net): Execution-plans were cached also the transient
is_skip_evictions()
configurations (instead of just whether no-outputs were asked). - doc(readme): explain “fork” status in the opening.
- ENH(travis): run full tests from Python-3.7–> Python-3.8.
v5.2.0 (27 Feb 2020, @ankostis): Map needs inputs –> args, SPELLCHECK¶
- FEAT(modifiers): optionals and new modifier
arg
can now fetch values from inputs into differently-named arguments of operation functions.- refact: decouple varargs from optional modifiers hierarchy.
- REFACT(OP): preparation of NEEDS –> function-args happens once for each argument, allowing to report all errors at once.
- feat(base): +MultiValueError exception class.
- DOC(modifiers,arch): modifiers were not included in “API reference”, nor in the glossary sections.
- FIX: spell-check everything, and add all custom words in the VSCode settings file
.vscode.settings.json
.
v5.1.0 (22 Jan 2020, @ankostis): accept named-tuples/objects provides¶
- ENH(OP): flag returns_dict handles also named-tuples & objects (
__dict__
).
v5.0.0 (31 Dec 2019, @ankostis): Method–>Parallel, all configs now per op flags; Screaming Solutions on fails/partials¶
BREAK(NETOP):
compose(method="parallel") --> compose(parallel=None/False/True)
and DROPnetop.set_execution_method(method)
; parallel now also controlled with the globalset_parallel_tasks()
configurations function.- feat(jetsam): report task executed in raised exceptions.
break(netop): rename
netop.narrowed() --> withset()
toi mimicOperation
API.break: rename flags:
reschedule --> rescheduleD
marshal --> marshalLED
.
break: rename global configs, as context-managers:
marshal_parallel_tasks --> tasks_marshalled
endure_operations --> operations_endured
FIX(net, plan,.TC): global skip evictions flag were not fully obeyed (was untested).
FIX(OP): revamped zipping of function outputs with expected provides, for all combinations of rescheduled,
NO_RESULT
& returns dictionary flags.configs:
- refact: extract configs in their own module.
- refact: make all global flags tri-state (
None, False, True
), allowing to “force” operation flags when not None. All default toNone
(false).
ENH(net, sol, logs): include a “solution-id” in revamped log messages, to facilitate developers to discover issues when multiple netops are running concurrently. Heavily enhanced log messages make sense to the reader of all actions performed.
ENH(plot): set toolltips with
repr(op)
to view all operation flags.FIX(TCs): close process-pools; now much more TCs for parallel combinations of threaded, process-pool & marshalled.
ENH(netop,net): possible to abort many netops at once, by resetting abort flag on every call of
NetworkOperation.compute()
(instead of on the first stopped netop).FEAT(SOL):
scream_if_incomplete()
will raise the newIncompleteExecutionError
exception if failures/partial-outs of endured/rescheduled operations prevented all operations to complete; exception message details causal errors and conditions.feat(build): +``all`` extras.
FAIL: x2 multi-threaded TCs fail spuriously with “inverse dag edges”:
test_multithreading_plan_execution()
test_multi_threading_computes()
both marked as
xfail
.
v4.4.1 (22 Dec 2019, @ankostis): bugfix debug print¶
- fix(net): had forgotten a debug-print on every operation call.
- doc(arch): explain parallel & the need for marshalling with process pools.
v4.4.0 (21 Dec 2019, @ankostis): RESCHEDULE for PARTIAL Outputs, on a per op basis¶
- [x] dynamic Reschedule after operations with partial outputs execute.
- [x] raise after jetsam.
- [x] plots link to legend.
- [x] refact netop
- [x] endurance per op.
- [x] endurance/reschedule for all netop ops.
- [x] merge _Rescheduler into Solution.
- [x] keep order of outputs in Solution even for parallels.
- [x] keep solution layers ordered also for parallel.
- [x] require user to create & enter pools.
- [x] FIX pickling THREAD POOL –>Process.
Details¶
- FIX(NET): keep Solution’s insertion order also for PARALLEL executions.
- FEAT(NET, OP): rescheduled operations with partial outputs;
they must have
FunctionalOperation.reschedule
set to true, or else they will fail. - FEAT(OP, netop): specify endurance/reschedule on a per operation basis, or collectively for all operations grouped under some netop.
- REFACT(NETOP):
- feat(netop): new method
NetworkOperation.compile()
, delegating to same-named method of network. - drop(net): method
Net.narrowed()
; remember netop.narrowed(outputs+predicate) and apply them on netop.compute() &netop.compile()
.- PROS: cache narrowed plans.
- CONS: cannot review network, must review plan of (new) netop.compile().
- drop(netop): inputs args in narrowed() didn’t make much sense, leftover from “unvarying netops”; but exist ni netop.compile().
- refact(netop): move net-assembly from compose() –> NetOp cstor; now reschedule/endured/merge/method args in cstor.
- feat(netop): new method
- NET,OP,TCs: FIX PARALLEL POOL CONCURRENCY
- Network:
- feat: +marshal +_OpTask
- refact: plan._call_op –> _handle_task
- enh: Make abort run variable a shared-memory
Value
.
- REFACT(OP,.TC): not a namedtuple, breaks pickling.
- ENH(pool): Pool
- FIX: compare Tokens with is –> ==, or else, it won’t work for sub-processes.
- TEST: x MULTIPLE TESTS
- +4 tags: parallel, thread, proc, marshal.
- many uses of exemethod.
- Network:
- FIX(build): PyPi README check did not detect forbidden
raw
directives, and travis auto-deployments were failing. - doc(arch): more terms.
v4.3.0 (16 Dec 2019, @ankostis): Aliases¶
- FEAT(OP): support “aliases” of provides, to avoid trivial pipe-through operations, just to rename & match operations.
v4.2.0 (16 Dec 2019, @ankostis): ENDURED Execution¶
- FEAT(NET): when
set_endure_operations()
configuration is set to true, a netop will keep on calculating solution, skipping any operations downstream from failed ones. The solution eventually collects all failures inSolution.failures
attribute. - ENH(DOC,plot): Links in Legend and Architecture Workflow SVGs now work, and delegate to architecture terms.
- ENH(plot): mark overwrites, failed & canceled in
repr()
(see endurance). - refact(conf): fully rename configuration operation
skip_evictions
. - REFACT(jetsam): raise after jetsam in situ, better for Readers & Linters.
- enh(net): improve logging.
v4.1.0 (13 Dec 2019, @ankostis): ChainMap Solution for Rewrites, stable TOPOLOGICAL sort¶
- FIX(NET): TOPOLOGICALLY-sort now break ties respecting operations insertion order.
- ENH(NET): new
Solution
class to collect all computation values, based on acollections.ChainMap
to distinguish outputs per operation executed:- ENH(NETOP):
compute()
returnSolution
, consolidating:- overwrites,
executed
operations, and- the generating plan.
- drop(net):
_PinInstruction
class is not needed. - drop(netop): overwrites_collector parameter; now in
Solution.overwrites()
. - ENH(plot):
Solution
is also aPlotter
; e.g. usesol.plot(...)`
.
- ENH(NETOP):
- DROP(plot): executed arg from plotting; now embedded in solution.
- ENH(PLOT.jupyter,doc): allow to set jupyter graph-styling selectively; fix instructions for jupyter cell-resizing.
- fix(plan): time-keeping worked only for sequential execution, not parallel. Refactor it to happen centrally.
- enh(NET,.TC): Add PREDICATE argument also for
compile()
. - FEAT(DOC): add GLOSSARY as new Architecture section, linked from API HEADERS.
v4.0.1 (12 Dec 2019, @ankostis): bugfix¶
- FIX(plan):
plan.repr()
was failing on empty plans. - fix(site): minor badge fix & landing diagram.
v4.0.0 (11 Dec 2019, @ankostis): NESTED merge, revert v3.x Unvarying, immutable OPs, “color” nodes¶
- BREAK/ENH(NETOP): MERGE NESTED NetOps by collecting all their operations
in a single Network; now children netops are not pruned in case
some of their needs are unsatisfied.
- feat(op): support multiple nesting under other netops.
- BREAK(NETOP): REVERT Unvarying NetOps+base-plan, and narrow Networks instead;
netops were too rigid, code was cumbersome, and could not really pinpoint
the narrowed needs always correctly (e.g. when they were also provides).
- A netop always narrows its net based on given inputs/outputs. This means that the net might be a subset of the one constructed out of the given operations. If you want all nodes, don’t specify needs/provides.
- drop 3
ExecutionPlan
attributes:plan, needs, plan
- drop recompile flag in
Network.compute()
. - feat(net): new method
Network.narrowed()
clones and narrows. Network()
cstor accepts a (cloned) graph to supportnarrowed()
methods.
- BREAK/REFACT(OP): simplify hierarchy, make
Operation
fully abstract, without name or requirements.- enh: make
FunctionalOperation
IMMUTABLE, by inheriting from class:.namedtuple.
- enh: make
- refact(net): consider as netop needs also intermediate data nodes.
- FEAT(#1, net, netop): support pruning based on arbitrary operation attributes (e.g. assign “colors” to nodes and solve a subset each time).
- enh(netop):
repr()
now counts number of contained operations. - refact(netop): rename
netop.narrow() --> narrowed()
- drop(netop): don’t topologically-sort sub-networks before merging them; might change some results, but gives control back to the user to define nets.
v3.1.0 (6 Dec 2019, @ankostis): cooler prune()
¶
- break/refact(NET): scream on
plan.execute()
(notnet.prune()
) so as calmly solve needs vs provides, based on the given inputs/outputs. - FIX(ot): was failing when plotting graphs with ops without fn set.
- enh(net): minor fixes on assertions.
v3.0.0 (2 Dec 2019, @ankostis): UNVARYING NetOperations, narrowed, API refact¶
NetworkOperations:
BREAK(NET): RAISE if the graph is UNSOLVABLE for the given needs & provides! (see “raises” list of
compute()
).BREAK:
NetworkOperation.__call__()
accepts solution as keyword-args, to mimic API ofOperation.__call__()
.outputs
keyword has been dropped.Tip
Use
NetworkOperation.compute()
when you ask different outputs, or set therecompile
flag if just different inputs are given.Read the next change-items for the new behavior of the
compute()
method.UNVARYING NetOperations:
- BREAK: calling method
NetworkOperation.compute()
with a single argument is now UNVARYING, meaning that all needs are demanded, and hence, all provides are produced, unless therecompile
flag is true oroutputs
asked. - BREAK: net-operations behave like regular operations when nested inside another netop, and always produce all their provides, or scream if less inputs than needs are given.
- ENH: a newly created or cloned netop can be
narrowed()
to specific needs & provides, so as not needing to pass outputs on every call tocompute()
. - feat: implemented based on the new “narrowed”
NetworkOperation.plan
attribute.
- BREAK: calling method
FIX: netop needs are not all optional by default; optionality applied only if all underlying operations have a certain need as optional.
FEAT: support function
**args
with 2 new modifiersvararg
&varargs
, acting likeoptional
(but without feeding into underlying functions like keywords).BREAK(yahoo#12): simplify
compose
API by turning it from class –> function; all args and operations are now given in a singlecompose()
call.REFACT(net, netop): make Network IMMUTABLE by appending all operations together, in
NetworkOperation
constructor.ENH(net): public-size
_prune_graph()
–>Network.prune()`()
which can be used to interrogate needs & provides for a given graph. It accepts None inputs & outputs to auto-derive them.
FIX(SITE): autodocs API chapter were not generated in at all, due to import errors, fixed by using autodoc_mock_imports on networkx, pydot & boltons libs.
enh(op): polite error-,msg when calling an operation with missing needs (instead of an abrupt
KeyError
).FEAT(CI): test also on Python-3.8
v2.3.0 (24 Nov 2019, @ankostis): Zoomable SVGs & more op jobs¶
- FEAT(plot): render Zoomable SVGs in jupyter(lab) notebooks.
- break(netop): rename execution-method
"sequential" --> None
. - break(netop): move
overwrites_collector
&method
args fromnetop.__call__()
–> cstor - refact(netop): convert remaining
**kwargs
into named args, tighten up API.
v2.2.0 (20 Nov 2019, @ankostis): enhance OPERATIONS & restruct their modules¶
- REFACT(src): split module
nodes.py
–>op.py
+ netop.py and moveOperation
frombase.py
–>op.py
, in order to break cycle of base(op) <– net <– netop, and keep utils only in base.py. - ENH(op): allow Operations WITHOUT any NEEDS.
- ENH(op): allow Operation FUNCTIONS to return directly Dictionaries.
- ENH(op): validate function Results against operation provides;
jetsam now includes results variables:
results_fn
&results_op
. - BREAK(op): drop unused Operation._after_init() pickle-hook; use dill instead.
- refact(op): convert
Operation._validate()
into a function, to be called by clients wishing to automate operation construction. - refact(op): replace
**kwargs
with named-args in class:FunctionalOperation, because it allowed too wide args, and offered no help to the user. - REFACT(configs): privatize
network._execution_configs
; expose more config-methods from base package.
v2.1.1 (12 Nov 2019, @ankostis): global configs¶
- BREAK: drop Python-3.6 compatibility.
- FEAT: Use (possibly multiple) global configurations for all networks,
stored in a
contextvars.ContextVar
. - ENH/BREAK: Use a (possibly) single execution_pool in global-configs.
- feat: add abort flag in global-configs.
- feat: add skip_evictions flag in global-configs.
v2.1.0 (20 Oct 2019, @ankostis): DROP BW-compatible, Restruct modules/API, Plan perfect evictions¶
The first non pre-release for 2.x train.
- BRAKE API: DROP Operation’s
params
- use functools.partial() instead. - BRAKE API: DROP Backward-Compatible
Data
&Operation
classes, - BRAKE: DROP Pickle workarounds - expected to use
dill
instead. - break(jetsam): drop “graphtik_` prefix from annotated attribute
- ENH(op): now
operation()
supported the “builder pattern” withoperation.withset()
. - REFACT: renamed internal package functional –> nodes and moved classes around,
to break cycles easier, (
base
works as supposed to), not to import early everything, but to fail plot early ifpydot
dependency missing. - REFACT: move PLAN and
compute()
up, fromNetwork --> NetworkOperation
. - ENH(NET): new PLAN BUILDING algorithm produces PERFECT EVICTIONS,
that is, it gradually eliminates from the solution all non-asked outputs.
- enh: pruning now cleans isolated data.
- enh: eviction-instructions are inserted due to two different conditions: once for unneeded data in the past, and another for unused produced data (those not belonging typo the pruned dag).
- enh: discard immediately irrelevant inputs.
- ENH(net): changed results, now unrelated inputs are not included in solution.
- refact(sideffect): store them as node-attributes in DAG, fix their combination with pinning & eviction.
- fix(parallel): eviction was not working due to a typo 65 commits back!
v2.0.0b1 (15 Oct 2019, @ankostis): Rebranded as Graphtik for Python 3.6+¶
Continuation of yahoo#30 as yahoo#31, containing review-fixes in huyng/graphkit#1.
Network¶
- FIX: multithreaded operations were failing due to shared
ExecutionPlan.executed
. - FIX: pruning sometimes were inserting plan string in DAG.
(not
_DataNode
). - ENH: heavily reinforced exception annotations (“jetsam”):
- FIX: (8f3ec3a) outer graphs/ops do not override the inner cause.
- ENH: retrofitted exception-annotations as a single dictionary, to print it in one shot (8f3ec3a & 8d0de1f)
- enh: more data in a dictionary
- TCs: Add thorough TCs (8f3ec3a & b8063e5).
- REFACT: rename Delete–>`Evict`, removed Placeholder from data nodes, privatize node-classes.
- ENH: collect “jetsam” on errors and annotate exceptions with them.
- ENH(sideffects): make them always DIFFERENT from regular DATA, to allow to co-exist.
- fix(sideffects): typo in add_op() were mixing needs/provides.
- enh: accept a single string as outputs when running graphs.
Testing & other code:¶
- TCs: pytest now checks sphinx-site builds without any warnings.
- Established chores with build services:
- Travis (and auto-deploy to PyPi),
- codecov
- ReadTheDocs
v1.3.0 (Oct 2019, @ankostis): NEVER RELEASED: new DAG solver, better plotting & “sideffect”¶
Kept external API (hopefully) the same, but revamped pruning algorithm and
refactored network compute/compile structure, so results may change; significantly
enhanced plotting. The only new feature actually is the sideffect
modifier.
Network:¶
FIX(yahoo#18, yahoo#26, yahoo#29, yahoo#17, yahoo#20): Revamped DAG SOLVER to fix bad pruning described in yahoo#24 & yahoo#25
Pruning now works by breaking incoming provide-links to any given intermediate inputs dropping operations with partial inputs or without outputs.
The end result is that operations in the graph that do not have all inputs satisfied, they are skipped (in v1.2.4 they crashed).
Also started annotating edges with optional/sideffects, to make proper use of the underlying
networkx
graph.REFACT(yahoo#21, yahoo#29): Refactored Network and introduced
ExecutionPlan
to keep compilation results (the oldsteps
list, plus input/output names).Moved also the check for when to evict a value, from running the execution-plan, to when building it; thus, execute methods don’t need outputs anymore.
ENH(yahoo#26): “Pin* input values that may be overwritten by calculated ones.
This required the introduction of the new
_PinInstruction
in the execution plan.FIX(yahoo#23, yahoo#22-2.4.3): Keep consistent order of
networkx.DiGraph
and sets, to generate deterministic solutions.Unfortunately, it non-determinism has not been fixed in < PY3.5, just reduced the frequency of spurious failures, caused by unstable dicts, and the use of subgraphs.
enh: Mark outputs produced by
NetworkOperation
’s needs asoptional
. TODO: subgraph network-operations would not be fully functional until “optional outputs” are dealt with (see yahoo#22-2.5).enh: Annotate operation exceptions with
ExecutionPlan
to aid debug sessions,drop: methods
list_layers()
/show layers()
not needed,repr()
is a better replacement.
Plotting:¶
ENH(yahoo#13, yahoo#26, yahoo#29): Now network remembers last plan and uses that to overlay graphs with the internals of the planing and execution:
- execution-steps & order
- evict & pin instructions
- given inputs & asked outputs
- solution values (just if they are present)
- “optional” needs & broken links during pruning
REFACT: Move all API doc on plotting in a single module, split in 2 phases, build DOT & render DOT
FIX(yahoo#13): bring plot writing into files up-to-date from PY2; do not create plot-file if given file-extension is not supported.
FEAT: path pydot library to support rendering in Jupyter notebooks.
Testing & other code:¶
- Increased coverage from 77% –> 90%.
ENH(yahoo#28): use
pytest
, to facilitate TCs parametrization.ENH(yahoo#30): Doctest all code; enabled many assertions that were just print-outs in v1.2.4.
FIX:
operation.__repr__()
was crashing when not all arguments had been set - a condition frequently met during debugging session or failed TCs (inspired by @syamajala’s 309338340).enh: Sped up parallel/multithread TCs by reducing delays & repetitions.
Tip
You need
pytest -m slow
to run those slow tests.
Chore & Docs:¶
- FEAT: add changelog in
CHANGES.rst
file, containing flowcharts to compare versionsv1.2.4 <--> v1.3..0
. - enh: updated site & documentation for all new features, comparing with v1.2.4.
- enh(yahoo#30): added “API reference’ chapter.
- drop(build):
sphinx_rtd_theme
library is the default theme for Sphinx now. - enh(build): Add
test
pip extras. - sound: https://www.youtube.com/watch?v=-527VazA4IQ, https://www.youtube.com/watch?v=8J182LRi8sU&t=43s
v1.2.4 (Mar 7, 2018)¶
1.2.2 (Mar 7, 2018, @huyng): Fixed versioning¶
Versioning now is manually specified to avoid bug where the version was not being correctly reflected on pip install deployments
1.2.1 (Feb 23, 2018, @huyng): Fixed multi-threading bug and faster compute through caching of find_necessary_steps¶
We’ve introduced a cache to avoid computing find_necessary_steps multiple times during each inference call.
This has 2 benefits:
- It reduces computation time of the compute call
- It avoids a subtle multi-threading bug in networkx when accessing the graph from a high number of threads.
1.2.0 (Feb 13, 2018, @huyng)¶
Added set_execution_method(‘parallel’) for execution of graphs in parallel.
1.1.0 (Nov 9, 2017, @huyng)¶
Update setup.py
1.0.4 (Nov 3, 2017, @huyng): Networkx 2.0 compatibility¶
Minor Bug Fixes:
- Compatibility fix for networkx 2.0
- net.times now only stores timing info from the most recent run
1.0.3 (Jan 31, 2017, @huyng): Make plotting dependencies optional¶
- Merge pull request yahoo#6 from yahoo/plot-optional
- make plotting dependencies optional
1.0.2 (Sep 29, 2016, @pumpikano): Merge pull request yahoo#5 from yahoo/remove-packaging-dep¶
- Remove ‘packaging’ as dependency