Difference between revisions of "Development Branches"

From VistrailsWiki
Jump to navigation Jump to search
 
(22 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This page has been deprecated. We should now use pull requests on github to track branches. Everything still on this page should be moved to github.
== Main branches ==
== Main branches ==


Line 17: Line 19:
== Development branches ==
== Development branches ==
master itself is an integration branch and is not generally stable. However development of complex features happen in their own branches, based off master or a release branch, which are listed below.
master itself is an integration branch and is not generally stable. However development of complex features happen in their own branches, based off master or a release branch, which are listed below.
=== add-gmaps-package ===
Adds a gmaps package that uses Qt's WebKit to render Google Maps in a spreadsheet cell.
=== automatic-loops ===
Execute module once for each input in a list without using a Map. Working. Merged into master.
=== automatic-loops-streaming ===
Based on automatic-loops, allows streaming inputs one at a time through the workflow. Working.
Adds new schema 1.0.4
Merged into master


=== controlflow-fake-signatures ===
=== controlflow-fake-signatures ===
This effort is about having Map work better with the cache and with ModuleSuspended/JobSubmission.
This effort is about having Map work better with the cache and with ModuleSuspended/JobSubmission.


'''controlflow-fake-signatures''' builds fake signatures for the looped module in FoldWithModule. This still doesn't use the cache but makes these signatures unique.
'''controlflow-fake-signatures''' builds fake signatures for the looped module in FoldWithModule. This still doesn't use the cache but makes these signatures unique. Merged into master.


'''jobs-use-signature''' uses the subpipeline signature as the job identifier. Checking for an existing job happens earlier, which allows a module with an already-submitted job to suspend again or finish without running its upstream at all (like persistence does once the file is cached).
'''jobs-use-signature''' uses the subpipeline signature as the job identifier. Checking for an existing job happens earlier, which allows a module with an already-submitted job to suspend again or finish without running its upstream at all (like persistence does once the file is cached). Merged into master.


'''jobs-use-signature_/connect-folded-module''' tries to make FoldWithModule build an actual runtime pipeline (a set of vistrails_module:Module and ModuleConnector objects, not vistrail.module:Module and Connection).
'''jobs-use-signature_/connect-folded-module''' tries to make FoldWithModule build an actual runtime pipeline (a set of vistrails_module:Module and ModuleConnector objects, not vistrail.module:Module and Connection).
Line 40: Line 31:
Next thing to try: make FoldWithModule instanciate a full-fledged pipeline an pass it through the interpreter (like what Group does, except that pipeline would be dynamically generated, and executed several times). There are issues here with logging, since we want to have a LoopExec with the global module IDs in the log, not a WorkflowExec and different (from the dynamic pipeline generation) module IDs.
Next thing to try: make FoldWithModule instanciate a full-fledged pipeline an pass it through the interpreter (like what Group does, except that pipeline would be dynamically generated, and executed several times). There are issues here with logging, since we want to have a LoopExec with the global module IDs in the log, not a WorkflowExec and different (from the dynamic pipeline generation) module IDs.


=== control-parameters ===
=== export-as-script ===
Adds a new type ModuleControlParam for storing controlflow settings on modules. It will be used with automatic-loops-streaming
Work on exporting and importing workflows as Python scripts (using [http://redbaron.readthedocs.org RedBaron])


=== dat ===
=== dat ===
This branch has changes needed by the DAT tool, and that generally makes VisTrails more customizable and useful when used as a library. These changes are intended to be merged back in the main line of development someday.
This branch has changes needed by the DAT tool, and that generally makes VisTrails more customizable and useful when used as a library. These changes are intended to be merged back in the main line of development someday.


=== hash-annotations ===
=== improve-vt-bundle ===
Adds annotations to module signatures
Work in progress on modifying the vt bundle structure to be extensible.  Adds serializer objects to facilitate the serialization of specific pieces of the bundle.
Deprecated. We should use control-parameters instead.


=== java-pkg ===
=== java-pkg ===
Port of the java packages from the 'java' branch. Work in progress.
Port of the java packages from the 'java' branch. Work in progress.
=== job-cache ===
Makes jobs independent of workflows so that they work better as a cache. WIP


=== mac-app ===
=== mac-app ===
Line 86: Line 73:


It is probably required for further work in stopping/resuming workflows, but is a bit complex.
It is probably required for further work in stopping/resuming workflows, but is a bit complex.
=== new-api ===
Rewrites vistrails.core.api as a nice, pythonic, user-friendly API for VisTrails, abstracting away internals.


=== new-matplotlib-pkg ===
=== new-matplotlib-pkg ===
Line 93: Line 83:
Profiles and tries to shorten the loading time of the VTK package.
Profiles and tries to shorten the loading time of the VTK package.


=== parameter-configuration-api ===
=== order-connections ===
Improves the widgets API to allow module developers to choose custom widgets for the ports of their modules (see [https://www.vistrails.org/ticket/699 issue 699]).
Work to make it possible to track the order of connections.  This would allow a ConcatenateStrings module to have a single port that took multiple connections and have the ordering be well-defined and modifiable.


Currently, it is only possible to provide a widget for a ''type'' (all ports of that type will have that widget) or to do limited configuration with entry_type. This branch makes it more extensible.
=== pkg-as-setuptools-plugins ===
Enables VisTrails to load packages from different Python packages, through the setuptools <tt>entry_points</tt> mechanism. This would allow to distribute packages outside of VisTrails easily.


=== persistent_archive ===
=== persistent_archive ===
Line 107: Line 98:


=== python3 ===
=== python3 ===
Python 3 port of VisTrails. Work in progress.
Python 3 support. This branch is archaic (used 2to3) but contain useful information. See [https://www.vistrails.org/ticket/674 issue 674].
 
Proper compatibility will have to wait for unicode to be supported, and __future__ imports to be in place.
 
==== future_imports ====
Adds __future__ imports to work towards Python 3 compatibility. Adds the <tt>add_future_imports.py</tt> script to automatically enable these imports.


See [https://www.vistrails.org/ticket/674 issue 674].
==== [[unicode]] ====
Tries to make VisTrails unicode-safe. This means eliminating bytestrings from the code where unicode strings were meant (PyQt4 already uses unicode). Work in progress.
 
See [https://www.vistrails.org/ticket/673 issue 673].


=== reset-cell-sizes-button ===
=== reset-cell-sizes-button ===
Line 122: Line 121:


Obviously doesn't work across sessions.
Obviously doesn't work across sessions.
=== [[Rewrite Startup Usage|rewrite-startup]] ===
Rewrites the startup logic (configuration and command-line option parsing).


=== richtextcell-xml-xsl ===
=== richtextcell-xml-xsl ===
Line 136: Line 132:
See [https://www.vistrails.org/ticket/732 issue 732]
See [https://www.vistrails.org/ticket/732 issue 732]


Merged in master, not in v2.1 yet.
=== sql-alchemy ===
 
Work to move to the database layer to something more abstract that can support multiple relational backends instead of being coded for mysql or another system.
=== unicode ===
Tries to make VisTrails unicode-safe. This means eliminating bytestrings from the code where unicode strings were meant (PyQt4 already uses unicode). Work in progress.
 
See [https://www.vistrails.org/ticket/673 issue 673].


=== use-uuid ===
=== use-uuid ===
Line 147: Line 139:


See [https://www.vistrails.org/ticket/355 ticket 355]
See [https://www.vistrails.org/ticket/355 ticket 355]
=== version-flag ===
Adds back the --version flag which disappeared with the rewrite-startup effort.


== Unknown branches (fixme?) ==
== Unknown branches (fixme?) ==
Line 154: Line 149:


=== fernando (2010-03-28) ===
=== fernando (2010-03-28) ===
=== improve-vt-bundle ===
=== matplotlib-add-helpers (2014-01-28) ===
=== matplotlib-add-helpers (2014-01-28) ===
=== order-connections ===
=== remove-weakref-descriptor ===
=== sahm-mac ===
=== sahm-mac ===
=== sql-alchemy ===
=== v2.0-prov ===
=== v2.0-prov ===
=== v2.0-with-layout (2012-08-22) ===
=== v2.0-with-layout (2012-08-22) ===
=== vtk-new-package ===


== To be deleted? ==
== To be deleted? ==
* sql-pkg-sqlalchemy: merged (rewrites sql package with SQLAlchemy)
(cleaned 2014-09-78)
* export-versiontree-dot: merged (exports version tree as Graphviz DOT file, [https://www.vistrails.org/ticket/843 issue 843])
* custom-colors: merged (adds hidden option to set custom colors on version tree, for publication purposes)
* export-versiontree-dot_/colors: merged (integration branch for export-versiontree-dot and custom-colors)
* readfile-module: merged (adds a ReadFile module, similar to WriteFile)
* save-module-moves: merged (shows a vistrail as modified when modules were moved, and flushes move actions when saving, [https://www.vistrails.org/ticket/853 issue 853])
* export-cells: merged (adds options to export a single spreadsheet cell to various formats)
* improves-logging: merged (improves core.debug. Backported to v2.1 with improves-logging_/v2.1)
* matplotlib-figure-creation: merged (attempt at fixing [https://www.vistrails.org/ticket/685 issue 685], which made matplotlib incompatible with the multithreaded-interpreter work)
* module-upgrade-recursion: merged (makes UpgradeWorkflowHandler.remap_module work recursively so that upgrades can be chained rather than rewritten; [https://www.vistrails.org/ticket/805 issue 805])
* rename-api: merged (renames some methods from CamelCaseNames to python_style_names)
* tabledata-merge-tabdata: merged (adds the tabdata work from David to Remi's tabledata package)
* ungroup-keep-disconnected-ports: merged (better group manipulation by keeping the InputPort/OutputPort modules when ungrouping, if corresponding group's port were unconnected)
* new-url-package: merged (rewrites the HTTP package as 'URL')
* input-module-no-subclass: merged in dont-use-modules-as-data (disallows connection of any type to Module input ports, only allows Module itself)
* dont-use-modules-as-data: merged (tries to not pass Module subclasses on connections, and to not use 'self' as an output port; [https://www.vistrails.org/ticket/804 issue 804])
* fix-upgrade-test: merged (fixes failing 'upgrade' test, [https://www.vistrails.org/ticket/845 issue 845])
* continue-dialog: merged in v2.1 (adds a module allowing to display a spreadsheet cell in a dialog window, with yes/no buttons)
* debugger: merged (starts the post-mortem Python debugger (pdb) after unexpected error)


== Related projects ==
== Related projects ==

Latest revision as of 13:22, 9 April 2015

This page has been deprecated. We should now use pull requests on github to track branches. Everything still on this page should be moved to github.

Main branches

master

This is the development branch of VisTrails. It's where the features are merged for the next release and tested/bugfixed for a future public version.

v2.1

This is the upcoming version, currently in beta. No more features should be added here, and once we think no more bug fixing is required, it will become the new stable version. Bug fixes will still happen on this branch, and would lead to the release of minor versions (2.1.1, 2.1.2, ...).

v2.0

This is the current stable version. No more features are to be added here. We add there the occasional bugfix, but this should slow down once 2.1 is released.

The list of features that are not present in 2.0 is available on the page New features.

uvcdat-*

These branches are used by the UV-CDAT project. They were forked from VisTrails long ago (Feb. 2012) and there is not intention to merge back, although they live in the same repository.

Development branches

master itself is an integration branch and is not generally stable. However development of complex features happen in their own branches, based off master or a release branch, which are listed below.

controlflow-fake-signatures

This effort is about having Map work better with the cache and with ModuleSuspended/JobSubmission.

controlflow-fake-signatures builds fake signatures for the looped module in FoldWithModule. This still doesn't use the cache but makes these signatures unique. Merged into master.

jobs-use-signature uses the subpipeline signature as the job identifier. Checking for an existing job happens earlier, which allows a module with an already-submitted job to suspend again or finish without running its upstream at all (like persistence does once the file is cached). Merged into master.

jobs-use-signature_/connect-folded-module tries to make FoldWithModule build an actual runtime pipeline (a set of vistrails_module:Module and ModuleConnector objects, not vistrail.module:Module and Connection).

Next thing to try: make FoldWithModule instanciate a full-fledged pipeline an pass it through the interpreter (like what Group does, except that pipeline would be dynamically generated, and executed several times). There are issues here with logging, since we want to have a LoopExec with the global module IDs in the log, not a WorkflowExec and different (from the dynamic pipeline generation) module IDs.

export-as-script

Work on exporting and importing workflows as Python scripts (using RedBaron)

dat

This branch has changes needed by the DAT tool, and that generally makes VisTrails more customizable and useful when used as a library. These changes are intended to be merged back in the main line of development someday.

improve-vt-bundle

Work in progress on modifying the vt bundle structure to be extensible. Adds serializer objects to facilitate the serialization of specific pieces of the bundle.

java-pkg

Port of the java packages from the 'java' branch. Work in progress.

mac-app

Alternative method to create a Mac OS X app bundle for VisTrails. Doesn't use py2app, builds Python as a framework, installs required packages and relinks Qt and vtk using install_name_tool.

TODO: move from shell script to Makefile or Python, make relinking more automatic.

madagascar

Contains a package wrapping madagascar. Basic modules are working but more work is needed.

matplotlib-add-helpers

Improvements to both the matplotlib package and a start at a general framework for semi-automated package generation.

multithreaded-interpreter

This changes the interpreter part of VisTrails to use a task system instead of relying on each module running its upstream counterparts recursively. It allows the use of multithreading at the module level (i.e. running different modules in parallel) and provides this via different schemes. The branch has schemes for threading and multiprocessing (through concurrent.futures) as well as IPython (which replaces the ParallelFlow package).

This is a legacy branch that is still relevant and functional, but is superseeded by multithreaded-interpreter-target-selection which provides more options.

More information is available on its specific page: multithreaded-interpreter.

multithreaded-interpreter-target-selection

It's a variant of the previous branch that changes the way schemes work. There can be different instances of each scheme (named targets), configured by the user, and each module can be assigned to a specific target, allowing fine-grained control of where each module should execute.

It also revamps the XML log and adds an XML configuration file for remote execution.

multithreaded-interpreter-responsive

This brings in background-interpreter, except that it works. The interpreter runs in a different thread, although modules that don't use the @parallelizable decorator will be run synchronously on the UI thread (thus blocking the UI for a time).

See issue 701.

It is probably required for further work in stopping/resuming workflows, but is a bit complex.

new-api

Rewrites vistrails.core.api as a nice, pythonic, user-friendly API for VisTrails, abstracting away internals.

new-matplotlib-pkg

New 'matplotlib' package, that replaces 'pylab'. Has been merged in v2.1 and released. The branch is still used by Sunitha Menon who is making more examples.

optimize-vtk-startup

Profiles and tries to shorten the loading time of the VTK package.

order-connections

Work to make it possible to track the order of connections. This would allow a ConcatenateStrings module to have a single port that took multiple connections and have the ordering be well-defined and modifiable.

pkg-as-setuptools-plugins

Enables VisTrails to load packages from different Python packages, through the setuptools entry_points mechanism. This would allow to distribute packages outside of VisTrails easily.

persistent_archive

Work on a new package to replace persistence.

See issue 755 and the file_archive tool.

pyside-compatibility

This branch aims at making VisTrails compatible with both PyQt4 and PySide. PySide currently crashes.

python3

Python 3 support. This branch is archaic (used 2to3) but contain useful information. See issue 674.

Proper compatibility will have to wait for unicode to be supported, and __future__ imports to be in place.

future_imports

Adds __future__ imports to work towards Python 3 compatibility. Adds the add_future_imports.py script to automatically enable these imports.

unicode

Tries to make VisTrails unicode-safe. This means eliminating bytestrings from the code where unicode strings were meant (PyQt4 already uses unicode). Work in progress.

See issue 673.

reset-cell-sizes-button

Adds a button to stretch the spreadsheet cells. Intended as a fix for 2.1 until the resizing code is fixed.

Currently missing an icon.

See issue 833.

resume-suspended-pipelines

Changes the interpreter to not clean not-cacheable modules from the persisted pipeline if the execution was suspended. This means that the next execution will resume close to where the last one stopped.

Obviously doesn't work across sessions.

richtextcell-xml-xsl

Adds an XSLCell module that shows a XML document formatted via an XSLT stylesheet.

See issue 292.

scripting-support

Work attempting to make it possible to convert from scripts to workflows and vice versa. Based off rename-api.

See issue 732

sql-alchemy

Work to move to the database layer to something more abstract that can support multiple relational backends instead of being coded for mysql or another system.

use-uuid

Use UUIDs for identifiers instead of incrementing integers. This would make things like merging vistrails or dealing with distributed versions of the same vistrail much easier (e.g. crowdLabs, subworkflows).

See ticket 355

version-flag

Adds back the --version flag which disappeared with the rewrite-startup effort.

Unknown branches (fixme?)

easy-install (2012-11-21)

Emanuele's branch trying to get easy-install to work on both user/system installations. We have now moved to using "pip" but some of this may still be relevant.

fernando (2010-03-28)

matplotlib-add-helpers (2014-01-28)

remove-weakref-descriptor

sahm-mac

v2.0-prov

v2.0-with-layout (2012-08-22)

To be deleted?

(cleaned 2014-09-78)

Related projects

These are projects part of the VisTrails effort that we are working on, but that are not technically a part of the VisTrails repository.

DAT

The Data Analysis Tool is inspired by UV-CDAT. It has a similar interface meant to create visualizations from Plots and Variables connected together by drag and drop, generating the VisTrails workflows on-the-fly. It lives in vgc.poly.edu:src/dat.git. The changes it requires on VisTrails have not been fully merged in and are in the dat branch.

file_archive

This is a file store with metadata that is meant to replace Git as the backend of the persistence package. It is developed as a stand-alone tool and library, allowing to search for and get files that were generated from VisTrails without having to go through its UI. An initial design document is on the Archive page.

See the Github project, issue 755 and branch persistent_archive.