VisTrails Home

Development

From VisTrailsWiki

(Difference between revisions)
Jump to: navigation, search
(Apr 1, 2011)
(October 30, 2017)
 
(1775 intermediate revisions not shown)
Line 1: Line 1:
-
== Roadmap ==
+
<div class="nonumtoc toclimit-1">
 +
__TOC__
 +
</div>
 +
See also the [https://github.com/VisTrails/VisTrails/wiki/Development Github wiki]
-
== Weekly Meetings ==
+
== 2017 ==
-
===Apr 1, 2011===
+
=== October 30, 2017 ===
-
* Workflow Execution Explorer demo (Tommy)
+
'''Updates'''
-
* 1.6.2 release (mainly with bug fixes) (Emanuele)
+
-
* VisMashup support in VisTrails discussion (Emanuele)
+
-
===Mar 22, 2011===
+
* [DK] PyQt5
 +
* [RR] Running Python3 code
-
* support a subset of VisMashup within VisTrails: re-use subworkflow infrastructure, and add the functions which create the application. Similar to subworkflows, mashups could live inside a vistrail
+
'''Items to Discuss'''
-
** [DK] Note that mashups are higher-level objects than workflows.  It would seem that mashups should include vistrails.
+
-
** [JF] My idea is really simple: when we create a subworkflow we are essentially creating a 'view' that hides away the details of the underlying modules and only exposes a subset of the parameters. What I would like to have is the component of VisMashup that takes a subworkflow specification and generates the app.  This would greatly simplify the publication of *interactive* results. And I believe, it would be simple to implement since it would re-use the existing infrastructure we have for subworkflows.
+
-
* support for global variables: maybe, an initial approach could be to have such variables associated with a vistrail. This would greatly simplify packaging of workflows, in particular, to deal with file locations in different systems
+
* Installation:  
-
** [DK] We have discussed this before as having '''workflow'''-global parameters and have proposed a method for accomplishing this---namely modify the interface to hide global parameters.  It may be possible to have them for entire vistrails, but note that this poses problems. Suppose a user that creates a workflow identifying 'x' (a Float) as a global variable, and then changes the workflow and identifies a new parameter as 'x' but its type is now a String.  A per-workflow definition of the global 'x' is fine, but a per-vistrail definition is not.  I agree that we should try to support some style of alias-like behavior so that a user can set parameters more globally, but it is important that these settings are saved as provenance as many of these might be per-session...
+
** Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
-
** [JF] I want global variables to simplify packaging of results. Having them on a per workflow basis would not solve the problem I had: In the WikiQuery case study I have a series of related workflows that all rely on the same set of input files; so in this case, I would need global variables at the VisTrails level.
+
** Update conda recipe (pyqt4 restriction for now)
-
** [JF] As for the issue DK raised, I think we could have a very simple solution: we check for conflicts with global variables and prevent the used from making mistakes
+
* Interpreter
 +
* Release
 +
* Integration
 +
** hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
 +
** independent pieces of the GUI could be reused if they were more independent.
-
* SubWorkflows in the database (Tommy)
+
=== September 6, 2017 ===
-
** abstractions==subworkflows? Some error messages still refer to abstractions
+
-
** Logic for merging subworkflows from different sources? (Upgrading issues?)
+
-
* Control Flow Assistant feedback (Tommy)
+
-
* From Matthias:
+
-
** Remote execution: make it easier to move executions to clusters, starting them from VisTrails, waiting for the results.  Can we make Joel's code for the wildfire simulations more flexible?
+
-
** Using VisTrails for validation: make it possible to take results of a workflow, and compare them with a previous result to validate that changes in code or VisTrails have not affected the results.
+
-
** Make it easier to select a persistent repository (test, release, etc.) so that the same workflow can be run against different repositories
+
-
** Better support for querying hundreds of data files (ie create a database of metadata that a user can use to filter the set of files to locate specific files quickly)
+
-
** Along with this, allow this to work over existing (non-persistent) repositories.
+
-
** A better python editor for PythonSource
+
-
===Mar 15, 2011===
+
'''Updates'''
-
* database to file: works for log now
+
* [RR] Fixes for scikit-learn, tables, usagestats
-
* database: save subworkflows to the database
+
-
* add uuids for vistrails, subworkflows, merges
+
-
* merging: same uuid and data changed
+
-
* look into distributed algorithm
+
-
* merge should possibly include logs
+
-
* allow additional metadata to be controlled, possible to have a 1-1 correspondence
+
-
* speedup on database queries to speed up joins over entities, combine different attributes
+
'''Items to Discuss'''
-
* loops on module executions
+
-
* traversing module executions
+
-
* build transitive closure
+
-
* build table to link the workflow_exec with module_exec
+
-
* fix for users-list problem with Embed GUI issue
+
* Stripped-down version of VisTrails to showcase new features?
-
* how to handle users that want to contribute code or examples
+
** include meta-vistrail, bundles
 +
** hide mashups, extra widgets, latex mode, etc.
 +
* Interpreter
-
* [Dan] committing control flow assistant soon, issue with database version (see trac commit from today)
+
=== August 16, 2017 ===
-
===Mar 8, 2011===
+
'''Updates'''
 +
* [DK] meta-vistrail
 +
** Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
 +
** Interface
-
* DB speedup finished? [Tommy]
+
'''Items to Discuss'''
-
** Speedup done.
+
* Notebook
-
** Looking into issues with saving log
+
* Interpreter
-
** Execution Provenance API
+
* Web visualization packages (e.g. bokeh)
 +
** [https://github.com/VisTrails/VisTrails/pull/969 PR 969] has more details
 +
* Pipeline templates
-
* Control flow [Dan]
+
=== August 2, 2017 ===
-
** Fixed issues
+
-
** Input List Override working
+
-
** Cartesian product now working
+
-
** Error message with version tree
+
-
** Weird issue with the group executing before the map starts
+
-
* Documentation [Claurissa]
+
'''Updates'''
-
** check in guide to git
+
* [DK] meta-vistrail work
 +
** Extend Vistrail to MetaVistrail to support meta-actions
 +
** Vistrail becomes less tied to workflows so it can also supporting versioning vistrails
-
===Mar 1, 2011===
+
'''Items to Discuss'''
-
* [[Documentation Log]]
+
* Interpreter
 +
** Using dask?
 +
** Streaming an issue
 +
* Notebooks
 +
** What is an output of a module? (need to expose a specific output port or the entire module)
 +
** Unifies some of the work from the script-workflow code
-
* Updating the relational DB (Tommy)
+
=== June 28, 2017 ===
-
** Import vistrail from DB is now very fast.
+
-
** Mostly done with export vistrail to DB. Still a few issues.
+
-
=== Feb 22, 2011 ===
+
'''Updates'''
 +
* [DK] Testing UUID/SQLAlchemy branch
 +
* [DK] Working on meta-versions
 +
** would be nice to have pipeline and vistrail separate
 +
** [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
 +
** [DK] Yes?
 +
* [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs
-
* Python Source widget (Emanuele)
+
'''Items to Discuss'''
-
* News: Claurissa will help us with documentation. She will start by translating the existing user guide into Sphynx, and then move on to updating the content as well as documenting the many features that are currently not described in the manual.
+
=== June 7, 2017 ===
-
** Please add any undocumented features as well any (informal/incomplete) descriptions you might have to the following page: [[Undocumented Features]]
+
-
** focus issue (will ask Huy if he knows a good solution for this)
+
'''Updates'''
 +
* [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs
-
* Browser updates (Tommy)
+
'''Items to Discuss'''
-
** project browser, workspace
+
* [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
 +
** Can designate in module description an output port
 +
** Value has to be serializable
 +
** Should use the OutputModes work to handle multiple modes, configuration, selection, ...
 +
* [RR] Is VisTrails with SQL usable concurrently?
 +
** [DK] Probably not, demo from [TE] exists
-
* Updating the relational DB (Tommy)
+
=== May 10, 2017 ===
-
** insert size: Tommy tested with 60,000 inserts
+
-
** loading: multiple selects and retrieve them all at once
+
-
** incremental updates, is_dirty, is_new
+
-
** no need to bump the schema version
+
-
** start thinking about better ways to do queries
+
-
** can prepare statements via low level mysql commands
+
-
* Loop Assistant (Dan)
+
'''Items to Discuss'''
-
** add tutorial for this loop
+
-
* mailing list emails
+
* Identifiers
-
** Dan: keep track of which emails and update the FAQ with answers
+
* Bundles & DB Code
-
** Could think about putting this into a system eventually, but for now it's low-traffic enough
+
* Interpreter
 +
* Merge / Versioning Version Trees
 +
* Python3
 +
* Workings of List
 +
* Packages: Wrappers + new Versions (VTK7)
 +
* Script Conversion
 +
* Intermediate Outputs: reuse output module work?
 +
** would be useful for both workflows (run to a particular checkpoint)
 +
** and useful for notebook interfaces
 +
* SQL backend: does it allow for concurrent access?
 +
** Yes, real-time collaboration work for concurrent single vistrail access
 +
** Should be able to access different vistrails at once without issues
-
=== Feb 8, 2011 ===
+
=== April 19, 2017 ===
-
* LaTex extension demo (Emanuele)
+
'''Updates'''
-
** New embed dialog
+
'''Items to Discuss'''
-
* Non-modal PythonSource windows ([https://vistrails.sci.utah.edu/ticket/408 link to discussion])
+
=== April 5, 2017 ===
-
** make it non-modal, single window
+
'''Updates'''
-
** update selection on click
+
-
* Where to keep User's guide and other documentation
+
'''Items to Discuss'''
-
** Sphinx, commit to doc directory
+
=== March 8, 2017 ===
-
* Log Explorer demo (Tommy)
+
'''Updates'''
 +
* [RR] Added a looping indicator on modules [https://github.com/VisTrails/VisTrails/pull/1217 #1217]
 +
** This makes it easier to understand what's looping, no need to follow edges and count
 +
* [RR] Refactoring CLTools for integration in Docker package
 +
* [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)
-
* [[Log_API]] RFC (Tommy)
+
'''Items to Discuss'''
 +
* [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers [https://github.com/VisTrails/VisTrails/issues/1216 #1216]
 +
** Can we move this up to the renderer modules?
 +
** Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)
-
* Control Flow Assistant (Daniel)
+
=== February 24, 2017 ===
-
=== Feb 1, 2011 ===
+
'''Updates'''
 +
* [RR] Rich-text module documentation ([https://github.com/VisTrails/VisTrails/pull/1210 #1210])
 +
** Went over packages and fixed format; maybe we should have a test to validate RST?
 +
* [RR] Single-instance shutdown with <tt>--remote-shutdown</tt> ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
 +
* [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment
-
* Execution Provenance View (Tommy)
+
'''Items to Discuss'''
 +
* [RR] There are issues with list-depth
 +
** Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
 +
** Issue with looping groups
-
* Control flow assistant (Daniel)
+
=== January 25, 2017 ===
-
* Script Interaction (Emanuele)
+
'''Updates'''
-
** CDAT feature that might be useful in other settings
+
-
* Port/parameter management (Dave)
 
-
** moved close button
 
-
** what about NULL parameters?
 
-
** global parameters
 
-
=== Jan 18, 2011 ===
+
'''Items to Discuss'''
-
* VisTrails browser (Tommy)
+
* Scaling VisTrails
-
** demo
+
* sql-alchemy, uuid, bundle branches/merges
-
** can import files and directories to the list of files
+
-
** can see and open vistrails, workflows
+
-
**  need to think about how to show execution information
+
-
* Control flow assistant (Daniel)
+
=== January 11, 2017 ===
-
** integration with parameter exploration
+
-
** can re-use infrastructure with parameter exploration
+
-
** can keep provenance for parameter explorations for free if we use the looping constructs
+
-
* New port/parameter management (Dave)
+
'''Updates'''
-
** demo
+
* [RR] New VisTrails version in Debian [https://packages.qa.debian.org/v/vistrails.html]
-
** ordering?
+
-
** ability to filter, sort parameters
+
-
** documentation
+
-
** required ports
+
-
** update modules to show when parameters are set on ports
+
-
** talk to Matthias
+
-
* 1.6.1 Release
+
'''Items to Discuss'''
-
** tomorrow, bug fixes + executable papers
+
* [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
-
** create tickets
+
** Have homemade workflow system based on R scripts and symlinks
 +
*** Each steps has multiple sets of parameters
 +
*** Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
 +
*** Gets all results in directory structure <code>step3-parameters/step2-parameters/step1-parameters/files</code>
 +
** VisTrails improves provenance capture and makes editing pipelines easier
 +
*** Need better support for aggregation (groupby operation might be good enough; pandas?)
-
*
+
== 2016 ==
 +
=== December 14, 2016 ===
-
=== Jan 11, 2011 ===
+
'''Updates'''
 +
* [RR] Single-instance code: over-engineered, but will work
 +
** (Have to avoid race conditions on startup, and OSes offer very different locking primitives)
-
* Release (Emanuele)
+
'''Items to Discuss'''
-
** 1.6 released 1/7/2011
+
* Email from Matt Dirks: combobox should work, ask for more code?
-
** there is a branch on git
+
* Email from marin.nl:
-
** nightly builds
+
** looks like they would benefit from new interpreter
-
** bundling ALPS python support into releases
+
** how to store their consolidated provenance when they exchange .vt files around?
-
* Control flow assistant (Dan)
+
=== November 30, 2016 ===
-
** focus on supporting the Map module
+
-
* VisTrails browser/repository (Tommy)
+
'''Updates'''
-
** open a vistrail get included in an index
+
-
** index includes vistrails, workflows, and executions
+
-
** use thumbnails for showing the workflows
+
-
** in the future, sort by projects
+
-
* Port and parameter updates (Dave)
+
'''Items to Discuss'''
-
** Colin Talbert at USGS-Fort Collins requests the same global parameters feature that Matthias has already mentioned.  We should start working on this functionality.
+
* [RR] Working on single instance code for MARIN ([https://github.com/VisTrails/VisTrails/pull/1205 #1205])
-
** Colin also asked about highlighting required ports
+
** Remote shutdown ready ([https://github.com/VisTrails/VisTrails/pull/1204 #1204])
-
** Changes to schema: add required flag, cardinality flag, and global parameters
+
** Still improving it
-
** Changes to execution: way to add global parameters and connections to workflow before execution begins (not display visually)
+
** Want to return something more structured to the client (success/fail code, stdout, stderr)
-
** Changes to interface: method to select ports and tie them to global parameters, global parameter display, highlight for ports to show required (red?), do we need to show cardinality?
+
** Should be usable without Qt
 +
* [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
 +
* [DK] Working on batch execution using console_mode
 +
* [DK] Settings aliased parameters when using core.api?
-
* Executable paper tasks
+
=== November 16, 2016 ===
-
=== Dec 21, 2010 ===
+
'''Updates'''
-
* saving parameters in a module (email from Tom Maxwell, from NASA)
+
* [RR] Issue [https://github.com/VisTrails/VisTrails/issues/1200 #1200] from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)
-
* explorations still do not detect MplFigureCell modules inside groups or subworkflows
+
-
=== Dec 14, 2010 ===
+
'''Items to Discuss'''
-
* [ES] Unit tests
+
* Email from Pieter: executing pipelines with parameters from command-line using single-instance
-
** Ran unit tests last week
+
** Seems to work, main instance gets full command-line and alters configuration temporarily
-
** Errors with old versions of pipelines, need to fix pipelines
+
** Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
-
** console_mode_test can't be enabled
+
** Actions:
 +
*** Can add switch to kill main instance
 +
*** Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
 +
*** Add client/server code to <tt>core</tt> that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like <tt>gui.application.VistrailsApplicationSingleton#noninteractiveMode()</tt>)?
-
* Data publishing with CrowdLabs
+
=== November 2, 2016 ===
-
** using persistent files and directories?
+
-
** couldn't think about how to synch user directories with server directories
+
-
** look into how to push data files across with git
+
-
** how to organize data (ie persistent files) from a specific workflow or all workflows with specific parameters
+
-
* [DR] Subworkflow upgrades: update--is it a wrap?
+
'''Updates'''
-
** functionality are completed
+
-
** one cosmetic issue -- disabling a package
+
-
** subworkflows move from top to bottom of the list
+
-
** [DR] Add documentation to wiki on subworkflow upgrade paths
+
-
* [DR] control flow assistant
+
* New version 2.2-2 of Debian package, still [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=796994 wrong copyright], and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748692 broken bundle installation]
-
** started looking into this
+
** Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
 +
* Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)
-
* [TE] [[Data/provenance browser]]: update
+
'''Items to Discuss'''
-
** looking at the old code
+
-
** some problems with the logs
+
-
** cannot find parent workflow
+
-
** adding project annotations to sort VisTrails into different projects, different vistrails in different project, need to have gui to do this
+
-
** synchronize when vistrail changes, especially with execution logs, create index schema for these files (vistrail and workflow exec)
+
-
** persistent data index, need to integrate eventually, scan through actions to generate list of files used
+
-
* PythonSource editor
+
=== October 5, 2016 ===
-
** seeing which PythonSource editor is open
+
-
** need to be able to have "save now" button or automatically save when someone presses "execute"
+
-
** generate id tag automatically, close the other window immediately
+
-
** highlight module being edited/configured, keep window open (like console, messages)
+
-
* Lack of global variables and bookmarks
+
'''Items to Discuss'''
-
** don't need to reconfigure executions to do this
+
-
** just don't display the "global" modules and connections
+
-
* [CS] look into 3 items for [MT]'s group: ports display, python source, global vars and bookmarks (and persistent files dump)
+
* working toward a 3.0 release; [https://github.com/VisTrails/VisTrails/projects/1 project board]
 +
** unique ids
 +
** bundles
 +
** GUI fixes
 +
** wrapping updates
 +
** python3
 +
** interpreter?
-
* [PM] Merging master and crowdlabs git branches. How should we merge the crowdlabs branch into the master branch?
+
* command-line input parameters to provenance
-
** problem is that there are changes in crowdlabs that can't go in the master branch (i.e. HTTPFile)
+
** [DK] we don't create a persisted workflow from those input params, i think
-
** [ES] just merge both so we don't have the crowdlabs branch
+
** [DK] these are already defined as aliases so we treat them differently form a normal input param
-
** [PM] change RepoSync to have a server config preference that is set to True for server, False for client, and RepoSync has an if in the compute that does one thing if server is true, another if server is false
+
** [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
-
** have a branch for workflow indexing so that patch lives in another place
+
** Do the vistrails variables values get stored in provenance? (what if values change?)
-
* Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.
+
=== September 14, 2016 ===
-
** Currently, non-optional ports are square, shown. Optionals are circles and must be enabled
+
-
** Add a third category for mandatory, problem is backward compat.
+
-
** [ES] Colors are already overloaded, not for ports, use shading instead?
+
-
** Also needs to be with global settings, etc.
+
-
* 1.6 release: [ES] already has 1.6 candidate
+
'''Updates'''
-
** don't include the new gui features yet
+
-
** freeze 1.6, just bug fixes
+
-
** merge crowdlabs branch
+
-
* [DK] persistent files on the server
+
'''Items to Discuss'''
-
** [MT] has problems with files on server, issue with /tmp/, running older python (2.5)
+
* [RR] [http://aosabook.org/en/vistrails.html VisTrails] in [http://aosabook.org/ The Architecture of Open Source Applications] Vol I (2012)
-
** stress test, test on [DR]'s machine
+
** Did we know/contribute to this?
 +
*** [DK] Yes. We wrote the text for the VisTrails chapter
 +
** Drafted response
 +
*** [DK] Looks good, will forward.
-
=== Dec 7, 2010 ===
+
=== August 31, 2016 ===
-
* [DR] Subworkflow upgrades
+
'''Updates'''
-
** Made sure that we don't rewrite actions but create new actions that show where subworkflows are upgraded
+
* [RR] Went over vgc.poly.edu/vistrails.org config:
-
** Document what is implemented and how it works (at high level), also what isn't yet implemented
+
** Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
 +
** Fixed usersguide scripts (won't disappear again)
 +
** Moved some websites over to HTTPS
 +
** [https://docs.google.com/document/d/18YZewjR2YY9cQ9pzdA-nU9jbu2L7jYcPlDi3iaPadwA/edit Websites]
 +
* [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs ([http://vistrails.readthedocs.io/ currently there]) and users' guide under the same RTD "project"
 +
* [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that
-
* [TE] Errors and web services
+
'''Items to Discuss'''
-
** allow user to decide to only display errors to messages window instead of also showing popup windows
+
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
-
** Allow add/delete of web services from GUI without going to preferences
+
** Get rid of more services? (svn, trac, ...)
-
** Qt dependencies for error messages gone
+
* [RR] Notebook execution works. How do we support more languages (than Python)?
 +
** Need: detection, metadata reader, code to input values
 +
* [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?
-
* [PM] Some specific questions
+
=== August 17, 2016 ===
-
** Question: Should v1_0_1 MySql schemas be upgraded to v1_0_2?
+
-
** db_connection.begin() being removed, stale results
+
-
** memory size is too big for the field size
+
-
** [DK] check on schema changes, see what is important for changes
+
-
* Question: do we display 'warnings' or only errors?
+
'''Updates'''
-
** Need to write ModuleErrors (maybe make ModuleWarnings) to the messages (debug) as well as adding as tool tips
+
* [TE] Fixed [https://github.com/VisTrails/VisTrails/commit/2cd42d156243067a7663bb0bf0156e5d87188c2f parameter order upgrade bug] that breaks VTK examples.
 +
* [TE] Matplotlib and sklearn are exportable using module-to-script api
 +
* [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
-
* Some suggestions/requests by Matthias:
+
'''Items to Discuss'''
-
** Support the grouping of vistrails and files into projects
+
* Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
-
*** Annotation here?
+
* Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
-
** Ability to set global parameters/bookmarks=workflows (e.g., a filename or path)
+
* [TE] Wrap-up notes
-
** Archive results -- create a big tar/zip file with all vt and results
+
-
** Ability to add a vt: if a node is deleted, all data associated with that should also be deleted
+
-
** Create a test suite: run a workflow, save and archive results (user could specify which results should be saved)
+
-
* [[Data/provenance browser]]
+
=== August 10, 2016 ===
-
** We need a better mechanism that allows users to search/query for workflows and provenance, for example search by project, parameter names and values
+
-
** One way to view this is to allow the creation of mappings between the vistrails/projects into different structures---like "smart folders", e.g., /projectN/vt1/wf3/exec23/O1....
+
-
** Support editing and synchronization in both ways (if edit smart folder, should update vt)
+
-
** Should be able to script the use of the browser, e.g., to test a new version of a workflow against a saved test suite
+
-
** Scientists like a directory structure and can go browse there like they browse files with finder/explorer
+
-
** Keep mappings from our datatypes to file structures
+
-
** Need an exportable archive (to send with a paper, allow browsing outside of VisTrails)
+
-
* On merging master and crowdlabs git branches
+
'''Updates'''
-
** some updates on the crowdlabs branch need to be merged/patched into the master branch
+
-
** how should this be done?
+
-
* PythonSource editor
+
'''Items to Discuss'''
-
** modal or multiple window editors?
+
* [TE] Python Wrapper and scripting
 +
** All changes merged into python-wrapper branch
 +
*** Tests pass
 +
*** Still need to update sub-branches
 +
** Minor fixes
 +
*** vtkPlot3DReader upgrade bug
 +
*** Added tuple exporting
 +
** Most of vtk examples can be exported [https://gist.github.com/rexissimus/5ac7374cde8525d84f9443f019d234e2 example]
 +
*** Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
 +
**** Could be a bug in the old vistrail, but not sure
 +
* [DK] Subworkflows and uuids
-
* Lack of global variables and bookmarks
+
=== August 3, 2016 ===
-
** Problem is that aliases don't change workflow specification
+
-
** Need to store these values in the execution log
+
-
** Keep alias dictionary for everything
+
-
** [PM] Not a need for global variables above the single workflow level
+
-
** This already exists at the conceptual level---one module with connections to other modules, this just is cluttered
+
-
* Unit tests
+
'''Updates'''
-
** [ES]: I ran the test suite on current master branch and 207 tests are executed of which 11 fail. Some of the tests are failing because console_mode_test package can be imported anymore (this is a package created specific to run console_mode tests).
+
-
** Most of the other problems are issues with
+
-
  InvalidPipeline: Pipeline cannot be instantiated:
 
-
  Missing version 0 of package <package>
 
-
* Data publishing with CrowdLabs
+
'''Items to Discuss'''
-
** Using git to manage data between local and server data stores might be too messy.
+
-
*** Does one create a new git repository for each vistrail on crowdlabs?
+
-
*** The local persistent git repo holds all persistent data used on that machine. How does one push/publish just one data product using git?
+
-
*** Can anybody think of a plausible git workflow to do this?
+
-
** Perhaps we should just look into adding directory support for RepoSync module along with a better data management UI.
+
-
** Are there OS independent file transfer protocols better than HTTP? Something like rsync might work nicely, but windows support is limited.
+
-
=== Nov 30, 2010 ===
+
* [TE] Wrapper patches now stored as [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/vtk_patch.py python code]
 +
** Connection variables now lowercase
 +
** Can we use code as output variables?
 +
*** E.g. "vtk_dataset_mapper.GetOutput(0)"
 +
*** Could mutate output when output method is only supposed to be called once
 +
*** Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
-
* Subworkflow Upgrades (Daniel)
+
=== July 27, 2016 ===
-
** Should be completed finished tonight
+
-
** Local subworkflow upgrades should be working now
+
-
** Need to test these after the checkin
+
-
** Using a dummy controller
+
-
* Assistant for control package (Daniel)
+
'''Updates'''
-
** Look at examples
+
-
** Try to make an assistant to modify workflow using currently selected modules as looping group
+
-
* [[Improve error handling/reporting]]
+
* [DK] DB bundle serialization implemented
-
** Tons of message boxes pop up when a workflow has a bunch of errors, UI response is bad after this...
+
-
** Update existing error messages
+
-
** Show Details rendering on Mac
+
-
** What to do when multiple errors occur in a single event?
+
-
** New dialog that shows "next message" but allows user to dismiss messages or view all in the messages window
+
-
** Color coding for messages: use gui/theme.py to configure: maybe try black for warning, gray for log level messages
+
-
* Data and workflow browser (Nivan will give a status update)
+
'''Items to Discuss'''
 +
* [TE] Finished using patch in script [https://github.com/VisTrails/VisTrails/tree/python-wrapper-script-patches python-wrapper-script-patches]
 +
** locale?
 +
** [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d new patching example] vs [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 old patching example]
 +
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/packages/vtk/vtk_wrapper/parse.py#L135-L273 Example of new patches]
 +
** Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
 +
** [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/api.py New api for executing standalone modules]
 +
*** Accessing module was disabled for core/api, but a different api for this is probably better anyway.
 +
*** api.Package now gives [https://gist.github.com/rexissimus/f0d6b026aa5a2a770847f3321744645d#file-new_patching-py-L26 directly callable module compute functions]
 +
*** Only works on modules not using interpreter functionality ('.interpreter', etc.)
 +
**** for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
 +
** Fixed more renaming issues - [https://github.com/VisTrails/VisTrails/blob/python-wrapper-script-patches/vistrails/core/scripting/scripts.py#L63-L64 function argument names should not be replaced]
 +
** Still need to do something nicer for the patch formatting
 +
*** patch keys currently contain non-python-variable characters but this can be fixed
 +
*** Need to replace string template with python variables that is replacable using RedBaron
 +
**** Patches could then look like:
-
* Saving Explorations (Feature Request from ALPS)
+
    def vtkInstance():
-
** We already save parameter explorations with the workflow in the version tree. As is, we save only the last exploration that was done for a given workflow.  If you ship the vistrail, the saved parameter explorations will be there.  Is this sufficient?
+
        self.vtkInstance = self
-
** [MT] Yes, that takes us a long way. Could one add a button in the explorations to save a certain set of explorations as a new version, like you do with the camera angle in the spreadsheet? That way one could store more than one set of exploration parameters without forcing a new version to be created with every change?
+
-
** Eventually we should provide a selector in parameter exploration
+
-
** Fastest to just allow a null action new version (a bit clunky) but wouldn't require too many changes
+
-
* Clean up/prune persistent directory
+
=== July 20, 2016 ===
-
** What are the concrete use cases for this?
+
-
** [MT] Here are two concrete use cases that appeared over the last week:
+
-
*** a project is done, the important results copied to an archive and I want to get rid of all its intermediate files, etc., to save space - but not those files associated with other projects. Or, similarly, I'm done testing some workflows, want to remove all (partially buggy or irrelevant) files, and then start with production runs.
+
-
*** there was a problem in a workflow(e.g. a bug in HTTPFile module with binary files or a bug in some user code). I want to force a certain persisted directory to be "forgotten", so that I can repair the wrong files. Bumping the version number of the ALPS package would be one way, but that is overkill since it would immediately invalidate all persisted files. Removing the whole persistence directory is just as radical.
+
-
** just testing simple tutorials with small files leaves me with 2 GB of persistent data after 10 days, in 142 persistent directories or files. I need some way to clean this up that is less radical then removing all persistent directories and files.  This could be done, by e.g. removing all those which do not occur in tagged or leaf workflows in a certain set of vistrails. Or marking those which occur in certain vistrails with certain labels, and then being able to prune all those with or without some label/tag. Another option might be having the ability to choose from various repositories.
+
-
** [DK] follow up on this
+
-
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
+
'''Updates'''
-
** work in progress
+
-
* Data Publishing with crowdLabs. (Phillip) 
+
'''Items to Discuss'''
-
** Do we want data publishing capabilities in VisTrails and crowdLabs?  '''YES!!!'''
+
* [TE] New patch method for python-wrapper [https://gist.github.com/rexissimus/009b303e4cadcd4b8f2fec6cf848cb2b example]
-
** If so...
+
** A patch is a string template with self/input/output variables for patching a class method
-
*** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
+
** Patches are stored in xml specification
-
*** Persistent Data works for experimental data. Can we use it for publishing data?
+
** Patches are assigned to methods in ClassSpec's
-
**** Publishing data or storing data on the server and pushing back and forth?
+
** Used both for execution in vistrails and when exporting as script
-
**** What is the current scheme for identifying data on the server?
+
*** Executed with "exec" when run in vistrails
-
**** Do we need to do better than the ids, versions, hashes that the peristence package provides?
+
*** Turns patches into natural calls in script
-
*** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?
+
** Wrapping VTK library no longer needed
-
** [ES] Feature request: Can RepoSync support a directory of files?
+
*** Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
-
*** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.
+
** Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
-
** persistence used more for exploratory
+
** Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
-
** want to focus on more archive
+
** TODO's
-
** move from exploratory data to published
+
*** Add to export_as_script
-
** Dataverse: essentially a site to curate data, archive data, host their own site, each university has their own server, linked together
+
*** Patch Modules that represent Python functions as well
-
** persistent store integration: maybe use git to do this, can add to local repositories to move to git
+
-
** use some of the dataverse: standard hash schemes, etc.
+
-
** privacy: can users push workflows without exposing data
+
-
** permissions for who can check things out: download permissions, upload permissions
+
-
** look into setting up persistent store on
+
-
* DisplayWall (Wendel)
+
* [vistrails-users] Read data from SQL example?
-
** new version of vistrails receives rotate/scale messages much more slowly
+
-
** using the vtk version that is already there
+
-
** comparing VisTrails 1.2 with nightly src
+
-
** test when the messages are received on the clients
+
-
** [ES] Changed something in the VTKCell, using the Qt interactor from vtk now, just uncomment this (vtkcell.py file)
+
-
** some commands are missed; check why this is happening
+
-
** preferences switch? 2x2 not 2x3
+
-
** check what modifications exist in the client code that's not exactly 1.2?
+
-
** new version of vtk package changed some things
+
-
** update vtk upgrade: resetcamera and addrgbpoint => _# versions
+
-
** Demos on displaywall: Wendel has to keep both VisTrails running and stop work when visitors stop by unannounced
+
-
* Relative paths in workflows, how to deal with them? (Emanuele)
+
=== July 13, 2016 ===
-
** [ES] Updated all examples to use HTTPFile for these
+
-
** some of these had to change completely (color widget), kept tree the same format
+
-
=== Nov 16, 2010 ===
+
'''Updates'''
-
* Data and workflow browser (Nivan)
+
* [DK] bundles on sql-alchemy branch
-
** Check into git branch (Dave)
+
** Bundle, BundleMapping, and BundleSerializer
-
** Add data support to the vistrails, workflows, and execs already exposed in the browser
+
** New BaseSerializers to deal with versions of directory, zip, db serializers
-
** Add thumbnails to the browser
+
** Can register the BundleSerializers with the base serializers
-
** Bittorrent support?
+
** Translation between bundle versions
 +
** Bundle.add_object "just works" if an appropriate mapping was registered
-
* Upgrade issues -- a wrap? (Daniel)
+
'''Items to Discuss'''
-
** Fixing some issues with namespaces
+
* [TE] Simplified running module using API for scripting [https://gist.github.com/rexissimus/b28d83ae516a519167f5751bb8ab3739 example]
-
** Upgrades for subworkflows that are included with packages, need to save these upgrades with the vt file as a "local abstraction" to ensure provenance.  May be able to tie these back to the package versions once the package is upgraded.
+
** API Module can be executed using '''compute''' method
 +
*** Takes list of outputs to compute + inputs as keyword arguments
 +
* Status of general cell synchronization (vistrails-users)
-
* Test Suite
+
=== July 6, 2016 ===
-
** Matthias has his own test suite now
+
-
** We should make sure that our tests are up-to-date and run them before releases
+
-
* Web services on users' guide (Tommy)
+
'''Updates'''
-
** Users' guide needs only a few changes.  One of the changes should be a note making it explicit that we are using a new library and the old one was deprecated.
+
-
** Do we want to update the 39-page guide as well?  For now, just update the users' guide.  Don't distribute this guide.  Just include it in a zip file that includes the old version.
+
-
* [[Improve error handling/reporting]] (Tommy)
+
* [DK] Working on bundles
-
** print statements, has a new window
+
-
** add a menu item to view message window and also keep all messages from the session
+
-
** look into writing QMessageBox errors and warning to message window
+
-
** write all messages to a single window, write critical messages in red?
+
-
** have a utility method that encapsulates the gui error display and the core.debug display (if not running gui, only write to core.debug)
+
-
** don't display stack trace in a qmessagebox?  -- try to emulate the invalid pipeline with "Show Details" button.
+
-
** make it easier for users to report errors by having a "copy" or "send to developers" button to report the details of an error.
+
-
* DisplayWall (Wendel)
+
'''Items to Discuss'''
-
** What examples should be in the SVN? If we decide to keep some examples in the svn, some of them use data that should be along.
+
* [TE] Using vistrails API in scripting [https://gist.github.com/rexissimus/034d9c4dedc3299a079e84981dedd8c7 example 1 (API with VTK)] [https://gist.github.com/rexissimus/a4a9ca6c08d4058c64967d2b35415fd5 example 2 (DownloadFile only)]
-
*** Indicate progress of HTTPFile
+
** Call module in script using vistrails API
-
*** Had to upgrade examples to work with new examples of VisTrails
+
** Can be used when no export method is defined.
-
*** VisTrails can send progress information with another thread like crowdlabs can do
+
* [TE] [https://github.com/VisTrails/VisTrails/commit/c76c109349769fee1280f85c22678380a2f9cd39 Server news bug] Critical?
-
*** Each machine has a .vistrails and needs to download own version of data, make a single .vistrails for all machines?
+
**[RR] No, unreleased
-
*** All slave machines can use the server as a cache?
+
* [DK] New bundle organization
-
** DisplayWall Client on the Apple Store? I dont like the idea of sharing the client source code. And putting the app without login/password will allow anyone to access our displaywall.
+
** BundleMapping contains individual mappings from raw objects to BundleObj
-
*** Add a password for this
+
** Must be separate in order to have in-memory, separate from serializer settings
-
*** Do user management?
+
-
* VTK package:
+
=== June 29, 2016 ===
-
** Have two ResetCamera modules now
+
-
** Need to write upgrade code
+
-
** Only with vtk 5.7 right now
+
-
** Encode library version in the package version?
+
-
** For reproducibility, we probably want to stick to specific versions
+
-
** We still need some ability to run new versions of vtk for testing, new features, etc., too...
+
-
* Relative paths in workflows, how to deal with them? (Emanuele)
+
'''Updates'''
-
** Should we use dataDirectory?
+
-
** Change examples to not use hard-coded files?
+
-
** Problem with workflows, not data files
+
-
** On Windows, use ../examples
+
-
** On Mac, use ../../../examples
+
-
** dataDirectory not used, I think
+
-
** create "$VT_EXAMPLES"?
+
-
** problem is using new versions of VisTrails, data may change so referencing old location will fail
+
-
** don't allow users to change this variable
+
-
** convert to use HTTPFile
+
-
* Data files and the GIT Repository (Emanuele)
+
'''Items to Discuss'''
-
** make a separate repository or use the submodule git functionality
+
* [TE] Export VTK as python script
-
** go with submodule
+
** Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) [https://gist.github.com/rexissimus/16caf5200802e150296f62a0c7844c5a example]
 +
** Preludes (imports) still need improvements
 +
** Works with vtkInteractionHandler (clipping plane)
 +
** Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
 +
** vtkRendererOutput Uses VTKRenderWindow as output
 +
*** Should support different output modes?
 +
*** Does not work with multiple outputs (Next cell is displayed when current is closed)
 +
*** Uses wrapped vtkInstance (as vvtk)
 +
**** Create new patch system with code that can be used both in vistrails and in a python script?
-
* Next release: What still needs to be done?
+
=== June 22, 2016 ===
-
** high priority from ALPS list
+
-
** Dan's fixes to the upgrade workflow
+
-
** Date for release?  Official release in beginning of December for ALPS
+
-
** All developers should run test suite on push
+
-
** Could make this somewhat automatic, run with nightly release script and email -dev list
+
-
** Could have different classes of tests, make sure that critical ones pass
+
-
** Tests have a bunch of errors currently
+
-
* Saving explorations:
+
'''Updates'''
-
** Keeps track of the latest exploration for each version
+
-
** Check with Matthias what the requirements for this are
+
-
** Have way to send just specification to a server (run this workflow with this parameter exploration)?
+
-
** Can we have an xml spec that specifies the parameter exploration (export parameter exploration)
+
-
** Currently persisting via xml and then converting to string so we already have an xml serialization
+
-
** Dan wrote API that takes xml string and populates the param_exp gui with the appropriate values
+
-
** Need to back the GUI state with core state to enable short-circuiting the gui
+
 +
'''Items to Discuss'''
 +
* [TE] Fix for upgraded analogies (PR [https://github.com/VisTrails/VisTrails/pull/1177 #1177])
 +
** Uses module location to find module remap (The best we can do right now?)
 +
** Also fixes remapped port names for connections to existing modules
 +
** May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
-
* Box input ports should be colored (e.g. black) if they are already set by an internal method. Another color (e.g. red) could be used to indicate mandatory input ports.
+
* [TE] Working on exporting wrapped libraries (VTK)
-
** Currently, non-optional ports are square, shown.  Optionals are circles and must be enabled
+
-
** Add a third category for mandatory, problem is backward compat.
+
-
** [ES] Colors are already overloaded, not for ports, use shading instead?
+
-
=== Nov 9, 2010 ===
+
=== June 15, 2016 ===
-
* FAQ
+
'''Updates'''
-
** As we reply to users queries, let's add the question and answer to the FAQ!
+
* [TE] Added depth logic to python script exporter ([https://github.com/VisTrails/VisTrails/commit/50f220e4f6abe5a7bb8689f73f456f43df2dbc12 full commit])
-
** Just a reminder
+
** Fully implements connection merging and list looping
-
* Caching of File module
+
'''Items to Discuss'''
-
** There was a message from a user who got confused because this module is cached by default (and silently). And in his application, since the file actually changed in between runs, he did not see the 'expected' result.
+
-
** Should we keep a hash for files and check whether they changed?
+
-
** Should we no cache File by default?
+
-
** [DK] File is supposed to have a special signature computation that detects changes in the file contents so this may be a bug instead.  It does work, it seems to be an issue with the create_file
+
-
** [ES] In this particular case, the file was empty. He deleted the file outside VisTrails and executed it again. As he set create_file to True, he was expecting the file to be created again.
+
-
** [DK] Add random salt to hash signature when the file doesn't exist and create_file is True
+
-
** [JF: DONE] Juliana will edit the FAQ to note that the signatures are dependent on modification times
+
-
* Claurissa will demo different ways to visualize the version tree
+
* [TE] Making analogy work on upgraded workflows ([https://github.com/VisTrails/VisTrails/issues/1175 #1175])
-
** weighting on various criteria for displaying nodes (importance from session, user, tags, etc.)
+
** Analogy only works We need to use the non-upgraded action chain
-
** other modes for viewing the versions (lists or timelines)
+
** But pipelines need to be valid (matcher uses module portspecs)
-
** often see only linear trees, teaching or thinking other than tree?
+
** Tried to get upgrade remap from upgrade action, but this seems impossible
 +
** Use module location instead?
 +
*** Location should be identical for one-to-one module upgrades
 +
=== June 1, 2016 ===
 +
'''Updates'''
 +
* VisTrails 2.2.4 announced (finally)
 +
** 5 survey replies
 +
** 47 downloads last week
-
* Upgrade issues (Dan)
+
'''Items to Discuss'''
-
** Vanilla upgrades should be in today's commit
+
* [TE] Export VTK as script (python-wrapper)
-
** Added code so when doing upgrade to check whether the latest code is upgraded
+
** to_python_script
-
** Two types of upgrades: (1) User changes the subworkflow, (2) Modules are out-of-date and need to be upgraded.
+
*** needs the pipeline (or at least the connections)
-
** Add higher-level features like "Upgrade All" for (1), "Import Changes to My Subworkflows", etc.
+
**** Should not  generate code for any possible VTK method (exporter detects which ports is used in the source)
-
** Want a 1-1 mapping between unique subworkflow id and a subworkflow file (even if we copy the same subworkflow to different vistrails)
+
**** VTK needs to know connection order
 +
**** Adding functions directly to the script results in nicer code
 +
** Add support for multiple input connections
 +
** Add support for list looping
-
* Offscreen bug  (update from Huy)
+
=== May 25, 2016 ===
-
** Mac issue when switching to Cocoa
+
-
** Huy added a QWidget for each of the render windows that is created, so should be fixed.
+
-
* Gesture support
+
'''Updates'''
-
** Someone on Windows check the pinch gesture support; it doesn't work for Matthias under Parallels
+
* Mailing-list fixed
-
** VTKCell is working now
+
* Getting stats from the "server news" mechanism (via [https://github.com/remram44/apache-log-info parser])
-
** Cell dependent, panning now maps to middle button, pinch on the base widget
+
-
** What are Matthias's requirements here?  Does touch gestures in the cells matter or only in the vt/wf views
+
-
* Thumbnails: [JF: this needs to go to trac, for 2.0?] [ES: 2 tickets were created for 2.0]
+
'''Items to Discuss'''
-
** Resolution? -- [ES] problem is the size of the vistrail, but this can be configured
+
* [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
-
** Can we export to PDF instead of PNG? [ES] cannot display the thumbnail as a PDF in Qt?
+
* [TE] Union ports
-
** Can we have the option to save higher-res or PDF versions of the thumbnails
+
** Show union ports in pipeline (DEMO)
-
** Can we have preferences that allow users to save a higher-res or PDF version, maybe a checkbox
+
-
** Maybe associate a high-res version with a spreadsheet cell so that we can save a high-res version on demand
+
-
** high-res version associated with a version but allow user to initiate action from spreadsheet
+
-
** can we have thumbnails dependent on camera position [ES] doesn't like, user won't see exactly that image upon re-executing
+
-
** compact vistrail option by removing thumbnails
+
-
* Tommy has rewritten the Web services module as well as updated all of the examples
+
=== May 18, 2016 ===
-
** Server for one of the examples seems to be down
+
-
** Need to update the manual
+
-
* [[Improve error handling/reporting]] (Tommy)
+
'''Items to Discuss'''
-
** Maybe display the intialization messages with the splash screen?
+
* [TE] mailing list still down - Julio working on it
-
** Debugging levels have no way to access debug level
+
* [TE] Improved union port
-
** Tommy takes first cut at trying to determine how to map print statements, notes those he is unsure of
+
** New schema 1.0.5 adds portitem.union string attribute
-
** Developer guidelines for debug usage, use them instead of print statements in the futur
+
** Works with pipeline view, parameter exploration view, and mashup view
 +
** Should work well for matplotlib
 +
** Use with VTK ports?
 +
*** It uses [item_1,item_2] style ports usually with different tuple length.
 +
* [TE] Remove emacs dependency from db generate [https://github.com/VisTrails/VisTrails/issues/1173 #1173]
-
* Merge functionality (Tommy) [JF: Cool!]
+
=== May 11, 2016 ===
-
** Can now merge an existing vistrail into your own vistrail using menu item
+
-
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
+
'''Items to Discuss'''
-
** [PM] I haven't had a chance to implement anything yet.
+
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
 +
** GUI-based solution [https://github.com/VisTrails/VisTrails/tree/union-port Branch union-port] Demo?
 +
*** Shows single port for all ports with same sort_key
 +
*** Select specific type when adding/showing
 +
*** Only one type should be visible on the module at any time
 +
*** Making tuple parameters union types is probably too complicated
 +
*** Should work with alternate specs like matplotlib (if sort_key is defined correctly)
-
* Data Publishing with crowdLabs. (Phillip)
+
* [TE] Feature suggestion: Let connections connect to functions/parameters
-
** Do we want data publishing capabilities in VisTrails and crowdLabs?  '''YES!!!'''
+
** Show functions and parameters as ports and toggle port visibility directly for functions and parameters
-
** If so...
+
** Use separate views for ports and functions (like before)
-
*** Currently limited data provenance for data. HTTPFile module lacks meta data, RepoSync module has poor usability.
+
*** Allow rearranging of functions
-
*** Persistent Data works for experimental data. Can we use it for publishing data?
+
** Pros:
-
**** Publishing data or storing data on the server and pushing back and forth?
+
*** Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
-
**** What is the current scheme for identifying data on the server?
+
**** We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
-
**** Do we need to do better than the ids, versions, hashes that the peristence package provides?
+
**** Makes List module simpler
-
*** Do we want to utilize data publishing software like [http://thedata.org/home The Dataverse Network] or maybe [http://www.fedora-commons.org/ Fedora Commons]?
+
*** Connect to parameter in tuple
-
** [ES] Feature request: Can RepoSync support a directory of files?
+
*** Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
-
*** [PM] It can't, and I'm not sure how add it. I also don't really like the current RepoSync interface. It's not obvious how your data is being handled, hence me bring up data publishing, so hopefully we can come up with something more usable/robust.
+
*** Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
 +
** Cons:
 +
*** Interface more complicated and crowded
 +
*** More difficult for users to grasp?
 +
*** No immediate use case
 +
*** Makes no union type for tuple parameters a bigger problem.
-
=== Nov 3, 2010 ===
+
* [TE] Publish user survey?
 +
** No response from vistrails release yet
 +
** Not announced on vistrails-users yet.
-
* Upgrading subworkflows (Dan)
+
=== May 4, 2016 ===
-
** add updated versions to subworkflow vistrail and push to registry
+
'''Updates'''
-
** fixed bug with version/descriptor redirects
+
* VisTrails 2.2.4 released
-
** key remaining task is to replace the box representing the subworkflow in the top-level workflow to reflect the change to the upgraded version of the subworkflow
+
-
** many corner cases, but we should get the vanilla upgrade (working) to git
+
-
** need to ensure that each subworkflow file has its own uuid; if a subworkflow exists in two different vistrails, it should have a different uuid (namespace).
+
-
** otherwise, we can get crosstalk where one vt can update the subworkflow from another vt (we can propgate changes to other vts via merges, but this should be a user choice...)
+
-
** enhancements:
+
-
*** allow users to merge subworkflows from a file into their own subworkflow (to incorporate outside changes)
+
-
*** latest version is the most recent non-upgraded version; if that version has an upgrade, use the upgrade
+
-
***'''Next step: work on usability for controlflow package; explore Dave's idea of an assistant'''
+
-
* Offscreen bug
+
'''Items to Discuss'''
-
** Huy is looking into this
+
* [TE] Union type [https://github.com/VisTrails/VisTrails/issues/1172 #1172]
-
* Crash due to issues in loaded user packages (Tommy has fixed this)
+
=== April 27, 2016 ===
 +
'''Updates'''
-
* Web services (Tommy)
+
'''Items to Discuss'''
-
** Package is completed, and all but one of our examples are working
+
* [TE] Wrapping Bokeh
-
** '''Tommy will test if the Web services will work when there is a proxy'''
+
** New method type "operation"
 +
*** Transforms Figure->Glyph->Output into Glyph->Figure->Output
 +
**** Glyph sets function name and arguments as output
 +
***** Not serializable, but should always be computed together with the subject of the operation.
 +
**** Figure applies Glyph operation on itself
 +
*** Useful for matplotlib?
 +
**** matplotlib currently find actors by searching upstream?
 +
**** But this is more difficult since matplotlib is not function-based.
 +
** New "union" port type
 +
*** Turns "x(float), x(list)" into "x (float, list)" and uses first set value
 +
**** Uses only first value in tuple
 +
**** Bokeh have many properties that support (list, string, Float)
 +
*** Does not work with depth=1 types (e.g. Float of depth 1)
 +
**** Turn Depth 1 types with known type into lists?
-
* Improve error handling/reporting (Tommy)
+
* [TE] New bugfix release
-
** we should have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages.
+
** Finalize survey
-
** the goal is to implement a function that will be the single point for error messages, and that work without emitting GUI signals in core. As Huy suggested, we should have a GUI-wrapper which will be a no-op when the GUI is not instantiated---this will get rid of some of the pyqt dependencies (at least for the error messages).
+
-
* Improvements to VisTrails server communication with clients to better handle errors/crashes (Phillip)
+
=== April 20, 2016 ===
-
** Currently the VisTrails servers consists of: a single-threaded instance with GUI; multiple GUI-less threads
+
'''Updates'''
-
** We will extend the server API to allow clients to check the server status. The server will provide a separate socket and the client will be able to 'ping' the server; if the server is working properly, it will respond; if it does not respond, the client will have a time out
+
* Merged upgrade fixes
-
** The API will also support the ability to kill both the GUI-less threads and the single-threaded GUI instance. We can then restart the server components using Emanuele's new script.
+
** Now has infinite loop test
-
** We should also allow users to set a per-workflow time-out; if the workflow execution takes longer than the pre-defined threshold, VisTrails will abort the execution
+
** loop test now correctly loops 5 times
 +
*** remap_module used wrong (current) package version when creating ModuleDescriptor
 +
* Fixed multi-step group upgrades (PR: [https://github.com/VisTrails/VisTrails/issues/1168 #1168])
 +
'''Items to Discuss'''
 +
* New bugfix release
 +
** Merge usagestats
 +
** Need to finalize survey
-
* Update on fixing VTK Package (Wendel)
+
=== April 13, 2016 ===
 +
'''Updates'''
-
** It seems that the ParaView package has the same issues.
+
'''Items to Discuss'''
-
** '''Wendel has already checked in the new wrapper into the trunk; Emanuele will test it'''
+
* [TE] Chaining upgrades done ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
 +
** Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
-
* Synchronizing VisTrails and ALPS releases (Emanuele)
+
* [TE] Iterating handle_invalid_pipeline ([https://github.com/VisTrails/VisTrails/pull/1165 #1165])
-
** VisTrails latex package was extended so that python is no longer required. If python is not present in the system, instead of issuing a request to execute the workflow and retrieve the image from the Web, a local, previously saved image will be used.  It is now also possible to embed the images of workflows into the latex file. Documentation about these and other features are currently in the README file and examples (example.tex) provided with the latex package.
+
** Not based on [https://github.com/VisTrails/VisTrails/tree/chain-upgrades-loop chain-upgrades-loop]
-
** Windows and Mac beta binaries already include ALPS
+
** Cherry-pick test to preserve source?
-
** Are we distributing ALPS as part of VisTrails then?  This seems backwards. ALPS should VisTrails as part of their stuff, no?
+
** Need maxPipelineFixAttempts option?
-
*** '''We will have separate binary distributions: one with and one without ALPS'''
+
*** The code is now smarter about recursion, so max attempts will "probably" never be reached.
 +
** Need [https://github.com/VisTrails/VisTrails/commit/0ef0667426937eddb1d1c1e397f0657ffec039a9 debug messages]?
 +
*** Would need to be moved into handle_invalid_pipeline
-
* Is a multi-touch interface to VisTrails possible now that it's supported by Qt? (Phillip)
+
=== April 6, 2016 ===
-
** Yes, Qt 4.6+ supports multi-tocuh --- '''Huy will look into this; the goal is to make this work both for the workflow builder/tree view and the spreadsheet'''
+
'''Updates'''
-
** We still need to figure out what makes sense for multi-touch, however, do we have requirements from ETH-Zurich?
+
* [TE] Query View improvements
-
*** We want to map the zoom, pan, click, etc to multi-touch, so that VisTrails can be intractable on a multi-touch screen without a mouse
+
** Done except for displaying "Version Info" and #1164
 +
* [RR] Working on finishing new interpreter (finally!)
-
* What is the status of the Vismashup i{Phone|Pad|Touch} app? (Phillip)
+
'''Items to Discuss'''
-
** If you mean mac binary, there's an alpha version here: http://www.sci.utah.edu/~emanuele/files/vismashup
+
* [TE] Chaining upgrades ([https://github.com/VisTrails/VisTrails/pull/1164 #1164])
-
** Do you mean iphone app? [Phillip: yes]
+
** Try latest upgrade first, then previous.
-
** '''Wendel will look into this'''
+
** If all fail, select latest upgrade?
-
*** One of the issues is how to effectively handle images that are larger than the memory on the iPods and iPads---we need to process these on the server or use a model that allows the image to be manipulated on the client
+
-
*** We also need to connect the app with the crowdlabs server, so that the app can get the list of mashups
+
-
*** Phillip will investigate the feasibility of working directly with JavaScript and bypass flash
+
 +
=== March 30, 2016 ===
 +
'''Updates'''
 +
'''Items to Discuss'''
-
=== Oct 19, 2010 ===
+
* [TE] Query View improvements
 +
** Unified do_version_switch and get_upgrades as validate_version()
 +
** Problems with query view using its own controller
 +
*** Fixed module info not being displayed for query results
 +
*** Still no version info for query results
 +
*** Still an issue with version query view not being updated
-
* Update on Trac and Roadmap (Emanuele and David)
+
=== March 23, 2016 ===
-
** Trac is now linked to git repository
+
'''Updates'''
-
** Roadmap on Trac has been cleaned up so we can hide completed milestones
+
-
** Tickets are being reassigned/revisited. Many tickets (24) are not associated with a milestone yet https://vistrails.sci.utah.edu/report/3
+
-
* Synchronizing VisTrails and ALPS releases
+
-
* Web services support (Tommy)
+
-
** discuss the interaction between the Web Services package and upgrades
+
-
* Subworkflows update (Daniel)
+
-
* Testing of new module drawing (Erik)
+
-
* Aliases
+
-
** Currently, an alias is stored on a parameter.  We need aliases to be stored at a higher level so that changing an alias is not a change to a parameter.  The one issue is that there is a link between parmaeters and aliases in that an alias can only exist for versions that have the specified parameter.  We might also have two versions where the alias points to different parameters.  We could just store aliases as a root-level workflow element so that the set of aliases is versioned corrected.
+
-
** In the current implementation, the aliases parsed from the parameters are stored in a dictionary in the workflow, so storing aliases as a root-level workflows element might be the way to go --[[User:Emanuele|Emanuele]] 19:50, 15 October 2010 (MDT)
+
-
* Error logging
+
-
** Suggest that we have a single point for all error messages to pass through that is linked to both core.debug and GUI elements that display error messages.  This should improve our error handling significantly
+
-
* PyQt dependencies in core
+
-
** Can we get rid of PyQt dependencies in core (and db)?  This would probably require moving the signals/slots that we currently use to a similar Python implementation which shouldn't be too difficult.  This would also require the specification of configuration widgets not as classes but rather some text that can be used to import the GUI elements only when using the GUI code.  However, what would happen with workflows that run in command-line mode but require some graphical output?
+
-
* Ports
+
-
** One annoying part of the current VisTrails model is that users are not given any visual indication that a port value is already set by a function or vice versa by a connection.  In addition, we have no way to specify or enforce cardinality on ports at design time.  It seems like we could allow developers to specify when a port should not be connected to more than one value (via a function or a connection), and give some visual feedback when a port has already been specified in one manner.
+
-
==== Meeting Notes ====
+
'''Items to Discuss'''
 +
* [RR] Usage reporting almost ready to go
 +
** Need to record whether specific features are used: paramexpore, mashup, query
 +
*** [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
 +
** How to record size of vistrail?
 +
*** [DK] IdScope already walks over vistrail, use that
 +
*** but total number of modules and versions doesn't give an idea of average size of individual pipelines
-
* Bug regarding userpackages at initial vistrails startup causing startup to fail.
+
=== March 16, 2016 ===
-
** Still an issue, ticket still open as a 1.6 milestone
+
'''Updates'''
-
* Copy-paste bug
+
-
** Shortcut not working on Mac.  Focus makes shortcut try to copy pipeline.
+
-
** Current fix appends to clipboard as users type - this is not a good fix.
+
-
* Synchronizing ALPS and VisTrails releases
+
-
** 1.6 scheduled for Dec. 1
+
-
* Webservices
+
-
** New library being used, but it's very low level.
+
-
** Simple types may cause problems with some web-services using complex or XML-based types.
+
-
** Need to make sure that this library can handle at least MOST of the web-services out there.
+
-
* Sub-workflows
+
-
** Daniel's changes seem functional and very slick.
+
-
** A little more work on it seems necessary.
+
-
** Need some easy GUI mechanism to delete a subworkflow.
+
-
** Is the exclamation point in the module draw the best way to handle this?  Make a tool-tip to explain it.
+
-
* VTK Wrapping
+
-
** Changes to VTK Python wrapping is causing headaches - particularly in backwards compatibility.
+
-
** Method typing has changed to expect things like "List" or "Vector" - makes things hard on the user when defining these inputs.
+
-
** Removing List and Vector types and replacing them with dynamically generated versions from Tuples.
+
-
* Aliases - Agenda pushed until after Vis.
+
-
* Error Logging - Unify some error handling to improve how exceptions are dealt with.  More on this after Vis.
+
-
* Refactoring out PyQt dependencies from Core/Db/etc - Need to move all the signals/slots into GUI elements.  Questions come when a workflow uses GUI elements as inputs.  - More after vis.
+
-
* Ports - We need a way of showing how a port is specified - function vs. input port.  Cardinality must be established in these cases.  Ordering on multiple connections should be handled in some way - whether it's just a documentation issue or a more fundamental one.
+
-
** Should we have a tutorial mode that bring up tooltips when someone does something new as if it were a live tutorial.
+
 +
'''Items to Discuss'''
 +
* [TE] Upgrades in Query Mode
 +
** Now using code similar to do_version_switch
 +
*** Uses handle_invalid_pipeline
 +
*** Differences: Not using current_pipeline, flushes changes
 +
*** New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
 +
*** Integrate to avoid repeated code?
 +
*** Use this for query mode?
 +
* [TE] VisTrails 2.2.4 ready for release?
 +
** tej package updated. Others?
 +
* [JF] upgrades and the version tree
 +
* [JF] provenance in Jupyter
 +
* [DK] bundles: [https://github.com/VisTrails/VisTrails/pull/1092 #1092]
-
=== Oct 12, 2010 ===
+
=== March 9, 2016 ===
 +
'''Updates'''
-
* Trac and Roadmap (Emanuele): Emanuele suggested we should create a development roadmap and make it a permanent item on the agenda. The idea is to go through the Trac tickets and use them to build the roadmap that would be made ''public''.
+
'''Items to Discuss'''
-
** go through trac and sort by priority
+
* [TE] Fixing search mode
-
** add other suggestions to roadmap as well
+
** String query now matches all by default
-
** admins can add milestones via the admin tab
+
*** This caused pipeline queries to never match
-
** ''David will assign tasks on list; those assigned need to follow up and check and fix them''
+
** String query can search: User/Notes/Tagname/Time/Module name
 +
*** Should we search module names by default? (Disabled by default because it is expensive)
 +
*** Fixed notes search
 +
** '>' parameter queries in documentation but has been removed
 +
** Fixed module matches for upgraded pipelines (By not upgrading pipeline)
 +
*** Upgrades still a problem
 +
*** Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
 +
*** [JF] Allow search on past tags
-
* Update on [[PythonSource error reporting and logging]] (Tommy)
+
* [TE] Create new bugfix release
-
** ''Need to make sure that all messages are printed through debug.X, so that the debug level (verboseness in preferences) is correctly used''
+
** Finish querying
-
** Can cut/copy/paste now
+
** Path issue on windows
-
** Can we prevent the delete from happening?
+
-
** Phillip noted the spyder project has an interactive python console: http://packages.python.org/spyder/
+
-
** Can press enter to get to the end
+
-
** Need to check if this works or if we need to change
+
-
** Now have ability to see stack trace from the GUI (triangle menu)
+
-
** ''Suggest using a dialog to display trace when this item is selected instead of printing to console''
+
-
** Also saving stack trace to the execution log
+
-
** Also printing debug information to vistrails log
+
-
** ''Add trac item to fix prints to use debug''
+
-
** ''Try to fix core.debug to have gui.debug to eliminate Qt dependency''
+
-
* Update on subworkflow (Daniel)
+
* [DK] Look into sqlalchemy and new bundle branches
-
** Notification when subworkflow is outdated triggered
+
-
** new_abstraction signal from registry in addition to new_module signal so that we can check if we need subworkflows to update
+
-
** How to display the possible upgrades (can be upgraded, can be upgraded but may break, etc.)
+
-
* New utility added by Carlos (currently at scripts/module_appearance) to draw fringes visually, which generates output that can be pasted directly into the add_module call.
+
=== March 2, 2016 ===
-
** ''Erik will test this on Mac and Windows.''
+
'''Updates'''
 +
* [TE] Fixed cache bug with port specs
 +
* [TE] VisTrails Next
 +
** Most packages now work
 +
** No qgis for Python 3
-
* Preparing a "Get Started" tutorial for SIGMOD repeatability
+
'''Items to Discuss'''
-
** Do we have instructions on how to use the latex package without crowdlabs?
+
** Always dereference symlinks? [https://github.com/VisTrails/VisTrails/issues/1146 #1146]
-
*** links to actual workflows in latex
+
-
*** have README and example for latex in the source
+
-
** Metadata associated with VisTrails: should we have the ability to add vistrail-level metadata? e.g., who created the vistrail, its purpose, etc.
+
-
*** have pointers to paper to workflow and vistrails
+
-
*** add pointers to papers
+
-
*** allow access to add/edit annotations at the vistrail level
+
-
*** ''add GUI element to allow people to edit/add annotations''
+
-
* Maintaining VTK package
+
=== February 23, 2016 ===
-
** have four classes that don't wrap, have error even in python level
+
'''Updates'''
-
** works well with VisTrails otherwise
+
* [TE] Fixed PythonSource 100% CPU bug
-
** haven't checked the new changes from the wrapping
+
* [TE] VisTrails Next
-
** ''Wendel will check on the new version''
+
** Updated [http://opendap.co-ops.nos.noaa.gov/axis/webservices/wind/ NOAA Web Service] examples
 +
*** Now uses REST service, tabledata, and Google Maps
 +
*** Web Service replaced with DownloadFile
-
* [[Web services package]] (Tommy)
+
'''Items to Discuss'''
 +
* [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
 +
** New 2.2 release soon?
 +
* [RR] Anonymous usage reports
 +
** Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features [https://github.com/VisTrails/VisTrails/pull/1154 #1154]
 +
* [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
 +
* noWorkflow - provenance for python scripts
 +
** Recent work on tracking dependencies
 +
* Also integrates ReproZip with Jupyter
 +
** [http://mybinder.org mybinder]
 +
*** Use with ReproZip to automatically build Jupyter notebook interfaces
 +
* Daves comments on Jupyter integration
 +
** It is possible to identify cell order on the server side
 +
** Not trivial to track order of cells on client side
 +
* How does Galaxy workflows relate to VisTrails?
 +
** Packaging in Galaxy is hard
 +
** Galaxy for biomedical workflows, difficulty making it general
 +
* Action items
 +
** How difficult to track cell dependencies in Jupyter notebooks?
 +
** Use noWorkflow to track cell dependencies?
 +
** How difficult to modify mybinder to use ReproZip packages?
 +
 
 +
=== February 17, 2016 ===
 +
'''Updates'''
 +
 
 +
'''Items to Discuss'''
 +
* [TE] Fixed Group memory usage
 +
** Modules in groups are supposed to be freed when done executing
 +
*** Prevented by logger that stored references to the modules (fixed, now using id(module))
 +
*** Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
 +
* [TE] VisTrails Next
 +
** Removed deprecated code
 +
*** Core imports that should be gui imports
 +
*** GetInputFromPort etc.
 +
**** PythonSources will be upgraded
 +
*** Fixes for Python < 3
 +
*** fixes for VTK < 6
 +
** TODOs
 +
*** Remove RemoteQ
 +
**** Port Hadoop to tej (Already have pbs)
 +
*** Merge package wrapper
 +
*** Finish SQLAlchemy branch
 +
*** Finish new bundle branch
 +
** [JF] Will contact Jeff and Matthias about moving to Python 3
 +
* [JF] Query by Example is broken
 +
* [JF] VisTrails Data Cleaning Project
 +
** Use VisTrails to process and explore task-specific data while keeping provenance.
 +
** Remi's Idea
 +
*** Use VisTrails as the backend in a Jupyter notebook
 +
*** Overcome problems with executing cells in random order
 +
**** Interesting research topic, but little interest has yet been noted
 +
*** Interesting to study how such a notebook would be used (Is there notebook usage data available?)
 +
*** Impose restrictions to notebook cell boundaries
 +
*** It will only work for functional operations
 +
*** Can we fix, or detect, non-functional (mutating) operations?
 +
*** Analyse code and data flow and see if it mutates data in unexpected ways?
 +
*** Mutable operations can be created in many ways in Python
 +
* Next meeting Tuesday 11:30
 +
 
 +
=== February 10, 2016 ===
 +
'''Updates'''
 +
 
 +
'''Items to Discuss'''
 +
* VisTrails Next
 +
** Test suite passes
 +
*** Python 3 less deterministic, module upgrade order random.
 +
** [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
 +
** All of our packages should run fine :)
 +
** VTK thumbnail comparison works
 +
*** Rendering bug on some systems
 +
**** QVTKCell now inherits QGLWidget
 +
*** Old thumbnails had wrong height due to spreadsheetwindow being too small
 +
**** Fixed by regenerating thumbnails
 +
*** Silenced deprecation messages when generating modules
 +
*** Many new classes in VTK 7
 +
** Remove deprecated code
 +
*** Core imports that should be gui imports
 +
*** GetInputFromPort etc.
 +
**** Will break existing PythonSources
 +
**** Try to upgrade?
 +
*** Old-style packages
 +
**** Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
 +
***** Read info.py without importing package?
 +
*** Fixes for Python < 3
 +
*** Fixes for Qt < 5
 +
*** fixes for VTK < 7
 +
*** Remove self port?
 +
 
 +
=== February 3, 2016 ===
 +
'''Updates'''
 +
 
 +
'''Items to Discuss'''
 +
* [vistrails-users] [http://lists.vistrails.org/pipermail/vistrails-users/2016-January/000402.html Rendering a .pvd (ParaView) file?]
 +
 
 +
* VisTrails Next
 +
** All tests but thumbnail comparison passes
 +
** Still issues with gui and more packages needs to be tested
 +
** PythonSource and MplSource upgraded with lib2to3
 +
** Keep contrib packages? Update?
 +
** [https://blog.kitware.com/vtk-7-0-0/ VTK 7 is released]
 +
*** Supports Python 3
 +
*** Will hopefully get into conda and others soon
 +
*** Deprecates a bunch of modules
 +
 
 +
* [https://travis-ci.org/VisTrails/VisTrails/builds/106743348 Travis builder] for VisTrails Next
 +
** Runs!
 +
** Uses conda
 +
*** Many levels of requirements
 +
**** apt packages
 +
**** conda packages
 +
**** python packages in requirements.txt
 +
**** python packages in setup.py
 +
*** Can probably drop conda. Default builder does not have Qt5, but beta builder do!
 +
*** But useful if we want to use conda in our own build process.
 +
** Lots of errors I do not get locally?
 +
 
 +
* [http://goo.gl/forms/Z4HBrwaBWL User Survey]
 +
** Have you used any alternatives to vistrails?
 +
*** Relevant? More options?
 +
** How often do you use any of these particular features in VisTrails?
 +
*** Add/Remove alternatives?
 +
** How important would you rank the following potential improvements to VisTrails?
 +
*** Add alternatives?
 +
 
 +
=== January 27, 2016 ===
 +
'''Updates'''
 +
* New interpreter
 +
** Working prototype soon
 +
** Looping not completely done (and port combination mode: pairwise/cartesian...)
 +
** Need to integrate with VisTrails
 +
*** Compatibility with existing modules
 +
*** Use log controller
 +
*** Want to cache at multiple levels (whole stream/single element of stream)
 +
*** Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)
 +
 
 +
'''Items to Discuss'''
 +
* User Survey
 +
 
 +
* ZMQ failing on windows (vistrails-users)
 +
 
 +
* [TE] [https://github.com/VisTrails/VisTrails/pull/1143 Python 3 branch]
 +
** Testsuite almost passes
 +
*** Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
 +
** Fixed issues
 +
*** Signatures are now bytes (convert to string to show in widget)
 +
*** Cannot compare different types, e.g.,  (None < 1)
 +
**** PortSpec.depth and min/maxConn could be None. (Backport!)
 +
**** cell.row could be None
 +
*** Lots of unicode/bytes fixes to serializations
 +
**** We have encryption implementations in the code! (keychain)
 +
*** Lots of small fixes to package method differences
 +
** Current issues
 +
*** logging widget empty?
 +
*** version tree signals messed up
 +
*** provenance view legend icons black
 +
*** Text do not fit in some widgets
 +
*** VTK thumbnail comparison fails
 +
**** VTK Cells have some faces in wrong direction?
 +
****  Because running in VM?
 +
** Installing on Ubuntu LTS/Latest Mint
 +
*** Install latest matplotlib from pip
 +
*** Install VTK 7.0.0rc2 from source
 +
*** No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)
-
=== Oct 5, 2010 ===
+
=== January 20, 2016 ===
 +
'''Updates'''
-
* Welcome Tommy!
+
'''Items to Discuss'''
 +
* [TE] Backporting features from Python 3 + PyQt5 + Qt5
 +
** Merged unicode with master ([https://github.com/VisTrails/VisTrails/tree/unicode unicode])
 +
*** One unicode test failing
 +
*** Problem saving to xml
 +
** Used [http://python-future.org/ python-future] to run futurize stage 1 to update legacy (pre 2.6) code ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized unicode-futurized])
 +
*** Updates print to functions 'from __future__ import print_function'
 +
*** Updates exception calls to use 'as'
 +
*** removes most usages of 'long' (mostly used unnecessarily)
 +
**** Only needed for 32-bit support
 +
*** Fixes relative imports (from __future__ import absolute_import)
 +
** Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 ([https://github.com/VisTrails/VisTrails/tree/unicode-futurized-qt5 unicode-futurized-qt5])
 +
*** Updated all signals to new-style
 +
* If we go 2 to 3, this could break packages
 +
** means we should schedule any package format changes for the same time
 +
** e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* [https://github.com/VisTrails/VisTrails/pull/1002 #1002])
 +
** any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
 +
** putting package identifier as xml?
 +
*** Python not so bad, plus some stuff in there is Python code (checking dependencies)
 +
** version the APIs?
 +
*** RR planning on versioning vistrails_module.Module, no point versioning the whole package?
 +
* file formats on 2 to 3:
 +
** no real translation/upgrade, but non-ascii values that work on 3 won't load in 2
-
* Update on subworkflow (Daniel)
+
=== January 13, 2016 ===
 +
'''Updates'''
-
* Issues raised by Matthias:
+
'''Items to Discuss'''
 +
* [TE] [https://gist.github.com/rexissimus/dfd27e466427529c385c Porting to Python 3 / PyQt5]
 +
** Basic GUI and execution works
 +
** Minimal work to support 2to3
 +
** Hard to support both PyQt4 and PyQt5
-
- need to automatically upgrade subworkflows. Currently it is necessary to manually upgrade a subworkflow whenever the version of one of the modules inside them changes.
+
=== January 6, 2016 ===
 +
'''Updates'''
 +
'''Items to Discuss'''
 +
* Updating VisTrails dependencies
 +
** Successfully built [https://gist.github.com/rexissimus/f1a1aada7de1c00a57c7 Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7]
 +
*** VTK 7 is the first supporting Python 3 but is only in release candidate phase.
 +
**** Next version of VTK 6 is rumored to support Python 3
 +
**** There is a development branch of VTK 6 supporting Python 3
 +
*** Can travis build this?
 +
**** We can skip tests for WebEngine and VTK if needed
 +
** Python 3 [https://python3wos.appspot.com/ showstoppers]
 +
*** No MySQL-python (sqlalchemy supports mysql through other drivers)
 +
*** No suds (there are active forks but we could also drop support?)
 +
* Options for python2/3 support:
 +
** Interpreter could be python2/python3 based on packages
 +
** six layer works but not the easiest to maintain (e.g. in reprozip)
 +
* Drop support for win-64
 +
* Update copyright headers to 2016
-
- explorations do not detect an MplFigureCell embedded in a subworkflow.
+
== Older meetings ==
 +
* [[Development/2015]]
 +
* [[Development/2014]]
 +
* [[Development/2013]]
 +
* [[Development/2012]]
 +
* [[Development/2011]]
 +
* [[Development/2010]]

Current revision as of 09:34, 30 October 2017

See also the Github wiki

2017

October 30, 2017

Updates

  • [DK] PyQt5
  • [RR] Running Python3 code

Items to Discuss

  • Installation:
    • Use conda? (macOS hack to have .app bundle load conda-installed vistrails)
    • Update conda recipe (pyqt4 restriction for now)
  • Interpreter
  • Release
  • Integration
    • hard to build same type of GUI in other apps that integrate core VisTrails (tend to use just the API)
    • independent pieces of the GUI could be reused if they were more independent.

September 6, 2017

Updates

  • [RR] Fixes for scikit-learn, tables, usagestats

Items to Discuss

  • Stripped-down version of VisTrails to showcase new features?
    • include meta-vistrail, bundles
    • hide mashups, extra widgets, latex mode, etc.
  • Interpreter

August 16, 2017

Updates

  • [DK] meta-vistrail
    • Working actions: normal action in vistrail, pruning version tree, tagging operations, and change
    • Interface

Items to Discuss

  • Notebook
  • Interpreter
  • Web visualization packages (e.g. bokeh)
  • Pipeline templates

August 2, 2017

Updates

  • [DK] meta-vistrail work
    • Extend Vistrail to MetaVistrail to support meta-actions
    • Vistrail becomes less tied to workflows so it can also supporting versioning vistrails

Items to Discuss

  • Interpreter
    • Using dask?
    • Streaming an issue
  • Notebooks
    • What is an output of a module? (need to expose a specific output port or the entire module)
    • Unifies some of the work from the script-workflow code

June 28, 2017

Updates

  • [DK] Testing UUID/SQLAlchemy branch
  • [DK] Working on meta-versions
    • would be nice to have pipeline and vistrail separate
    • [RR] Then do we only have meta-actions, any change to the pipeline is a meta-action that adds a version that does the change?
    • [DK] Yes?
  • [RR] Notebook interface (Vizier) making good progress, figuring out cell outputs

Items to Discuss

June 7, 2017

Updates

  • [DK] Merged UUID/SQLAlchemy branch with new bundle format branch, getting rid of the bugs

Items to Discuss

  • [RR] How to specify an "intermediate output" that should be presented to the user, from any module? (useful for notebook interface)
    • Can designate in module description an output port
    • Value has to be serializable
    • Should use the OutputModes work to handle multiple modes, configuration, selection, ...
  • [RR] Is VisTrails with SQL usable concurrently?
    • [DK] Probably not, demo from [TE] exists

May 10, 2017

Items to Discuss

  • Identifiers
  • Bundles & DB Code
  • Interpreter
  • Merge / Versioning Version Trees
  • Python3
  • Workings of List
  • Packages: Wrappers + new Versions (VTK7)
  • Script Conversion
  • Intermediate Outputs: reuse output module work?
    • would be useful for both workflows (run to a particular checkpoint)
    • and useful for notebook interfaces
  • SQL backend: does it allow for concurrent access?
    • Yes, real-time collaboration work for concurrent single vistrail access
    • Should be able to access different vistrails at once without issues

April 19, 2017

Updates

Items to Discuss

April 5, 2017

Updates

Items to Discuss

March 8, 2017

Updates

  • [RR] Added a looping indicator on modules #1217
    • This makes it easier to understand what's looping, no need to follow edges and count
  • [RR] Refactoring CLTools for integration in Docker package
  • [RR] Will try to refactor autogenerated package logic next (need to update TensorFlow package for TF 1.0, should really be using this)

Items to Discuss

  • [RR] Problem with vtkRendererToSpreadsheet output mode: does it's own ModuleConnector handling, interfering with looping/streaming logic, to add the module ID on the renderers #1216
    • Can we move this up to the renderer modules?
    • Thankfully RR couldn't find much of this madness in the rest of code (outside of controlflow and parallelflow)

February 24, 2017

Updates

  • [RR] Rich-text module documentation (#1210)
    • Went over packages and fixed format; maybe we should have a test to validate RST?
  • [RR] Single-instance shutdown with --remote-shutdown (#1204)
  • [RR] Bundle installation not super reliable; probably need to go in and add checks for every usual environment

Items to Discuss

  • [RR] There are issues with list-depth
    • Some VTK ports marked with depth=1 don't actually accept multiple inputs (vtkDataSetMapper)
    • Issue with looping groups

January 25, 2017

Updates


Items to Discuss

  • Scaling VisTrails
  • sql-alchemy, uuid, bundle branches/merges

January 11, 2017

Updates

  • [RR] New VisTrails version in Debian [1]

Items to Discuss

  • [RR] Collaboration with school of medicine to port HiC-bench to VisTrails
    • Have homemade workflow system based on R scripts and symlinks
      • Each steps has multiple sets of parameters
      • Submits PBS jobs for each step with each set of parameters, using PBS job metadata to wait on previous step
      • Gets all results in directory structure step3-parameters/step2-parameters/step1-parameters/files
    • VisTrails improves provenance capture and makes editing pipelines easier
      • Need better support for aggregation (groupby operation might be good enough; pandas?)

2016

December 14, 2016

Updates

  • [RR] Single-instance code: over-engineered, but will work
    • (Have to avoid race conditions on startup, and OSes offer very different locking primitives)

Items to Discuss

  • Email from Matt Dirks: combobox should work, ask for more code?
  • Email from marin.nl:
    • looks like they would benefit from new interpreter
    • how to store their consolidated provenance when they exchange .vt files around?

November 30, 2016

Updates

Items to Discuss

  • [RR] Working on single instance code for MARIN (#1205)
    • Remote shutdown ready (#1204)
    • Still improving it
    • Want to return something more structured to the client (success/fail code, stdout, stderr)
    • Should be usable without Qt
  • [DK] Shutting down VisTrails has issues: cleans up interpreter (to remove files) but also resets persistent pipeline for no reason
  • [DK] Working on batch execution using console_mode
  • [DK] Settings aliased parameters when using core.api?

November 16, 2016

Updates

  • [RR] Issue #1200 from @samtux: need to set PyQt to API version 2 before ever importing it; using require_pyqt_api2() does that (also can install bundle)

Items to Discuss

  • Email from Pieter: executing pipelines with parameters from command-line using single-instance
    • Seems to work, main instance gets full command-line and alters configuration temporarily
    • Not sure what his error is (VisTrails returns little information in a lot of cases); though execution error should return something more meaningful
    • Actions:
      • Can add switch to kill main instance
      • Can improve single-instance code to return more meaningful errors + machine-readable output from "client" VisTrails
      • Add client/server code to core that doesn't use Qt? Also need a run.py-line entrypoint that doesn't use gui (calls console mode like gui.application.VistrailsApplicationSingleton#noninteractiveMode())?

November 2, 2016

Updates

  • New version 2.2-2 of Debian package, still wrong copyright, and broken bundle installation
    • Finally replied, says copyright issue will be fixed soon, working on including VisTrails 2.2.4 for Debian Stretch
  • Working on Docker package, low hanging fruit because we have CLTools, and useful for some workflow (also step towards CWL support)

Items to Discuss

October 5, 2016

Items to Discuss

  • working toward a 3.0 release; project board
    • unique ids
    • bundles
    • GUI fixes
    • wrapping updates
    • python3
    • interpreter?
  • command-line input parameters to provenance
    • [DK] we don't create a persisted workflow from those input params, i think
    • [DK] these are already defined as aliases so we treat them differently form a normal input param
    • [DK] still have the potential problem of allowing a user to view the workflow as it was run with a given set of aliases
    • Do the vistrails variables values get stored in provenance? (what if values change?)

September 14, 2016

Updates

Items to Discuss

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites
  • [RR] users' guide on readthedocs: doesn't seem like we can have both the internal docs (currently there) and users' guide under the same RTD "project"
  • [DK] UUID support almost complete (use-uuid branch), bundle translation then fits more easily on top of that

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)
  • [RR] Notebook execution works. How do we support more languages (than Python)?
    • Need: detection, metadata reader, code to input values
  • [RR] ALPS and VisTrails 2.1 issue: should be using 2.2?

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Personal tools