VisTrails Home

Development

From VisTrailsWiki

(Difference between revisions)
Jump to: navigation, search
(August 31, 2016: Websites summary)
m (Move 2015 meetings to separate page)
Line 625: Line 625:
* Drop support for win-64
* Drop support for win-64
* Update copyright headers to 2016
* Update copyright headers to 2016
-
 
-
== 2015 ==
 
-
 
-
=== December 30, 2015 ===
 
-
'''Updates'''
 
-
* Bokeh package finished
 
-
** Can output to Browser/IPython/Spreadsheet (Limited functionality in spreadsheet)
 
-
'''Items to Discuss'''
 
-
* Updating VisTrails dependencies
 
-
** Bokeh needs QWebEngine (Chromium) in Qt5 which requires PyQt5
 
-
*** Many plotting libraries only work in browsers
 
-
*** But Chromium may be a big and controversial dependency
 
-
*** Or create a browser version of the spreadsheet?
 
-
** Some users have requested Python 3
 
-
** Not backwards compatible?
 
-
** Target for VisTrails 3.0 with new interpreter?
 
-
 
-
=== December 16, 2015 ===
 
-
'''Updates'''
 
-
'''Items to Discuss'''
 
-
* Users list question
 
-
* [TE] Wrapping Bokeh
 
-
** Added bokeh property parser
 
-
*** complements numpydoc parser
 
-
*** Not all attributes are available, partially because I am using a pre-release version.
 
-
** Wrapped modules
 
-
*** bokeh.plotting finished (Backwards like matplotlib: Create plot then draw on it)
 
-
*** bokeh.charts mostly done (Missing arguments)
 
-
** QWebView mostly works (but no zooming or buttons)
 
-
*** It would be better to use external browser or QWebEngineView
 
-
** Added improvements to wrapper
 
-
*** Generalized port translations (Color/Path to native types)
 
-
*** Can use multiple docstring/property parsers simultaneously
 
-
 
-
=== December 9, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Looking into wrapper
 
-
** Will port TensorFlow
 
-
** Doing a pass on the wrapper code & doc
 
-
*** UX improvements
 
-
** Factorize high-level docstring parsers into core.wrapper? (sphinx, numpy, google)
 
-
** Problem with name vs module_name (fails sklearn tests because class_by_name() is broken)
 
-
** Looking into compressing specs, lazy registration of modules
 
-
 
-
=== December 2, 2015 ===
 
-
'''Updates'''
 
-
* [TE] Package wrapping
 
-
** Wrapped most of numpy/scipy using __all__
 
-
** Added class attribute and method access for classes (Can add to class or to separate inspector module)
 
-
** TODO: Document wrapping procedure
 
-
 
-
'''Items to Discuss'''
 
-
 
-
=== November 25, 2015 ===
 
-
'''Updates'''
 
-
* [RR] New interpreter is coming along
 
-
** Everything is a stream
 
-
** Task system, eventually work-stealing parallelism
 
-
** Streams no longer advance in lock-step
 
-
*** Allows for constructions like filter, join, sorted-merge
 
-
** depth>1 still to be tested, no current plan to allow this through the module interface
 
-
*** Through looping though, can do streams of streams
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Package wrapping
 
-
** Wrapped most numpy functions (no polynomials)
 
-
** and half of scipy classes and functions
 
-
** Added functions manually from documentation (no parseable lists?)
 
-
** Focusing on data flow constructs (no property/method access for classes)
 
-
*** Using spec diff (empty for now) and dynamic parser (stores spec in .vistrails/)
 
-
** Python class/function wrapper now stable
 
-
** Add google doc parser?
 
-
*** Is there a parser available?
 
-
 
-
=== November 18, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Package wrapping
 
-
** Numpy class wrapping works
 
-
*** Basic type is List, since most types are array_likes.
 
-
*** TODO: More classes and functions
 
-
*** Some operations are in-place and some aren't, and docstring not super clear
 
-
*** [RR] It seems only methods mutate, and most have a numpy.xx function equivalent; just manually go through the methods and remove the mutating ones?
 
-
** Class wrapper is modular
 
-
*** Docstring parser and type string parser can be customized
 
-
*** Classes can have optional attribute/method ports
 
-
**** Inspectors and attribute/method modules can be created separately
 
-
** Use PythonCalc as an example
 
-
*** Can be re-implemented as a function with a parseable docstring
 
-
*** [RR] PythonCalc is just an example, and it might actually make sense to get rid of it, or do a proper math package with scalar operations (as separate modules, no combobox)
 
-
** [RR] TensorFlow to use wrapping as well
 
-
*** Very simple (only types 'tensor' & 'variable')
 
-
 
-
=== November 11, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Package wrapping
 
-
** Wrapping numpy's classes using numpydoc (ndarray)
 
-
*** Wrapping constructor arguments, attribute getters/setters as module ports
 
-
**** Then how to access attributes afterwards?
 
-
***** Use input value port to class modules?
 
-
*** Wrap methods as modules? E.g., `ndarray.shape`.
 
-
*** Function wrapping is almost a subset of class wrapping
 
-
**** May be able to use the same parsing/execution methods
 
-
** VTK's non-getter/setter methods could be wrapped as modules
 
-
*** We could then remove extra logic in interpreter for keeping function order
 
-
** Bokeh uses autogenerated docstrings
 
-
*** We may be able to read the specification directly
 
-
 
-
* [RR] [https://github.com/VisTrails/VisTrails/pull/1138 TensorFlow package]
 
-
** Basic setup working, can execute the Mandelbrot example
 
-
** Will autogenerate the operations
 
-
 
-
=== November 4, 2015 ===
 
-
'''Updates'''
 
-
* [RR] Interpreter work requires a fix for [https://github.com/VisTrails/VisTrails/issues/1137 DB issue #1137]
 
-
* [DK] Kitware's Resonant [http://www.kitware.com/source/home/post/173]
 
-
** [https://github.com/girder/girder Girder]: data management system
 
-
** [https://github.com/Kitware/romanesco Romanesco]: execution engine, uses [http://www.celeryproject.org/ Celery] for task management
 
-
** Resonant Flow: web app for editing and executing workflows
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Added OSX Lion (10.7) VM on build machine using vagrant (seems ok with license?)
 
-
** New builds works on Lion
 
-
** Change the minimum requirement to 10.7?
 
-
* [TE] Package wrapping
 
-
* Added upgrade suggestions using 2 spec versions ([http://pastebin.com/30XUU8Gp example])
 
-
** using name edit distance to find matches
 
-
** Could use something better like port similarity for modules, and type similarity for ports
 
-
** Should write python upgrade code?
 
-
** Added spec to sklearn package
 
-
*** Can now diff spec versions and keep the spec static
 
-
*** There will be problems with downgrades and package versions
 
-
 
-
=== October 29, 2015 ===
 
-
'''Updates'''
 
-
* [TE] nightly-build-mac fixed (Needed to approve xcode license)
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] matplotlib wrapper
 
-
** ported to general spec
 
-
** Diff can be used on general spec
 
-
*** Only needed minor changes
 
-
*** Fixed indexing bug that corrupted specs
 
-
*** Can now move diff tools to core/wrapper
 
-
** Add new plots?
 
-
*** New plots in mpl 1.3: eventplot
 
-
*** New plots in mpl 1.4: angle_spectrum, magnitude_spectrum, phase_spectrum', violinplot
 
-
*** No new plots in mpl 1.5 (from looking at boilerplate.py)
 
-
 
-
=== October 21, 2015 ===
 
-
'''Updates'''
 
-
* [RR] Alexis has arrived, will be working with [RR] on a more efficient interpreter
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package]
 
-
** Supporting multiple package versions
 
-
*** Check which version can be loaded [https://github.com/VisTrails/VisTrails/pull/1135 #1135]
 
-
*** We cannot show version requirements in the list of packages because old packages are loaded when importing the codepath
 
-
** Visual diff does not work well, but we can use the spec differ to see differences.
 
-
** Unifying vtk and matplotlib wrapping specs (WIP)
 
-
*** And porting matplotlibs spec differ to work on the general spec
 
-
*** Spec differ can then be used on vtk and other packages in the future
 
-
*** Will enable automatic upgrade generation
 
-
* [RR] Rework interpreter
 
-
** restore abstraction between interpreter/module code
 
-
** build looping, streaming, caching into the interpreter
 
-
** rework cache
 
-
** look into parallelism once basic functionality is there
 
-
 
-
=== October 14, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] [https://github.com/VisTrails/VisTrails/pull/1133 New matplotlib package] (adding numpydoc parser)
 
-
 
-
=== October 7, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] [https://github.com/VisTrails/VisTrails/tree/fix-matplotlib-parser New matplotlib parser] (adding numpydoc parser)
 
-
** Added numpydoc attribute parser for plots.
 
-
*** TODO: Need better port spec reconciliation with call signature parser.
 
-
*** We can create a general numpydoc parser, but attribute types are unique to matplotlib.
 
-
** Needs package versioning
 
-
*** New spec will not support matplotlib < v1.4 due to changed path to axes classes.
 
-
*** Load spec version corresponding to installed matplotlib version?
 
-
*** Need version downgrades?
 
-
 
-
=== September 30, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Updating Matplotlib parser
 
-
** Matplotlib docstring parser fails on numpy docstrings
 
-
** I have [https://github.com/VisTrails/VisTrails/commit/eae8b0c16260810587e47c41cab53552cbb3cf0a added basic numpydoc parsing]
 
-
*** Only used by a few docstrings so far
 
-
*** Parsing uses many sources (class tables, signatures, ACCEPTS, method docstrings, definition parsing).
 
-
** Caswell said they were thinking about moving to traitlets, but this is not ready yet
 
-
 
-
=== September 23, 2015 ===
 
-
'''Updates'''
 
-
* [TE] VisTrails 2.2.3 released
 
-
** Also have pushed to PyPI, binstar, etc.
 
-
** Sourceforge vs. GitHub
 
-
*** should be able to host releases on GitHub
 
-
*** nightly binaries pushed to sf each night
 
-
*** old binaries?
 
-
* [TE] Re-wrapping MatPlotlib
 
-
** Keep static generation - Docstrings are brittle beween matplotlib versions
 
-
** Keep generating the executable classes - but create functions, not vistrails Modules
 
-
*** Why is there so much patching?
 
-
** Re-implemented Alternate PortSpec for InputPortSpec
 
-
*** It will now inherit specs from parent (No need to reimplement)
 
-
** Module upgrades?
 
-
 
-
'''Items to Discuss'''
 
-
 
-
=== September 16, 2015 ===
 
-
'''Updates'''
 
-
* [TE] Job Monitor tests and documentation done
 
-
** OK to leave document package in packages directory?
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Library wrapping: How to do code patching? [https://docs.google.com/drawings/d/1Sh3eOXot6v2y880jHKQrqXqAwr-0aoqYligg3z8G8MY/edit?usp=sharing Wrapping diagram]
 
-
** Insert code into module template (Old VTK code)
 
-
** Patch the library that is called (New VTK code)
 
-
** Somehow store code in spec and apply when executed?
 
-
*** Executed code needs at a minimum access to inputs, outputs, current class
 
-
*** Use standard input/output dict and obj reference that the code operates on?
 
-
*** Can this be done while keeping the execution abstract?
 
-
 
-
* Do new release now?
 
-
 
-
=== September 9, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* New release?
 
-
** [RR] wants reprounzip
 
-
** tej changes & doc
 
-
** job monitor & jobmixin fixes
 
-
** mongodb
 
-
** reprounzip
 
-
** warning: MplFigure type is int
 
-
** tabledata: add Reshape, DictoToTable & ListToTable, fix ListToTable with numpy arrays
 
-
** PythonSource can have same name for an input & output
 
-
** don't show spreadsheet at exit
 
-
 
-
=== September 2, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Limit autosaves? [https://github.com/VisTrails/VisTrails/pull/1126 #1126]
 
-
 
-
=== August 26, 2015 ===
 
-
'''Updates'''
 
-
* [TE] Added Job support to Parameter Explorations (Requested by Colin), and Mashups
 
-
** specify job ids, need to specify different ids for parameter explorations since they have the same version id
 
-
** how to deal with parameters passed in on command line
 
-
* [General] provenance: should be creating a new version when we execute workflow with changed parameters?
 
-
** currently, custom_params annotation stores this in provenance currently
 
-
** [RR] API doesn't record provenance if passing in parameters: http://git.io/vsAA5
 
-
'''Items to Discuss'''
 
-
 
-
=== August 19, 2015 ===
 
-
'''Updates'''
 
-
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
 
-
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
 
-
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
 
-
* [TE] Working on Job Monitor documentation
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] JobMixin and JobMonitor: stable now?
 
-
** Definitely needs more tests
 
-
* [TE] Stop testing VisTrails 2.0?
 
-
 
-
=== August 12, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Internal docs [https://github.com/VisTrails/VisTrails/pull/1116 #1116]
 
-
** On ReadTheDocs: [http://vistrails.readthedocs.org/ vistrails.readthedocs.org]
 
-
* [RR] JobMixin and JobMonitor: stable now?
 
-
** Definitely needs more tests
 
-
* [RR] Improvements to tej, users' guide entry & example [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
 
-
* [RR] Build broken on Travis; because of IPython 4 released today? ([https://travis-ci.org/VisTrails/VisTrails/builds/75279795 build 992]; [https://github.com/VisTrails/VisTrails/issues/1123 #1123])
 
-
 
-
=== August 5, 2015 ===
 
-
'''Updates'''
 
-
* [TE] Fixes to Jobs
 
-
** Could not delete jobs
 
-
** Could not run job in group
 
-
** Job not reset when calling ModuleSuspended
 
-
** Added deleting job from context menu
 
-
* [RR] Writing documentation for everything [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
 
-
 
-
'''Items to Discuss'''
 
-
 
-
=== July 29, 2015 ===
 
-
'''Updates'''
 
-
* RR still looking into new interpreter thing
 
-
** Goal is to take out scheduling logic from Module so it can be split in multiple processes, and so that smarter strategies can be added in time
 
-
** This means some work on packages
 
-
** Spreadsheet can live in kernel process? Still some UI stuff that will take work (changing configuration, persistent archive's viewer, ...)
 
-
* [TE] Problems running examples [https://github.com/VisTrails/VisTrails/pull/1111 #1111]
 
-
** Testing of more examples requires additional packages on the test machines.
 
-
** Fixed faulty line-ending in PythonSource:s failing on Python 2.6.
 
-
** Test suite now testing SUDSWebServices (If web service is down, test suite will fail)
 
-
** preferences.py test failed reloading 'dialogs' package, switched to using 'URL'.
 
-
 
-
'''Items to Discuss'''
 
-
 
-
=== July 22, 2015 ===
 
-
'''Updates'''
 
-
* [TE] mailing lists back online
 
-
* [TE] Working on [https://github.com/VisTrails/VisTrails/issues/1107 #1107]
 
-
* [RR] Working on ReproZip package
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] MongoDB package [https://github.com/VisTrails/VisTrails/pull/1106 #1106]
 
-
* [RR] Example for tej docs? [https://github.com/VisTrails/VisTrails/pull/1105 #1105]
 
-
 
-
=== July 15, 2015 ===
 
-
'''Updates'''
 
-
* 2.2.2 released
 
-
 
-
'''Items to Discuss'''
 
-
* RR is considering executing everything in an IPython kernel (i.e. separate Python interpreter, like the one spawned for a notebook)
 
-
** It's a separate process that we can restart without restarting the app/gui
 
-
** We can isolate the execution environment (e.g. for the server)
 
-
** We can run the whole thing remotely (if your data is elsewhere, just run VisTrails locally and the pipeline elsewhere)
 
-
** Pipeline execution no longer makes the interface hang, it just makes the kernel hang (but that's fine)
 
-
** We can use notebooks as modules (probably way nicer than the PythonSource module)
 
-
** We can run multiple kernels so long as the ports carry things that are serializable
 
-
*** meaning we can put the multithreaded-interpreter without all the hacky parts it has now
 
-
*** we can run IPython kernels in all the languages IPython supports, [https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages currently 46]
 
-
 
-
=== July 8, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Buildbot github hook not working after IP address change
 
-
* [TE] No reference to VistrailsApplication ([https://github.com/VisTrails/VisTrails/issues/1103 #1103])
 
-
** [RR] Added PR [https://github.com/VisTrails/VisTrails/pull/1104 #1104]
 
-
* [TE]  Reopening VT file after saving with bundled subworkflow won't offer subworkflow upgrade ([https://github.com/VisTrails/VisTrails/issues/1102 #1102])
 
-
** allow manual delete to fix right now
 
-
** fix this on top of the use-uuid branch
 
-
* [TE]  Release VisTrails 2.2.1? ([https://github.com/VisTrails/VisTrails/blob/v2.2/CHANGELOG CHANGELOG])
 
-
** [https://github.com/VisTrails/VisTrails/pull/1093 output-modules-use-global-dir]
 
-
** [https://github.com/VisTrails/VisTrails/milestones/version%202.2 Current issues for 2.2]
 
-
 
-
=== July 1, 2015 ===
 
-
'''Updates'''
 
-
* [TE] PROV fixed
 
-
* [TE] Working on subworkflow issues
 
-
* [RR] Considering reworking the controller (log vs exception problem, retained upgrades causing interferences) and also the interpreter (IPython?)
 
-
 
-
'''Items to Discuss'''
 
-
 
-
=== June 24, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* UV-CDAT
 
-
* [TE] Can a cyclic workflow be valid? ([https://github.com/VisTrails/VisTrails/issues/1097 #1097])
 
-
** focus on disabling the ability to create cyclic pipelines because more things break than just this with a cyclic pipeline
 
-
* [RR] Relative paths ([https://github.com/VisTrails/VisTrails/pull/1057 #1057])
 
-
** This interacts with the new bundle; how do handle packing files inside the VT bundle?
 
-
 
-
=== June 17, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] current_version and reusing existing upgrades are broken ([https://github.com/VisTrails/VisTrails/issues/1095 ticket #1095])
 
-
** Could be that export to PROV is using an unflushed upgrade pipeline?
 
-
** current_version would then be correct.
 
-
** It may work to flush the actions before exporting?
 
-
 
-
=== June 10, 2015 ===
 
-
'''Updates'''
 
-
* DAT: fixed VTK issue on Linux and Mac
 
-
* Still crashes on Windows. Need help! Reminder: this works in the VisTrails spreadsheet (QCellPresenter), although no widget get changed there during a drag
 
-
** Is it a VTK bug?
 
-
** Is it simply impossible to change widgets during the drag, should we do it a different way?
 
-
** Did I miss something that is done in VisTrails but somehow not in DAT?
 
-
** Low prio, UV-CDAT doesn't run on Windows anyway
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 Updated pull request])
 
-
* Single instance code and batch mode
 
-
 
-
=== June 3, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] Executable Paper ([https://github.com/VisTrails/VisTrails/issues/1086 ticket #1088] [https://github.com/VisTrails/VisTrails/pull/1088 pr])
 
-
** Requires fixes to command line parameters, Output modules, and batch mode
 
-
** How to test this
 
-
** Updated missing/outdated flags
 
-
** Fixed view issues when generating graphs
 
-
* [TE] batch mode
 
-
** SpreadsheetOutput not enabled in batch mode. Should we check is_running_gui instead?
 
-
** Other instance setting flags from caller
 
-
** Is graphsAsPdf replacing spreadsheetDumpPdf?
 
-
** graphsAsPdf true by Default?
 
-
** Batch mode executing by default (Not needed when generating graphs)
 
-
** Re-added workflowInfo as withWorkflowInfo for writing graph and xml workflow
 
-
** [DK] batch mode should be outputting to files or stdout, shouldn't always trigger SpreadsheetMode
 
-
** execute flag, maybe make execution the default and allow a "--no-execute" if you only want to capture graphs, for example
 
-
 
-
=== May 27, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [vistrails-users] API question
 
-
**  http://lists.vistrails.org/pipermail/vistrails-users/2015-May/000357.html
 
-
* Status of DAT
 
-
** Can add VCS plot to DAT but need to work on configuration windows
 
-
** Documentation for graphics templates for UV-CDAT/vcs?
 
-
* Qt support?
 
-
 
-
=== May 20, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Reviving the DAT, integrating scripting & porting UV-CDAT!
 
-
** On [https://github.com/VisTrails/DAT GitHub] ([https://github.com/VisTrails/DAT/issues issues])
 
-
** Merging 2 years of development taking longer than expected, but getting there. The plan is to get the patches in VisTrails and never fork again, we never want to get in UV-CDAT's situation (and don't need to).
 
-
** VTK cell works fine on Linux but there was flickering on Mac & Windows before; still issues on Mac (Windows status unknown)
 
-
** Can get a VCS plot soon (but will need VTK cell fix)
 
-
** Integrate in UV-CDAT's build system (so we have cdms, VCS, ...) -> RR can do this, low priority
 
-
** How do we integrate scripting?
 
-
*** We want to be able to seamlessly make changes to a plot by changing Python code
 
-
*** Define new plots by entering Python code without writing modules/packages?
 
-
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
 
-
* [RR] Subworkflow issues
 
-
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
 
-
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
 
-
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
 
-
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
 
-
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen
 
-
 
-
=== May 13, 2015 ===
 
-
'''Updates'''
 
-
* [RR] Test skipping whitelist ([https://github.com/VisTrails/VisTrails/pull/1069 #1069]) -- low priority
 
-
* [RR] Custom matplotlib modules can't be compatible with both 2.1 and 2.2 ([https://github.com/VisTrails/VisTrails/issues/1067 #1067]); should be fixed for ALPS ([https://github.com/VisTrails/VisTrails/issues/1070 #1070])
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] UV-CDAT [https://github.com/UV-CDAT/uvcdat/issues/1275 needs] [https://github.com/VisTrails/VisTrails/pull/1073 #1073], please take a look
 
-
* [RR] Subworkflow issues
 
-
** [https://github.com/VisTrails/VisTrails/issues/1065 #1065]: missing attribute (fixed)
 
-
** [https://github.com/VisTrails/VisTrails/issues/1066 #1066]: groups in subworkflows were ignored when finding dependencies (fixed)
 
-
** [https://github.com/VisTrails/VisTrails/issues/725 #725]: "upgrade subworkflow" button not popping up (fixed); also, namespace issue
 
-
** [https://github.com/VisTrails/VisTrails/issues/1071 #1071]: subworkflows don't go through upgrades
 
-
* [RR] [https://github.com/VisTrails/VisTrails/issues/1074 #1074]: can't load and edit a single pipeline if upgrade happen
 
-
 
-
=== May 7, 2015 ===
 
-
'''Updates'''
 
-
* [RR] Export/import workflow to Python working!
 
-
 
-
'''Items to Discuss'''
 
-
* [Claudio] UV-CDAT
 
-
** The UV-CDAT project is the biggest user base of VisTrails
 
-
** VisTrails package management provides a lot of friction towards people plugging in their code
 
-
** Need to make it easy to integrate your random Python scripts in the system without having to deal with all the boilerplate, at least in the first step
 
-
** [RR] argues that modules are still good; UV-CDAT shouldn't move towards a purely script-based backend
 
-
** [RR] export/import with Python could reduce a lot of that friction by allowing 1) to edit workflow as Python 2) to open up boxes automatically if needed code doesn't match actual modules
 
-
** ...
 
-
 
-
=== April 29, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] <strike>Abstractions</strike> subworkflows status ([https://github.com/VisTrails/VisTrails/labels/subworkflows tickets])
 
-
* [RR] matplotlib compatibility (2.1 & 2.2), [https://github.com/VisTrails/VisTrails/issues/1067 #1067]
 
-
** RR to try and fix ALPS matplotlib modules
 
-
 
-
=== April 22, 2015 ===
 
-
'''Updates'''
 
-
* 2.2.0 has been released!
 
-
* Windows issue (via email from Ryan)
 
-
** issue with manifest file (may be a new file in VTK6?)
 
-
** Tommy has regenerated new Windows builds
 
-
* Binaries, pypi, and conda released
 
-
* [RR] Export as script
 
-
* Python sources using VTK need to switch to SetInputData (users should be aware of this)
 
-
 
-
=== April 15, 2015 ===
 
-
'''Updates'''
 
-
* Ready for 2.2.0 (apart from binary/deps issues)
 
-
** Missing some libs (scikit-learn, tej, tdparser, SQLAlchemy+connectors
 
-
** Windows: runvistrails.py is no longer used, so the PATH is wrong
 
-
** Windows: pip is broken, but it probably wouldn't work anyway because of permission issues (disable this?)
 
-
* Queries, upgrades and getPipeline() usage ([https://github.com/VisTrails/VisTrails/issues/1054 #1054])
 
-
** Getting a pipeline with getPipeline() is not safe: it might return an invalid pipeline
 
-
** This is used in many places throughout the code, like queries
 
-
** Upgrading would require going through the controller, but that creates new actions
 
-
* [TC] Avoid copying a module's output if it's used as input by exactly one downstream module ([https://github.com/VisTrails/VisTrails/issues/1060 #1060]) (useful for big numpy arrays you can update in-place)
 
-
 
-
=== April 8, 2015 ===
 
-
'''Updates'''
 
-
* [DK] Merged RR's changes for output modules (1012 and 1013)
 
-
** RR will merge remaining changes, then create v2.2 branch!
 
-
 
-
'''Items to Discuss'''
 
-
* BNL need numpy array to VTK image
 
-
** Looks like VTK has some helpers for this
 
-
** We will help if issues arise
 
-
** Will contribute back to VisTrails package
 
-
* Upgrade issue: [https://github.com/VisTrails/VisTrails/issues/1017 #1017]
 
-
** Automatic upgrades should happen between versions of provided upgrades
 
-
** Our existing upgrades work around this so it doesn't need to be 2.2.0
 
-
* Corner-case VTK modules
 
-
** No longer need VTKCell input port, so don't interfere with registry and API anymore
 
-
** Still work weirdly, people probably shouldn't use them
 
-
** But we have lots of clunky modules since we wrap the whole of VTK; some people might rely on this and know how to use them, let's keep them anyway
 
-
** Ready for 2.2.0
 
-
 
-
=== April 1, 2015 ===
 
-
'''Updates'''
 
-
* [RR] UV-CDAT: bugfixing for 2.2, long-term plans: implement scripting import/export in VisTrails, port to UV-CDAT
 
-
* Possibly, try to move to regular VisTrails to use new features
 
-
 
-
'''Items to Discuss'''
 
-
* 2.2 release: nothing much is pending anymore, release next week?
 
-
** [TE] vtkExporter classes [https://github.com/VisTrails/VisTrails/issues/1032 #1032]
 
-
** [RR] Hiding upgrades in version tree might make it (or might be 2.2.1 so we can test it out) [https://github.com/VisTrails/VisTrails/issues/949 #949]
 
-
** Output modules changes to go in
 
-
* add note to documentation about order of parameters in VTK
 
-
* add issue about exporter upgrades if not already there
 
-
 
-
=== March 25, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* 2.2 release
 
-
** Checklist on Github: https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist
 
-
** Review output modules
 
-
** RR has a couple more issues to fix
 
-
** Ready to go -- sign app package for OSX? [https://github.com/VisTrails/VisTrails/issues/984 #984] We need access to the Apple Membership team
 
-
 
-
=== March 18, 2015 ===
 
-
'''Updates'''
 
-
* [TE] VTK6 works
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Release v2.1.5 with backported tabledata?
 
-
** MTA example needs updated tabledata (for JoinTables)
 
-
** Google Maps package still not available
 
-
* [RR] Work torward v2.2.0?
 
-
** changes:
 
-
*** new persistence
 
-
*** API changes
 
-
*** output module changes (upgrades?), maintain cells but try to upgrade
 
-
*** not wrapping stuff
 
-
*** VTK6? yes
 
-
*** JobSubmission stuff?
 
-
*** relabeling for upgrades [https://github.com/VisTrails/VisTrails/issues/949 #949]
 
-
** makes sense, needs the tree view code to be updated, check selection
 
-
** See [https://github.com/VisTrails/VisTrails/wiki/2.2-Release-Checklist 2.2 checklist]
 
-
* Discussion of [https://github.com/VisTrails/VisTrails/issues/1016 #1016]
 
-
** plumbing between outputs and output modes, how to define a mode that works for many outputs without writing for each output?
 
-
 
-
=== March 11, 2015 ===
 
-
'''Updates'''
 
-
* [RR] persistent_archive done; merge? ([https://github.com/VisTrails/VisTrails/pull/755 #755])
 
-
** note about the focus events for widgets
 
-
** TE be aware of file_archive for future binaries that include persistent_archive
 
-
* [TE] [https://github.com/VisTrails/VisTrails/pull/998 New VTK package] finished
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails? (see last week; decision needed)
 
-
** RR email to JF about this
 
-
** yes; email sent to Github
 
-
* [RR] Switching order of output ports ([https://github.com/VisTrails/VisTrails/pull/1006 #1006])
 
-
** added port specs are sorted at a separate spot (Module.*_port_specs properties) than those in the registry (which are sorted in the registry), but those two lists are just combined without respect to sort keys
 
-
** need to determine whether the two lists should be merged or remain distinct
 
-
** should make sure that order of input ports and output ports makes parallel connections for things with same order
 
-
** DK suggests breaking backward compatibility here: workflows still run, can fix easily if a problem in an existing package.
 
-
* [RR] Question about output modes ([https://github.com/VisTrails/VisTrails/pull/1007 #1007]), how to integrate in API ([https://github.com/VisTrails/VisTrails/pull/24 #24])
 
-
** Should ImageFileMode be removed? ("image" is not a mode, "file" is)
 
-
** ImageOutput missing?
 
-
** Feel free to change how formats works
 
-
* [TE] Test suite segfaults on Fedora 17 virtualbox. Install newer version? (Support ended in 2013)
 
-
 
-
=== March 4, 2015 ===
 
-
'''Updates'''
 
-
* [TE] VTK wrapper
 
-
** Works on VTK 5.10
 
-
** Still need to test VTK 6
 
-
** New general wrappers for python functions and classes
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Unmark UV-CDAT/VisTrails as a fork of VisTrails/VisTrails?
 
-
** When [http://i.imgur.com/UTY2qfg.png filing a pull request] from UV-CDAT/VisTrails, VisTrails/VisTrails is selected by default
 
-
** People keep forgetting to change the default ([https://github.com/VisTrails/VisTrails/pull/956 #956] [https://github.com/VisTrails/VisTrails/pull/968 #968] [https://github.com/VisTrails/VisTrails/pull/999 #999] [https://github.com/VisTrails/VisTrails/pull/1000 #1000] [https://github.com/VisTrails/VisTrails/pull/1003 #1003] [https://github.com/VisTrails/VisTrails/pull/1004 #1004] [https://github.com/VisTrails/VisTrails/pull/1005 #1005])
 
-
** Only way to change that is to not have it marked as a fork of VisTrails/VisTrails
 
-
** Github staff [https://gist.github.com/remram44/b364c24ef1e462ce6e2f can make that change for us]; should we do it?
 
-
** Juliana: comments on this? visibility vs. convenience/annoyance for developers
 
-
* [RR] What about [https://github.com/VisTrails/VisTrails/issues/991 #991]?
 
-
 
-
=== February 25, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] vtkviewcell for infovis support, can we unify with VTKCell?
 
-
** need to test this
 
-
* [TE] vtk wrapping
 
-
** Mostly finished
 
-
** VTK 5.10 produce incorrect results with old wrapping
 
-
*** Old wrapper is based mostly on VTK 4
 
-
*** Most vtk_examples affected
 
-
*** [http://www.vtk.org/Wiki/VTK/VTK_6_Migration/Replacement_of_SetInput]
 
-
*** should be able to upgrade from SetInput to SetInputData (need to drop GetOutput and replace with self ports)
 
-
*** can we change vtkInstance to just return self and not wrap things
 
-
** terminator example not working under 5.8?
 
-
* How does VTK wrapping fit into general wrapping framework?
 
-
* [RR] new persistence package
 
-
 
-
=== February 18, 2015 ===
 
-
'''Updates'''
 
-
* [RR] New VisTrails API and IPython integration ([https://github.com/VisTrails/VisTrails/pull/24 #24])
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] VTK wrapping
 
-
** [https://gist.github.com/rexissimus/16b74693545b5e7c1035 Benchmarking vtk package]
 
-
*** Old: 24.7 seconds
 
-
*** New: 10.5 seconds (Except first time that adds 8 sec)
 
-
**** The parsing that calls is_abstract (that tries to instanciate all vtk classes) is now only run the first time.
 
-
**** get_items_from_sigstring takes 2 seconds, maybe we can use a lookup dict for already computed sigstrings?
 
-
** Now using a general python function wrapper
 
-
*** VTK classes are wrapped into python function that does not depend on vistrails
 
-
*** VTK functions can be executed without vistrails
 
-
*** The spec maps functions into vistrails modules, but can also describe wrapping
 
-
*** A general python function wrapper that supports
 
-
**** kwarg inputs
 
-
**** single, list, dict outputs
 
-
**** callback for progress reporting
 
-
**** temporary file generator for using FilePool
 
-
**** optional output generation
 
-
*** Creating specs:
 
-
***** Create spec by hand
 
-
***** Auto-create spec outline (TODO) and manually finish it
 
-
***** Dynamically create spec (VTK)
 
-
***** Implement documentation wrappers (Can use scikitlearn wrapper to wrap numpydoc) (TODO)
 
-
***** Classes as bad functions needs to be wrapped in new functions before they are wrapped. This is different for each package.
 
-
****** Classes is hard: Like VTK, and matplotlib. Scikit-learn does still not wrap classes
 
-
***** Spec diffing and patching could be done using code from matplotlib.
 
-
** Still needs upgrades from old VTK package
 
-
*** Is it possible to dynamically wrap functions, e.g, you see a SetFunc and just remove the 'Set' prefix. Or do you need to create a complete mapping?
 
-
 
-
=== February 11, 2015 ===
 
-
'''Updates'''
 
-
* Update from Friday's meeting
 
-
** discussed VisTrails internals
 
-
** discussed wrapping
 
-
*** xml discussion, hard to modify because tied to db code
 
-
*** TE has made it possible to add the schema-defined attributes to the intermediate representation
 
-
*** higher-level operations on the port specs
 
-
** make sure the simple case works
 
-
*** [JF] take a simple package with documentation and figure out what the base case for wrapping is
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] VTK wrapping
 
-
** Dynamic loading works
 
-
** Reading XML is fast enough, but [https://gist.github.com/rexissimus/8272d3c776ccfa115d82 unserializing data is slow]
 
-
** Working on patterns for patching
 
-
** Matplotlib has many advanced patterns like argument ordering, nested arguments, alternateSpecs, output types.
 
-
*** Having all this in a general wrapper might confuse users?
 
-
** [RR] Delay module (except for identifiers) until you need it---e.g. don't deal with port specs, etc. until necessary
 
-
* Scripting Support [https://github.com/VisTrails/VisTrails/pull/950 #950]
 
-
** [RR] Issue with getting code from modules
 
-
** Design a simple solution
 
-
** [JF] Couldn't you use modules as black boxes without conversion, just to call into modules/subworkflows easily from e.g. IPython?
 
-
*** [RR] This is a job for the API, and a very separate use case. See [https://github.com/VisTrails/VisTrails/pull/24 #24]
 
-
 
-
=== February 4, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* Wrapping
 
-
** Format to use? Currently XML (like current matplotlib)
 
-
*** JSON and YAML have simple "to python dictionary" methods
 
-
*** But don't stream
 
-
*** YAML a lot easier for humans
 
-
** [DK] vtk-new-package also changes parameter names, creates enumerations
 
-
*** intermediate schema needs to be extensible
 
-
*** packages will want to store there specific infos for compute() method generation
 
-
*** also might have specs-altering info, like matplotlib's alternateSpec
 
-
** representation to code , registry already has schema for some aspects
 
-
** [RR] We might want to see if Module subclasses can be created lazily
 
-
*** no need to create all the classes just to register them in the registry and never actually use most of them
 
-
*** future effort
 
-
* [RR] Where should VisTrails packages live?
 
-
** tej installs as 'vistrailspkg.tej', TE installed it as 'userpackages.tej'
 
-
** Currently, standard packages are 'vistrails.packages.', user packages are 'userpackages.' and packages loaded through pkg_resources might be anything
 
-
** [RR] Use 'vistrailspkg.' everywhere?
 
-
** Long-term effort to simplify package distribution/installation (and have VisTrails get them automatically?)
 
-
 
-
=== January 28, 2015 ===
 
-
'''Updates'''
 
-
* T. Caswell to come visit on Fri 6 to discuss wrapping work
 
-
 
-
'''Items to Discuss'''
 
-
* [TE] New VTK wrapping
 
-
** Current code by DK seems a good deal faster
 
-
** Generates XML that can be patched/tweaked, generates Python code from it, VisTrails only loads generated Python code
 
-
** RR would rather have VisTrails load intermediate representation (= XML) directly, wants to make sure this is not slower
 
-
** The goal is to turn the intermediate step into something generic that would be used for every wrapped package (vtk, numpy, matplotlib, sklearn, java) instead of each having its own
 
-
** TC has its own code at [https://github.com/Nikea/VTTools/blob/master/vttools/wrap_lib.py github:VTTools] which parses numpy docstrings and generates modules, doesn't yet handle classes or persist anything
 
-
* Web crawler
 
-
** Right now, TE starts jobs for "start crawler", "stop crawler", "install classifier"
 
-
** RR would rather have the crawling be a job as far as VisTrails and tej are concerned
 
-
** The whole thing would be one pipeline: load examples, train classifier, start crawler [check for job, kill previous one, upload model, start processes], get snapshot, visualize
 
-
*** Need some support in tej and job submission system: long-running jobs, stop a job (wait for it to finish?), restart a job even though results are cached
 
-
 
-
=== January 21, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Unified wrapping method discussion [https://github.com/VisTrails/VisTrails/issues/991 #991]
 
-
** TE to work on reusable method with intermediate representation, starting with VTK
 
-
* [RR] Examples for scikit-learn: JF has an old example using Weka with parameter exploration (not currently in source tree)
 
-
** AM's examples are enough
 
-
* [AM] scikit-learn package is done, merge it in? [https://github.com/VisTrails/VisTrails/pull/955#issuecomment-70286788 #955]
 
-
** RR will merge
 
-
* [RR] What should copyright headers say? [https://github.com/VisTrails/VisTrails/pull/994#issuecomment-70173935 #994]
 
-
** Let's keep everything in there: Utah/Poly/NYU
 
-
 
-
=== January 14, 2015 ===
 
-
'''Updates'''
 
-
* [TE] Working on classifier
 
-
* [RR] Scripting integration, work in progress
 
-
 
-
'''Items to Discuss'''
 
-
* [RR] Unified wrapping method discussion ([https://github.com/VisTrails/VisTrails/issues/991 #991])
 
-
** Let's talk next week, [AM] and [DK] are not here
 
-
 
-
=== January 7, 2015 ===
 
-
'''Updates'''
 
-
 
-
'''Items to Discuss'''
 
-
* make sure that we address critical issues, questions, and pending review branches in a timely manner
 
-
* scripting support
 
-
** [RR] no issues if we want to just keep annotations in the generated code to allow the link back to a workflow
 
-
** [RR] can translate from workflow to script, working on script to workflow
 
-
** will work for parameter value changes, structural changes require changes to the annotations
 
-
** need to publish best practices here
 
-
** would be cool to do looping in scripts (easier interface than with workflows)
 
-
* notebook support (convert form notebook to workflow)
 
-
** RR will sync with FC on this
 
-
* Issue with console in built-from-scratch
 
-
** [TC] iPython rearranged some of the completion stuff in 2.2 and 2.3
 
-
** binary has old version of iPython -> 1.0.0, should we update?
 
-
* [TC] automated wrapping of numpy and scipy
 
-
** discovered a bunch of malformed documentation in numpy and scipy
 
-
** has github repo for vistrails tools
 
-
** example modules wrap a bunch of R stuff (not baked in, just how things are)
 
-
** will be pushing wrapping logic up
 
-
** port names forbidden (window and domain)
 
-
** have an import hook to get from yaml directly to VisTrails Modules
 
-
** should work for any python modules with well-formed numpy docstrings.
 
-
* [Action] should make it clear in documentation that Constant now means serializable not that the value doesn't change (e.g. List)
 
-
* [TC] might be interesting to try to build components of matplotlib and accumulate in figure (long-term project, but thinking about how this might work)
 
-
* [TE] build and build scripts
 
-
** completely automatic, buildbot
 
-
** need to set the build machines for the environment we want for the binary
 
-
** would virtualenv work here?
 
-
** [TC] anaconda can pin versions, potential path to test different configurations
 
-
** Q: upload nightly binary builds? A: makes sense, make sure they are well-labeled
 
-
* sourceforge stats: e.g. http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
 
-
* package issues (see Remi's message)
 
-
* [TE] Scope of tej
 
-
** Support single ssh commands?
 
-
** Queue can be used as a remote machine (crawler is using queue.call*)
 
-
* SourceForge stats: http://sourceforge.net/projects/vistrails/files/vistrails/nightly/vistrails-src-nightly.tar.gz/stats/timeline?dates=2014-01-01+to+2015-01-07
 
== Older meetings ==
== Older meetings ==
 +
* [[Development/2015]]
* [[Development/2014]]
* [[Development/2014]]
* [[Development/2013]]
* [[Development/2013]]

Revision as of 15:14, 30 August 2016

See also the Github wiki

2016

August 31, 2016

Updates

  • [RR] Went over vgc.poly.edu/vistrails.org config:
    • Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
    • Fixed usersguide scripts (won't disappear again)
    • Moved some websites over to HTTPS
    • Websites

Items to Discuss

  • [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
    • Get rid of more services? (svn, trac, ...)

August 17, 2016

Updates

  • [TE] Fixed parameter order upgrade bug that breaks VTK examples.
  • [TE] Matplotlib and sklearn are exportable using module-to-script api
  • [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.

Items to Discuss

  • Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
  • Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
  • [TE] Wrap-up notes

August 10, 2016

Updates

Items to Discuss

  • [TE] Python Wrapper and scripting
    • All changes merged into python-wrapper branch
      • Tests pass
      • Still need to update sub-branches
    • Minor fixes
      • vtkPlot3DReader upgrade bug
      • Added tuple exporting
    • Most of vtk examples can be exported example
      • Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
        • Could be a bug in the old vistrail, but not sure
  • [DK] Subworkflows and uuids

August 3, 2016

Updates


Items to Discuss

  • [TE] Wrapper patches now stored as python code
    • Connection variables now lowercase
    • Can we use code as output variables?
      • E.g. "vtk_dataset_mapper.GetOutput(0)"
      • Could mutate output when output method is only supposed to be called once
      • Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)

July 27, 2016

Updates

  • [DK] DB bundle serialization implemented

Items to Discuss

   def vtkInstance():
       self.vtkInstance = self

July 20, 2016

Updates

Items to Discuss

  • [TE] New patch method for python-wrapper example
    • A patch is a string template with self/input/output variables for patching a class method
    • Patches are stored in xml specification
    • Patches are assigned to methods in ClassSpec's
    • Used both for execution in vistrails and when exporting as script
      • Executed with "exec" when run in vistrails
      • Turns patches into natural calls in script
    • Wrapping VTK library no longer needed
      • Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
    • Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
    • Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
    • TODO's
      • Add to export_as_script
      • Patch Modules that represent Python functions as well
  • [vistrails-users] Read data from SQL example?

July 13, 2016

Updates

  • [DK] bundles on sql-alchemy branch
    • Bundle, BundleMapping, and BundleSerializer
    • New BaseSerializers to deal with versions of directory, zip, db serializers
    • Can register the BundleSerializers with the base serializers
    • Translation between bundle versions
    • Bundle.add_object "just works" if an appropriate mapping was registered

Items to Discuss

  • [TE] Simplified running module using API for scripting example
    • API Module can be executed using compute method
      • Takes list of outputs to compute + inputs as keyword arguments
  • Status of general cell synchronization (vistrails-users)

July 6, 2016

Updates

  • [DK] Working on bundles

Items to Discuss

  • [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
    • Call module in script using vistrails API
    • Can be used when no export method is defined.
  • [TE] Server news bug Critical?
    • [RR] No, unreleased
  • [DK] New bundle organization
    • BundleMapping contains individual mappings from raw objects to BundleObj
    • Must be separate in order to have in-memory, separate from serializer settings

June 29, 2016

Updates

Items to Discuss

  • [TE] Export VTK as python script
    • Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
    • Preludes (imports) still need improvements
    • Works with vtkInteractionHandler (clipping plane)
    • Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
    • vtkRendererOutput Uses VTKRenderWindow as output
      • Should support different output modes?
      • Does not work with multiple outputs (Next cell is displayed when current is closed)
      • Uses wrapped vtkInstance (as vvtk)
        • Create new patch system with code that can be used both in vistrails and in a python script?

June 22, 2016

Updates

Items to Discuss

  • [TE] Fix for upgraded analogies (PR #1177)
    • Uses module location to find module remap (The best we can do right now?)
    • Also fixes remapped port names for connections to existing modules
    • May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
  • [TE] Working on exporting wrapped libraries (VTK)

June 15, 2016

Updates

  • [TE] Added depth logic to python script exporter (full commit)
    • Fully implements connection merging and list looping

Items to Discuss

  • [TE] Making analogy work on upgraded workflows (#1175)
    • Analogy only works We need to use the non-upgraded action chain
    • But pipelines need to be valid (matcher uses module portspecs)
    • Tried to get upgrade remap from upgrade action, but this seems impossible
    • Use module location instead?
      • Location should be identical for one-to-one module upgrades

June 1, 2016

Updates

  • VisTrails 2.2.4 announced (finally)
    • 5 survey replies
    • 47 downloads last week

Items to Discuss

  • [TE] Export VTK as script (python-wrapper)
    • to_python_script
      • needs the pipeline (or at least the connections)
        • Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
        • VTK needs to know connection order
        • Adding functions directly to the script results in nicer code
    • Add support for multiple input connections
    • Add support for list looping

May 25, 2016

Updates

  • Mailing-list fixed
  • Getting stats from the "server news" mechanism (via parser)

Items to Discuss

  • [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
  • [TE] Union ports
    • Show union ports in pipeline (DEMO)

May 18, 2016

Items to Discuss

  • [TE] mailing list still down - Julio working on it
  • [TE] Improved union port
    • New schema 1.0.5 adds portitem.union string attribute
    • Works with pipeline view, parameter exploration view, and mashup view
    • Should work well for matplotlib
    • Use with VTK ports?
      • It uses [item_1,item_2] style ports usually with different tuple length.
  • [TE] Remove emacs dependency from db generate #1173

May 11, 2016

Items to Discuss

  • [TE] Union type #1172
    • GUI-based solution Branch union-port Demo?
      • Shows single port for all ports with same sort_key
      • Select specific type when adding/showing
      • Only one type should be visible on the module at any time
      • Making tuple parameters union types is probably too complicated
      • Should work with alternate specs like matplotlib (if sort_key is defined correctly)
  • [TE] Feature suggestion: Let connections connect to functions/parameters
    • Show functions and parameters as ports and toggle port visibility directly for functions and parameters
    • Use separate views for ports and functions (like before)
      • Allow rearranging of functions
    • Pros:
      • Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
        • We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
        • Makes List module simpler
      • Connect to parameter in tuple
      • Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
      • Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
    • Cons:
      • Interface more complicated and crowded
      • More difficult for users to grasp?
      • No immediate use case
      • Makes no union type for tuple parameters a bigger problem.
  • [TE] Publish user survey?
    • No response from vistrails release yet
    • Not announced on vistrails-users yet.

May 4, 2016

Updates

  • VisTrails 2.2.4 released

Items to Discuss

April 27, 2016

Updates

Items to Discuss

  • [TE] Wrapping Bokeh
    • New method type "operation"
      • Transforms Figure->Glyph->Output into Glyph->Figure->Output
        • Glyph sets function name and arguments as output
          • Not serializable, but should always be computed together with the subject of the operation.
        • Figure applies Glyph operation on itself
      • Useful for matplotlib?
        • matplotlib currently find actors by searching upstream?
        • But this is more difficult since matplotlib is not function-based.
    • New "union" port type
      • Turns "x(float), x(list)" into "x (float, list)" and uses first set value
        • Uses only first value in tuple
        • Bokeh have many properties that support (list, string, Float)
      • Does not work with depth=1 types (e.g. Float of depth 1)
        • Turn Depth 1 types with known type into lists?
  • [TE] New bugfix release
    • Finalize survey

April 20, 2016

Updates

  • Merged upgrade fixes
    • Now has infinite loop test
    • loop test now correctly loops 5 times
      • remap_module used wrong (current) package version when creating ModuleDescriptor
  • Fixed multi-step group upgrades (PR: #1168)

Items to Discuss

  • New bugfix release
    • Merge usagestats
    • Need to finalize survey

April 13, 2016

Updates

Items to Discuss

  • [TE] Chaining upgrades done (#1164)
    • Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
  • [TE] Iterating handle_invalid_pipeline (#1165)
    • Not based on chain-upgrades-loop
    • Cherry-pick test to preserve source?
    • Need maxPipelineFixAttempts option?
      • The code is now smarter about recursion, so max attempts will "probably" never be reached.
    • Need debug messages?
      • Would need to be moved into handle_invalid_pipeline

April 6, 2016

Updates

  • [TE] Query View improvements
    • Done except for displaying "Version Info" and #1164
  • [RR] Working on finishing new interpreter (finally!)

Items to Discuss

  • [TE] Chaining upgrades (#1164)
    • Try latest upgrade first, then previous.
    • If all fail, select latest upgrade?

March 30, 2016

Updates

Items to Discuss

  • [TE] Query View improvements
    • Unified do_version_switch and get_upgrades as validate_version()
    • Problems with query view using its own controller
      • Fixed module info not being displayed for query results
      • Still no version info for query results
      • Still an issue with version query view not being updated

March 23, 2016

Updates

Items to Discuss

  • [RR] Usage reporting almost ready to go
    • Need to record whether specific features are used: paramexpore, mashup, query
      • [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
    • How to record size of vistrail?
      • [DK] IdScope already walks over vistrail, use that
      • but total number of modules and versions doesn't give an idea of average size of individual pipelines

March 16, 2016

Updates

Items to Discuss

  • [TE] Upgrades in Query Mode
    • Now using code similar to do_version_switch
      • Uses handle_invalid_pipeline
      • Differences: Not using current_pipeline, flushes changes
      • New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
      • Integrate to avoid repeated code?
      • Use this for query mode?
  • [TE] VisTrails 2.2.4 ready for release?
    • tej package updated. Others?
  • [JF] upgrades and the version tree
  • [JF] provenance in Jupyter
  • [DK] bundles: #1092

March 9, 2016

Updates

Items to Discuss

  • [TE] Fixing search mode
    • String query now matches all by default
      • This caused pipeline queries to never match
    • String query can search: User/Notes/Tagname/Time/Module name
      • Should we search module names by default? (Disabled by default because it is expensive)
      • Fixed notes search
    • '>' parameter queries in documentation but has been removed
    • Fixed module matches for upgraded pipelines (By not upgrading pipeline)
      • Upgrades still a problem
      • Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
      • [JF] Allow search on past tags
  • [TE] Create new bugfix release
    • Finish querying
    • Path issue on windows
  • [DK] Look into sqlalchemy and new bundle branches

March 2, 2016

Updates

  • [TE] Fixed cache bug with port specs
  • [TE] VisTrails Next
    • Most packages now work
    • No qgis for Python 3

Items to Discuss

    • Always dereference symlinks? #1146

February 23, 2016

Updates

  • [TE] Fixed PythonSource 100% CPU bug
  • [TE] VisTrails Next
    • Updated NOAA Web Service examples
      • Now uses REST service, tabledata, and Google Maps
      • Web Service replaced with DownloadFile

Items to Discuss

  • [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
    • New 2.2 release soon?
  • [RR] Anonymous usage reports
    • Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
  • [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
  • noWorkflow - provenance for python scripts
    • Recent work on tracking dependencies
  • Also integrates ReproZip with Jupyter
    • mybinder
      • Use with ReproZip to automatically build Jupyter notebook interfaces
  • Daves comments on Jupyter integration
    • It is possible to identify cell order on the server side
    • Not trivial to track order of cells on client side
  • How does Galaxy workflows relate to VisTrails?
    • Packaging in Galaxy is hard
    • Galaxy for biomedical workflows, difficulty making it general
  • Action items
    • How difficult to track cell dependencies in Jupyter notebooks?
    • Use noWorkflow to track cell dependencies?
    • How difficult to modify mybinder to use ReproZip packages?

February 17, 2016

Updates

Items to Discuss

  • [TE] Fixed Group memory usage
    • Modules in groups are supposed to be freed when done executing
      • Prevented by logger that stored references to the modules (fixed, now using id(module))
      • Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
  • [TE] VisTrails Next
    • Removed deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • PythonSources will be upgraded
      • Fixes for Python < 3
      • fixes for VTK < 6
    • TODOs
      • Remove RemoteQ
        • Port Hadoop to tej (Already have pbs)
      • Merge package wrapper
      • Finish SQLAlchemy branch
      • Finish new bundle branch
    • [JF] Will contact Jeff and Matthias about moving to Python 3
  • [JF] Query by Example is broken
  • [JF] VisTrails Data Cleaning Project
    • Use VisTrails to process and explore task-specific data while keeping provenance.
    • Remi's Idea
      • Use VisTrails as the backend in a Jupyter notebook
      • Overcome problems with executing cells in random order
        • Interesting research topic, but little interest has yet been noted
      • Interesting to study how such a notebook would be used (Is there notebook usage data available?)
      • Impose restrictions to notebook cell boundaries
      • It will only work for functional operations
      • Can we fix, or detect, non-functional (mutating) operations?
      • Analyse code and data flow and see if it mutates data in unexpected ways?
      • Mutable operations can be created in many ways in Python
  • Next meeting Tuesday 11:30

February 10, 2016

Updates

Items to Discuss

  • VisTrails Next
    • Test suite passes
      • Python 3 less deterministic, module upgrade order random.
    • [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
    • All of our packages should run fine :)
    • VTK thumbnail comparison works
      • Rendering bug on some systems
        • QVTKCell now inherits QGLWidget
      • Old thumbnails had wrong height due to spreadsheetwindow being too small
        • Fixed by regenerating thumbnails
      • Silenced deprecation messages when generating modules
      • Many new classes in VTK 7
    • Remove deprecated code
      • Core imports that should be gui imports
      • GetInputFromPort etc.
        • Will break existing PythonSources
        • Try to upgrade?
      • Old-style packages
        • Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
          • Read info.py without importing package?
      • Fixes for Python < 3
      • Fixes for Qt < 5
      • fixes for VTK < 7
      • Remove self port?

February 3, 2016

Updates

Items to Discuss

  • VisTrails Next
    • All tests but thumbnail comparison passes
    • Still issues with gui and more packages needs to be tested
    • PythonSource and MplSource upgraded with lib2to3
    • Keep contrib packages? Update?
    • VTK 7 is released
      • Supports Python 3
      • Will hopefully get into conda and others soon
      • Deprecates a bunch of modules
  • Travis builder for VisTrails Next
    • Runs!
    • Uses conda
      • Many levels of requirements
        • apt packages
        • conda packages
        • python packages in requirements.txt
        • python packages in setup.py
      • Can probably drop conda. Default builder does not have Qt5, but beta builder do!
      • But useful if we want to use conda in our own build process.
    • Lots of errors I do not get locally?
  • User Survey
    • Have you used any alternatives to vistrails?
      • Relevant? More options?
    • How often do you use any of these particular features in VisTrails?
      • Add/Remove alternatives?
    • How important would you rank the following potential improvements to VisTrails?
      • Add alternatives?

January 27, 2016

Updates

  • New interpreter
    • Working prototype soon
    • Looping not completely done (and port combination mode: pairwise/cartesian...)
    • Need to integrate with VisTrails
      • Compatibility with existing modules
      • Use log controller
      • Want to cache at multiple levels (whole stream/single element of stream)
      • Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)

Items to Discuss

  • User Survey
  • ZMQ failing on windows (vistrails-users)
  • [TE] Python 3 branch
    • Testsuite almost passes
      • Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
    • Fixed issues
      • Signatures are now bytes (convert to string to show in widget)
      • Cannot compare different types, e.g., (None < 1)
        • PortSpec.depth and min/maxConn could be None. (Backport!)
        • cell.row could be None
      • Lots of unicode/bytes fixes to serializations
        • We have encryption implementations in the code! (keychain)
      • Lots of small fixes to package method differences
    • Current issues
      • logging widget empty?
      • version tree signals messed up
      • provenance view legend icons black
      • Text do not fit in some widgets
      • VTK thumbnail comparison fails
        • VTK Cells have some faces in wrong direction?
        • Because running in VM?
    • Installing on Ubuntu LTS/Latest Mint
      • Install latest matplotlib from pip
      • Install VTK 7.0.0rc2 from source
      • No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)

January 20, 2016

Updates

Items to Discuss

  • [TE] Backporting features from Python 3 + PyQt5 + Qt5
    • Merged unicode with master (unicode)
      • One unicode test failing
      • Problem saving to xml
    • Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
      • Updates print to functions 'from __future__ import print_function'
      • Updates exception calls to use 'as'
      • removes most usages of 'long' (mostly used unnecessarily)
        • Only needed for 32-bit support
      • Fixes relative imports (from __future__ import absolute_import)
    • Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
      • Updated all signals to new-style
  • If we go 2 to 3, this could break packages
    • means we should schedule any package format changes for the same time
    • e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
    • any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
    • putting package identifier as xml?
      • Python not so bad, plus some stuff in there is Python code (checking dependencies)
    • version the APIs?
      • RR planning on versioning vistrails_module.Module, no point versioning the whole package?
  • file formats on 2 to 3:
    • no real translation/upgrade, but non-ascii values that work on 3 won't load in 2

January 13, 2016

Updates

Items to Discuss

January 6, 2016

Updates Items to Discuss

  • Updating VisTrails dependencies
    • Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
      • VTK 7 is the first supporting Python 3 but is only in release candidate phase.
        • Next version of VTK 6 is rumored to support Python 3
        • There is a development branch of VTK 6 supporting Python 3
      • Can travis build this?
        • We can skip tests for WebEngine and VTK if needed
    • Python 3 showstoppers
      • No MySQL-python (sqlalchemy supports mysql through other drivers)
      • No suds (there are active forks but we could also drop support?)
  • Options for python2/3 support:
    • Interpreter could be python2/python3 based on packages
    • six layer works but not the easiest to maintain (e.g. in reprozip)
  • Drop support for win-64
  • Update copyright headers to 2016

Older meetings

Personal tools