Difference between revisions of "Development"
m (Move 2015 meetings to separate page) |
|||
Line 18: | Line 18: | ||
* [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/ | * [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/ | ||
** Get rid of more services? (svn, trac, ...) | ** Get rid of more services? (svn, trac, ...) | ||
* [RR] Notebook execution works. How do we support more languages (than Python)? | |||
** Need: detection, metadata reader, code to input values | |||
=== August 17, 2016 === | === August 17, 2016 === |
Revision as of 14:45, 31 August 2016
See also the Github wiki
2016
August 31, 2016
Updates
- [RR] Went over vgc.poly.edu/vistrails.org config:
- Added TLS certificate (vistrails.org, www.vistrails.org, lists.vistrails.org, vgc.poly.edu and a few others)
- Fixed usersguide scripts (won't disappear again)
- Moved some websites over to HTTPS
- Websites
Items to Discuss
- [RR] Move www.vistrails.org (wiki) to HTTPS? There is a trac at https://www.vistrails.org/
- Get rid of more services? (svn, trac, ...)
- [RR] Notebook execution works. How do we support more languages (than Python)?
- Need: detection, metadata reader, code to input values
August 17, 2016
Updates
- [TE] Fixed parameter order upgrade bug that breaks VTK examples.
- [TE] Matplotlib and sklearn are exportable using module-to-script api
- [TE] Need to add to_python_script to pythonfunction.py to get bokeh and numpy export working.
Items to Discuss
- Version checks: Please use a method instead of hard-coded `try: int(v) except ValueError: ...` logic!
- Mashups: why do they share the same id scope?? (see MashupsManager.createMashupController)
- [TE] Wrap-up notes
August 10, 2016
Updates
Items to Discuss
- [TE] Python Wrapper and scripting
- All changes merged into python-wrapper branch
- Tests pass
- Still need to update sub-branches
- Minor fixes
- vtkPlot3DReader upgrade bug
- Added tuple exporting
- Most of vtk examples can be exported example
- Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
- Could be a bug in the old vistrail, but not sure
- Sometimes connections are wrong, executing workflow sometimes works, otherwise copy/pasting is necessary
- All changes merged into python-wrapper branch
- [DK] Subworkflows and uuids
August 3, 2016
Updates
Items to Discuss
- [TE] Wrapper patches now stored as python code
- Connection variables now lowercase
- Can we use code as output variables?
- E.g. "vtk_dataset_mapper.GetOutput(0)"
- Could mutate output when output method is only supposed to be called once
- Need to be able to rename output port as python code(e.g. when there are 2 "vtk_dataset_mapper"s)
July 27, 2016
Updates
- [DK] DB bundle serialization implemented
Items to Discuss
- [TE] Finished using patch in script python-wrapper-script-patches
- locale?
- new patching example vs old patching example
- Example of new patches
- Legacy '.vtkInstance' is now added as a patch "self.vtkInstance=self" (see new patching example)
- New api for executing standalone modules
- Accessing module was disabled for core/api, but a different api for this is probably better anyway.
- api.Package now gives directly callable module compute functions
- Only works on modules not using interpreter functionality ('.interpreter', etc.)
- for zip file modules and others that need temp files and use the interpreter for this, need to modify the module, but could work around with python libraries for managing temp files
- Fixed more renaming issues - function argument names should not be replaced
- Still need to do something nicer for the patch formatting
- patch keys currently contain non-python-variable characters but this can be fixed
- Need to replace string template with python variables that is replacable using RedBaron
- Patches could then look like:
def vtkInstance(): self.vtkInstance = self
July 20, 2016
Updates
Items to Discuss
- [TE] New patch method for python-wrapper example
- A patch is a string template with self/input/output variables for patching a class method
- Patches are stored in xml specification
- Patches are assigned to methods in ClassSpec's
- Used both for execution in vistrails and when exporting as script
- Executed with "exec" when run in vistrails
- Turns patches into natural calls in script
- Wrapping VTK library no longer needed
- Removes ".vtkInstance" (Automatically removed from PythonSources and vtkInteractionHandlers)
- Removed fix_classes.py (Subclassing of VTK classes to patch methods directly)
- Also used for port translations (With patch names like "basic:Color#input") (No more ugly input_t method definitions)
- TODO's
- Add to export_as_script
- Patch Modules that represent Python functions as well
- [vistrails-users] Read data from SQL example?
July 13, 2016
Updates
- [DK] bundles on sql-alchemy branch
- Bundle, BundleMapping, and BundleSerializer
- New BaseSerializers to deal with versions of directory, zip, db serializers
- Can register the BundleSerializers with the base serializers
- Translation between bundle versions
- Bundle.add_object "just works" if an appropriate mapping was registered
Items to Discuss
- [TE] Simplified running module using API for scripting example
- API Module can be executed using compute method
- Takes list of outputs to compute + inputs as keyword arguments
- API Module can be executed using compute method
- Status of general cell synchronization (vistrails-users)
July 6, 2016
Updates
- [DK] Working on bundles
Items to Discuss
- [TE] Using vistrails API in scripting example 1 (API with VTK) example 2 (DownloadFile only)
- Call module in script using vistrails API
- Can be used when no export method is defined.
- [TE] Server news bug Critical?
- [RR] No, unreleased
- [DK] New bundle organization
- BundleMapping contains individual mappings from raw objects to BundleObj
- Must be separate in order to have in-memory, separate from serializer settings
June 29, 2016
Updates
Items to Discuss
- [TE] Export VTK as python script
- Works on Terminator (Isosurface, Volume Rendering, clipping plane, combined rendering) example
- Preludes (imports) still need improvements
- Works with vtkInteractionHandler (clipping plane)
- Resulting clipping plane is incorrect? (supposed to hide one side of the plane)
- vtkRendererOutput Uses VTKRenderWindow as output
- Should support different output modes?
- Does not work with multiple outputs (Next cell is displayed when current is closed)
- Uses wrapped vtkInstance (as vvtk)
- Create new patch system with code that can be used both in vistrails and in a python script?
June 22, 2016
Updates
Items to Discuss
- [TE] Fix for upgraded analogies (PR #1177)
- Uses module location to find module remap (The best we can do right now?)
- Also fixes remapped port names for connections to existing modules
- May still fail on complex upgrades like adding a function on an upgraded module that has a different port name
- [TE] Working on exporting wrapped libraries (VTK)
June 15, 2016
Updates
- [TE] Added depth logic to python script exporter (full commit)
- Fully implements connection merging and list looping
Items to Discuss
- [TE] Making analogy work on upgraded workflows (#1175)
- Analogy only works We need to use the non-upgraded action chain
- But pipelines need to be valid (matcher uses module portspecs)
- Tried to get upgrade remap from upgrade action, but this seems impossible
- Use module location instead?
- Location should be identical for one-to-one module upgrades
June 1, 2016
Updates
- VisTrails 2.2.4 announced (finally)
- 5 survey replies
- 47 downloads last week
Items to Discuss
- [TE] Export VTK as script (python-wrapper)
- to_python_script
- needs the pipeline (or at least the connections)
- Should not generate code for any possible VTK method (exporter detects which ports is used in the source)
- VTK needs to know connection order
- Adding functions directly to the script results in nicer code
- needs the pipeline (or at least the connections)
- Add support for multiple input connections
- Add support for list looping
- to_python_script
May 25, 2016
Updates
- Mailing-list fixed
- Getting stats from the "server news" mechanism (via parser)
Items to Discuss
- [RR] Alexis doing writeup on VisTrails, curious about interesting streaming use-cases (and potential future users)
- [TE] Union ports
- Show union ports in pipeline (DEMO)
May 18, 2016
Items to Discuss
- [TE] mailing list still down - Julio working on it
- [TE] Improved union port
- New schema 1.0.5 adds portitem.union string attribute
- Works with pipeline view, parameter exploration view, and mashup view
- Should work well for matplotlib
- Use with VTK ports?
- It uses [item_1,item_2] style ports usually with different tuple length.
- [TE] Remove emacs dependency from db generate #1173
May 11, 2016
Items to Discuss
- [TE] Union type #1172
- GUI-based solution Branch union-port Demo?
- Shows single port for all ports with same sort_key
- Select specific type when adding/showing
- Only one type should be visible on the module at any time
- Making tuple parameters union types is probably too complicated
- Should work with alternate specs like matplotlib (if sort_key is defined correctly)
- GUI-based solution Branch union-port Demo?
- [TE] Feature suggestion: Let connections connect to functions/parameters
- Show functions and parameters as ports and toggle port visibility directly for functions and parameters
- Use separate views for ports and functions (like before)
- Allow rearranging of functions
- Pros:
- Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
- We can also have, e.g., a line joining the ports on the module to indicate they are part of a list.
- Makes List module simpler
- Connect to parameter in tuple
- Functions that are visible on the module can connect to an actual function, and not just to the first one it finds.
- Can replace operation modules (in python-wrapper) in some cases for, e.g., numpy.
- Supports ordered connections. Create 2 functions, make them visible as ports, make connections to them.
- Cons:
- Interface more complicated and crowded
- More difficult for users to grasp?
- No immediate use case
- Makes no union type for tuple parameters a bigger problem.
- [TE] Publish user survey?
- No response from vistrails release yet
- Not announced on vistrails-users yet.
May 4, 2016
Updates
- VisTrails 2.2.4 released
Items to Discuss
- [TE] Union type #1172
April 27, 2016
Updates
Items to Discuss
- [TE] Wrapping Bokeh
- New method type "operation"
- Transforms Figure->Glyph->Output into Glyph->Figure->Output
- Glyph sets function name and arguments as output
- Not serializable, but should always be computed together with the subject of the operation.
- Figure applies Glyph operation on itself
- Glyph sets function name and arguments as output
- Useful for matplotlib?
- matplotlib currently find actors by searching upstream?
- But this is more difficult since matplotlib is not function-based.
- Transforms Figure->Glyph->Output into Glyph->Figure->Output
- New "union" port type
- Turns "x(float), x(list)" into "x (float, list)" and uses first set value
- Uses only first value in tuple
- Bokeh have many properties that support (list, string, Float)
- Does not work with depth=1 types (e.g. Float of depth 1)
- Turn Depth 1 types with known type into lists?
- Turns "x(float), x(list)" into "x (float, list)" and uses first set value
- New method type "operation"
- [TE] New bugfix release
- Finalize survey
April 20, 2016
Updates
- Merged upgrade fixes
- Now has infinite loop test
- loop test now correctly loops 5 times
- remap_module used wrong (current) package version when creating ModuleDescriptor
- Fixed multi-step group upgrades (PR: #1168)
Items to Discuss
- New bugfix release
- Merge usagestats
- Need to finalize survey
April 13, 2016
Updates
Items to Discuss
- [TE] Chaining upgrades done (#1164)
- Replace "Upgrade" description annotation with "upgraded_from" annotation to identify which tags to hide?
- [TE] Iterating handle_invalid_pipeline (#1165)
- Not based on chain-upgrades-loop
- Cherry-pick test to preserve source?
- Need maxPipelineFixAttempts option?
- The code is now smarter about recursion, so max attempts will "probably" never be reached.
- Need debug messages?
- Would need to be moved into handle_invalid_pipeline
April 6, 2016
Updates
- [TE] Query View improvements
- Done except for displaying "Version Info" and #1164
- [RR] Working on finishing new interpreter (finally!)
Items to Discuss
- [TE] Chaining upgrades (#1164)
- Try latest upgrade first, then previous.
- If all fail, select latest upgrade?
March 30, 2016
Updates
Items to Discuss
- [TE] Query View improvements
- Unified do_version_switch and get_upgrades as validate_version()
- Problems with query view using its own controller
- Fixed module info not being displayed for query results
- Still no version info for query results
- Still an issue with version query view not being updated
March 23, 2016
Updates
Items to Discuss
- [RR] Usage reporting almost ready to go
- Need to record whether specific features are used: paramexpore, mashup, query
- [DK] Record if LaTeX extension is used? But hard to distinguish from direct batch invocation
- How to record size of vistrail?
- [DK] IdScope already walks over vistrail, use that
- but total number of modules and versions doesn't give an idea of average size of individual pipelines
- Need to record whether specific features are used: paramexpore, mashup, query
March 16, 2016
Updates
Items to Discuss
- [TE] Upgrades in Query Mode
- Now using code similar to do_version_switch
- Uses handle_invalid_pipeline
- Differences: Not using current_pipeline, flushes changes
- New controller method get_pipeline optimized using cached pipelines (but not current_pipeline)
- Integrate to avoid repeated code?
- Use this for query mode?
- Now using code similar to do_version_switch
- [TE] VisTrails 2.2.4 ready for release?
- tej package updated. Others?
- [JF] upgrades and the version tree
- [JF] provenance in Jupyter
- [DK] bundles: #1092
March 9, 2016
Updates
Items to Discuss
- [TE] Fixing search mode
- String query now matches all by default
- This caused pipeline queries to never match
- String query can search: User/Notes/Tagname/Time/Module name
- Should we search module names by default? (Disabled by default because it is expensive)
- Fixed notes search
- '>' parameter queries in documentation but has been removed
- Fixed module matches for upgraded pipelines (By not upgrading pipeline)
- Upgrades still a problem
- Add "Upgrade all menu option?" (Or upgrade all on open when `hideUpgrades`)
- [JF] Allow search on past tags
- String query now matches all by default
- [TE] Create new bugfix release
- Finish querying
- Path issue on windows
- [DK] Look into sqlalchemy and new bundle branches
March 2, 2016
Updates
- [TE] Fixed cache bug with port specs
- [TE] VisTrails Next
- Most packages now work
- No qgis for Python 3
Items to Discuss
- Always dereference symlinks? #1146
February 23, 2016
Updates
- [TE] Fixed PythonSource 100% CPU bug
- [TE] VisTrails Next
- Updated NOAA Web Service examples
- Now uses REST service, tabledata, and Google Maps
- Web Service replaced with DownloadFile
- Updated NOAA Web Service examples
Items to Discuss
- [RR] Couple of fixes are in for API, and tej&reprounzip on the Mac binary. Also new release of tej, and working on anonymous usage reports.
- New 2.2 release soon?
- [RR] Anonymous usage reports
- Sends out info like number of modules & versions, packages used, Python & libs versions, execution time, specific features #1154
- [JF] Bring in noWorkflow to track variable dependencies, for the considered Jupyter integration?
- noWorkflow - provenance for python scripts
- Recent work on tracking dependencies
- Also integrates ReproZip with Jupyter
- mybinder
- Use with ReproZip to automatically build Jupyter notebook interfaces
- mybinder
- Daves comments on Jupyter integration
- It is possible to identify cell order on the server side
- Not trivial to track order of cells on client side
- How does Galaxy workflows relate to VisTrails?
- Packaging in Galaxy is hard
- Galaxy for biomedical workflows, difficulty making it general
- Action items
- How difficult to track cell dependencies in Jupyter notebooks?
- Use noWorkflow to track cell dependencies?
- How difficult to modify mybinder to use ReproZip packages?
February 17, 2016
Updates
Items to Discuss
- [TE] Fixed Group memory usage
- Modules in groups are supposed to be freed when done executing
- Prevented by logger that stored references to the modules (fixed, now using id(module))
- Prevented by list iterator bug that stored list of modules on the 'self' port bug (fixed)
- Modules in groups are supposed to be freed when done executing
- [TE] VisTrails Next
- Removed deprecated code
- Core imports that should be gui imports
- GetInputFromPort etc.
- PythonSources will be upgraded
- Fixes for Python < 3
- fixes for VTK < 6
- TODOs
- Remove RemoteQ
- Port Hadoop to tej (Already have pbs)
- Merge package wrapper
- Finish SQLAlchemy branch
- Finish new bundle branch
- Remove RemoteQ
- [JF] Will contact Jeff and Matthias about moving to Python 3
- Removed deprecated code
- [JF] Query by Example is broken
- [JF] VisTrails Data Cleaning Project
- Use VisTrails to process and explore task-specific data while keeping provenance.
- Remi's Idea
- Use VisTrails as the backend in a Jupyter notebook
- Overcome problems with executing cells in random order
- Interesting research topic, but little interest has yet been noted
- Interesting to study how such a notebook would be used (Is there notebook usage data available?)
- Impose restrictions to notebook cell boundaries
- It will only work for functional operations
- Can we fix, or detect, non-functional (mutating) operations?
- Analyse code and data flow and see if it mutates data in unexpected ways?
- Mutable operations can be created in many ways in Python
- Next meeting Tuesday 11:30
February 10, 2016
Updates
Items to Discuss
- VisTrails Next
- Test suite passes
- Python 3 less deterministic, module upgrade order random.
- [RR] updating packages: Reprounzip (pushed), tej, persistent_archive, ...
- All of our packages should run fine :)
- VTK thumbnail comparison works
- Rendering bug on some systems
- QVTKCell now inherits QGLWidget
- Old thumbnails had wrong height due to spreadsheetwindow being too small
- Fixed by regenerating thumbnails
- Silenced deprecation messages when generating modules
- Many new classes in VTK 7
- Rendering bug on some systems
- Remove deprecated code
- Core imports that should be gui imports
- GetInputFromPort etc.
- Will break existing PythonSources
- Try to upgrade?
- Old-style packages
- Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
- Read info.py without importing package?
- Or only allow a new style, like __init__.py renamed to info.py? (merge with identifiers.py?)
- Fixes for Python < 3
- Fixes for Qt < 5
- fixes for VTK < 7
- Remove self port?
- Test suite passes
February 3, 2016
Updates
Items to Discuss
- [vistrails-users] Rendering a .pvd (ParaView) file?
- VisTrails Next
- All tests but thumbnail comparison passes
- Still issues with gui and more packages needs to be tested
- PythonSource and MplSource upgraded with lib2to3
- Keep contrib packages? Update?
- VTK 7 is released
- Supports Python 3
- Will hopefully get into conda and others soon
- Deprecates a bunch of modules
- Travis builder for VisTrails Next
- Runs!
- Uses conda
- Many levels of requirements
- apt packages
- conda packages
- python packages in requirements.txt
- python packages in setup.py
- Can probably drop conda. Default builder does not have Qt5, but beta builder do!
- But useful if we want to use conda in our own build process.
- Many levels of requirements
- Lots of errors I do not get locally?
- User Survey
- Have you used any alternatives to vistrails?
- Relevant? More options?
- How often do you use any of these particular features in VisTrails?
- Add/Remove alternatives?
- How important would you rank the following potential improvements to VisTrails?
- Add alternatives?
- Have you used any alternatives to vistrails?
January 27, 2016
Updates
- New interpreter
- Working prototype soon
- Looping not completely done (and port combination mode: pairwise/cartesian...)
- Need to integrate with VisTrails
- Compatibility with existing modules
- Use log controller
- Want to cache at multiple levels (whole stream/single element of stream)
- Non-runtime attributes of modules (cache check, automatic conversion check, constant from/to string)
Items to Discuss
- User Survey
- ZMQ failing on windows (vistrails-users)
- [TE] Python 3 branch
- Testsuite almost passes
- Failing are unicode filenames, some users guide examples, and thumbnail comparisons.
- Fixed issues
- Signatures are now bytes (convert to string to show in widget)
- Cannot compare different types, e.g., (None < 1)
- PortSpec.depth and min/maxConn could be None. (Backport!)
- cell.row could be None
- Lots of unicode/bytes fixes to serializations
- We have encryption implementations in the code! (keychain)
- Lots of small fixes to package method differences
- Current issues
- logging widget empty?
- version tree signals messed up
- provenance view legend icons black
- Text do not fit in some widgets
- VTK thumbnail comparison fails
- VTK Cells have some faces in wrong direction?
- Because running in VM?
- Installing on Ubuntu LTS/Latest Mint
- Install latest matplotlib from pip
- Install VTK 7.0.0rc2 from source
- No QtWebEngine binary (controversial) (Must install sip/PyQt5/Qt5 from source)
- Testsuite almost passes
January 20, 2016
Updates
Items to Discuss
- [TE] Backporting features from Python 3 + PyQt5 + Qt5
- Merged unicode with master (unicode)
- One unicode test failing
- Problem saving to xml
- Used python-future to run futurize stage 1 to update legacy (pre 2.6) code (unicode-futurized)
- Updates print to functions 'from __future__ import print_function'
- Updates exception calls to use 'as'
- removes most usages of 'long' (mostly used unnecessarily)
- Only needed for 32-bit support
- Fixes relative imports (from __future__ import absolute_import)
- Fixing code that is deprecated in Qt 4.8 and removed in Qt 5 (unicode-futurized-qt5)
- Updated all signals to new-style
- Merged unicode with master (unicode)
- If we go 2 to 3, this could break packages
- means we should schedule any package format changes for the same time
- e.g., imports should be unrelated to code path (function to load module from package identifier?) (take them out of vistrails.* #1002)
- any structural changes (e.g. the __init__ versus init thing) (RR thinks it makes sense, though having both named "init" is unfortunate)
- putting package identifier as xml?
- Python not so bad, plus some stuff in there is Python code (checking dependencies)
- version the APIs?
- RR planning on versioning vistrails_module.Module, no point versioning the whole package?
- file formats on 2 to 3:
- no real translation/upgrade, but non-ascii values that work on 3 won't load in 2
January 13, 2016
Updates
Items to Discuss
- [TE] Porting to Python 3 / PyQt5
- Basic GUI and execution works
- Minimal work to support 2to3
- Hard to support both PyQt4 and PyQt5
January 6, 2016
Updates Items to Discuss
- Updating VisTrails dependencies
- Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
- VTK 7 is the first supporting Python 3 but is only in release candidate phase.
- Next version of VTK 6 is rumored to support Python 3
- There is a development branch of VTK 6 supporting Python 3
- Can travis build this?
- We can skip tests for WebEngine and VTK if needed
- VTK 7 is the first supporting Python 3 but is only in release candidate phase.
- Python 3 showstoppers
- No MySQL-python (sqlalchemy supports mysql through other drivers)
- No suds (there are active forks but we could also drop support?)
- Successfully built Win7-64/Python3/PyQt5/Qt5/WebEngine/VTK7
- Options for python2/3 support:
- Interpreter could be python2/python3 based on packages
- six layer works but not the easiest to maintain (e.g. in reprozip)
- Drop support for win-64
- Update copyright headers to 2016