Remi: Moved from Development

2015-01-08T16:20:04Z

New page

=== June 25, 2014 ===
'''Updates'''
* [DK] master merged into output-modules

'''Items to Discuss'''
* [DK] Global configuration settings for output-modules when using inheritance:
** Example: Allow different file.suffix='.dat', imageFile.suffix='.png'?
** For an individual output module, we only have one setting since imageFile derives from file, but how should this work if we a global setting for moduleA=vtkRendererOutput and moduleB=FileOutput?
** decided that overriding for specific subclases makes sense
* [TE] automatic-looping
** get_input now merges all connections when depth>0
** Typechecking now respects configuration flags - fix for float/int reverted
** Issues with allowing List<->"Type of depth 1" conversions
*** Treat list as Variant of depth 1
*** List->Variant need to be treated differently otherwise it would loop over the variant
** wouldn't be able to do implicit looping over List/Variant because we don't know what the list depth of incoming list objects?
* [RR] matplotlib issue: log doesn't reflect the actual execution
** can we update logging provenance here?
** other packages (VTK) have similar issues since the renderer actually causes some of the upstream internal modules to do computations.

=== June 18, 2014 ===

'''Updates'''
* [TE] Chris is using BatchQ

'''Items to Discuss'''
* [TE] automatic-loops-changes
** Mixing 0 and 1 depth types now works
** Removed use of moduleInfo
** transfer_attrs copies specs and control parameters
** Use get_input_list to support multiple connections
* [RR] matplotlib issues
** pushed branch to create a function that does this
** [http://mpld3.github.io mpld3] -- do they use same matplotlib model where figure must exist?
* [DK] list of depth 1, can we pass a list here
** exception of List input port type?
** will this work? allows only
* [TE] numpy types in matplotlib?
** e.g. different float types
** workaround for numpy for List types already
** what about adding other types later? how would a package developer do this?
** Converter modules could work here but add more modules? hide the extra modules?
** Unify type checking that occurs over connections (controlled by settings) and Loop/depth type checking method (now in vistrails.core.modules.vistrails_module)
* [RR] depth 1 port going to List head port
* [TE] variant for (float, integer) tuple
* [RR] multiple connections into a depth-1 port only use one of the values
** should we have a self.get_input_list(flatten=True) option?
* [DK] output-modules: should work on so we can merge

=== June 11, 2014 ===

'''Updates'''
* [TE] RemoteQ jobs can now skip upstream
* [TE] Fixed zipfile bug corrupting vt files
* [TE] Fixed ungrouping caused by
* [TE] Google maps binary available
** Updated for zipfile bug but not for ungrouping bug yet

'''Items to Discuss'''

* [TE] File as parameter is confusing
** signature changes with mtime
** Make File non-constant and use Path instead
* [TE] Talked with Chris at LSU
** PBS Jobs 20 min to a few hours
** VisTrails does most of what they need
*** Suggest we focus on bug-fixing and documentation
** Want interactivity
*** Fetch and visualize files while PBS Job is running
** May use Hadoop in the future

=== June 4, 2014 ===

'''Updates'''
* [TE] Added skip upstream for RemoteQ jobs
* [DK] tabledata/gmaps updates

'''Items to Discuss'''
* [TE] Test run.py [https://www.vistrails.org/ticket/863 #863]
** Changes in run.py are currently not detected by test suite
** Other changes in configuration also don't get tested
** What about importing unittest2 in 2.6?
* [DK] userpackages/packages shadowing [https://www.vistrails.org/ticket/864 #864]
** add the directory
** being able to see both the package that exists in userpackages and packages
** checking loading/unloading
* [DK] need to improve upgrade utility methods
* [RR] Removing zip/unzip executables [https://www.vistrails.org/ticket/862 #862]
** check in the old commit logs and/or trac to figure out what the old problem was
* [RR] Rolling out conda packages: anyone has experience with Anaconda?

=== May 28, 2014 ===

'''Updates'''
* [DK] rewrite-startup close to stable, fixed issue with going back and forth from v2.1 (trying to resolve package config settings issue), means output-modules is close, too

'''Items to Discuss'''
* [DK] extra_info on modules: is it really module-specific info?
* [DK] programmatic package configuration settings without direct imports?
* [TE] job-cache
** New control parameter for caching module outputs to disk
** checks for Constant modules, so any Constant module can be cached
** used for any module to save state of, not necessarily jobs
** is_serializable flag?
** will this be package-specific?
** have an is_pickeable decorator and allow more complex is_serializable interface code
* [TE] LSU feedback (Jinghua and Chris)
** Have used PBS with VisTrails (by running commands over SSH)
** Want to use distributed workflows with vistrails (more than one machine) Could use IPython?
** Using MOAB (Meta scheduler) Module would be useful
** Interested in moving files back and forth
** Have experience with SAGA (Could likely make use of it)

=== May 21, 2014 ===

'''Updates'''
* [DK] output-modules, MplFigureToFile
* [TE] VisTrails 2.1.2 Released (100 downloads so far)
'''Items to Discuss'''
* [DK] persistence across machines (USGS use case)
* [TE] Merge automatic-loops-streaming into master?
** Implicit looping
** Streaming
** ControlParameters
** New schema version 1.0.4
** Possible conflict with multithreaded-interpreter?
* control parameters can also be used for other things like caching?
* [TE] job-cache
** Make module jobs independent of workflow
** Ability to cache jobs for all modules with constant outputs. Set in module/PythonSource/ControlParameter
** Needs branches: controlflow-fake-signatures, jobs-use-signature_/connect-folded-module
* [TE] [vistrails-users] Michele: VisTrails on remote machine
* [TE] [http://saga-project.github.io/saga-python/ SAGA-Python]: A Light-Weight Access Layer for Distributed Computing Infrastructure
** Alternative to support distributed computing in VisTrails?
** Support for most job schedulers
** Could replace BatchQ/SSH/PBS/LSF/HTTP modules
** No Hadoop support
** Open Grid standard
** Compatibility with VisTrails/ModuleSuspended?
* [RR] Debian vistrails package

=== May 14, 2014 ===

'''Updates'''
* CrowdLabs mashups done
* v2.1.2 release ready
* [DK] work on rewrite-startup, make sure preferences work in new style; output-modules
'''Items to Discuss'''
* vagrant for reproducibility
** simpler interface for exploring the experiments
** [RR] generating a VisTrails workflow from reprozip was planned, but not integration of the tracer in VisTrails, to trace VisTrails workflows -- it probably makes sense. See https://github.com/remram44/reprozip-ptrace/issues/8

=== May 7, 2014 ===

'''Updates'''
* CrowdLabs mashups
** Uploading works
** HTML looping works
** Flash removed (actually hidden)
* [DK] output-modules
** Running in batch mode switches to file mode
** Allows parameters to be specified on command-line
** Work on GUI for parameters
'''Items to Discuss'''
* VTK 6
* [DK] output-modules: target single module parameters?
** [RR] use labels to name specific modules, useful in command-line
* scripting support

=== Apr 30, 2014 ===

'''Updates'''
* Does spreadsheet resizing work correctly? Is it too slow? ([https://www.vistrails.org/ticket/833 #833])
** looks like on downsize the image is smaller than it should be and then flickers to flll space (upsize is ok)
* [TE] Mashups on Crowdlabs
** HTML mashups is now rendered in the crowdlabs template
** Added embedding html mashups (using jsonp)
** Fixed uploading mashups with a vistrail
** Investigating issue with saving mashups to DB
* [DK] rewrite-startup, output-modules
** working on command-line and global parameters

'''Items to Discuss'''
* 2.1.2 release
** some major bugs a
* VTK 6
* iPython integration
** how to migrate from scripts to workflows
* Introducing vistrails as a normal python package
** check on anaconda support for Qt, VTK support?
* output-modules
** make sure that filenames can be set
** output directory
** interpreter should return a dictionary with python objects produced (what is input to the output modules?)
** unify OutputPort and OutputModule? Right now, act differently, but function in similar ways
* directions: scripting support, large-scale data analysis, reproducibility

=== Apr 23, 2014 ===

'''Updates'''
* debugger
** have to enable the configuration option, starts python debugger in console
** have similar flag for test suite
* Branches to be merged in: new-url-package
** API calls don't go through upgrades currently, can we support this so we don't break scripts?
** added HTTPS certificate verification
** new dependencies for binaries?
* dont-use-modules-as-data?
** branch is ready?
** changes a lot of packages,

'''Items to Discuss'''
* output-specific modules (e.g. vtkInteractionHandler)
** only need to be enabled when spreadsheet is active, but if we create a workflow with such a module, it would be nice to still be able to run that workflow in batch mode
** not verify entire workflow?
** load module logic on-demand so we don't need dependencies?
* reload packages
** interactions with spreadsheet, stale controller objects
** spreadsheet may need to get information about pipeline when copying cells (get from controller?)
* [TE] server mode
** Changed to non-interactive
** Pass debug messages back?
** Auto-load packages security?
*** startup.xml file gets corrupted?
*** using same xml file for multiple instances
** Locale errors
*** crowdlabs defaults to english, but vistrails server does not.
*** set LANGUAGE will fix it
*** should we set this environment variable in VisTrails
*** decided to chose English for now, can reevaluate when unicode branch is ready
** Default to HTML5 interface?
*** Flash interface as backup
*** look at animations for mashups in HTML5
* module descriptor weakrefs
* VTK6
* continue-dialog branch
** changes SpreadsheetCell modules
** merge into 2.1 for the LSU folks
* output modules: where should the output module live? only one package?

=== Apr 16, 2014 ===

'''Updates'''
* Branches to be merged in: readfile-module, save-module-moves ([https://www.vistrails.org/ticket/853 #853])
* CrowdLabs needs update to VisTrails 2.1 with schema 1.0.3
* [DK] OutputModules (FileOutput)

'''Items to Discuss'''
* LSU meetings
** Pausing workflow/confirmation module ([https://www.vistrails.org/ticket/854 #854])
*** Checking an intermediate result before carrying on with the workflow
*** Their problems: Qt event loop, sinks ordering
** CLTools problem
** crowdLabs usage? (they have 100+GB files)
*** Parameter exploration
** Running mathematica on a cluster (JobSubmission, RemoteQ?)
** Exporting VTK visualization (vti/vtp?) for Kitware's [http://www.kiwiviewer.org/ KiwiViewer]
** Wrapping SNARK09 (C++ code) as VisTrails modules
* eo4vistrails (Derek, Terence)
** Add FTP module?

=== Apr 9, 2014 ===

'''Updates'''
* Branches to be merged in: export-cells, reset-cell-sizes-button (-> v2.1; icon?), export-versiontree-dot & custom-colors, input-module-no-subclass
* [DK] UV-CDAT release, OutputModules (see below)

'''Items to Discuss'''
* In-place spreadsheet updates: [https://www.vistrails.org/ticket/847 #847]
** need to be able to replacement mode toggle on and off
** Colin has a working solution, is there a need to do this globally?
* Autosave for existing vistrails not working <strike>[https://www.vistrails.org/ticket/849 #849]</strike>
* Other tickets from USGS
** Reason for pipeline scroll area being 100 times greater than pipeline?
** File Selection Dialog <strike>[https://www.vistrails.org/ticket/846 #846]</strike>
** Copy/paste with retained connections [https://www.vistrails.org/ticket/851 #851]
** Auto-connect ports when moving modules [https://www.vistrails.org/ticket/852 #852]
*** Also think about fixing some of the port-snapping for selecting a port on the same module when there are multiples of the same type
* Unicode support
** question on mailing list
** [RR} dates is something else, should be able to put on v2.1
** fix for LC on master, port to v2.1?
** should try to persist dates in numeric formats, but GUI level can be language-specific.
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
* [TE] Analogies does not delete annotations [https://www.vistrails.org/ticket/848 #848]
** Code never worked as intended
* [TE] Add eo4vistrails features to vistrails? https://github.com/ict4eo/eo4vistrails
** Look at these and try to determine what may be appropriate
** data types are probably most important, doesn't look like anything there
* [DK] [[OutputModules]]

=== Apr 2, 2014 ===

'''Updates'''
* [TE] Added annotations and control parameters to visual diff
** three tabs
** TODO: analogies
*** just work on making sure the actions execute, don't worry about the matching
* [TE] Build documentation using [http://128.238.102.101:9050/builders/usersguide-master buildbot]
** makefile is run directly so no need to patch (no merges)
** the nightly build still generates the pages, and the cron job now pulls the built pages
** cron job is set to pull built pages after nightly build
* [TE] Updated [http://128.238.102.101:9050/builders/coverage code coverage tester] (currently 33%)
** coverage is done manually, it is not done nightly
** takes 15-minutes to run, on vis-7.
* [DK] Output modules

'''Items to Discuss'''
* Unicode support
** question on mailing list
** fix for LC on master, port to v2.1?
* Output modules
** outputs, modes, mode configurations
** spreadsheet as a package
** output mode configuration dependencies
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
** trying to complete output modules before new VTK package

=== Mar 26, 2014 ===

'''Updates'''
* Branches to be merged in: application-no-default-argv, export-cells, testsuite-module-imports, reset-cell-sizes-button (icon?), export-versiontree-dot, custom-colors
* [TE] Added branch control-parameters (used by list handling)
** For setting hashed control options on modules (How to combine lists, while looping, caching)
** Missing: Include in diff/merges/analogies?

'''Items to Discuss'''
* iPython
** script to workflow bridge
** access to VisTrails commands in iPython
** changes to the API
* Multi-faceted output
* [RR] Documentation buildbot
** [TE] currently [https://www.vistrails.org/#Testing here]
** problem is that we don't get a message when the documentation failed to build
** what about copying the output from a build machine to vistrails.org?
* [RR] Last(?) migration issue: [https://www.vistrails.org/ticket/837 #837]
** [DK] to look at
* VTK 6 support [https://www.vistrails.org/ticket/739 #739]
* spreadsheet export button in UV-CDAT
* Develop menu?

=== Mar 19, 2014 ===

'''Updates'''
* [DK] UV-CDAT #407 fix, version tree rewrites

'''Items to Discuss'''
* [TE] Cleaning up test suite log
** Deprecated core.modules.constant_configuration imports
*** check whether the deprecations for vistrails.core.vistrails_module.VistrailsModule camel-case are similar
** KeyError (Ticket [https://www.vistrails.org/ticket/837 #837]), one of [DK]'s commits
*** [DK] to look at
** Skip modules not to be imported (or fix/remove)
** Python invalid drawable (Probably from mashups)
** Missing version 0.8 of package org.vistrails.vistrails.tests.upgrade (bug?)
* [TE] Add annotations to signature? (hash-annotations)
** for the streaming/automatic looping for new values on the modules
** could be any module
** label doesn't affect caching
** prefix, flag involved, add fields to module about looping
** schema change? problem is that don't want to have to recreate module each time
** backward compatibility
** new type of "control parameter"
* VTK 6?
** See Ticket [https://www.vistrails.org/ticket/739 #739]
* UV-CDAT #408: [RR] has worked on this
** support adding more formats to export
** decided to have single menu item, and put all available formats in the dialog
* Spreadsheet resizing bug
** if you resized the sheet, would reset to equal-size cells
** spreadsheet-resizing branch changed this to allow non-equal-size cells on resize
** has some math issues because doing this dynamically difficult
** button to resize to equal-size on branch
** updating the documentation
** [TE] can we store the locations more precisely? (problem might be if a header column is also resized)

=== Mar 13, 2014 ===

'''Updates'''
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829]); fixed(?), merge?
** [DK] Yes, merge
* [RR] Will merge in tabledata additions
* [RR] New SQL package, using tabledata and sqlalchemy. Should be fully backward compatible(?)
** Check if Fernando has anything here from the Oracle workflow and database provenance
* [RR] Searching in vistrail and regex ([https://www.vistrails.org/ticket/373#comment:7 #373], [https://www.vistrails.org/ticket/779 779]); fixed?
** Error was hidden (entering '?' searched for '.*?.*' which was a valid regex but still didn't do what was expected)
* [TE] IPython console now shows <STANDARD OUTPUT> before printing from stdout/stderr
* [TE] Vistrail variables fixed
** Deleting a module with a variable did not remove the variable from the workflow
** Copying modules with variables now works

'''Items to Discuss'''
* [RR] Do not run modules upstream of submitted job
** jobs-use-signatures: Uses the subpipeline signature to identify jobs
** works: if job is submitted, check it without running upstream (we don't need the parameters this time(?))
** doesn't work in Map
** jobs-use-signatures_/connect-folded-module
*** create a connection from folded module to InputList upstream
*** looped module can run upstream if it needs, else we don't
*** problem: the Map still needs the list, for length and type-checking
** Could we rearchitect the looping behavior? Basically, just copy the pipelines and run them with each entry from the loop
*** Provenance could expand
* [RR] module-upgrade-recursion status? (not yet merged)
** [DK] to look at and merge (should be done)
* [DK] UV-CDAT Issues: [https://github.com/UV-CDAT/uvcdat/issues/407 #407] and [https://github.com/UV-CDAT/uvcdat/issues/408 #408]
* [RR] Thumbnail comparison bug: different references are selected on master and dont-use-modules-as-data! (pipeline results are identical) Problem with upgrade code?
** using different reference images
** [TE] to look at
** look at the console_mode, how it gets back the upgraded version id
** filed as [https://www.vistrails.org/ticket/831 #831]
* [TE] Vistrail variables
** Normally only one hidden module exist for each vistrail variable, but pasting a module with a vistrail variable will create a new hidden module. This does not break anything, but should be taken into account when modifying the code.
** When a vistrail variable module is created, an input named "value" is created. A bug with default values gave it the value "None". This is fixed. To make validation of existing vistrail variables work we need to check if the value is "None", but this would not allow "None" as a string.
** [DK] to look into this, we cannot have "None" there.
* [JF] NSF Big Data proposals (due in June)
* uuid status
** need to check tests and features outside of normal execution

=== Mar 5, 2014 ===

'''Updates'''

'''Items to Discuss'''
* [RR] New SQL package, using tabledata and sqlalchemy
** No upgrade possible? (output changed from List (of rows) to Table)
*** Should we make it a new package?
*** Will try to make backwards compatible
* [TE] Change pinning to always show pinned panels? ([https://www.vistrails.org/ticket/828 #828])
** Yes, it is a better behavior
* [RR] Package dependency ordering issue ([https://www.vistrails.org/ticket/829 #829])
** Will try to fix
* [RR] Merging multithreaded-interpreter and JobManager
** Need examples of JobSubmission
* [TE] Merging automatic-loops-streaming
** Try to merge with multithreaded-interpreter
* Wiki menu looks strange in Chrome
** [TE] Will try to fix

=== Feb 26, 2014 ===

'''Updates'''
* Branch to be merged in: hidden-port-icon (v2.1)
* [RR] tabledata-merge-tabdata: added tests, fixed issues (todo: json)
* [TE] Added package RemoteQ (BatchQ/PBS/Hadoop)
** new Mac binary that includes everything here
* [DK] gmaps, climate proposal

'''Items to Discuss'''
* Climate proposal
** recommendation and best practices
** automatically segmenting the version tree, better modes of interaction
* [RR] tabledata
** selections, projections, joins
** writing out table as csv?
** much improved bugs
** [DK] should merge aggregate module int
** [RR] working on different json representations
* [TE] Hadoop configuration
** has package configuration now
** still have machine module to hardcode a configuration for a workflow
** http://www.vistrails.org/index.php/Hadoop_Package
* Data cleaning/repo:
** always need the vistrails engine to run things
** scripting interface

=== Feb 19, 2014 ===

'''Updates'''
* [TE] automatic-loops-streaming works
* [TE] Hadoop package:
** Merged BatchQ-PBS and hadoop into RemoteQ
* [DK] improvements to tabledata, gmaps
* [RR] java-pkg is done

'''Items to Discuss'''
* [TE] automatic-loops-streaming
** Add documentation
** Perhaps a matplotlib example where the plot updates as more data streams
** Do we want to add additional controlflow features?
*** Implemented while module, but this needs to be improved because of annotations
*** annotations don't invalidate cache
*** could include annotations in hasher for cache updates
*** how does if module work?
** [RR] pairwise for three ports?
*** the list modules do this, but we may need an interface for this
** http://dev.mygrid.org.uk/wiki/display/taverna/List+handlings
* [TE] Hadoop package:
** Best way to separate task and configuration parameters?
** URI is different for different clusters
** use package configuration variables
** move from checkbox to allow more general configuration?
** use configuration file for this?
* [RR] working to merge multithreading code with job-monitor, look at streaming
* [RR] java-pkg
** works for some classes (call constructors, setters, getters)
** cannot call any methods from VisTrails, don't want to mutate
** can call these methods from your own modules, also insert code in Python into these packages
** may add in an interface for loading jar files and wrapping them

=== Feb 11, 2014 ===

'''Updates'''
* [TE] automatic-loops-streaming - partially works
** Using a python generator for each streaming module
** TODO: Streaming Sinks
*** last module collecting values
** Logging/Feedback performance
** Demo?
* [DK] vtk-new-package
* [RR] java package: conflicts with dlls and jpype, going to work

'''Items to Discuss'''
* [TE] Hadoop package:
** a few installation issues, vistrails-hadoop pkg on git but was not accessible to public
** same package for local cluster and aws? Yes, but need to change two lines, path to configuration files on server
** could config files be set in a module instead or in the preferences?
** directories are hardcoded in hadoop package, need to be parameters
** anything that can change in different installations should be parameters
** can be hard to figure out what does change
* [DK] merge tabledata-merge-tabdata, matplotlib-add-helpers?
** examples are not very useful, no example with multiple plotline modules?
** Update matplotlib examples to better show off capabilities
* [DK] package loading speed, kwargs
** kwargs can be significantly slower
* java issues:
** [RR] should work, just a few installation issues
** machine learning in python: scikit-learn (http://scikit-learn.org/stable/)

=== Feb 5, 2014 ===

'''Updates'''
* [TE] automatic-loops
** PortSpec-based version now works
*** Connections show how modules will be iterated
** Streaming is tricky
* [DK] Examples, merging tabledata

'''Items to Discuss'''
* mailing list question
** no resopnse yet
* UV-CDAT: warnings when downsizing spreadsheet
** problem: warnings will pop up a lot
** keep offscreen cache of (x,y) locations in spreadsheet?
** cache pipeline instead of full rendered pipeline
* [TE] streaming with branches becomes interesting
** only want to update upstream branches once
** Taverna has streaming-style (pipelining)
* PythonSource deleting port?
* [RR] Java: possibly considering the matplotlib-style port creation versus java-style ports

=== Jan 29, 2014 ===

'''Updates'''
* [TE] automatic-loops - Groups and Multi-level looping now works
* [DK] matplotlib-add-helpers: Rework the semi-automatic generators
* [RR] java-pkg: bringing in old code, spreadsheet still works, probably going for dynamic generation of packages
** looking at jpype versus pyjnius

'''Items to Discuss'''
* [RR] str-format-module: in master. Merge in v2.1? Having to open the configuration window and click on a button is awkward
** [DK] watch out for dynamically created format strings, could cause inconsistency at compute time if the format string doesn't match the ports
** Go ahead to merge since similar problem with PythonSource
** What about using number of connections to enforce this? (set to 0 for config-style ports?)
* Streaming in VisTrails
** Q: how is parallelism working in VisTrails? A: both theading and multiprocessing, but separate branch
** for streaming, order of input and outut must be preserved
** module can generate a number of pieces, a module that collects needs to know when the stream ends
** need to preserve order of map operations and
** goal is to reduce the storage costs, want to make sure that we don't keep entire data in memory, don't generate the entire list at beginning
** hyperflow needs to know when a stream starts and when it ends: start and end token, hyperflow wraps data in a token
** Huy advises getting rid of the list and using tokens instead?
** interpreter can call compute-upstream and just tell interpreter to go upstream
** VTK streaming interpreter: might be in svn
** Remi: just be able to pass an object on the conection that knows the shape of the data and how to get next element (basically an iterator)
*** where do we call get-next? module in the collector calls
** main advantage of streaming is being able to do aggregation during the compute (otherwise, we're doing map-reduce)
** the iterator encapsulates the calls to the module's compute
** issue with logging? already rewritten for parallel, should work here
** Examples: triangle_area, image histograms
** Slowdown caused by lists? colors? logging?
** Schema changes: need to record depth, what type of semantics for combining port values
** Also probably means that we don't pass just raw values, need to wrap with depth and iterator
** Issues Huy raised:
*** issue with multithreading and streaming both enabled
*** triangle with modules: need to be careful about order with items coming from two different modules (e.g. BuildTable in Rémi's example)
* UV-CDAT

=== Jan 22, 2014 ===

'''Updates'''
* [RR] Branches to be merged in: reload-disabled-package ([https://www.vistrails.org/ticket/714 #714]), ungroup-keep-disconnected-ports_/hybrid
* [TE] VisTrails 2.1.1 released
* [DK] matplotlib, uuid work

'''Items to Discuss'''
* [TE] Automatic Looping
** Using ListOf module as input will trigger the looping
*** Runtime type checking because we don't have a ListOfT class for each type.
*** [DK] USGS interested in working with this
*** Add a merge module to make it clear where the iteration stops and ends
*** Have different type of ports
*** Remi to talk to Huy about streaming ideas
*** can we detect the iteration and draw those modules differently
*** decided that some fold/merge module would be useful, even if we don't always need to use it explicitly
* [RR] Ticking in matplotlib
** formatters, locators, tick scale
* [RR] Sort tickets? There are [https://www.vistrails.org/report/18 issues for the 2.0 milestone] and [https://www.vistrails.org/report/17 issues that affect released versions]. Tickets assigned to 2.1 are [https://www.vistrails.org/report/19 here].
* [DK] To contact USGS about the new persistence package and get feedback
* [DK] Merge weather and subway examples to use new tabledata package
* Look into pandas http://pandas.pydata.org/
** could be helpful to support some of the table operations
** in the future, we may want to support it for data analysis and stats (but for this, additional dependencies are required)

=== Jan 15, 2014 ===

'''Updates'''
* [RR] improves-logging:
** Pass exceptions to it directly for single line, or traceback.format_exc() for traceback
** WARNING level now printed in the console by default, -V 1: INFO (log() calls), -V 2: DEBUG (debug() calls)
** Messages view now shows whatever is selected, no matter the console log level
** Python warnings get captured
** Use warnings.warn(..., category=VistrailsWarning|VistrailsDeprecation) to warn once
** Backport to v2.1? (so deprecation warnings actually show up)
*** -> Yes, include in v2.1.1
* [TE] Hadoop package documented here: http://www.vistrails.org/index.php/Hadoop_Package
* [DK] UV-CDAT, LSU

'''Items to Discuss'''
* [RR] input-module-no-subclass: input ports of type 'Module' shouldn't accept any output? These are used for 'self'->controlflow module connections...
** controlflow modules have a function port that accepts a module
** check for modules that might use Module for something else
** suggestion is to use Variant for any type being supported (StandardOutput, List, others)
** warns for now (doesn't actually disallow the connection until next version)
** -> David wants to look at this more
* [RR] ungroup-keep-disconnected-ports: alternative ungrouping, keeps pipeline disconnected and preserves InputPort/OutputPort modules
** ideally, want to edit in place, could try to emulate the edit subworkflow code here, just have to save things back to the original vistrail
** -> hybrid approach, materialize the unconnected ports
* Bugfix release? Would solve issues with server and parameter exploration. (also, logging improvement?)
** [TE] Also fixes to workspace
* [DK] font issue for OS X 10.9
* Look at Java support in Java?
** can we leverage Rémi's past work here? on Weka
** pyjnius: cannot subclass a Java class in Python, should be able to create and call out to java classes
** issue with Jython is the interface
** also had java-based spreadsheet
* also look at moving VTK to a matplotlib style generation

=== Jan 8, 2014 ===

'''Updates'''
* [TE] Hadoop package: Added -combine option for [http://www.vistrails.org/index.php/Cs9223_Mapreduce_Assignment Cs9223_Mapreduce_Assignment]
** tricky to set up the package still, need documentation to understand how to assemble workflows from scratch
* [RR] Logging improvements underway (branch 'improves-logging')
** Fixes some logging gotchas
** Better exception printing (pass them directly to critical|warning|log)
** WARNING is now the default level for the console (only CRITICAL displays a popup)
* [RR] Alternative Mac .app ([https://docs.google.com/uc?id=0B3ucPz7GSthBZG5TS2pXUVdUWEk&export=download example dmg])
** Standard build of Python (currently as a Mac .framework)
** Have to relink libraries with install_name_tool
** Seems to work
** Advantages: standard, full build of Python; pip will work
** Last problem: if Qt is installed, it gets loaded and conflicts with bundled Qt (I missed a dylib link)
* [DK] Work with USGS, refactoring version trees

'''Items to Discuss'''
* Colin's feedback on loops is that it has taken some time to wrap his head around what is going on
** what does Taverna do? automatically determines when a collection is being input and does iteration over the input and determines what output looks like
** look at what this would take, figure out what the best way to attack this ([[Automatic loops]])
* Colin also asked about use of persistence in another module

=== Jan 3, 2014 ===

'''Updates'''

* [TE] VisTrails 2.1 released
* [DK] UUID branch, USGS work, upgrade recursion
** van Wijk on tree vis, and Buneman on XML Updates
'''Items to Discuss'''

Development/2014 - Revision history

Remi at 15:20, 4 February 2015

Remi: Moved from Development